evaluating quant research

50 %
50 %
Information about evaluating quant research
Entertainment

Published on August 20, 2007

Author: Roxie

Source: authorstream.com

Slide1:  Reviewer’s Secrets Revealed! James R. Van Scotter, Ph.D., ISDS Department, Louisiana State University About This Presentation:  About This Presentation Is it about learning how to review papers? About This Presentation:  About This Presentation ..or, is it about how reviewers evaluate your papers? About This Presentation:  About This Presentation …or, both! Basic Questions for Reviewers:  Should this paper be published? (Why or why not?) Does it contribute to scientific knowledge? (How?) Can the paper be strengthened? Basic Questions for Reviewers Criteria:  Schwab’s (1985) Criteria 1. Technical merit Internal validity, construct validity, causal issues. 2. Craftsmanship Organization, clarity 3. Significance or importance Tests a reasonable theory in non-trivial way. Improves an existing body of empirical research. Has implications for an important policy issue. Criteria Source: Schwab, D. (1985). Reviewing empirically-based manuscripts: Perspectives on Process. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 171-181. Criteria:  Why I Recommended Rejecting Your Manuscript… No theory Theory should explain what the variables mean, how they are related and why they are expected to be related to one another. Concepts and operationalization not aligned Variables do not reflect the concepts or the logical links have not been developed. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Why I Recommended Rejecting Your Manuscript… 3. Insufficient definition - Theory Concepts are not defined or justified. It is not clear why variables were included. Some were included simply because they had appeared in other studies. 4. Insufficient rationale – Design Lack of explanation for study procedures, description of sample, etc. Nothing is obvious or assumed. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Why I Recommended Rejecting Your Manuscript… 5. Insufficient research design Usually fatal. Using survey questions to study subtle processes. Using undergrad sample for strategy research (CEO, CIO level). 6. Lack of organization and flow No single story line. Mismatch between tests and hypotheses. 7. Amateurish style and tone Overly negative approach to previous literature. Exaggerated sense of importance. Logical gaps in writing. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Leading Reasons for Rejection by JIS Lack of theory Weakness in Design Poor Motivation for the Study Other Reasons for Rejection Lack of Fit with Journal Insufficient Development Insufficient Contribution Methodological Problems Criteria Source: Murthy, U. andamp; Wiggins, C., (2002). Why manuscripts are rejected: An analysis of JIS rejections. Journal of Information Systems, 16, 1, 41-48. Review Process:  Peer review is a complex social act. All referees are not experts in the areas they review. Recommendations are based on the editor’s or reviewer’s personal preferences. 3. Reviewers’ recommendations may conflict. 4. When recommendations become too detailed, reviewers become ghost writers. 5. Nobel prize winning research has been rejected. Review Process Source: Bedeian, A. (2004). Peer Review and the social construction of knowledge In the management discipline. Academy of Management Learning and Education, 3, 2, 198-216. Review Process:  Survey of Reviewer Practices Typical reviewer spends 5.4 hours per paper. They reviewed 1.5 papers per month. Two dominant roles: Critic and Coach. Review Process Source: Jauch, L. andamp; Wall, J. (1989). What do they do when they get your manuscript: A survey of Academy of Management Reviewer Practices. Academy of Management Journal, 32, 1, 157-173. Process:  Developmental Reviewing More than just avoiding a negative tone. Two main ingredients: Helping the author(s) understand what is wrong. Helping the author(s) figure out how to fix it. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. Process:  Developmental Reviewing 3. Procedure: Start with overall evaluation. Describe aspects of the paper that are strong/weak. Identify major and minor issues. Suggest solutions for methods problems. Suggest areas that should be cut or expanded. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. Process:  Developmental Reviewing 4. Specific recommendations. No sample is perfect. Help the author(s) make the best use of the data. b. Give examples of improvements (e.g., clearer hypotheses) you would like to see. If appropriate, provide references for papers with an alternative perspective. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. What to Do:  What to Do What to Do:  What to Do 'Science depends on qualitative, common-sense knowing … (Campbell, 1979, p.50)' Campbell, D. T. (1979). Degrees of freedom and the case study, in T. D. Cook, andamp; C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research, Beverly Hills, CA: Sage Publications, pp.49-67. Influential Articles:  Analysis of Influential Articles 1. Obvious contribution to knowledge 2. Results of major theoretical significance 3. Useful new theory/theoretical framework 4. Generates new research 5. Provides new and exciting ideas 6. Integrates several areas 7. Integrates existing theories into a simpler framework Influential Articles Source: Sternberg, R. andamp; Gordeeva, T. (1996). The anatomy of impact: What Makes an article influential. Psychological Science, 7, 2, 69-75. Slide19:  A research paper should be like a present. The general concept should be appealing, the contents should be valuable, and everything necessary should be wrapped up in the package. Six General Goals of Science:  Six General Goals of Science Organize andamp; categorize things (typologies) Explain past events Predict future events Control future events Provide a sense of understanding Generalize results Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?:  How does science accomplish its goals? Formulating hypotheses andamp; descriptive statements that: Are derived from theories Are falsifiable Can be tested empirically Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?:  2. Developing models that summarize important attributes/relationships Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  3. Collecting data Manipulating relevant variables Measuring relevant variables Controlling and/or measuring nuisance variables Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  4. Using empirical data to test the hypotheses Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  5. Generalizing research results to a larger context Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? General Principles:  General Principles General Principles:  General Principles Research as story telling and good writing. 2. Favor simple explanations (parsimony). 3. Use models to summarize important relationships. Portions adapted from: Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers. General Principles:  General Principles Collecting data to test models/hypotheses. 5. Using comparison as the evaluation framework. 6. Ruling out alternative explanations. Portions adapted from: Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers. Story Telling:  Story Telling From general to specific From specific to general Hour glass shape Story Telling:  Story Telling Introduction Literature Review Methods Results Discussion Conclusion andamp; Recommendations Story Telling:  1. Why is the topic important and interesting? Theoretical andamp; practical value Appropriateness for journal Timeliness Novelty Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  2. Literature review Link to important literature Logic for hypotheses andamp; propositions Context for study (samples/methods) Use literature to develop arguments Thorough exposition Accurate interpretation (including nuances) c. Lead to a conclusion about what is needed d. Establish incremental value of the paper Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  3. Conceptual development Adequacy andamp; scope of research hypotheses Clarity and logical coherence (one main story line) Explains how things work without leaps of faith or logical gaps 4. Have a point (not just a point of view) Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Parsimony:  Parsimony Simple explanations are better. Simple explanations are more likely to be testable. Parsimony:  Parsimony Parsimony is related to goals of scientific research Simplifying complex issues Separating important from unimportant factors Generalizing across time, place, samples Providing a benchmark Slide36:  Theoretical statement goes here Theoretical statements should be concise enough to fit on a bumper sticker. …and simple enough that your grandmother could understand them. Models:  Models Simplify communication of complex relationships Consistent with idea that measures of the same variable are interchangeable Encourage thinking about alternative explanations Models:  Models 4. Testing process must be guided by theory, not the modeling method. 5. All 6 models above have the same fit (Bollen, 1989, p.70) Methods:  Methods Sample design 1. Representativeness andamp; generalizability Appropriateness for research question Demographics for subjects Research setting andamp; context 2. Suitability for tests andamp; inference (statistical assumptions) Sampling method (random, purposive, convenience) Comparison with population, or samples in related studies Sufficiency (sample size, response rate, attrition) Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement Justify constructs Purpose Theory andamp; Prior research Operationalization of constructs Use well known instruments when possible Use archival data when appropriate Use multiple sources of measures when possible Avoid single item measures Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement 3. Distinguish between measures of perceptions, intentions versus behaviors and outcomes. 4. Avoid criterion contamination or assess andamp; report it. 5. Use appropriate level of analysis; avoid ecological fallacy. Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement 6. Provide appropriate reliability evidence. Internal consistency Inter-rater Test-retest, Alternate forms Average Variance Extracted 7. Provide appropriate validity evidence. Convergent andamp; discriminant Construct Content Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design 1. Use pilot studies Test manipulations/treatments and instruments Test sampling methods and procedures 2. Use standard administration methods to avoid biasing results 3. Use ethical procedures; comply with rules for treatment of human subjects Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design (cross-sectional studies) 4. Use designs appropriate to the hypotheses Tests of alternative models/explanations Differential predictions NOT the general null hypothesis 5. Control for statistical and sampling artifacts Restriction in range Common method bias Uses appropriate control variables Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design (experimental/quasi-experimental) 6. Use random assignment procedures; provide information on comparability of groups. 7. Use adequate manipulations/perform checks. 8. Avoid threats to validity. Internal andamp; external threats Experimenter bias Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Framework for Evaluation:  Framework for Evaluation Empirical research tests relationships between observed variables Framework for Evaluation:  Framework for Evaluation Theories focus on relationships between Latent (unobservable) variables Framework for Evaluation:  Framework for Evaluation Latent Independent Variable Latent Dependent Variable Observed Independent Variable Observed Dependent Variable What we want to know What we can test Construct validity Construct validity Statistical Conclusion Validity Statistical Conclusion Validity Construct validity Methods:  Methods Comparative Evaluation 1. Tests of different groups Natural categories Treatment/control groups; multiple treatment groups 2. Tests of alternative hypotheses Competing explanations Nested models Goal is to find evidence that contradicts one explanation and supports another Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Data Analysis Use appropriate statistics Demonstrate awareness of major assumptions Conduct supplemental analyses suggested by the study 2. Avoid capitalization on chance Examine outliers as needed Control experiment-wise error rate 3. Report effect sizes, confidence intervals, significance levels Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Ruling Out Alternatives:  Ruling Out Alternatives Some sources of errors Design Sample Instruments Experimenter andamp; situational errors Demand artifacts Results:  Results Summarize results briefly Link findings to hypotheses and interpret them Relate findings to other studies Derive specific theoretical and practical implications Provide suggestions for future research Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  Story Telling Writing quality, simple sentences, proper grammar, active voice Logical organization andamp; sense of direction 3. Succinct and shows attention to detail 4. Tell a consistent, integrated story Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  Why should this paper NOT be published? Does the theoretical development make sense? Do the authors explain why AB? Are there any logical gaps or leaps of faith? Story Telling Story Telling:  Why should this paper NOT be published? Do the authors test what they said they would? Are the data and methods appropriate? What are the theoretical implications? What does this paper add to the literature? Story Telling Additional:  Journals with 75% or more articles by IS Authors Journal Journal Pct Ranking MIS Quarterly .96 A Information Systems Research .95 A Journal of Management Information Systems .92 A Communications of the AIS .91 B The Data Base for Advances in Info Systems .84 A International Journal of Electronic Commerce .80 A Journal of the AIS .80 B Journal of Global Information Management .76 ? Additional Source: Chua, C., Cao, L., Cousins, K., andamp; Straub, D. (2002). Measuring researcher production in information systems. Journal of the AIS, 3, 145-215. Publish or Perish:  The 7 P’s Proper Prior Planning Prevents Poor Publication Performance Publish or Perish Source: Unknown Additional References:  Additional References Bhattacharjee, S., Tung, Y., andamp; Pathak, B. (2004). Author experiences with the IS journal review process. Communications of the AIS, 13, 629-653. Mingers, J. (2002). The long and winding road: Getting papers published In top journals. Communications of the AIS, 8, 330-339. Slide59:  The End

Add a comment

Comments

scarpe-hogan.it | 19/06/15
Whoa! This blog looks exactly like my old one! It's on a entirely different topic but it has pretty much the same page layout and design. Outstanding choice of colors! scarpe-hogan.it http://www.scarpe-hogan.it
tiffany and co jewelry | 21/07/15
I really like your blog! Just wanted to write something. [url=http://www.tiffanyjewelrysuk.org]tiffany and co jewelry[/url]

Related presentations

Related pages

Evaluating Quantitative Research - aea267.k12.ia.us

Evaluating Quantitative Research Quantitative research articles should include the following sections: an abstract, literature review, method, results, and ...
Read more

Measuring research: A guide to research evaluation ...

Measuring research. A guide to research evaluation frameworks and tools. by Susan Guthrie, Watu Wamae, Stephanie Diepeveen, Steven Wooding, Jonathan Grant.
Read more

Evaluation Tool for Quantitative Research Studies - USIR

Evaluation Tool for Quantitative Research Studies Evaluation Tool for Quantitative Research Studies Prof Andrew Long, School of Healthcare, University of Leeds
Read more

The Strengths and Weaknesses of Research Methodology ...

The Strengths and Weaknesses of Research Methodology: Comparison and Complimentary between Qualitative and Quantitative Approaches Looi Theam Choy
Read more

Educational Research: Planning, Conducting, and Evaluating ...

Buy Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research (4th Edition) on Amazon.com FREE SHIPPING on qualified ...
Read more

Criteria Qualitative Research Quantitative Research

Criteria Qualitative Research Quantitative Research Purpose To understand & interpret social interactions. To test hypotheses, look at cause & effect,
Read more

ResearchGate - Share and discover research

Find over 100+ million publications, 11+ million researchers and 1 million answers to research questions. ResearchGate is a network dedicated to science ...
Read more

8 Integrating Qualitative and Quantitative Approaches in ...

Integrating Qualitative and Quantitative Approaches ... opment research and program evaluation ... hensive mixed-method strategy for evaluating programs ...
Read more

Introduction to Mixed Methods in Impact Evaluation

Introduction 1 Part I. Why Mixed Methods? 3 1.1. What is a mixed methods impact evaluation design? 3 1.2. The limitations of an exclusive reliance on QUANT ...
Read more