ReviewA scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews
Introduction
The use of mixed methods research increases in health sciences (Creswell et al., 2004). Mixed methods research is defined as a combination of qualitative and quantitative methods conducted by a researcher or researcher team, for the broad purpose of gaining breadth and depth of understanding or corroboration, within a single study or closely related studies (Johnson et al., 2007). For example, mixed methods research may combine a quantitative cross-sectional survey on the accessibility of mental healthcare with a qualitative ethnographic study to better understand the lack of access in a culturally diverse community (Groleau et al., 2007). While combining qualitative and quantitative methods constitutes a longstanding practice in evaluation and research (Greene, 2006, Pluye et al., 2009b), it has only recently been conceptualized in terms of mixed methods research (Tashakkori and Teddlie, 2003).
The division of methods within health sciences as qualitative or quantitative has its roots in the different ‘world views’ of constructivism and logical empiricism, which are usually presented as competing paradigms (Creswell and Plano Clark, 2007, Greene, 2007, Johnson et al., 2007, Pluye et al., 2009b, Teddlie and Tashakkori, 2003). Constructivism is associated with idealism, relativism and (inter)subjectivity, while logical empiricism is associated with materialism, realism and objectivity. Constructivism is most frequently associated with inductive qualitative studies, and logical empiricism is most frequently associated with deductive quantitative studies. Mixed methods may be conceived as methods that loop between constructivism and logical empiricism, and include the notion that something can be “both socially constructed and yet real” (Hacking, 1999, p. 119).
There is no consensus on criteria for appraising the methodological quality of mixed methods research (O’Cathain et al., 2008). While there are general criteria for planning, designing, reporting and globally assessing mixed methods research (Creswell and Plano Clark, 2007, O’Cathain et al., 2008), researchers must refine further the criteria for evaluating its quality (Creswell et al., 2004). In the present paper, we examine systematic Mixed Studies Reviews, and criteria for appraising the quality of qualitative, quantitative and mixed methods studies.
A mixed studies review (MSR) is a literature review that concomitantly examines qualitative, quantitative and mixed methods primary studies. We found examples of MSR in education, health, management and social sciences such as reviews to synthesize existing knowledge about an intervention or a program that is to be evaluated. We conceptualize MSR as mixed methods research where data consist of the text of publications reporting qualitative and quantitative studies and/or mixed methods studies.
In the present paper, we review MSRs in health sciences, define three categories of MSRs (convenience, reproducible and systematic), and provide guidance on processes that should be included and reported. Our main objective is to critically scrutinize the different ‘quality appraisal tools’ used in systematic MSRs, and propose a scoring system for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies in a systematic MSR. This scoring system may also be used to appraise the methodological quality of qualitative, quantitative and mixed methods components of mixed methods research.
Section snippets
Mixed studies review as an emerging form of literature review in health sciences
We use the term MSR to refer to reviews with many different names such as integrative review, meta-needs assessment, mixed approaches to evidence synthesis, mixed methods review, mixed methods synthesis, mixed research synthesis, and realist review. These terms and corresponding references are presented in Table 1. This profusion of terminology is testament to the significant desire for better comprehension and conceptualization of MSR. We suggest the term ‘Mixed Studies Review’ as a generic
Data sources and sampling
We identified and examined 59 MSRs indexed in MEDLINE up to March 2006.
Description of the sample
Of the 59 retained health-related MSRs, 56 (95%) were published after 2000. MSRs were conducted in the following disciplines: nursing (24%), psychosocial and behavioural research (19%), health services and policy research (14%), population health (8%) and aging research (8%). First authors were affiliated with US and UK universities for 23 (39%) and 17 (29%) reviews, respectively. Other first authors’ affiliation countries were Netherlands (N = 5), Sweden (n = 4), Canada (n = 3), Finland (n = 2),
Discussion
Results support the usability of the proposed 15 characteristics for concomitantly appraising the methodological quality of qualitative, quantitative and mixed methods studies in a SMSR context. The content validity of this initial set of criteria is supported by the literature and our review of SMSRs in health sciences. This minimum set may decrease the burden on reviewers. A revised set of criteria is presented in the next Section 5.1.
In our sample, the fact that only two of 17 SMSRs (12%)
Conclusion
The rationale for conducting mixed methods research lies in combining the strengths of both qualitative and quantitative studies. Typically, the former studies provide in-depth descriptions of complex phenomena that are context-specific, but may suggest theoretical and methodological lessons transferable to other contexts, while the latter examine observations or causal relationships that may be generalized using statistical inferences. In the present paper, we define MSR as a form of mixed
Conflict of interest
None.
Acknowledgements
Pierre Pluye and Marie-Pierre Gagnon hold a New Investigator Award from the Canadian Institutes of Health Research. This work was also supported by the “Fonds de recherche en santé du Québec”. Authors gratefully acknowledge the suggestions of Bryce Mansell, Laura Shea, Robbyn Seller and Tara Bambrick. Preliminary versions of this paper have been presented at the Conference of Qualitative Inquiry (Urbana-Champaign, 2006), the Canadian Cochrane Symposium (Ottawa, 2007), and the Mixed Methods
References (64)
- et al.
Mixed-method evaluation: developing quality criteria through concept mapping
Evaluation Practice
(1994) Meta-needs assessment
Evaluation and Program Planning
(2000)- et al.
Assessing the quality of reports of randomized clinical trials: is blinding necessary?
Controlled Clinical Trials
(1996) - et al.
Research synthesis reviews: an illustrated critique of “hidden” judgments, choices, and compromises
Clinical Psychology Review
(1987) - et al.
Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed methods studies
International Journal of Medical Informatics
(2005) - et al.
Comparability work and the management of difference in research synthesis studies
Social Science Medicine
(2007) - et al.
Scoping studies: towards a methodological framework
International Journal of Social Research Methodology
(2005) Doing A Literature Review in Health and Social Care: A Practical Guide
(2007)- et al.
Using meta ethnography to synthesise qualitative research: a worked example
Journal of Health Services Research and Policy
(2002) Integrating quantitative and qualitative research: how is it done?
Qualitative Research
(2006)
Data analysis strategies for mixed-method evaluation designs
Educational Evaluation and Policy Analysis
Reliability and Validity Assessment
What is This Thing Called Science?
A brief history of research synthesis
Evaluation and The Health Professions
Cochrane Reviewers’ Handbook 4.1.6 [Updated January 2003]
Synthesizing Research: A Guide for Literature Reviews
The Conduct of Systematic Research Reviews for SCI Knowledge Reviews
Designing and Conducting Mixed Methods Research
Designing a mixed methods study in primary care
Annals of Family Medicine
Including qualitative research in systematic reviews: opportunities and problems
Journal of Evaluation in Clinical Practice
Synthesising qualitative and quantitative evidence: a review of possible methods
Journal of Health Services Research and Policy
Conducting Research Literature Reviews: From the Internet to Paper
Methodological strategies for the identification and synthesis of ‘evidence’ to support decision-making in relation to complex healthcare systems and practices
Nursing Inquiry
Writing Literature Reviews: A Guide for Students of the Social and Behavioral Sciences
Health Sciences Literature Review Made Easy: The Matrix Method
Toward a methodology of mixed methods social inquiry
Research in the Schools
Mixed Methods in Social Inquiry
A mixed methods approach to the cultural understanding of distress and the non-use of mental health services
Journal of Mental Health
The Social Construction of What?
Applying systematic review methods to studies of people's views: an example from public health research
Journal of Epidemiology Community Health
Methodological issues in combining diverse study types in systematic reviews
International Journal of Social Research Methodology
Cited by (727)
Facilitators and constraints to adult sports participation: A systematic review
2024, Psychology of Sport and ExerciseSystematic Review of Pain Research Among Limited English Proficiency Patient Populations in Health care
2024, Pain Management NursingTeaching Strategies for Professional Identity Education in Pharmacy: A Scoping Review
2024, American Journal of Pharmaceutical EducationCharacterization and critical appraisal of physiotherapy intervention research in Nigeria: a systematic review
2024, BMC Musculoskeletal DisordersNarrative medicine interventions for oncology clinicians: a systematic review
2024, Supportive Care in Cancer