Assessing Mixed Methods Research in Mental Health

We conducted a literature review of mental health services research publications over a five-year period (Jan 2005–Dec 2009), using the PubMed Central database. Data were taken from the full text of the research article. Criteria for identification and selection of articles included reports of original research and one of the following: (1) studies that were specifically identified as using mixed methods, either through keywords or description in the title; (2) qualitative studies conducted as part of larger projects, including randomized controlled trials, which also included use of quantitative methods; or (3) studies that “quantitized” qualitative data (Miles and Huberman 1994 ) or “qualitized” quantitative data (Tashakkori and Teddlie 1998 ). Per criteria used by McKibbon and Gadd (2004 (link)), the analysis had to be fairly substantial—for example, a simple descriptive analysis of baseline demographics of the participants was not sufficient to be included as a mixed method article. Further, qualitative studies that were not clearly linked to quantitative studies or methods were excluded from our review.
We next assessed the use of mixed methods in each study to determine their structure, function, and process. A taxonomy of these elements of mixed method designs and definition of terms is provided in Table 1 below. Procedures for assessing the reliability of the classification procedures are described elsewhere (Palinkas et al. 2010 ). Assessment of the structure of the research design was based on Morse’s (1991 (link)) taxonomy that gives emphasis to timing (e.g., using methods in sequence [represented by a “→”symbol] versus using them simultaneous [represented by a “+” symbol]), and to weighting (e.g., primary method [represented by capital letters like “QUAN”] versus secondary [represented in small case letters like “qual”]). Assessment of the function of mixed methods was based on whether the two methods were being used to answer the same question or to answer related questions and whether the intention of using mixed methods corresponded to any of the five types of mixed methods designs described by Greene et al. (1989 ) (Triangulation or Convergence, Complementarity, Expansion, Development, and Initiation or Sampling). Finally, the process or strategies for combining qualitative and quantitative data was assessed using the typology proposed by Cresswell and Plano Clark (2007 ): merging or converging the two datasets by actually bringing them together, connecting the two datasets by having one build upon the other, or embedding one dataset within the other so that one type of data provides a supportive role for the other dataset.Table 1

Taxonomy of mixed method designs

Element	Category	Definition
Structure	QUAL → quan	Sequential collection and analysis of quantitative and qualitative data, beginning with qualitative data, for primary purpose of exploration/hypothesis generation
	qual → QUAN	Sequential collection and analysis of quantitative and qualitative data, beginning with qualitative data, for primary purpose of confirmation/hypothesis testing
	Quan → QUAL	Sequential collection and analysis of quantitative and qualitative data, beginning with quantitative data, for primary purpose of exploration/hypothesis generation
	QUAN → qual	Sequential collection and analysis of quantitative and qualitative data, beginning with quantitative data, for primary purpose of confirmation/hypothesis testing
	Qual + QUAN	Simultaneous collection and analysis of quantitative and qualitative data for primary purpose of confirmation/hypothesis testing
	QUAL + quan	Simultaneous collection and analysis of quantitative and qualitative data for primary purpose of exploration/hypothesis generation
	QUAN + QUAL	Simultaneous collection and analysis of quantitative and qualitative data, giving equal weight to both types of data
Function	Convergence	Using both types of methods to answer the same question, either through comparison of results to see if they reach the same conclusion (triangulation) or by converting a data set from one type into another (e.g. quantifying qualitative data or qualifying quantitative data)
	Complementarity	Using each set of methods to answer a related question or series of questions for purposes of evaluation (e.g., using quantitative data to evaluate outcomes and qualitative data to evaluate process) or elaboration (e.g., using qualitative data to provide depth of understanding and quantitative data to provide breadth of understanding)
	Expansion	Using one type of method to answer questions raised by the other type of method (e.g., using qualitative data set to explain results of analysis of quantitative data set)
	Development	Using one type of method to answer questions that will enable use of the other method to answer other questions (e.g., develop data collection measures, conceptual models or interventions)
	Sampling	Using one type of method to define or identify the participant sample for collection and analysis of data representing the other type of method (e.g., selecting interview informants based on responses to survey questionnaire)
Process	Merge	Merge or converge the two datasets by actually bringing them together (e.g., convergence—triangulation to validate one dataset using another type of dataset)
	Connect	Have one dataset build upon another data set (e.g., complementarity—elaboration, transformation, expansion, initiation or sampling)
	Embed	Conduct one study within another so that one type of data provides a supportive role to the other dataset (e.g., complementarity—evaluation: a qualitative study of implementation process embedded within an RCT of implementation outcome)