Janice Thomas and Stella George
PM research raises many questions where the basic science and theories have not yet been developed. Mixed methods studies allow researchers to both develop and test theory. Researchers often conduct qualitative studies to develop theories based in the experience of managing projects and then larger sample size quantitative studies to generalise these theories. Sometimes quantitative survey research generates unexpected theories that need to be explained through qualitative study. This chapter presents guidance on how to design a mixed methods research (MMR) approach and explores an example of how this is done in practice.
understand what is meant by MMR, when it can be applied and explanation of same;
design a MMR study in project;
identify some of the likely challenges and potential mitigation strategies available to a research when undertaking a mixed methods approach.
Keywords: mixed methods, qualitative research, quantitative research, research design
Introduction to Mixed Methods Research (MMR)
MMR is becoming more common in many disciplines as research questions become more complex (a definition is offered below). As we come to ask more sophisticated questions about PM, we too find a greater need to mix research design elements of both a qualitative and quantitative nature, where neither approach individually would adequately address the research question. Mixed methods studies take into account culture and context as well as measurement. These research designs are tricky as all research methods employed must be used intentionally and systematically to the highest level of competence (Cresswell and Plano Clark, 2011).
Mixed methods research is, generally speaking, an approach to knowledge (theory and practice) that attempts to consider multiple viewpoints, perspectives, positions and standpoints (always including the standpoints of qualitative and quantitative research. (Johnson, Onwuegbuzie and Turner, 2007: p. 113)
Many of the challenges associated with mixed methods approaches arise from the necessary bridging of what some might consider incommensurable differences in ontology and epistemology (as discussed in detail in Part one of the book). Mixed methods requires a hybrid ontology that allows us to explore situations that are clearly social in nature while attempting to come up with reliable means of measuring or quantifying at least some aspects of the phenomenon. Similarly the epistemological position of mixed methods varies between the logical empiricism of attempting to observe empirical facts tempered by the knowledge that many managerial phenomenon are constructed through constructions of meanings by the social actors embedded in the phenomenon.
When No Other Method Will Do
Social science typically involves (1) deductive approaches using quantitative research methods, or (2) inductive approaches using qualitative methods. However, contributions can also be made when (3) quantitative methods have laid the ground for inductive theory building or (4) qualitative methods are used in deductive theory testing (Bitektine, 2008). Increasing the number of methods used in any one study clearly increases the complexity of the research design. However, there are many research situations where a mixed method is the only real option for addressing the research question.
the study is requires a range of perspectives and requires critical multiplism to make valuable contributions to our understanding of complex phenomenon (Johnson et al., 2007);
previous qualitative and quantitative research has only partly addressed the research question or may have produced contradictory results;
capture of relevant, reliable and honest data can only be achieved by using a range of data collection methods;
various philosophical perspectives on the research question result in conflicting perspectives of a phenomena. The implications of philosophical assumptions are more likely raised through conflicting data (Srnka and Koeszegi, 2007).
Mixed method approaches are particularly valued for their capacity to triangulate both data and method (Jack and Raturi, 2006). Multiple methods allow the researcher to use one method ‘as a validation process that ensures that the explained variances are the result of the underlying phenomenon or trait and not the method’ (Johnson et al., 2007: p. 113). By extending generalisability of the theory to those aspects of the social phenomena that are not amenable to quantification, theory testing using qualitative methods can reduce the need for ‘leaps of faith’ when translating quantitative results to other less-measurable aspects of social process (Bitektine, 2008).
Not Without a Downside
Clearly a mixed methods design is a valuable approach to developing deep rich understandings of a phenomenon and testing their generalisability. However, the flexibility and potential of a mixed method approach does not mean that it should be used in all circumstances because its benefits do not come without relatively large practical issues. For these reasons mixed method approaches are recognised as difficult to complete successfully.
MMR projects require skill in multiple methods which can be difficult to find as very few individuals have experience in both quantitative and qualitative methods (Bryman, 2007). This is the primary reason why this approach is not often suggested for thesis research.
Mixed methods studies require more effort on the part of the researcher than single method studies. Multimethod studies consume more time and resources as there is necessary duplication of effort in conducting multiple analsyes in order to facilitate triangulation.
Integration of data from both methods is difficult, and there are few good examples of integrative results (Bryman, 2007). Different analysis can yield apparently contradictory results that require further investigation to explain or may nullify your research.
The audience of the results of the project may prefer one type of data over another (Bryman, 2007) and the nature of the data means that one set is more interesting than the other to some researchers (Bryman, 2007).
Using both qualitative and quantitative methods simultaneously to answer a research question can be controversial. Some believe that the assumptions underlying differing methodologies are inherently opposed, and cannot be meaningfully combined.
It may be hard to publish results from a mixed method study as certain fields require data presentation within limited formats.
The additional cost to the researcher and difficulties in conducting and publishing this research serves as a caution to all researchers contemplating such a study and makes it critical that we only pursue this option when the benefits outweigh the costs. Thus, it is very important to carefully construct your research question before deciding on your methodological approach.
How to Mix Methods
Mixed methods approaches are often used in combination, by alternating between methods – typically using qualitative research for theory generation and quantitative methods for theory testing. A grounded theory approach as described predominantly in Eisenhardt’s grounded theory work is commonly used. This approach designs research to observe ‘phenomena with multiple methods to ground the theory development process in different versions of an existing reality’ (Jack and Raturi, 2006: p. 356). Theory is created by ‘observing patterns within systematically collected empirical data’ (Eisenhardt and Graebner, 2007: p. 30) by ‘recursively iterating between (and thus constantly comparing) theory and data during analysis, and theoretically sampling cases…’ (Eisenhardt and Graebner, 2007: p. 30). In a mixed methods study the iteration and theory building is strengthened by comparative analysis between quantitative and qualitative findings to create strong and detailed theory.
First, what could be called a two solitudes approach, whereby both quantitative and qualitative studies feed the results but they are done separately. The first approach is multidisciplinary where two (or more) different studies are completed independently and the results are combined after the fact. This can be a very time-effective way of completing a project as there is no dependency between the qualitative and quantitative studies
A sequential approach can be used, where qualitative analysis feeds quantitative analysis (or vice versa) to produces results. This approach also allows each methodology to be completed separately but the results of the first study (either quantitative or qualitative) informs the second study (either qualitative or quantitative). This approach is interdisciplinary in nature as the researchers need to work together at the planning and handoff points. This approach also allows you to maintain more separation between the types of studies and the underlying ontological and epistemological assumptions.
Third, an integrated approach where both quantitative and qualitative analysis are conducted simultaneously and interact with each other in producing the final results. The third approach is highly interdisciplinary, or even trans-disciplinary, where all data collection occurs at the same time for the different analytical methods, usually through a common set of instruments. This approach to mixed methods is much more complex to design as it requires more coordination and integration between studies and more flexibility in ontological and epistemological positions. The synthesis of these trans-disciplinary approaches adds both value and costs to this form of mixed methods study.
Structuring a Mixed Methods Project
Where research is not theory-driven but is about understanding a particular phenomenon, the research question drives the most appropriate methods for the study. A mixed method study only becomes appropriate when the questions being asked cannot be answered effectively by the use of a single method. There is no rigid formula for designing a mixed methods study, but the following checklist should provide some guidance in designing a mixed method project (Table 23.1). You can read in more detail on this topic in Cresswell and Plano Clark (2011) and Teddie and Tashakkori (2009).
How much consideration is given to the credibility, reliability and validity of the data? Credible data is data that is relevant to the study and also believable, convincing, probable and trustworthy.
Reliability relates to whether the data is suitable (captured in an appropriate way, and in sufficient quantity) and recorded in a way that it is fit to be relied upon during analysis.
Validity refers to a clear repeatable process that another researcher could recover (ideally replicate) the study and achieve comparable results (Checkland and Holwell, 1998). Meeting these requirements influences the empirical methods available to you.
Is triangulation used to increase credibility, reliability and validity? Mixed methods approaches provide the ability to investigate the same question from a number of perspectives. Analysing data from these different perspectives allows us to build a stronger case for or against the research hypotheses. What is an interesting result from one perspective can be explained with a different perspective applied to the same or related data. This triangulation of data from a variety of sources is an integral part of ensuring that the data collected is reliable and credible.
How generalisable are the results (in positivist quantitative studies)? Generalisability refers to the ability of the research to make claims based on the study to populations outside the original study. Generalisability usually has methodological implications in terms of sample size and nature of the measures and instruments used.
How transferable are the results (usually a question asked of qualitative studies)? Transferability is often used as a measure of the usefulness and the degree to which the findings could apply to situations outside the original study. Highly detailed descriptions of the research site and methods allows the reader to identify how the situation is the same or different from their own. In comparing the specifics of the research situation with one in which they are familiar, the reader may be able to infer that the results would be the same or similar in their own situation. Transferable studies have strong face validity, in that they resonate with knowledgeable readers.
How appropriate are the methods of analysis to the data collected and the questions asked? The selection of empirical methods should include consideration as to how data is collected. Will the collection method influence the number, quality and veracity of replies? Could some replies omit more compromising information? How could a change in method avoid potential problems? The sources of data sought (numbers of responses required, range of participants on same question(s)) will also practically influence the methods chosen (surveys, interviews – paper, online, phone, in person, documents, observations and so on) as will the attempt to create multiple measures of key concepts.
Having reviewed the guidance available to us on how to conduct mixed methods studies, and some of the quality indicators, let us turn now to an example and see how this guidance played out against reality of our project.
Example Case – A Mixed Method Project Framework
The Researching the Value of Project Management initiative (Value project) was a large-scale interdisciplinary (and sometimes trans-disciplinary) research project partially funded by PMI between 2005 and 2008. The project involved 48 researchers from 14 countries working in 18 teams to develop 65 detailed case studies. The research essentially needed to be a longitudinal study executed in a short timeframe across a wide geography.
The case study is a large, some may say mega, research project and some of you as students may be wondering about its relevance to planning your own research. While you may only run into a few of these issues on a smaller mixed methods study, our case provides the opportunity to examine all of the complexity associated with mixed methods studies and how we dealt with them. Having examined all of our challenges, we hope you can think about what you are most likely to face and how you will deal with these challenges in planning your own study.
Step 1 – What are You Interested in Exploring? What is the High-Level Research Question?
The overarching question of interest was given to us through the request for proposal process. The professional association wanted to know what value PM delivers to organisations. Specifically they wanted to quantify that value.
A long-standing and tricky problem known to be important to the organisation, the research essentially needed to be a longitudinal study executed in a short timeframe. All previous studies had failed to answer the question because no baseline data exists. It was known that some of the relevant measures are intangible and often seen as non-credible; isolating answers from overall context and the results of other changes in an open system is difficult to achieve.
there is no one right way to measure value or PM outside of an organisational context;
both value and PM are multidimensional constructs and, therefore, a variety of disciplines across the management spectrum have contributions to make in answering these questions;
the data to answer these questions is not easily available in most organisations.
Based on a pragmatic approach to the ontological and epistemological choices made in the design of the research and the belief that multi-paradigm inquiry is not only possible but is necessary to ‘offer insights into the characteristic contradictions and tensions embodied in contemporary organisation’ (Reed, 1985: p. 201), a mixed methods approach was identified from the outset as the best way to answer the question.
Recognising at the same time the difficulties associated with mixed methods studies we felt we had two major advantages to make this project possible. First, one of the co-leads was well trained and experienced in using both qualitative and quantitative methods. Second, the size and nature of the multidisciplinary team ensured that we had the expertise in various methodologies to manage a complex cross-method study.
Step 2 – What Questions do you need to Answer to Answer this Overarching Question?
What is PM in organisations? Is PM a measurable thing?
Is it different in different organisations/contexts? What organisational context differences effect the value of PM?
What is value? Value to whom? From what perspective? How is value categorised? Measured?
The common conceptual model originated in the early literature reviews conducted by the principal investigators in preparation for proposing on this study and is illustrated opposite (Figure 23.4).
The conceptual model allowed team members from all disciplines a touchstone to return to throughout the project to stay focused on the questions we were trying to answer.
Step 3 – What Data do you need to Answer these Questions?
Deciding how to use a mixed methods approach will be to some extent driven by your data sources. Three influences should be considered when determining your approach. The first is consideration to which data that would both be relevant to the research question and available for collection. The second is to the methods and instruments used to access and collect this data as this will impact its reliability and veracity. The final factor will be the method of analysis to be applied to this data in order to make sense of it and hopefully answer your research question.
Our approach to answer these questions was to get all these experts in a room for three days to work our way through the three questions and what data we would need to answer these questions from different disciplinary approaches. The discussion was long (three days), and sometimes heated (conflicting epistemological and ontological positions). However, this first workshop set the tone for the type of flexibility and cooperation that would be needed to complete the project. The lesson is to deal with the big, hairy questions of how you will satisfy people coming at your research from different perspectives early rather than late and then keep all the players focused on the research questions.
Step 4: What are the Appropriate Methods for Collecting this Data?
There were at least three levels of data (individual, group, organisational) required to answer our questions. Our project investigated value at the organisation level and multiple case study approach was deemed appropriate. The professional society sponsoring this research wanted generalisable data, so a multiple case study approach was necessary –preferably with enough cases to allow for some statistical analysis of data at the organisational level. Case study methodology is often the foundation of mixed methodology studies. Data for each case study would come from:
interview – to access perceptions and feelings as well as facts;
individual surveys, for example, ‘Do you use this tool?’;
organisational surveys, for example, ‘So you provide these types of training?’;
document review – to assess formalised actions;
observation – to assess culture and actual actions, situational choices.
Ensuring reliability, credibility and validity of our results was important. Approaches used included: researchers were trained in common data capture instruments and tools; stakeholder and advisory group assessment of process of data capture and analysis calculations and assumptions; multiple forms of evidence sought and triangulation in analysis at every opportunity; conservative (understatement) use of valuation and benefits; cautious and critical treatment of unsubstantiated outliers.
Instrument Development and Testing
Based on the data requirements and methods discussion defined in the workshop, instruments for data collection were created by a small team employing the help of method experts. Once the complete instrument package was completed (including five different interview templates, an organisational survey, five different online survey scripts, a document review strategy, and a form for collecting information on practices documented, talked about and actually done in each organisation) that then went out to the entire team for feedback.
A pilot phase (of five cases) was used to test the reviewed instruments to determine data blind spots. The pilot cases were reviewed with the entire research team (at a workshop) in order to test the ability of the research instruments to collect the data we needed to answer the research questions and evaluate our ability to answer our research questions with the data we would be able to collect. We used this point of review – reality check – as a decision stage gate for the project. In our case there were some serious limitations in our ability to collect data on quantitative costs associated with PM implementations and some difficulty in estimating quantitative benefits. We believed that what could be accomplished would be valuable but the decision was the project sponsor’s to make. They decided we should continue with the revised scope of delivery. Carrying out this type of review is very important in order to fall back to what is doable and realign stakeholder expectations as early in the process as possible.
Figure 23.5 illustrates the six types of data instruments developed for this study and the magnitude of the data collection undertaken. Data management is an issue for any research project. For a mixed method study, the magnitude of types of data as well as data elements makes this an even more serious consideration. We built a database that contained all quantitative and qualitative data referenced by case number in order to be able to manage the electronic and paper data.
Project design strategy
We chose to start ‘from qualitative material and transform it into numerical data to be used in further quantitative analysis aimed at deriving generalisable results’ (Srnka and Koeszegi, 2007: p. 33) as well as separately collecting quantitative data. Both qualitative and quantitative data was analysed separately and jointly in a combination of the sequential and combined approaches to mixed methods studies in order to provide an overall picture of the value phenomenon. Adopting this generalisation approach to MMR design allows us to conduct discovery-oriented research that provides significant research insights while also allowing us to derive generalisable results from largely qualitative data (Srnka and Koeszegi, 2007).
Given the complexity of the research question and the multidisciplinary team, Phase 1 was designed to answer two questions: Can we figure out a way to answer these questions that would satisfy a multidisciplinary stakeholder group? Is the design developed do-able? Phase 1 included conceptual and methodological design of the project and testing of this design through pilot case data collection and analysis. Phase 1 completed with the stage gate decision discussed above.
Phase 2 involved an exploratory, theory development stage based on collection of multiple case studies and was to conclude with a large-scale survey-based stage to test the theories developed. However, as part of the stage gate decision, the original proposal of testing the developed theory with a large sample size (>200 organisations) was postponed in order to meet sponsor deadlines. Theories generated from the case study analysis were tested against the quantitative date collected to date. Further theory testing was left for future researchers.
The lesson here is that it is very easy to be overly optimistic in planning a mixed methods study as it is easy to underestimate the added complexity of dealing with two or more different types of analysis. Make sure you use good PM techniques and build off ramps and decision points into your project strategy that your committee and sponsors agree to. Having agreement early in your project on how these decisions will be made and by whom will save a lot of grief later on.
Step 5: Data Collection and Analysis of the Data
The primary method for this study was comparative mixed methods, detailed cases on a large scale. Each team (16 in total) completed a cross-case analysis of the cases they collected. It was critical for each team to follow the project design to ensure the data would be comparative.
In addition, a qualitative and quantitative team worked across the whole data set, sometimes in isolation (content analyses, discursive analysis to identify trends in value perceptions and to analyse the consistency of perceptions across organisations and quantitative: descriptive statistics, to correlation and regression) and in the later stages in collaboration. The lesson for a smaller stage study is likely that you need to devote time to each type of analysis separately before you try to synthesise the results into a coherent whole. Make sure you know what the qualitative and quantitative findings are independently before trying to make sense of the whole story.
Step 6 – Draw Conclusions, Report and Share Findings
In our example case, the content and the process for accomplishing the project was planned as part of the methodological choice. ‘… methodological choices started to drive specific methods’ (Thomas and Mullaly, 2008, p 55). The research did not follow a structured grounded theory approach, such as those set out in the work of Glaser and Strauss (1967); Glaser (1992); Strauss and Corbin (1998) but followed Eisenhardt’s lead and ‘engage(d) in systematic data collection and theory development processes that are reported with transparent description … of cross case comparison techniques. The key here is to convey the rigor, creativity, and open-mindedness of the research processes while sidestepping confusion and philosophical pitfalls’ (Eisenhardt and Graebner, 2007). ‘Standard data was collected from each participating organisation by every case team. In addition, each researcher collected any additional data they deemed necessary to explore the value phenomenon from their own unique disciplinary/theoretical perspective. Cross-case comparison using both qualitative and quantitative methods formed the foundation for both theory generation and theory testing’ Thomas and Mullaly (2008: p. 55).
Drawing Conclusions and Getting Published
The final workshop of our project was dedicated to joint sense-making based on the results of the individual case study analysis, the comparative case analysis by each research team, and the cross-case qualitative and quantitative analysis. In this way we used triangulation of both data and method to draw conclusions across all the work done.
So far this study has resulted in one book, several book chapters, many conference papers, a special edition of a journal, and more than 15 published papers. That said, meeting the page limitations and framing papers for publication in organisation theory journals has proven as difficult as the guidance suggests. It is difficult after completing a study of this magnitude to pick out very small insights to focus on for publication when the richness of the data available begs for greater description. Pick you target audience wisely and be prepared to put in the work to tailor your message.
Adopting a mixed methods approach adds a level of complexity to all elements of research design. However, there are some research questions in PM that require this level of inquiry. If yours is one of these, make sure you build a strong supervisory team with the skills needed to guide you through the use of multiple methods or a research team that can provide these skill sets. From our experience, mixed methods are a challenging approach to take but worth it in terms of the sophistication and detailed understanding that they generate.
Tips and Questions
Tips for Students
Further information on MMR design can be gained from the references provided. In particular a fuller introduction to MMR can be found Creswell and Plano Clark (2011) Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications, Inc. A great overview presentation of Cresswell’s work can be found in the online presentation http://prezi.com/qsksml6l-_vi/introduction-to-mixed-methods-research.
Another useful source is Teddie, C. and Yahakkori, A (2009)Foundations of Mixed Methods. Thousand Oaks, CA: Sage Publications. If you are interested in exploring the methods used in the Value Project in more detail, please see Thomas and Mullaly (2008).
Tips for Supervisors
Students adopting a mixed methods approach are tricky to supervise as expertise in both quantitative and qualitative methods are required on the committee. This can often cause dissension on the committee itself over what constitutes appropriate analysis to answer research questions. The student will require more sophisticated people skills to manage the committee through the process.
Research questions need to be defined in ways that make it clear what types of data and analysis is required to answer them. Often defining research questions such that different methods are used to answer each question makes it easier to address the issues in Tip 1 as different committee members can focus on different questions.
Make sure that your student understands both the benefits and costs of MMR, and that the research they want to undertake truly requires a more complex mixed methods approach.
When are you most likely to adopt a mixed methods approach?
What are the three greatest challenges to conducting MMR and how can you deal with them?
How do you deal with the epistemological challenges presented by adoption of a mixed methods approach?
 From World Bank Indicators website: http://data.worldbank.org/indicator/NE.GDI.TOTL.ZS accessed on 31 March 2012.
 Dewey (1938) specifies that a situation ‘is not a single object or event or set of objects and events. For we never experience nor form judgments about objects and events in isolation, but only in connection with a contextual whole. This latter is what is called a “situation”’ (Dewey, 1938: p. 66).
 NAP is a competence network bringing together companies from the entire value chain in the Dutch process industry.
 The NETLIPSE Consortium consisted of: Department for Transport (UK), Ministry of Infrastructure and the Environment (NL), National Laboratory for Civil Engineering (PT), Road and Bridge Research Institute (PL), Swiss Federal Institute of Technology (CH), Erasmus University (NL), KPC (CH), AT Osborne B.V. (NL).