sample questions for mixed methods research

• What is mixed methods research?

Last updated

20 February 2023

Reviewed by

Miroslav Damyanov

By blending both quantitative and qualitative data, mixed methods research allows for a more thorough exploration of a research question. It can answer complex research queries that cannot be solved with either qualitative or quantitative research .

Analyze your mixed methods research

Dovetail streamlines analysis to help you uncover and share actionable insights

Mixed methods research combines the elements of two types of research: quantitative and qualitative.

Quantitative data is collected through the use of surveys and experiments, for example, containing numerical measures such as ages, scores, and percentages. 

Qualitative data involves non-numerical measures like beliefs, motivations, attitudes, and experiences, often derived through interviews and focus group research to gain a deeper understanding of a research question or phenomenon.

Mixed methods research is often used in the behavioral, health, and social sciences, as it allows for the collection of numerical and non-numerical data.

• When to use mixed methods research

Mixed methods research is a great choice when quantitative or qualitative data alone will not sufficiently answer a research question. By collecting and analyzing both quantitative and qualitative data in the same study, you can draw more meaningful conclusions. 

There are several reasons why mixed methods research can be beneficial, including generalizability, contextualization, and credibility. 

For example, let's say you are conducting a survey about consumer preferences for a certain product. You could collect only quantitative data, such as how many people prefer each product and their demographics. Or you could supplement your quantitative data with qualitative data, such as interviews and focus groups , to get a better sense of why people prefer one product over another.

It is important to note that mixed methods research does not only mean collecting both types of data. Rather, it also requires carefully considering the relationship between the two and method flexibility.

You may find differing or even conflicting results by combining quantitative and qualitative data . It is up to the researcher to then carefully analyze the results and consider them in the context of the research question to draw meaningful conclusions.

When designing a mixed methods study, it is important to consider your research approach, research questions, and available data. Think about how you can use different techniques to integrate the data to provide an answer to your research question.

• Mixed methods research design

A mixed methods research design  is   an approach to collecting and analyzing both qualitative and quantitative data in a single study.

Mixed methods designs allow for method flexibility and can provide differing and even conflicting results. Examples of mixed methods research designs include convergent parallel, explanatory sequential, and exploratory sequential.

By integrating data from both quantitative and qualitative sources, researchers can gain valuable insights into their research topic . For example, a study looking into the impact of technology on learning could use surveys to measure quantitative data on students' use of technology in the classroom. At the same time, interviews or focus groups can provide qualitative data on students' experiences and opinions.

• Types of mixed method research designs

Researchers often struggle to put mixed methods research into practice, as it is challenging and can lead to research bias. Although mixed methods research can reveal differences or conflicting results between studies, it can also offer method flexibility.

Designing a mixed methods study can be broken down into four types: convergent parallel, embedded, explanatory sequential, and exploratory sequential.

Convergent parallel

The convergent parallel design is when data collection and analysis of both quantitative and qualitative data occur simultaneously and are analyzed separately. This design aims to create mutually exclusive sets of data that inform each other. 

For example, you might interview people who live in a certain neighborhood while also conducting a survey of the same people to determine their satisfaction with the area.

Embedded design

The embedded design is when the quantitative and qualitative data are collected simultaneously, but the qualitative data is embedded within the quantitative data. This design is best used when you want to focus on the quantitative data but still need to understand how the qualitative data further explains it.

For instance, you may survey students about their opinions of an online learning platform and conduct individual interviews to gain further insight into their responses.

Explanatory sequential design

In an explanatory sequential design, quantitative data is collected first, followed by qualitative data. This design is used when you want to further explain a set of quantitative data with additional qualitative information.

An example of this would be if you surveyed employees at a company about their satisfaction with their job and then conducted interviews to gain more information about why they responded the way they did.

Exploratory sequential design

The exploratory sequential design collects qualitative data first, followed by quantitative data. This type of mixed methods research is used when the goal is to explore a topic before collecting any quantitative data.

An example of this could be studying how parents interact with their children by conducting interviews and then using a survey to further explore and measure these interactions.

Integrating data in mixed methods studies can be challenging, but it can be done successfully with careful planning.

No matter which type of design you choose, understanding and applying these principles can help you draw meaningful conclusions from your research.

• Strengths of mixed methods research

Mixed methods research designs combine the strengths of qualitative and quantitative data, deepening and enriching qualitative results with quantitative data and validating quantitative findings with qualitative data. This method offers more flexibility in designing research, combining theory generation and hypothesis testing, and being less tied to disciplines and established research paradigms.

Take the example of a study examining the impact of exercise on mental health. Mixed methods research would allow for a comprehensive look at the issue from different angles. 

Researchers could begin by collecting quantitative data through surveys to get an overall view of the participants' levels of physical activity and mental health. Qualitative interviews would follow this to explore the underlying dynamics of participants' experiences of exercise, physical activity, and mental health in greater detail.

Through a mixed methods approach, researchers could more easily compare and contrast their results to better understand the phenomenon as a whole.  

Additionally, mixed methods research is useful when there are conflicting or differing results in different studies. By combining both quantitative and qualitative data, mixed methods research can offer insights into why those differences exist.

For example, if a quantitative survey yields one result while a qualitative interview yields another, mixed methods research can help identify what factors influence these differences by integrating data from both sources.

Overall, mixed methods research designs offer a range of advantages for studying complex phenomena. They can provide insight into different elements of a phenomenon in ways that are not possible with either qualitative or quantitative data alone. Additionally, they allow researchers to integrate data from multiple sources to gain a deeper understanding of the phenomenon in question.  

• Challenges of mixed methods research

Mixed methods research is labor-intensive and often requires interdisciplinary teams of researchers to collaborate. It also has the potential to cost more than conducting a stand alone qualitative or quantitative study . 

Interpreting the results of mixed methods research can be tricky, as it can involve conflicting or differing results. Researchers must find ways to systematically compare the results from different sources and methods to avoid bias.

For example, imagine a situation where a team of researchers has employed an explanatory sequential design for their mixed methods study. After collecting data from both the quantitative and qualitative stages, the team finds that the two sets of data provide differing results. This could be challenging for the team, as they must now decide how to effectively integrate the two types of data in order to reach meaningful conclusions. The team would need to identify method flexibility and be strategic when integrating data in order to draw meaningful conclusions from the conflicting results.

• Advanced frameworks in mixed methods research

Mixed methods research offers powerful tools for investigating complex processes and systems, such as in health and healthcare.

Besides the three basic mixed method designs—exploratory sequential, explanatory sequential, and convergent parallel—you can use one of the four advanced frameworks to extend mixed methods research designs. These include multistage, intervention, case study , and participatory. 

This framework mixes qualitative and quantitative data collection methods in stages to gather a more nuanced view of the research question. An example of this is a study that first has an online survey to collect initial data and is followed by in-depth interviews to gain further insights.

Intervention

This design involves collecting quantitative data and then taking action, usually in the form of an intervention or intervention program. An example of this could be a research team who collects data from a group of participants, evaluates it, and then implements an intervention program based on their findings .

This utilizes both qualitative and quantitative research methods to analyze a single case. The researcher will examine the specific case in detail to understand the factors influencing it. An example of this could be a study of a specific business organization to understand the organizational dynamics and culture within the organization.

Participatory

This type of research focuses on the involvement of participants in the research process. It involves the active participation of participants in formulating and developing research questions, data collection, and analysis.

An example of this could be a study that involves forming focus groups with participants who actively develop the research questions and then provide feedback during the data collection and analysis stages.

The flexibility of mixed methods research designs means that researchers can choose any combination of the four frameworks outlined above and other methodologies , such as convergent parallel, explanatory sequential, and exploratory sequential, to suit their particular needs.

Through this method's flexibility, researchers can gain multiple perspectives and uncover differing or even conflicting results when integrating data.

When it comes to integration at the methods level, there are four approaches.

Connecting involves collecting both qualitative and quantitative data during different phases of the research.

Building involves the collection of both quantitative and qualitative data within a single phase.

Merging involves the concurrent collection of both qualitative and quantitative data.

Embedding involves including qualitative data within a quantitative study or vice versa.

• Techniques for integrating data in mixed method studies

Integrating data is an important step in mixed methods research designs. It allows researchers to gain further understanding from their research and gives credibility to the integration process. There are three main techniques for integrating data in mixed methods studies: triangulation protocol, following a thread, and the mixed methods matrix.

Triangulation protocol

This integration method combines different methods with differing or conflicting results to generate one unified answer.

For example, if a researcher wanted to know what type of music teenagers enjoy listening to, they might employ a survey of 1,000 teenagers as well as five focus group interviews to investigate this. The results might differ; the survey may find that rap is the most popular genre, whereas the focus groups may suggest rock music is more widely listened to. 

The researcher can then use the triangulation protocol to come up with a unified answer—such as that both rap and rock music are popular genres for teenage listeners. 

Following a thread

This is another method of integration where the researcher follows the same theme or idea from one method of data collection to the next. 

A research design that follows a thread starts by collecting quantitative data on a specific issue, followed by collecting qualitative data to explain the results. This allows whoever is conducting the research to detect any conflicting information and further look into the conflicting information to understand what is really going on.

For example, a researcher who used this research method might collect quantitative data about how satisfied employees are with their jobs at a certain company, followed by qualitative interviews to investigate why job satisfaction levels are low. They could then use the results to explore any conflicting or differing results, allowing them to gain a deeper understanding of job satisfaction at the company. 

By following a thread, the researcher can explore various research topics related to the original issue and gain a more comprehensive view of the issue.

Mixed methods matrix

This technique is a visual representation of the different types of mixed methods research designs and the order in which they should be implemented. It enables researchers to quickly assess their research design and adjust it as needed. 

The matrix consists of four boxes with four different types of mixed methods research designs: convergent parallel, explanatory sequential, exploratory sequential, and method flexibility. 

For example, imagine a researcher who wanted to understand why people don't exercise regularly. To answer this question, they could use a convergent parallel design, collecting both quantitative (e.g., survey responses) and qualitative (e.g., interviews) data simultaneously.

If the researcher found conflicting results, they could switch to an explanatory sequential design and collect quantitative data first, then follow up with qualitative data if needed. This way, the researcher can make adjustments based on their findings and integrate their data more effectively.

Mixed methods research is a powerful tool for understanding complex research topics. Using qualitative and quantitative data in one study allows researchers to understand their subject more deeply. 

Mixed methods research designs such as convergent parallel, explanatory sequential, and exploratory sequential provide method flexibility, enabling researchers to collect both types of data while avoiding the limitations of either approach alone.

However, it's important to remember that mixed methods research can produce differing or even conflicting results, so it's important to be aware of the potential pitfalls and take steps to ensure that data is being correctly integrated. If used effectively, mixed methods research can offer valuable insight into topics that would otherwise remain largely unexplored.

What is an example of mixed methods research?

An example of mixed methods research is a study that combines quantitative and qualitative data. This type of research uses surveys, interviews, and observations to collect data from multiple sources.

Which sampling method is best for mixed methods?

It depends on the research objectives, but a few methods are often used in mixed methods research designs. These include snowball sampling, convenience sampling, and purposive sampling. Each method has its own advantages and disadvantages.

What is the difference between mixed methods and multiple methods?

Mixed methods research combines quantitative and qualitative data in a single study. Multiple methods involve collecting data from different sources, such as surveys and interviews, but not necessarily combining them into one analysis. Mixed methods offer greater flexibility but can lead to differing or conflicting results when integrating data.

Editor’s picks

Last updated: 11 January 2024

Last updated: 15 January 2024

Last updated: 25 November 2023

Last updated: 12 May 2023

Last updated: 30 April 2024

Last updated: 18 May 2023

Last updated: 10 April 2023

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next.

Users report unexpectedly high data usage, especially during streaming sessions.

Users find it hard to navigate from the home page to relevant playlists in the app.

It would be great to have a sleep timer feature, especially for bedtime listening.

I need better filters to find the songs or artists I’m looking for.

Log in or sign up

Get started for free

Qualitative Research

• What is Qualitative Research
• PEO for Qualitative Questions
• SPIDER for Mixed Methods Qualitative Research Questions
• Finding Qualitative Research Articles
• Critical Appraisal of Qualitative Research Articles
• Mixed Methods Research
• Qualitative Synthesis

SPIDER: Mixed Methods Qualitative Research Questions

SPIDER  is a search strategy for finding research to answer a mixed-method qualitative research question.

• Sample:  Sample is similar to Patient/Population of PICO. This is the group of people you are interesting in studying qualitatively. For example, in the qualitative research question, "What are the barriers felt by clinicians that lead to the reluctance to use EBP in practice?", S = clinicians  
• Phenomenon of Interest:  The Phenomenon of Interest can be similar to Intervention of PICO. This can be viewed as the topic of the research. For example, in the qualitative research question, "What are the barriers felt by clinicians that lead to the reluctance to use EBP in practice?", PI = Use of EBP  
• Focus Groups
• Observations  
• Evaluation:  Evaluation is similar to Outcomes of PICO. For example, in the qualitative research question, "What are the barriers felt by clinicians that lead to the reluctance to use EBP in practice?", E = Barriers to use of EBP  
• Phenomenology
• Ethnography
• Grounded theory

Using SPIDER for Search Terms

Now that you have your qualitative research questions broken into SPiDER, you can now think about your search strategy and search terms.

For example, in the qualitative research question, " What are the barriers felt by clinicians that lead to the  reluctance to use EBP in practice?"

NOTE: You might not use all these terms in your search.

FINAL BOOLEAN SEARCH:

(clinician* OR health care professional OR health care provider) AND (evidence-based practice OR EBP) AND  ( interview* OR focus group*) AND (barrier* OR hinder* OR resist*) AND phenomenology

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence  synthesis.  Quality Health Research, 22 (10), 1435-1443. doi: 10.1177/1049732312452938

• << Previous: PEO for Qualitative Questions
• Next: Finding Qualitative Research Articles >>
• Last Updated: Aug 28, 2023 2:47 PM
• URL: https://researchguides.gonzaga.edu/qualitative

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

116k Accesses

92 Citations

13 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

What is Qualitative in Qualitative Research

Qualitative Research: Ethical Considerations

Criteria for Good Qualitative Research: A Comprehensive Review

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Community Engagement Program

Supporting bi-directional community engagement to improve the relevance, quality, and impact of research.

• Getting Started
• Resources for Equity in Research
• Community-Engaged Student Practice Placement
• Maternal Health Equity
• Youth Mental Health
• Leadership and Membership
• Past Members
• Study Review Rubric
• Community Ambassador Initiative
• Implementation Science Working Group
• Past Webinars & Podcasts
• Policy Atlas
• Community Advisory Board

For more information:

Mixed methods research.

According to the National Institutes of Health , mixed methods strategically integrates or combines rigorous quantitative and qualitative research methods to draw on the strengths of each. Mixed method approaches allow researchers to use a diversity of methods, combining inductive and deductive thinking, and offsetting limitations of exclusively quantitative and qualitative research through a complementary approach that maximizes strengths of each data type and facilitates a more comprehensive understanding of health issues and potential resolutions.¹ Mixed methods may be employed to produce a robust description and interpretation of the data, make quantitative results more understandable, or understand broader applicability of small-sample qualitative findings.

Integration

This refers to the ways in which qualitative and quantitative research activities are brought together to achieve greater insight. Mixed methods is not simply having quantitative and qualitative data available or analyzing and presenting data findings separately. The integration process can occur during data collection, analysis, or in the presentation of results.

¹ NIH Office of Behavioral and Social Sciences Research: Best Practices for Mixed Methods Research in the Health Sciences

Basic Mixed Methods Research Designs 

View image description .

Five Key Questions for Getting Started

• What do you want to know?
• What will be the detailed quantitative, qualitative, and mixed methods research questions that you hope to address?
• What quantitative and qualitative data will you collect and analyze?
• Which rigorous methods will you use to collect data and/or engage stakeholders?
• How will you integrate the data in a way that allows you to address the first question?

Rationale for Using Mixed Methods

• Obtain different, multiple perspectives: validation
• Build comprehensive understanding
• Explain statistical results in more depth
• Have better contextualized measures
• Track the process of program or intervention
• Study patient-centered outcomes and stakeholder engagement

Sample Mixed Methods Research Study

The EQUALITY study used an exploratory sequential design to identify the optimal patient-centered approach to collect sexual orientation data in the emergency department.

Qualitative Data Collection and Analysis : Semi-structured interviews with patients of different sexual orientation, age, race/ethnicity, as well as healthcare professionals of different roles, age, and race/ethnicity.

Builds Into : Themes identified in the interviews were used to develop questions for the national survey.

Quantitative Data Collection and Analysis : Representative national survey of patients and healthcare professionals on the topic of reporting gender identity and sexual orientation in healthcare.

Other Resources:

  Introduction to Mixed Methods Research : Harvard Catalyst’s eight-week online course offers an opportunity for investigators who want to understand and apply a mixed methods approach to their research.

Best Practices for Mixed Methods Research in the Health Sciences [PDF] : This guide provides a detailed overview of mixed methods designs, best practices, and application to various types of grants and projects.

Mixed Methods Research Training Program for the Health Sciences (MMRTP ): Selected scholars for this summer training program, hosted by Johns Hopkins’ Bloomberg School of Public Health, have access to webinars, resources, a retreat to discuss their research project with expert faculty, and are matched with mixed methods consultants for ongoing support.

Michigan Mixed Methods : University of Michigan Mixed Methods program offers a variety of resources, including short web videos and recommended reading.

To use a mixed methods approach, you may want to first brush up on your qualitative skills. Below are a few helpful resources specific to qualitative research:

• Qualitative Research Guidelines Project : A comprehensive guide for designing, writing, reviewing and reporting qualitative research.
• Fundamentals of Qualitative Research Methods – What is Qualitative Research : A six-module web video series covering essential topics in qualitative research, including what is qualitative research and how to use the most common methods, in-depth interviews, and focus groups.

View PDF of the above information.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

• Open access
• Published: 01 May 2024

Evaluation and students’ perception of a health equity education program in physical therapy: a mixed methods pilot study

• Alexis A. Wright 1 ,
• Dominique Reynolds 1 &
• Megan Donaldson 2  

BMC Medical Education volume  24 , Article number:  481 ( 2024 ) Cite this article

187 Accesses

Metrics details

Health equity is a common theme discussed in health professions education, yet only some researchers have addressed it in entry-level education.

The purpose of this study is to serve as an educational intervention pilot to 1) evaluate students’ perception of the effectiveness of the DPT program in providing a foundation for health equity education, with or without the benefit of a supplemental resource and 2) establishing priorities for the program related to educating students on health inequities in physical therapy clinical practice.

A mixed method design with a focus-group interview was utilized to explore students’ perceptions of the DPT program's commitment to advancing health equity.

A three-staged sequential mixed methods study was conducted. Stage 1 began with quantitative data collection after completing the DEI Bundle utilizing the Tripod DEI survey. Stage 2 involved identifying themes from the Tripod Survey data and creating semi-structured interview questions. Stage 3 consisted of a focus group interview process.

A total of 78 students completed the Tripod DEI survey upon completing 70% of the curriculum. Thirty-five students, eight core faculty, 13 associated faculty, and four clinical instructors completed the APTA DEI Bundle Course Series. According to the Tripod DEI Survey results, program stakeholders found the program’s commitment to DEI and overall climate to be inclusive, fair, caring, safe, welcoming, and understanding of individuals from different backgrounds, including a sense of student belonging where students feel valued and respected. Three themes emerged from the qualitative focus group interviews, including the value of inclusivity, health equity curricular foundations, and DEI in entry-level DPT education.

Conclusions

This study highlights the value of incorporating health equity and DEI topics into curricula while fostering an incluse program culture.

Peer Review reports

Introduction

Racial and ethnic disparities in healthcare are a longstanding and well-documented crisis in the United States [ 1 ]. A strategic goal of the American Physical Therapy Association (APTA) is to increase diversity, equity, and inclusion within the profession to serve society's health better. At its core, physical therapy is rooted in optimizing overall health and decreasing preventable illness and injury. Additionally, physical therapists are trained to be adaptive and respond to patients' social and environmental influences that impact health outcomes. These foundational traits uniquely position healthcare providers with the skills to respond to health inequities. Education and training for health providers are rarely studied to determine the effectiveness or implementation of the educational training [ 1 , 2 ]. Specifically, diversity, equity, and inclusion (DEI) education training provides a basis to confront systemic racism and improve health equity, and physical therapy programs are being called to action [ 2 ]. However, the measurement of learners’ awareness and perceived effectiveness of educational interventions has lagged [ 1 ].

The literature review on this topic includes a study by the Institute of Medicine (IOM), which has provided recommendations for addressing and eliminating racial/ethnic disparities in healthcare. These recommendations include increasing healthcare providers’ awareness of racial/ethnic disparities in healthcare and educating health providers on health disparities, cultural competence, and the impact of race/ethnicity on clinical decision-making [ 3 ] A developing entry-level Doctor of Physical Therapy program intentionally designed curricula aligned with the IOM recommendations. Curricular topics were informed by the Clinical Prevention and Population Health Curriculum Framework, a product of the Healthy People Curriculum Task Force established in 2002 by the Association for Prevention Teaching and Research (APTR) [ 4 ]. Knowledge-based activities were designed to further awareness and understanding of the social determinants of health, health prevention, cultural awareness, health inequities, healthcare accessibility, systems thinking, and implicit and explicit bias among entry-level DPT students. The theoretical framework of the DPT curriculum is based on a theoretical framework of constructivism, which refers to the belief that learners actively construct knowledge by linking new information to what they have previously learned and by incorporating new experiences into their knowledge base and that learners’ knowledge structures are continually constructed and reconstructed [ 5 ].

Additionally, co-curricular educational activities were promoted throughout the program.

The theoretical framework for co-curricular educational activities is based on relational learning. Specifically, this model has been used for health promotion and inclusion [ 6 , 7 ]. The co-curriculum does what the standard academic curriculum generally does not: it is developmental, transformative, and future-focused. For example, as a program, sessions were provided for learners to attend speaker sessions on DEI topics, apply for leadership roles (including the Diversity, Equity, and Anti-Racism (DEAR) Council), and engage in service activities, all grounded in an expectation of professional behaviors that encourage intellectual discussions on complex topics in an environment free of criticism, discrimination, harassment or any other emotional or physical harm.

The purpose of this study is to serve as an educational intervention pilot to 1) evaluate students’ perceptions of the effectiveness of the DPT program in providing a foundation for health equity education, with or without the benefit of a supplemental resource, and 2) establish priorities for the program related to educating students on health inequities in physical therapy clinical practice.

Materials and methods

Participants and study design.

Determining the research question(s) is vital in the mixed research process. Research questions are pivotal in the mixed research process, which is interactive, emergent, fluid, and evolving [ 8 ]. As Leech and Onwuegbuzie [ 8 ] defined, “mixed methods research questions combine or mix both the quantitative and qualitative research questions necessitating the resulting data be collected and analyzed.” Mixed research sampling designs can be classified according to (a) the time orientation of the components (e.g., whether the qualitative and quantitative phases occur concurrently or sequentially) and (b) the relationship of the qualitative and quantitative samples (e.g., identical vs. parallel vs. nested vs. multilevel).

Design:  To address the objectives of this study, a partially mixed-method design with a sequential and nested relationship was selected. The nested structure implies that individuals chosen for one phase of the study (qualitative focus group interviews) constitute a subset of those selected in the preceding phase (participants in the quantitative surveys) [ 8 , 9 ]. Nonetheless, qualitative and quantitative research methodologies hold equal significance in this study's design and analytical approach.

Sampling Strategy: Participant enrichment refers to the mixing of qualitative and quantitative techniques for the rationale of optimizing the sample. Beginning with Phase 1, a total of 153 participants, including students (81) from the Class of 2022 (as pre-professionals) and 2) program faculty (16), associated faculty (36), and clinical instructors (20) (as post-professionals) were offered the option to participate in this mixed methods study. An email describing the purpose of the study was sent to all participants.

Within mixed-method designs, instrument fidelity is essential and used by researchers to maximize the appropriateness and utility of the quantitative and qualitative instruments used in the study. These included the Tripod DEI survey, the APTA Diversity, Equity, and Inclusion (DEI) Bundle, and the qualitative semi-guided interview process. Stage 1 began with quantitative data collection after completing the Diversity, Equity, and Inclusion Bundle utilizing the Tripod DEI survey. Stage 2 involved identifying themes from the Tripod Survey data and creating semi-structured interview questions. Stage 3 consisted of the focus group interview process. See further details outlining the timeline and phases of the study in Fig.  1 . Timeline and Process for Study.

Timeline and Process for Study

The research implementation began with the quantitative survey, in which all students were surveyed using the Tripod DEI survey, which was deployed after semester 4 of the program, reflecting 70% completion of the curriculum [ 10 ]. Students were allowed to participate in the voluntary, supplementary APTA DEI Bundle beginning in Semester 5 [ 11 ]. Before participating in the APTA DEI Bundle, the Tripod DEI Survey was readministered to all students, program faculty, associated faculty, and clinical instructors who elected to participate [ 10 ]. Following completion of the APTA DEI Bundle, the Tripod DEI Survey was readministered a second time to all students, program faculty, associated faculty, and clinical instructors who completed the APTA DEI Bundle course series [ 10 , 11 ]. The pre-test and post-test methodologies explored differences between adding the American Physical Therapy Association DEI Bundle to the program’s curriculum and co-curricular activities [ 11 ].

The study commenced once approval to conduct it was obtained from the Institutional Review Board at the university. After the submission was reviewed, the Tufts University IRB office determined that the proposed activity was not deemed human research as defined by DHHS and FDA regulations. (IRB ID:STUDY00002820).

Research planning: quantitative study instrument

Tripod Education Partners works with programs to gather, organize, and report on student and teacher perspectives [ 10 ]. The Tripod DEI survey captures student perceptions of how diversity, equity, and inclusion issues play out in their school. The survey collects feedback from teachers about their experiences as teachers and perspectives about strengths and opportunities for improvement. Permission and funding for survey distribution were obtained before disseminating the survey.

The survey consisted of a total of 38 questions with eight distinct measures including 1) School commitment to DEI ( N  = 3), 2) School climate overall ( N  = 4), 3) School climate for DEI ( N  = 4), 4) Classroom teaching supporting DEI ( N  = 7), 5) Co-Curricular activities supporting DEI ( N  = 3), 6) Everyday discrimination by students ( N  = 6), 7) Everyday discrimination by teachers ( N  = 6), 8) Meaningful interactions across difference N  = 5) (Tripod Education Partners,2019). School commitment to DEI is scored on a Likert scale from 1 (totally untrue) to 5 (totally true). School climate overall and DEI are scored as ordinal variables, with 2 being more favorable. Classroom teaching supporting DEI is scored on a Likert scale from 1 (none) to 5 (all). Co-curricular activities supporting DEI is scored on a Likert scale from 1 (my school doesn’t sponsor things like this) to 6 (very often). Everyday discrimination by students and teachers and meaningful interactions across differences are scored on a Likert scale from 1 (never) to 5 (very often).

The “overall sense of belonging” ( N  = 3) was scored on a Likert scale from 1 (totally untrue) to 5 (totally true).

The Tripod DEI survey development shows good construct validity and internal consistency [ 10 ]. Diverse student populations are at the center of the survey. Reports disaggregate findings by social identities across various groups, including but not limited to race, gender, and socioeconomic status. This breakdown allows programs to pinpoint groups of students reporting less-than-positive experiences and take action to address their needs.

Research planning: description of the DEI training bundle

The optional training program was conducted through asynchronous electronic delivery of the APTA DEI bundle [ 11 ]. This program is a three-part series exploring foundational concepts related to diversity, equity, and inclusion and is led by Diana Lautenberger, MA, co-lead of the American Medical Colleges' leadership development seminar program. The three-part series utilizes a highly reflective approach whereby participants learn about identity, privilege, bias, and allyship as foundational pillars to achieving DEI. In addition, participants engage in self-reflection throughout the series to apply concepts to their clinical and personal lives to create more respectful and inclusive environments.

The series consists of three two-hour sessions: Part 1 – Unconscious Bias in the Health Professions; Part 2 – Power, Privilege, and Microaggressions; Part 3 – Responding to Microaggressions Through Allyship. The elements of this bundle listed objectives for the learners to 1) understand how their various identities carry social capital or power, 2) describe aspects of a dominant culture that advantage some and disadvantage others, and 3) utilize allyship and bystander intervention strategies that reduce harm to create more respectful and inclusive environments [ 11 ]. This program requires the completion of an assessment from the training. Viewers who completed all three sessions and scored at least 70% on each session's assessment (built into the modules) were also allowed to earn 0.6 CEUs (six contact hours) and a certificate of completion.

Research planning: qualitative focus group interviews

Using an explanatory sequential mixed methods study, the qualitative portion aimed to further understand the students’ perceptions, establish priorities for the program related to educating students on health inequities in physical therapy clinical practice, and evaluate the effectiveness of adding the DEI Bundle. Based on the results of the quantitative portion of the study, two researchers created questions that would be used in the focus group interviews. The a priori semi-structured question guide in Table  1 was designed to allow emergent focus group discussion to explore concepts further.

Data analysis plan

Quantitative data collection and analysis.

The data analysis program IBM SPSS 28.0 was utilized to store and analyze data from the Tripod DEI survey. For all the Tripod DEI survey subscales, items were summed, and scores were calculated. Descriptive statistics were utilized to calculate means, standard deviations, and 95% confidence intervals for each of the eight domains and Overall Sense of Belonging. Paired sample t-tests were conducted to compare pre-test and post-test scores. Summary independent samples t-tests compared the entire sample data ( N  = 81) to the post-DEI Bundle Series data.

Qualitative data collection and analysis

The semi-structured focus group interview guide questions (Table  1 ) were designed after the quantitative data collection was completed, and the data assessment revolved around concepts collected from the survey data.

A variety of data collection strategies were used, including (a) a mixture of open- and closed-ended items within the questionnaires that guided the focus group interview process, (b) a mixture of a priori (from the quantitative results) and additional emergent/flowing focus-group strategies through a semi-guided interview process. The Standards for Reporting Qualitative Research (SRQR) checklist was utilized for reporting.

Given the small sample size, no statistical software was utilized. Coding was used to assign labels to data segments to capture their meaning and allow comparison to identify themes or patterns. Both researchers used qualitative content analysis to systematically categorize transcribed content into topic areas from the thick descriptions provided. Qualitative fields were created to organize data by topic counts of language content areas (such as “DEI” and “belonging” quotes). The preliminary or open coding was done first and then refined to a higher level to reflect broader categories. All coding stages were done separately and then together to ensure improved accuracy. Then, the researchers used the comparison analysis and consensus approach to categorize and interpret data to identify patterns and content themes during the analysis. The analysis used a matrix table as a visual spreadsheet, where the rows represented participants, and the columns represented codes identified.

Researcher characteristics and reflexivity: The background and experience of the researchers could have influenced the research as two of the researchers had routine involvement with the participants within the study. The same researchers that conducted the study design and implementation conducted the focus group interviews via Zoom while participants were on clinical rotations. The focus-group interviews were audio-recorded and transcribed by an administrative coordinator who supported the faculty and had limited student interactions during daily work.

Techniques to enhance trustworthiness: The research team, consistent throughout the study, undertook the quantitative and qualitative data analysis. To maintain objectivity, they devised a set of a priori questions for interviews, steering clear of leading inquiries or interpretations. Subsequently, they conducted content analysis directly from transcriptions. Reflexivity strategies encompassed credibility checks via member validation and a post-session peer debriefing (between researchers), ensuring accuracy in focus group interviews. The research coordinator, unbiased to quantitative analysis, remained uninvolved in question formulation, solely providing session transcriptions for analysis. Furthermore, thick descriptions were provided, and qualitative counts of language content areas were evenly applied to promote the transferability of qualitative findings. By integrating these measures, the study aimed to mitigate inherent limitations in its design and bolster the credibility, transferability, dependability, and confirmability of its qualitative research, thus enhancing the trustworthiness and reliability of its findings.

Quantitative analysis and results

A total of 78 students completed the Tripod DEI survey upon completing Semester 4 of the curriculum. A total of 42 students, eight core faculty, 16 associated faculty, and four clinical instructors elected to participate and complete the voluntary, supplementary pre-APTA DEI bundle Tripod DEI survey beginning Semester five. A total of 35 students, eight core faculty, 13 associated faculty, and four clinical instructors completed the APTA DEI Bundle Course Series. Thirty-two students, eight core faculty, 13 associated faculty, and four clinical instructors completed the post-APTA DEI Bundle Tripod DEI Survey.

Student results

Demographics of the full sample of 78 students can be found in Table  2 .

Survey results following the completion of Semester 4 are summarized below and reported as mean, standard deviation.

School Commitment to DEI (1 = totally untrue; to 5 = totally true)

Students generally found the program's commitment to DEI to be inclusive, fair, and understanding of individuals from different backgrounds (M = 4.1, SD = 0.9) or “mostly true”.

School Climate Overall (1 = less favorable; 2 = favorable)

Students reported the program's climate/culture as caring, respectful, safe, and welcoming (M = 2.0, SD = 0.1) where 2 is scored as caring, respectful, safe, and welcoming.

School Climate for DEI (1 = less favorable; 2 = favorable)

Students rated the program's climate/culture for DEI as “equally fair” to all students, regardless of their social identity (M = 1.9, SD = 0.2). This included questions related to race, ethnicity, sexual orientation, socioeconomic status, and gender where 2 is scored as equally fair to all students.

Classroom teaching Supporting DEI (1 = none; 5 = all)

Classroom teaching supporting diversity, equity, and inclusion rated “most but not all” (M = 4.1, SD = 0.8) faculty as having integrated material on different social identities, discussing issues of social inequality, and using student-centered teaching methods. This included questions related to helping students think about how to improve the world, leading discussions about why some people have difficult lives and other people have easier lives, connecting content from the classroom to problems or issues in the world as well as the student’s own life and interests, helping students think about how to improve other people’s lives, assigning readings or materials about people from different backgrounds or places, and taught about influential people from many different cultures.

Co-Curricular Activities Supporting DEI (1 = my school doesn’t sponsor things like this; 6 = very often)

With regards to co-curricular activities supporting diversity, equity, and inclusion, students reported on average that they “hardly ever” participated in a school-sponsored group for students of different racial, ethnic, socioeconomic, gender, sexual orientation, or ability groups; attended a school-sponsored event related to diversity, fairness, or inclusion; or participated in a program sponsored group working to make the world a better place (M = 3.3, SD = 1.0).

Everyday Discrimination by Students (1 = never; 5 = very often)

Students reported “never to hardly ever” regarding everyday discrimination by students regarding courtesy, respect, intelligence, being better than others, being bullied or threatened, and insults (M = 1.8, SD = 0.7).

Everyday Discrimination by Teachers (1 = never; 5 = very often)

Students reported “never to hardly ever” regarding everyday discrimination by faculty regarding courtesy, respect, intelligence, being better than others, being bullied or threatened, and insults (M = 1.4, SD = 0.6).

Meaningful Interactions Across Differences (1 = never; 5 = very often)

Students rated the program as “fairly often” with regards to meaningful interactions across differences, including honest discussions with other students whose religion was different from their own, whose families have more or less money than their own, whose culture is different from their own, and whose race is different from their own (M = 3.8, SD = 0.9).

Belonging (1 = totally untrue; 5 = totally true)

Finally, the students rated the program as “mostly true to totally true” concerning their sense of belonging in the program, whereby the student feels valued, respected, and a sense of belonging (M = 4.4, SD = 0.8).

Comparison of tripod survey pre-post

Thirty-two students elected to participate and complete the APTA DEI Bundle Series with completed pre- and post-Bundle Series survey data. Demographic information on student participation in the DEI Bundle can be found in Table  3 . After completing the APTA DEI Bundle Series, we found no significant difference in any of the eight domains or Sense of Belonging. We found no significant differences in any domain between the full sample ( N  = 78) and the post-DEI Bundle Series data sample ( N  = 32).

Post-professional stakeholder results

Twenty-five of our post-professional stakeholders elected to participate and complete the APTA DEI Bundle Series with completed pre- and post-Bundle Series survey data. After completing the APTA DEI Bundle Series, we found no significant difference in any of the eight domains or Sense of Belonging.

Similarly, the post-professional stakeholders generally found the program's commitment to DEI to be inclusive, fair, and understanding of individuals from different backgrounds (M = 4.2, SD = 1.2).

Post-professionals reported the program’s climate/culture overall as caring, respectful, safe, and welcoming (M = 2.0, SD = 0.0).

Post-professionals rated the program’s climate/culture for DEI as “equally fair” to all students, regardless of their social identity (M = 2.0, SD = 0.1). This included questions related to race, ethnicity, sexual orientation, socioeconomic status, and gender.

Classroom Teaching Supporting DEI (1 = none; 5 = all)

Post-professionals rated climate for DEI Classroom teaching supporting diversity, equity, and inclusion rated “most but not all faculty” (M = 3.8, SD = 1.0) as having integrated material on different social identities, discussing issues of social inequality, and using student-centered teaching methods. This included questions related to helping them think about how to improve the world, leading discussions about why some people have difficult lives and other people have easier lives, connecting content from the classroom to problems or issues in the world as well as the student’s own life and interests, helping students think about how to improve other people’s lives, assigning readings or materials about people from different backgrounds or places, and taught about influential people from many different cultures.

With regards to co-curricular activities supporting diversity, equity, and inclusion, post professionals reported on average that they “hardly ever participated” in a school-sponsored group for students of different racial, ethnic, socioeconomic, gender, sexual orientation, or ability groups; attended a school-sponsored event related to diversity, fairness, or inclusion; or participated in a program sponsored group working to make the world a better place (M = 2.9, SD = 1.0).

Post professionals reported “never to hardly ever” concerning everyday discrimination by students (M = 1.3, SD = 0.5).

Post professionals reported “never to hardly ever” concerning everyday discrimination by teachers (M = 1.4, SD = 0.5).

Post professionals rated the program as “fairly often” with regards to meaningful interactions across differences, including honest discussions with other students whose religion was different from their own, whose families have more or less money than their own, whose culture is different from their own, and whose race is different from their own (M = 3.1, SD = 0.9).

Finally, the post professionals rated the program as “mostly true to totally true” regarding their sense of belonging in the program, whereby the student feels valued, respected, and a sense of belonging (M = 4.5, SD = 1.0).

Result of qualitative focus group content analysis

From those participants completing the quantitative portion of the study, a nested sub-group of students ( n  = 9) volunteered to participate in the semi-structured focus group interview following the completion of the DEI Bundle. Demographic information on student participation in the interviews can be found in Table  4 .

There was a rich discussion with the interview guide around the topics 1) DEI with or without the training supplement related to health equity in physical therapy and 2) the program’s commitment to training students on topics associated with health equity. Three themes emerged from the qualitative focus group interviews based on the final qualitative content analysis.

Theme 1: student’s perceived value of inclusivity

Theme one was the value of inclusivity with three associated sub-themes of fairness, actions, and communication. In higher education, inclusivity is the ongoing process of improving the education system to meet the needs of all students, especially those in marginalized groups. Inclusivity involves reimagining educational services to cater to a diverse audience and making learning materials and teaching methods accessible to as many students as possible. This includes considering a range of diverse student identities, including race, gender, sexuality, and abilities. “ The program does make an effort, especially with adjuncts that we bring in, ableism talks, and people from different backgrounds speaking to us in classes on Zoom .”

Additionally, providing sessions to improve inclusivity and communicating and demonstrating actions consistent with the value of inclusivity is essential to the participants. “ Being a member of the gay community, having a faculty in class that you feel you belong in and are not outcasted in is super important .” Participants valued being included during activities and communicating support during school and personal life challenges. The participants recognized the challenge of finding people from different backgrounds who meet the expectations and specialties to teach within the program. They identified that, at times, visual diversity was limited within the core faculty but felt an intention of more inclusivity of race and ethnicity within the associated faculty roles or lecturers.

Within the value of inclusivity, there is also an inherent limitation to who can afford the DPT graduate-level program at a private university. Hybrid education offers more geographical convenience and reaches a more diverse student group; however, current students feel that money concerns could be a barrier to inclusivity, especially those in marginalized groups. “Program doesn’t have control over the cost of tuition but does communicate what is available as far as opportunities for financial aid.” However, they felt that communication about costs for the hybrid program and what financial aid was available was essential.

Theme 2: student’s perceived value of health equity curricular foundations

Theme two was the value of health equity curricular foundations with three sub-themes of representation in assignments, system resources, and practice issues. Health equity is the goal of helping people reach their highest level of health. It means everyone has a fair chance to achieve optimal health regardless of race, ethnicity, gender identity, or socioeconomic status. Health equity can be promoted through DEI initiatives, which focus on representing the acceptance and inclusiveness of people. The focus group reported health equity topics associated with race, social determinants, and access were satisfactorily addressed within the curriculum. However, there were opportunities to gain additional insights on improving formative activities to be more integrated with how health issues affect those with visual diversity. “ Activities within the program should also include skin tone other than white throughout systems-focused curriculum case studies, mannequins, and simulation/ standardized patients .”

Theme 3: student’s perceive value of DEI in entry-level PT education

Lastly, one remaining theme specifically addressed DEI supplementation to the curriculum. Theme three is the value of DEI in entry-level physical therapy education, with three sub-themes emerging on the timing of content, planned redundancy of learning, and the limited value of a stand-alone DEI bundle. The students in the focus group had a consensus on their perceived confidence and appropriate knowledge of social determinants of health when working with the underserved population during their clinical education exposures. However, the focus group agreed with “ concerns about generalizing their feelings to all classmates, as some students may have had different experiences based on their final clinical education setting and exposure .”

Additionally, according to the student perception, inclusivity and health equity values should be blended across the curriculum so that support and the training of those with different backgrounds can be promoted through DEI initiatives. Curriculum initiatives were given rich context regarding the program and curriculum that would be more “ inclusive and supportive of a health equity curricular track and activities threaded throughout the curriculum rather than a stand-alone module .” There was a consensus from the focus group that mirrored the quantitative results that there was a perceived “ limited value in the DEI Bundle as a stand-alone module outside of the curriculum .” Instead, the students preferred the curriculum designed to include the topics sufficiently within systems and population coursework.

The mixed methods analysis allows a better explanation of the student’s perceptions by blending the results from this study's qualitative and quantitative study portions. It was found in both portions of the study design that the program climate/culture is essential, especially as students relate inclusivity and accepting others when learning to value DEI from a health equity perspective. Students further strengthened their perceived value for their education and blended content topics across the curriculum as they related to health equity and diversity. Still, they found value when more than just content was presented. Students felt that there was a program culture, planned curriculum content, and co-curricular (outside of a class) support for health equity and inclusivity of the population's health care providers serve. As educators look to streamline variation in essential content across healthcare disciplines, utilizing a structured format (toolkit or bundle) could benefit students educationally but may be valued less by them.

Our study aimed to explore the students’ perceptions and establish priorities for the program regarding educating students on health inequities in physical therapy clinical practice.

Health equity is a common theme discussed in health professions education, yet only some have published the methods to address it in entry-level education. National organizations recommend that medical schools and health professions train students in the social determinants of health. This provides the opportunity to educate the next generation of healthcare professionals about sensitive yet essential issues.

Given the complexity of this topic, we utilized a three-staged sequential mixed methods approach to generate the results presented in this study. We found the program’s commitment to DEI and overall climate to be inclusive, fair, caring, safe, welcoming, and understanding of individuals from different backgrounds, including a sense of student belonging where students feel valued and respected. Additionally, the sample provided feedback on the educational approach and format, which was provided with the DEI Bundle. The modular-based curricular approach (not integrated through a course) was used in this study. Thus, the results of the APTA’s DEI Bundle should be considered, given the context of the study, regarding the curricular delivery and format as an “addition to” approach. Given this format, the DEI Bundle was insignificant due to the threaded curricular approach already within the program, as assessed on the Tripod DEI survey or qualitative focus group theme. This approach aligns with other recommendations for curriculum approaches to health equity [ 12 ] that integrate health equity content longitudinally and alongside other topics. The goal would be to eliminate views of health equity and healthcare as separate [ 13 ].

Limited studies explore health equity topics' style, content, and delivery through the healthcare professional’s entry-level educational program. However, the Association of American Medical Colleges recommends that medical educators expose their students to content about health disparities [ 14 ]. There are some challenges to implementing the recommendations [ 15 ], which are further complicated by the lack of recommendations regarding format, delivery, and the requisite degree of competency, which are poorly defined. Several resources are provided but not easily found across all health professions disciplines. However, several studies highlight the importance of health equity education, its impact on therapeutic relationships (trust and caring), and identify the consequences of implicit bias on patient adherence and outcomes [ 16 ].

Significant work must be done to unite all the health professions on strategies for implementing the health equity curriculum. However, an external resource strategy or modular-based approach could be effective, given limited resources and a lack of topic expertise within the program faculty. Still, it should be used with an integrated approach and placed intentionally within the curriculum design. It should have more opportunities for integration across courses, with case studies to facilitate thinking and reasoning and culminate in a competency type of assessment. Curriculum toolkits provided by professional associations may be one way to unite the disciplines to support health equity education in the health professions [ 17 ]. An excellent example of this approach is the American Academy of Family Practitioners Health Equity Curricular Toolkit, which has over 40 content experts [ 18 , 19 ]. A threaded curriculum with a program culture and willingness to utilize health equity curriculum toolkits are essential for our next generation of health practitioners. These toolkits are resources for learning and reducing the variability in education [ 18 ]. Exploring outcomes associated with toolkits may be an option to begin to explore best practices in curriculum delivery to maximize learning outcomes and competency on health equity [ 20 ]. Lastly, any health equity resource or curricular approach should facilitate the exploration of some of the most pressing questions around social determinants of health, vulnerable populations, economics, and policy from an evidence-informed perspective.

Limitations

There are several limitations that we would like to address. Within the quantitative portion of the study, the Tripod DEI survey adequately assessed overall student perception of the DPT program commitment to DEI; however, it may need more responsiveness surrounding the APTA DEI Bundle. Within any mixed methods design approach, it is important to address data fidelity during the qualitative portion. A non-investigator conducted both the survey distribution and outcome assessment; however, the focus group interviews were conducted by two study investigators. Additionally, both researchers are on the leadership team within the program, which may compromise the fidelity, trustworthiness, or sharing from the participants during this experience. It is a limitation in the study that the researchers also are involved in the education. Although a safe space and relational learning theory approach is utilized within the program, this may have limited some of the exploration of the topics/themes if the participants were sensitive. From what was shared in the focus groups, a non-investigator recorded and transcribed the data analysis portion. The second limitation of the qualitative focus groups was the limited number and need for more diversity within the sample. Specifically, the individuals who made time to participate in the qualitative focus group were not significantly diverse regarding their race or sex. The third limitation is the inability to identify the number of students who respond based on their participation in additional co-curricular activities to supplement their learning in DEI.

However, significant work must be done to unite all the health professions on strategies for implementing health equity curricula. It was essential to gain insight from the students’ perception and establish priorities on the current curriculum and entry-level education program culture related to educating students on health inequities in physical therapy clinical practice. However, given limited resources and a lack of topic expertise for health equity content among program administrators and faculty, an external resource strategy or modular-based approach could be effective. However, based on our study, the program culture is important as it relates to DEI from a health equity perspective. It should be evident to students as we influence them to become the next generation of health professionals.

Lastly, the intentional curriculum design should have more opportunities for integration across courses with case studies and culminate in a competency type of assessment, even if an external resource is used. Resources are available to support health equity education in the health professions, including health equity curriculum toolkits, which provide free links and resources for learning and may help to reduce the variability in education [ 15 ]. Any health equity resource or curricular approach should facilitate faculty’s willingness to include some of the most pressing questions around social determinants of health, vulnerable populations, economics, and policy within their current or future developed curriculum. However, motivating incremental changes in entry-level professional teaching methods and working intentionally to integrate health equity into the clinic- and classroom-based environments are tangible next steps. Identifying best practices from education to implementation has yet to be well known, and this study only provided a pilot for future studies.

Availability of data and materials

The data supporting this study's findings are available from the corresponding author upon request.

Fitzgerald C, Hurst S. Implicit bias in healthcare professionals: a systematic review. BMC Med Ethics. 2017;18:19.

Article   Google Scholar  

Matthews ND, Rowley KM, Dusing SC, Krause L, Yamaguchi N, Gordon J. Beyond a Statement of Support: Changing the Culture of Equity, Diversity, and Inclusion in Physical Therapy. Phys Ther. 2021;101(12):pzab212. https://doi.org/10.1093/ptj/pzab212 .

Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care. Unequal treatment: confronting racial and ethnic disparities in health care. In: Smedley BD, Stith AY, Nelson AR, editors. Washington (DC): National Academies Press (US); 2003. PMID: 25032386.

“Clinical Prevention and Population Health Curriculum Framework.” Association for Prevention Teaching and Research. https://www.teachpopulationhealth.org/ .

Dong H, Lio J, Sherer R, Jiang I. Some Learning Theories for Medical Educators. Med Sci Educ. 2021;31(3):1157–72. https://doi.org/10.1007/s40670-021-01270-6 . PMID:34457959; PMCID:PMC8368150.

Nieminen, J.H. Assessment for Inclusion: rethinking inclusive assessment in higher education, Teaching in Higher Education. 2022. https://doi.org/10.1080/13562517.2021.2021395 .

Rogers DT. The working alliance in teaching and learning: theoretical clarity and research implications. International Journal for the Scholarship of Teaching and Learning. 2009;3(2):Article 28.

Leech NL, Onwuegbuzie AJ. Guidelines for Conducting and Reporting Mixed Research in the Field of Counseling and Beyond. J Couns Dev. 2010;88:61–9.

Leech NL, Onwuegbuzie AJ. A typology of mixed methods research designs. Quality and Quantity: International Journal of Methodology. 2009;43:265–75.

Tripod Education Partners. Tripod’s Diversity, Equity, and Inclusion (DEI Survey) Technical Manual; 2019.

Google Scholar  

American Physical Therapy Association DEI Bundle. October 22, 2021. APTA's New DEI Course Series Helps You Work Toward More Inclusivity. 2022.  https://www.apta.org/news/2021/10/22/dei-bundle .

Butler, L., Arya, V., Nonyel, N., & Moore, T. The Rx-HEART Framework to Address Health Equity and Racism Within Pharmacy Education. American Journal of Pharmaceutical Education. 2021:85. https://doi.org/10.5688/ajpe8590 .

Landry AM. Integrating Health Equity Content Into Health Professions Education. AMA J Ethics. 2021;23(3):E229–234.

Association of American Medical Colleges. Quality improvement and patient safety competencies across the learning continuum. Association of American Medical Colleges: New and Emerging Areas in Medicine Series; 2019.

Liburd L, Hall J, Mpofu J, Williams S, Bouye K, Penman-Aguilar A. Addressing Health Equity in Public Health Practice: Frameworks, Promising Strategies, and Measurement Considerations. Annu Rev Public Health. 2020. https://doi.org/10.1146/annurev-publhealth-040119-094119 .

Paterick TE, Patel N, Tajik AJ, Chandrasekaran K. Improving health outcomes through patient education and partnerships with patients. Proc (Bayl Univ Med Cent). 2017;30(1):112–3.

Ueffing E, Tugwell P, Roberts J, Walker P, Hamel N, Welch V. Equity-oriented toolkit for health technology assessment and knowledge translation: application to scaling up of training and education for health workers. Hum Resour Health. 2009;7:67–67. https://doi.org/10.1186/1478-4491-7-67 .

American Academy of Family Practitioners. Health Equity Curricular Toolkit. Accessed 6 Jun 2023. https://www.aafp.org/family-physician/patient-care/the-everyone-project/health-equity-tools.html .

Martinez-Bianchi V, Frank B, Edgoose J, Michener L, Rodiguez M, Gottlieb L, Reddick B, Kelly C, Yu K, Davis S, Carr J, Lee JW, Smith KL, New RD. Addressing Family Medicine’s Capacity to Improve Health Equity through Collaboration. Accountability and Coalition-Building Fam Med. 2019;51(2):198–203.

Porroche-Escudero A, Popay J. The Health Inequalities Assessment Toolkit: supporting integration of equity into applied health research. J Public Health (Oxf). 2020;43:567–72. https://doi.org/10.1093/pubmed/fdaa047 .

Download references

Acknowledgements

Not applicable (NA).

This project was funded through an internal Tufts University School of Medicine Innovations in Diversity Education Awards (IDEAS) program.

Author information

Authors and affiliations.

Doctor of Physical Therapy Program, Department of Rehabilitation Sciences, Tufts University, Boston, MA, 02111, USA

Alexis A. Wright & Dominique Reynolds

Department of Rehabilitation Sciences, Medical University of South Carolina, Charleston, SC, 29425, USA

Megan Donaldson

You can also search for this author in PubMed   Google Scholar

Contributions

Authors AW and MD contributed to the work's concept, design, and data analysis; drafting and final approval of the work; and agree to be accountable for all aspects of the work. Author DR contributed to acquiring the data for the work; reviewing and final approval of the work; and agrees to be accountable for the accuracy.

Corresponding author

Correspondence to Alexis A. Wright .

Ethics declarations

Ethics approval and consent to participate.

This research study was performed following the Declaration of Helsinki. This research study was submitted to the Tufts University Social, Behavioral, & Educational Research IRB on 6/9/2022 and was determined not to meet the criteria for human research—study Protocol STUDY00002820. The IRB determined “that the proposed activity is not research as defined by DHHS and FDA regulations. IRB review and approval by this organization is not required. This determination applies only to the activities described in the IRB submission and does not apply should any changes be made.” The research team did follow all the requirements for ethical research, including the consent process. Informed consent to participate in the study was obtained from all participant data reported in this study.

Consent for publication

Competing interests, additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wright, A.A., Reynolds, D. & Donaldson, M. Evaluation and students’ perception of a health equity education program in physical therapy: a mixed methods pilot study. BMC Med Educ 24 , 481 (2024). https://doi.org/10.1186/s12909-024-05471-6

Download citation

Received : 27 June 2023

Accepted : 25 April 2024

Published : 01 May 2024

DOI : https://doi.org/10.1186/s12909-024-05471-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Doctor of physical therapy
• Health equity
• Health professions

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 07 May 2024

Understanding caregiver burden and quality of life in Kerala’s primary palliative care program: a mixed methods study from caregivers and providers’ perspectives

• Arsha Kochuvilayil 1 &
• Ravi Prasad Varma 1  

International Journal for Equity in Health volume  23 , Article number:  92 ( 2024 ) Cite this article

Metrics details

Family caregivers are vital for long-term care for persons with serious health-related suffering in Kerala. Long-term caregiving and ageing may become burdensome and detrimental to patients and caregivers. We compared the caregiver burden and quality-of-life of ageing caregivers with younger caregivers. We also explored the palliative care nurses’ perceptions of the family caregivers’ issues.

We did a mixed method study focusing on two groups: (i) three in-depth interviews and a cross-sectional survey among 221 caregivers of palliative care patients in five randomly selected panchayats (most peripheral tier of three-tier local self-government system in India concerned with governance of a village or small town) of Kollam district, Kerala, as part of development and validation of the Achutha Menon Centre Caregiver Burden Inventory; (ii) five in-depth interviews with purposively selected primary palliative care nurses as part of a study on local governments and palliative care. We used a structured interview schedule to collect cross-sectional data on sociodemographic and caregiving-related characteristics, caregiver burden, and health-related quality of life using the EuroQol EQ5D5L and interview guidelines on caregiver issues tailored based on participant type for qualitative interviews.

Older caregivers comprised 28.1% of the sample and had significantly poorer health and quality-of-life attributes. More senior caregivers experiencing caregiver burden had the lowest mean scores of 0.877 (Standard deviation (SD 0.066, 95% confidence intervals (CI) 0.854–0.899) followed by younger caregivers with high burden (0.926, SD 0.090, 95% CI 0.907–0.945), older caregivers with low burden (0.935, SD 0.058, 95% CI 0.912–0.958) and younger caregivers with low burden (0.980, SD 0.041, 95% CI 0.970–0.990). Caregivers faced physical, psychological, social, and financial issues, leading to a caregiver burden. The relationships between the palliative care nurses and family caregivers were complex, and nurses perceived caregiver burden, but there were no specific interventions to address this.

In our study from Kollam, Kerala, three out of ten caregivers of palliative care patients were 60 years of age or older. They had significantly lower health-related quality of life, particularly if they perceived caregiver burden. Despite being recognized by palliative care nurses, caregiver issues were not systematically addressed. Further research and suitable interventions must be developed to target such problems in the palliative care programme in Kerala.

Norman Daniels “We should not allow misfortune to beget injustice” [ 1 ].

In Low- or Middle-Income Countries (LMIC), when a person becomes bedridden or homebound due to chronic illness or injury, family members are likely to be tasked with caring for a dependent [ 2 ]. State involvement still needs to improve in such situations, but the local government (LG) driven primary palliative care programme in Kerala state, India, has been functioning for nearly 30 years as a well-acknowledged approach for community-based sustainable palliative care [ 3 , 4 , 5 ]. Governance in India comprises powers that are divided between a central government of the country, more regional state governments with separate legislatures, and local governments with locally elected representatives. Local governments oversee governance administration and developmental activities within specific jurisdictions like villages or towns, overseeing local infrastructure and services. Health is considered a subject of interest for the state governments. Kerala initiated decentralization reforms in several sectors including health care, where substantial funds and many functions were transferred to local governments [ 6 ]. The Kerala primary palliative care programme evolved with the support of local governments. Bedridden or homebound patients with serious health-related concerns requiring long-term symptom management are usually registered under this programme [ 5 ]. Pain and symptom management, psychological support for patient and family and provision of assistive aids and medicines are integral parts of the services rendered [ 3 , 4 ]. However, even in this setting palliative care patients are highly dependent on others, primarily family caregivers, for their daily activities [ 4 ]. Family caregivers also help with medical and nursing care requirements [ 7 ]. Consequently, palliative care nurses often train family caregivers on simple and practical strategies of caregiving [ 4 ]. Thus, family caregivers play an integral role in translating programme services into better outcomes for the patient.

At times, for some such caregivers, this caregiving can become a burden, a multidimensional form of distress affecting their physical, psychological, social and financial well-being [ 2 , 8 , 9 ]. Perceived caregiver burden is associated with increased mortality, [ 10 , 11 , 12 ] poor health outcomes, including anxiety and depression [ 13 , 14 ] and reduced quality-of-life among family caregivers [ 15 ]. Several studies have explored caregiver burden and associated factors [ 16 , 17 ], but few studies have looked at these issues from the providers’ perceptive in LMIC [ 18 ]. Palliative care nurses have a limited understanding of caregiver burden and related issues. Patients remain the focus of care, while caregivers and their issues may go largely unnoticed [ 19 ].

Caregivers themselves may be sufferers of chronic diseases. This may be particularly true of Kerala, where the population aged 60 and above comprised 16.5% of the people in 2021 anisre expected to reach 20.9% by 2031 in Kerala [ 20 ]. Ageing caregivers may experience an increased impact of the consequences of caregiving along with physiological ageing, isolation and comorbidities [ 21 ]. With advancing age, multimorbidity is common among the ageing population [ 22 ]. Changing family structures due to migration and the increased number of women entering the workforce lead to many households having only ageing persons. Caring for a bedridden or homebound person by an ageing spouse is likely to be high in the Kerala population. Most such caregivers see ‘caregiving’ as their responsibility and feel obligated to provide care for their dependent. Spouse caregivers frequently report being more stressed and burdened compared to adult-child caregivers [ 9 ]. Ageing spousal carers may be at risk of increased cognitive impairment, loneliness, sadness, and anxiety compared to demographically matched ageing non-caregivers [ 23 ]. Also, our earlier analysis of depression among women caregivers had shown increasing odds of depression for higher age groups. These initial results underscore the significance of considering age as a potential factor that may contribute to varying experiences of burden among caregivers [ 13 ]. Age is usually treated as a confounder in studies on caregiving and adjusted at the time of analysis, and age-specific findings are not often reported [ 24 ]. Recently, however, research attention to the importance of ageing on caregiving outcomes is increasing [ 25 ]. There is a clear need to explore differences in experiences and needs of different age groups within the caregiver population so that targeted interventions and support strategies may be developed.

The World Health Organization in 2002 had recommended that services for chronic care should foster continuity of care and personal connection between the caregiver and the care recipient [ 26 ]. This will require functional relationships between the palliative care nurses and family caregivers, necessitating effective communication and rapport building by the nurse [ 27 ]. How the programme and its frontline representative, the palliative care nurse, perceive family caregivers, the caregiving role and caregiver issues are not adequately explored. A 2019 palliative care policy document from Kerala mentions caregiver support but this is still in a very early stage in the programme [ 28 ]. In this context, we studied the caregiver burden and quality of life of caregivers aged 60 years or above compared to younger caregivers of palliative care patients in Kerala. We also explored the perspectives of palliative care nurses on family caregiver issues in home care settings and whether these perspectives are reflected in the services offered by the nurses and the programme.

The palliative care programme

All panchayats in Kerala have a home care team that is led by a trained palliative care nurse. The nurse conducts periodic home visits along with the field staff of the local primary health centre, elected LG members and community volunteers. Each palliative care nurse schedules the home visits, directs patient health assessment and management and maintains several registers, one of which is the nominal register with patient name, contact information, diagnosis, and remarks on main service provision (e.g., catheter change, wound dressing etc.). We used the patient register of selected panchayats to identify patients and contact their caregivers for enrolment in the study.

The details of the sampling strategy for the cross-sectional survey have been published earlier [ 8 ].. The basis for sample size was adequacy for factor analysis– a sample size of 200 was deemed adequate for factor analysis with 25 items, achieving an item-to-participant ratio of at least 1:8 [ 34 ]. As male caregivers were very few, all male caregivers as reported by palliative care nurses were approached. Women caregivers were selected purposively from the list of patients in each panchayat palliative care registry to represent both cancer and non-cancer conditions.

Regarding sample size for the in-depth interviews, the primary objective of the in-depth interviews with caregivers was scrutiny of the representation of caregiver burden domains identified from the literature, and no new domains emerged after three interviews. For palliative care nurses, perceptions of caregiver burden were first identified and coded from literature and a draft thematic framework was prepared a priori. The first nurse interviewed belonged to the panchayats selected for the quantitative survey. During that interview, the interviewer (AK) felt that the nurse was fully aware of the caregiver issues encountered during the cross-sectional survey by the investigator and was giving responses conforming to the interviewer’s expectations. Therefore, four remaining nurses were purposively selected from panchayats in the same district that were not part of the quantitative study. Interviews were conducted to explore new categories and themes and data collection was stopped when no new categories emerged for two interviews.

Design and data collection techniques

An integrative knowledge synthesis using mixed methods was carried out using analysis of a cross-sectional survey and qualitative exploration using in-depth interviews. This analysis used data from two study components done by the investigators, one on caregivers of palliative care patients and one on palliative care nurses. Table  1 summarizes the participant profile and data collection techniques for each study component.

Data collection from caregivers

The caregiver survey and interviews took place between January and February 2020. The investigators collected data for a study on developing and validating a Caregiver Burden Inventory in early 2020, published earlier [ 8 ]. The portion of that data used here comprised three in-depth interviews (IDI) with caregivers of palliative care patients and cross-sectional survey data of caregiver burden and related issues of 221 caregivers in five randomly selected panchayats in Kollam district, Kerala, India. This analysis focused on a comparison of findings of the cross-sectional survey on the caregivers aged above 60 years with younger or middle-aged caregivers aged between 18 and 59 years. All family caregivers of patients registered under the palliative care programme, aged 18 and above, who identified themselves as the primary caregivers and are providing care for not less than three months were included in the study. Those caregivers having a condition that limits their participation in the study and those caring for a critically ill care recipient during the study period are excluded from the study. An interview schedule was used to collect the sociodemographic information, care recipient and caregiver issues, and caregiver burden based on the Achutha Menon Centre Caregiver Burden Inventory, a nine-item inventory for assessing caregiver burden that had two domains– (i) physical, psychological, and spiritual aspects and (ii) financial aspects. Each item was scored on a 4-point Likert scale from zero to three. A caregiver could potentially score between zero (lowest possible burden level) and 27 (highest possible burden score). Quality of life also was assessed using the Malayalam version of the EuroQol EQ-5D 5-level version (EQ5D5L) [ 29 ]. We used the EQ-5D-5L Indian value set to convert responses to utility values [ 30 ]. The EQ-5D-5L is a widely accepted five-dimension HRQoL measure that covers mobility, self-care, usual activities, pain, anxiety/depression, and overall health state. It is easy to apply in younger and older populations and persons with less education [ 31 ]. It has good psychometric properties and the index values and dimensions have been found to strongly correlate with other measures of global health indicators, physical/functional health, pain, daily activities, and clinical/biological variables [ 32 ].

Data collection from palliative care nurses

The researchers were part of a team working on decentralization and health in Kerala, in which one of the researched themes was the primary palliative care programme [ 33 ]. One of the themes selected for enquiry was caregiver issues. Five primary palliative care nurses (Table  1 ) with at least one year experience were purposively selected and interviewed to get an insightful account of their experiences with caregiver issues. Interviews were conducted telephonically due to COVID-19-related restrictions in 2020 and early 2021.

Data analysis

To assess the validity of the EQ-5D-5L, we performed internal consistency checks and factor analysis of the five items of the EQ-5D-5L for the whole sample and the two age groups of interest separately (up to 59 years and 60 years and above). We extracted one factor from observed item values using principal axis factoring with direct oblimin rotation and correlated it with the utility scores obtained from the Indian value set of the EQ-5D-5L.

For the quantitative data, general characteristics and caregiver issues were summarised as frequencies and proportions or means and standard deviations, along with 95 per cent confidence intervals. Burden scores were converted to a categorical variable using tertiles, and labelled as low, moderate and high burden. Chi-square or Fisher exact tests were done to compare proportions. Analysis of variance (ANOVA) and posthoc Bonferroni tests were done to compare means. IBM SPSS version 25 was used for the quantitative analysis. Qualitative analysis was done manually.

All recordings of the IDIs were translated to English and initially coded by the same researcher (AK) who maintained an audit trail to map the interview transcripts and related codes to categories and themes. The approach to coding and categorising was inductive for the caregiver interviews and deductive for the palliative care nurse interviews. Information extracted from the literature review was used to generate a codebook for qualitative analysis to portray caregiver issues and perspectives of the nurse. The search was limited to articles in English, and title and abstract mention of caregiver issues along with provider perspective. Both investigators reviewed the shortlisted papers, and prepared codes, categories and themes through an iterative process. Existing codes were verified and additional codes, if any, were explored through triangulation with transcripts from caregiver issues mentioned by palliative care nurses in the main decentralization study. (See Additional file 1 ) After describing the findings based on this approach, we referred to Eva Kittay’s critique of Daniels and Nussbaum, based on the burden of caregiving and its effect on the caregiver’s opportunities while interpreting our findings from the study [ 35 ].

Subjectivities of the researchers

AK conducted all interviews and both investigators were involved in the analysis and interpretations. Both investigators hold basic biomedical degrees and subsequently public health qualifications. The research experience of both researchers has been predominantly post-positivist. We believe that our experiences around epidemiological surveys would have shaped the data collection and interpretations in a predominantly biomedical perspective with some consideration of social determinants shaped by our experience level. However, our ongoing engagement with palliative care and caregivers’ issues also brings in some relational approaches and interpretations characteristic of literature on caring.

Ethical aspects

All prospective study participants were assured of their autonomy, benefits and risks, privacy and confidentially and non-effect on care or benefits before obtaining informed consent. Informed consent, written or electronically documented, was obtained from all study participants. The Institutional Ethics Committee of the Sree Chitra Tirunal Institute for Medical Sciences and Technology, Trivandrum cleared all tools of the scale development phase. (Letter number SCTIMST/IEC/1444/NOVEMBER-2019 dated 14 November 2019). The proposal and tools of the palliative care nurse interviews, part of the decentralization project, were reviewed and cleared by the institutional ethics committee of Health Action by People Thiruvananthapuram. (IEC EC2/P1/SEP/2020/HAP dated 10 December 2020). While these were originally independent studies, clearance was obtained from the Institutional Ethics Committee of the Sree Chitra Tirunal Institute for Medical Sciences and Technology, Trivandrum (Letter number SCTIMST/IEC/2048/MAY-2023 dated 17 June 2023) for a synthesis exercise as part of formative research for a forthcoming study on caregiver burden assessment and intervention.

Validity of the EQ-5D-5L in our study sample

We report the Cronbach’s alpha for internal consistency, the eigen value for the extracted factor, the factor loadings of the extracted factor onto each item of the EQ-5D-5L and Pearson’s correlation coefficient between the extracted factor and utility scores in Table  2 . Internal consistency was moderate to good, eigenvalue was more than one and there was a high correlation between the factor derived from observed values and utility score values taken from the India value set. Factor loadings for pain/ discomfort and anxiety/ depression were relatively higher in the younger age group while for usual activities, factor loadings were higher in the older caregiver group.

Findings from cross-sectional survey among caregivers

Palliative care recipients had various diagnoses ranging from stroke (23.9%), to cancer (12.8%) followed by other conditions. The mean age of the caregivers was 51.2 years (Standard Deviation (SD 12.7). The mean age of the older group was 66.2 years (SD 7.1) and of younger or middle-aged caregivers was 45.3 (SD 9.0). Caregiver ages ranged from 25 to 88 years. Demographic characteristic of the caregivers according to their age category is given the Table  3 . Most caregivers were women, but in the older age group, the proportion of men was significantly higher. Older caregivers were significantly less educated and less likely to be married, but the social class was comparable.

Table  4 depicts the distribution of variables related to caregiving. Nearly all caregivers in both groups were the sole caregiver for their care recipient. A significantly higher proportion of older caregivers were giving care to their spouses. Care requirements were significantly higher for the care recipients of younger caregivers, but most other variables were comparable. A higher proportion of older caregivers reported being satisfied with their caregiving activities.

Older caregivers reported poorer states for all variables related to self-reported morbidity and quality of life attributes measured using the EQ5D5L, except for self-care. (Table  5 ) Nearly three-fourths of older caregivers reported mobility issues; over half had pain or felt anxious or depressed.

The mean EQ-5D-5L utility score for the caregivers was 0.936 (SD 0.078, 95% CI 0.926–0.947). On comparing the caregiver’s age and burden experienced with the utility score, we found that the burden level impacted the perceived quality of life, irrespective of the caregiver’s age. As shown in Fig.  1 , younger caregivers generally had a better quality of life than older caregivers, and those with low caregiver burden had better utility scores than those with moderate to high levels of caregiver burden. Younger caregivers who perceived a high burden level had lower mean utility scores (0.926, SD 0.090, 0.907–0.945) than younger caregivers who perceived a low burden (0.980, SD 0.041, 0.970–0.990). Likewise, older caregivers who perceived a higher burden level had a lower mean utility score (0.877, SD 0.066, 0.854–0.899) than their counterparts with a low burden (0.935, SD 0.058, 0.912–0.958). Except for the difference in means between older caregivers with low burden and younger caregivers with moderate to high burden, all mean differences were statistically significant. ( p  < 0.001)

Means and 95% confidence intervals of EQ5D5L utility scores for caregivers grouped based on age category and burden level

Table  6 maps the support these dyads received regarding palliative care nurse visits, assistive devices, food kits or support from non-governmental charitable organisations. The frequency of nurse visits (monthly or above) was determined almost exclusively by patient need and was not associated with caregiver burden level. Among other forms of support, receiving food kits from the LG was found to be significantly higher when high levels of caregiving burden were present.

Themes from in depth interviews with caregivers

“i do everything for her/him”.

All caregivers mentioned doing “everything” for the care recipient, including all activities of daily living, medications, and procedures like skin care.

CG1: “I do everything for her…I bathe her… take her to the toilet…help her to change her dress…. Give her food. Everything…”. CG2: “I’ve cared for my husband for the last 10 years. he is entirely dependent on me… everything…I clean him…bathe him… give him food…everything”.

Caregiving became physically and psychologically demanding

Doing “everything” involved physically demanding activities that reportedly led to chronic body pain for the caregiver.

CG2: “…a constant pain on my legs… I always lift him alone, there will not be anybody home…”.

Other issues mentioned included sleep deprivation, financial and job-related issues, and limitations to social participation due to caregiving. Care recipients could also have a temperament that made caregiving challenging.

CG2: “He is always very angry. He always shouts at me and my son… I’m always worried… I do not know what to do…”.

Care team is only patient-focused, caregiver issues are not addressed

The palliative care team when they visit would do patient-centred procedures, dispense medicines, and provide advice for improving patient care.

CG3: “…people from health (services) come once a month and change the urine tube…. They give medicines also…”.

Some advice provided could not be implemented, often due to affordability issues.

CG2: “…they give instructions about how to do physiotherapy…but it is no use…once in a month we used to call a physiotherapist…but it is expensive…”.

Themes from palliative care nurse interviews

We shortlisted 17 articles for further analysis. Four were from Kerala and the rest were from outside India. Caregiver issues highlighted included burden, burnout, and health and wellbeing-related issues. Four themes on the care provider perspective were initially decided upon, namely: (i) exposition of caregiver burden by providers (ii) nature of family caregiver-health provider relationships (iii) factors that enable or hinder caregiver support from providers (iv) specific interventions that foster caregiver endurance.

Each provider interview took about 40 min, ranging from 35 to 50 min. Open codes from documents were binned into existing categories in the schema or new categories were added, if felt necessary. (See Additional file 1 ) No codes fell into the theme “specific interventions that foster caregiver endurance”. Brief descriptions of the findings were as follows:

Accurate exposition of the caregiver burden by palliative care nurses

All nurses highlighted the “burden” experienced by the family caregivers, mainly expressed as socioeconomic deprivation and challenges.

“Issues like no secure house, no food due to lack of income… patients who cannot buy expensive medication and continue their treatment… bystanders struggling for their children’s education…” (PN2– when reporting quotes abbreviation PN indicates participant attribute - palliative care nurse). “They talk about the difficulties of not being able to go to work leaving their Amma (mother)” (PN1).

Added to this were disruptions and conflicts that the caregivers must handle along with the caregiving role.

Caregivers cannot sleep, they cannot look after their home and other household works, they cannot do their own activities like taking care of children (PN1).

Nurses often found themselves encountering conflicts, either between the caregiver and the patient or among family members taking the main caregiver responsibility. Sometimes patient behaviours were distressing for caregivers.

Sometimes patients will be so “violent” because of their condition; sometimes the patient’s condition is so bad… This also reflects on the caregivers. This affects them and they may also become frustrated. (PN1)

The caregiver role often limited the caregivers to their homes and restricted their social life. Societal perceptions of caring often deepened this social restriction. Nurses clearly described difficulties associated with long-term caregiving including physical pain, psychological distress, individual life disruptions, economic, and social challenges. Some caregivers had become sick from the long haul of physical exhaustion.

I know caregivers like these…so desperate and hopeless… (PN5)

Nurses also felt that caregivers often neglect their well-being and prioritise their patient’s care.

Disparate relationships between caregivers and health providers and the system

Nurse representations of caregiver-provider relationships were complex, ranging from excellent cordiality to open conflicts. Nurses were at times “being like a family member” and at other times involved in verbal altercations and in extreme situations, involvement of law enforcement when neglect of the care recipient was perceived. A consistent part of the relationship, however, was the instrumental contribution expected from the caregiver in caring for the care recipient. Family caregivers were taken for granted as resource persons for caring for the patient and interactions mostly involved general instructions on caregiving or specific training for skin care, wound care, or catheter care. Some task-shifting often happened from the nurses to capable caregivers.

“We made them do these in front of us… The caregiver has taken care of the patient so well.” (PN1, mentioning an example of caregiver education for wound dressing).

Referral for palliative care itself might be perceived by family members as further care was largely up to themselves. It would often take multiple visits to discern all such concerns.

“…they also share their concerns… as palliative (is understood as) end-of-life care…so these makes them worried…” (PN5).

The first time they won’t say everything… after numerous visits, they tell us everything (PN3)

When disagreements were encountered, nurses tried to resolve them by working for a healthy relationship between the caregiver and the care recipient. A somewhat stereotypical portrayal of caregiving emerged in the discourse, where caregiving was a moral imperative of the family, often women. The “best” caregivers were those who fulfilled this expected role well.

“I strongly believe that we should take care of our own parents” (PN2).

“There are no issues or problems for caregivers who are not working” (PN2)

“She is a widow…has two kids…the patient is her late husband’s mother…she (caregiver) is working… she does everything for her patient; only after that she leaves for work… When we visit the patient…it’s so clean and we never feel it’s a room of a bedridden patient…there are caregivers like this” (PN3).

Some caregivers were hesitant to build relationships with palliative care nurses. Nurses too might choose against investing time and visits for getting better acquainted with the caregiver. Caregivers who were demanding and making decisions independent of the nurse were considered problematic.

“They (caregivers) “torture” us by making calls to the panchayat member (the elected LG representatives who helm the programme)…” (PN4).

Families perceived as neglecting the care recipient were labelled as outright problematic. At times, nurses tend to establish an authoritarian role in such instances.

“I say to them if you did not take care of your parents, your seven generations will suffer…” (From additional codes as indicated in the additional file, said by a nurse based on the spiritual belief on results of bad deeds being passed on to future generations) (See Additional file 1 ). “I say, “If you didn’t take care of them, I will inform to (the elected LG representatives) and doctor…If… your mother is lying in (urine and faeces), then you will be taken by police” (From additional codes) (See Additional file 1 ).

But palliative care nurses were often the first in the health system to recognize patient negligence and abuse by the family.

Caregivers who followed their instructions well and include nurses in treatment-related decisions were considered dependable. Yet, once good communication and rapport were established, caregivers often began to consider the nurse “like family” and this was highly valued by nurses, who mentioned several “friendships” that continued long after the death of the patient.

“(When her) daughter (finished school) she (caregiver) asked me which (field of education) is good for her daughter… now, following my advice, the daughter is doing nursing in the district hospital.” (PN3).

Systemic factors often hinder caregiver support

By systemic factors, we mean programmatic focus on the patient, lack of training, lack of time and limited attention to support schemes involving caregiver issues and burden. As such, there were no caregiver-specific initiatives or systematic documentation of caregiver issues. Caregiver support when existed was reactive rather than proactive. Caregivers were mostly given instructional support and/ or instrumental assistance for aiding patient care like medicines, cotton pads, gauze, catheters, Ryle’s tubes, or mobility aids. Communication and consoling were perceived as the main form of intervention by palliative care nurses.

“Their (caregivers) blood pressure will increase because of this lack of sleep. So, during our home visit we will check their BP also…” (PN1).

However, nurses informed eligible caregivers and families about beneficial schemes (‘ Ashwasakiranam ’, a state government-initiated financial assistance scheme for primary caregivers of palliative patients with cancer) or helpful charity organizations, if any.

Lack of time was the main impediment in addressing caregiver issues. Additionally, inadequate training and resources for giving caregiver support were also mentioned. Nurses suggested some systemic failures in recognizing the medical and social issues of caregivers.

“Some of the caregivers, have issues like CKD (chronic kidney disease), cancers or heart problems, but we cannot register them with the palliative care programme.” (PN5).

The main LG support specifically mentioning caregivers was the annual Kudumbasangamom (family gathering) with some recreational programmes, that too in the pre-pandemic days. Some LGs had schemes for self-employment generation for patients or caregivers, to make some products that could be sold for money. LGs support for hosting such schemes was patchy.

“But there was no adequate support from our panchayat for selling their product or purchasing the raw materials…no support for promoting these initiatives.” (PN1).

In this mixed methods study, we attempted to compare caregiver issues between older and younger caregivers in the palliative care program in Kerala. We also tried to document provider-side perspectives on family caregiver issues as articulated by palliative care nurses. The family caregiver issues we identified included physical, psychological, social, and financial issues, much like those reported by Ferrell and Wittenberg in their review of family caregiver trials in cancer patients [ 36 ]. As expected, older caregivers were more susceptible to health-related problems at this age. Irrespective of age, those who experienced a higher burden level had poorer quality of life. When combined, with higher burden experience, older caregivers had the poorest quality of life. This might be brought on by the physical demands of providing care as well as the ageing process’s effects on health.

The absence of any specific service or programme that enables caregiver endurance or any mention of systematic documentation of caregiver issues is a programmatic shortcoming. Nurses gave more attention to patients with skilled care needs and the level of caregiver burden was probably not a factor in determining their visits. Nurses’ tendency for “non-inviting interactions” with family members of patients, by prioritising medical and technical tasks, has been reported earlier from institutional settings [ 37 ]. But nurses recognised most caregiver issues and mentioned insufficient time to address them. Healthcare providers in similar programmes may not even have time for meeting their personal needs due to work demands [ 19 ]. Nurse perceptions about caregiving-related challenges mentioned social determinants of health but also mirrored prevalent socio-cultural and patriarchal norms. Family caregiver-centric studies are rare from LMIC, but available studies reflected socioeconomic deprivation and intense gender-role-driven concentration of caregiving in women [ 38 ]. Nurses however actively tried to improve the family caregivers’ skills in caregiving. This is important to prevent and delay burnout [ 39 ]. Additionally, they provide psychological support, often bonding well with caregivers long after they are bereaved [ 40 ]. Receiving interventions like food kits was significantly higher when the perceived caregiver burden was high. Caregiver burden is multi-dimensional and includes financial difficulties [ 8 ]. LGs generally focus more on the poorest and this finding is expected. The interventions remain basic, but it is promising that LGs can prioritise families with high caregiver burdens for interventions.

Poor households might disproportionately access the LG-run palliative care service, as the services are free of cost. Such households may already have high burden due to pre-existing structural and social disadvantages. Yet, even if caregiving was not causal for the problems expressed, the perceived burden would still be detrimental to quality of life. The directive principles of state policy of the constitution of India clearly list the fundamental rights of citizens and the responsibility of the state to protect citizens unable to access the minimal provisions for social and economic well-being. These principles also mention the autonomy of LGs [ 41 ]. It is thus a moral requirement of the LG-run palliative care programme to focus on the needs of families in addition to the patients.

Our findings draw attention to an important element of long-term care that is somewhat neglected– caregiver impact. Caregiving is a moral responsibility between individuals and at the collective level, as all individuals need care and are dependent at some point in their lives. But caregiving is a mix of reward and burden. Caregivers remain seen as a means to an end when in reality the caregiver is also an end in herself or himself. Allocation of caregiving responsibility is heavily gendered, rendering it as a form of inequity. Potential disadvantages of women may get compounded when she gets restricted to the caregiver role– lesser education, or work opportunities, and often treated as if she is unemployed or not doing economically productive work– leading to depression and a low sense of worth. Another aspect of caregiving that has implications for equity is the way society often works, based on normative or normal people. This may become unfair to suffering people as well as their caregivers, and the burden may be considered inevitable. The family caregiver is not a biological extension of the care recipient’s situation, to be moulded to sustain the biological functions of the care recipient. Neither is caregiving by a family member a law of nature that cannot be changed. This is a situation shaped by relationships between people and societies and the values and practices thereof. Moral requirements of caregiving should also consider what is lost to a caregiver and provide respect for the caregiver. Solutions may be explored by forming partnerships between the caregiver and others and by tapping into existing community resources. This has to happen without diminishing the relationship between the caregiver and the care recipient [ 26 ].

Norman Daniels proposes a lifespan approach of justice that may be useful to consider in this setting [ 42 ]. As individuals get older, their needs changes. When the society itself in an ageing society, that too brings in a new set of needs. In such a situation, reasoning has to be applied on how competing needs are to be met. Competing needs would be between different age groups or between care recipients and those giving care. Some needs would inevitable not be met when social obligations are to be met, but there should be fairness in the terms involved, and adequate social support to prevent issues like burnouts. Identifying beneficial interventions will remain an ethical challenge due to three aspects: (i) the vulnerability of the care recipient should not be exploited (Daniels); (ii) the voice of the caregiver has to be used for meeting the needs of the care recipient, as the capabilities of the latter have diminished (Kittay); (iii) the caregiver too has interests that would often be diminished (Kittay). The caregiver burden is disproportionately a woman’s issue because most of the caregiving work is rendered by women, many of whom are older persons. Discussions of fairness and equity often focus on fair distribution of goods like education and health. As Kittay points out in response to Norman Daniels and Nussbaum, conventional approaches to justice focusing on fair sharing of goods and aiming for equality of opportunity or capability do not talk about fair sharing of burden. In ageing societies, considerations of the distribution of burden may be as important as the distribution of goods.

The CARE framework refers to caregivers as “hidden patients” and recommends a framework comprising Caregiver well-being, Advanced care planning, Respite, and Education for planning to address caregiver issues [ 43 ] The first attribute in addressing family caregiver-related issues is an assessment of need. Symptom severity of care recipients, marginalized families and caregivers with significant psychosocial issues have been suggested as potential indicators of high caregiver issues [ 44 , 45 ]. The deployment of tools like carer support needs assessment tool might help identify support needs and decrease caregiver strain [ 46 ]. Newer modalities like an app-based assessment are being tested in Sweden for family caregivers of patients with dementia [ 47 ]. Examples of successful caregiver interventions from LMIC countries are generally few. The trials covered in the review by Ferrell and Wittenberg were mostly from high-income countries [ 36 ]. In New Zealand three themes of advice for caregivers were considered most useful by providers– caring for oneself physically, emotionally, and spiritually; learning practical skills; and knowing what to expect and plan for as the family member’s health declines [ 48 ]. Researchers from the Netherlands recommended appreciation, information, practical support, and opportunities for time off (like respite care) as useful to lessen caregiver problems [ 49 ]. An intervention based on group sessions for caregivers in South Korea also showed promising physical and psychological outcomes [ 50 ].

Most of these examples are based on individual-level interventions. Krieger et al. reported the need for comprehensive caregiver support at two levels– the individual caregiver level, and the system level [ 51 ]. The United States of America (USA) has had several legislative and programmatic structures aimed at minimizing caregiver distress [ 52 ]. Caregivers of veterans in the USA have specific support like training, financial support, and assistance of a caregiver support coordinator, although Zebrak mentions about the lack of coordination between such policies [ 53 , 54 ]. The National Health Service in the United Kingdom has some specific measures to support caregivers [ 55 ]. The National Institute for Health and Care Excellence, UK has included an assessment of caregivers’ quality-of-life in economic evaluation in its health technology evaluation manual published in January 2022 [ 56 , 57 ].

The primary palliative care programme in Kerala is run by the LGs with support from the health department. Each LG unit sets aside resources from its annual fund allocation to support the wages of the palliative care nurse, travel costs, and costs of equipment, materials, and drugs for home-based care. Additional community-based resources are also mobilised by some LGs. In Kerala, the decentralized health system and the agency available with LGs for extending welfare measures to the needy using locally identified resources offers promise for good interventions [ 6 ]. Caregiver training and certification could be done, and a list of authorised paid caregiver schemes could be piloted, with efforts to include men in the initiative [ 58 ]. Facilities for respite care [ 59 ] may offer some personal space and time for caregivers, or additional appropriate practical help [ 60 ] could be offered. Building the competency of caregivers could extend to self-care in addition to patient care [ 39 ]. The formation of caregiver peer groups could be another intervention that facilitates information sharing, coping and increased social interactions [ 39 , 61 ]. A specialist support service like a caregiver support coordinator or group could be initiated by the district-level health structures of the National Health Mission or the LG or by NGOs. Despite limited evidence of the success of such interventions on a large scale, it is useful to remember the economic value of family caregivers to the health system and community [ 45 ].

Limitations

The quantitative data being cross-sectional, the temporality of the associations we saw cannot be ascertained. Poor health may cause poor quality of life and that may precipitate caregiver burden rather than burden resulting in poor quality of life. However, the implication for the health system remains somewhat the same– poor health, poor quality of life and high caregiver burden need attention whatever the order of their occurrence. Another limitation of the study is the lack of direct interaction with palliative nurses due to COVID-19-related restrictions. The interviews were possibly influenced by the previous experience of the researchers on caregiver issues. Physical visits to the settings and interactions with a wider group of stakeholders from the health department, the LG and other community representatives would have provided richer descriptions of caregiver issues and more quintessential details of caregiver-provider/ system interactions. At the analysis stage, we did not do a multivariable analysis to account for potential confounding or effect modification as data were not primarily collected to explore these aspects. The largely deductive qualitative analysis based on a priori themes is another limitation. As our focus was on validating our literature-generated construct of caregiver burden, we did not explore the experiences of elderly caregivers at that stage of the study and this is a drawback of this synthesis. Yet, we feel that our findings offer some insights that can be used to inform future research in this area.

Caregivers aged 60 years or above made up three out of ten caregivers, with over half caring for their spouse, in this study setting. This is one of the first studies using Indian values of EQ5D5L utility scores for studying the quality of life of caregivers. Older caregivers reported a poor health-related quality-of-life and were experiencing a dual burden of caregiving and poor health, also having chronic health issues needing to take care of others while having to take care of others. The complex dynamics of caregiving by elderly caregivers have not been explored much, suggesting opportunities for future studies to explore these issues and develop targeted interventions for their specific needs. Potential interventions could be Respite care and support services for older caregivers that could offer temporary relief and help caregivers take breaks from caregiving responsibilities. Peer support groups could be another approach that can help caregivers to cope better with the burden. Also, comprehensive geriatric health and wellness programmes encompassing preventive, promotive, curative, rehabilitative and palliative care that jointly cater to patients and caregivers together are needed in settings with high ageing and chronic health conditions.

Data availability

The corresponding author will provide the transcripts, data set and analysis of this current work on reasonable request.

Abbreviations

Analysis of variance

Five level EQ-5D version of EuroQol

In-depth interviews

Low- or Middle-Income Countries

Local Government

Palliative Nurse

Standard Deviation

United Kingdom

United States of America

Daniels N. Just health: Meeting health needs fairly. New York: Cambridge University Press; 2007. p.13.

Ugargol AP, Bailey A. Family caregiving for older adults: gendered roles and caregiver burden in emigrant households of Kerala, India. Asian Popul Stud. 2018;14:194–210.

Article   Google Scholar  

Abdul Azeez EP, Anbu Selvi G. What determines the sustainability of community-based palliative care operations? Perspectives of the social work professionals. Asian Soc Work Policy Rev. 2019;13:334–42.

Philip RR, Philip S, Tripathy JP, Manima A, Venables E. Twenty years of home-based palliative care in Malappuram, Kerala, India: a descriptive study of patients and their care-givers. BMC Palliat Care. 2018;17:26.

Article   PubMed   PubMed Central   Google Scholar  

Sallnow L, Kumar S, Numpeli M. Home-based palliative care in Kerala, India: the neighbourhood network in palliative care. Prog Palliat Care. 2010;18:14–7.

Anju R, Sadanandan R, Vijayakumar K, Raman Kutty V, Soman B, Ravindran RM, et al. Decentralisation, health and sustainable development goal 3. Public Health Action. 2023;13(Suppl 1):51–6.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Reinhard SC, Given B, Petlick NH, Bemis A. Supporting Family Caregivers in Providing Care. In: Hughes RG, editor. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008. http://www.ncbi.nlm.nih.gov/books/NBK2665/ Accessed 15 Mar 2023.

Kochuvilayil A, Varma RP. Caregiver Burden among Informal caregivers in the Kerala Palliative Care Program: Development and Validation of the Achutha Menon Centre-Caregiver Burden Inventory. J Palliat Med. 2021;24:1197–205.

Article   PubMed   Google Scholar  

Oldenkamp M, Hagedoorn M, Slaets J, Stolk R, Wittek R, Smidt N. Subjective burden among spousal and adult-child informal caregivers of older adults: results from a longitudinal cohort study. BMC Geriatr. 2016;16:208.

Roth DL, Haley WE, Hovater M, Perkins M, Wadley VG, Judd S. Family caregiving and all-cause mortality: findings from a population-based propensity-matched analysis. Am J Epidemiol. 2013;178:1571–8.

Schulz R, Beach SR. Caregiving as a risk factor for mortality: the Caregiver Health effects Study. JAMA. 1999;282:2215–9.

Article   CAS   PubMed   Google Scholar  

Perkins M, Howard VJ, Wadley VG, Crowe M, Safford MM, Haley WE, et al. Caregiving strain and all-cause mortality: evidence from the REGARDS study. J Gerontol B Psychol Sci Soc Sci. 2013;68:504–12.

Kochuvilayil A, Varma RP. Factors Associated with Screening positive for Depression among women caregivers of primary Palliative Care patients in Kerala, India. J Palliat Care. 2022;37:510–17.

Govina O, Vlachou E, Kalemikerakis I, Papageorgiou D, Kavga A, Konstantinidis T. Factors Associated with anxiety and depression among family caregivers of patients undergoing Palliative Radiotherapy. Asia Pac J Oncol Nurs. 2019;6:283–91.

Srivastava G, Tripathi RK, Tiwari SC, Singh B, Tripathi SM. Caregiver Burden and Quality of Life of Key Caregivers of patients with dementia. Indian J Psychol Med. 2016;38:133–6.

Devassy SM, Green D, Cheguvera N, Yohannan SV, Grills N, Joubert L. The lived experience of carers who assist people with disability in Ernakulam, Kerala, India. Disabil Rehabil. 2022;44:6660–7.

Usha K. PA26 unmet needs and stress among caregivers of bedridden stroke patients in north Kerala - a community based study. BMJ Support Palliat Care. 2015;5(Suppl 1):A27.

Krutter S, Schaffler-Schaden D, Essl-Maurer R, Wurm L, Seymer A, Kriechmayr C, et al. Comparing perspectives of family caregivers and healthcare professionals regarding caregiver burden in dementia care: results of a mixed methods study in a rural setting. Age Ageing. 2020;49:199–207.

Useros MV, Espín AA, Parra EC, Martínez AB. Family caregivers: nurses’ perception and attitudes. Social Med. 2012;6:151–61.

Google Scholar  

Kerala State Planning Board: Economic Review. 2021. 2022. https://spb.kerala.gov.in/sites/default/files/2022-03/ECNO_%20ENG_21_%20Vol_1.pdf . Accessed 15 Mar 2023.

Neri AL, Yassuda MS, Fortes-Burgos AC, Mantovani EP, Arbex FS, de Souza Torres SV, et al. Relationships between gender, age, family conditions, physical and mental health, and social isolation of elderly caregivers. Int Psychogeriatr. 2012;24:472–83.

Pati S, Agrawal S, Swain S, Lee JT, Vellakkal S, Hussain MA, et al. Non communicable disease multimorbidity and associated health care utilization and expenditures in India: cross-sectional study. BMC Health Serv Res. 2014;14:451.

Lavela SL, Ather N. Psychological health in older adult spousal caregivers of older adults. Chronic Illn. 2010;6:67–80.

Tough H, Brinkhof MWG, Siegrist J, Fekete C, SwiSCI Study Group. Social inequalities in the burden of care: a dyadic analysis in the caregiving partners of persons with a physical disability. Int J Equity Health. 2019;19:3.

Melo LA, Jesus ITM, Orlandi FS, et al. Frailty, depression, and quality of life: a study with elderly caregivers. Rev Bras Enferm. 2020;73Suppl(3):e20180947.

World Health Organization: Ethical choices in long-term care: What does justice require? Pp 19. Geneva: World Health Organization. 2002. https://apps.who.int/iris/bitstream/handle/10665/42614/9291562285.pdf;sequence=1 Accessed 15 Mar 2023.

Jika BM, Khan HT, Lawal M. Exploring experiences of family caregivers for older adults with chronic illness: a scoping review. Geriatr Nurs. 2021;42:1525–32.

Government of Kerala: Kerala State Policy on Palliative Care. 2019. https://palliumindia.org/wp-content/uploads/2020/05/Kerala-State-Palliative-Care-Policy-2019.pdf Accessed 15 Mar 2023.

Herdman M, Fox-Rushby J, Rabin R, Badia X, Selai C. Producing other language versions of the EQ-5D. In: Brooks R, Rabin R, de Charro F, editors. The measurement and valuation of health status using EQ-5D: a European perspective. Dordrecht: Kluwer; 2003. pp. 183–90.

Chapter   Google Scholar  

Jyani G, Sharma A, Prinja S, Kar SS, Trivedi M, Patro BK, et al. Development of an EQ-5D value set for India using an Extended Design (DEVINE) study: the Indian 5-Level version EQ-5D Value Set. Value Health. 2022;25:1218–26.

Marten O, Brand L, Greiner W. Feasibility of the EQ-5D in the elderly population: a systematic review of the literature. Qual Life Res. 2022;31:1621–37.

Feng YS, Kohlmann T, Janssen MF, Buchholz I. Psychometric properties of the EQ-5D-5L: a systematic review of the literature. Qual Life Res. 2021;30:647–73.

Kochuvilayil A, Rajalakshmi S, Krishnan A, Vijayanand SM, Kutty VR, Iype T, et al. Palliative care management committees: a model of collaborative governance for primary health care. Public Health Action. 2023;13:12–8.

Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60:34–42.

Kittay EF. Can contractualism justify state-supported long-term care policies? Or, I’d rather be some mother’s child. A reply to Nussbaum and Daniels. Ethical choices in long-term care. What Does Justice Require. 2002:77–83.

Wittenberg E, Buller H, Ferrell B, Koczywas M, Borneman T. Understanding Family Caregiver Communication to provide family-centered Cancer Care. Semin Oncol Nurs. 2017;33:507–16.

Söderström IM, Benzein E, Saveman BI. Nurses’ experiences of interactions with family members in intensive care units. Scand J Caring Sci. 2003;17:185–92.

Kipp W, Chacko S, Laing L, Kabagambe G. Adolescent reproductive health in Uganda: issues related to access and quality of care. Int J Adolesc Med Health. 2007;19:383–93.

Solli H, Hvalvik S. Nurses striving to provide caregiver with excellent support and care at a distance: a qualitative study. BMC Health Serv Res. 2019;19:893.

Salin S, Kaunonen M, Astedt-Kurki P. Nurses’ perceptions of their relationships with informal carers in institutional respite care for older people. Nurs Res Pract. 2013;2013:967084.

PubMed   PubMed Central   Google Scholar  

Choudhry S, Khosla M, Mehta PB. The Oxford Handbook of the Indian Constitution. 1st ed. Oxford: Oxford University Press; 2016.

Book   Google Scholar  

Daniels N. Just health: Meeting health needs fairly. New York: Cambridge University Press; 2007. p.162.

Holliday AM, Quinlan CM, Schwartz AW. The hidden patient: the CARE framework to care for caregivers. J Family Med Prim Care. 2022;11:5–9.

Valero-Cantero I, Wärnberg J, Carrión-Velasco Y, Martínez-Valero FJ, Casals C, Vázquez-Sánchez MÁ. Predictors of sleep disturbances in caregivers of patients with advanced cancer receiving home palliative care: a descriptive cross-sectional study. Eur J Oncol Nurs. 2021;51:101907.

Hudson PL, Thomas K, Trauer T, Remedios C, Clarke D. Psychological and social profile of family caregivers on commencement of palliative care. J Pain Symptom Manage. 2011;41:522–34.

Aoun SM, Grande G, Howting D, Deas K, Toye C, Troeung L, et al. The impact of the carer support needs assessment tool (CSNAT) in community palliative care using a stepped wedge cluster trial. PLoS ONE. 2015;10:e0123012.

Kabir ZN, Leung AYM, Grundberg Å, Boström AM, Lämås K, Kallström AP, et al. Care of family caregivers of persons with dementia (CaFCa) through a tailor-made mobile app: study protocol of a complex intervention study. BMC Geriatr. 2020;20:305.

Angelo JK, Egan R, Reid K. Essential knowledge for family caregivers: a qualitative study. Int J Palliat Nurs. 2013;19:383–8.

Bijnsdorp FM, Pasman HRW, Boot CRL, van Hooft SM, van Staa A, Francke AL. Profiles of family caregivers of patients at the end of life at home: a Q-methodological study into family caregiver’ support needs. BMC Palliat Care. 2020;19:51.

Han EJ, Park M, Park S, Giap TT, Han D. Randomized Controlled Trial of the Caregiver Orientation for Mobilizing Personal Assets and Strengths for Self-Care (COMPASS) for Caregiving Journey: A National Family Caregiver Support Program in a Long-Term Care Insurance System. J Am Med Dir Assoc. 2020;21:1906-13.e3.

Krieger T, Feron F, Dorant E. Two-level multi-methodological evaluation of a new complex primary support programme for stroke care-givers in Germany. Ageing Soc. 2022;42:1–31.

Rozario PA, Palley E. When the private sphere goes public: exploring the issues facing family caregiver organizations in the development of long-term care policies. Soc Work Public Health. 2008;23:49–68.

Smith VA, Lindquist J, Miller KEM, Shepherd-Banigan M, Olsen M, Campbell-Kotler M, et al. Comprehensive Family Caregiver Support and Caregiver Well-Being: preliminary evidence from a Pre-post-survey Study with a non-equivalent Control Group. Front Public Health. 2019;7:122.

Zebrak KA, Campione JR. The Effect of National Family Caregiver Support Program Services on Caregiver Burden. J Appl Gerontol. 2021;40:963–71.

National Health Service: Support and benefits for carers. (2018) https://www.nhs.uk/conditions/social-care-and-support-guide/support-and-benefits-for-carers/ Accessed 20 Mar 2023.

Pennington BM. Inclusion of Carer Health-Related Quality of Life in National Institute for Health and Care Excellence appraisals. Value Health. 2020;23:1349–57.

Dawoud D, Lamb A, Moore A, Bregman C, Rupniewska E, Paling T, et al. Capturing what matters: updating NICE methods guidance on measuring and valuing health. Qual Life Res. 2022;31:2167–73.

Cross ER, Emanuel L. Providing inbuilt economic resilience options: an obligation of comprehensive cancer care. Cancer 2008113(12 Suppl):3548–55. https://doi.org/10.1002/cncr.23943 .

Roberts E, Struckmeyer KM. The impact of Respite Programming on Caregiver Resilience in Dementia Care: a qualitative examination of Family Caregiver perspectives. Inquiry. 2018;55:46958017751507.

PubMed   Google Scholar  

Simon C, Kumar S, Kendrick T. Who cares for the carers? The district nurse perspective. Fam Pract. 2002;19:29–35. https://doi.org/10.1093/fampra/19.1.29 .

Greenwood N, Habibi R, Mackenzie A, Drennan V, Easton N. Peer support for carers: a qualitative investigation of the experiences of carers and peer volunteers. Am J Alzheimers Dis Other Demen. 2013;28:617–26.

Download references

Acknowledgements

The authors thank the Department of Health and Family Welfare, Government of Kerala, the Local Self Government Department of the Government of Kerala and the Panchayats for granting permission to undertake the study.

AK received partial financial support from the project Local Government and Health in Kerala, implemented by Health Action by People (HAP), Thiruvananthapuram, Kerala, for conducting the palliative care nurse interviews. The Local Government and Health project was supported by the Health Systems Transformation Platform, through a financial contribution from the Sir Ratan Tata Trust. The funders had no role in data collection and analysis or preparation of the manuscript.

Author information

Authors and affiliations.

Achutha Menon Centre for Health Science Studies, Sree Chitra Tirunal Institute for Medical Sciences and Technology, Trivandrum, Thiruvananthapuram, Kerala, India

Arsha Kochuvilayil & Ravi Prasad Varma

You can also search for this author in PubMed   Google Scholar

Contributions

AK conducted the interviews, undertook the analysis and wrote the first draft of the manuscript. RPV supervised the work and contributed to the conceptualization, design, analysis and revision of the manuscript. Both authors have reviewed and approved the manuscript in its present form including the revisions.

Corresponding author

Correspondence to Ravi Prasad Varma .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethics approval and consent to participate

All study participants provided written or electronic documentation of their informed consent. The study draws on data from two previous studies conducted by the researchers. Both previous studies were cleared by the respective Institutional Ethics Committee (Letter dated November 14, 2019, with number SCTIMST/IEC/1444; and letter dated 10 December 2020, numbered IEC EC2/P1/SEP/2020/HAP). The synthesis is part of the formative work towards the doctoral dissertation of Dr Arsha Kochuvilayil and the protocol and tools were reviewed and cleared by the Institutional Ethics Committee of the Sree Chitra Tirunal Institute for Medical Sciences and Technology, Trivandrum. (Letter number SCTIMST/IEC/2048/MAY-2023 dated 17 June 2023).

Consent for publication

Not applicable.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Additional File 1. Coding Schema for Coding Palliative Care Nurse Interviews.

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kochuvilayil, A., Varma, R.P. Understanding caregiver burden and quality of life in Kerala’s primary palliative care program: a mixed methods study from caregivers and providers’ perspectives. Int J Equity Health 23 , 92 (2024). https://doi.org/10.1186/s12939-024-02155-x

Download citation

Received : 23 March 2023

Accepted : 18 March 2024

Published : 07 May 2024

DOI : https://doi.org/10.1186/s12939-024-02155-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Family caregivers
• Caregiver burden
• Quality of life
• Utility scores
• Palliative nurse
• Palliative care

International Journal for Equity in Health

ISSN: 1475-9276

• General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Mixed Methods Research | Definition, Guide, & Examples

Mixed Methods Research | Definition, Guide, & Examples

Published on 4 April 2022 by Tegan George . Revised on 25 October 2022.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, benefits of mixed methods research, disadvantages of mixed methods research, frequently asked questions about mixed methods research.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalisability : Qualitative research usually has a smaller sample size , and thus is not generalisable . In mixed methods research, this comparative weakness is mitigated by the comparative strength of ‘large N’, externally valid quantitative research.
• Contextualisation: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help ‘put meat on the bones’ of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions. Mixed methods can be very challenging to put into practice, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism, run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyse them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyse cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyse accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyse both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualise your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses. Then you can use the quantitative data to test or confirm your qualitative findings.

‘Best of both worlds’ analysis

Combining the two types of data means you benefit from both the detailed, contextualised insights of qualitative data and the generalisable, externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalisable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labour-intensive. Collecting, analysing, and synthesising two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organisations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analysed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analysed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualise your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analysed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

George, T. (2022, October 25). Mixed Methods Research | Definition, Guide, & Examples. Scribbr. Retrieved 6 May 2024, from https://www.scribbr.co.uk/research-methods/mixed-methods/

Is this article helpful?

Tegan George

This paper is in the following e-collection/theme issue:

Published on 7.5.2024 in Vol 26 (2024)

This is a member publication of National University of Singapore

Effectiveness of an Artificial Intelligence-Assisted App for Improving Eating Behaviors: Mixed Methods Evaluation

Authors of this article:

Original Paper

• Han Shi Jocelyn Chew 1 , PhD   ; 
• Nicholas WS Chew 2 , MBBS   ; 
• Shaun Seh Ern Loong 3 , MBBS   ; 
• Su Lin Lim 4 , PhD   ; 
• Wai San Wilson Tam 1 , PhD   ; 
• Yip Han Chin 3 , MBBS   ; 
• Ariana M Chao 5 , PhD   ; 
• Georgios K Dimitriadish 6 , MBBS, MSc   ; 
• Yujia Gao 7   ; 
• Jimmy Bok Yan So 8 , MB ChB, FRCS, MPH   ; 
• Asim Shabbir 8 , MBBS, MMed, FRCS   ; 
• Kee Yuan Ngiam 9 , MBBS, FRCS  

1 Alice Lee Centre for Nursing Studies, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

2 Department of Cardiology, National University Hospital, Singapore, Singapore

3 Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore

4 Department of Dietetics, National University Hospital, Singapore, Singapore

5 School of Nursing, Johns Hopkins University, Baltimore, MD, United States

6 Department of Endocrinology ASO/EASO COM, King's College Hospital NHS Foundation Trust, London, United Kingdom

7 Division of Hepatobiliary & Pancreatic Surgery, Department of Surgery, National University Hospital, Singapore, Singapore

8 Division of General Surgery (Upper Gastrointestinal Surgery), Department of Surgery, National University Hospital, Singapore, Singapore

9 Division of Thyroid & Endocrine Surgery, Department of Surgery, National University Hospital, Singapore, Singapore

Corresponding Author:

Han Shi Jocelyn Chew, PhD

Alice Lee Centre for Nursing Studies

Yong Loo Lin School of Medicine

National University of Singapore

Level 3, Clinical Research Centre

Block MD11, 10 Medical Drive

Singapore, 117597

Phone: 65 65168687

Email: [email protected]

Background: A plethora of weight management apps are available, but many individuals, especially those living with overweight and obesity, still struggle to achieve adequate weight loss. An emerging area in weight management is the support for one’s self-regulation over momentary eating impulses.

Objective: This study aims to examine the feasibility and effectiveness of a novel artificial intelligence–assisted weight management app in improving eating behaviors in a Southeast Asian cohort.

Methods: A single-group pretest-posttest study was conducted. Participants completed the 1-week run-in period of a 12-week app-based weight management program called the Eating Trigger-Response Inhibition Program (eTRIP). This self-monitoring system was built upon 3 main components, namely, (1) chatbot-based check-ins on eating lapse triggers, (2) food-based computer vision image recognition (system built based on local food items), and (3) automated time-based nudges and meal stopwatch. At every mealtime, participants were prompted to take a picture of their food items, which were identified by a computer vision image recognition technology, thereby triggering a set of chatbot-initiated questions on eating triggers such as who the users were eating with. Paired 2-sided t tests were used to compare the differences in the psychobehavioral constructs before and after the 7-day program, including overeating habits, snacking habits, consideration of future consequences, self-regulation of eating behaviors, anxiety, depression, and physical activity. Qualitative feedback were analyzed by content analysis according to 4 steps, namely, decontextualization, recontextualization, categorization, and compilation.

Results: The mean age, self-reported BMI, and waist circumference of the participants were 31.25 (SD 9.98) years, 28.86 (SD 7.02) kg/m 2 , and 92.60 (SD 18.24) cm, respectively. There were significant improvements in all the 7 psychobehavioral constructs, except for anxiety. After adjusting for multiple comparisons, statistically significant improvements were found for overeating habits (mean –0.32, SD 1.16; P <.001), snacking habits (mean –0.22, SD 1.12; P <.002), self-regulation of eating behavior (mean 0.08, SD 0.49; P =.007), depression (mean –0.12, SD 0.74; P =.007), and physical activity (mean 1288.60, SD 3055.20 metabolic equivalent task-min/day; P <.001). Forty-one participants reported skipping at least 1 meal (ie, breakfast, lunch, or dinner), summing to 578 (67.1%) of the 862 meals skipped. Of the 230 participants, 80 (34.8%) provided textual feedback that indicated satisfactory user experience with eTRIP. Four themes emerged, namely, (1) becoming more mindful of self-monitoring, (2) personalized reminders with prompts and chatbot, (3) food logging with image recognition, and (4) engaging with a simple, easy, and appealing user interface. The attrition rate was 8.4% (21/251).

Conclusions: eTRIP is a feasible and effective weight management program to be tested in a larger population for its effectiveness and sustainability as a personalized weight management program for people with overweight and obesity.

Trial Registration: ClinicalTrials.gov NCT04833803; https://classic.clinicaltrials.gov/ct2/show/NCT04833803

Introduction

Overweight and obesity remain a public health concern that affects slightly more than half of the global adult population [ 1 ]. Across 52 Organization for Economic Co-operation and Development, Group of Twenty, and European Union 28 countries, treating conditions related to overweight and obesity costs US $425 billion per year, based on purchasing power parity. Each US dollar used to prevent obesity results in a 6-fold return in economic benefits [ 2 ]. Strategies for maintaining a healthy weight range from policy mandates on nutritional food labeling [ 3 ] to clinical treatments focused on lifestyle modifications, pharmacotherapy, and bariatric surgery [ 4 ]. However, the effectiveness of such strategies is limited by insurance coverage [ 5 ] and challenges with weight loss maintenance [ 6 - 9 ]. Some participants have been reported to regain up to 100% of their initial weight loss within 5 years [ 9 , 10 ].

With the rapid digitalization and smartphone penetration worldwide, weight loss apps have been gaining popularity, as they help overcome the temporospatial challenges of in-person weight loss programs [ 11 ]. For instance, participants enrolled in conventional weight management programs typically attend multiple face-to-face sessions at designated facilities, which could be burdensome and inconvenient as one needs to schedule appointments and travel to the facility that may be beyond one’s usual mobility pattern. Moreover, such programs are resource-intensive, requiring a multidisciplinary team of trained health care professionals (eg, physicians, dietitians, physiotherapists, nurses), infrastructure (eg, counselling room), and equipment (eg, weighing scale, stadiometer) to maintain. Well-known apps that support weight loss in the market include MyFitnessPal [ 12 ], MyPlate Calorie Tracker [ 13 ], and Fitbit [ 14 ]. In Singapore, Healthy 365 [ 15 ] is available for the public, while nBuddy [ 16 ] is used for the clinical population. These apps mostly focus on calorie tracking, health status tracking, and progress monitoring. Increasingly, apps are enhanced with features that allow intuitive synchronization of health metrics across apps to provide a more holistic progress monitoring experience. With a fee, some apps even match users to a health coach who would provide personalized weight management plans to support weight loss. However, there is a need for apps that include monitoring and support for one’s self-regulation over momentary eating impulses, which are often triggered and influenced by dietary lapse triggers such as visual food cues, eating out, negative affect, and sleep deprivation [ 17 - 20 ]. Self-regulation of eating behaviors during weight loss treatment commonly includes portion control, increasing fruit and vegetable consumption, reducing unhealthy food (sugar-sweetened beverages and high-fat food items) consumption, and reducing overall caloric consumption [ 17 ]. Therefore, we aimed to examine the feasibility and effectiveness of a novel artificial intelligence (AI)-assisted weight management app on improving eating behaviors and to explore the mechanism by which this app influences eating behaviors, as hypothesized in our earlier work [ 21 ].

Study Design

A single-group pretest-posttest study was conducted and reported according to the TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) checklist ( Multimedia Appendix 1 ) [ 22 ]. Despite the limitations of the study design, it was deemed the most appropriate and feasible experimental study design for a preliminary understanding of the usability, acceptability, and effectiveness of the app [ 23 ].

Participant Recruitment

Participants older than 21 years with BMI ≥23 kg/m 2 and not undergoing a commercial weight loss program were recruited from January 2022 to October 2022 through social media platforms and physical recruitment at a local tertiary hospital’s specialist weight management clinic in Singapore. Using G*Power (version 3.1.9.7) [ 24 ], to detect a small effect size of 0.2 at .05 significance level and 80% power while accounting for an attrition rate of 20%, 248 participants are required. To be conservative, 250 participants were recruited.

Intervention

Immediately after completing the pretest questionnaire, participants were onboarded to the Eating Trigger-Response Inhibition Program (eTRIP) app by a trained research assistant to complete the 1-week run-in of the program. During the onboarding, participants were invited to enter their anthropometric details, desired weight loss goals, and motivation. They were also encouraged to personalize certain app functions such as the timing of the check-in prompts and preferred name for interaction with a chatbot. At every mealtime (at least 3 times a day), the participants were prompted to take a picture of their food, which was immediately recognized by a food-based computer vision image recognition technology, which then triggered a set of chatbot-initiated questions on eating triggers (eg, how they are feeling). These questions were developed based on our past work on eating behaviors [ 25 - 30 ]. Participants were able to view their image-based food log and eating habits on a dashboard, reflect upon their eating habits throughout the day, and set their goals and action plans for the next day. On the 8th day, all participants’ user accounts were locked, and they were unable to make any changes but were able to still view their check-in logs. Participants could also provide feedback on the app by filling out the comments section in one of the app pages. Participants were reimbursed SGD 25 (SGD 1=US $0.74) for completing this program.

The eTRIP app was developed as a 12-week AI-assisted, app-based, self-regulation program targeted at improving weight loss through healthy eating. eTRIP was developed largely based on a modified temporal self-regulation theory [ 31 , 32 ], behavioral change taxonomy [ 33 ], and our previous work on healthy eating and weight loss [ 27 - 29 , 34 , 35 ]. This includes studies on people with overweight and obesity in the areas of personal motivators, self-regulation facilitators, and barriers [ 27 ]; the potential of AI, apps, and chatbots in improving weight loss [ 6 , 25 , 29 ]; perceptions and needs of AI to increase its adoption in weight management [ 26 ]; and the essential elements of a weight loss app [ 28 ]. The development of eTRIP was split into 2 phases: (1) development of an AI-assisted self-monitoring system and (2) development of an AI-assisted behavioral nudging system. In this paper, we report the feasibility and effectiveness of an AI-assisted self-monitoring system after a 1-week run-in. The self-monitoring system is built upon 3 main components, namely, (1) chatbot-based check-ins on eating lapse triggers, (2) food-based computer vision image recognition (system built based on local food items), and (3) automated time-based nudges and meal stopwatch.

All participants completed the same self-report questionnaire before and after the 1-week run-in of the app, which reflected their sociodemographic profile, BMI, waist circumference, intention to improve eating behaviors, habits of overeating (Self-Report Habit Index) [ 36 ], habits of snacking [ 36 ], consideration of future consequences (Consideration of Future Consequences Scale-6 items) [ 37 ], self-regulation of eating behavior (Self-Regulation of Eating Behavior Questionnaire) [ 38 ], physical activity (International Physical Activity Questionnaire-Short Form) [ 39 ], anxiety symptoms (Generalized Anxiety Disorder-2 items) [ 40 ], and depressive symptoms (Patient Health Questionnaire-2 items) [ 41 ]. Details are reported in Multimedia Appendix 2 . The primary outcomes were overeating habits, snacking habits, immediate thinking, self-regulation of eating habits, depression, anxiety, and physical activity. The secondary outcomes were their subscale scores.

Data Analysis

SPSS statistical software (version 27; IBM Corp) [ 42 ] was used for the analyses. The baseline characteristics of the participants were presented in mean (SD) and frequency (%). Paired 2-sided t tests were used to compare the differences in the psychobehavioral constructs before and after the 7-day program, including overeating habits, snacking habits, consideration of future consequences, self-regulation of eating behaviors, anxiety, depression, and physical activity. To account for the increased risk of a type 1 error due to multiple comparisons [ 43 ], the Bonferroni-corrected significant level was set to P ≤.007. Qualitative feedback were analyzed using content analysis according to 4 steps, namely, decontextualization, recontextualization, categorization, and compilation [ 44 ]. Feedback was first consolidated verbatim and read iteratively by 2 coders (Nagadarshini Nicole Rajasegaran and HSJC). The verbatim feedback was then analyzed independently by 2 reviewers into meaning units. Meaning units were then reconstituted, categorized, and reported as themes and subthemes.

Ethics Approval

This single-group pretest-posttest study was approved by the National Healthcare Group Domain Specific Review Board (ref 2020/01439), registered with the ClinicalTrials.gov (ref NCT04833803) on April 6, 2021.

Baseline Characteristics of the Participants

A total of 251 participants were enrolled in this study (Chew HSJ, unpublished data, 2023); 20 (7.9%) participants dropped out of the 1-week program due to the inability to perform check-ins every day. Among those who completed the program (n=231), 1 participant was removed from the analyses due to ineligibility. The mean age, self-reported BMI, and waist circumference of the participants was 31.25 (SD 9.98) years, 28.86 (SD 7.02) kg/m 2 , and 92.6 (SD 18.24) cm, respectively ( Table 1 ). Approximately 47.8% (111/230) of the participants were males, indicating a good mix of participants from both sexes, and most of the participants were single (169/230, 73.6%), Chinese (181/230, 78.7%), and had a university education (148/230, 64.1%).

a SGD 1=US $0.74.

Mean Baseline Scores on Each Outcome Variable

The mean baseline scores on each outcome variable of the participants who completed and who dropped out from the 1-week program were calculated ( Table 2 ). As the dropout rate was only 8.4% (20/251), statistical comparisons between those who dropped out and those who completed the program was not necessary.

a SRHI: Self-Report Habit Index.

b CFCS-6: Consideration of Future Consequences Scale-6 items.

c SREBQ: Self-Regulation of Eating Behavior Questionnaire.

d IPAQ-SF: International Physical Activity Questionnaire-Short Form.

e MET: metabolic equivalent task.

Pretest and Posttest Mean Differences

There were significant improvements in all the 7 psychobehavioral constructs, except for anxiety. After adjusting for multiple comparisons, there were only statistically significant improvements in the overeating habit, snacking habit, self-regulation of eating behavior, depression, and physical activity ( Table 3 ). Forty-one participants reported skipping at least 1 meal (ie, breakfast, lunch, or dinner), summing to a total of 578 (67.1%) of the 862 meals skipped.

b Significant at P <.007.

c CFCS-6: Consideration of Future Consequences Scale-6 items.

d Consideration of Future Consequences Scale-6 immediate subscale.

e Consideration of Future Consequences Scale-6 future subscale.

f SREBQ: Self-Regulation of Eating Behavior Questionnaire.

g IPAQ-SF: International Physical Activity Questionnaire-Short Form.

User Engagement

Among those who completed the program, 97% (46,867/48,316) chatbot-based questions were completed. As participants were given the option to add additional check-ins for snacks, the percentage of completed check-ins could not be accurately computed.

Qualitative Feedback

Of the 230 participants, 80 (34.8%) provided textual feedback that indicated satisfactory experience with eTRIP. Four themes emerged, namely, (1) becoming more mindful of self-monitoring, (2) personalized reminders with prompts and chatbot, (3) food logging with image recognition, and (4) engaging with a simple, easy, and appealing user interface.

Becoming More Mindful of Self-Monitoring

By checking in with the app for every meal, the participants mentioned being more aware of their unhealthy eating habits and more mindful of their next meal. One participant said, “It (eTRIP) incentivizes me to stick to my diet plan because I am reminded of my diet plan daily. Ticking the box that indicates ‘I did not meet my diet plan’ made me guilty and it motivates me to opt for healthier food choice the next time round” (Female, Chinese, 22 years old). Another participant said, “I really liked the eTRIP app! Has a lot of potential for further expansion and use by more people. I like how it sends prompts during selected times of the day to be careful of what we see on social media. The rating of our mood before meals also helps me know how mood can affect my eating patterns. Lastly, the stopwatch function is great because it reminds me to eat more mindfully” (Male, Chinese, 27 years old).

Personalized Reminders With Prompts and Chatbot

Some participants mentioned the appreciation for reminders to check in with themselves in terms of the triggers of overeating. One participant said, “I like that there’s a reminder to check in for every meal and users get to decide what time the app should prompt!” (Female, Indian, 27 years old). Some also suggested to develop the prompting system to prompt based on the user’s previous check-in timings to optimize the prediction of mealtimes and prompt the check-in sessions intuitively. One participant suggested, “I think what would make this better is if you could aggregate the time the meals are entered from the past few days and estimate the time the user will normally eat and auto-adjust the timing…” (Female, Chinese, 23 years old). Others suggested to include reminders of how to make their meal options healthier, “Might be good to have reminders that reminds us to eat healthy with some tips on how to choose food” (Male, Chinese, 25 years old).

Food Logging With Image Recognition

Many participants highlighted their appreciation for the image recognition-based food logging, as it was accurate and convenient for food logging. One participant said, “It is very accurate in determining the food I’ve eaten just from the picture, and this saved me a lot of time from typing out the food I’ve eaten” (Male, Chinese, 21 years old).

Engaging With a Simple, Easy, and Appealing User Interface

All the participants who commented on the user experience expressed being impressed with the user interface and structure. One participant said, “the flow was smooth, quite clear. Graphics were cute. Very easy to input my info (information) especially from the homepage, I like how there’s the ability to skip a meal” (Female, Malay, 25 years old). Another participant said, “The app is very smart, … yes it’s very easy to fill and I loss (lost) like 0.5kg?” (Female, Chinese, 25 years old).

Participants’ Suggestions

In terms of the areas for improvement, the participants preferred to have (1) more options and rating scales for each domain of eating trigger instead of typing out in the “others” field (although there was a stored text for repeated entries); (2) summary of the instances where one was able to achieve the goal of the day, which the user sets daily for the next day (based on a user preset list of goals); (3) examples of standard portions and frequency of meals; and (4) feedback on how to improve upon the unhealthy meals logged.

Real-time interventions that can effectively address eating lapse triggers and improve eating behavior self-regulation, lapse events, weight loss, and weight maintenance remain unclear [ 45 ]. OnTrack, a just-in-time adaptive intervention that has been tested, is a smartphone app that uses machine learning to predict dietary lapses based on the repeated assessments of lapse triggers (ecological momentary assessment). OnTrack is used in conjunction with existing weight loss apps such as WeightWatchers app and provides personalized recommendations to prevent dietary lapses. The compliance rate for completing the lapse trigger survey in OnTrack was 62.9% over 3 months, and the studied sample was mostly females who were Whites [ 46 ]. Evidence has shown that factors influencing obesity and overweight are population-specific, influenced by socioeconomic, cultural, and genetic factors among others [ 45 , 47 ]. Singapore is a multiethnic society with a unique food culture influenced by various racial beliefs and traditions [ 48 ]. The differences in geographical, social, environmental, and genetic characteristics could define a different set of triggers and response to such weight loss apps.

Principal Findings

In this paper, we report the effectiveness of a weeklong AI-assisted weight loss app for improving overeating habits, snacking habits, immediate thinking, self-regulation of eating habits, depression, and physical activity. Interestingly, there were no significant improvements in the anxiety symptoms before and after using eTRIP, potentially due to the already low level of anxiety in those who completed the program (ie, ceiling effect). We also report corresponding qualitative user feedback on the experience with using eTRIP, where the users appreciated the app for enabling them to become more mindful of self-monitoring; personalized reminders with prompts and chatbot; food logging with image recognition; and engaging with a simple, easy, and appealing user interface. The significant improvements observed among the participants in this study reveal the potential of this app to influence weight loss in the context of a Southeast Asian cohort with overweight and obesity. The qualitative feedback also informs future app development to enhance user engagement and reduce dropout rates.

Eating habits contribute to overweight and obesity [ 49 , 50 ]. Encouraged by an obesogenic environment, overeating is commonly triggered by situational factors such as food novelty or variety, social company (eg, eating with certain people), affect emotional states (which trigger emotional eating), and distractions (eg, concurrent tasks) [ 30 , 50 - 53 ]. Other studies have suggested that people at risk for obesity exhibit hyperresponsivity in the neural reward system to calorie-dense foods, which is associated with increased food consumption [ 54 ]. Alongside users’ feedback that the app made them more mindful of their eating patterns, the significant improvement in overeating and snacking habits could have been due to an increased awareness of one’ maladaptive eating habits and subsequently, the motivation to change. This coincides with a review that reported the effectiveness of mindful eating interventions on reducing food consumption in people with overweight and obesity [ 55 ]. Our qualitative findings showed that by self-monitoring one’s eating behavior through chatbot-initiated check-ins, one could enhance mindful eating and reduce overeating without the need for undergoing mindful eating training. This could eventually lead to a reduction in total food consumption and weight loss. However, more quantitative evidence is needed to support this point.

It is noteworthy that some participants reported skipping meals as planned, which might have led to reduced energy consumption. However, this has to be examined further, as studies have shown that the calories avoided during a skipped meal may be compensated by an increase in snacking or overeating during mealtimes [ 56 ]. One additional element that can be explored in future studies is the effectiveness of promoting healthy snacking, which includes snacking on foods rich in proteins, fruits, vegetables, and whole grains, as opposed to nutrient-poor and energy-dense foods [ 57 , 58 ]. These healthier alternatives have been found not only to be associated with earlier satiety but also to be more nutritious, with their contents being more consistent with the established dietary recommendations and guidelines [ 59 , 60 ]. This strategy can be explored in conjunction with the current approach to decrease participants’ overall snacking habits.

The improvement in the self-regulation of eating habits could be attributed to several factors, including the app content focused on reminding participants of their weight loss goals and to adopt healthier eating habits of less snacking and overeating during mealtimes. In particular, in commonly stigmatized populations like those with overweight and obesity, personalization of interventions enhances one’s feeling of being taken care of, nurtured, and respected, providing them with a sense of confidence [ 61 ]. This may have improved individuals’ willingness to engage with the eTRIP content, knowing that they would be well-respected and seen as individuals through personalized chatbot conversations and reminders [ 62 ]. Other studies have shown that personalized eHealth interventions are more effective than conventional programs in enhancing weight loss maintenance, BMI, waist circumference, and various other metabolic indicators [ 63 ].

In conjunction with improvements in eating habits, participants also engaged in greater levels of physical activity by the end of this study. Increased health consciousness and self-education about the impacts and types of physical activity are factors that may explain the observed increased level of physical activity among the participants [ 64 ]. Various studies have found that a combination of diet and exercise is superior to diet-only interventions in inducing weight loss [ 65 ]. The level of physical activity is also an important factor for improving long-term weight loss [ 66 - 68 ]. Moderate amounts of physical activity were observed to prevent weight regain after weight loss [ 65 , 68 , 69 ]. In addition, the American College of Sports Medicine recommends 200-300 minutes of moderate physical activity a week to prevent similar weight regain [ 70 ]. In our study, low and moderate levels of exercise were seen to increase significantly among the participants. Although the sustainability of the increase in the exercise levels is still unknown, the preliminary data are encouraging to show the potential of the app in impacting physical activity. The amounts of high levels of exercise were, however, not impacted significantly. Additional interventions, including the provision of educational materials about the benefits of and types of exercise, along with personalized reminders for exercise can potentially further increase the success of the app in increasing moderate and high levels of exercise among its participants [ 71 , 72 ].

In addition to improvements in the eating habits and levels of physical activity, there were changes in the psychological factors among the participants. The mean depressive symptoms were significantly decreased at the end of the weeklong program. Various studies have shown that healthy living characterized by various factors such as healthy eating and sufficient levels of physical activity have the potential to positively impact psychological factors such as mood and emotions [ 73 ]. Healthy eating with adherence to dietary recommendations has been found to reduce the levels of inflammation, increase the levels of various micronutrients such as vitamins, and regulate the levels of simple sugars, all of which are protective against mental illnesses, especially depression [ 73 - 76 ]. Studies have also shown that the use of chatbots in the app may decrease depressive symptoms among some participants. Chatbots provide individuals the ability to provide self-care in an environment that is neither costly nor stigmatizing [ 77 ]. This may enable participants to be more open with their emotions, as well as to have an outlet to gain relief through their interaction with the chatbot as a proxy of human interaction [ 77 ]. Studies have shown that improvements in psychological factors such as depressive symptoms have been positively associated with weight loss and maintenance, further increasing the effectiveness of weight loss efforts among participants with overweight and obesity [ 45 ].

Strengths and Limitations

This study was the first to characterize the effectiveness of an AI-assisted weight loss app in the context of a Southeast Asian cohort. One strength of this study was the demographics of the participants, which was generally representative of the Singaporean population in terms of sex and ethnicity. The consideration of population-specific determinants of obesity and overweight during the design of the app would have also increased its applicability in this population [ 78 ], having considered the various nuances and practical needs of its target demographic. For example, the food image recognition system, which was built based on local food items, reduced the amount of time and effort required for the logging of food, improved the usability of the intervention, and enhanced user experience. The success of this app thus provides evidence that the consideration of population-specific underpinnings and practical requirements were essential toward the successful design and implementation of a weight loss intervention [ 79 ].

Although the app presents significant potential in this weeklong trial, this study is limited due to its short time frame. This presents with difficulties in understanding the midterm to long-term impacts of using the app. However, it is reassuring that despite the limited time frame of this study, the various behavioral and psychological indicators were observed to be significantly improved. Through the feedback gathered from the participants, the app may be improved in specific aspects, including (1) refining choices available for the various survey fields such as the provision of a drop-down menu for the selection of weight loss goals; (2) providing additional feedback and weekly summaries to the participants for knowledge of their progress in various aspects; and (3) providing educational materials to provide participants with the means to improve, especially for what to do after eating lapses and suggestions for healthy snacking. Another limitation was the lack of feedback quotes from older individuals as opposed to those from younger individuals. This could be due to various reasons, of which decreased media literacy among older individuals might present additional obstacles for the provision of feedback [ 80 ]. Lastly, we did not collect information on the participants’ medical and pharmacological history, where certain diseases and drugs are known to influence weight gain through various metabolic and neural pathways. Weight and waist circumference were also self-reported, and thus, data from these measures should be interpreted cautiously.

This study was the first to characterize the effectiveness of an AI-assisted weight loss app in the context of a Southeast Asian cohort. The positive findings of this study show the feasibility of implementing this app and the large potential it has in impacting weight loss efforts, especially among individuals with overweight and obesity. Efforts should be made to lengthen and upscale this program for a greater understanding of the midterm to long-term effects of this app.

Conflicts of Interest

AMC has served on advisory boards to Eli Lilly and Boehringer Ingelheim and received grant support, on behalf of the University of Pennsylvania, from Eli Lilly and WW (Weight Watchers). No other authors declare conflicts of interest.

TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) checklist.

Details on outcome measures.

• Abdelaal M, le Roux CW, Docherty NG. Morbidity and mortality associated with obesity. Ann Transl Med. Apr 2017;5(7):161. [ FREE Full text ] [ CrossRef ] [ Medline ]
• OECD. The heavy burden of obesity: the economics of prevention. OECD Health Policy Studies. Oct 10, 2019.:1-100. [ FREE Full text ] [ CrossRef ]
• Hawkes C, Smith TG, Jewell J, Wardle J, Hammond RA, Friel S, et al. Smart food policies for obesity prevention. The Lancet. Jun 2015;385(9985):2410-2421. [ CrossRef ]
• Johns DJ, Hartmann-Boyce J, Jebb SA, Aveyard P, Behavioural Weight Management Review Group. Diet or exercise interventions vs combined behavioral weight management programs: a systematic review and meta-analysis of direct comparisons. J Acad Nutr Diet. Oct 2014;114(10):1557-1568. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Clarke B, Kwon J, Swinburn B, Sacks G. Understanding the dynamics of obesity prevention policy decision-making using a systems perspective: A case study of Healthy Together Victoria. PLoS One. 2021;16(1):e0245535. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chew HSJ, Koh WL, Ng JSHY, Tan KK. Sustainability of weight loss through smartphone apps: systematic review and meta-analysis on anthropometric, metabolic, and dietary outcomes. J Med Internet Res. Sep 21, 2022;24(9):e40141. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Booth HP, Prevost TA, Wright AJ, Gulliford MC. Effectiveness of behavioural weight loss interventions delivered in a primary care setting: a systematic review and meta-analysis. Fam Pract. Dec 2014;31(6):643-653. [ FREE Full text ] [ CrossRef ] [ Medline ]
• LeBlanc ES, Patnode CD, Webber EM, Redmond N, Rushkin M, O'Connor EA. Behavioral and pharmacotherapy weight loss interventions to prevent obesity-related morbidity and mortality in adults: updated evidence report and systematic review for the US Preventive Services Task Force. JAMA. Sep 18, 2018;320(11):1172-1191. [ CrossRef ] [ Medline ]
• MacLean PS, Wing RR, Davidson T, Epstein L, Goodpaster B, Hall KD, et al. NIH working group report: Innovative research to improve maintenance of weight loss. Obesity (Silver Spring). Jan 2015;23(1):7-15. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Daley A, Jolly K, Madigan C, et al. A brief behavioural intervention to promote regular self-weighing to prevent weight regain after weight loss: a RCT. Public Health Research. 2019.:7. [ CrossRef ] [ Medline ]
• Hadžiabdić MO, Mucalo I, Hrabač P, Matić T, Rahelić D, Božikov V. Factors predictive of drop-out and weight loss success in weight management of obese patients. J Hum Nutr Diet. Feb 2015;28 Suppl 2:24-32. [ CrossRef ] [ Medline ]
• Evans D. MyFitnessPal. Br J Sports Med. Jan 27, 2016;51(14):1101-1102. [ CrossRef ]
• Garcia R. Utilization, Integration & Evaluation of LIVESTRONG's MyPlate Telehealth Technology. URL: https:/​/www.​researchgate.net/​publication/​273635099_Utilization_Integration_Evaluation_of_LIVESTRONG's_MyPlate_Telehealth_Technology [accessed 2023-04-01]
• Hartman SJ, Nelson SH, Weiner LS. Patterns of Fitbit use and activity levels throughout a physical activity intervention: exploratory analysis from a randomized controlled trial. JMIR Mhealth Uhealth. Feb 05, 2018;6(2):e29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yao J, Tan CS, Chen C, Tan J, Lim N, Müller-Riemenschneider F. Bright spots, physical activity investments that work: National Steps Challenge, Singapore: a nationwide mHealth physical activity programme. Br J Sports Med. Sep 2020;54(17):1047-1048. [ CrossRef ] [ Medline ]
• Lim SL, Johal J, Ong KW, Han CY, Chan YH, Lee YM, et al. Lifestyle intervention enabled by mobile technology on weight loss in patients with nonalcoholic fatty liver disease: randomized controlled trial. JMIR Mhealth Uhealth. Apr 13, 2020;8(4):e14802. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Birkett D. Disinhibition. In: The Psychiatry of Stroke. London, UK. Routledge; 2012;150-161.
• Carels RA, Hoffman J, Collins A, Raber AC, Cacciapaglia H, O'Brien WH. Ecological momentary assessment of temptation and lapse in dieting. Eat Behav. 2001;2(4):307-321. [ CrossRef ] [ Medline ]
• Kwasnicka D, Dombrowski SU, White M, Sniehotta FF. N-of-1 study of weight loss maintenance assessing predictors of physical activity, adherence to weight loss plan and weight change. Psychol Health. Jun 2017;32(6):686-708. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McKee HC, Ntoumanis N, Taylor IM. An ecological momentary assessment of lapse occurrences in dieters. Ann Behav Med. Dec 2014;48(3):300-310. [ CrossRef ] [ Medline ]
• Chew HSJ, Loong SSE, Lim SL, Tam WSW, Chew NWS, Chin YH, et al. Socio-demographic, behavioral and psychological factors associated with high BMI among adults in a Southeast Asian multi-ethnic society: a structural equation model. Nutrients. Apr 10, 2023;15(8):1826. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Des Jarlais DC, Lyles C, Crepaz N, TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. Mar 2004;94(3):361-366. [ CrossRef ] [ Medline ]
• Knapp T. Why is the one-group pretest-posttest design still used? Clin Nurs Res. Oct 2016;25(5):467-472. [ CrossRef ] [ Medline ]
• Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. May 2007;39(2):175-191. [ CrossRef ] [ Medline ]
• Chew HSJ. The use of artificial intelligence-based conversational agents (chatbots) for weight loss: scoping review and practical recommendations. JMIR Med Inform. Apr 13, 2022;10(4):e32578. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chew HSJ, Achananuparp P. Perceptions and needs of artificial intelligence in health care to increase adoption: scoping review. J Med Internet Res. Jan 14, 2022;24(1):e32939. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chew HSJ, Gao Y, Shabbir A, Lim SL, Geetha K, Kim G, et al. Personal motivation, self-regulation barriers and strategies for weight loss in people with overweight and obesity: a thematic framework analysis. Public Health Nutr. Feb 22, 2022;25(9):2426-2435. [ CrossRef ]
• Chew H, Lim S, Kim G, Kayambu G, So BYJ, Shabbir A, et al. Essential elements of weight loss apps for a multi-ethnic population with high BMI: a qualitative study with practical recommendations. Transl Behav Med. Apr 03, 2023;13(3):140-148. [ CrossRef ] [ Medline ]
• Chew HSJ, Ang WHD, Lau Y. The potential of artificial intelligence in enhancing adult weight loss: a scoping review. Public Health Nutr. Feb 17, 2021;24(8):1993-2020. [ CrossRef ]
• Chew HSJ, Lau ST, Lau Y. Weight-loss interventions for improving emotional eating among adults with high body mass index: A systematic review with meta-analysis and meta-regression. Eur Eat Disord Rev. Jul 2022;30(4):304-327. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chew HSJ, Sim KLD, Choi KC, Chair SY. Effectiveness of a nurse-led temporal self-regulation theory-based program on heart failure self-care: A randomized controlled trial. Int J Nurs Stud. Mar 2021;115:103872. [ CrossRef ] [ Medline ]
• Hall PA, Fong GT. Temporal self-regulation theory: A model for individual health behavior. Health Psychology Review. Mar 2007;1(1):6-52. [ CrossRef ]
• Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol. May 2008;27(3):379-387. [ CrossRef ] [ Medline ]
• Chew HSJ, Li J, Chng S. Improving adult eating behaviours by manipulating time perspective: a systematic review and meta-analysis. Psychol Health. Jan 24, 2023.:1-17. [ CrossRef ] [ Medline ]
• Chew HSJ, Rajasegaran NN, Chng S. Effectiveness of interactive technology-assisted interventions on promoting healthy food choices: a scoping review and meta-analysis. Br J Nutr. Jan 25, 2023;130(7):1250-1259. [ CrossRef ]
• Verplanken B, Orbell S. Reflections on past behavior: a self‐report index of habit strength. J Applied Social Pyschol. Jul 31, 2006;33(6):1313-1330. [ CrossRef ]
• Chng S, Chew HSJ, Joireman J. When time is of the essence: Development and validation of brief consideration of future (and immediate) consequences scales. Personality and Individual Differences. Feb 2022;186:111362. [ CrossRef ]
• Kliemann N, Beeken RJ, Wardle J, Johnson F. Development and validation of the Self-Regulation of Eating Behaviour Questionnaire for adults. Int J Behav Nutr Phys Act. Aug 02, 2016;13:87. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Craig CL, Marshall AL, Sjöström M, Bauman AE, et al. International Physical Activity Questionnaire: 12-country reliability and validity. Medicine & Science in Sports & Exercise. 2003;35(8):1381-1395. [ CrossRef ]
• Kroenke K, Spitzer RL, Williams JB, Monahan PO, Löwe B. Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann Intern Med. Mar 06, 2007;146(5):317-325. [ CrossRef ] [ Medline ]
• Kroenke K, Spitzer RL, Williams JBW. The Patient Health Questionnaire-2: validity of a two-item depression screener. Med Care. Nov 2003;41(11):1284-1292. [ CrossRef ] [ Medline ]
• IBM SPSS statistics for Windows. IBM Corp. May 21, 2021. URL: https://www.ibm.com/products/spss-statistics [accessed 2024-03-28]
• VanderWeele T, Mathur M. Some desirable properties of the Bonferroni correction: is the Bonferroni correction really so bad? Am J Epidemiol. Mar 01, 2019;188(3):617-618. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bengtsson M. How to plan and perform a qualitative study using content analysis. NursingPlus Open. 2016;2:8-14. [ CrossRef ]
• Varkevisser RDM, van Stralen MM, Kroeze W, Ket JCF, Steenhuis IHM. Determinants of weight loss maintenance: a systematic review. Obes Rev. Feb 2019;20(2):171-211. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Forman EM, Goldstein SP, Crochiere RJ, Butryn ML, Juarascio AS, Zhang F, et al. Randomized controlled trial of OnTrack, a just-in-time adaptive intervention designed to enhance weight loss. Transl Behav Med. Nov 25, 2019;9(6):989-1001. [ CrossRef ] [ Medline ]
• Qasim A, Turcotte M, de Souza RJ, Samaan MC, Champredon D, Dushoff J, et al. On the origin of obesity: identifying the biological, environmental and cultural drivers of genetic risk among human populations. Obes Rev. Feb 2018;19(2):121-149. [ CrossRef ] [ Medline ]
• Executive summary on National Population Health Survey 2016/17. Singapore MoH. URL: https:/​/www.​moh.gov.sg/​docs/​librariesprovider5/​resources-statistics/​reports/​executive-summary-nphs-2016_17.​pdf [accessed 2023-04-01]
• McCrory MA, Suen VM, Roberts SB. Biobehavioral influences on energy intake and adult weight gain. The Journal of Nutrition. Dec 2002;132(12):3830S-3834S. [ CrossRef ]
• Davis C, Levitan RD, Muglia P, Bewell C, Kennedy JL. Decision-making deficits and overeating: a risk model for obesity. Obes Res. Jun 2004;12(6):929-935. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hetherington MM. Cues to overeat: psychological factors influencing overconsumption. Proc. Nutr. Soc. Feb 28, 2007;66(1):113-123. [ CrossRef ]
• Borer K. Understanding human physiological limitations and societal pressures in favor of overeating helps to avoid obesity. Nutrients. Jan 22, 2019;11(2):227. [ CrossRef ]
• Borer KT. Why we eat too much, have an easier time gaining than losing weight, and expend too little energy: suggestions for counteracting or mitigating these problems. Nutrients. Oct 26, 2021;13(11):3812. [ CrossRef ]
• Stice E, Burger K. Neural vulnerability factors for obesity. Clin Psychol Rev. Mar 2019;68:38-53. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Warren JM, Smith N, Ashwell M. A structured literature review on the role of mindfulness, mindful eating and intuitive eating in changing eating behaviours: effectiveness and associated potential mechanisms. Nutr. Res. Rev. Jul 18, 2017;30(2):272-283. [ CrossRef ]
• Savige G, Macfarlane A, Ball K, Worsley A, Crawford D. Snacking behaviours of adolescents and their association with skipping meals. Int J Behav Nutr Phys Act. Sep 17, 2007;4:36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Njike VY, Smith TM, Shuval O, Shuval K, Edshteyn I, Kalantari V, et al. Snack food, satiety, and weight. Adv Nutr. Sep 2016;7(5):866-878. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Larson N, Story M. A review of snacking patterns among children and adolescents: what are the implications of snacking for weight status? Child Obes. Apr 2013;9(2):104-115. [ CrossRef ] [ Medline ]
• Lucan SC, Karpyn A, Sherman S. Storing empty calories and chronic disease risk: snack-food products, nutritive content, and manufacturers in Philadelphia corner stores. J Urban Health. May 2010;87(3):394-409. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hess J, Jonnalagadda S, Slavin J. What is a snack, why do we snack, and how can we choose better snacks? The definitions of snacking, motivations to snack, contributions to dietary intake, and recommendations for improvement. The FASEB Journal. Apr 2016;30(S1):466-475. [ CrossRef ]
• Coulter A, Entwistle V, Eccles A, Ryan S, Shepperd S, Perera R. Personalised care planning for adults with chronic or long-term health conditions. Cochrane Database Syst Rev. 2015;2015(3):1-117. [ CrossRef ]
• Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, Kinnersley P, et al. Shared decision making: a model for clinical practice. J Gen Intern Med. Oct 2012;27(10):1361-1367. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lau Y, Chee DGH, Chow XP, Cheng LJ, Wong SN. Personalised eHealth interventions in adults with overweight and obesity: A systematic review and meta-analysis of randomised controlled trials. Prev Med. Mar 2020;132:106001. [ CrossRef ] [ Medline ]
• Wang Q, Egelandsdal B, Amdam GV, Almli VL, Oostindjer M. Diet and physical activity apps: perceived effectiveness by app users. JMIR Mhealth Uhealth. Apr 07, 2016;4(2):e33. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim B, Choi D, Jung C, Kang S, Mok J, Kim C. Obesity and physical activity. J Obes Metab Syndr. Mar 2017;26(1):15-22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anderson JW, Konz EC, Frederich RC, Wood CL. Long-term weight-loss maintenance: a meta-analysis of US studies. Am J Clin Nutr. Nov 2001;74(5):579-584. [ CrossRef ] [ Medline ]
• Fogelholm M, Kukkonen-Harjula K. Does physical activity prevent weight gain--a systematic review. Obes Rev. Oct 2000;1(2):95-111. [ CrossRef ] [ Medline ]
• Thomas DM, Bouchard C, Church T, Slentz C, Kraus WE, Redman LM, et al. Why do individuals not lose more weight from an exercise intervention at a defined dose? An energy balance analysis. Obesity Reviews. Jun 11, 2012;13(10):835-847. [ CrossRef ]
• Saris WHM, Blair SN, van Baak MA, Eaton SB, Davies PSW, Di Pietro L, et al. How much physical activity is enough to prevent unhealthy weight gain? Outcome of the IASO 1st Stock Conference and consensus statement. Obes Rev. May 2003;4(2):101-114. [ CrossRef ] [ Medline ]
• Haskell W, Lee I, Pate R, Powell K, Blair S, Franklin B. Physical activity and public health: updated recommendation for adults from the American College of Sports Medicine and the American Heart Association. Med Sci Sports Exerc. 2007;39(8):1423-1434. [ CrossRef ]
• Fredriksson SV, Alley SJ, Rebar AL, Hayman M, Vandelanotte C, Schoeppe S. How are different levels of knowledge about physical activity associated with physical activity behaviour in Australian adults? PLoS One. 2018;13(11):e0207003. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muntaner-Mas A, Sanchez-Azanza VA, Ortega FB, Vidal-Conti J, Borràs PA, Cantallops J, et al. The effects of a physical activity intervention based on a fatness and fitness smartphone app for University students. Health Informatics J. 2021;27(1):1460458220987275. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hautekiet P, Saenen ND, Martens DS, Debay M, Van der Heyden J, Nawrot TS, et al. A healthy lifestyle is positively associated with mental health and well-being and core markers in ageing. BMC Med. Sep 29, 2022;20(1):328. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lassale C, Batty GD, Baghdadli A, Jacka F, Sánchez-Villegas A, Kivimäki M, et al. Healthy dietary indices and risk of depressive outcomes: a systematic review and meta-analysis of observational studies. Mol Psychiatry. Jul 2019;24(7):965-986. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Petridou ET, Kousoulis AA, Michelakos T, Papathoma P, Dessypris N, Papadopoulos FC, et al. Folate and B12 serum levels in association with depression in the aged: a systematic review and meta-analysis. Aging Ment Health. Sep 2016;20(9):965-973. [ CrossRef ] [ Medline ]
• Wang C, Yang T, Wang G, Zhao Y, Yang L, Bi B. Association between dietary patterns and depressive symptoms among middle-aged adults in China in 2016-2017. Psychiatry Res. Feb 2018;260:123-129. [ CrossRef ] [ Medline ]
• Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry. Jul 2019;64(7):456-464. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Population in brief. National Population and Talent Division SG. 2022. URL: https://www.strategygroup.gov.sg/files/media-centre/publications/population-in-brief-2022.pdf [accessed 2023-04-01]
• Hebden L, Cook A, van der Ploeg HP, King L, Bauman A, Allman-Farinelli M. A mobile health intervention for weight management among young adults: a pilot randomised controlled trial. J Hum Nutr Diet. Aug 2014;27(4):322-332. [ CrossRef ] [ Medline ]
• Rasi P, Vuojärvi H, Rivinen S. Promoting media literacy among older people: a systematic review. Adult Education Quarterly. May 25, 2020;71(1):37-54. [ CrossRef ]

Abbreviations

Edited by A Mavragani; submitted 26.01.23; peer-reviewed by YC Liu, V Jennings; comments to author 07.12.23; revised version received 12.12.23; accepted 12.03.24; published 07.05.24.

©Han Shi Jocelyn Chew, Nicholas WS Chew, Shaun Seh Ern Loong, Su Lin Lim, Wai San Wilson Tam, Yip Han Chin, Ariana M Chao, Georgios K Dimitriadish, Yujia Gao, Jimmy Bok Yan So, Asim Shabbir, Kee Yuan Ngiam. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.