how to write a comparative case study

• AI Content Shield
• AI KW Research
• AI Assistant
• SEO Optimizer
• AI KW Clustering
• Customer reviews
• The NLO Revolution
• Press Center
• Help Center
• Content Resources
• Facebook Group

Writing a Comparative Case Study: Effective Guide

Table of Contents

As a researcher or student, you may be required to write a comparative case study at some point in your academic journey. A comparative study is an analysis of two or more cases. Where the aim is to compare and contrast them based on specific criteria. We created this guide to help you understand how to write a comparative case study . This article will discuss what a comparative study is, the elements of a comparative study, and how to write an effective one. We also include samples to help you get started.

What is a Comparative Case Study?

A comparative study is a research method that involves comparing two or more cases to analyze their similarities and differences . These cases can be individuals, organizations, events, or any other unit of analysis. A comparative study aims to gain a deeper understanding of the subject matter by exploring the differences and similarities between the cases.

Elements of a Comparative Study

Before diving into the writing process, it’s essential to understand the key elements that make up a comparative study. These elements include:

• Research Question : This is the central question the study seeks to answer. It should be specific and clear, and the basis of the comparison.
• Cases : The cases being compared should be chosen based on their significance to the research question. They should also be similar in some ways to facilitate comparison.
• Data Collection : Data collection should be comprehensive and systematic. Data collected can be qualitative, quantitative, or both.
• Analysis : The analysis should be based on the research question and collected data. The data should be analyzed for similarities and differences between the cases.
• Conclusion : The conclusion should summarize the findings and answer the research question. It should also provide recommendations for future research.

How to Write a Comparative Study

Now that we have established the elements of a comparative study, let’s dive into the writing process. Here is a detailed approach on how to write a comparative study:

Choose a Topic

The first step in writing a comparative study is to choose a topic relevant to your field of study. It should be a topic that you are familiar with and interested in.

Define the Research Question

Once you have chosen a topic, define your research question. The research question should be specific and clear.

Choose Cases

The next step is to choose the cases you will compare. The cases should be relevant to your research question and have similarities to facilitate comparison.

Collect Data

Collect data on each case using qualitative, quantitative, or both methods. The data collected should be comprehensive and systematic.

Analyze Data

Analyze the data collected for each case. Look for similarities and differences between the cases. The analysis should be based on the research question.

Write the Introduction

The introduction should provide background information on the topic and state the research question.

Write the Literature Review

The literature review should give a summary of the research that has been conducted on the topic.

Write the Methodology

The methodology should describe the data collection and analysis methods used.

Present Findings

Present the findings of the analysis. The results should be organized based on the research question.

Conclusion and Recommendations

Summarize the findings and answer the research question. Provide recommendations for future research.

Sample of Comparative Case Study

To provide a better understanding of how to write a comparative study , here is an example: Comparative Study of Two Leading Airlines: ABC and XYZ

Introduction

The airline industry is highly competitive, with companies constantly seeking new ways to improve customer experiences and increase profits. ABC and XYZ are two of the world’s leading airlines, each with a distinct approach to business. This comparative case study will examine the similarities and differences between the two airlines. And provide insights into what works well in the airline industry.

Research Questions

What are the similarities and differences between ABC and XYZ regarding their approach to business, customer experience, and profitability?

Data Collection and Analysis

To collect data for this comparative study, we will use a combination of primary and secondary sources. Primary sources will include interviews with customers and employees of both airlines, while secondary sources will include financial reports, marketing materials, and industry research. After collecting the data, we will use a systematic and comprehensive approach to data analysis. We will use a framework to compare and contrast the data, looking for similarities and differences between the two airlines. We will then organize the data into categories: customer experience, revenue streams, and operational efficiency.

After analyzing the data, we found several similarities and differences between ABC and XYZ. Similarities Both airlines offer a high level of customer service, with attentive flight attendants, comfortable seating, and in-flight entertainment. They also strongly focus on safety, with rigorous training and maintenance protocols in place. Differences ABC has a reputation for luxury, with features such as private suites and shower spas in first class. On the other hand, XYZ has a reputation for reliability and efficiency, with a strong emphasis on on-time departures and arrivals. In terms of revenue streams, ABC derives a significant portion of its revenue from international travel. At the same time, XYZ has a more diverse revenue stream, focusing on domestic and international travel. ABC also has a more centralized management structure, with decision-making authority concentrated at the top. On the other hand, XYZ has a more decentralized management structure, with decision-making authority distributed throughout the organization.

This comparative case study provides valuable insights into the airline industry and the approaches taken by two leading airlines, ABC and Delta. By comparing and contrasting the two airlines, we can see the strengths and weaknesses of each method. And identify potential strategies for improving the airline industry as a whole. Ultimately, this study shows that there is no one-size-fits-all approach to doing business in the airline industry. And that success depends on a combination of factors, including customer experience, operational efficiency, and revenue streams.

Wrapping Up

A comparative study is an effective research method for analyzing case similarities and differences. Writing a comparative study can be daunting, but proper planning and organization can be an effective research method. Define your research question, choose relevant cases, collect and analyze comprehensive data, and present the findings. The steps detailed in this blog post will help you create a compelling comparative study that provides valuable insights into your research topic . Remember to stay focused on your research question. And use the data collected to provide a clear and concise analysis of the cases being compared.

Abir Ghenaiet

Abir is a data analyst and researcher. Among her interests are artificial intelligence, machine learning, and natural language processing. As a humanitarian and educator, she actively supports women in tech and promotes diversity.

Explore All Write A Case Study Articles

How to write a leadership case study (sample) .

Writing a case study isn’t as straightforward as writing essays. But it has proven to be an effective way of…

• Write A Case Study

Top 5 Online Expert Case Study Writing Services 

It’s a few hours to your deadline — and your case study college assignment is still a mystery to you.…

Examples Of Business Case Study In Research

A business case study can prevent an imminent mistake in business. How? It’s an effective teaching technique that teaches students…

How to Write a Multiple Case Study Effectively

Have you ever been assigned to write a multiple case study but don’t know where to begin? Are you intimidated…

How to Write a Case Study Presentation: 6 Key Steps

Case studies are an essential element of the business world. Understanding how to write a case study presentation will give…

How to Write a Case Study for Your Portfolio

Are you ready to showcase your design skills and move your career to the next level? Crafting a compelling case…

• school Campus Bookshelves
• menu_book Bookshelves
• perm_media Learning Objects
• login Login
• how_to_reg Request Instructor Account
• hub Instructor Commons

Margin Size

• Download Page (PDF)
• Download Full Book (PDF)
• Periodic Table
• Physics Constants
• Scientific Calculator
• Reference & Cite
• Tools expand_more
• Readability

selected template will load here

This action is not available.

2.3: Case Selection (Or, How to Use Cases in Your Comparative Analysis)

• Last updated
• Save as PDF
• Page ID 135832

• Dino Bozonelos, Julia Wendt, Charlotte Lee, Jessica Scarffe, Masahiro Omae, Josh Franco, Byran Martin, & Stefan Veldhuis
• Victor Valley College, Berkeley City College, Allan Hancock College, San Diego City College, Cuyamaca College, Houston Community College, and Long Beach City College via ASCCC Open Educational Resources Initiative (OERI)

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

\( \newcommand{\Span}{\mathrm{span}}\)

\( \newcommand{\id}{\mathrm{id}}\)

\( \newcommand{\kernel}{\mathrm{null}\,}\)

\( \newcommand{\range}{\mathrm{range}\,}\)

\( \newcommand{\RealPart}{\mathrm{Re}}\)

\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

\( \newcommand{\Argument}{\mathrm{Arg}}\)

\( \newcommand{\norm}[1]{\| #1 \|}\)

\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

\( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

\( \newcommand{\vectorC}[1]{\textbf{#1}} \)

\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

Learning Objectives

By the end of this section, you will be able to:

• Discuss the importance of case selection in case studies.
• Consider the implications of poor case selection.

Introduction

Case selection is an important part of any research design. Deciding how many cases, and which cases, to include, will clearly help determine the outcome of our results. If we decide to select a high number of cases, we often say that we are conducting large-N research. Large-N research is when the number of observations or cases is large enough where we would need mathematical, usually statistical, techniques to discover and interpret any correlations or causations. In order for a large-N analysis to yield any relevant findings, a number of conventions need to be observed. First, the sample needs to be representative of the studied population. Thus, if we wanted to understand the long-term effects of COVID, we would need to know the approximate details of those who contracted the virus. Once we know the parameters of the population, we can then determine a sample that represents the larger population. For example, women make up 55% of all long-term COVID survivors. Thus, any sample we generate needs to be at least 55% women.

Second, some kind of randomization technique needs to be involved in large-N research. So not only must your sample be representative, it must also randomly select people within that sample. In other words, we must have a large selection of people that fit within the population criteria, and then randomly select from those pools. Randomization would help to reduce bias in the study. Also, when cases (people with long-term COVID) are randomly chosen they tend to ensure a fairer representation of the studied population. Third, your sample needs to be large enough, hence the large-N designation for any conclusions to have any external validity. Generally speaking, the larger the number of observations/cases in the sample, the more validity we can have in the study. There is no magic number, but if using the above example, our sample of long-term COVID patients should be at least over 750 people, with an aim of around 1,200 to 1,500 people.

When it comes to comparative politics, we rarely ever reach the numbers typically used in large-N research. There are about 200 fully recognized countries, with about a dozen partially recognized countries, and even fewer areas or regions of study, such as Europe or Latin America. Given this, what is the strategy when one case, or a few cases, are being studied? What happens if we are only wanting to know the COVID-19 response in the United States, and not the rest of the world? How do we randomize this to ensure our results are not biased or are representative? These and other questions are legitimate issues that many comparativist scholars face when completing research. Does randomization work with case studies? Gerring suggests that it does not, as “any given sample may be widely representative” (pg. 87). Thus, random sampling is not a reliable approach when it comes to case studies. And even if the randomized sample is representative, there is no guarantee that the gathered evidence would be reliable.

One can make the argument that case selection may not be as important in large-N studies as they are in small-N studies. In large-N research, potential errors and/or biases may be ameliorated, especially if the sample is large enough. This is not always what happens, errors and biases most certainly can exist in large-N research. However, incorrect or biased inferences are less of a worry when we have 1,500 cases versus 15 cases. In small-N research, case selection simply matters much more.

This is why Blatter and Haverland (2012) write that, “case studies are ‘case-centered’, whereas large-N studies are ‘variable-centered’". In large-N studies we are more concerned with the conceptualization and operationalization of variables. Thus, we want to focus on which data to include in the analysis of long-term COVID patients. If we wanted to survey them, we would want to make sure we construct questions in appropriate ways. For almost all survey-based large-N research, the question responses themselves become the coded variables used in the statistical analysis.

Case selection can be driven by a number of factors in comparative politics, with the first two approaches being the more traditional. First, it can derive from the interests of the researcher(s). For example, if the researcher lives in Germany, they may want to research the spread of COVID-19 within the country, possibly using a subnational approach where the researcher may compare infection rates among German states. Second, case selection may be driven by area studies. This is still based on the interests of the researcher as generally speaking scholars pick areas of studies due to their personal interests. For example, the same researcher may research COVID-19 infection rates among European Union member-states. Finally, the selection of cases selected may be driven by the type of case study that is utilized. In this approach, cases are selected as they allow researchers to compare their similarities or their differences. Or, a case might be selected that is typical of most cases, or in contrast, a case or cases that deviate from the norm. We discuss types of case studies and their impact on case selection below.

Types of Case Studies: Descriptive vs. Causal

There are a number of different ways to categorize case studies. One of the most recent ways is through John Gerring. He wrote two editions on case study research (2017) where he posits that the central question posed by the researcher will dictate the aim of the case study. Is the study meant to be descriptive? If so, what is the researcher looking to describe? How many cases (countries, incidents, events) are there? Or is the study meant to be causal, where the researcher is looking for a cause and effect? Given this, Gerring categorizes case studies into two types: descriptive and causal.

Descriptive case studies are “not organized around a central, overarching causal hypothesis or theory” (pg. 56). Most case studies are descriptive in nature, where the researchers simply seek to describe what they observe. They are useful for transmitting information regarding the studied political phenomenon. For a descriptive case study, a scholar might choose a case that is considered typical of the population. An example could involve researching the effects of the pandemic on medium-sized cities in the US. This city would have to exhibit the tendencies of medium-sized cities throughout the entire country. First, we would have to conceptualize what we mean by ‘a medium-size city’. Second, we would then have to establish the characteristics of medium-sized US cities, so that our case selection is appropriate. Alternatively, cases could be chosen for their diversity . In keeping with our example, maybe we want to look at the effects of the pandemic on a range of US cities, from small, rural towns, to medium-sized suburban cities to large-sized urban areas.

Causal case studies are “organized around a central hypothesis about how X affects Y” (pg. 63). In causal case studies, the context around a specific political phenomenon or phenomena is important as it allows for researchers to identify the aspects that set up the conditions, the mechanisms, for that outcome to occur. Scholars refer to this as the causal mechanism , which is defined by Falleti & Lynch (2009) as “portable concepts that explain how and why a hypothesized cause, in a given context, contributes to a particular outcome”. Remember, causality is when a change in one variable verifiably causes an effect or change in another variable. For causal case studies that employ causal mechanisms, Gerring divides them into exploratory case-selection, estimating case-selection, and diagnostic case-selection. The differences revolve around how the central hypothesis is utilized in the study.

Exploratory case studies are used to identify a potential causal hypothesis. Researchers will single out the independent variables that seem to affect the outcome, or dependent variable, the most. The goal is to build up to what the causal mechanism might be by providing the context. This is also referred to as hypothesis generating as opposed to hypothesis testing. Case selection can vary widely depending on the goal of the researcher. For example, if the scholar is looking to develop an ‘ideal-type’, they might seek out an extreme case. An ideal-type is defined as a “conception or a standard of something in its highest perfection” (New Webster Dictionary). Thus, if we want to understand the ideal-type capitalist system, we want to investigate a country that practices a pure or ‘extreme’ form of the economic system.

Estimating case studies start with a hypothesis already in place. The goal is to test the hypothesis through collected data/evidence. Researchers seek to estimate the ‘causal effect’. This involves determining if the relationship between the independent and dependent variables is positive, negative, or ultimately if no relationship exists at all. Finally, diagnostic case studies are important as they help to “confirm, disconfirm, or refine a hypothesis” (Gerring 2017). Case selection can also vary in diagnostic case studies. For example, scholars can choose an least-likely case, or a case where the hypothesis is confirmed even though the context would suggest otherwise. A good example would be looking at Indian democracy, which has existed for over 70 years. India has a high level of ethnolinguistic diversity, is relatively underdeveloped economically, and a low level of modernization through large swaths of the country. All of these factors strongly suggest that India should not have democratized, or should have failed to stay a democracy in the long-term, or have disintegrated as a country.

Most Similar/Most Different Systems Approach

The discussion in the previous subsection tends to focus on case selection when it comes to a single case. Single case studies are valuable as they provide an opportunity for in-depth research on a topic that requires it. However, in comparative politics, our approach is to compare. Given this, we are required to select more than one case. This presents a different set of challenges. First, how many cases do we pick? This is a tricky question we addressed earlier. Second, how do we apply the previously mentioned case selection techniques, descriptive vs. causal? Do we pick two extreme cases if we used an exploratory approach, or two least-likely cases if choosing a diagnostic case approach?

Thankfully, an English scholar by the name of John Stuart Mill provided some insight on how we should proceed. He developed several approaches to comparison with the explicit goal of isolating a cause within a complex environment. Two of these methods, the 'method of agreement' and the 'method of difference' have influenced comparative politics. In the 'method of agreement' two or more cases are compared for their commonalities. The scholar looks to isolate the characteristic, or variable, they have in common, which is then established as the cause for their similarities. In the 'method of difference' two or more cases are compared for their differences. The scholar looks to isolate the characteristic, or variable, they do not have in common, which is then identified as the cause for their differences. From these two methods, comparativists have developed two approaches.

What Is the Most Similar Systems Design (MSSD)?

This approach is derived from Mill’s ‘method of difference’. In a Most Similar Systems Design Design, the cases selected for comparison are similar to each other, but the outcomes differ in result. In this approach we are interested in keeping as many of the variables the same across the elected cases, which for comparative politics often involves countries. Remember, the independent variable is the factor that doesn’t depend on changes in other variables. It is potentially the ‘cause’ in the cause and effect model. The dependent variable is the variable that is affected by, or dependent on, the presence of the independent variable. It is the ‘effect’. In a most similar systems approach the variables of interest should remain the same.

A good example involves the lack of a national healthcare system in the US. Other countries, such as New Zealand, Australia, Ireland, UK and Canada, all have robust, publicly accessible national health systems. However, the US does not. These countries all have similar systems: English heritage and language use, liberal market economies, strong democratic institutions, and high levels of wealth and education. Yet, despite these similarities, the end results vary. The US does not look like its peer countries. In other words, why do we have similar systems producing different outcomes?

What Is the Most Different Systems Design (MDSD)?

This approach is derived from Mill’s ‘method of agreement’. In a Most Different System Design, the cases selected are different from each other, but result in the same outcome. In this approach, we are interested in selecting cases that are quite different from one another, yet arrive at the same outcome. Thus, the dependent variable is the same. Different independent variables exist between the cases, such as democratic v. authoritarian regime, liberal market economy v. non-liberal market economy. Or it could include other variables such as societal homogeneity (uniformity) vs. societal heterogeneity (diversity), where a country may find itself unified ethnically/religiously/racially, or fragmented along those same lines.

A good example involves the countries that are classified as economically liberal. The Heritage Foundation lists countries such as Singapore, Taiwan, Estonia, Australia, New Zealand, as well as Switzerland, Chile and Malaysia as either free or mostly free. These countries differ greatly from one another. Singapore and Malaysia are considered flawed or illiberal democracies (see chapter 5 for more discussion), whereas Estonia is still classified as a developing country. Australia and New Zealand are wealthy, Malaysia is not. Chile and Taiwan became economically free countries under the authoritarian military regimes, which is not the case for Switzerland. In other words, why do we have different systems producing the same outcome?

Comparative Case Studies: Methodological Discussion

• Open Access
• First Online: 25 May 2022

Cite this chapter

You have full access to this open access chapter

• Marcelo Parreira do Amaral 7  

Part of the book series: Palgrave Studies in Adult Education and Lifelong Learning ((PSAELL))

12k Accesses

6 Citations

Case Study Research has a long tradition and it has been used in different areas of social sciences to approach research questions that command context sensitiveness and attention to complexity while tapping on multiple sources. Comparative Case Studies have been suggested as providing effective tools to understanding policy and practice along three different axes of social scientific research, namely horizontal (spaces), vertical (scales), and transversal (time). The chapter, first, sketches the methodological basis of case-based research in comparative studies as a point of departure, also highlighting the requirements for comparative research. Second, the chapter focuses on presenting and discussing recent developments in scholarship to provide insights on how comparative researchers, especially those investigating educational policy and practice in the context of globalization and internationalization, have suggested some critical rethinking of case study research to account more effectively for recent conceptual shifts in the social sciences related to culture, context, space and comparison. In a third section, it presents the approach to comparative case studies adopted in the European research project YOUNG_ADULLLT that has set out to research lifelong learning policies in their embeddedness in regional economies, labour markets and individual life projects of young adults. The chapter is rounded out with some summarizing and concluding remarks.

You have full access to this open access chapter,  Download chapter PDF

Similar content being viewed by others

Introduction to the Book and the Comparative Study

Theoretical and Methodological Considerations

Main findings and discussion.

• Case-based research
• Comparative case studies

1 Introduction

Exploring landscapes of lifelong learning in Europe is a daunting task as it involves a great deal of differences across places and spaces; it entails attending to different levels and dimensions of the phenomena at hand, but not least it commands substantial sensibility to cultural and contextual idiosyncrasies. As such, case-based methodologies come to mind as tested methodological approaches to capturing and examining singular configurations such as the local settings in focus in this volume, in which lifelong learning policies for young people are explored in their multidimensional reality. The ensuing question, then, is how to ensure comparability across cases when departing from the assumption that cases are unique. Recent debates in Comparative and International Education (CIE) research are drawn from that offer important insights into the issues involved and provide a heuristic approach to comparative cases studies. Since the cases focused on in the chapters of this book all stem from a common European research project, the comparative case study methodology allows us to at once dive into the specifics and uniqueness of each case while at the same time pay attention to common treads at the national and international (European) levels.

The chapter, first, sketches the methodological basis of case-based research in comparative studies as a point of departure, also highlighting the requirements in comparative research. In what follows, second, the chapter focuses on presenting and discussing recent developments in scholarship to provide insights on how comparative researchers, especially those investigating educational policy and practice in the context of globalization and internationalization, have suggested some critical rethinking of case study research to account more effectively for recent conceptual shifts in the social sciences related to culture, context, space and comparison. In a third section, it presents the approach to comparative case studies adopted in the European research project YOUNG_ADULLLT that has set out to research lifelong learning policies in their embeddedness in regional economies, labour markets and individual life projects of young adults. The chapter is rounded out with some summarizing and concluding remarks.

2 Case-Based Research in Comparative Studies

In the past, comparativists have oftentimes regarded case study research as an alternative to comparative studies proper. At the risk of oversimplification: methodological choices in comparative and international education (CIE) research, from the 1960s onwards, have fallen primarily on either single country (small n) contextualized comparison, or on cross-national (usually large n, variable) decontextualized comparison (see Steiner-Khamsi, 2006a , 2006b , 2009). These two strands of research—notably characterized by Development and Area Studies on the one side and large-scale performance surveys of the International Association for the Evaluation of Educational Achievement (IEA) type, on the other—demarcated their fields by resorting to how context and culture were accounted for and dealt with in the studies they produced. Since the turn of the century, though, comparativists are more comfortable with case study methodology (see Little, 2000 ; Vavrus and Bartlett 2006 , 2009 ; Bartlett & Vavrus, 2017 ) and diagnoses of an “identity crisis” of the field due to a mass of single-country studies lacking comparison proper (see Schriewer, 1990 ; Wiseman & Anderson, 2013 ) started dying away. Greater acceptance of and reliance on case-based methodology has been related with research on policy and practice in the context of globalization and coupled with the intention to better account for culture and context, generating scholarship that is critical of power structures, sensitive to alterity and of other ways of knowing.

The phenomena that have been coined as constituting “globalization” and “internationalization” have played, as mentioned, a central role in the critical rethinking of case study research. In researching education under conditions of globalization, scholars placed increasing attention on case-based approaches as opportunities for investigating the contemporary complexity of policy and practice. Further, scholarly debates in the social sciences and the humanities surrounding key concepts such as culture, context, space, and place but also comparison have also contributed to a reconceptualization of case study methodology in CIE. In terms of the requirements for such an investigation, scholarship commands an adequate conceptualization that problematizes the objects of study and that does not take them as “unproblematic”, “assum[ing] a constant shared meaning”; in short, objects of study that are “fixed, abstract and absolute” (Fine, quoted in Dale & Robertson, 2009 , p. 1114). Case study research is thus required to overcome methodological “isms” in their research conceptualization (see Dale & Robertson, 2009 ; Robertson & Dale, 2017 ; see also Lange & Parreira do Amaral, 2018 ). In response to these requirements, the approaches to case study discussed in CIE depart from a conceptualization of the social world as always dynamic, emergent, somewhat in motion, and always contested. This view considers the fact that the social world is culturally produced and is never complete or at a standstill, which goes against an understanding of case as something fixed or natural. Indeed, in the past cases have often been understood almost in naturalistic ways, as if they existed out there, waiting for researchers to “discover” them. Usually, definitions of case study also referred to inquiry that aims at elucidating features of a phenomenon to yield an understanding of why, how and with what consequences something happens. One can easily find examples of cases understood simply as sites to observe/measure variables—in a nomothetic cast—or examples, where cases are viewed as specific and unique instances that can be examined in the idiographic paradigm. In contrast, rather than taking cases as pre-existing entities that are defined and selected as cases, recent case-oriented research has argued for a more emergent approach which recognizes that boundaries between phenomenon and context are often difficult to establish or overlap. For this reason, researchers are incited to see this as an exercise of “casing”, that is, of case construction. In this sense, cases here are seen as complex systems (Ragin & Becker, 1992 ) and attention is devoted to the relationships between the parts and the whole, pointing to the relevance of configurations and constellations within as well as across cases in the explanation of complex and contingent phenomena. This is particularly relevant for multi-case, comparative research since the constitution of the phenomena that will be defined, as cases will differ. Setting boundaries will thus also require researchers to account for spatial, scalar (i.e., level or levels with which a case is related) and temporal aspects.

Further, case-based research is also required to account for multiple contexts while not taking them for granted. One of the key theoretical and methodological consequences of globalization for CIE is that it required us to recognize that it alters the nature and significance of what counts as contexts (see Parreira do Amaral, 2014 ). According to Dale ( 2015 ), designating a process, or a type of event, or a particular organization, as a context, entails bestowing a particular significance on them, as processes, events, and so on that are capable of affecting other processes and events. The key point is that rather than being so intrinsically, or naturally, contexts are constructed as “contexts”. In comparative research, contexts have been typically seen as the place (or the variables) that enable us to explain why what happens in one case is different from what happens another case; what counts as context then is seen as having the same effect everywhere, although the forms it takes vary substantially (see Dale, 2015 ). In more general terms, recent case study approaches aim at accounting for the increasing complexity of the contexts in which they are embedded, which, in turn, is related to the increasing impact of globalization as the “context of contexts” (Dale, 2015 , p. 181f; see also Carter & Sealey, 2013 ; Mjoset, 2013 ). It also aims at accounting for overlapping contexts. Here it is important to note that contexts are not only to be seen in spatio-geographical terms (i.e., local, regional, national, international), but contexts may also be provided by different institutional and/or discursive contexts that create varying opportunity structures (Dale & Parreira do Amaral, 2015 ; see also Chap. 2 in this volume). What one can call temporal contexts also plays an important role, for what happens in the case unfolds as embedded not only in historical time, but may be related to different temporalities (see the concept of “timespace” as discussed by Lingard & Thompson, 2016 ) and thus are influenced by path dependence or by specific moments of crisis (Rhinard, 2019 ; see also McLeod, 2016 ). Moreover, in CIE research, the social-cultural production of the world is influenced by developments throughout the globe that take place at various places and on several scales, which in turn influence each other, but in the end, become locally relevant in different facets. As Bartlett and Vavrus write, “context is not a primordial or autonomous place; it is constituted by social interactions, political processes, and economic developments across scales and times.” ( Bartlett & Vavrus, 2017 , p. 14). Indeed, in this sense, “context is not a container for activity, it is the activity” (Bartlett & Vavrus, 2017 , p. 12, emphasis in orig.).

Also, dealing with the complexity of education policy and practice requires us to transcend the dichotomy of idiographic versus nomothetic approaches to causation. Here, it can be argued that case studies allow us to grasp and research the complexity of the world, thus offering conceptual and methodological tools to explore how phenomena viewed as cases “depend on all of the whole, the parts, the interactions among parts and whole, and the interactions of any system with other complex systems among which it is nested and with which it intersects” (Byrne, 2013 , p. 2). The understanding of causation that undergirds recent developments in case-based research aims at generalization, yet it resists ambitions to establishing universal laws in social scientific research. Focus is placed on processes while tracking the relevant factors, actors and features that help explain the “how” and the “why” questions (Bartlett and Vavrus 2017 , p. 38ff), and on “causal mechanisms”, as varying explanations of outcomes within and across cases, always contingent on interaction with other variables and dependent contexts (see Byrne, 2013 ; Ragin, 2000 ). In short, the nature of causation underlying the recent case study approaches in CIE is configurational and not foundational.

This is also in line with how CIE research regards education practice, research, and policy as a socio-cultural practice. And it refers to the production of social and cultural worlds through “social actors, with diverse motives, intentions, and levels of influence, [who] work in tandem with and/or in response to social forces” (Bartlett and Vavrus 2017 , p. 1). From this perspective, educational phenomena, such as in policymaking, are seen as a “deeply political process of cultural production engaged in and shaped by social actors in disparate locations who exert incongruent amounts of influence over the design, implementation, and evaluation of policy” ( Bartlett & Vavrus, 2017 , p. 1f). Culture here is understood in non-static and complex ways that reinforce the “importance of examining processes of sense-making as they develop over time, in distinct settings, in relation to systems of power and inequality, and in increasingly interconnected conversation with actors who do not sit physically within the circle drawn around the traditional case” (Bartlett & Vavrus, 2017 , p. 11, emphasis in orig.).

In sum, the approaches to case study put forward in CIE provide conceptual and methodological tools that allow for an analysis of education in the global context throughout scale, space, and time, which is always regarded as complexly integrated and never as isolated or independent. The following subsection discusses Comparative Case Studies (CCS) as suggested in recent comparative scholarship, which aims at attending to the methodological requirements discussed above by integrating horizontal, vertical, and transversal dimensions of comparison.

2.1 Comparative Case Studies: Horizontal, Vertical and Transversal Dimensions

Building up on their previous work on vertical case studies (Bartlett and Vavrus 2017 ; Vavrus & Bartlett, 2006 , 2009 ), Frances Vavrus and Lesley Bartlett have proposed a comparative approach to case study research that aims at meeting the requirements of culture and context sensitive research as discussed in this special issue.

As a research approach, CCS offers two theoretical-methodological lenses to research education as a socio-cultural practice. These lenses represent different views on the research object and account for the complexity of education practice, policy, and research in globalized contexts. The first lens is “context-sensitive”, which focuses on how social practices and interactions constitute and produce social contexts. As quoted above, from the perspective of a socio-cultural practice, “context is not a container for activity, it is the activity” (Vavrus and Bartlett 2017: 12, emphasis in orig.). The settings that influence and condition educational phenomena are culturally produced in different and sometimes overlapping (spatial, institutional, discursive, temporal) contexts as just mentioned. The second CCS lens is “culture-sensitive” and focuses on how socio-cultural practices produce social structures. As such, culture is a process that is emergent, dynamic, and constitutive of meaning-making as well as social structuration.

The CCS approach aims at studying educational phenomena throughout scale, time, and space by providing three axes for a “studying through” of the phenomena in question. As stated by Lesley Bartlett and Frances Vavrus with reference to comparative analyses of global education policy:

the horizontal axis compares how similar policies unfold in distinct locations that are socially produced […] and ‘complexly connected’ […]. The vertical axis insists on simultaneous attention to and across scales […]. The transversal comparison historically situates the processes or relations under consideration (Bartlett and Vavrus 2017 : 3, emphasis in orig.).

These three axes allow for a methodological conceptualization of “policy formation and appropriation across micro-, meso-, and macro levels” by not theorizing them as distinct or unrelated (Bartlett and Vavrus 2017 , p. 4). In following Latour, they state:

the macro is neither “above” nor “below” the intersections but added to them as another of their connections’ […]. In CCS research, one would pay close attention to how actions at different scales mutually influence one another (Bartlett and Vavrus 2017 , p. 13f, emphasis in orig.)

Thus, these three axes contain

processes across space and time; and [the CCS as a research design] constantly compares what is happening in one locale with what has happened in other places and historical moments. These forms of comparison are what we call horizontal, vertical, and transversal comparisons (Bartlett and Vavrus 2017 , p. 11, emphasis in orig.)

In terms of the three axes along with comparison is organized, the authors state that horizontal comparison commands attention to how historical and contemporary processes have variously influenced the “cases”, which might be constructed by focusing “people, groups of people, sites, institutions, social movements, partnerships, etc.” (Bartlett and Vavrus 2017 , p. 53) Horizontal comparisons eschew pressing categories resultant from one case others, which implies including multiple cases at the same scale in a comparative case study, while at the same time attending to “valuable contextual information” about each of them. Horizontal comparisons use units of analysis that are homologous, that is, equivalent in terms of shape, function, or institutional/organizational nature (for instance, schools, ministries, countries, etc.) ( Bartlett & Vavrus, 2017 , p. 53f). Similarly, comparative case studies may also entail tracing a phenomenon across sites, as in multi-sited ethnography (see Coleman & von Hellermann, 2012 ; Marcus, 1995 ).

Vertical comparison, in turn, does not simply imply the comparison of levels; rather it involves analysing networks and their interrelationships at different scales. For instance, in the study of policymaking in a specific case, vertical comparison would consider how actors at different scales variably respond to a policy issued at another level—be it inter−/supranational or at the subnational level. CCS assumes that their different appropriation of policy as discourse and as practice is often due to different histories of racial, ethnic, or gender politics in their communities that appropriately complicate the notion of a single cultural group (Bartlett and Vavrus 2017 , p. 73f). Establishing what counts as context in such a study would be done “by tracing the formation and appropriation of a policy” at different scales; and “by tracing the processes by which actors and actants come into relationship with one another and form non-permanent assemblages aimed at producing, implementing, resisting, and appropriating policy to achieve particular aims” ( Bartlett & Vavrus, 2017 , p. 76). A further element here is that, in this way, one may counter the common problem that comparison of cases (oftentimes countries) usually overemphasizes boundaries and treats them as separated or as self-sustaining containers, when, in reality, actors and institutions at other levels/scales significantly impact policymaking (Bartlett & Vavrus, 2017 ).

In terms of the transversal axis of comparison, Bartlett and Vavrus argue that the social phenomena of interest in a case study have to be seen in light of their historical development (Bartlett & Vavrus, 2017 , p. 93), since these “historical roots” impacted on them and “continues to reverberate into the present, affecting economic relations and social issues such as migration and educational opportunities.” As such, understanding what goes on in a case requires to “understand how it came to be in the first place.” ( Bartlett & Vavrus, 2017 , p. 93) argue:

history offers an extensive fount of evidence regarding how social institutions function and how social relations are similar and different around the world. Historical analysis provides an essential opportunity to contrast how things have changed over time and to consider what has remained the same in one locale or across much broader scales. Such historical comparison reveals important insights about the flexible cultural, social, political, and economic systems humans have developed and sustained over time (Bartlett & Vavrus, 2017 , p. 94).

Further, time and space are intimately related and studying the historical development of the social phenomena of interest in a case study “allows us to assess evidence and conflicting interpretations of a phenomenon,” but also to interrogate our own assumptions about them in contemporary times (Bartlett and Vavrus 2017 ), thus analytically sharpening our historical analyses.

As argued by the authors, researching the global dimension of education practice, research or policy aims at a “studying through” of phenomena horizontally, vertically, and transversally. That is, comparative case study builds on an emergent research design and on a strong process orientation that aims at tracing not only “what”, but also “why” and “how” phenomena emerge and evolve. This approach entails “an open-ended, inductive approach to discover what […] meanings and influences are and how they are involved in these events and activities—an inherently processual orientation” (Bartlett and Vavrus 2017 , p. 7, emphasis in orig.).

The emergent research design and process orientation of the CCS relativizes a priori, somewhat static notions of case construction in CIE and emphasizes the idea of a processual “casing”. The process of casing put forward by CCS has to be understood as a dynamic and open-ended embedding of “cased” research phenomena within moments of scale, space, and time that produce varying sets of conditions or configurations.

In terms of comparison, the primary logic is well in line with more sophisticated approaches to comparison that not simply establish relationships between observable facts or pre-existing cases; rather, the comparative logic aims at establishing “relations between sets of relationships”, as argued by Jürgen Schriewer:

[the] specific method of science dissociates comparison from its quasi-natural union with resemblances; the interest in identifying similarities shifts from the level of factual contents to the level of generalizable relationships. […] One of the primary ways of extending their scope, or examining their explanatory power, is the controlled introduction of varying sets of conditions. The logic of relating relationships, which distinguishes the scientific method of comparison, comes close to meeting these requirements by systematically exploring and analysing sociocultural differences with respect to scrutinizing the credibility of theories, models or constructs (Schriewer, 1990 , p. 36).

The notion of establishing relations between sets of relationships allows to treat cases not as homogeneous (thus avoiding a universalizing notion of comparison); it establishes comparability not along similarity but based on conceptual, functional and/or theoretical equivalences and focuses on reconstructing ‘varying sets of conditions’ that are seen as relevant in social scientific explanation and theorizing, and to which then comparative case studies may contribute.

The following section aims presents the adaptation and application of a comparative case study approach in the YOUNG_ADULLLT research project.

3 Exploring Landscapes of Lifelong Learning through Case Studies

This section illustrates the usage of comparative case studies by drawing from research conducted in a European research project upon which the chapters in this volume are based. The project departed from the observation that most current European lifelong learning (LLL) policies have been designed to create economic growth and, at the same time, guarantee social inclusion and argued that, while these objectives are complementary, they are, however, not linearly nor causally related and, due to distinct orientations, different objectives, and temporal horizons, conflicts and ambiguities may arise. The project was designed as a mixed-method comparative study and aimed at results at the national, regional, and local levels, focusing in particular on policies targeting young adults in situations of near social exclusion. Using a multi-level approach with qualitative and quantitative methods, the project conducted, amongst others, local/regional 18 case studies of lifelong learning policies through a multi-method and multi-level design (see Parreira do Amaral et al., 2020 for more information). The localisation of the cases in their contexts was carried out by identifying relevant areas in terms of spatial differentiation and organisation of social and economic relations. The so defined “functional regions” allowed focus on territorial units which played a central role within their areas, not necessarily overlapping with geographical and/or administrative borders. Footnote 1

Two main objectives guided the research: first, to analyse policies and programmes at the regional and local level by identifying policymaking networks that included all social actors involved in shaping, formulating, and implementing LLL policies for young adults; second, to recognize strengths and weaknesses (overlapping, fragmented or unfocused policies and projects), thus identifying different patterns of LLL policymaking at regional level, and investigating their integration with the labour market, education and other social policies. The European research project focused predominantly on the differences between the existing lifelong learning policies in terms of their objectives and orientations and questioned their impact on young adults’ life courses, especially those young adults who find themselves in vulnerable positions. What concerned the researchers primarily was the interaction between local institutional settings, education, labour markets, policymaking landscapes, and informal initiatives that together nurture the processes of lifelong learning. They argued that it is by inquiring into the interplay of these components that the regional and local contexts of lifelong learning policymaking can be better assessed and understood. In this regard, the multi-layered approach covered a wide range of actors and levels and aimed at securing compatibility throughout the different phases and parts of the research.

The multi-level approach adopted aimed at incorporating the different levels from transnational to regional/local to individual, that is, the different places, spaces, and levels with which policies are related. The multi-method design was used to bring together the results from the quantitative, qualitative and policy/document analysis (for a discussion: Parreira do Amaral, 2020 ).

Studying the complex relationships between lifelong learning (LLL) policymaking on the one hand, and young adults’ life courses on the other, requires a carefully established research approach. This task becomes even more challenging in the light of the diverse European countries and their still more complex local and regional structures and institutions. One possible way of designing a research framework able to deal with these circumstances clearly and coherently is to adopt a multi-level or multi-layered approach. This approach recognises multiple levels and patterns of analysis and enables researchers to structure the workflow according to various perspectives. It was this multi-layered approach that the research consortium of YOUNG_ADULLLT adopted and applied in its attempts to better understand policies supporting young people in their life course.

3.1 Constructing Case Studies

In constructing case studies, the project did not apply an instrumental approach focused on the assessment of “what worked (or not)?” Rather, consistently with Bartlett and Vavrus’s proposal (Bartlett & Vavrus, 2017 ), the project decided to “understand policy as a deeply political process of cultural production engaged in and shaped by social actors in disparate locations who exert incongruent amounts of influence over the design, implementation, and evaluation of policy” ( Bartlett & Vavrus, 2017 , p. 1f). This was done in order to enhance the interactive and relational dimension among actors and levels, as well as their embeddedness in local infra-structures (education, labour, social/youth policies) according to project’s three theoretical perspectives. The analyses of the information and data integrated by our case study approach aimed at a cross-reading of the relations among the macro socio-economic dimensions, structural arrangements, governance patterns, addressee biographies and mainstream discourses that underlie the process of design and implementation of the LLL policies selected as case study. The subjective dimensions of agency and sense-making animated these analyses, and the multi-level approach contextualized them from the local to the transnational levels. Figure 3.1 below represents the analytical approach to the research material gathered in constructing the case studies. Specifically, it shows the different levels, from the transnational level down to the addressees.

Multi-level and multi-method approach to case studies in YOUNG_ADULLLT. Source: Palumbo et al., 2019

The project partners aimed at a cross-dimensional construction of the case studies, and this implied the possibility of different entry points, for instance by moving the analytical perspective top-down or bottom-up, as well as shifting from left to right of the matrix and vice versa. Considering the “horizontal movement”, the multidimensional approach has enabled taking into consideration the mutual influence and relations among the institutional, individual, and structural dimensions (which in the project corresponded to the theoretical frames of CPE, LCR, and GOV). In addition, the “vertical movement” from the transnational to the individual level and vice versa was meant to carefully carry out a “study of flows of influence, ideas, and actions through these levels” (Bartlett and Vavrus 2017 , p. 11), emphasizing the correspondences/divergences among the perspectives of different actors at different levels. The transversal dimension, that is, the historical process, focused on the period after the financial crisis of 2007/2008 as it has impacted differently on the social and economic situations of young people, often resulting in stern conditions and higher competition in education and labour markets, which also called for a reassessment of existing policies targeting young adults in the countries studied.

Concerning the analyses, a further step included the translation of the conceptual model illustrated in Fig. 3.1 above into a heuristic table used to systematically organize the empirical data collected and guide the analyses cases constructed as multi-level and multidimensional phenomena, allowing for the establishment of interlinkages and relationships. By this approach, the analysis had the possibility of grasping the various levels at which LLL policies are negotiated and displaying the interplay of macro-structures, regional environments and institutions/organizations as well as individual expectations. Table 3.1 illustrates the operationalization of the data matrix that guided the work.

In order to ensure the presentability and intelligibility of the results, Footnote 2 a narrative approach to case studies analysis was chosen whose main task was one of “storytelling” aimed at highlighting what made each case unique and what difference it makes for LLL policymaking and to young people’s life courses. A crucial element of this entails establishing relations “between sets of relationships”, as argued above.

LLL policies were selected as starting points from which the cases themselves could be constructed and of which different stories could be developed. That stories can be told differently does not mean that they are arbitrary, rather this refers to different ways of accounting for the embedding of the specific case to its context, namely the “diverging policy frameworks, patterns of policymaking, networks of implementation, political discourses and macro-structural conditions at local level” (see Palumbo et al., 2020 , p. 220). Moreover, developing different narratives aimed at representing the various voices of the actors involved in the process—from policy-design and appropriation through to implementation—and making the different stakeholders’ and addressees’ opinions visible, creating thus intelligible narratives for the cases (see Palumbo et al., 2020 ). Analysing each case started from an entry point selected, from which a story was told. Mainly, two entry points were used: on the one hand, departing from the transversal dimension of the case and which focused on the evolution of a policy in terms of its main objectives, target groups, governance patterns and so on in order to highlight the intended and unintended effects of the “current version” of the policy within its context and according to the opinions of the actors interviewed. On the other hand, biographies were selected as starting points in an attempt to contextualize the life stories within the biographical constellations in which the young people came across the measure, the access procedures, and how their life trajectories continued in and possibly after their participation in the policy (see Palumbo et al., 2020 for examples of these narrative strategies).

4 Concluding Remarks

This chapter presented and discussed the methodological basis and requirements of conducting case studies in comparative research, such as those presented in the subsequent chapters of this volume. The Comparative Case Study approach suggested in the previous discussion offers productive and innovative ways to account sensitively to culture and contexts; it provides a useful heuristic that deals effectively with issues related to case construction, namely an emergent and dynamic approach to casing, instead of simply assuming “bounded”, pre-defined cases as the object of research; they also offer a helpful procedural, configurational approach to “causality”; and, not least, a resourceful approach to comparison that allows researchers to respect the uniqueness and integrity of each case while at the same time yielding insights and results that transcend the idiosyncrasy of the single case. In sum, CCS offers a sound approach to CIE research that is culture and context sensitive.

For a discussion of the concept of functional region, see Parreira do Amaral et al., 2020 .

This analytical move is in line with recent developments that aim at accounting for a cultural turn (Jameson, 1998 ) or ideational turn (Béland & Cox, 2011 ) in policy analysis methodology, called interpretive policy analysis (see Münch, 2016 ).

Bartlett, L., & Vavrus, F. (2017). Rethinking case study research. A comparative approach . Routledge.

Google Scholar  

Béland, D., & Cox, R. H. (Eds.). (2011). Ideas and politics in social science research . Oxford University Press.

Byrne, D. (2013). Introduction. Case-based methods: Why we need them; what they are; how to do them. In D. Byrne & C. C. Ragin (Eds.), The SAGE handbook of case-based methods (pp. 1–13). SAGE.

Carter, B., & Sealey, A. (2013). Reflexivity, realism and the process of casing. In D. Byrne & C. C. Ragin (Eds.), The SAGE handbook of case-based methods (pp. 69–83). SAGE.

Coleman, S., & von Hellermann, P. (Eds.). (2012). Multi-sited ethnography: Problems and possibilities in the translocation of research methods . Routledge.

Dale, R., & Parreira do Amaral, M. (2015). Discursive and institutional opportunity structures in the governance of educational trajectories. In M. P. do Amaral, R. Dale, & P. Loncle (Eds.), Shaping the futures of young Europeans: Education governance in eight European countries (pp. 23–42). Symposium Books.

Dale, R., & Robertson, S. (2009). Beyond methodological ʻismsʼ in comparative education in an era of globalisation. In R. Cowen & A. M. Kazamias (Eds.), International handbook of comparative education (pp. 1113–1127). Springer Netherlands.

Chapter   Google Scholar  

Dale, R. (2015). Globalisierung in der Vergleichenden Erziehungswissenschaft re-visited: Die Relevanz des Kontexts des Kontexts. In M. P. do Amaral & S.K. Amos (Hrsg.) (Eds.), Internationale und Vergleichende Erziehungswissenschaft. Geschichte, Theorie, Methode und Forschungsfelder (pp. 171–187). Waxmann.

Jameson, F. (1998). The cultural turn. Selected writings on the postmodern. 1983–1998 . Verso.

Lange, S., & Parreira do Amaral, M. (2018). Leistungen und Grenzen internationaler und vergleichender Forschung— Regulative Ideen für die methodologische Reflexion? Tertium Comparationis, 24 (1), 5–31.

Lingard, B., & Thompson, G. (2016). Doing time in the sociology of education. British Journal of Sociology of Education, 38 (1), 1–12.

Article   Google Scholar  

Little, A. (2000). Development studies and comparative education: Context, content, comparison and contributors. Comparative Education, 36 (3), 279–296.

Marcus, G. E. (1995). Ethnography in/of the world system: The emergence of multi-sited ethnography. Annual Review of Anthropology, 24 , 95–117.

McLeod, J. (2016). Marking time, making methods: Temporality and untimely dilemmas in the sociology of youth and educational change. British Journal of Sociology of Education, 38 (1), 13–25.

Mjoset, L. (2013). The Contextualist approach to social science methodology. In D. Byrne & C. C. Ragin (Eds.), The SAGE handbook of case-based methods (pp. 39–68). SAGE.

Münch, S. (2016). Interpretative Policy-Analyse. Eine Einführung . Springer VS.

Book   Google Scholar  

Palumbo, M., Benasso, S., Pandolfini, V., Verlage, T., & Walther, A. (2019). Work Package 7 Cross-case and cross-national Report. YOUNG_ADULLLT Working Paper. http://www.young-adulllt.eu/publications/working-paper/index.php . Accessed 31 Aug 2021.

Palumbo, M., Benasso, S., & Parreira do Amaral, M. (2020). Telling the story. Exploring lifelong learning policies for young adults through a narrative approach. In M. P. do Amaral, S. Kovacheva, & X. Rambla (Eds.), Lifelong learning policies for young adults in Europe. Navigating between knowledge and economy (pp. 217–239). Policy Press.

Parreira do Amaral, M. (2014). Globalisierung im Fokus vergleichender Forschung. In C. Freitag (Ed.), Methoden des Vergleichs. Komparatistische Methodologie und Forschungsmethodik in interdisziplinärer Perspektive (pp. 117–138). Budrich.

Parreira do Amaral, M. (2020). Lifelong learning policies for young adults in Europe: A conceptual and methodological discussion. In M. P. do Amaral, S. Kovacheva, & X. Rambla (Eds.), Lifelong learning policies for young adults in Europe. Navigating between knowledge and economy (pp. 3–20). Policy Press.

Parreira do Amaral, M., Lowden, K., Pandolfini, V., & Schöneck, N. (2020). Coordinated policy-making in lifelong learning: Functional regions as dynamic units. In M. P. do Amaral, S. Kovacheva, & X. Rambla (Eds.), Lifelong learning policies for young adults in Europe navigating between knowledge and economy (pp. 21–42). Policy Press.

Ragin, C. C., & Becker, H. (1992). What is a case? Cambridge UP.

Ragin, C. C. (2000). Fuzzy set social science . Chicago UP.

Rhinard, M. (2019). The Crisisification of policy-making in the European Union. Journal of Common Market Studies, 57 (3), 616–633.

Robertson, S., & Dale, R. (2017). Comparing policies in a globalizing world. Methodological reflections. Educação & Realidade, 42 (3), 859–876.

Schriewer, J. (1990). The method of comparison and the need for externalization: Methodological criteria and sociological concepts. In J. Schriewer & B. Holmes (Eds.), Theories and methods in comparative education (pp. 25–83). Lang.

Steiner-Khamsi, G. (2006a). The development turn in comparative and international education. European Education: Issues and Studies, 38 (3), 19–47.

Steiner-Khamsi, G. (2006b). U.S. social and educational research during the cold war: An interview with Harold J. Noah. European Education: Issues and Studies, 38 (3), 9–18.

Vavrus, F., & Bartlett, L. (2006). Comparatively knowing: Making a case for vertical case study. Current Issues in Comparative Education, 8 (2), 95–103.

Vavrus, F., & Bartlett, L. (Eds.). (2009). Critical approaches to comparative education: Vertical case studies from Africa, Europe, the Middle East, and the Americas . Palgrave Macmillan.

Wiseman, A. W., & Anderson, E. (2013). Introduction to part 3: Conceptual and methodological developments. In A. W. Wiseman & E. Anderson (Eds.), Annual review of comparative and international education 2013 (international perspectives on education and society, Vol. 20) (pp. 85–90). Emerald Group Publishing Limited.

Download references

Author information

Authors and affiliations.

Institute of Education, University of Münster, Münster, Germany

Marcelo Parreira do Amaral

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marcelo Parreira do Amaral .

Editor information

Editors and affiliations.

Department of Educational Sciences, University of Genoa, Genova, Italy

Sebastiano Benasso

Faculty of Teacher Education, University of Zagreb, Zagreb, Croatia

Dejana Bouillet

Faculty of Psychology and Education Sciences, University of Porto, Porto, Portugal

Tiago Neves

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and permissions

Copyright information

© 2022 The Author(s)

About this chapter

do Amaral, M.P. (2022). Comparative Case Studies: Methodological Discussion. In: Benasso, S., Bouillet, D., Neves, T., Parreira do Amaral, M. (eds) Landscapes of Lifelong Learning Policies across Europe. Palgrave Studies in Adult Education and Lifelong Learning. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-96454-2_3

Download citation

DOI : https://doi.org/10.1007/978-3-030-96454-2_3

Published : 25 May 2022

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-030-96453-5

Online ISBN : 978-3-030-96454-2

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Utility Menu

GA4 Tracking Code

Gen ed writes, writing across the disciplines at harvard college.

• Comparative Analysis

What It Is and Why It's Useful

Comparative analysis asks writers to make an argument about the relationship between two or more texts. Beyond that, there's a lot of variation, but three overarching kinds of comparative analysis stand out:

• Coordinate (A ↔ B): In this kind of analysis, two (or more) texts are being read against each other in terms of a shared element, e.g., a memoir and a novel, both by Jesmyn Ward; two sets of data for the same experiment; a few op-ed responses to the same event; two YA books written in Chicago in the 2000s; a film adaption of a play; etc. 
• Subordinate (A  → B) or (B → A ): Using a theoretical text (as a "lens") to explain a case study or work of art (e.g., how Anthony Jack's The Privileged Poor can help explain divergent experiences among students at elite four-year private colleges who are coming from similar socio-economic backgrounds) or using a work of art or case study (i.e., as a "test" of) a theory's usefulness or limitations (e.g., using coverage of recent incidents of gun violence or legislation un the U.S. to confirm or question the currency of Carol Anderson's The Second ).
• Hybrid [A  → (B ↔ C)] or [(B ↔ C) → A] , i.e., using coordinate and subordinate analysis together. For example, using Jack to compare or contrast the experiences of students at elite four-year institutions with students at state universities and/or community colleges; or looking at gun culture in other countries and/or other timeframes to contextualize or generalize Anderson's main points about the role of the Second Amendment in U.S. history.

"In the wild," these three kinds of comparative analysis represent increasingly complex—and scholarly—modes of comparison. Students can of course compare two poems in terms of imagery or two data sets in terms of methods, but in each case the analysis will eventually be richer if the students have had a chance to encounter other people's ideas about how imagery or methods work. At that point, we're getting into a hybrid kind of reading (or even into research essays), especially if we start introducing different approaches to imagery or methods that are themselves being compared along with a couple (or few) poems or data sets.

Why It's Useful

In the context of a particular course, each kind of comparative analysis has its place and can be a useful step up from single-source analysis. Intellectually, comparative analysis helps overcome the "n of 1" problem that can face single-source analysis. That is, a writer drawing broad conclusions about the influence of the Iranian New Wave based on one film is relying entirely—and almost certainly too much—on that film to support those findings. In the context of even just one more film, though, the analysis is suddenly more likely to arrive at one of the best features of any comparative approach: both films will be more richly experienced than they would have been in isolation, and the themes or questions in terms of which they're being explored (here the general question of the influence of the Iranian New Wave) will arrive at conclusions that are less at-risk of oversimplification.

For scholars working in comparative fields or through comparative approaches, these features of comparative analysis animate their work. To borrow from a stock example in Western epistemology, our concept of "green" isn't based on a single encounter with something we intuit or are told is "green." Not at all. Our concept of "green" is derived from a complex set of experiences of what others say is green or what's labeled green or what seems to be something that's neither blue nor yellow but kind of both, etc. Comparative analysis essays offer us the chance to engage with that process—even if only enough to help us see where a more in-depth exploration with a higher and/or more diverse "n" might lead—and in that sense, from the standpoint of the subject matter students are exploring through writing as well the complexity of the genre of writing they're using to explore it—comparative analysis forms a bridge of sorts between single-source analysis and research essays.

Typical learning objectives for single-sources essays: formulate analytical questions and an arguable thesis, establish stakes of an argument, summarize sources accurately, choose evidence effectively, analyze evidence effectively, define key terms, organize argument logically, acknowledge and respond to counterargument, cite sources properly, and present ideas in clear prose.

Common types of comparative analysis essays and related types: two works in the same genre, two works from the same period (but in different places or in different cultures), a work adapted into a different genre or medium, two theories treating the same topic; a theory and a case study or other object, etc.

How to Teach It: Framing + Practice

Framing multi-source writing assignments (comparative analysis, research essays, multi-modal projects) is likely to overlap a great deal with "Why It's Useful" (see above), because the range of reasons why we might use these kinds of writing in academic or non-academic settings is itself the reason why they so often appear later in courses. In many courses, they're the best vehicles for exploring the complex questions that arise once we've been introduced to the course's main themes, core content, leading protagonists, and central debates.

For comparative analysis in particular, it's helpful to frame assignment's process and how it will help students successfully navigate the challenges and pitfalls presented by the genre. Ideally, this will mean students have time to identify what each text seems to be doing, take note of apparent points of connection between different texts, and start to imagine how those points of connection (or the absence thereof)

• complicates or upends their own expectations or assumptions about the texts
• complicates or refutes the expectations or assumptions about the texts presented by a scholar
• confirms and/or nuances expectations and assumptions they themselves hold or scholars have presented
• presents entirely unforeseen ways of understanding the texts

—and all with implications for the texts themselves or for the axes along which the comparative analysis took place. If students know that this is where their ideas will be heading, they'll be ready to develop those ideas and engage with the challenges that comparative analysis presents in terms of structure (See "Tips" and "Common Pitfalls" below for more on these elements of framing).

Like single-source analyses, comparative essays have several moving parts, and giving students practice here means adapting the sample sequence laid out at the " Formative Writing Assignments " page. Three areas that have already been mentioned above are worth noting:

• Gathering evidence : Depending on what your assignment is asking students to compare (or in terms of what), students will benefit greatly from structured opportunities to create inventories or data sets of the motifs, examples, trajectories, etc., shared (or not shared) by the texts they'll be comparing. See the sample exercises below for a basic example of what this might look like.
• Why it Matters: Moving beyond "x is like y but also different" or even "x is more like y than we might think at first" is what moves an essay from being "compare/contrast" to being a comparative analysis . It's also a move that can be hard to make and that will often evolve over the course of an assignment. A great way to get feedback from students about where they're at on this front? Ask them to start considering early on why their argument "matters" to different kinds of imagined audiences (while they're just gathering evidence) and again as they develop their thesis and again as they're drafting their essays. ( Cover letters , for example, are a great place to ask writers to imagine how a reader might be affected by reading an their argument.)
• Structure: Having two texts on stage at the same time can suddenly feel a lot more complicated for any writer who's used to having just one at a time. Giving students a sense of what the most common patterns (AAA / BBB, ABABAB, etc.) are likely to be can help them imagine, even if provisionally, how their argument might unfold over a series of pages. See "Tips" and "Common Pitfalls" below for more information on this front.

Sample Exercises and Links to Other Resources

• Common Pitfalls
• Advice on Timing
• Try to keep students from thinking of a proposed thesis as a commitment. Instead, help them see it as more of a hypothesis that has emerged out of readings and discussion and analytical questions and that they'll now test through an experiment, namely, writing their essay. When students see writing as part of the process of inquiry—rather than just the result—and when that process is committed to acknowledging and adapting itself to evidence, it makes writing assignments more scientific, more ethical, and more authentic. 
• Have students create an inventory of touch points between the two texts early in the process.
• Ask students to make the case—early on and at points throughout the process—for the significance of the claim they're making about the relationship between the texts they're comparing.
• For coordinate kinds of comparative analysis, a common pitfall is tied to thesis and evidence. Basically, it's a thesis that tells the reader that there are "similarities and differences" between two texts, without telling the reader why it matters that these two texts have or don't have these particular features in common. This kind of thesis is stuck at the level of description or positivism, and it's not uncommon when a writer is grappling with the complexity that can in fact accompany the "taking inventory" stage of comparative analysis. The solution is to make the "taking inventory" stage part of the process of the assignment. When this stage comes before students have formulated a thesis, that formulation is then able to emerge out of a comparative data set, rather than the data set emerging in terms of their thesis (which can lead to confirmation bias, or frequency illusion, or—just for the sake of streamlining the process of gathering evidence—cherry picking). 
• For subordinate kinds of comparative analysis , a common pitfall is tied to how much weight is given to each source. Having students apply a theory (in a "lens" essay) or weigh the pros and cons of a theory against case studies (in a "test a theory") essay can be a great way to help them explore the assumptions, implications, and real-world usefulness of theoretical approaches. The pitfall of these approaches is that they can quickly lead to the same biases we saw here above. Making sure that students know they should engage with counterevidence and counterargument, and that "lens" / "test a theory" approaches often balance each other out in any real-world application of theory is a good way to get out in front of this pitfall.
• For any kind of comparative analysis, a common pitfall is structure. Every comparative analysis asks writers to move back and forth between texts, and that can pose a number of challenges, including: what pattern the back and forth should follow and how to use transitions and other signposting to make sure readers can follow the overarching argument as the back and forth is taking place. Here's some advice from an experienced writing instructor to students about how to think about these considerations:

a quick note on STRUCTURE

     Most of us have encountered the question of whether to adopt what we might term the “A→A→A→B→B→B” structure or the “A→B→A→B→A→B” structure.  Do we make all of our points about text A before moving on to text B?  Or do we go back and forth between A and B as the essay proceeds?  As always, the answers to our questions about structure depend on our goals in the essay as a whole.  In a “similarities in spite of differences” essay, for instance, readers will need to encounter the differences between A and B before we offer them the similarities (A d →B d →A s →B s ).  If, rather than subordinating differences to similarities you are subordinating text A to text B (using A as a point of comparison that reveals B’s originality, say), you may be well served by the “A→A→A→B→B→B” structure.  

     Ultimately, you need to ask yourself how many “A→B” moves you have in you.  Is each one identical?  If so, you may wish to make the transition from A to B only once (“A→A→A→B→B→B”), because if each “A→B” move is identical, the “A→B→A→B→A→B” structure will appear to involve nothing more than directionless oscillation and repetition.  If each is increasingly complex, however—if each AB pair yields a new and progressively more complex idea about your subject—you may be well served by the “A→B→A→B→A→B” structure, because in this case it will be visible to readers as a progressively developing argument.

As we discussed in "Advice on Timing" at the page on single-source analysis, that timeline itself roughly follows the "Sample Sequence of Formative Assignments for a 'Typical' Essay" outlined under " Formative Writing Assignments, " and it spans about 5–6 steps or 2–4 weeks. 

Comparative analysis assignments have a lot of the same DNA as single-source essays, but they potentially bring more reading into play and ask students to engage in more complicated acts of analysis and synthesis during the drafting stages. With that in mind, closer to 4 weeks is probably a good baseline for many single-source analysis assignments. For sections that meet once per week, the timeline will either probably need to expand—ideally—a little past the 4-week side of things, or some of the steps will need to be combined or done asynchronously.

What It Can Build Up To

Comparative analyses can build up to other kinds of writing in a number of ways. For example:

• They can build toward other kinds of comparative analysis, e.g., student can be asked to choose an additional source to complicate their conclusions from a previous analysis, or they can be asked to revisit an analysis using a different axis of comparison, such as race instead of class. (These approaches are akin to moving from a coordinate or subordinate analysis to more of a hybrid approach.)
• They can scaffold up to research essays, which in many instances are an extension of a "hybrid comparative analysis."
• Like single-source analysis, in a course where students will take a "deep dive" into a source or topic for their capstone, they can allow students to "try on" a theoretical approach or genre or time period to see if it's indeed something they want to research more fully.
• DIY Guides for Analytical Writing Assignments

• Types of Assignments
• Unpacking the Elements of Writing Prompts
• Formative Writing Assignments
• Single-Source Analysis
• Research Essays
• Multi-Modal or Creative Projects
• Giving Feedback to Students

Assignment Decoder

Sign up for our newsletter

• Evidence & Evaluation
• Evaluation guidance
• Impact evaluation with small cohorts (evaluation guidance)
• Get started with small n evaluation

Comparative Case Study

Share content.

A comparative case study (CCS) is defined as ‘the systematic comparison of two or more data points (“cases”) obtained through use of the case study method’ (Kaarbo and Beasley 1999, p. 372). A case may be a participant, an intervention site, a programme or a policy. Case studies have a long history in the social sciences, yet for a long time, they were treated with scepticism (Harrison et al. 2017). The advent of grounded theory in the 1960s led to a revival in the use of case-based approaches. From the early 1980s, the increase in case study research in the field of political sciences led to the integration of formal, statistical and narrative methods, as well as the use of empirical case selection and causal inference (George and Bennett 2005), which contributed to its methodological advancement. Now, as Harrison and colleagues (2017) note, CCS:

“Has grown in sophistication and is viewed as a valid form of inquiry to explore a broad scope of complex issues, particularly when human behavior and social interactions are central to understanding topics of interest.”

It is claimed that CCS can be applied to detect causal attribution and contribution when the use of a comparison or control group is not feasible (or not preferred). Comparing cases enables evaluators to tackle causal inference through assessing regularity (patterns) and/or by excluding other plausible explanations. In practical terms, CCS involves proposing, analysing and synthesising patterns (similarities and differences) across cases that share common objectives.

What is involved?

Goodrick (2014) outlines the steps to be taken in undertaking CCS.

Key evaluation questions and the purpose of the evaluation: The evaluator should explicitly articulate the adequacy and purpose of using CCS (guided by the evaluation questions) and define the primary interests. Formulating key evaluation questions allows the selection of appropriate cases to be used in the analysis.

Propositions based on the Theory of Change: Theories and hypotheses that are to be explored should be derived from the Theory of Change (or, alternatively, from previous research around the initiative, existing policy or programme documentation).

Case selection: Advocates for CCS approaches claim an important distinction between case-oriented small n studies and (most typically large n) statistical/variable-focused approaches in terms of the process of selecting cases: in case-based methods, selection is iterative and cannot rely on convenience and accessibility. ‘Initial’ cases should be identified in advance, but case selection may continue as evidence is gathered. Various case-selection criteria can be identified depending on the analytic purpose (Vogt et al., 2011). These may include:

• Very similar cases
• Very different cases
• Typical or representative cases
• Extreme or unusual cases
• Deviant or unexpected cases
• Influential or emblematic cases

Identify how evidence will be collected, analysed and synthesised: CCS often applies mixed methods.

Test alternative explanations for outcomes: Following the identification of patterns and relationships, the evaluator may wish to test the established propositions in a follow-up exploratory phase. Approaches applied here may involve triangulation, selecting contradicting cases or using an analytical approach such as Qualitative Comparative Analysis (QCA). Download a Comparative Case Study here Download a longer briefing on Comparative Case Studies here

Useful resources

A webinar shared by Better Evaluation with an overview of using CCS for evaluation.

A short overview describing how to apply CCS for evaluation:

Goodrick, D. (2014). Comparative Case Studies, Methodological Briefs: Impact Evaluation 9 , UNICEF Office of Research, Florence.

An extensively used book that provides a comprehensive critical examination of case-based methods:

Byrne, D. and Ragin, C. C. (2009). The Sage handbook of case-based methods . Sage Publications.

Home » Blog » A Step-by-Step Guide to Writing a Comparative Analysis

A Step-by-Step Guide to Writing a Comparative Analysis

Table of Contents

How to Write a Comparative Analysis with Examples

Writing a comparative analysis in a research paper is not as difficult as many people might tend to think. With some tips, it is possible to write an outstanding comparative review. There are steps that must be utilized to attain this result. They are as detailed in this article.

Within the literary, academic, and journalistic world, analysis allows exposing ideas and arguments in front of a context, making it an important material for discussion within professional work.

Within this genre, we can find a comparative analysis. For some authors, the comparative essay is defined as the text where two opposing positions are proposed or where two theses are verified. The author intends to make the reader reflect on a specific topic through this comparison. It consists of giving a written opinion about two positions, which are compared between them to conclude. Do you know how to write a comparative essay? In this article, we will explain how to do it step by step.

So, let’s see the guidelines you must follow to achieve a good comparative analysis.

How to Write a Good Comparative Analysis

The structure.

The approach is generally developed in the first paragraph or at the beginning of the work. Its objective is to propose the author’s position regarding a specific subject. Generally, this approach specifies the objective to be achieved. You must be clear about what topic you will deal with, what you want to explain, and what perspectives will be used in your comparative analysis, and you must also define who you write for.

As it is a comparative text, it begins with a general observation that can serve as a context for both approaches, then begins by establishing the arguments in each of the two cases. Do not forget to compare both objects of study according to each argument or idea to develop.

Let it be the reader himself who finds or defines his position in this essay and chooses one of the two alternatives.

In this entry, there are two possibilities of approach: one deductive and the other inductive. The deductive method raises the issue, and you use your analysis of the variables to guide the reader to draw their conclusions or fix a position on the issue. While the inductive method starts with an argument, developing each variable until the topic’s approach or problem is reached. The two ways of approaching the subject are viable. Choose the one that is easiest for you to work with.

At the end of this section, your audience should:

• First, clearly understand what topics you will cover in your essay, what you want to explain, and under what positions or perspectives you will do it. It begins with a general observation that establishes the similarity between the two subjects and then moves the essay’s focus to the concrete.
• The reader should understand which points will be examined and which will not be examined in the comparison. At the end of the introduction, state your preference, or describe the two subjects’ meaning.
• Your readers should be able to describe the ideas you will treat. Make a detailed exposition of its characteristics, history, consequences, and development that you consider appropriate. Your comparative analysis should expose the characteristics of the second position on which you want to speak as much as in the first one.

Development of Body

Generally, in the body of the essay, the author presents all the arguments that support his thesis, which gives him a reflective and justifying body of the author’s initial statement. Depending on the length of the work, which can range from two to 15 pages, each paragraph or before a title corresponds to an argument’s development.

After speaking on the subject, the author must close the essay, conclude, show the findings of his work, and/or show the conclusions he reached. You must write a final closing paragraph as a conclusion, exposing a confrontation between the two positions. Try to create a fight between them so that the reader gets involved. The conclusion should give a brief and general summary of the most important similarities and differences. It should end with a personal statement, an opinion, and the “what then?” – what is important about the two things being compared.

Readers should be left feeling that this essay’s different threads have been put together coherently, that they have learned something – and they must be sure that this is the end – that they do not look around for missing pages. And finally, your assessment must explain your solidarity position and why you prefer it to the other.

Examples of How to Write a Comparative Analysis

Comparative analysis example 1:.

Paragraph 1: Messi’s preferred position / Ronaldo’s preferred position.

Paragraph 2: Messi’s play style / Ronaldo’s play style.

Paragraph 3: Messi aerial game / Ronaldo aerial game.

Comparative Analysis Example 2:

Paragraph 1: Messi teamwork.

Paragraph 2: Ronaldo’s teamwork.

Paragraph 3: Messi stopped the ball.

Paragraph 4: Ronaldo’s stopped the ball.

Paragraph 5: Messi’s achievements.

Paragraph 6: Ronaldo’s achievements.

Few Important Rules for Comparative analysis

Even if the exercise sounds simple, a few rules should be followed to help your audience as best as possible make the best decision.

1. Clearly state your position

The first question is, “Why are you doing a comparison analysis”? To highlight your view or ideas over another, or simply to compare two (or more) solutions that do not belong to you? You must clearly state your position to your reader, and so does your credibility.

Be honest and state, for example:

• The idea you are trying to espouse
• The framework you are using
• The reason why you are doing this comparison is the objective

In addition to the above, you must be consistent with the exposition of your ideas.

2. Stay objective

Even if you include your personal ideology in your comparison, stay objective. Your readers will not appreciate it when you point out all the disadvantages of one idea while you display the advantages of the other. Your comparison will turn into advertising. You have to raise weak points and strong points on both sides.

These analyses are always subjective, so you must clarify which position convinces you the most.

3. Think about audience’s expectations

The research paper is intended for your readers, meaning you must consider their expectations when writing your review. Put aside your desire to sell your desired idea and take your readers’ perspective:

• What information are they interested in?
• What are their criteria?
• What do they want to know?
• What do they want from the product or service?

Again, it is about being objective in all your statements.

4. Organize information

It is important to structure your comments for your readers to want to read your comparative analysis. The idea is to make it easy for your readers to navigate your paper and get them to find the information that interests them quickly.

5. End with a conclusion

You’ve tried to be as objective as possible throughout your comparison, and now is the time to let go, as we have mentioned many times in this post. In your conclusion, you can go directly to your readers and give your opinion. With a few tips, you can also encourage them to go towards one or the other idea.

Note: If time is not an issue, the best way to review the essay is to leave it for one day. Go for a walk, eat something, have fun, and forget. Then it’s time to return to the text, find and fix problems. This must be done separately; first, find all the problems you can without correcting them. Although doing it simultaneously is tempting, it is smarter to do it separately. It is effective and fast.

Tips on Comparative Analysis

Be concise or accurate in your analysis and dissertation of the topic.

Sometimes the authors believe that the more elaborate the language and the more extensive the writing, the better the writers or essayists. On the contrary, a good essay refers to an exact topic analysis, where the reader can dynamically advance the work and understand the author’s position.

Use only the arguments necessary to explain the topic, do not talk too much. You risk being redundant or repetitive, making the text-heavy when reading and understanding it.

Write in Short Sentences

Just as we recommend that you do not redound in your texts, we also encourage you to write with short sentences. They give dynamism to the text. Communication is direct. The reader advances in the text and understands much more.

Include Reflections in Your Text

Supporting your approach with reflections or quotes from authors makes your essay more important. Above all, use those arguments that justify or strengthen your position regarding one thesis or the other.

Text Revision

Since comparative analysis can tend to be a subjective work, you must let it “sit” for a day or a few hours and read it again. This exercise will allow you to make corrections. Modify those aspects that are not clear enough for you. And you can improve it in a few words. Once you do this exercise, you can submit it just like this.

If you like this article, see others like it:

Balancing Work and Life: Achieving Success in Nigeria’s Competitive Job Market

2024 complete nigeria current affairs pdf free download, 2024 nigeria current affairs quiz questions & answers, learn how to trade forex: a beginner’s guide, 5 corporate team building activities ideas for introverted employees, related topics, how to search for journals for a research project, signs it’s time to re-evaluate your career goals, 5 common career change fears and what to do, how to plan an affordable vacation as a student, 5 things to consider when choosing a new career path.

What is comparative analysis? A complete guide

Last updated

18 April 2023

Reviewed by

Jean Kaluza

Short on time? Get an AI generated summary of this article instead

Comparative analysis is a valuable tool for acquiring deep insights into your organization’s processes, products, and services so you can continuously improve them. 

Similarly, if you want to streamline, price appropriately, and ultimately be a market leader, you’ll likely need to draw on comparative analyses quite often.

When faced with multiple options or solutions to a given problem, a thorough comparative analysis can help you compare and contrast your options and make a clear, informed decision.

If you want to get up to speed on conducting a comparative analysis or need a refresher, here’s your guide.

Make comparative analysis less tedious

Dovetail streamlines comparative analysis to help you uncover and share actionable insights

• What exactly is comparative analysis?

A comparative analysis is a side-by-side comparison that systematically compares two or more things to pinpoint their similarities and differences. The focus of the investigation might be conceptual—a particular problem, idea, or theory—or perhaps something more tangible, like two different data sets.

For instance, you could use comparative analysis to investigate how your product features measure up to the competition.

After a successful comparative analysis, you should be able to identify strengths and weaknesses and clearly understand which product is more effective.

You could also use comparative analysis to examine different methods of producing that product and determine which way is most efficient and profitable.

The potential applications for using comparative analysis in everyday business are almost unlimited. That said, a comparative analysis is most commonly used to examine

Emerging trends and opportunities (new technologies, marketing)

Competitor strategies

Financial health

Effects of trends on a target audience

Free AI content analysis generator

Make sense of your research by automatically summarizing key takeaways through our free content analysis tool.

• Why is comparative analysis so important? 

Comparative analysis can help narrow your focus so your business pursues the most meaningful opportunities rather than attempting dozens of improvements simultaneously.

A comparative approach also helps frame up data to illuminate interrelationships. For example, comparative research might reveal nuanced relationships or critical contexts behind specific processes or dependencies that wouldn’t be well-understood without the research.

For instance, if your business compares the cost of producing several existing products relative to which ones have historically sold well, that should provide helpful information once you’re ready to look at developing new products or features.

• Comparative vs. competitive analysis—what’s the difference?

Comparative analysis is generally divided into three subtypes, using quantitative or qualitative data and then extending the findings to a larger group. These include

Pattern analysis —identifying patterns or recurrences of trends and behavior across large data sets.

Data filtering —analyzing large data sets to extract an underlying subset of information. It may involve rearranging, excluding, and apportioning comparative data to fit different criteria. 

Decision tree —flowcharting to visually map and assess potential outcomes, costs, and consequences.

In contrast, competitive analysis is a type of comparative analysis in which you deeply research one or more of your industry competitors. In this case, you’re using qualitative research to explore what the competition is up to across one or more dimensions.

For example

Service delivery —metrics like the Net Promoter Scores indicate customer satisfaction levels.

Market position — the share of the market that the competition has captured.

Brand reputation —how well-known or recognized your competitors are within their target market.

• Tips for optimizing your comparative analysis

Conduct original research

Thorough, independent research is a significant asset when doing comparative analysis. It provides evidence to support your findings and may present a perspective or angle not considered previously. 

Make analysis routine

To get the maximum benefit from comparative research, make it a regular practice, and establish a cadence you can realistically stick to. Some business areas you could plan to analyze regularly include:

Profitability

Competition

Experiment with controlled and uncontrolled variables

In addition to simply comparing and contrasting, explore how different variables might affect your outcomes.

For example, a controllable variable would be offering a seasonal feature like a shopping bot to assist in holiday shopping or raising or lowering the selling price of a product.

Uncontrollable variables include weather, changing regulations, the current political climate, or global pandemics.

Put equal effort into each point of comparison

Most people enter into comparative research with a particular idea or hypothesis already in mind to validate. For instance, you might try to prove the worthwhileness of launching a new service. So, you may be disappointed if your analysis results don’t support your plan.

However, in any comparative analysis, try to maintain an unbiased approach by spending equal time debating the merits and drawbacks of any decision. Ultimately, this will be a practical, more long-term sustainable approach for your business than focusing only on the evidence that favors pursuing your argument or strategy.

Writing a comparative analysis in five steps

To put together a coherent, insightful analysis that goes beyond a list of pros and cons or similarities and differences, try organizing the information into these five components:

1. Frame of reference

Here is where you provide context. First, what driving idea or problem is your research anchored in? Then, for added substance, cite existing research or insights from a subject matter expert, such as a thought leader in marketing, startup growth, or investment

2. Grounds for comparison Why have you chosen to examine the two things you’re analyzing instead of focusing on two entirely different things? What are you hoping to accomplish?

3. Thesis What argument or choice are you advocating for? What will be the before and after effects of going with either decision? What do you anticipate happening with and without this approach?

For example, “If we release an AI feature for our shopping cart, we will have an edge over the rest of the market before the holiday season.” The finished comparative analysis will weigh all the pros and cons of choosing to build the new expensive AI feature including variables like how “intelligent” it will be, what it “pushes” customers to use, how much it takes off the plates of customer service etc.

Ultimately, you will gauge whether building an AI feature is the right plan for your e-commerce shop.

4. Organize the scheme Typically, there are two ways to organize a comparative analysis report. First, you can discuss everything about comparison point “A” and then go into everything about aspect “B.” Or, you alternate back and forth between points “A” and “B,” sometimes referred to as point-by-point analysis.

Using the AI feature as an example again, you could cover all the pros and cons of building the AI feature, then discuss the benefits and drawbacks of building and maintaining the feature. Or you could compare and contrast each aspect of the AI feature, one at a time. For example, a side-by-side comparison of the AI feature to shopping without it, then proceeding to another point of differentiation.

5. Connect the dots Tie it all together in a way that either confirms or disproves your hypothesis.

For instance, “Building the AI bot would allow our customer service team to save 12% on returns in Q3 while offering optimizations and savings in future strategies. However, it would also increase the product development budget by 43% in both Q1 and Q2. Our budget for product development won’t increase again until series 3 of funding is reached, so despite its potential, we will hold off building the bot until funding is secured and more opportunities and benefits can be proved effective.”

Should you be using a customer insights hub?

Do you want to discover previous research faster?

Do you share your research findings with others?

Do you analyze research data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 18 April 2023

Last updated: 27 February 2023

Last updated: 6 February 2023

Last updated: 15 January 2024

Last updated: 6 October 2023

Last updated: 5 February 2023

Last updated: 16 April 2023

Last updated: 7 March 2023

Last updated: 9 March 2023

Last updated: 12 December 2023

Last updated: 11 March 2024

Last updated: 13 May 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next.

Users report unexpectedly high data usage, especially during streaming sessions.

Users find it hard to navigate from the home page to relevant playlists in the app.

It would be great to have a sleep timer feature, especially for bedtime listening.

I need better filters to find the songs or artists I’m looking for.

Log in or sign up

Get started for free

How do I write a comparative analysis?

A comparative analysis is an essay in which two things are compared and contrasted. You may have done a "compare and contrast" paper in your English class, and a comparative analysis is the same general idea, but as a graduate student you are expected to produce a higher level of analysis in your writing. You can follow these guidelines to get started. 

• Conduct your research. Need help? Ask a Librarian!
• Brainstorm a list of similarities and differences. The Double Bubble  document linked below can be helpful for this step.
• Write your thesis. This will be based on what you have discovered regarding the weight of similarities and differences between the things you are comparing. 
• Alternating (point-by-point) method: Find similar points between each subject and alternate writing about each of them.
• Block (subject-by-subject) method: Discuss all of the first subject and then all of the second.
• This page from the University of Toronto gives some great examples of when each of these is most effective.
• Don't forget to cite your sources! 

Visvis, V., & Plotnik, J. (n.d.). The comparative essay . University of Toronto. https://advice.writing.utoronto.ca/types-of-writing/comparative-essay/

Walk, K. (1998). How to write a comparative analysis . Harvard University. https://writingcenter.fas.harvard.edu/pages/how-write-comparative-analysis

Links & Files

• Double_Bubble_Map.docx
• Health Sciences
• Reading and Writing
• Graduate Writing
• Last Updated Sep 06, 2023
• Views 129907
• Answered By Kerry Louvier

FAQ Actions

• Share on Facebook

Comments (0)

Hello! We're here to help! Please log in to ask your question.

Need an answer now? Search our FAQs !

How can I find my course textbook?

You can expect a prompt response, Monday through Friday, 8:00 AM-4:00 PM Central Time (by the next business day on weekends and holidays).

Questions may be answered by a Librarian, Learning Services Coordinator, Instructor, or Tutor. 

Organizing Your Social Sciences Research Paper: Writing a Case Study

• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Reading Research Effectively
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• What Is Scholarly vs. Popular?
• Qualitative Methods
• Quantitative Methods
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Annotated Bibliography
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• Types of Structured Group Activities
• Group Project Survival Skills
• Multiple Book Review Essay
• Reviewing Collected Essays
• Writing a Case Study
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Research Proposal
• Bibliography

The term case study refers to both a method of analysis and a specific research design for examining a problem, both of which are used in most circumstances to generalize across populations. This tab focuses on the latter--how to design and organize a research paper in the social sciences that analyzes a specific case.

A case study research paper examines a person, place, event, phenomenon, or other type of subject of analysis in order to extrapolate  key themes and results that help predict future trends, illuminate previously hidden issues that can be applied to practice, and/or provide a means for understanding an important research problem with greater clarity. A case study paper usually examines a single subject of analysis, but case study papers can also be designed as a comparative investigation that shows relationships between two or among more than two subjects. The methods used to study a case can rest within a quantitative, qualitative, or mixed-method investigative paradigm.

Case Studies . Writing@CSU. Colorado State University; Mills, Albert J. , Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010 ; “What is a Case Study?” In Swanborn, Peter G. Case Study Research: What, Why and How? London: SAGE, 2010.

How to Approach Writing a Case Study Research Paper

General information about how to choose a topic to investigate can be found under the " Choosing a Research Problem " tab in this writing guide. Review this page because it may help you identify a subject of analysis that can be investigated using a single case study design.

However, identifying a case to investigate involves more than choosing the research problem . A case study encompasses a problem contextualized around the application of in-depth analysis, interpretation, and discussion, often resulting in specific recommendations for action or for improving existing conditions. As Seawright and Gerring note, practical considerations such as time and access to information can influence case selection, but these issues should not be the sole factors used in describing the methodological justification for identifying a particular case to study. Given this, selecting a case includes considering the following:

• Does the case represent an unusual or atypical example of a research problem that requires more in-depth analysis? Cases often represent a topic that rests on the fringes of prior investigations because the case may provide new ways of understanding the research problem. For example, if the research problem is to identify strategies to improve policies that support girl's access to secondary education in predominantly Muslim nations, you could consider using Azerbaijan as a case study rather than selecting a more obvious nation in the Middle East. Doing so may reveal important new insights into recommending how governments in other predominantly Muslim nations can formulate policies that support improved access to education for girls.
• Does the case provide important insight or illuminate a previously hidden problem? In-depth analysis of a case can be based on the hypothesis that the case study will reveal trends or issues that have not been exposed in prior research or will reveal new and important implications for practice. For example, anecdotal evidence may suggest drug use among homeless veterans is related to their patterns of travel throughout the day. Assuming prior studies have not looked at individual travel choices as a way to study access to illicit drug use, a case study that observes a homeless veteran could reveal how issues of personal mobility choices facilitate regular access to illicit drugs. Note that it is important to conduct a thorough literature review to ensure that your assumption about the need to reveal new insights or previously hidden problems is valid and evidence-based.
• Does the case challenge and offer a counter-point to prevailing assumptions? Over time, research on any given topic can fall into a trap of developing assumptions based on outdated studies that are still applied to new or changing conditions or the idea that something should simply be accepted as "common sense," even though the issue has not been thoroughly tested in practice. A case may offer you an opportunity to gather evidence that challenges prevailing assumptions about a research problem and provide a new set of recommendations applied to practice that have not been tested previously. For example, perhaps there has been a long practice among scholars to apply a particular theory in explaining the relationship between two subjects of analysis. Your case could challenge this assumption by applying an innovative theoretical framework [perhaps borrowed from another discipline] to the study a case in order to explore whether this approach offers new ways of understanding the research problem. Taking a contrarian stance is one of the most important ways that new knowledge and understanding develops from existing literature.
• Does the case provide an opportunity to pursue action leading to the resolution of a problem? Another way to think about choosing a case to study is to consider how the results from investigating a particular case may result in findings that reveal ways in which to resolve an existing or emerging problem. For example, studying the case of an unforeseen incident, such as a fatal accident at a railroad crossing, can reveal hidden issues that could be applied to preventative measures that contribute to reducing the chance of accidents in the future. In this example, a case study investigating the accident could lead to a better understanding of where to strategically locate additional signals at other railroad crossings in order to better warn drivers of an approaching train, particularly when visibility is hindered by heavy rain, fog, or at night.
• Does the case offer a new direction in future research? A case study can be used as a tool for exploratory research that points to a need for further examination of the research problem. A case can be used when there are few studies that help predict an outcome or that establish a clear understanding about how best to proceed in addressing a problem. For example, after conducting a thorough literature review [very important!], you discover that little research exists showing the ways in which women contribute to promoting water conservation in rural communities of Uganda. A case study of how women contribute to saving water in a particular village can lay the foundation for understanding the need for more thorough research that documents how women in their roles as cooks and family caregivers think about water as a valuable resource within their community throughout rural regions of east Africa. The case could also point to the need for scholars to apply feminist theories of work and family to the issue of water conservation.

Eisenhardt, Kathleen M. “Building Theories from Case Study Research.” Academy of Management Review 14 (October 1989): 532-550; Emmel, Nick. Sampling and Choosing Cases in Qualitative Research: A Realist Approach . Thousand Oaks, CA: SAGE Publications, 2013; Gerring, John. “What Is a Case Study and What Is It Good for?” American Political Science Review 98 (May 2004): 341-354; Mills, Albert J. , Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010; Seawright, Jason and John Gerring. "Case Selection Techniques in Case Study Research." Political Research Quarterly 61 (June 2008): 294-308.

Structure and Writing Style

The purpose of a paper in the social sciences designed around a case study is to thoroughly investigate a subject of analysis in order to reveal a new understanding about the research problem and, in so doing, contributing new knowledge to what is already known from previous studies. In applied social sciences disciplines [e.g., education, social work, public administration, etc.], case studies may also be used to reveal best practices, highlight key programs, or investigate interesting aspects of professional work. In general, the structure of a case study research paper is not all that different from a standard college-level research paper. However, there are subtle differences you should be aware of. Here are the key elements to organizing and writing a case study research paper.

I.  Introduction

As with any research paper, your introduction should serve as a roadmap for your readers to ascertain the scope and purpose of your study . The introduction to a case study research paper, however, should not only describe the research problem and its significance, but you should also succinctly describe why the case is being used and how it relates to addressing the problem. The two elements should be linked. With this in mind, a good introduction answers these four questions:

• What was I studying? Describe the research problem and describe the subject of analysis you have chosen to address the problem. Explain how they are linked and what elements of the case will help to expand knowledge and understanding about the problem.
• Why was this topic important to investigate? Describe the significance of the research problem and state why a case study design and the subject of analysis that the paper is designed around is appropriate in addressing the problem.
• What did we know about this topic before I did this study? Provide background that helps lead the reader into the more in-depth literature review to follow. If applicable, summarize prior case study research applied to the research problem and why it fails to adequately address the research problem. Describe why your case will be useful. If no prior case studies have been used to address the research problem, explain why you have selected this subject of analysis.
• How will this study advance new knowledge or new ways of understanding? Explain why your case study will be suitable in helping to expand knowledge and understanding about the research problem.

Each of these questions should be addressed in no more than a few paragraphs. Exceptions to this can be when you are addressing a complex research problem or subject of analysis that requires more in-depth background information.

II.  Literature Review

The literature review for a case study research paper is generally structured the same as it is for any college-level research paper. The difference, however, is that the literature review is focused on providing background information and  enabling historical interpretation of the subject of analysis in relation to the research problem the case is intended to address . This includes synthesizing studies that help to:

• Place relevant works in the context of their contribution to understanding the case study being investigated . This would include summarizing studies that have used a similar subject of analysis to investigate the research problem. If there is literature using the same or a very similar case to study, you need to explain why duplicating past research is important [e.g., conditions have changed; prior studies were conducted long ago, etc.].
• Describe the relationship each work has to the others under consideration that informs the reader why this case is applicable . Your literature review should include a description of any works that support using the case to study the research problem and the underlying research questions.
• Identify new ways to interpret prior research using the case study . If applicable, review any research that has examined the research problem using a different research design. Explain how your case study design may reveal new knowledge or a new perspective or that can redirect research in an important new direction.
• Resolve conflicts amongst seemingly contradictory previous studies . This refers to synthesizing any literature that points to unresolved issues of concern about the research problem and describing how the subject of analysis that forms the case study can help resolve these existing contradictions.
• Point the way in fulfilling a need for additional research . Your review should examine any literature that lays a foundation for understanding why your case study design and the subject of analysis around which you have designed your study may reveal a new way of approaching the research problem or offer a perspective that points to the need for additional research.
• Expose any gaps that exist in the literature that the case study could help to fill . Summarize any literature that not only shows how your subject of analysis contributes to understanding the research problem, but how your case contributes to a new way of understanding the problem that prior research has failed to do.
• Locate your own research within the context of existing literature [very important!] . Collectively, your literature review should always place your case study within the larger domain of prior research about the problem. The overarching purpose of reviewing pertinent literature in a case study paper is to demonstrate that you have thoroughly identified and synthesized prior studies in the context of explaining the relevance of the case in addressing the research problem.

III.  Method

In this section, you explain why you selected a particular subject of analysis to study and the strategy you used to identify and ultimately decide that your case was appropriate in addressing the research problem. The way you describe the methods used varies depending on the type of subject of analysis that frames your case study.

If your subject of analysis is an incident or event . In the social and behavioral sciences, the event or incident that represents the case to be studied is usually bounded by time and place, with a clear beginning and end and with an identifiable location or position relative to its surroundings. The subject of analysis can be a rare or critical event or it can focus on a typical or regular event. The purpose of studying a rare event is to illuminate new ways of thinking about the broader research problem or to test a hypothesis. Critical incident case studies must describe the method by which you identified the event and explain the process by which you determined the validity of this case to inform broader perspectives about the research problem or to reveal new findings. However, the event does not have to be a rare or uniquely significant to support new thinking about the research problem or to challenge an existing hypothesis. For example, Walo, Bull, and Breen conducted a case study to identify and evaluate the direct and indirect economic benefits and costs of a local sports event in the City of Lismore, New South Wales, Australia. The purpose of their study was to provide new insights from measuring the impact of a typical local sports event that prior studies could not measure well because they focused on large "mega-events." Whether the event is rare or not, the methods section should include an explanation of the following characteristics of the event: a) when did it take place; b) what were the underlying circumstances leading to the event; c) what were the consequences of the event.

If your subject of analysis is a person. Explain why you selected this particular individual to be studied and describe what experience he or she has had that provides an opportunity to advance new understandings about the research problem. Mention any background about this person which might help the reader understand the significance of his/her experiences that make them worthy of study. This includes describing the relationships this person has had with other people, institutions, and/or events that support using him or her as the subject for a case study research paper. It is particularly important to differentiate the person as the subject of analysis from others and to succinctly explain how the person relates to examining the research problem.

If your subject of analysis is a place. In general, a case study that investigates a place suggests a subject of analysis that is unique or special in some way and that this uniqueness can be used to build new understanding or knowledge about the research problem. A case study of a place must not only describe its various attributes relevant to the research problem [e.g., physical, social, cultural, economic, political, etc.], but you must state the method by which you determined that this place will illuminate new understandings about the research problem. It is also important to articulate why a particular place as the case for study is being used if similar places also exist [i.e., if you are studying patterns of homeless encampments of veterans in open spaces, why study Echo Park in Los Angeles rather than Griffith Park?]. If applicable, describe what type of human activity involving this place makes it a good choice to study [e.g., prior research reveals Echo Park has more homeless veterans].

If your subject of analysis is a phenomenon. A phenomenon refers to a fact, occurrence, or circumstance that can be studied or observed but with the cause or explanation to be in question. In this sense, a phenomenon that forms your subject of analysis can encompass anything that can be observed or presumed to exist but is not fully understood. In the social and behavioral sciences, the case usually focuses on human interaction within a complex physical, social, economic, cultural, or political system. For example, the phenomenon could be the observation that many vehicles used by ISIS fighters are small trucks with English language advertisements on them. The research problem could be that ISIS fighters are difficult to combat because they are highly mobile. The research questions could be how and by what means are these vehicles used by ISIS being supplied to the militants and how might supply lines to these vehicles be cut? How might knowing the suppliers of these trucks from overseas reveal larger networks of collaborators and financial support? A case study of a phenomenon most often encompasses an in-depth analysis of a cause and effect that is grounded in an interactive relationship between people and their environment in some way.

NOTE:   The choice of the case or set of cases to study cannot appear random. Evidence that supports the method by which you identified and chose your subject of analysis should be linked to the findings from the literature review. Be sure to cite any prior studies that helped you determine that the case you chose was appropriate for investigating the research problem.

IV.  Discussion

The main elements of your discussion section are generally the same as any research paper, but centered around interpreting and drawing conclusions about the key findings from your case study. Note that a general social sciences research paper may contain a separate section to report findings. However, in a paper designed around a case study, it is more common to combine a description of the findings with the discussion about their implications. The objectives of your discussion section should include the following:

Reiterate the Research Problem/State the Major Findings Briefly reiterate the research problem you are investigating and explain why the subject of analysis around which you designed the case study were used. You should then describe the findings revealed from your study of the case using direct, declarative, and succinct proclamation of the study results. Highlight any findings that were unexpected or especially profound.

Explain the Meaning of the Findings and Why They are Important Systematically explain the meaning of your case study findings and why you believe they are important. Begin this part of the section by repeating what you consider to be your most important or surprising finding first, then systematically review each finding. Be sure to thoroughly extrapolate what your analysis of the case can tell the reader about situations or conditions beyond the actual case that was studied while, at the same time, being careful not to misconstrue or conflate a finding that undermines the external validity of your conclusions.

Relate the Findings to Similar Studies No study in the social sciences is so novel or possesses such a restricted focus that it has absolutely no relation to previously published research. The discussion section should relate your case study results to those found in other studies, particularly if questions raised from prior studies served as the motivation for choosing your subject of analysis. This is important because comparing and contrasting the findings of other studies helps to support the overall importance of your results and it highlights how and in what ways your case study design and the subject of analysis differs from prior research about the topic.

Consider Alternative Explanations of the Findings It is important to remember that the purpose of social science research is to discover and not to prove. When writing the discussion section, you should carefully consider all possible explanations for the case study results, rather than just those that fit your hypothesis or prior assumptions and biases. Be alert to what the in-depth analysis of the case may reveal about the research problem, including offering a contrarian perspective to what scholars have stated in prior research.

Acknowledge the Study's Limitations You can state the study's limitations in the conclusion section of your paper but describing the limitations of your subject of analysis in the discussion section provides an opportunity to identify the limitations and explain why they are not significant. This part of the discussion section should also note any unanswered questions or issues your case study could not address. More detailed information about how to document any limitations to your research can be found here .

Suggest Areas for Further Research Although your case study may offer important insights about the research problem, there are likely additional questions related to the problem that remain unanswered or findings that unexpectedly revealed themselves as a result of your in-depth analysis of the case. Be sure that the recommendations for further research are linked to the research problem and that you explain why your recommendations are valid in other contexts and based on the original assumptions of your study.

V.  Conclusion

As with any research paper, you should summarize your conclusion in clear, simple language; emphasize how the findings from your case study differs from or supports prior research and why. Do not simply reiterate the discussion section. Provide a synthesis of key findings presented in the paper to show how these converge to address the research problem. If you haven't already done so in the discussion section, be sure to document the limitations of your case study and needs for further research.

The function of your paper's conclusion is to: 1)  restate the main argument supported by the findings from the analysis of your case; 2) clearly state the context, background, and necessity of pursuing the research problem using a case study design in relation to an issue, controversy, or a gap found from reviewing the literature; and, 3) provide a place for you to persuasively and succinctly restate the significance of your research problem, given that the reader has now been presented with in-depth information about the topic.

Consider the following points to help ensure your conclusion is appropriate:

• If the argument or purpose of your paper is complex, you may need to summarize these points for your reader.
• If prior to your conclusion, you have not yet explained the significance of your findings or if you are proceeding inductively, use the conclusion of your paper to describe your main points and explain their significance.
• Move from a detailed to a general level of consideration of the case study's findings that returns the topic to the context provided by the introduction or within a new context that emerges from your case study findings.

Note that, depending on the discipline you are writing in and your professor's preferences, the concluding paragraph may contain your final reflections on the evidence presented applied to practice or on the essay's central research problem. However, the nature of being introspective about the subject of analysis you have investigated will depend on whether you are explicitly asked to express your observations in this way.

Problems to Avoid

Overgeneralization One of the goals of a case study is to lay a foundation for understanding broader trends and issues applied to similar circumstances. However, be careful when drawing conclusions from your case study. They must be evidence-based and grounded in the results of the study; otherwise, it is merely speculation. Looking at a prior example, it would be incorrect to state that a factor in improving girls access to education in Azerbaijan and the policy implications this may have for improving access in other Muslim nations is due to girls access to social media if there is no documentary evidence from your case study to indicate this. There may be anecdotal evidence that retention rates were better for girls who were on social media, but this observation would only point to the need for further research and would not be a definitive finding if this was not a part of your original research agenda.

Failure to Document Limitations No case is going to reveal all that needs to be understood about a research problem. Therefore, just as you have to clearly state the limitations of a general research study , you must describe the specific limitations inherent in the subject of analysis. For example, the case of studying how women conceptualize the need for water conservation in a village in Uganda could have limited application in other cultural contexts or in areas where fresh water from rivers or lakes is plentiful and, therefore, conservation is understood differently than preserving access to a scarce resource.

Failure to Extrapolate All Possible Implications Just as you don't want to over-generalize from your case study findings, you also have to be thorough in the consideration of all possible outcomes or recommendations derived from your findings. If you do not, your reader may question the validity of your analysis, particularly if you failed to document an obvious outcome from your case study research. For example, in the case of studying the accident at the railroad crossing to evaluate where and what types of warning signals should be located, you failed to take into consideration speed limit signage as well as warning signals. When designing your case study, be sure you have thoroughly addressed all aspects of the problem and do not leave gaps in your analysis.

Case Studies . Writing@CSU. Colorado State University; Gerring, John. Case Study Research: Principles and Practices . New York: Cambridge University Press, 2007; Merriam, Sharan B. Qualitative Research and Case Study Applications in Education . Rev. ed. San Francisco, CA: Jossey-Bass, 1998; Miller, Lisa L. “The Use of Case Studies in Law and Social Science Research.” Annual Review of Law and Social Science 14 (2018): TBD; Mills, Albert J., Gabrielle Durepos, and Eiden Wiebe, editors. Encyclopedia of Case Study Research . Thousand Oaks, CA: SAGE Publications, 2010; Putney, LeAnn Grogan. "Case Study." In Encyclopedia of Research Design , Neil J. Salkind, editor. (Thousand Oaks, CA: SAGE Publications, 2010), pp. 116-120; Simons, Helen. Case Study Research in Practice . London: SAGE Publications, 2009;  Kratochwill,  Thomas R. and Joel R. Levin, editors. Single-Case Research Design and Analysis: New Development for Psychology and Education .  Hilldsale, NJ: Lawrence Erlbaum Associates, 1992; Swanborn, Peter G. Case Study Research: What, Why and How? London : SAGE, 2010; Yin, Robert K. Case Study Research: Design and Methods . 6th edition. Los Angeles, CA, SAGE Publications, 2014; Walo, Maree, Adrian Bull, and Helen Breen. “Achieving Economic Benefits at Local Events: A Case Study of a Local Sports Event.” Festival Management and Event Tourism 4 (1996): 95-106.

Writing Tip

At Least Five Misconceptions about Case Study Research

Social science case studies are often perceived as limited in their ability to create new knowledge because they are not randomly selected and findings cannot be generalized to larger populations. Flyvbjerg examines five misunderstandings about case study research and systematically "corrects" each one. To quote, these are:

Misunderstanding 1 :  General, theoretical [context-independent knowledge is more valuable than concrete, practical (context-dependent) knowledge. Misunderstanding 2 :  One cannot generalize on the basis of an individual case; therefore, the case study cannot contribute to scientific development. Misunderstanding 3 :  The case study is most useful for generating hypotheses; that is, in the first stage of a total research process, whereas other methods are more suitable for hypotheses testing and theory building. Misunderstanding 4 :  The case study contains a bias toward verification, that is, a tendency to confirm the researcher’s preconceived notions. Misunderstanding 5 :  It is often difficult to summarize and develop general propositions and theories on the basis of specific case studies [p. 221].

While writing your paper, think introspectively about how you addressed these misconceptions because to do so can help you strengthen the validity and reliability of your research by clarifying issues of case selection, the testing and challenging of existing assumptions, the interpretation of key findings, and the summation of case outcomes. Think of a case study research paper as a complete, in-depth narrative about the specific properties and key characteristics of your subject of analysis applied to the research problem.

Flyvbjerg, Bent. “Five Misunderstandings About Case-Study Research.” Qualitative Inquiry 12 (April 2006): 219-245.

• << Previous: Reviewing Collected Essays
• Next: Writing a Field Report >>
• Last Updated: Jan 17, 2023 10:50 AM
• URL: https://libguides.pointloma.edu/ResearchPaper

Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > The Case for Case Studies
• > Selecting Cases for Comparative Sequential Analysis

Book contents

• The Case for Case Studies
• Strategies for Social Inquiry
• Copyright page
• Contributors
• Preface and Acknowledgments
• 1 Using Case Studies to Enhance the Quality of Explanation and Implementation
• Part I Internal and External Validity Issues in Case Study Research
• Part II Ensuring High-Quality Case Studies
• 6 Descriptive Accuracy in Interview-Based Case Studies
• 7 Selecting Cases for Comparative Sequential Analysis
• 8 The Transparency Revolution in Qualitative Social Science
• Part III Putting Case Studies to Work: Applications to Development Practice

7 - Selecting Cases for Comparative Sequential Analysis

Novel Uses for Old Methods

from Part II - Ensuring High-Quality Case Studies

Published online by Cambridge University Press:  05 May 2022

Pavone analyzes how our evolving understanding of case-based causal inference via process-tracing should alter how we select cases for comparative inquiry. The chapter explicates perhaps the most influential and widely used means to conduct qualitative research involving two or more cases: Mill’s methods of agreement and difference. It then argues that the traditional use of Millian methods of case selection can lead us to treat cases as static units to be synchronically compared rather than as social processes unfolding over time. As a result, Millian methods risk prematurely rejecting and otherwise overlooking (1) ordered causal processes, (2) paced causal processes, and (3) equifinality, or the presence of multiple pathways that produce the same outcome. To address these issues, the chapter develops a set of recommendations to ensure the alignment of Millian methods of case selection with within-case sequential analysis.

7.1 Introduction

In the lead article of the first issue of Comparative politics , Harold Lasswell posited that the “scientific approach” and the “comparative method” are one and the same ( Reference Lasswell Lasswell 1968 : 3). So important is comparative case study research to the modern social sciences that two disciplinary subfields – comparative politics in political science and comparative-historical sociology – crystallized in no small part because of their shared use of comparative case study research ( Reference Collier and Finifter Collier 1993 ; Reference Adams, Clemens, Orloff, Adams, Clemens and Orloff Adams, Clemens, and Orloff 2005 : 22–26; Reference Mahoney and Thelen Mahoney and Thelen 2015 ). As a result, a first-principles methodological debate emerged about the appropriate ways to select cases for causal inquiry. In particular, the diffusion of econometric methods in the social sciences exposed case study researchers to allegations that they were “selecting on the dependent variable” and that “selection bias” would hamper the “answers they get” ( Reference Geddes Geddes 1990 ). Lest they be pushed to randomly select cases or turn to statistical and experimental approaches, case study researchers had to develop a set of persuasive analytic tools for their enterprise.

It is unsurprising, therefore, that there has been a profusion of scholarship discussing case selection over the years. Footnote 1 Reference Gerring and Cojocaru Gerring and Cojocaru (2015) synthesize this literature by deriving no less than five distinct types (representative, anomalous, most-similar, crucial, and most-different) and eighteen subtypes of cases, each with its own logic of case selection. It falls outside the scope of this chapter to provide a descriptive overview of each approach to case selection. Rather, the purpose of the present inquiry is to place the literature on case selection in constructive dialogue with the equally lively and burgeoning body of scholarship on process tracing ( Reference George and Bennett George and Bennett 2005 ; Reference Brady and Collier Brady and Collier 2010 ; Reference Beach and Pedersen Beach and Pedersen 2013 ; Reference Bennett and Checkel Bennett and Checkel 2015 ). I ask a simple question: Should our evolving understanding of causation and our toolkit for case-based causal inference courtesy of process-tracing scholars alter how scholars approach case selection? If so, why, and what may be the most fruitful paths forward?

To propose an answer, this chapter focuses on perhaps the most influential and widely used means to conduct qualitative research involving two or more cases: Mill’s methods of agreement and difference. Also known as the “most-different systems/cases” and “most-similar systems/cases” designs, these strategies have not escaped challenge – although, as we will see, many of these critiques were fallaciously premised on case study research serving as a weaker analogue to econometric analysis. Here, I take a different approach: I argue that the traditional use of Millian methods of case selection can indeed be flawed, but rather because it risks treating cases as static units to be synchronically compared rather than as social processes unfolding over time. As a result, Millian methods risk prematurely rejecting and otherwise overlooking (1) ordered causal processes, (2) paced causal processes, and (3) equifinality, or the presence of multiple pathways that produce the same outcome. While qualitative methodologists have stressed the importance of these processual dynamics, they have been less attentive to how these factors may problematize pairing Millian methods of case selection with within-case process tracing (e.g., Reference Hall, Mahoney and Rueschemeyer Hall 2003 ; Reference Tarrow Tarrow 2010 ; Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 ). This chapter begins to fill that gap.

Taking a more constructive and prescriptive turn, the chapter provides a set of recommendations for ensuring the alignment of Millian methods of case selection with within-case sequential analysis. It begins by outlining how the deductive use of processualist theories can help reformulate Millian case selection designs to accommodate ordered and paced processes (but not equifinal processes). More originally, the chapter concludes by proposing a new, alternative approach to comparative case study research: the method of inductive case selection . By making use of Millian methods to select cases for comparison after a causal process has been identified within a particular case, the method of inductive case selection enables researchers to assess (1) the generalizability of the causal sequences, (2) the logics of scope conditions on the causal argument, and (3) the presence of equifinal pathways to the same outcome. In so doing, scholars can convert the weaknesses of Millian approaches into strengths and better align comparative case study research with the advances of processualist researchers.

Organizationally, the chapter proceeds as follows. Section 7.2 provides an overview of Millian methods for case selection and articulates how the literature on process tracing fits within debates about the utility and shortcomings of the comparative method. Section 7.3 articulates why the traditional use of Millian methods risks blinding the researcher to ordered, paced, and equifinal causal processes, and describes how deductive, processualist theorizing helps attenuate some of these risks. Section 7.4 develops a new inductive method of case selection and provides a number of concrete examples from development practice to illustrate how it can be used by scholars and policy practitioners alike. Section 7.5 concludes.

7.2 Case Selection in Comparative Research

7.2.1 case selection before the processual turn.

Before “process tracing” entered the lexicon of social scientists, the dominant case selection strategy in case study research sought to maximize causal leverage via comparison, particularly via the “methods of agreement and difference” of John Stuart Reference Mill Mill (1843 [1974] : 388–391).

In Mill’s method of difference, the researcher purposively chooses two (or more) cases that experience different outcomes, despite otherwise being very similar on a number of relevant dimensions. Put differently, the researcher seeks to maximize variation in the outcome variable while minimizing variation amongst a set of plausible explanatory variables. It is for this reason that the approach also came to be referred to as the ‘most-similar systems’ or ‘most-similar cases’ design – while Mill’s nomenclature highlights variation in the outcome of interest, the alternative terminology highlights minimal variation amongst a set of possible explanatory factors. The underlying logic of this case selection strategy is that because the cases are so similar, the researcher can subsequently probe for the explanatory factor that actually does exhibit cross-case variation and isolate it as a likely cause.

Mill’s method of agreement is the mirror image of the method of difference. Here, the researcher chooses two (or more) cases that experience similar outcomes despite being very different on a number of relevant dimensions. That is, the researcher seeks to minimize variation in the outcome variable while maximizing variation amongst a set of plausible explanatory variables. An alternative, independent variable-focused terminology for this approach was developed – the ‘most-different systems’ or ‘most-different cases’ design – breeding some confusion. The underlying logic of this case selection strategy is that it helps the researcher isolate the explanatory factor that is similar across the otherwise different cases as a likely cause. Footnote 2

Figure 7.1 Case selection setup under Mill’s methods of difference and agreement

Mill himself did not believe that such methods could yield causal inferences outside of the physical sciences ( Reference Mill Mill 1843 [1974] : 452). Nevertheless, in the 1970s a number of comparative social scientists endorsed Millian methods as the cornerstones of the comparative method. For example, Reference Przeworski and Teune Przeworski and Teune (1970) advocated in favor of the most-different cases design, whereas Reference Lijphart Lijphart (1971) favored the most-similar cases approach. In so doing, scholars sought case selection techniques that would be as analogous as possible to regression analysis: focused on controlling for independent variables across cases, maximizing covariation between the outcome and a plausible explanatory variable, and treating cases as a qualitative equivalent to a row of dataset observations. It is not difficult to see why this contributed to the view that case study research serves as the “inherently flawed” version of econometrics ( Reference Adams, Clemens, Orloff, Adams, Clemens and Orloff Adams, Clemens, and Orloff 2005 : 25; Reference Tarrow Tarrow 2010 ). Indeed, despite his prominence as a case study researcher, Reference Lijphart Lijphart (1975 : 165; Reference Lijphart 1971 : 685) concluded that “because the comparative method must be considered the weaker method,” then “if at all possible one should generally use the statistical (or perhaps even the experimental) method instead.” As Reference Hall, Mahoney and Rueschemeyer Hall (2003 : 380; 396) brilliantly notes, case study research

was deeply influenced by [Lijphart’s] framing of it … [where] the only important observations to be drawn from the cases are taken on the values of the dependent variable and a few explanatory variables … From this perspective, because the number of pertinent observations available from small-N comparison is seriously limited, the analyst lacks the degrees of freedom to consider more than a few explanatory variables, and the value of small-N comparison for causal inference seems distinctly limited.

In other words, the predominant case selection approach through the 1990s sought to do its best to reproduce a regression framework in a small-N setting – hence Lijphart’s concern with the “many variables, small number of cases” problem, which he argued could only be partially mitigated if, inter alia , the researcher increases the number of cases and decreases the number of variables across said cases ( Reference Lijphart 1971 : 685–686). Later works embraced Lijphart’s formulation of the problem even as they sought to address it: for example, Reference Eckstein, Greenstein and Polsby Eckstein (1975 : 85) argued that a “case” could actually be comprised of many “cases” if the unit of analysis shifted from being, say, the electoral system to, say, the voter. Predictably, such interventions invited retorts: Reference Lieberson Lieberson (1994) , for example, claimed that Millian methods’ inability to accommodate probabilistic causation, Footnote 3 interaction effects, and multivariate analysis would remain fatal flaws.

7.2.2 Enter Process Tracing

It is in this light that ‘process tracing’ – a term first used by Reference Hobarth Hobarth (1972) but popularized by Reference George and Lauren George (1979 ) and particularly Reference George and Bennett George and Bennett (2005) , Reference Brady and Collier Brady and Collier (2010) , Reference Beach and Pedersen Beach and Pedersen (2013) , and Reference Bennett and Checkel Bennett and Checkel (2015) – proved revolutionary for the ways in which social scientists conceive of case study research. Cases have gradually been reconceptualized not as dataset observations but as concatenations of concrete historical events that produce a specific outcome ( Reference Mahoney Goertz and Mahoney 2012 ). That is, cases are increasingly treated as social processes, where a process is defined as “a particular type of sequence in which the temporally ordered events belong to a single coherent pattern of activity” ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 214). Although there exist multiple distinct conceptions of process tracing – from Bayesian approaches ( Reference Bennett, Bennett and Checkel Bennett 2015 ) to set-theoretic approaches ( Reference Mahoney, Kimball and Koivu Mahoney et al. 2009 ) to mechanistic approaches ( Reference Beach and Pedersen Beach and Pedersen 2013 ) to sequentialist approaches ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 ) – their overall esprit is the same: reconstructing the sequence of events and interlinking causal logics that produce an outcome – isolating the ‘causes of effects’ – rather than probing a variable’s mean impact across cases via an ‘effects of causes’ approach. Footnote 4

For this intellectual shift to occur, processualist social scientists had to show how a number of assumptions underlying Millian comparative methods – as well as frequentist approaches more generally – are usually inappropriate for case study research. For example, the correlational approach endorsed by Reference Przeworski and Teune Przeworski and Teune (1970) , Reference Lijphart Lijphart (1971) , and Reference Eckstein, Greenstein and Polsby Eckstein (1975) treats observational units as homogeneous and independent ( Reference Hall, Mahoney and Rueschemeyer Hall 2003 : 382; Reference Mahoney Goertz and Mahoney 2012 ). Unit homogeneity means that “different units are presumed to be fully identical to each other in all relevant respects except for the values of the main independent variable,” such that each observation contributes equally to the confidence we have in the accuracy and magnitude of our causal estimates ( Reference Brady and Collier Brady and Collier 2010 : 41–42). Given this assumption, more observations are better – hence, Reference Lijphart Lijphart (1971) ’s dictum to “increase the number of cases” and, in its more recent variant, to “increase the number of observations” ( Reference King, Keohane and Verba King, Keohane, and Verba 1994 : 208–230). By independence, we mean that “for each observation, the value of a particular variable is not influenced by its value in other observations”; thus, each observation contributes “new information about the phenomenon in question” ( Reference Brady and Collier Brady and Collier 2010 : 43).

By contrast, practitioners of process tracing have shown that treating cases as social processes implies that case study observations are often interdependent and derived from heterogeneous units ( Reference Mahoney Goertz and Mahoney 2012 ). Unit heterogeneity means that not all historical events, and the observable evidence they generate, are created equal. Hence, some observations may better enable the reconstruction of a causal process because they are more proximate to the central events under study. Correlatively, this is why historians accord greater ‘weight’ to primary than to secondary sources, and why primary sources concerning actors central to a key event are more important than those for peripheral figures ( Reference Trachtenberg Trachtenberg 2009 ; Reference Tansey Tansey 2007 ). In short, while process tracing may yield a bounty of observable evidence, we seek not to necessarily increase the number, but rather the quality, of observations. Finally, by interdependence we mean that because time is “fateful” ( Reference Sewell Sewell 2005 : 6), antecedent events in a sequence may influence subsequent events. This “fatefulness” has multiple sources. For instance, historical institutionalists have shown how social processes can exhibit path dependencies where the outcome of interest becomes a central driver of its own reproduction ( Reference Pierson Pierson 1996 ; Reference Pierson Pierson 2000 ; Reference Mahoney Mahoney 2000 ; Reference Hall, Mahoney and Rueschemeyer Hall 2003 ; Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 ). At the individual level, processual sociologists have noted that causation in the social world is rarely a matter of one billiard ball hitting another, as in Reference Hume Hume’s (1738 [2003]) frequentist concept of “constant conjunction.” Rather, it hinges upon actors endowed with memory, such that the micro-foundations of social causation rest on individuals aware of their own historicality ( Reference Sewell Sewell 2005 ; Reference Abbott Abbott 2001 ; Reference Abbott 2016 ).

At its core, eschewing the independence and unit homogeneity assumptions simply means situating case study evidence within its spatiotemporal context ( Reference Hall, Mahoney and Rueschemeyer Hall 2003 ; Reference Falleti and Lynch Falleti and Lynch 2009 ). This commitment is showcased by the language which process-sensitive case study researchers use when making causal inferences. First, rather than relating ‘independent variables’ to ‘dependent variables’, they often privilege the contextualizing language of relating ‘events’ to ‘outcomes’ ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 ). Second, they prefer to speak not of ‘dataset observations’ evocative of cross-sectional analysis, but of ‘causal process observations’ evocative of sequential analysis ( Reference Brady and Collier Brady and Collier 2010 ; Reference Mahoney Goertz and Mahoney 2012 ). Third, they may substitute the language of ‘causal inference via concatenation’ – a terminology implying that unobservable causal mechanisms are embedded within a sequence of observable events – for that of ‘causal inference via correlation’, evocative of the frequentist billiard-ball analogy ( Reference Waldner and Kincaid Waldner 2012 : 68). The result is that case study research is increasingly hailed as a “distinctive approach that offers a much richer set of observations, especially about causal processes, than statistical analyses normally allow” ( Reference Hall, Mahoney and Rueschemeyer Hall 2003 : 397).

7.3 Threats to Processual Inference and the Role of Theory

While scholars have shown how process-tracing methods have reconceived the utility of case studies for causal inference, there remains some ambiguity about the implications for case selection, particularly using Millian methods. While several works have touched upon this theme (e.g., Reference Hall, Mahoney and Rueschemeyer Hall 2003 ; Reference George and Bennett George and Bennett 2005 ; Reference Levy Levy 2008 ; Reference Tarrow Tarrow 2010 ), the contribution that most explicitly wrestles with this topic is Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney (2015) , who acknowledge that “the application of Millian methods for sequential arguments has not been systematically explored, although we believe it is commonly used in practice” ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 226). Falleti and Mahoney argue that process tracing can remedy the weaknesses of Millian approaches: “When used in isolation, the methods of agreement and difference are weak instruments for small-N causal inference … small-N researchers thus normally must combine Millian methods with process tracing or other within-case methods to make a positive case for causality” ( Reference Falleti, Mahoney, Mahoney and Thelen 2015 : 225–226). Their optimism about the synergy between Millian methods and process tracing leads them to conclude that “by fusing these two elements, the comparative sequential method merits the distinction of being the principal overarching methodology for [comparative-historical analysis] in general” ( Reference Falleti, Mahoney, Mahoney and Thelen 2015 : 236).

Falleti and Mahoney’s contribution is the definitive statement of how comparative case study research has long abandoned its Lijphartian origins and fully embraced treating cases as social processes. It is certainly true that process-tracing advocates have shown that some past critiques of Millian methods may not have been as damning as they first appeared. For example, Reference Lieberson Lieberson’s (1994) critique that Millian case selection requires a deterministic understanding of causation has been countered by set-theoretic process tracers who note that causal processes can indeed be conceptualized as concatenations of necessary and sufficient conditions ( Reference Mahoney Goertz and Mahoney 2012 ; Reference Mahoney and Vanderpoel Mahoney and Vanderpoel 2015 ). After all, “at the individual case level, the ex post (objective) probability of a specific outcome occurring is either 1 or 0” ( Reference Mahoney Mahoney 2008 : 415). Even for those who do not explicitly embrace set-theoretic approaches and prefer to perform a series of “process tracing tests” (such as straw-in-the-wind, hoop, smoking gun, and doubly-decisive tests), the objective remains to evaluate the deterministic causal relevance of a historical event on the next linkage in a sequence ( Reference Collier Collier 2011 ; Reference Mahoney Mahoney 2012 ). In this light, Millian methods appear to have been thrown a much-needed lifeline.

Yet processualist researchers have implicitly exposed new, and perhaps more damning, weaknesses in the traditional use of the comparative method. Here, Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney (2015) are less engaged in highlighting how their focus on comparing within-case sequences should push scholars to revisit strategies for case selection premised on assumptions that process-tracing advocates have undermined. In this light, I begin by outlining three hitherto underappreciated threats to inference associated with the traditional use of Millian case selection: potentially ignoring (1) ordered and (2) paced causal processes, and ignoring (3) the possibility of equifinality. I then demonstrate how risks (1) and (2) can be attenuated deductively by formulating processualist theories and tweaking Millian designs for case selection.

Risk 1: Ignoring Ordered Processes

Process-sensitive social scientists have long noted that “the temporal order of the events in a sequence [can be] causally consequential for the outcome of interest” ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 218; see also Reference Pierson Pierson 2004 : 54–78). For example, where individual acts of agency play a critical role – such as political elites’ response to a violent protest – “reordering can radically change [a] subject’s understanding of the meaning of particular events,” altering their response and the resulting outcomes ( Reference Abbott Abbott 1995 : 97).

An evocative illustration is provided by Reference Sewell Sewell’s (1996) analysis of how the storming of the Bastille in 1789 produced the modern concept of “revolution.” After overrunning the fortress, the crowd freed the few prisoners held within it; shot, stabbed, and beheaded the Bastille’s commander; and paraded his severed head through the streets of Paris ( Reference Sewell Sewell 1996 : 850). When the French National Assembly heard of the taking of the Bastille, it first interpreted the contentious event as “disastrous news” and an “excess of fury”; yet, when the king subsequently responded by retreating his troops to their provincial barracks, the Assembly recognized that the storming of the Bastille had strengthened its hand, and proceeded to reinterpret the event as a patriotic act of protest in support of political change ( Reference Sewell Sewell 1996 : 854–855). The king’s reaction to the Bastille thus bolstered the Assembly’s resolve to “invent” the modern concept of revolution as a “legitimate rising of the sovereign people that transformed the political system of a nation” ( Reference Sewell Sewell 1996 : 854–858). Proceeding counterfactually, had the ordering of events been reversed – had the king withdrawn his troops before the Bastille had been stormed – the National Assembly would have had little reason to interpret the popular uprising as a patriotic act legitimating reform rather than a violent act of barbarism.

Temporal ordering may also alter a social process’s political outcomes through macro-level mechanisms. For example, consider Reference Falleti Falleti’s (2005 , Reference Falleti 2010 ) analysis of the conditions under which state decentralization – the devolution of national powers to subnational administrative bodies – increases local political autonomy in Latin America. Through process tracing, Falleti demonstrates that when fiscal decentralization precedes electoral decentralization, local autonomy is increased, since this sequence endows local districts with the monetary resources necessary to subsequently administer an election effectively. However, when the reverse occurs, such that electoral decentralization precedes fiscal decentralization, local autonomy is compromised. For although the district is being offered the opportunity to hold local elections, it lacks the monetary resources to administer them effectively, endowing the national government with added leverage to impose conditions upon the devolution of fiscal resources.

For our purposes, what is crucial to note is not simply that temporal ordering matters, but that in ordered processes it is not the presence or absence of events that is most consequential for the outcome of interest. For instance, in Falleti’s analysis both fiscal and electoral decentralization occur. This means that a traditional Millian framework risks dismissing some explanatory events as causally irrelevant on the grounds that their presence is insufficient for explicating the outcome of interest (see Figure 7.2 ).

Figure 7.2 How ordered processes risk being ignored by a Millian setup

The way to deductively attenuate the foregoing risk is to develop an ordered theory and then modify the traditional Millian setup to assess the effect of ordering on an outcome of interest. That is, deductive theorizing aimed at probing the causal effect of ordering can guide us in constructing an appropriate Millan case selection design, such as that in Figure 7.3 . In this example, we redefine the fourth independent variable to measure not the presence or absence of a fourth event, but rather to measure the ordering of two previously defined events (in this case, events 1 and 2). This case selection setup would be appropriate if deductive theorizing predicts that the outcome of interest is produced when event 1 is followed by event 2 (such that, unless this specific ordering occurs, the presence of events 1 and 2 is insufficient to generate the outcome). In other words, if Millian methods are to be deductively used to select cases for comparison, the way to guard against prematurely dismissing the causal role of temporal ordering is to explicitly theorize said ordering a priori . If this proves difficult, or if the researcher lacks sufficient knowledge to develop such a theory, it is advisable to switch to the more inductive method for case selection outlined in the next section .

Figure 7.3 Deductively incorporating ordered processes within a Millian setup

Risk 2: Ignoring Paced Processes

Processualist researchers have also emphasized that, beyond temporal order, “the speed or duration of events … is causally consequential” ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 219). For example, social scientists have long distinguished an “eventful temporality” ( Reference Sewell Sewell 1996 ) from those “big, slow moving” incremental sequences devoid of rapid social change ( Reference Pierson, Mahoney and Rueschemeyer Pierson 2003 ). For historical institutionalists, this distinction is illustrated by “critical junctures” – defined as “relatively short periods of time during which there is a substantially heightened probability that agents’ choices will affect the outcome of interest” ( Reference Capoccia and Kelemen Capoccia and Kelemen 2007 : 348; Reference Capoccia, Mahoney and Thelen Capoccia 2015 : 150–151) – on the one hand, and those “causal forces that develop over an extended period of time,” such as “cumulative” social processes, sequences involving “threshold effects,” and “extended causal chains” on the other hand ( Reference Pierson Pierson 2004 : 82–90; Reference Mahoney, Thelen, Mahoney and Thelen Mahoney and Thelen 2010 ).

An excellent illustration is provided by Reference Beissinger Beissinger (2002) ’s analysis of the contentious events that led to the collapse of the Soviet State. Descriptively, the sequence of events has its origins in the increasing transparency of Soviet institutions and freedom of expression accompanying Gorbachev’s Glasnost ( Reference Beissinger Beissinger 2002 : 47). As internal fissures within the Politburo began to emerge in 1987, Glasnost facilitated media coverage of the split within the Soviet leadership ( Reference Beissinger 2002 : 64). In response, “interactive attempts to contest the state grew regularized and began to influence one another” ( Reference Beissinger 2002 : 74). These challenging acts mobilized around previously dormant national identities, and for the first time – often out of state incompetence – these early protests were not shut down ( Reference Beissinger 2002 : 67). Protests reached a boiling point in early 1989 as the first semicompetitive electoral campaign spurred challengers to mobilize the electorate and cultivate grievances in response to regime efforts to “control nominations and electoral outcomes” ( Reference Beissinger 2002 : 86). By 1990 the Soviet State was crumbling, and “in many parts of the USSR demonstration activity … had become a normal means for dealing with political conflict” ( Reference Beissinger 2002 : 90).

Crucially, Beissinger stresses that to understand the causal dynamics of the Soviet State’s collapse, highlighting the chronology of events is insufficient. The 1987–1990 period comprised a moment of “thickened history” wherein “what takes place … has the potential to move history onto tracks otherwise unimaginable … all within an extremely compressed period of time” ( Reference Beissinger 2002 : 27). Information overload, the density of interaction between diverse social actors, and the diffusion of contention engendered “enormous confusion and division within Soviet institutions,” allowing the hypertrophy of challenging acts to play “an increasingly significant role in their own causal structure” ( Reference Beissinger 2002 : 97, 27). In this light, the temporal compression of a sequence of events can bolster the causal role of human agency and erode the constraints of social structure. Proceeding counterfactually, had the exact same sequence of contentious events unfolded more slowly, it is doubtful that the Soviet State would have suddenly collapsed.

Many examples of how the prolongation of a sequence of events can render them invisible, and thus produce different outcomes, could be referenced. Consider, for example, how global climate change – which is highlighted by Reference Pierson Pierson (2004 : 81) as a prototypical process with prolonged time horizons – conditions the psychological response of social actors. As a report from the American Psychological Association underscores, “climate change that is construed as rapid is more likely to be dreaded,” for “people often apply sharp discounts to costs or benefits that will occur in the future … relative to experiencing them immediately” ( Reference Swim Swim et al. 2009 : 24–25; Reference Loewenstein and Elster Loewenstein and Elster 1992 ). This logic is captured by the metaphor of the “boiling frog”: “place a frog in a pot of cool water, and gradually raise the temperature to boiling, and the frog will remain in the water until it is cooked” ( Reference Boyatzis Boyatzis 2006 : 614).

What is important to note is that, once more, paced processes are not premised on the absence or presence of their constitutive events being causally determinative; rather, they are premised on the duration of events (or their temporal separation) bearing explanatory significance. Hence the traditional approach to case selection risks neglecting the causal impact of temporal duration on the outcome of interest (see Figure 7.4 ).

Figure 7.4 Paced processes risk being ignored by a Millian setup

Here, too, the way to deductively assess the causal role of pacing on an outcome of interest is to explicitly develop a paced theory before selecting cases for empirical analysis. On the one hand, we might theorize that it is the duration of a given event that is causally consequential; on the other hand, we might theorize that it is the temporal separation of said event from other events that is significant. Figure 7.5 suggests how a researcher can assess both theories through a revised Millian design. In the first example, we define a fourth independent variable measuring not the presence of a fourth event, but rather the temporal duration of a previously defined event (in this case, event 1). This would be an appropriate case selection design to assess a theory predicting that the outcome of interest occurs when event 1 unfolds over a prolonged period of time (such that if event 1 unfolds more rapidly, its mere occurrence is insufficient for the outcome). In the second example, we define a fourth independent variable measuring the temporal separation between two previously defined events (in this case, events 1 and 2). This would be an appropriate case selection design for a theory predicting that the outcome of interest only occurs when event 1 is temporally distant to event 2 (such that events 1 and 2 are insufficient for the outcome if they are proximate). Again, if the researcher lacks a priori knowledge to theorize how a paced process may be generating the outcome, it is advisable to adopt the inductive method of case selection described in Section 7.4 .

Figure 7.5 Deductively incorporating paced processes within a Millian setup

Risk 3: Ignoring Equifinal Causal Processes

Finally, researchers have noted that causal processes may be mired by equifinality: the fact that “multiple combinations of values … produce the same outcome” ( Reference Mahoney Mahoney 2008 : 424; see also Reference George and Bennett George and Bennett 2005 ; Reference Mahoney Goertz and Mahoney 2012 ). More formally, set-theoretic process tracers account for equifinality by emphasizing that, in most circumstances, “necessary” conditions or events are actually INUS conditions – individually necessary components of an unnecessary but sufficient combination of factors ( Reference Mahoney and Vanderpoel Mahoney and Vanderpoel 2015 : 15–18).

One of the reasons why processualist social scientists increasingly take equifinality seriously is the recognition that causal mechanisms may be context-dependent. Sewell’s work stresses that “the consequences of a given act … are not intrinsic to the act but rather will depend on the nature of the social world within which it takes place” ( Reference Sewell Sewell 2005 : 9–10). Similarly, Reference Falleti and Lynch Falleti and Lynch (2009 : 2; 11) argue that “causal effects depend on the interaction of specific mechanisms with aspects of the context within which these mechanisms operate,” hence the necessity of imposing “scope conditions” on theory building. One implication is that the exact same sequence of events in two different settings may produce vastly different causal outcomes. The flip side of this conclusion is that we should not expect a given outcome to always be produced by the same sequence of events.

For example, consider Sewell’s critique of Reference Skocpol Skocpol (1979) ’s States and Social Revolutions for embracing an “experimental temporality.” Skocpol deploys Millian methods of case selection to theorize that the great social revolutions – the French, Russian, and Chinese revolutions – were caused by a conjunction of three necessary conditions: “(1) military backwardness, (2) politically powerful landlord classes, and (3) autonomous peasant communities” ( Reference Sewell Sewell 2005 : 93). Yet to permit comparison, Skocpol assumes that the outcomes of one revolution, and the processes of historical change more generally, have no effect on a subsequent revolution ( Reference Sewell Sewell 2005 : 94–95). This approach amounts to “cutting up the congealed block of historical time into artificially interchangeable units,” ignoring the fatefulness of historical sequences ( Reference Sewell Sewell 2005 ). For example, the Industrial Revolution “intervened” between the French and Russian Revolutions, and consequently one could argue that “the revolt of the Petersburg and Moscow proletariat was a necessary condition for social revolution in Russia in 1917, even if it was not a condition for the French Revolution in 1789” ( Reference Sewell Sewell 2005 : 94–95). What Sewell is emphasizing, in short, is that peasant rebellion is an INUS condition (as is a proletariat uprising), rather than a necessary condition.

Another prominent example of equifinality is outlined by Reference Collier Collier’s (1999 : 5–11) review of the diverse pathways through which democratization occurs. In the elite-driven pathway, emphasized by Reference O’Donnell and Schmitter O’Donnell and Schmitter (1986 ), an internal split amongst authoritarian incumbents emerges; this is followed by liberalizing efforts by some incumbents, which enables the resurrection of civil society and popular mobilization; finally, authoritarian incumbents negotiate a pacted transition with opposition leaders. By contrast, in the working-class-driven pathway, emphasized by Reference Rueschemeyer, Stephens and Stephens Rueschemeyer, Stephens, and Stephens (1992) , a shift in the material balance of power in favor of the democracy-demanding working class and against the democracy-resisting landed aristocracy causes the former to overpower the latter, and via a democratic revolution from below a regime transition occurs. Crucially, Reference Collier Collier (1999 : 12) emphasizes that these two pathways need not be contradictory (or exhaustive): the elite-driven pathway appears more common in the Latin American context during the second wave of democratization, whereas the working-class-driven pathway appears more common in Europe during the first wave of democratization.

What is crucial is that Millian case selection is premised on there being a single cause underlying the outcome of interest. As a result, Millian methods risk dismissing a set of events as causally irrelevant ex ante in one case simply because that same set of events fails to produce the outcome in another case (see Figure 7.6 ). Unlike ordered and paced processes, there is no clear way to leverage deductive theorizing to reconfigure Millian methods for case selection and accommodate equifinality. However, I argue that the presence of equifinal pathways can be fruitfully probed if we embrace a more inductive approach to comparative case selection, as the next section outlines.

Figure 7.6 Equifinal causal processes risk being ignored by a Millian setup

7.4 A New Approach: The Method of Inductive Case Selection

If a researcher wishes to guard against ignoring consequential temporal dynamics but lacks the a priori knowledge necessary to develop a processual theory and tailor their case selection strategy, is there an alternative path forward? Yes, indeed: I suggest that researchers could wield most-similar or most-different cases designs to (1) probe causal generalizability, (2) reveal scope conditions, and (3) explore the presence of equifinality. Footnote 5 To walk through this more inductive case selection approach, I engage some case studies from development practice to illustrate how researchers and practitioners alike could implement and benefit from the method.

7.4.1 Tempering the Deductive Use of Millian Methods

To begin, one means to ensure against a Millian case selection design overlooking an ordered, paced, or equifinal causal process (in the absence of deductive theorizing) is to be wary of leveraging the methods of agreement and difference to eliminate potential explanatory factors ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 225–226). That is, the decision to discard an explanatory variable or historical event as causally unnecessary (via the method of agreement) or insufficient (via the method of difference) may be remanded to the process-tracing stage, rather than being made ex ante at the case selection stage.

Notice how this recommendation is particularly intuitive in light of the advances in process-tracing methods. Before this burgeoning literature existed, Millian methods were called upon to accomplish two things at once: (1) provide a justification for selecting two or more cases for social inquiry, and (2) yield causal leverage via comparison and the elimination of potential explanatory factors as unnecessary or insufficient. But process-tracing methodologists have showcased how the analysis of temporal variation disciplined via counterfactual analysis, congruence testing, and process-tracing tests renders within-case causal inference possible even in the absence of an empirical comparative case ( Reference George and Bennett George and Bennett 2005 ; Reference Gerring Gerring 2007 ; Reference Collier Collier 2011 ; Reference Mahoney Mahoney 2012 ; Reference Beach and Pedersen Beach and Pedersen 2013 ; Reference Bennett and Checkel Bennett and Checkel 2015 ; Reference Levy Levy 2015 ). That is, the ability to make causal inferences need not be primarily determined at the case selection stage.

The foregoing implies that if a researcher does not take temporal dynamics into account when developing their theory, the use of Millian methods should do no more than to provisionally discount the explanatory purchase of a given explanatory factor. The researcher should then bear in mind that as the causal process is reconstructed from a given outcome, the provisionally discounted factor may nonetheless be shown to be of causal relevance – particularly if the underlying process is ordered or paced, or if equifinal pathways are possible.

Despite these limitations, Millian methods might fruitfully serve additional functions from the standpoint of case selection, particularly if researchers shift (1) when and (2) why they make use of them. First, Millian methods may be as – if not more – useful after process tracing of a particular case is completed rather than to set the stage for within-case analysis. Such a chronological reversal – process tracing followed by Millian case selection, instead of Millian case selection followed by process tracing – inherently embraces a more inductive, theory-building approach to case study research ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 229–231) which, I suspect, is far more commonly used in practice than is acknowledged. I refer to this approach as the method of inductive case selection , wherein “theory-building process tracing” ( Reference Beach and Pedersen Beach and Pedersen 2013 : 16–18) of a single case is subsequently followed by the use of a most-similar or most-different cases design.

7.4.2 Getting Started: Selecting the Initial Case

The method of inductive case selection begins by assuming that the researcher has justifiable reasons for picking a particular case for process tracing and is subsequently looking to contextualize the findings or build a theory outwards. Hence, the first step involves picking an initial case. Qualitative methodologists have already supplied a number of plausible logics for selecting a single case, and I describe three nonexhaustive possibilities here: (1) theoretical or historical importance; (2) policy relevance and salience; and (3) empirically puzzling nature.

First, an initial case may be selected due to its theoretical or historical importance. Reference Eckstein, Greenstein and Polsby Eckstein (1975) , for example, defines an idiographic case study as a case where the specific empirical events/outcome serve as a central referent for a scholarly literature. As an illustration, Reference Gerring and Cojocaru Gerring and Cojocaru (2015 : 11) point to Reference North and Weingast North and Weingast (1989) ’s influential study of how the Glorious Revolution in seventeenth-century Britain favorably shifted the constitutional balance of power for the government to make credible commitments to protecting property rights (paving the way for the financial revolution of the early eighteenth century). Given that so much of the scholarly debate amongst economic historians centers on the institutional foundations of economic growth, North and Weingast’s case study was “chosen (it would appear) because of its central importance in the [historical political economy] literature on the topic, and because it is … a prominent and much-studied case” ( Reference Gerring and Cojocaru Gerring and Cojocaru 2015 : 11). In other words, Reference North and Weingast North and Weingast (1989) ’s study is idiographic in that it “aim[s] to explain and/or interpret a single historical episode,” but it remains “theory-guided” in that it “focuses attention on some theoretically specified aspects of reality and neglects others” ( Reference Levy Levy 2008 : 4).

While the causes of the Glorious Revolution are a much-debated topic amongst economic historians, they have less relevance to researchers and practitioners focused on assessing the effects of contemporary public policy interventions. Hence, a second logic for picking a first case for process tracing is its policy relevance and salience. Reference George and Bennett George and Bennett (2005 : 263–286) define a policy-relevant case study as one where the outcome is of interest to policy-makers and its causes are at least partially amenable to policy manipulation. For example, one recent World Bank case study ( Reference El-Saharty and Nagaraj El-Saharty and Nagaraj 2015 ) analyzes how HIV/AIDS prevalence amongst vulnerable subpopulations – particularly female sex workers – can be reduced via targeted service delivery. To study this outcome, two states in India – Andhra Pradesh and Karnataka – were selected for process tracing. There are three reasons why this constitutes an appropriate policy-relevant case selection choice. First, the outcome of interest – a decline in HIV/AIDS prevalence amongst female sex workers – was present in both Indian states. Second, because India accounts for almost 17.5 percent of the world population and has a large population of female sex workers, this outcome was salient to the government ( Reference El-Saharty and Nagaraj El-Saharty and Nagaraj 2015 : 3). Third, the Indian government had created a four-phase National AIDS Control Program (NACP) spanning from 1986 through 2017, meaning that at least one set of possible explanatory factors for the decline in HIV/AIDS prevalence comprised policy interventions that could be manipulated. Footnote 6

A third logic for picking an initial case for process tracing is its puzzling empirical nature. One obvious instantiation is when an exogenous shock or otherwise significant event/policy intervention yields a different outcome from the one scholars and practitioners expected. Footnote 7 For example, in 2004 the federal government of Nigeria partnered with the World Bank to improve the share of Nigeria’s urban population with access to piped drinking water. This partnership – the National Urban Water Sector Reform Project (NUWSRP1) – aimed to “increase access to piped water supply in selected urban areas by improving the reliability and financial viability of selected urban water utilities” and by shifting resources away from “infrastructure rehabilitation” that had failed in the past ( Reference Hima and Santibanez Hima and Santibanez 2015 : 2). Despite $200 million worth of investments, ultimately the NUWSRP1 “did not perform as strongly on the institutional reforms needed to ensure sustainability” ( Reference Hima and Santibanez Hima and Santibanez 2015 ). Given this puzzling outcome, the World Bank conducted an intensive case study to ask why the program did “not fully meet its essential objective of achieving a sustainable water delivery service” ( Reference Hima and Santibanez Hima and Santibanez 2015 ). Footnote 8

The common thread of these three logics for selecting an initial case is that the case itself is theoretically or substantively important and that its empirical dynamics – underlying either the outcome itself or its relationship to some explanatory events – are not well understood. That being said, the method of inductive case selection merely presumes that there is some theoretical, policy-related, empirical, or normative justification to pick the initial case.

7.4.3 Probing Generalizability Via a Most-Similar Cases Design

It is after picking an initial case that the method of inductive case selection contributes novel guidelines for case study researchers by reconfiguring how Millian methods are used. Namely, how should one (or more) additional cases be selected for comparison, and why? This question presumes that the researcher wishes to move beyond an idiographic, single-case study for the purposes of generating inferences that can travel. Yet in this effort, we should take seriously process-tracing scholars’ argument that causal mechanisms are often context-dependent. As a result, the selection of one or more comparative cases is not meant to uncover universally generalizable abstractions; rather, it is meant to contextualize the initial case within a set or family of cases that are spatiotemporally bounded.

That being said, the first logical step is to understand whether the causal inferences yielded by the process-traced case can indeed travel to other contexts ( Reference Goertz Goertz 2017 : 239). This constitutes the first reconfiguration of Millian methods: the use of comparative case studies to assess generalizability. Specifically, after within-case process tracing reveals a factor or sequence of factors as causally important to an outcome of interest, the logic is to select a case that is as contextually analogous as possible such that there is a higher probability that the causal process will operate similarly in the second case. This approach exploits the context-dependence of causal mechanisms to the researcher’s advantage: Similarity of context increases the probability that a causal mechanism will operate similarly across both cases. By “context,” it is useful to follow Reference Falleti and Lynch Falleti and Lynch (2009 : 14) and to be

concerned with a variety of contextual layers: those that are quite proximate to the input (e.g., in a study of the emergence of radical right-wing parties, one such layer might be the electoral system); exogenous shocks quite distant from the input that might nevertheless effect the functioning of the mechanism and, hence, the outcome (e.g., a rise in the price of oil that slows the economy and makes voters more sensitive to higher taxes); and the middle-range context that is neither completely exogenous nor tightly coupled to the input and so may include other relevant institutions and structures (the tax system, social solidarity) as well as more atmospheric conditions, such as rates of economic growth, flows of immigrants, trends in partisan identification, and the like.

For this approach to yield valuable insights, the researcher focuses on ‘controlling’ for as many of these contextual explanatory factors (crudely put, for as many independent variables) as possible. In other words, the researcher selects a most-similar case: if the causal chain similarly operates in the second case, this would support the conclusion that the causal process is likely at work across the constellation of cases bearing ‘family resemblances’ to the process-traced case ( Reference Soifer Soifer 2020 ). Figure 7.7 displays the logic of this design:

Figure 7.7 Probing generalizability by selecting a most-similar case

As in Figure 7.7 , suppose that process tracing of Case 1 reveals that some sequence of events (in this example, event 4 followed by event 5) caused the outcome of interest. The researcher would then select a most-similar case (a case with similar values/occurrences of other independent variables/events (here, IV1–IV3) that might also influence the outcome). The researcher would then scout whether the sequence in Case 1 (event 4 followed by event 5) also occurs in the comparative case. If it does, the expectation for a minimally generalizable theory is that it would produce a similar outcome in Case 2 as in Case 1. Correlatively, if the sequence does not occur in Case 2, the expectation is that it would not experience the same outcome as Case 1. These findings would provide evidence that the explanatory sequence (event 4 followed by event 5) has causal power that is generalizable across a set of cases bearing family resemblances.

For example, suppose a researcher studying democratization in Country A finds evidence congruent with the elite-centric theory of democratization of Reference O’Donnell and Schmitter O’Donnell and Schmitter (1986 ) described previously. To assess causal generalizability, the researcher would subsequently select a case – Country B – that is similar in the background conditions that the literature has shown to be conducive to democratization, such as level of GDP per capita ( Reference Przeworski and Limongi Przeworski and Limongi 1997 ; Reference Boix and Stokes Boix and Stokes 2003 ) or belonging to the same “wave” of democratization via spatial and temporal proximity ( Reference Collier, Rustow and Erickson Collier 1991 ; Reference Huntington Huntington 1993 ). Notice that these background conditions in Case B have to be at least partially exogenous to the causal process whose generalizability is being probed – that is, they cannot constitute the events that directly comprise the causal chain revealed in Case A. One way to think about them is as factors that in Case A appear to have been necessary, but less proximate and important, conditions for the outcome. Here, importance is determined by the “extent that they are [logically/counterfactually] present only when the outcome is present” ( Reference Mahoney, Kimball and Koivu Mahoney et al. 2009 : 119), whereas proximity is determined by the degree to which the condition is “tightly coupled” with the chain of events directly producing the outcome ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 233).

An example related to the impact of service delivery in developmental contexts can be drawn from the World Bank’s case study of HIV/AIDS interventions in India. Recall that this case study actually spans across two states: Andhra Pradesh and Karnataka. In a traditional comparative case study setup, the selection of both cases would seem to yield limited insights. After all, they are contextually similar: “Andhra Pradesh and Karnataka … represent the epicenter of the HIV/AIDS epidemic in India. In addition, they were early adopters of the targeted interventions”; and they also experience a similar outcome: “HIV/AIDS prevalence among female sex workers declined from 20 percent to 7 percent in Andhra Pradesh and from 15 percent to 5 percent in Karnataka between 2003 and 2011” ( Reference El-Saharty and Nagaraj El-Saharty and Nagaraj 2015 : 7; 3). In truth, this comparative case study design makes substantial sense: had the researchers focused on the impact of the Indian government’s NACP program only in Andhra Pradesh or only in Karnataka, one might have argued that there was something unique about either state that rendered it impossible to generalize the causal inferences. By instead demonstrating that favorable public health outcomes can be traced to the NACP program in both states, the researchers can support the argument that the intervention would likely prove successful in other contexts to the extent that they are similar to Andhra Pradesh and Karnataka.

One risk of the foregoing approach is highlighted by Reference Sewell Sewell (2005 : 95–96): contextual similarity may suggest cross-case interactions that hamper the ability to treat the second, most-similar case as if it were independent of the process-traced case. For example, an extensive body of research has underscored how protests often diffuse across proximate spatiotemporal contexts through mimicry and the modularity of repertoires of contention ( Reference Tilly Tilly 1995 ; Reference Tarrow Tarrow 1998 ). And, returning to the World Bank case study of HIV/AIDS interventions in Andhra Pradesh and Karnataka, one concern is that because these states share a common border, cross-state learning or other interactions might limit the value-added of a comparative design over a single case study, since the second case may not constitute truly new data. The researcher should be highly sensitive to this possibility when selecting and subsequently process tracing the most-similar case: the greater the likelihood of cross-case interactions, the lesser the likelihood that it is a case-specific causal process – as opposed to cross-case diffusion mechanism – that is doing most of the explanatory work.

Conversely, if the causal chain is found to operate differently in the second, most-similar case, then the researcher can make an argument for rejecting the generalizability of the causal explanation with some confidence. The conclusion would be that the causal process is sui generis and requires the “localization” of the theoretical explanation for the outcome of interest ( Reference Tarrow Tarrow 2010 : 251–252). In short, this would suggest that the process-traced case is an exceptional or deviant case, given a lack of causal generalizability even to cases bearing strong family resemblances. Here, we are using the ‘strong’ notion of ‘deviant’: the inability of a causal process to generalize to similar contexts substantially decreases the likelihood that “other cases” could be explained with reference to (or even in opposition to) the process-traced case.

There is, of course, the risk that by getting mired in the weeds of the first case, the researcher is unable to recognize how the overall chronology of events and causal logics in the most-similar case strongly resembles the process-traced case. That is, a null finding of generalizability in a most-similar context calls on the researcher to probe whether they have descended too far down the “ladder of generality,” requiring more abstract conceptual categories to compare effectively ( Reference Sartori Sartori 1970 ; Reference Collier and Levitsky Collier and Levitsky 1997 ).

7.4.4 Probing Scope Conditions and Equifinality Via a Most-Different Cases Design

A researcher that has process-traced a given case and revealed a factor or sequence of factors as causally relevant may also benefit from leveraging a most-different cases approach. This case selection technique yields complementary insights to the most-similar cases design described in the previous section , but its focus is altogether different: instead of uncovering the degree to which an identified causal process travels, the objective is to try to understand where and why it fails to travel and whether alternative pathways to the same outcome may be possible.

More precisely, by selecting a case that differs substantially from the process-traced case in background characteristics, the researcher maximizes contextual heterogeneity and the likelihood that the causal process will not generalize to the second case ( Reference Soifer Soifer 2020 ). Put differently, the scholar would be selecting a least-likely case for generalizability, because the context-dependence of causal mechanisms renders it unlikely that the same sequence of events will generate the same outcome in the second case. This would offer a first cut at establishing “scope conditions” upon the generalizability of the theory ( Reference Tarrow Tarrow 2010 : 251) by isolating which contextual factors prevented the process from producing the outcome in the most-different case.

Figure 7.8 provides a visual illustration of what this design could look like. Suppose, once more, that process tracing in Case 1 has revealed that some event 4 followed by event 5 generated the outcome of interest. To maximize the probability that we will be able to place scope conditions on this finding, we would select a comparative case that is most different to the process-traced case (a case with different values/occurrences of other independent variables/events [denoted as IV1–IV3 in Figure 7.8 ] that might also influence the outcome) but which also experienced the sequence of event 4 followed by event 5. Given the contextual differences between these two cases, the likelihood that the same sequence will produce the same outcome in both is low, which then opens up opportunities for the researcher to probe the logic of scope conditions. In this endeavor, temporality can serve as a useful guide: a means for restricting the set of potential contextual factors that prevented the causal process from reproducing the outcome in Case 2 is to identify at what chronological point the linkages between events 4 and 5 on the one hand and the outcome of interest on the other hand branched off from the way they unfolded in Case 1. The researcher can then scout which contextual factors exuded the greatest influence at that temporal location and identify them as central to the scope conditions to be placed upon the findings.

Figure 7.8 Probing scope conditions by selecting a most-different case

To provide an example for how this logic of inquiry can work, consider a recent case study focused on understanding the effectiveness of Mexico’s conditional cash transfer program – Opportunitades , the first program of its kind – in providing monetary support to the female heads of Indigenous households ( Reference Alva Estrabridis and Ortega Nieto Alva Estrabridis and Ortega Nieto 2015 ). The program suffered from the fact that Indigenous beneficiaries dropped out at higher rates than their non-Indigenous counterparts. In 2009 the World Bank spearheaded an Indigenous Peoples Plan (IPP) to bolster service delivery of cash transfers to Indigenous populations, which crucially included “catering to indigenous peoples in their native languages and disseminating information in their languages” ( Reference Alva Estrabridis and Ortega Nieto Alva Estrabridis and Ortega Nieto 2015 : 2). A subsequent impact evaluation found that “[w]hen program messages were offered in beneficiaries’ mother tongues, they were more convincing, and beneficiaries tended to participate and express themselves more actively” ( Reference Alva Estrabridis and Ortega Nieto Alva Estrabridis and Ortega Nieto 2015 ; Reference Mir, Gámez, Loyola, Martí and Veraza Mir et al. 2011 ).

Researchers might well be interested in the portability of the foregoing finding, in which case the previously described most-similar cases design is appropriate – for example, a comparison with the Familias en Accion program in Colombia may be undertaken ( Reference Attanasio, Battistin, Fitzsimons, Mesnard and Vera-Hernandez. Attanasio et al. 2005 ). But they might also be interested in the limits of the policy intervention – in understanding where and why it is unlikely to yield similar outcomes. To assess the scope conditions upon the “bilingualism” effect of cash transfer programs, a most-different cases design is appropriate. Thankfully, conditional cash transfer programs are increasingly common even in historical, cultural, and linguistic contexts markedly different from Mexico, most prominently in sub-Saharan Africa ( Reference Lagarde, Haines and Palmer Lagarde et al. 2007 ; Reference Garcia and Moore Garcia and Moore 2012 ). Selecting a comparative case from sub-Saharan Africa should prove effective for probing scope conditions: the more divergent the contextual factors, the less likely it is that the policy intervention will produce the same outcome in both contexts.

On the flip side, in the unlikely event that part or all of the causal process is nonetheless reproduced in the most-different case, the researcher would obtain a strong signal that they have identified one of those rare causal explanations of general scope. In coming to this conclusion, however, the researcher should be wary of “conceptual stretching” ( Reference Sartori Sartori 1970 : 1034), such that there is confidence that the similarity in the causal chain across the most-different cases lies at the empirical level and is not an artificial by-product of imprecise conceptual categories ( Reference Bennett and Checkel Bennett and Checkel 2015 : 10–11). Here process tracing, by pushing researchers to not only specify a sequence of “tightly-coupled” events ( Reference Falleti, Mahoney, Mahoney and Thelen Falleti and Mahoney 2015 : 233), but also to collect observable implications about the causal mechanisms concatenating these events, can guard against conceptual stretching. By opening the “black box” of causation through detailed within-case analysis, process tracing limits the researcher’s ability to posit “pseudo-equivalences” across contexts ( Reference Sartori Sartori 1970 : 1035).

Selecting a most-different case vis-à-vis the process-traced case is also an excellent strategy for probing equifinality – for maximizing the likelihood that the scholar will be able to probe multiple causal pathways to the same outcome. To do so, it is not sufficient to merely ensure divergence in background conditions; it is equally necessary to follow Mill’s method of agreement by ensuring that the outcome in the process-traced case is also present in the second, most-different case. By ensuring minimal variation in outcome, the scholar guarantees that process tracing the second case will lead to the desired destination; by ensuring maximal variation in background conditions, the scholar substantially increases the likelihood that process tracing will reveal a slightly or significantly different causal pathway to said destination. Should an alternative route to the outcome be found, then its generalizability could be assessed using the most-similar cases approach described previously.

Figure 7.9 visualizes what this case selection design might look like. Here, as in previous examples, suppose process tracing in Case 1 provides evidence that event 4 followed by event 5 produced the outcome of interest. The researcher then selects a case with the same outcome, but with different values/occurrences of some independent variables/events (in this case, IV1–IV3) that may influence the outcome. Working backwards from the outcome to reconstruct the causal chain that produced it, the researcher then probes whether (i) the sequence (event 4 followed by event 5) also occurred in Case 2, and (ii) whether the outcome of interest can be retraced to said sequence. Given the contextual dissimilarities between these most-different cases, such a finding is rather unlikely, which would subsequently enable to the researcher to probe whether some other factor (perhaps IV2/event 2 in the example of Figure 7.9 ) produced the outcome in the comparative case instead, which would comprise clear evidence of equifinality.

Figure 7.9 Probing equifinality by selecting a most-different case with the same outcome

To return to the concrete example of Mexico’s conditional cash transfer program’s successful outreach to marginalized populations via bilingual service provision, an alternative route to the same outcome might be unearthed if a cash transfer program without bilingual outreach implemented in a country characterized by different linguistic, gender, and financial decision-making norms proves similarly successful in targeting marginalized populations. Several factors – including recruitment procedures, the size of the cash transfers, the requirements for participation, and the supply of other benefits ( Reference Lagarde, Haines and Palmer Lagarde et al. 2007 : 1902) – could interact with the different setting to produce similar intervention outcomes, regardless of whether multilingual services are provided. Such a finding would suggest that these policy interventions can be designed in multiple ways and still prove effective.

To conclude, the method of inductive case selection complements within-case analysis by supplying a coherent logic for probing generalizability, scope conditions, and equifinality. To summarize, Figure 7.10 provides a roadmap of this approach to comparative case selection.

Figure 7.10 Case selection roadmap to assess generalizability, scope conditions, equifinality

In short, if the researcher has the requisite time and resources, a multistage use of Millian methods to conduct four comparative case studies could prove very fertile. The researcher would begin by selecting a second, most-similar case to assess causal generalizability to a family of cases similar to the process-traced case; subsequently, a third, most-different case would be selected to surface possible scope conditions blocking the portability of the theory to divergent contexts; and a fourth, most-different case experiencing the same outcome would be picked to probe equifinal pathways. This sequential, four-case comparison would substantially improve the researcher’s ability to map the portability and contours of both their empirical analysis and their theoretical claims. Footnote 9

7.5 Conclusion

The method of inductive case selection converts process tracing meant to simply “craft a minimally sufficient explanation of a particular outcome” into a methodology used to build and refine a causal theory – a form of “theory-building process-tracing” ( Reference Beach and Pedersen Beach and Pedersen 2013 : 16–18). Millian methods are called upon to probe the portability of a particular causal process or causal mechanism and to specify the logics of its relative contextual-dependence. In so doing, they enable theory-building without presuming that the case study researcher holds the a priori knowledge necessary to account for complex temporal dynamics at the deductive theorizing stage. Both of these approaches – deductive, processualist theorizing on the one hand, and the method of inductive case selection on the other hand – provide some insurance against Millian methods leading the researcher into ignoring the ordered, paced, or equifinal structure that may underlie the pathway(s) to the outcome of interest. But, I would argue, the more inductive approach is uniquely suited for research that is not only process-sensitive, but also open to novel insights supplied by the empirical world that may not be captured by existing theories.

Furthermore, case study research often does (and should!) proceed with the scholar outlining why an outcome is of interest, and then seeking ways to not only make inferences about what produced said outcome (via process tracing) but situating it within a broader empirical and theoretical landscape (via the method of inductive case selection). This approach pushes scholars to answer that pesky yet fundamental question – why should we care or be interested in this case/outcome? – before disciplining their drive for generalizable causal inferences. After all, the deductive use of Millian methods tells us nothing about why we should care about the cases selected, yet arguably this is an essential component of any case selection justification. By deploying a most-similar or most-different cases design after an initial case has been justifiably selected due to its theoretical or historical importance, policy relevance, or puzzling empirical nature, the researcher is nudged toward undertaking case study research yielding causal theories that are not only comparatively engaged, but also substantively interesting.

The method of inductive case selection is most useful when the foregoing approach constitutes the esprit of the case study researcher. Undoubtedly, deductively oriented case study research (see Reference Lieberman Lieberman 2005 ; Reference Lieberman, Mahoney and Thelen 2015 ) and traditional uses of Millian methods will continue to contribute to social scientific understanding. Nevertheless, the perils of ignoring important sequential causal dynamics – particularly in the absence of good, processualist theories – should caution researchers to proceed with the greatest of care. In particular, researchers should be willing to revise both theory building and research design to its more inductive variant should process tracing reveal temporal sequences that eschew the analytic possibilities of the traditional comparative method.

I would like to thank Jennifer Widner and Michael Woolcock for the invitation to write this chapter, and Daniel Ortega Nieto for pointing me to case studies conducted by the World Bank’s Global Delivery Initiative that I use as illustrative examples, as well as Jack Levy, Hillel Soifer, Andrew Moravcsik, Cassandra Emmons, Rory Truex, Dan Tavana, Manuel Vogt, and Killian Clarke for constructive feedback.

1 See, for example, Reference Przeworski and Teune Przeworski and Teune (1970) , Reference Lijphart Lijphart (1971) , Reference Eckstein, Greenstein and Polsby Eckstein (1975) , Reference Yin Yin (1984) , Reference Geddes Geddes (1990) , Reference Collier and Finifter Collier (1993) , Reference Faure Faure (1994) , Reference George and Bennett George and Bennett (2005) , Reference Flyvbjerg Flyvbjerg (2006) , Reference Levy Levy (2008) , Reference Seawright and Gerring Seawright and Gerring (2008) , Reference Gerring Gerring (2007) , Reference Brady and Collier Brady and Collier (2010) , and Reference Tarrow Tarrow (2010) .

2 Some scholars, such as Reference Faure Faure (1994) , distinguish Mill’s dependent-variable driven methods of agreement and difference from the independent-variable driven most-similar and most-different systems designs, suggesting they are distinct. But because, as Figure 7.1 shows, Mill’s dependent-variable driven methods also impose requirements on the array of independent variables to permit causal inference via exclusion, this distinction is not particularly fertile.

3 In Mill’s method of difference, factors present in both cases are eliminated for being insufficient for the outcome (in the method of agreement, factors that vary across the cases are eliminated for being unnecessary).

4 Note that Mill himself distinguished between deductively assessing the average “effect of causes” and inductively retracing the “causes of effects” using the methods of agreement and disagreement ( Reference Mill Mill 1843 [1974] , pp. 449, 764).

5 The proposed approach bears several similarities to Reference Soifer Soifer’s (2020) fertile analysis of how “shadow cases” in comparative research can contribute to theory-building and empirical analysis.

6 This study found that the expansion of clinical services into government facilities embedded in the public health system, the introduction of peer educators, and the harmonization of large quantities of public health data underlay the timing and breadth of the decline in HIV/AIDS amongst female sex workers.

7 What Reference Levy Levy (2008 :13) calls a “deviant” case – which “focus[es] on observed empirical anomalies in existing theoretical propositions” – would also fit within the category of a puzzling case.

8 Process tracing revealed that a conjunction of factors – management turnover and a lackluster culture of staff performance at the state level, inadequate coordination at the federal level, premature disbursement of funds, and citizen aversion to the commercialization of the public water supply – underlay the initially perplexing underperformance of the urban water delivery project.

9 Many thanks to Rory Truex for highlighting this implication of the roadmap in Figure 7.5 .

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Selecting Cases for Comparative Sequential Analysis
• By Tommaso Pavone
• Edited by Jennifer Widner , Princeton University, New Jersey , Michael Woolcock , Daniel Ortega Nieto
• Book: The Case for Case Studies
• Online publication: 05 May 2022
• Chapter DOI: https://doi.org/10.1017/9781108688253.008

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 25 February 2020

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

• Bella Reichard   ORCID: orcid.org/0000-0001-5057-4019 1 ,
• Mark S Reed 1 ,
• Jenn Chubb 2 ,
• Ged Hall   ORCID: orcid.org/0000-0003-0815-2925 3 ,
• Lucy Jowett   ORCID: orcid.org/0000-0001-7536-3429 4 ,
• Alisha Peart 4 &
• Andrea Whittle 1  

Palgrave Communications volume  6 , Article number:  31 ( 2020 ) Cite this article

24k Accesses

14 Citations

82 Altmetric

Metrics details

• Language and linguistics

This paper reports on two studies that used qualitative thematic and quantitative linguistic analysis, respectively, to assess the content and language of the largest ever sample of graded research impact case studies, from the UK Research Excellence Framework 2014 (REF). The paper provides the first empirical evidence across disciplinary main panels of statistically significant linguistic differences between high- versus low-scoring case studies, suggesting that implicit rules linked to written style may have contributed to scores alongside the published criteria on the significance, reach and attribution of impact. High-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach than low-scoring cases. High-scoring case studies contained attributional phrases which were more likely to attribute research and/or pathways to impact, and they were written more coherently (containing more explicit causal connections between ideas and more logical connectives) than low-scoring cases. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, and often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than typically associated with academic writing. High-scoring case studies in two Main Panels were significantly easier to read than low-scoring cases on the Flesch Reading Ease measure, although both high-scoring and low-scoring cases tended to be of “graduate” reading difficulty. The findings of our work enable impact case study authors to better understand the genre and make content and language choices that communicate their impact as effectively as possible. While directly relevant to the assessment of impact in the UK’s Research Excellence Framework, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

Similar content being viewed by others

Research impact evaluation and academic discourse

Demystifying the process of scholarly peer-review: an autoethnographic investigation of feedback literacy of two award-winning peer reviewers

Aspiring to greater intellectual humility in science

Introduction.

Academics are under increasing pressure to engage with non-academic actors to generate “usable” knowledge that benefits society and addresses global challenges (Clark et al., 2016 ; Lemos, 2015 ; Rau et al., 2018 ). This is largely driven by funders and governments that seek to justify the societal value of public funding for research (Reed et al., 2020 ; Smith et al., 2011 ) often characterised as ‘impact’. While this has sometimes been defined narrowly as reflective of the need to demonstrate a return on public investment in research (Mårtensson et al., 2016 ; Tsey et al., 2016 ; Warry, 2006 ), there is also a growing interest in the evaluation of “broader impacts” from research (cf. Bozeman and Youtie, 2017 ; National Science Foundation, 2014 ), including less tangible but arguably equally relevant benefits for society and culture. This shift is exemplified by the assessment of impact in the UK’s Research Excellence Framework (REF) in 2014 and 2021, the system for assessing the quality of research in UK higher education institutions, and in the rise of similar policies and evaluation systems in Australia, Hong Kong, the United States, Horizon Europe, The Netherlands, Sweden, Italy, Spain and elsewhere (Reed et al., 2020 ).

The evaluation of research impact in the UK has been criticised by scholars largely for its association with a ‘market logic’ (Olssen and Peters, 2005 ; Rhoads and Torres, 2005 ). Critics argue that a focus of academic performativity can be seen to “destabilise” professional identities (Chubb and Watermeyer, 2017 ), which in the context of research impact evaluation can further “dehumanise and deprofessionalise” academic performance (Watermeyer, 2019 ), whilst leading to negative unintended consequences (which Derrick et al., 2018 , called “grimpact”). MacDonald ( 2017 ), Chubb and Reed ( 2018 ) and Weinstein et al. ( 2019 ) reported concerns from researchers that the impact agenda may be distorting research priorities, “encourag[ing] less discovery-led research” (Weinstein et al., 2019 , p. 94), though these concerns were questioned by University managers in the same study who were reported to “not have enough evidence to support that REF was driving specific research agendas in either direction” (p. 94), and further questioned by Hill ( 2016 ).

Responses to this critique have been varied. Some have called for civil disobedience (Watermeyer, 2019 ) and organised resistance (Back, 2015 ; MacDonald, 2017 ) against the impact agenda. In a review of Watermeyer ( 2019 ), Reed ( 2019 ) suggested that attitudes towards the neoliberal political roots of the impact agenda may vary according to the (political) values and beliefs of researchers, leading them to pursue impacts that either support or oppose neoliberal political and corporate interests. Some have defended the benefits of research impact evaluation. For example, Weinstein et al. ( 2019 ) found that “a focus on changing the culture outside of academia is broadly valued” by academics and managers. The impact agenda might enhance stakeholder engagement (Hill, 2016 ) and give “new currency” to applied research (Chubb, 2017 ; Watermeyer, 2019 ). Others have highlighted the long-term benefits for society of incentivising research impact, including increased public support and funding for a more accountable, outward-facing research system (Chubb and Reed, 2017 ; Hill, 2016 ; Nesta, 2018 ; Oancea, 2010 , 2014 ; Wilsdon et al., 2015 ).

In the UK REF, research outputs and impact are peer reviewed at disciplinary level in ‘Units of Assessment’ (36 in 2014, 34 in 2021), grouped into four ‘Main Panels’. Impact is assessed through case studies that describe the effects of academic research and are given a score between 1* (“recognised but modest”) and 4* (“outstanding”). The case studies follow a set structure of five sections: 1—Summary of the impact; 2—Underpinning research; 3—References to the research; 4—Details of the impact; 5—Sources to corroborate the impact (HEFCE, 2011 ). The publication of over 6000 impact case studies in 2014 Footnote 1 by Research England (formerly Higher Education Funding Council for England, HEFCE) was unique in terms of its size, and unlike the recent selective publication of high-scoring case studies from Australia’s 2018 Engagement and Impact Assessment, both high-scoring and low-scoring case studies were published. This provides a unique opportunity to evaluate the construction of case studies that were perceived by evaluation panels to have successfully demonstrated impact, as evidenced by a 4* rating, and to compare these to case studies that were judged as less successful.

The analysis of case studies included in this research is based on the definition of impact used in REF2014, as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (HEFCE, 2011 , p. 26). According to REF2014 guidance, the primary functions of an impact case study were to articulate and evidence the significance and reach of impacts arising from research beyond academia, clearly demonstrating the contribution that research from a given institution contributed to those impacts (HEFCE, 2011 ).

In addition to these explicit criteria driving the evaluation of impact in REF2014, a number of analyses have emphasised the role of implicit criteria and subjectivity in shaping the evaluation of impact. For example, Pidd and Broadbent ( 2015 ) emphasised the implicit role a “strong narrative” plays in high-scoring case studies (p. 575). This was echoed by the fears of one REF2014 panellist interviewed by Watermeyer and Chubb ( 2018 ) who said, “I think with impact it is literally so many words of persuasive narrative” as opposed to “giving any kind of substance” (p. 9). Similarly, Watermeyer and Hedgecoe ( 2016 ), reporting on an internal exercise at Cardiff University to evaluate case studies prior to submission, emphasised that “style and structure” were essential to “sell impact”, and that “case studies that best sold impact were those rewarded with the highest evaluative scores” (p. 651).

Recent research based on interviews with REF2014 panellists has also emphasised the subjectivity of the peer-review process used to evaluate impact. Derrick’s ( 2018 ) research findings based on panellist interviews and participant observation of REF2014 sub-panels argued that scores were strongly influenced by who the evaluators were and how the group assessed impact together. Indeed, a panellist interviewed by Watermeyer and Chubb ( 2018 ) concurred that “the panel had quite an influence on the criteria” (p. 7), including an admission that some types of (more intangible) evidence were more likely to be overlooked than other (more concrete) forms of evidence, “privileg[ing] certain kinds of impact”. Other panellists interviewed spoke of their emotional and intellectual vulnerability in making judgements about an impact criterion that they had little prior experience of assessing (Watermeyer and Chubb, 2018 ). Derrick ( 2018 ) argued that this led many evaluators to base their assessments on more familiar proxies for excellence linked to scientific excellence, which led to biased interpretations and shortcuts that mimicked “groupthink” (p. 193).

This paper will for the first time empirically assess the content and language of the largest possible sample of research impact case studies that received high versus low scores from assessment panels in REF2014. Combining qualitative thematic and quantitative linguistic analysis, we ask:

How do high-scoring versus low-scoring case studies articulate and evidence impacts linked to underpinning research?

Do high-scoring and low-scoring case studies have differences in their linguistic features or styles?

Do high-scoring and low-scoring case studies have lexical differences (words and phrases that are statistically more likely to occur in high- or low-scoring cases) or text-level differences (including reading ease, narrative clarity, use of cohesive devices)?

By answering these questions, our goal is to provide evidence for impact case study authors and their institutions to reflect on in order to optimally balance the content and to use language that communicates their impact as effectively as possible. While directly relevant to the assessment of impact in the UK’s REF, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

Research design and sample

The datasets were generated by using published institutional REF2014 impact scores to deduce the scores of some impact case studies themselves. Although scores for individual case studies were not made public, we were able to identify case studies that received the top mark of 4* based on the distribution of scores received by some institutions, where the whole submission by an institution in a given Unit of Assessment was awarded the same score. In those 20 Units of Assessment (henceforth UoA) where high-scoring case studies could be identified in this way, we also accessed all case studies known to have scored either 1* or 2* in order to compare the features of high-scoring case studies to those of low-scoring case studies.

We approached our research questions with two separate studies, using quantitative linguistic and qualitative thematic analysis respectively. The thematic analysis, explained in more detail in the section “Qualitative thematic analysis” below, allowed us to find answers to research question 1 (see above). The quantitative linguistic analysis was used to extract and compare typical word combinations for high-scoring and low-scoring case studies, as well as assessing their readability. It mainly addressed research questions 2 and 3.

The quantitative linguistic analysis was based on a sample of all identifiable high-scoring case studies in any UoA ( n  = 124) and all identifiable low-scoring impact case studies in those UoAs where high-scoring case studies could be identified ( n  = 93). As the linguistic analysis focused on identifying characteristic language choices in running text, only those sections designed to contain predominantly text were included (1—Summary of the impact; 2—Underpinning research; 4—Details of the impact). Figure 1 shows the distribution of case studies across Main Panels in the quantitative analysis. Table 1 summarises the number of words included in the analysis.

Distribution of case studies across Main Panels used for the linguistic analysis sample.

In order to detect patterns of content in high-scoring and low-scoring case studies across all four Main Panels, a sub-sample of case studies was selected for a qualitative thematic analysis. This included 60% of high-scoring case studies and 97% of low-scoring case studies from the quantitative analysis, such that only UoAs were included where both high-scoring and low-scoring case studies are available (as opposed to the quantitative sample, which includes all available high-scoring case studies). Further selection criteria were then designed to create a greater balance in the number of high-scoring and low-scoring case studies across Main Panels. Main Panels A (high) and C (low) were particularly over-represented, so a lower proportion of those case studies were selected and 10 additional high-scoring case studies were considered in Panel B, including institutions where at least 85% of the case studies scored 4* and the remaining scores were 3*. As this added a further UoA, we could also include 14 more low-scoring case studies in Main Panel B. This resulted in a total of 85 high-scoring and 90 low-scoring case studies. Figure 2 shows the distribution of case studies across Main Panels in the thematic analysis, illustrating the greater balance compared to the sample used in the quantitative analysis. The majority (75%) of the case studies analysed are included in both samples (Table 2 ).

Distribution of case studies across Main Panels used for the thematic analysis sample.

Quantitative linguistic analysis

Quantitative linguistic analysis can be used to make recurring patterns in language use visible and to assess their significance. We treated the dataset of impact case studies as a text collection (the ‘corpus’) divided into two sections, namely high-scoring and low-scoring case studies (the two ‘sub-corpora’), in order to explore the lexical profile and the readability of the case studies.

One way to explore the lexical profile of groups of texts is to generate frequency-based word lists and compare these to word lists from a reference corpus to determine which words are characteristic of the corpus of interest (“keywords”, cf. Scott, 1997 ). Another way is to extract word combinations that are particularly frequent. Such word combinations, called “lexical bundles”, are “extended collocations” (Hyland, 2008 , p. 41) that appear across a set range of texts (Esfandiari and Barbary, 2017 ). We merged these two approaches in order to uncover meanings that could not be made visible through the analysis of single-word frequencies, comparing lexical bundles from each sub-corpus to the other. Lexical bundles of 2–4 words were extracted with AntConc (specialist software developed by Anthony, 2014 ) firstly from the corpus of all high-scoring case studies and then separately from the sub-corpora of high-scoring case studies in Main Panel A, C and D. Footnote 2 The corresponding lists were extracted from low-scoring case studies overall and separated by panel. The lists of lexical bundles for each of the high-scoring corpus parts were then compared to the corresponding low-scoring parts (High-Overall vs. Low-Overall, High-Main Panel A vs. Low-Main Panel A, etc.) to detect statistically significant over-use and under-use in one set of texts relative to another.

Two statistical measures were used in the analysis of lexical bundles. Log Likelihood was used as a measure of the statistical significance of frequency differences (Rayson and Garside, 2000 ), with a value of >3.84 corresponding to p  < 0.05. This measure had the advantage, compared to the more frequently used chi-square test, of not assuming a normal distribution of data (McEnery et al., 2006 ). The Log Ratio (Hardie, 2014 ) was used as a measure of effect size, which quantifies the scale, rather than the statistical significance, of frequency differences between two datasets. The Log Ratio is technically the binary log of the relative risk, and a value of >0.5 or <−0.5 is considered meaningful in corpus linguistics (Hardie, 2014 ), with values further removed from 0 reflecting a bigger difference in the relative frequencies found in each corpus. There is currently no agreed standard effect size measure for keywords (Brezina, 2018 , p. 85) and the Log Ratio was chosen because it is straightforward to interpret. Each lexical bundle that met the ‘keyness’ threshold (Log Likelihood > 3.84 in the case of expected values > 12, with higher significance levels needed for expected values < 13—see Rayson et al., 2004 , p. 8) was then assigned a code according to its predominant meaning in the texts, as reflected in the contexts captured in the concordance lines extracted from the corpus.

In the thematic analysis, it appeared that high-scoring case studies were easier to read. In order to quantify the readability of the texts, we therefore analysed them using the Coh-Metrix online tool (www.cohmetrix.com, v3.0) developed by McNamara et al. ( 2014 ). This tool provides 106 descriptive indices of language features, including 8 principal component scores developed from combinations of the other indices (Graesser et al., 2011 ). We selected these principal component scores as comprehensive measures of “reading ease” because they assess multiple characteristics of the text, up to whole-text discourse level (McNamara et al., 2014 , p. 78). This was supplemented by the traditional and more wide-spread Flesch Reading Ease score of readability measuring the lengths of words and sentences, which are highly correlated with reading speed (Haberlandt and Graesser, 1985 ). The selected measures were compared across corpus sections using t -tests to evaluate significance. The effect size was measured using Cohen’s D , following Brezina ( 2018 , p. 190), where D  > 0.3 indicates a small, D  > 0.5 a medium, and D  > 0.8 a high effect size. As with the analysis of lexical bundles, comparisons were made between high- and low-scoring case studies in each of Main Panels A, C and D, as well as between all high-scoring and all low-scoring case studies across Main Panels.

Qualitative thematic analysis

While a quantitative analysis as described above can make differences in the use of certain words visible, it does not capture the narrative or content of the texts under investigation. In order to identify common features of high-scoring and low-scoring case studies, thematic analysis was chosen to complement the quantitative analysis by identifying patterns and inferring meaning from qualitative data (Auerbach and Silverstein, 2003 ; Braun and Clarke, 2006 ; Saldana, 2009 ). To familiarise themselves with the data and for inter-coder reliability, two research team members read a selection of REF2014 impact case studies from different Main Panels, before generating initial codes for each of the five sections of the impact case study template. These were discussed with the full research team, comprising three academic and three professional services staff who had all read multiple case studies themselves. They were piloted prior to defining a final set of themes and questions against which the data was coded (based on the six-step process outlined by Braun and Clarke, 2006 ) (Table 3 ). An additional category was used to code stylistic features, to triangulate elements of the quantitative analysis (e.g. readability) and to include additional stylistic features difficult to assess in quantitative terms (e.g. effective use of testimonials). In addition to this, 10 different types of impact were coded for, based on Reed’s ( 2018 ) typology: capacity and preparedness, awareness and understanding, policy, attitudinal change, behaviour change and other forms of decision-making, other social, economic, environmental, health and wellbeing, and cultural impacts. There was room for coders to include additional insights arising in each section of the case study that had not been captured in the coding system; and there was room to summarise other key factors they thought might account for high or low scores.

Coders summarised case study content pertaining to each code, for example by listing examples of effective or poor use of structure and formatting as they arose in each case study. Coders also quoted the original material next to their summaries so that their interpretation could be assessed during subsequent analysis. This initial coding of case study text was conducted by six coders, with intercoder reliability (based on 10% of the sample) assessed at over 90%. Subsequent thematic analysis within the codes was conducted by two of the co-authors. This involved categorising coded material into themes as a way of assigning meaning to features that occurred across multiple case studies (e.g. categorising types of corroborating evidence typically used in high-scoring versus low-scoring case studies).

Results and discussion

In this section, we integrate findings from the quantitative linguistic study and the qualitative analysis of low-scoring versus high-scoring case studies. The results are discussed under four headings based on the key findings that emerged from both analyses. Taken together, these findings provide the most comprehensive evidence to date of the characteristics of a top-rated (4*) impact case study in REF2014.

Highly-rated case studies provided specific, high-magnitude and well-evidenced articulations of significance and reach

One finding from our qualitative thematic analysis was that 84% of high-scoring cases articulated benefits to specific groups and provided evidence of their significance and reach, compared to 32% of low-scoring cases which typically focused instead on the pathway to impact, for example describing dissemination of research findings and engagement with stakeholders and publics without citing the benefits arising from dissemination or engagement. One way of conceptualising this difference is using the content/process distinction: whereas low-scoring cases tended to focus on the process through which impact was sought (i.e. the pathway used), the high-scoring cases tended to focus on the content of the impact itself (i.e. what change or improvement occurred as a result of the research).

Examples of global reach were evidenced across high-scoring case studies from all panels (including Panel D for Arts and Humanities research), but were less often claimed or evidenced in low-scoring case studies. Where reach was more limited geographically, many high-scoring case studies used context to create robust arguments that their reach was impressive in that context, describing reach for example in social or cultural terms or arguing for the importance of reaching a narrow but hard-to-reach or otherwise important target group.

Table 4 provides examples of evidence from high-scoring cases and low-scoring cases that were used to show significance and reach of impacts in REF2014.

Findings from the quantitative linguistic analysis in Table 5 show how high-scoring impact case studies contained more phrases that specified reach (e.g. “in England and”, “in the US”), compared to low-scoring case studies that used the more generic term “international”, leaving the reader in doubt about the actual reach. They also include more phrases that implicitly specified the significance of the impact (e.g. “the government’s” or “to the House of Commons”), compared to low-scoring cases which provided more generic phrases, such as “policy and practice”, rather than detailing specific policies or practices that had been changed.

The quantitative linguistics analysis also identified a number of words and phrases pertaining to engagement and pathways, which were intended to deliver impact but did not actually specify impact (Table 6 ). A number of phrases contained the word “dissemination”, and there were several words and phrases specifying types of engagement that could be considered more one-way dissemination than consultative or co-productive (cf. Reed et al.’s ( 2018 ) engagement typology), e.g. “the book” and “the event”. The focus on dissemination supports the finding from the qualitative thematic analysis that low-scoring case tended to focus more on pathways or routes than on impact. Although it is not possible to infer this directly from the data, it is possible that this may represent a deeper epistemological position underpinning some case studies, where impact generation was seen as one-way knowledge or technology transfer, and research findings were perceived as something that could be given unchanged to publics and stakeholders through dissemination activities, with the assumption that this would be understood as intended and lead to impact.

It is worth noting that none of the four UK countries appear significantly more often in either high-scoring or low-scoring case studies (outside of the phrase “in England and”). Wales ( n  = 50), Scotland ( n  = 71) and Northern Ireland ( n  = 32) appear slightly more often in high-scoring case studies, but the difference is not significant (England: n  = 162). An additional factor to take into account is that our dataset includes only submissions that are either high-scoring or low-scoring, and the geographical spread of the submitting institutions was not a factor in selecting texts. There was a balanced number of high-scoring and low-scoring case studies in the sample from English, Scottish and Welsh universities, but no guaranteed low-scoring submissions from Northern Irish institutions. The REF2014 guidance made it clear that impacts in each UK country would be evaluated equally in comparison to each other, the UK and other countries. While the quantitative analysis of case studies from our sample only found a statistically significant difference for the phrase “in England and”, this, combined with the slightly higher number of phrases containing the other countries of the UK in high-scoring case studies, might indicate that this panel guidance was implemented as instructed.

Figures 3 – 5 shows which types of impact could be identified in high-scoring or low-scoring case studies, respectively, in the qualitative thematic analysis (based on Reed’s ( 2018 ) typology of impacts). Note that percentages do not add up to 100% because it was possible for each case study to claim more than one type of impact (high-scoring impact case studies described on average 2.8 impacts, compared to an average of 1.8 impacts described by low-scoring case studies) Footnote 3 . Figure 3 shows the number of impacts per type as a percentage of the total number of impacts claimed in high-scoring versus low-scoring case studies. This shows that high-scoring case studies were more likely to claim health/wellbeing and policy impacts, whereas low-scoring case studies were more likely to claim understanding/awareness impacts. Looking at this by Main Panel, over 50% of high-scoring case studies in Main Panel A claimed health/wellbeing, policy and understanding/awareness impacts (Fig. 4 ), whereas over 50% of low-scoring case studies in Main Panel A claimed capacity building impacts (Fig. 5 ). There were relatively high numbers of economic and policy claimed in both high-scoring and low-scoring case studies in Main Panels B and C, respectively, with no impact type dominating strongly in Main Panel D (Figs. 4 and 5 ).

Number of impacts claimed in high- versus low-scoring case studies by impact type.

Percentage of high-scoring case studies that claimed different types of impact.

Percentage of low-scoring case studies that claimed different types of impact.

Highly-rated case studies used distinct features to establish links between research (cause) and impact (effect)

Findings from the quantitative linguistic analysis show that high-scoring case studies were significantly more likely to include attributional phrases like “cited in”, “used to” and “resulting in”, compared to low-scoring case studies (Table 7 provides examples for some of the 12 phrases more frequent in high-scoring case studies). However, there were some attributional phrases that were more likely to be found in low-scoring case studies (e.g. “from the”, “of the research” and “this work has”—total of 9 different phrases).

To investigate this further, all 564 and 601 instances Footnote 4 of attributional phrases in high-scoring and low-scoring case studies, respectively, were analysed to categorise the context in which they were used, to establish the extent to which these phrases in each corpus were being used to establish attribution to impacts. The first word or phrase preceding or succeeding the attributional content was coded. For example, if the attributional content was “used the”, followed by “research to generate impact”, the first word succeeding the attributional content (in this case “research”) was coded rather than the phrase it subsequently led to (“generate impact”). According to a Pearson Chi Square test, high-scoring case studies were significantly more likely to establish attribution to impact than low-scoring cases ( p  < 0.0001, but with a small effect size based on Cramer’s V  = 0.22; bold in Table 8 ). 18% ( n  = 106) of phrases in the low-scoring corpus established attribution to impact, compared to 37% ( n  = 210) in the high-scoring corpus, for example, stating that research, pathway or something else led to impact. Instead, low-scoring case studies were more likely to establish attribution to research (40%; n  = 241) compared to high-scoring cases (28%; n  = 156; p  < 0.0001, but with a small effect size based on Cramer’s V  = 0.135). Both high- and low-scoring case studies were similarly likely to establish attribution to pathways (low: 32%; n  = 194; high: 31% n  = 176).

Moreover, low-scoring case studies were more likely to include ambiguous or uncertain phrases. For example, the phrase “a number of” can be read to imply that it is not known how many instances there were. This occurred in all sections of the impact case studies, for example in the underpinning research section as “The research explores a number of themes” or in the summary or details of the impact section as “The work has also resulted in a number of other national and international impacts”, or “has influenced approaches and practices of a number of partner organisations”. Similarly, “an impact on” could give the impression that the nature of the impact is not known. This phrase occurred only in summary and details of the impact sections, for example, “These activities have had an impact on the professional development”, “the research has had an impact on the legal arguments”, or “there has also been an impact on the work of regional agency”.

In the qualitative thematic analysis, we found that only 50% of low-scoring case studies clearly linked the underpinning research to claimed impacts (compared to 97% of high-scoring cases). This gave the impression of over-claimed impacts in some low-scoring submissions. For example, one case study claimed “significant impacts on [a country’s] society” based on enhancing the security of a new IT system in the department responsible for publishing and archiving legislation. Another claimed “economic impact on a worldwide scale” based on billions of pounds of benefits, calculated using an undisclosed method by an undisclosed evaluator in an unpublished final report by the research team. One case study claimed attribution for impact based on similarities between a prototype developed by the researchers and a product subsequently launched by a major corporation, without any evidence that the product as launched was based on the prototype. Similar assumptions were made in a number of other case studies that appeared to conflate correlation with causation in their attempts to infer attribution between research and impact. Table 9 provides examples of different ways in which links between research and impact were evidenced in the details of the research section.

Table 10 shows how corroborating sources were used to support these claims. 82% of high-scoring case studies compared to 7% of low-scoring cases were identified in the qualitative thematic analysis as having generally high-quality corroborating evidence. In contrast, 11% of high-scoring case studies, compared to 71% of low-scoring cases, were identified as having corroborating evidence that was vague and/or poorly linked to claimed impacts. Looking at only case studies that claim policy impact, 11 out of 26 high-scoring case studies in the sample described both policy and implementation (42%), compared to just 5 out of 29 low-scoring case studies that included both policy and implementation (17%; the remainder described policy impacts only with no evidence of benefits arising from implementation). High- scoring case studies were more likely to cite evidence of impacts rather than just citing evidence pertaining to the pathway (which was more common in low-scoring cases). High-scoring policy case studies also provided evidence pertaining to the pathway, but because they typically also included evidence of policy change, this evidence helped attribute policy impacts to research.

Highly-rated case studies were easy to understand and well written

In preparation for the REF, many universities invested heavily in writing assistance (Coleman, 2019 ) to ensure that impact case studies were “easy to understand and evaluation-friendly” (Watermeyer and Chubb, 2018 ) for the assessment panels, which comprised academics and experts from other sectors (HEFCE, 2011 , p. 6). With this in mind, we investigated readability and style, both in the quantitative linguistic and in the qualitative thematic analysis.

High-scoring impact case studies scored more highly on the Flesch Reading Ease score, a readability measure based on the length of words and sentences. The scores in Table 11 are reported out of 100, with a higher score indicating that a text is easier to read. While the scores reveal a significant difference between 4* and 1*/2* impact case studies, they also indicate that impact case studies are generally on the verge of “graduate” difficulty (Hartley, 2016 , p. 1524). As such our analysis should not be understood as suggesting that these technical documents should be adjusted to the readability of a newspaper article, but they should be maintained at interested and educated non-specialist level.

Interestingly, there were differences between the main panels. Footnote 5 In Social Science and Humanities case studies (Main Panels C and D), high-scoring impact case studies scored significantly higher on reading ease than low-scoring ones. There was no significant difference in Main Panel A between 4* and 1*/2* cases. However, all Main Panel A case studies showed, on average, lower reading ease scores than the low-scoring cases in Main Panels C and D. This means that their authors used longer words and sentences, which may be explained in part by more and longer technical terms needed in Main Panel A disciplines; the difference between high- and low-scoring case studies in Main Panels C and D may be explained by the use of more technical jargon (confirmed in the qualitative analysis).

The Flesch Reading Ease measure assesses the sentence- and word-level, rather than capturing higher-level text-processing difficulty. While this is recognised as a reliable indicator of comparative reading ease, and the underlying measures of sentence-length and word-length are highly correlated with reading speed (Haberlandt and Graesser, 1985 ), Hartley ( 2016 ) is right in his criticism that the tool takes neither the meaning of the words nor the wider text into account. The Coh-Metrix tool (McNamara et al., 2014 ) provides further measures for reading ease based on textual cohesion in these texts compared to a set of general English texts. Of the eight principal component scores computed by the tool, most did not reveal a significant difference between high- and low-scoring case studies or between different Main Panels. Moreover, in most measures, impact case studies overall were fairly homogenous compared to the baseline of general English texts. However, there were significant differences between high- and low-scoring impact case studies in two of the measures: “deep cohesion” and “connectivity” (Table 12 ).

“Deep cohesion” shows whether a text makes causal connections between ideas explicit (e.g. “because”, “so”) or leaves them for the reader to infer. High-scoring case studies had a higher level of deep cohesion compared to general English texts (Graesser et al., 2011 ), while low-scoring case studies tended to sit below the general English average. In addition, Main Panel A case studies (Life Sciences), which received the lowest scores in Flesch Reading Ease, on average scored higher on deep cohesion than case studies in more discursive disciplines (Main Panel C—Social Sciences and Main Panel D—Arts and Humanities). “Connectivity” measures the level of explicit logical connectives (e.g. “and”, “or” and “but”) to show relations in the text. Impact case studies were low in connectivity compared to general English texts, but within each of the Main Panels, high-scoring case studies had more explicit connectivity than low-scoring case studies. This means that Main Panel A case studies, while using on average longer words and sentences as indicated by the Flesch Reading Ease scores, compensated for this by making causal and logical relationships more explicit in the texts. In Main Panels C and D, which on average scored lower on these measures, there was a clearer difference between high- and low-scoring case studies than in Main Panel A, with high-scoring case studies being easier to read.

Linked to this, low-scoring case studies across panels were more likely than high-scoring case studies to contain phrases linked to the research process (suggesting an over-emphasis on the research rather than the impact, and a focus on process over findings or quality; Table 18 ) and filler-phrases (Table 13 ).

High-scoring case studies were more likely to clearly identify individual impacts via subheadings and paragraph headings ( p  < 0.0001, with effect size measure Log Ratio 0.54). The difference is especially pronounced in Main Panel D (Log Ratio 1.53), with a small difference in Main Panel C and no significant difference in Main Panel A. In Units of Assessment combined in Main Panel D, a more discursive academic writing style is prevalent (see e.g. Hyland, 2002 ) using fewer visual/typographical distinctions such as headings. The difference in the number of headings used in case studies from those disciplines suggests that high-scoring case studies showed greater divergence from disciplinary norms than low-scoring case studies. This may have allowed them to adapt the presentation of their research impact to the audience of panel members to a greater extent than low-scoring case studies.

The qualitative thematic analysis of Impact Case Studies indicates that it is not simply the number of subheadings that matters, although this comparison is interesting especially in the context of the larger discrepancy in Main Panel D. Table 14 summarises formatting that was considered helpful and unhelpful from the qualitative analysis.

The observations in Tables 11 – 13 stem from quantitative linguistic analysis, which, while enabling statistical testing, does not show directly the effect of a text on the reader. When conducting the qualitative thematic analysis, we collected examples of formatting and stylistic features from the writing and presentation of high and low-scoring case studies that might have affected clarity of the texts (Tables 14 and 15 ). Specifically, 38% of low-scoring case studies made inappropriate use of adjectives to describe impacts (compared to 20% of high-scoring; Table 16 ). Inappropriate use of adjectives may have given an impression of over-claiming or created a less factual impression than case studies that used adjectives more sparingly to describe impacts. Some included adjectives to describe impacts in testimonial quotes, giving third-party endorsement to the claims rather than using these adjectives directly in the case study text.

Highly-rated case studies were more likely to describe underpinning research findings, rather than research processes

To be eligible, case studies in REF2014 had to be based on underpinning research that was “recognised internationally in terms of originality, significance and rigour” (denoted by a 2* quality profile, HEFCE, 2011 , p. 29). Ineligible case studies were excluded from our sample (i.e. those in the “unclassifiable” quality profile), so all the case studies should have been based on strong research. Once this research quality threshold had been passed, scores were based on the significance and reach of impact, so case studies with higher-rated research should not, in theory, get better scores on the basis of their underpinning research. However, there is evidence that units whose research outputs scored well in REF2014 also performed well on impact (unpublished Research England analysis cited in Hill, 2016 ). This observation only shows that high-quality research and impact were co-located, rather than demonstrating a causal relationship between high-quality research and highly rated impacts. However, our qualitative thematic analysis suggests that weaker descriptions of research (underpinning research was not evaluated directly) may have been more likely to be co-located with lower-rated impacts at the level of individual case studies. We know that the majority of underpinning research in the sample was graded 2* or above (because we excluded unclassifiable case studies from the analysis) but individual ratings for outputs in the underpinning research section are not provided in REF2014. Therefore, the qualitative analysis looked for a range of indicators of strong or weak research in four categories: (i) indicators of publication quality; (ii) quality of funding sources; (iii) narrative descriptions of research quality; and (iv) the extent to which the submitting unit (versus collaborators outside the institution) had contributed to the underpinning research. As would be expected (given that all cases had passed the 2* threshold), only a small minority of cases in the sample gave grounds to doubt the quality of the underpinning research. However, both our qualitative and quantitative analyses identified research-related differences between high- and low-scoring impact case studies.

Based on our qualitative thematic analysis of indicators of research quality, a number of low-scoring cases contained indications that underpinning research may have been weak. This was very rare in high-scoring cases. In the most extreme case, one case study was not able to submit any published research to underpin the impact, relying instead on having secured grant funding and having a manuscript under review. Table 17 describes indicators that underpinning research may have been weaker (presumably closer to the 2* quality threshold for eligibility). It also describes the indications of higher quality research (which were likely to have exceeded the 2* threshold) that were found in the rest of the sample. High-scoring case studies demonstrated the quality of the research using a range of direct and indirect approaches. Direct approaches included the construction of arguments that articulated the originality, significance and rigour of the research in the “underpinning research” section of the case study (sometimes with reference to outputs that were being assessed elsewhere in the exercise to provide a quick and robust check on quality ratings). In addition to this, a wide range of indirect proxies were used to infer quality, including publication venue, funding sources, reviews and awards.

These indicators are of particular interest given the stipulation in REF2021 that case studies must provide evidence of research quality, with the only official guidance suggesting that this is done via the use of indicators. The indicators identified in Table 17 overlap significantly with example indicators proposed by panels in the REF2021 guidance. However, there are also a number of additional indicators, which may be of use for demonstrating the quality of research in REF2021 case studies. In common with proposed REF2021 research quality indicators, many of the indicators in Table 17 are highly context dependent, based on subjective disciplinary norms that are used as short-cuts to assessments of quality by peers within a given context. Funding sources, publication venues and reviews that are considered prestigious in one disciplinary context are often perceived very differently in other disciplinary contexts. While REF2021 does not allow the use of certain indicators (e.g. journal impact factors), no comment is given on the appropriateness of the suggested indicators. While this may be problematic, given that an indicator by definition sign-posts, suggests or indicates by proxy rather than representing the outcome of any rigorous assessment, we make no comment on whether it is appropriate to judge research quality via such proxies. Instead, Table 17 presents a subjective, qualitative identification of indicators of high or low research quality, which were as far as possible considered within the context of disciplinary norms in the Units of Assessments to which the case studies belonged.

The quantitative linguistic analysis also found differences between the high-scoring and low-scoring case studies relating to underpinning research. There were significantly more words and phrases in low-scoring case studies compared to high-scoring cases relating to research outputs (e.g. “the paper”, “peer-reviewed”, “journal of”, “et al”), the research process (e.g. “research project”, “the research”, “his work”, “research team”) and descriptions of research (“relationship between”, “research into”, “the research”) (Table 18 ). The word “research” itself appears frequently in both (high: 91× per 10,000 words; low: 110× per 10,000 words), which is nevertheless a small but significant over-use in the low-scoring case studies (effect size measure log ratio = 0.27, p  < 0.0001).

There are two alternative ways to interpret these findings. First, the qualitative research appears to suggest a link between higher-quality underpinning research and higher impact scores. However, the causal mechanism is not clear. An independent review of REF2014 commissioned by the UK Government (Stern, 2016 ) proposed that underpinning research should only have to meet the 2* threshold for rigour, as the academic significance and novelty of the research is not in theory a necessary precursor to significant and far-reaching impact. However, a number of the indications of weaker research in Table 17 relate to academic significance and originality, and many of the indicators that suggested research exceeded the 2* threshold imply academic significance and originality (e.g. more prestigious publication venues often demand stronger evidence of academic significance and originality in addition to rigour). As such, it may be possible to posit two potential causal mechanisms related to the originality and/or significance of research. First, it may be argued that major new academic breakthroughs may be more likely to lead to impacts, whether directly in the case of applied research that addresses societal challenges in new and important ways leading to breakthrough impacts, or indirectly in the case of major new methodological or theoretical breakthroughs that make new work possible that addresses previously intractable challenges. Second, the highest quality research may have sub-consciously biased reviewers to view associated impacts more favourably. Further research would be necessary to test either mechanism.

However, these mechanisms do not explain the higher frequency of words and phrases relating to research outputs and process in low-scoring case studies. Both high-scoring and low-scoring cases described the underpinning research, and none of the phrases that emerged from the analysis imply higher or lower quality of research. We hypothesised that this may be explained by low-scoring case studies devoting more space to underpinning research at the expense of other sections that may have been more likely to contribute towards scores. Word limits were “indicative”, and the real limit of “four pages” in REF2014 (extended to five pages in REF2021) was operationalised in various way. However, a t -test found no significant difference between the underpinning research word counts (mean of 579 and 537 words in high and low-scoring case studies, respectively; p  = 0.11). Instead, we note that words and phrases relating to research in the low-scoring case studies focused more on descriptions of research outputs and processes rather than descriptions of research findings or the quality of research, as requested in REF2014 guidelines. Given that eligibility evidenced in this section is based on whether the research findings underpin the impacts and the quality of the research (HEFCE, 2011 ), we hypothesise that the focus of low-scoring case studies on research outputs and processes was unnecessary (at best) or replaced or obscured research findings (at worst). This could be conceptualised as another instance of the content/process distinction, whereby high-scoring case studies focused on what the research found and low-scoring case studies focused on the process through which the research was conducted and disseminated. It could be concluded that this tendency may have contributed towards lower scores if unnecessary descriptions of research outputs and process, which would not have contributed towards scores, used up space that could otherwise have been used for material that may have contributed towards scores.

Limitations

These findings may be useful in guiding the construction and writing of case studies for REF2021 but it is important to recognise that our analyses are retrospective, showing examples of what was judged to be ‘good’ and ‘poor’ practice in the authorship of case studies for REF2014. Importantly, the findings of this study should not be used to infer a causal relationship between the linguistic features we have identified and the judgements of the REF evaluation panel. Our quantitative analysis has identified similarities and differences in their linguistic features, but there are undoubtedly a range of considerations taken into account by evaluation panels. It is also not possible to anticipate how REF2021 panels will interpret guidance and evaluate case studies, and there is already evidence that practice is changing significantly across the sector. This shift in expectations regarding impact is especially likely to be the case in research concerned with public policy, which are increasingly including policy implementation as well as design in their requirements, and research involving public engagement, which is increasingly being expected to provide longitudinal evidence of benefits and provide evidence of cause and effect. We are unable to say anything conclusive from our sample about case studies that focused primarily on public engagement and pedagogy because neither of these types of impact were common enough in either the high-scoring or low-scoring sample to infer reliable findings. While this is the largest sample of known high-scoring versus low-scoring case studies ever analysed, it is important to note that this represents <3% of the total case studies submitted to REF2014. Although the number of case studies was fairly evenly balanced between Main Panels in the thematic analysis, the sample only included a selection of Units of Assessment from each Main Panel, where sufficient numbers of high and low-scoring cases could be identified (14 and 20 out of 36 Units of Assessment in the qualitative and quantitative studies, respectively). As such, caution should be taken when generalising from these findings.

This paper provides empirical insights into the linguistic differences in high-scoring and low-scoring impact case studies in REF2014. Higher-scoring case studies were more likely to have articulated evidence of significant and far-reaching impacts (rather than just presenting the activities used to reach intended future impacts), and they articulated clear evidence of causal links between the underpinning research and claimed impacts. While a cause and effect relationship between linguistic features, styles and the panel’s evaluation cannot be claimed, we have provided a granularity of analysis that shows how high-scoring versus low-scoring case studies attempted to meet REF criteria. Knowledge of these features may provide useful lessons for future case study authors, submitting institutions and others developing impact assessments internationally. Specifically, we show that high-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach, compared to low-scoring cases, which were more likely to provide less specific and lower-magnitude articulations of significance and reach. Lower-scoring case studies were more likely to focus on pathways to impact rather than articulating clear impact claims, with a particular focus on one-way modes of knowledge transfer. High-scoring case studies were more likely to provide clear links between underpinning research and impacts, supported by high-quality corroborating evidence, compared to low-scoring cases that often had missing links between research and impact and were more likely to be underpinned by corroborating evidence that was vague and/or not clearly linked to impact claims. Linked to this, high-scoring case studies were more likely to contain attributional phrases, and these phrases were more likely to attribute research and/or pathways to impact, compared to low-scoring cases, which contained fewer attributional phrases, which were more likely to provide attribution to pathways rather than impact. Furthermore, there is evidence that high-scoring case studies had more explicit causal connections between ideas and more logical connective words (and, or, but) than low-scoring cases.

However, in addition to the explicit REF2014 rules, which appear to have been enacted effectively by sub-panels, there is evidence that implicit rules, particularly linked to written style, may also have played a role. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than might be normally expected in academic writing (cf. e.g. Vold, 2006 ; Yang et al., 2015 ). Low-scoring case studies were more likely to contain filler phrases that could be described as “academese” (Biber and Gray, 2019 , p. 1), more likely to use unsubstantiated or vague adjectives to describe impacts, and were less likely to signpost readers to key points using sub-headings and paragraph headings. High-scoring case studies in two Main Panels (out of the three that could be analysed in this way) were significantly easier to read, although both high- and low-scoring case studies tended to be of “graduate” (Hartley, 2016 ) difficulty.

These findings suggest that aspects of written style may have contributed towards or compromised the scores of some case studies in REF2014, in line with previous research emphasising the role of implicit and subjective factors in determining the outcomes of impact evaluation (Derrick, 2018 ; Watermeyer and Chubb, 2018 ). If this were the case, it may raise questions about whether case studies are an appropriate way to evaluate impact. However, metric-based approaches have many other limitations and are widely regarded as inappropriate for evaluating societal impact (Bornmann et al., 2018 ; Pollitt et al., 2016 ; Ravenscroft et al., 2017 ; Wilsdon et al., 2015 ). Comparing research output evaluation systems across different countries, Sivertsen ( 2017 ) presents the peer-review-based UK REF as “best practice” compared to the metrics-based systems elsewhere. Comparing the evaluation of impact in the UK to impact evaluations in USA, the Netherlands, Italy and Finland, Derrick ( 2019 ) describes REF2014 and REF2021 as “the world’s most developed agenda for evaluating the wider benefits of research and its success has influenced the way many other countries define and approach the assessment of impact”.

We cannot be certain about the extent to which linguistic features or style shaped the judgement of REF evaluators, nor can such influences easily be identified or even consciously recognised when they are at work (cf. research on sub-conscious bias and tacit knowledge; the idea that “we know more than we can say”—Polanyi, 1958 cited in Goodman, 2003 , p. 142). Nonetheless, we hope that the granularity of our findings proves useful in informing decisions about presenting case studies, both for case study authors (in REF2021 and other research impact evaluations around the world) and those designing such evaluation processes. In publishing this evidence, we hope to create a more “level playing field” between institutions with and without significant resources available to hire dedicated staff or consultants to help write their impact case studies.

Data availability

The dataset analysed during the current study corresponds to the publicly available impact case studies defined through the method explained in Section “Research design and sample” and Table 2 . A full list of case studies included can be obtained from the corresponding author upon request.

https://impact.ref.ac.uk/casestudies/search1.aspx

For Main Panel B, only six high-scoring and two low-scoring case studies are clearly identifiable and available to the public (cf. Fig. 1 ). The Main Panel B dataset is therefore too small for separate statistical analysis, and no generalisations should be made on the basis of only one high-scoring and one low-scoring submission.

However, in the qualitative analysis, there were a similar number of high-scoring case studies that were considered to have reached this score due to a clear focus on one single, highly impressive impact, compared to those that were singled out for their impressive range of different impacts.

Note that there were more instances of the smaller number of attributional phrases in the low-scoring corpus.

For Main Panel B, only six high-scoring and two low-scoring case studies are clearly identifiable and available to the public. The Main Panel B dataset is therefore too small for separate statistical analysis, and no generalisations should be made on the basis of only one high-scoring and one low-scoring submission.

Anthony L (2014) AntConc, 3.4.4 edn. Waseda University, Tokyo

Google Scholar  

Auerbach CF, Silverstein LB (2003) Qualitative data: an introduction to coding and analyzing data in qualitative research. New York University Press, New York, NY

Back L (2015) On the side of the powerful: the ‘impact agenda’ and sociology in public. https://www.thesociologicalreview.com/on-the-side-of-the-powerful-the-impact-agenda-sociology-in-public/ . Last Accessed 24 Jan 2020

Biber D, Gray B (2019) Grammatical complexity in academic English: linguistic change in writing. Cambridge University Press, Cambridge

Bornmann L, Haunschild R, Adams J (2018) Do altmetrics assess societal impact in the same way as case studies? An empirical analysis testing the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF). J Informetr 13(1):325–340

Article   Google Scholar  

Bozeman B, Youtie J (2017) Socio-economic impacts and public value of government-funded research: lessons from four US National Science Foundation initiatives. Res Policy 46(8):1387–1398

Braun V, Clarke V (2006) Using thematic analysis in psychology. Quale Res Psychol 3(2):77–101

Brezina V (2018) Statistics in corpus linguistics: a practical guide. Cambridge University Press, Cambridge

Book   Google Scholar  

Chubb J (2017) Instrumentalism and epistemic responsibility: researchers and the impact agenda in the UK and Australia. University of York

Chubb J, Watermeyer R (2017) Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Stud High Educ 42(2):2360–2372

Chubb J, Reed MS (2017) Epistemic responsibility as an edifying force in academic research: investigating the moral challenges and opportunities of an impact agenda in the UK and Australia. Palgrave Commun 3:20

Chubb J, Reed MS (2018) The politics of research impact: academic perceptions of the implications for research funding, motivation and quality. Br Politics 13(3):295–311

Clark WC et al. (2016) Crafting usable knowledge for sustainable development. Proc Natl Acad Sci USA 113(17):4570–4578

Article   ADS   CAS   PubMed   Google Scholar  

Coleman I (2019) The evolution of impact support in UK universities. Cactus Communications Pvt. Ltd

Derrick G (2018) The evaluators’ eye: impact assessment and academic peer review. Palgrave Macmillan

Derrick G (2019) Cultural impact of the impact agenda: implications for social sciences and humanities (SSH) research. In: Bueno D et al. (eds.), Higher education in the world, vol. 7. Humanities and higher education: synergies between science, technology and humanities. Global University Network for Innovation (GUNi)

Derrick G et al. (2018) Towards characterising negative impact: introducing Grimpact. In: Proceedings of the 23rd international conference on Science and Technology Indicators (STI 2018). Centre for Science and Technology Studies (CWTS), Leiden, The Netherlands

Esfandiari R, Barbary F (2017) A contrastive corpus-driven study of lexical bundles between English writers and Persian writers in psychology research articles. J Engl Academic Purp 29:21–42

Goodman CP (2003) The tacit dimension. Polanyiana 2(1):133–157

Graesser AC, McNamara DS, Kulikowich J (2011) Coh-Metrix: providing multi-level analyses of text characteristics. Educ Res 40:223–234

Haberlandt KF, Graesser AC (1985) Component processes in text comprehension and some of their interactions. J Exp Psychol: Gen 114(3):357–374

Hardie A (2014) Statistical identification of keywords, lockwords and collocations as a two-step procedure. ICAME 35, Nottingham

Hartley J (2016) Is time up for the Flesch measure of reading ease? Scientometrics 107(3):1523–1526

HEFCE (2011) Assessment framework and guidance on submissions. Ref. 02.2011

Hill S (2016) Assessing (for) impact: future assessment of the societal impact of research. Palgrave Commun 2:16073

Hyland K (2002) Directives: argument and engagement in academic writing. Appl Linguist 23(2):215–238

Hyland K (2008) As can be seen: lexical bundles and disciplinary variation. Engl Specif Purp 27(1):4–21

Lemos MC (2015) Usable climate knowledge for adaptive and co-managed water governance. Curr Opin Environ Sustain 12:48–52

MacDonald R (2017) “Impact”, research and slaying Zombies: the pressures and possibilities of the REF. Int J Sociol Soc Policy 37(11–12):696–710

Mårtensson P et al. (2016) Evaluating research: a multidisciplinary approach to assessing research practice and quality. Res Policy 45(3):593–603

McEnery T, Xiao R, Tono Y (2006) Corpus-based language studies: an advanced resource book. Routledge, Abingdon

McNamara DS et al. (2014) Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press, New York, NY

National Science Foundation (2014) Perspectives on broader impacts

Nesta (2018) Seven principles for public engagement in research and innovation policymaking. https://www.nesta.org.uk/documents/955/Seven_principles_HlLwdow.pdf . Last Accessed 12 Dec 2019

Oancea A (2010) The BERA/UCET review of the impacts of RAE 2008 on education research in UK higher education institutions. ERA/UCET, Macclesfield

Oancea (2014) Research assessment as governance technology in the United Kingdom: findings from a survey of RAE 2008 impacts. Z Erziehungswis 17(S6):83–110

Olssen M, Peters MA (2005) Neoliberalism, higher education and the knowledge economy: from the free market to knowledge capitalism. J Educ Policy 20(3):313–345

Pidd M, Broadbent J (2015) Business and management studies in the 2014 Research Excellence Framework. Br J Manag 26:569–581

Pollitt A et al. (2016) Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers. BMJ Open 6(8):e010916

Article   PubMed   PubMed Central   Google Scholar  

Rau H, Goggins G, Fahy F (2018) From invisibility to impact: recognising the scientific and societal relevance of interdisciplinary sustainability research. Res Policy 47(1):266–276

Ravenscroft J et al. (2017) Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements. PLoS ONE 12(3):e0173152

Article   PubMed   PubMed Central   CAS   Google Scholar  

Rayson P, Garside R (2000) Comparing corpora using frequency profiling, Workshop on Comparing Corpora, held in conjunction with the 38th annual meeting of the Association for Computational Linguistics (ACL 2000), Hong Kong, pp. 1–6

Rayson P, Berridge D, Francis B (2004) Extending the Cochran rule for the comparison of word frequencies between corpora. In: Purnelle G, Fairon C, Dister A (eds.), Le poids des mots: Proceedings of the 7th international conference on statistical analysis of textual data (JADT 2004) (II). Presses universitaires de Louvain, Louvain-la-Neuve, Belgium, pp. 926–936

Reed MS (2018) The research impact handbook, 2nd edn. Fast Track Impact, Huntly, Aberdeenshire

Reed MS (2019) Book review: new book calls for civil disobedience to fight “dehumanising” impact agenda. Fast Track Impact

Reed MS et al. (under review) Evaluating research impact: a methodological framework. Res Policy

Rhoads R, Torres CA (2005) The University, State, and Market: The Political Economy of Globalization in the Americas. Stanford University Press, Stanford

Saldana J (2009) The Coding Manual for Qualitative Researchers. Sage, Thousand Oaks

Scott M (1997) PC analysis of key words—and key key words. System 25(2):233–245

Sivertsen G (2017) Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective. Palgrave Commun 3:17078

Smith S, Ward V, House A (2011) ‘Impact’ in the proposals for the UK’s Research Excellence Framework: shifting the boundaries of academic autonomy. Res Policy 40(10):1369–1379

Stern LN (2016) Building on success and learning from experience: an independent review of the Research Excellence Framework

Tsey K et al. (2016) Evaluating research impact: the development of a research for impact tool. Front Public Health 4:160

Vold ET (2006) Epistemic modality markers in research articles: a cross-linguistic and cross-disciplinary study. Int J Appl Linguist 16(1):61–87

Warry P (2006) Increasing the economic impact of the Research Councils (the Warry report). Research Council UK, Swindon

Watermeyer R (2019) Competitive accountability in academic life: the struggle for social impact and public legitimacy. Edward Elgar, Cheltenham

Watermeyer R, Hedgecoe A (2016) ‘Selling ‘impact’: peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. J Educ Policy 31:651–665

Watermeyer R, Chubb J (2018) Evaluating ‘impact’ in the UK’s Research Excellence Framework (REF): liminality, looseness and new modalities of scholarly distinction. Stud Higher Educ 44(9):1–13

Weinstein N et al. (2019) The real-time REF review: a pilot study to examine the feasibility of a longitudinal evaluation of perceptions and attitudes towards REF 2021

Wilsdon J et al. (2015) Metric tide: report of the independent review of the role of metrics in research assessment and management

Yang A, Zheng S, Ge G (2015) Epistemic modality in English-medium medical research articles: a systemic functional perspective. Engl Specif Purp 38:1–10

Download references

Acknowledgements

Thanks to Dr. Adam Mearns, School of English Literature, Language & Linguistics at Newcastle University for help with statistics and wider input to research design as a co-supervisor on the Ph.D. research upon which this article is based.

Author information

Authors and affiliations.

Newcastle University, Newcastle, UK

Bella Reichard, Mark S Reed & Andrea Whittle

University of York, York, UK

University of Leeds, Leeds, UK

Northumbria University, Newcastle, UK

Lucy Jowett & Alisha Peart

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mark S Reed .

Ethics declarations

Competing interests.

MR is CEO of Fast Track Impact Ltd, providing impact training to researchers internationally. JC worked with Research England as part of the Real-Time REF Review in parallel with the writing of this article. BR offers consultancy services reviewing REF impact case studies.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Reichard, B., Reed, M.S., Chubb, J. et al. Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Commun 6 , 31 (2020). https://doi.org/10.1057/s41599-020-0394-7

Download citation

Received : 10 July 2019

Accepted : 09 January 2020

Published : 25 February 2020

DOI : https://doi.org/10.1057/s41599-020-0394-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Case Studies and Comparative Analysis

A case study is an in-depth, detailed examination of a particular case (or cases) within a real-world context. Generally, a case study can highlight an individual, group, organization, event, belief system, or action. A case study does not necessarily have to be one observation, but may include many observations (one or multiple individuals and entities across multiple time periods, all within the same case study). Comparative analysis, on the other hand, is a method that compares two or more of anything (documents, data sets, political systems etc.) – though sometimes a form of comparative analysis is used to compare two or more cases studies, demonstrating the links between these two methods. Text adapted from Wikipedia .

The resources below have been curated by the E-International Relations team. You can find more resources on our methods homepage .

Types of Case Study by Graham R Gibbs (YouTube)

Comparative Politics: What and how to compare by Patrick Theiner (YouTube)

Comparative Case Studies – Avoid bias, ensure case studies represent population, not anomalies by Delwyn Goodrick (YouTube)

Websites Writing a Case Study by PLNU (Website).

Qualitative Comparative Analysis by Wendy Olsen (Website).

Further Reading on E-International Relations

• Quantitative Analysis Tools
• Qualitative, Quantitative and Mixed Methods Approaches
• Multimedia and Textual Analysis
• Positivism, Post-Positivism and Interpretivism
• Research Ethics
• Ethnography in International Relations

Please Consider Donating

Before you download your free e-book, please consider donating to support open access publishing.

E-IR is an independent non-profit publisher run by an all volunteer team. Your donations allow us to invest in new open access titles and pay our bandwidth bills to ensure we keep our existing titles free to view. Any amount, in any currency, is appreciated. Many thanks!

Donations are voluntary and not required to download the e-book - your link to download is below.

This website may not work correctly because your browser is out of date. Please update your browser .

Comparative case studies

Comparative case studies can be useful to check variation in program implementation. 

Comparative case studies are another way of checking if results match the program theory. Each context and environment is different. The comparative case study can help the evaluator check whether the program theory holds for each different context and environment. If implementation differs, the reasons and results can be recorded. The opposite is also true, similar patterns across sites can increase the confidence in results.

Evaluators used a comparative case study method for the National Cancer Institute’s (NCI’s) Community Cancer Centers Program (NCCCP). The aim of this program was to expand cancer research and deliver the latest, most advanced cancer care to a greater number of Americans in the communities in which they live via community hospitals. The evaluation examined each of the program components (listed below) at each program site. The six program components were:

• increasing capacity to collect biospecimens per NCI’s best practices;
• enhancing clinical trials (CT) research;
• reducing disparities across the cancer continuum;
• improving the use of information technology (IT) and electronic medical records (EMRs) to support improvements in research and care delivery;
• improving quality of cancer care and related areas, such as the development of integrated, multidisciplinary care teams; and
• placing greater emphasis on survivorship and palliative care.

The evaluators use of this method assisted in providing recommendations at the program level as well as to each specific program site.

Advice for choosing this method

• Compare cases with the same outcome but differences in an intervention (known as MDD, most different design)
• Compare cases with the same intervention but differences in outcomes (known as MSD, most similar design)

Advice for using this method

• Consider the variables of each case, and which cases can be matched for comparison.
• Provide the evaluator with as much detail and background on each case as possible. Provide advice on possible criteria for matching.

National Cancer Institute, (2007).  NCI Community Cancer Centers Program Evaluation (NCCCP) . Retrieved from website: https://digitalscholarship.unlv.edu/jhdrp/vol8/iss1/4/

Expand to view all resources related to 'Comparative case studies'

• Broadening the range of designs and methods for impact evaluations

'Comparative case studies' is referenced in:

Framework/guide.

• Rainbow Framework :  Check the results are consistent with causal contribution
• Sustained and Emerging Impacts Evaluation (SEIE)

Back to top

© 2022 BetterEvaluation. All right reserved.

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 10 methods for comparative studies.

Francis Lau and Anne Holbrook .

10.1. Introduction

In eHealth evaluation, comparative studies aim to find out whether group differences in eHealth system adoption make a difference in important outcomes. These groups may differ in their composition, the type of system in use, and the setting where they work over a given time duration. The comparisons are to determine whether significant differences exist for some predefined measures between these groups, while controlling for as many of the conditions as possible such as the composition, system, setting and duration.

According to the typology by Friedman and Wyatt (2006) , comparative studies take on an objective view where events such as the use and effect of an eHealth system can be defined, measured and compared through a set of variables to prove or disprove a hypothesis. For comparative studies, the design options are experimental versus observational and prospective versus retro­­spective. The quality of eHealth comparative studies depends on such aspects of methodological design as the choice of variables, sample size, sources of bias, confounders, and adherence to quality and reporting guidelines.

In this chapter we focus on experimental studies as one type of comparative study and their methodological considerations that have been reported in the eHealth literature. Also included are three case examples to show how these studies are done.

10.2. Types of Comparative Studies

Experimental studies are one type of comparative study where a sample of participants is identified and assigned to different conditions for a given time duration, then compared for differences. An example is a hospital with two care units where one is assigned a cpoe system to process medication orders electronically while the other continues its usual practice without a cpoe . The participants in the unit assigned to the cpoe are called the intervention group and those assigned to usual practice are the control group. The comparison can be performance or outcome focused, such as the ratio of correct orders processed or the occurrence of adverse drug events in the two groups during the given time period. Experimental studies can take on a randomized or non-randomized design. These are described below.

10.2.1. Randomized Experiments

In a randomized design, the participants are randomly assigned to two or more groups using a known randomization technique such as a random number table. The design is prospective in nature since the groups are assigned concurrently, after which the intervention is applied then measured and compared. Three types of experimental designs seen in eHealth evaluation are described below ( Friedman & Wyatt, 2006 ; Zwarenstein & Treweek, 2009 ).

Randomized controlled trials ( rct s) – In rct s participants are randomly assigned to an intervention or a control group. The randomization can occur at the patient, provider or organization level, which is known as the unit of allocation. For instance, at the patient level one can randomly assign half of the patients to receive emr reminders while the other half do not. At the provider level, one can assign half of the providers to receive the reminders while the other half continues with their usual practice. At the organization level, such as a multisite hospital, one can randomly assign emr reminders to some of the sites but not others. Cluster randomized controlled trials ( crct s) – In crct s, clusters of participants are randomized rather than by individual participant since they are found in naturally occurring groups such as living in the same communities. For instance, clinics in one city may be randomized as a cluster to receive emr reminders while clinics in another city continue their usual practice. Pragmatic trials – Unlike rct s that seek to find out if an intervention such as a cpoe system works under ideal conditions, pragmatic trials are designed to find out if the intervention works under usual conditions. The goal is to make the design and findings relevant to and practical for decision-makers to apply in usual settings. As such, pragmatic trials have few criteria for selecting study participants, flexibility in implementing the intervention, usual practice as the comparator, the same compliance and follow-up intensity as usual practice, and outcomes that are relevant to decision-makers.

10.2.2. Non-randomized Experiments

Non-randomized design is used when it is neither feasible nor ethical to randomize participants into groups for comparison. It is sometimes referred to as a quasi-experimental design. The design can involve the use of prospective or retrospective data from the same or different participants as the control group. Three types of non-randomized designs are described below ( Harris et al., 2006 ).

Intervention group only with pretest and post-test design – This design involves only one group where a pretest or baseline measure is taken as the control period, the intervention is implemented, and a post-test measure is taken as the intervention period for comparison. For example, one can compare the rates of medication errors before and after the implementation of a cpoe system in a hospital. To increase study quality, one can add a second pretest period to decrease the probability that the pretest and post-test difference is due to chance, such as an unusually low medication error rate in the first pretest period. Other ways to increase study quality include adding an unrelated outcome such as patient case-mix that should not be affected, removing the intervention to see if the difference remains, and removing then re-implementing the intervention to see if the differences vary accordingly. Intervention and control groups with post-test design – This design involves two groups where the intervention is implemented in one group and compared with a second group without the intervention, based on a post-test measure from both groups. For example, one can implement a cpoe system in one care unit as the intervention group with a second unit as the control group and compare the post-test medication error rates in both units over six months. To increase study quality, one can add one or more pretest periods to both groups, or implement the intervention to the control group at a later time to measure for similar but delayed effects. Interrupted time series ( its ) design – In its design, multiple measures are taken from one group in equal time intervals, interrupted by the implementation of the intervention. The multiple pretest and post-test measures decrease the probability that the differences detected are due to chance or unrelated effects. An example is to take six consecutive monthly medication error rates as the pretest measures, implement the cpoe system, then take another six consecutive monthly medication error rates as the post-test measures for comparison in error rate differences over 12 months. To increase study quality, one may add a concurrent control group for comparison to be more convinced that the intervention produced the change.

10.3. Methodological Considerations

The quality of comparative studies is dependent on their internal and external validity. Internal validity refers to the extent to which conclusions can be drawn correctly from the study setting, participants, intervention, measures, analysis and interpretations. External validity refers to the extent to which the conclusions can be generalized to other settings. The major factors that influence validity are described below.

10.3.1. Choice of Variables

Variables are specific measurable features that can influence validity. In comparative studies, the choice of dependent and independent variables and whether they are categorical and/or continuous in values can affect the type of questions, study design and analysis to be considered. These are described below ( Friedman & Wyatt, 2006 ).

Dependent variables – This refers to outcomes of interest; they are also known as outcome variables. An example is the rate of medication errors as an outcome in determining whether cpoe can improve patient safety. Independent variables – This refers to variables that can explain the measured values of the dependent variables. For instance, the characteristics of the setting, participants and intervention can influence the effects of cpoe . Categorical variables – This refers to variables with measured values in discrete categories or levels. Examples are the type of providers (e.g., nurses, physicians and pharmacists), the presence or absence of a disease, and pain scale (e.g., 0 to 10 in increments of 1). Categorical variables are analyzed using non-parametric methods such as chi-square and odds ratio. Continuous variables – This refers to variables that can take on infinite values within an interval limited only by the desired precision. Examples are blood pressure, heart rate and body temperature. Continuous variables are analyzed using parametric methods such as t -test, analysis of variance or multiple regression.

10.3.2. Sample Size

Sample size is the number of participants to include in a study. It can refer to patients, providers or organizations depending on how the unit of allocation is defined. There are four parts to calculating sample size. They are described below ( Noordzij et al., 2010 ).

Significance level – This refers to the probability that a positive finding is due to chance alone. It is usually set at 0.05, which means having a less than 5% chance of drawing a false positive conclusion. Power – This refers to the ability to detect the true effect based on a sample from the population. It is usually set at 0.8, which means having at least an 80% chance of drawing a correct conclusion. Effect size – This refers to the minimal clinically relevant difference that can be detected between comparison groups. For continuous variables, the effect is a numerical value such as a 10-kilogram weight difference between two groups. For categorical variables, it is a percentage such as a 10% difference in medication error rates. Variability – This refers to the population variance of the outcome of interest, which is often unknown and is estimated by way of standard deviation ( sd ) from pilot or previous studies for continuous outcome.

Sample Size Equations for Comparing Two Groups with Continuous and Categorical Outcome Variables.

An example of sample size calculation for an rct to examine the effect of cds on improving systolic blood pressure of hypertensive patients is provided in the Appendix. Refer to the Biomath website from Columbia University (n.d.) for a simple Web-based sample size / power calculator.

10.3.3. Sources of Bias

There are five common sources of biases in comparative studies. They are selection, performance, detection, attrition and reporting biases ( Higgins & Green, 2011 ). These biases, and the ways to minimize them, are described below ( Vervloet et al., 2012 ).

Selection or allocation bias – This refers to differences between the composition of comparison groups in terms of the response to the intervention. An example is having sicker or older patients in the control group than those in the intervention group when evaluating the effect of emr reminders. To reduce selection bias, one can apply randomization and concealment when assigning participants to groups and ensure their compositions are comparable at baseline. Performance bias – This refers to differences between groups in the care they received, aside from the intervention being evaluated. An example is the different ways by which reminders are triggered and used within and across groups such as electronic, paper and phone reminders for patients and providers. To reduce performance bias, one may standardize the intervention and blind participants from knowing whether an intervention was received and which intervention was received. Detection or measurement bias – This refers to differences between groups in how outcomes are determined. An example is where outcome assessors pay more attention to outcomes of patients known to be in the intervention group. To reduce detection bias, one may blind assessors from participants when measuring outcomes and ensure the same timing for assessment across groups. Attrition bias – This refers to differences between groups in ways that participants are withdrawn from the study. An example is the low rate of participant response in the intervention group despite having received reminders for follow-up care. To reduce attrition bias, one needs to acknowledge the dropout rate and analyze data according to an intent-to-treat principle (i.e., include data from those who dropped out in the analysis). Reporting bias – This refers to differences between reported and unreported findings. Examples include biases in publication, time lag, citation, language and outcome reporting depending on the nature and direction of the results. To reduce reporting bias, one may make the study protocol available with all pre-specified outcomes and report all expected outcomes in published results.

10.3.4. Confounders

Confounders are factors other than the intervention of interest that can distort the effect because they are associated with both the intervention and the outcome. For instance, in a study to demonstrate whether the adoption of a medication order entry system led to lower medication costs, there can be a number of potential confounders that can affect the outcome. These may include severity of illness of the patients, provider knowledge and experience with the system, and hospital policy on prescribing medications ( Harris et al., 2006 ). Another example is the evaluation of the effect of an antibiotic reminder system on the rate of post-operative deep venous thromboses ( dvt s). The confounders can be general improvements in clinical practice during the study such as prescribing patterns and post-operative care that are not related to the reminders ( Friedman & Wyatt, 2006 ).

To control for confounding effects, one may consider the use of matching, stratification and modelling. Matching involves the selection of similar groups with respect to their composition and behaviours. Stratification involves the division of participants into subgroups by selected variables, such as comorbidity index to control for severity of illness. Modelling involves the use of statistical techniques such as multiple regression to adjust for the effects of specific variables such as age, sex and/or severity of illness ( Higgins & Green, 2011 ).

10.3.5. Guidelines on Quality and Reporting

There are guidelines on the quality and reporting of comparative studies. The grade (Grading of Recommendations Assessment, Development and Evaluation) guidelines provide explicit criteria for rating the quality of studies in randomized trials and observational studies ( Guyatt et al., 2011 ). The extended consort (Consolidated Standards of Reporting Trials) Statements for non-pharmacologic trials ( Boutron, Moher, Altman, Schulz, & Ravaud, 2008 ), pragmatic trials ( Zwarestein et al., 2008 ), and eHealth interventions ( Baker et al., 2010 ) provide reporting guidelines for randomized trials.

The grade guidelines offer a system of rating quality of evidence in systematic reviews and guidelines. In this approach, to support estimates of intervention effects rct s start as high-quality evidence and observational studies as low-quality evidence. For each outcome in a study, five factors may rate down the quality of evidence. The final quality of evidence for each outcome would fall into one of high, moderate, low, and very low quality. These factors are listed below (for more details on the rating system, refer to Guyatt et al., 2011 ).

Design limitations – For rct s they cover the lack of allocation concealment, lack of blinding, large loss to follow-up, trial stopped early or selective outcome reporting. Inconsistency of results – Variations in outcomes due to unexplained heterogeneity. An example is the unexpected variation of effects across subgroups of patients by severity of illness in the use of preventive care reminders. Indirectness of evidence – Reliance on indirect comparisons due to restrictions in study populations, intervention, comparator or outcomes. An example is the 30-day readmission rate as a surrogate outcome for quality of computer-supported emergency care in hospitals. Imprecision of results – Studies with small sample size and few events typically would have wide confidence intervals and are considered of low quality. Publication bias – The selective reporting of results at the individual study level is already covered under design limitations, but is included here for completeness as it is relevant when rating quality of evidence across studies in systematic reviews.

The original consort Statement has 22 checklist items for reporting rct s. For non-pharmacologic trials extensions have been made to 11 items. For pragmatic trials extensions have been made to eight items. These items are listed below. For further details, readers can refer to Boutron and colleagues (2008) and the consort website ( consort , n.d.).

Title and abstract – one item on the means of randomization used. Introduction – one item on background, rationale, and problem addressed by the intervention. Methods – 10 items on participants, interventions, objectives, outcomes, sample size, randomization (sequence generation, allocation concealment, implementation), blinding (masking), and statistical methods. Results – seven items on participant flow, recruitment, baseline data, numbers analyzed, outcomes and estimation, ancillary analyses, adverse events. Discussion – three items on interpretation, generalizability, overall evidence.

The consort Statement for eHealth interventions describes the relevance of the consort recommendations to the design and reporting of eHealth studies with an emphasis on Internet-based interventions for direct use by patients, such as online health information resources, decision aides and phr s. Of particular importance is the need to clearly define the intervention components, their role in the overall care process, target population, implementation process, primary and secondary outcomes, denominators for outcome analyses, and real world potential (for details refer to Baker et al., 2010 ).

10.4. Case Examples

10.4.1. pragmatic rct in vascular risk decision support.

Holbrook and colleagues (2011) conducted a pragmatic rct to examine the effects of a cds intervention on vascular care and outcomes for older adults. The study is summarized below.

Setting – Community-based primary care practices with emr s in one Canadian province. Participants – English-speaking patients 55 years of age or older with diagnosed vascular disease, no cognitive impairment and not living in a nursing home, who had a provider visit in the past 12 months. Intervention – A Web-based individualized vascular tracking and advice cds system for eight top vascular risk factors and two diabetic risk factors, for use by both providers and patients and their families. Providers and staff could update the patient’s profile at any time and the cds algorithm ran nightly to update recommendations and colour highlighting used in the tracker interface. Intervention patients had Web access to the tracker, a print version mailed to them prior to the visit, and telephone support on advice. Design – Pragmatic, one-year, two-arm, multicentre rct , with randomization upon patient consent by phone, using an allocation-concealed online program. Randomization was by patient with stratification by provider using a block size of six. Trained reviewers examined emr data and conducted patient telephone interviews to collect risk factors, vascular history, and vascular events. Providers completed questionnaires on the intervention at study end. Patients had final 12-month lab checks on urine albumin, low-density lipoprotein cholesterol, and A1c levels. Outcomes – Primary outcome was based on change in process composite score ( pcs ) computed as the sum of frequency-weighted process score for each of the eight main risk factors with a maximum score of 27. The process was considered met if a risk factor had been checked. pcs was measured at baseline and study end with the difference as the individual primary outcome scores. The main secondary outcome was a clinical composite score ( ccs ) based on the same eight risk factors compared in two ways: a comparison of the mean number of clinical variables on target and the percentage of patients with improvement between the two groups. Other secondary outcomes were actual vascular event rates, individual pcs and ccs components, ratings of usability, continuity of care, patient ability to manage vascular risk, and quality of life using the EuroQol five dimensions questionnaire ( eq-5D) . Analysis – 1,100 patients were needed to achieve 90% power in detecting a one-point pcs difference between groups with a standard deviation of five points, two-tailed t -test for mean difference at 5% significance level, and a withdrawal rate of 10%. The pcs , ccs and eq-5D scores were analyzed using a generalized estimating equation accounting for clustering within providers. Descriptive statistics and χ2 tests or exact tests were done with other outcomes. Findings – 1,102 patients and 49 providers enrolled in the study. The intervention group with 545 patients had significant pcs improvement with a difference of 4.70 ( p < .001) on a 27-point scale. The intervention group also had significantly higher odds of rating improvements in their continuity of care (4.178, p < .001) and ability to improve their vascular health (3.07, p < .001). There was no significant change in vascular events, clinical variables and quality of life. Overall the cds intervention led to reduced vascular risks but not to improved clinical outcomes in a one-year follow-up.

10.4.2. Non-randomized Experiment in Antibiotic Prescribing in Primary Care

Mainous, Lambourne, and Nietert (2013) conducted a prospective non-randomized trial to examine the impact of a cds system on antibiotic prescribing for acute respiratory infections ( ari s) in primary care. The study is summarized below.

Setting – A primary care research network in the United States whose members use a common emr and pool data quarterly for quality improvement and research studies. Participants – An intervention group with nine practices across nine states, and a control group with 61 practices. Intervention – Point-of-care cds tool as customizable progress note templates based on existing emr features. cds recommendations reflect Centre for Disease Control and Prevention ( cdc ) guidelines based on a patient’s predominant presenting symptoms and age. cds was used to assist in ari diagnosis, prompt antibiotic use, record diagnosis and treatment decisions, and access printable patient and provider education resources from the cdc . Design – The intervention group received a multi-method intervention to facilitate provider cds adoption that included quarterly audit and feedback, best practice dissemination meetings, academic detailing site visits, performance review and cds training. The control group did not receive information on the intervention, the cds or education. Baseline data collection was for three months with follow-up of 15 months after cds implementation. Outcomes – The outcomes were frequency of inappropriate prescribing during an ari episode, broad-spectrum antibiotic use and diagnostic shift. Inappropriate prescribing was computed by dividing the number of ari episodes with diagnoses in the inappropriate category that had an antibiotic prescription by the total number of ari episodes with diagnosis for which antibiotics are inappropriate. Broad-spectrum antibiotic use was computed by all ari episodes with a broad-spectrum antibiotic prescription by the total number of ari episodes with an antibiotic prescription. Antibiotic drift was computed in two ways: dividing the number of ari episodes with diagnoses where antibiotics are appropriate by the total number of ari episodes with an antibiotic prescription; and dividing the number of ari episodes where antibiotics were inappropriate by the total number of ari episodes. Process measure included frequency of cds template use and whether the outcome measures differed by cds usage. Analysis – Outcomes were measured quarterly for each practice, weighted by the number of ari episodes during the quarter to assign greater weight to practices with greater numbers of relevant episodes and to periods with greater numbers of relevant episodes. Weighted means and 95% ci s were computed separately for adult and pediatric (less than 18 years of age) patients for each time period for both groups. Baseline means in outcome measures were compared between the two groups using weighted independent-sample t -tests. Linear mixed models were used to compare changes over the 18-month period. The models included time, intervention status, and were adjusted for practice characteristics such as specialty, size, region and baseline ari s. Random practice effects were included to account for clustering of repeated measures on practices over time. P -values of less than 0.05 were considered significant. Findings – For adult patients, inappropriate prescribing in ari episodes declined more among the intervention group (-0.6%) than the control group (4.2%)( p = 0.03), and prescribing of broad-spectrum antibiotics declined by 16.6% in the intervention group versus an increase of 1.1% in the control group ( p < 0.0001). For pediatric patients, there was a similar decline of 19.7% in the intervention group versus an increase of 0.9% in the control group ( p < 0.0001). In summary, the cds had a modest effect in reducing inappropriate prescribing for adults, but had a substantial effect in reducing the prescribing of broad-spectrum antibiotics in adult and pediatric patients.

10.4.3. Interrupted Time Series on EHR Impact in Nursing Care

Dowding, Turley, and Garrido (2012) conducted a prospective its study to examine the impact of ehr implementation on nursing care processes and outcomes. The study is summarized below.

Setting – Kaiser Permanente ( kp ) as a large not-for-profit integrated healthcare organization in the United States. Participants – 29 kp hospitals in the northern and southern regions of California. Intervention – An integrated ehr system implemented at all hospitals with cpoe , nursing documentation and risk assessment tools. The nursing component for risk assessment documentation of pressure ulcers and falls was consistent across hospitals and developed by clinical nurses and informaticists by consensus. Design – its design with monthly data on pressure ulcers and quarterly data on fall rates and risk collected over seven years between 2003 and 2009. All data were collected at the unit level for each hospital. Outcomes – Process measures were the proportion of patients with a fall risk assessment done and the proportion with a hospital-acquired pressure ulcer ( hapu ) risk assessment done within 24 hours of admission. Outcome measures were fall and hapu rates as part of the unit-level nursing care process and nursing sensitive outcome data collected routinely for all California hospitals. Fall rate was defined as the number of unplanned descents to the floor per 1,000 patient days, and hapu rate was the percentage of patients with stages i-IV or unstageable ulcer on the day of data collection. Analysis – Fall and hapu risk data were synchronized using the month in which the ehr was implemented at each hospital as time zero and aggregated across hospitals for each time period. Multivariate regression analysis was used to examine the effect of time, region and ehr . Findings – The ehr was associated with significant increase in document rates for hapu risk (2.21; 95% CI 0.67 to 3.75) and non-significant increase for fall risk (0.36; -3.58 to 4.30). The ehr was associated with 13% decrease in hapu rates (-0.76; -1.37 to -0.16) but no change in fall rates (-0.091; -0.29 to 011). Hospital region was a significant predictor of variation for hapu (0.72; 0.30 to 1.14) and fall rates (0.57; 0.41 to 0.72). During the study period, hapu rates decreased significantly (-0.16; -0.20 to -0.13) but not fall rates (0.0052; -0.01 to 0.02). In summary, ehr implementation was associated with a reduction in the number of hapu s but not patient falls, and changes over time and hospital region also affected outcomes.

10.5. Summary

In this chapter we introduced randomized and non-randomized experimental designs as two types of comparative studies used in eHealth evaluation. Randomization is the highest quality design as it reduces bias, but it is not always feasible. The methodological issues addressed include choice of variables, sample size, sources of biases, confounders, and adherence to reporting guidelines. Three case examples were included to show how eHealth comparative studies are done.

• Baker T. B., Gustafson D. H., Shaw B., Hawkins R., Pingree S., Roberts L., Strecher V. Relevance of consort reporting criteria for research on eHealth interventions. Patient Education and Counselling. 2010; 81 (suppl. 7):77–86. [ PMC free article : PMC2993846 ] [ PubMed : 20843621 ]
• Columbia University. (n.d.). Statistics: sample size / power calculation. Biomath (Division of Biomathematics/Biostatistics), Department of Pediatrics. New York: Columbia University Medical Centre. Retrieved from http://www ​.biomath.info/power/index.htm .
• Boutron I., Moher D., Altman D. G., Schulz K. F., Ravaud P. consort Group. Extending the consort statement to randomized trials of nonpharmacologic treatment: Explanation and elaboration. Annals of Internal Medicine. 2008; 148 (4):295–309. [ PubMed : 18283207 ]
• Cochrane Collaboration. Cochrane handbook. London: Author; (n.d.) Retrieved from http://handbook ​.cochrane.org/
• consort Group. (n.d.). The consort statement . Retrieved from http://www ​.consort-statement.org/
• Dowding D. W., Turley M., Garrido T. The impact of an electronic health record on nurse sensitive patient outcomes: an interrupted time series analysis. Journal of the American Medical Informatics Association. 2012; 19 (4):615–620. [ PMC free article : PMC3384108 ] [ PubMed : 22174327 ]
• Friedman C. P., Wyatt J.C. Evaluation methods in biomedical informatics. 2nd ed. New York: Springer Science + Business Media, Inc; 2006.
• Guyatt G., Oxman A. D., Akl E. A., Kunz R., Vist G., Brozek J. et al. Schunemann H. J. grade guidelines: 1. Introduction – grade evidence profiles and summary of findings tables. Journal of Clinical Epidemiology. 2011; 64 (4):383–394. [ PubMed : 21195583 ]
• Harris A. D., McGregor J. C., Perencevich E. N., Furuno J. P., Zhu J., Peterson D. E., Finkelstein J. The use and interpretation of quasi-experimental studies in medical informatics. Journal of the American Medical Informatics Association. 2006; 13 (1):16–23. [ PMC free article : PMC1380192 ] [ PubMed : 16221933 ]
• The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. London: 2011. (Version 5.1.0, updated March 2011) Retrieved from http://handbook ​.cochrane.org/
• Holbrook A., Pullenayegum E., Thabane L., Troyan S., Foster G., Keshavjee K. et al. Curnew G. Shared electronic vascular risk decision support in primary care. Computerization of medical practices for the enhancement of therapeutic effectiveness (compete III) randomized trial. Archives of Internal Medicine. 2011; 171 (19):1736–1744. [ PubMed : 22025430 ]
• Mainous III A. G., Lambourne C. A., Nietert P.J. Impact of a clinical decision support system on antibiotic prescribing for acute respiratory infections in primary care: quasi-experimental trial. Journal of the American Medical Informatics Association. 2013; 20 (2):317–324. [ PMC free article : PMC3638170 ] [ PubMed : 22759620 ]
• Noordzij M., Tripepi G., Dekker F. W., Zoccali C., Tanck M. W., Jager K.J. Sample size calculations: basic principles and common pitfalls. Nephrology Dialysis Transplantation. 2010; 25 (5):1388–1393. Retrieved from http://ndt ​.oxfordjournals ​.org/content/early/2010/01/12/ndt ​.gfp732.short . [ PubMed : 20067907 ]
• Vervloet M., Linn A. J., van Weert J. C. M., de Bakker D. H., Bouvy M. L., van Dijk L. The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: A systematic review of the literature. Journal of the American Medical Informatics Association. 2012; 19 (5):696–704. [ PMC free article : PMC3422829 ] [ PubMed : 22534082 ]
• Zwarenstein M., Treweek S., Gagnier J. J., Altman D. G., Tunis S., Haynes B., Oxman A. D., Moher D. for the consort and Pragmatic Trials in Healthcare (Practihc) groups. Improving the reporting of pragmatic trials: an extension of the consort statement. British Medical Journal. 2008; 337 :a2390. [ PMC free article : PMC3266844 ] [ PubMed : 19001484 ] [ CrossRef ]
• Zwarenstein M., Treweek S. What kind of randomized trials do we need? Canadian Medical Association Journal. 2009; 180 (10):998–1000. [ PMC free article : PMC2679816 ] [ PubMed : 19372438 ]

Appendix. Example of Sample Size Calculation

This is an example of sample size calculation for an rct that examines the effect of a cds system on reducing systolic blood pressure in hypertensive patients. The case is adapted from the example described in the publication by Noordzij et al. (2010) .

(a) Systolic blood pressure as a continuous outcome measured in mmHg

Based on similar studies in the literature with similar patients, the systolic blood pressure values from the comparison groups are expected to be normally distributed with a standard deviation of 20 mmHg. The evaluator wishes to detect a clinically relevant difference of 15 mmHg in systolic blood pressure as an outcome between the intervention group with cds and the control group without cds . Assuming a significance level or alpha of 0.05 for 2-tailed t -test and power of 0.80, the corresponding multipliers 1 are 1.96 and 0.842, respectively. Using the sample size equation for continuous outcome below we can calculate the sample size needed for the above study.

n = 2[(a+b)2σ2]/(μ1-μ2)2 where

n = sample size for each group

μ1 = population mean of systolic blood pressures in intervention group

μ2 = population mean of systolic blood pressures in control group

μ1- μ2 = desired difference in mean systolic blood pressures between groups

σ = population variance

a = multiplier for significance level (or alpha)

b = multiplier for power (or 1-beta)

Providing the values in the equation would give the sample size (n) of 28 samples per group as the result

n = 2[(1.96+0.842)2(202)]/152 or 28 samples per group

(b) Systolic blood pressure as a categorical outcome measured as below or above 140 mmHg (i.e., hypertension yes/no)

In this example a systolic blood pressure from a sample that is above 140 mmHg is considered an event of the patient with hypertension. Based on published literature the proportion of patients in the general population with hypertension is 30%. The evaluator wishes to detect a clinically relevant difference of 10% in systolic blood pressure as an outcome between the intervention group with cds and the control group without cds . This means the expected proportion of patients with hypertension is 20% (p1 = 0.2) in the intervention group and 30% (p2 = 0.3) in the control group. Assuming a significance level or alpha of 0.05 for 2-tailed t -test and power of 0.80 the corresponding multipliers are 1.96 and 0.842, respectively. Using the sample size equation for categorical outcome below, we can calculate the sample size needed for the above study.

n = [(a+b)2(p1q1+p2q2)]/χ2

p1 = proportion of patients with hypertension in intervention group

q1 = proportion of patients without hypertension in intervention group (or 1-p1)

p2 = proportion of patients with hypertension in control group

q2 = proportion of patients without hypertension in control group (or 1-p2)

χ = desired difference in proportion of hypertensive patients between two groups

Providing the values in the equation would give the sample size (n) of 291 samples per group as the result

n = [(1.96+0.842)2((0.2)(0.8)+(0.3)(0.7))]/(0.1)2 or 291 samples per group

From Table 3 on p. 1392 of Noordzij et al. (2010).

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Lau F, Holbrook A. Chapter 10 Methods for Comparative Studies. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Types of Comparative Studies
• Methodological Considerations
• Case Examples
• Example of Sample Size Calculation

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 10 Methods for Comparative Studies - Handbook of eHealth Evaluation: An ... Chapter 10 Methods for Comparative Studies - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Degree Level

FINANCIAL AID

STUDENT SERVICES

PROFESSIONAL DEVELOPMENT

Crafting Compelling Narratives: How to Write a Business Case Study

June 13th, 2024 by JWU

Working in the business world, there is a lot that you need to know. It is not just about the technical aspects of running a business and handling finances, but marketing and beyond. One of the most versatile and useful skills you can develop as a business professional is that of being able to write a simple business case study. These can be powerful marketing tools that can help you build your brand’s reputation, increase customer loyalty, and demonstrate your brand’s unique value proposition.

So, what is a case study in business and why does it matter? We are covering all of this, as well as tips on how to write a business case study, below.

What Is a Business Case Study?

Specifically, a  business case study  refers to a publication that covers how a company or brand responded to a specific problem or situation in a successful way. For example, a case study might explain how a company’s services helped a client solve a problem or achieve a desired outcome. These case studies can then be published on business websites, blogs, and even shared on social media as a form of effective marketing.

In many ways, a business case study is a cohesive combination of brand information and customer testimonials that can help promote a company’s products/services in a positive light.

Why Are Business Case Studies Important?

Business case studies are important for businesses of all sizes. When they are well written, these publications can help showcase a brand’s unique expertise while building trust with clients. Likewise, these case studies can demonstrate the potential real-world results that clients can expect when they work with your business. This is often more persuasive than making simple claims alone.

Common Elements:

The exact elements of a business case study can vary based on the exact scenario and products/services being covered. However, most successful business case studies will include most or all these components:

• A compelling storyline
• Client testimonials or interviews
• A clear call to action
• Visuals or other presentations of data

Why Writing Business Case Studies Is Important for Your Organization

There are several reasons as to why business professionals should be able to write compelling case studies as part of their everyday jobs. Let’s dive into why writing is so crucial here:

Showcase Your Value Proposition

First, business case studies can be extremely effective when it comes to showcasing your brand’s unique value proposition. Case studies can help demonstrate how your brand has successfully addressed specific pain points with examples and tangible results.

Build Credibility

Meanwhile, business case studies can also be an excellent way to build credibility for your brand in a way that is more compelling and persuasive than more “traditional” marketing strategies. This is because with a business case study, you can use real-world examples to show potential clients first-hand what your company’s products/services can do for them.

Differentiate Yourself

Writing and publishing business case studies can also help your company set itself apart from its competitors in any space. This is because a professionally written business case study will showcase the real value of your brand, proving that you are not just another company making promises. Instead, a case study provides real-world examples and applications that demonstrate your brand’s history of success.

Multiple Use Cases

When it comes to marketing tools, a business case study is also one of the most versatile options out there. Even once a business case study is written and published, it can be repurposed for any number of applications and content types. From a single case study, for example, you might be able to reuse the content for your company’s website, social media page, sales presentations, and much more.

Steps for Writing an Effective Business Case Study

Now that you have a better understanding of what a business case study entails and why these are such crucial tools for your business, you may be wondering where to start when it comes to writing one. Writing effective business case studies is something that will take some time and practice on your part. Still, there are some tips and best practices you can follow to write better case studies today.

Client Selection

First, understand the importance of selecting the right client to highlight for your case study. You will want to make sure that you choose a case with a clear problem, a compelling solution you offered, and quantifiable results. From there, you will need to reach out to the client personally and make sure they are willing to participate in any interviews or write a testimonial as needed.

Get Client Buy-In

Of course, it is not enough to get a client to agree to participate. Ideally, you will want a client enthusiastic about being part of the case study. This will ensure that you get the best content possible when it comes to quotes, data approval, and the like. Not sure where to begin with client buy-in? Refer to some of your happiest clients, including those who have left your company positive reviews online, and go from there.

The Storytelling Approach

Another critical component of an effective business study is storytelling. With great storytelling, your finished case study will be more of a narrative and not just a list of facts. As you craft the storyline for your narrative, try to include the following components:

• The Challenge – What pain point did the client face?
• The Solution – How did your company’s product or service address it? This section should be as specific and in-depth as possible.
• The Results – What was the outcome of your company working with the client? This section should contain quantifiable metrics and the impact your involvement had on the client’s business.
• Testimonials – Being able to incorporate direct client quotes can add a lot of authenticity to your case study.

Keep It Focused

When writing a business case study, it is also essential to keep your document as focused as possible. At the same time, be careful not to overpromise or include absolutes that could mislead potential clients.

Visual Appeal

A business case study is not compelling if nobody is reading it. This is why it is key to break up larger blocks of text with plenty of eye-catching visuals. This adds visual interest and is more likely to keep readers engaged. Whenever possible, be sure to incorporate relevant images, infographics, and other visuals to break up chunks of text.

Have a Strong Call to Action

Finally, do not forget the clear call to action. Do not assume that readers will be able to read your mind and take the next step on their own. Instead, clearly guide readers on what you want them to do after reading your case study, whether it is reaching out to you, requesting a quote, or signing up for an email list.

Beyond the Basics: Tips for Excellence

In addition to the above best practices for crafting a compelling business case study, there are a few additional tips you can follow to take your writing to the next level.

Tailor to Your Target Audience

Whenever possible, try to write in a way that targets the most specific audience in your case study. Of course, this requires you to have a solid understanding of who your target audience is and what their specific pain points are. From there, you can speak their language and address their most pressing concerns to yield results.

Search Engine Optimization (SEO)

Business case studies can also be an excellent opportunity to improve your company’s  SEO . As you write your case study, try to naturally incorporate target keywords as much as possible. This way, when it comes time to publish your case study on your website or blog, you will also have the potential to improve your search engine rankings and drive more organic traffic to your site.

Promote Your Case Studies

Finally, understand that even once your case study is published, your work is not done. Take time to promote and share your case studies as much as possible, even going as far as to repurpose them into different content mediums from time to time. By including business case studies in your toolkit and sharing them actively, you can get as much mileage as possible out of them.

Sharpen Your Business Acumen at JWU

Now you know what is a case study in business. So, when carefully crafted and thoughtfully executed, a business case study can work wonders when it comes to increasing customer loyalty and building a positive reputation for your brand among stakeholders and potential clients. Of course, writing a business case study can be easier said than done. This is why it is so important to gain practice in business writing.

A formal education in business can help you gain the practical skills you need to write compelling cast studies and take your career to the next level. At Johnson & Wales University, we’re proud to offer both an  online bachelor’s in Business Administration  and an  online MBA program  to help you take your education in the right direction. For more information about completing your degree online, complete the  Request Info form , call 855-JWU-1881 , or email  [email protected] .

By clicking Get Started below, I consent to receive recurring marketing/promotional e-mails, phone calls, and SMS/text messages from Johnson & Wales University (JWU) about any educational/programmatic purpose (which relates to my inquiry of JWU) at the e-mail/phone numbers (landline/mobile) provided, including calls or texts made using an automatic telephone dialing system and/or artificial/prerecorded voice messages. My consent applies regardless of my inclusion on any state, federal, or other do-not-call lists. Consent is not a condition for receipt of any good or service. Carrier charges may apply. Terms and conditions apply .

Request info

Now accepting applications

AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries

A new study reveals the need for benchmarking and public evaluations of AI tools in law.

Artificial intelligence (AI) tools are rapidly transforming the practice of law. Nearly  three quarters of lawyers plan on using generative AI for their work, from sifting through mountains of case law to drafting contracts to reviewing documents to writing legal memoranda. But are these tools reliable enough for real-world use?

Large language models have a documented tendency to “hallucinate,” or make up false information. In one highly-publicized case, a New York lawyer  faced sanctions for citing ChatGPT-invented fictional cases in a legal brief;  many similar cases have since been reported. And our  previous study of general-purpose chatbots found that they hallucinated between 58% and 82% of the time on legal queries, highlighting the risks of incorporating AI into legal practice. In his  2023 annual report on the judiciary , Chief Justice Roberts took note and warned lawyers of hallucinations. 

Across all areas of industry, retrieval-augmented generation (RAG) is seen and promoted as the solution for reducing hallucinations in domain-specific contexts. Relying on RAG, leading legal research services have released AI-powered legal research products that they claim  “avoid” hallucinations and guarantee  “hallucination-free” legal citations. RAG systems promise to deliver more accurate and trustworthy legal information by integrating a language model with a database of legal documents. Yet providers have not provided hard evidence for such claims or even precisely defined “hallucination,” making it difficult to assess their real-world reliability.

AI-Driven Legal Research Tools Still Hallucinate

In a new  preprint study by  Stanford RegLab and  HAI researchers, we put the claims of two providers, LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI)), to the test. We show that their tools do reduce errors compared to general-purpose AI models like GPT-4. That is a substantial improvement and we document instances where these tools provide sound and detailed legal research. But even these bespoke legal AI tools still hallucinate an alarming amount of the time: the Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time.

Read the full study, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

To conduct our study, we manually constructed a pre-registered dataset of over 200 open-ended legal queries, which we designed to probe various aspects of these systems’ performance.

Broadly, we investigated (1) general research questions (questions about doctrine, case holdings, or the bar exam); (2) jurisdiction or time-specific questions (questions about circuit splits and recent changes in the law); (3) false premise questions (questions that mimic a user having a mistaken understanding of the law); and (4) factual recall questions (questions about simple, objective facts that require no legal interpretation). These questions are designed to reflect a wide range of query types and to constitute a challenging real-world dataset of exactly the kinds of queries where legal research may be needed the most.

Figure 1: Comparison of hallucinated (red) and incomplete (yellow) answers across generative legal research tools.

These systems can hallucinate in one of two ways. First, a response from an AI tool might just be  incorrect —it describes the law incorrectly or makes a factual error. Second, a response might be  misgrounded —the AI tool describes the law correctly, but cites a source which does not in fact support its claims.

Given the critical importance of authoritative sources in legal research and writing, the second type of hallucination may be even more pernicious than the outright invention of legal cases. A citation might be “hallucination-free” in the narrowest sense that the citation  exists , but that is not the only thing that matters. The core promise of legal AI is that it can streamline the time-consuming process of identifying relevant legal sources. If a tool provides sources that  seem authoritative but are in reality irrelevant or contradictory, users could be misled. They may place undue trust in the tool's output, potentially leading to erroneous legal judgments and conclusions.

Figure 2:  Top left: Example of a hallucinated response by Westlaw's AI-Assisted Research product. The system makes up a statement in the Federal Rules of Bankruptcy Procedure that does not exist (and Kontrick v. Ryan, 540 U.S. 443 (2004) held that a closely related bankruptcy deadline provision was not jurisdictional). Top right: Example of a hallucinated response by LexisNexis's Lexis+ AI. Casey and its undue burden standard were overruled by the Supreme Court in Dobbs v. Jackson Women's Health Organization, 597 U.S. 215 (2022); the correct answer is rational basis review. Bottom left: Example of a hallucinated response by Thomson Reuters's Ask Practical Law AI. The system fails to correct the user’s mistaken premise—in reality, Justice Ginsburg joined the Court's landmark decision legalizing same-sex marriage—and instead provides additional false information about the case. Bottom right: Example of a hallucinated response from GPT-4, which generates a statutory provision that has not been codified.

RAG Is Not a Panacea

Figure 3: An overview of the retrieval-augmentation generation (RAG) process. Given a user query (left), the typical process consists of two steps: (1) retrieval (middle), where the query is embedded with natural language processing and a retrieval system takes embeddings and retrieves the relevant documents (e.g., Supreme Court cases); and (2) generation (right), where the retrieved texts are fed to the language model to generate the response to the user query. Any of the subsidiary steps may introduce error and hallucinations into the generated response. (Icons are courtesy of FlatIcon.)

Under the hood, these new legal AI tools use retrieval-augmented generation (RAG) to produce their results, a method that many tout as a potential solution to the hallucination problem. In theory, RAG allows a system to first  retrieve the relevant source material and then use it to  generate the correct response. In practice, however, we show that even RAG systems are not hallucination-free. 

We identify several challenges that are particularly unique to RAG-based legal AI systems, causing hallucinations. 

First, legal retrieval is hard. As any lawyer knows, finding the appropriate (or best) authority can be no easy task. Unlike other domains, the law is not entirely composed of verifiable  facts —instead, law is built up over time by judges writing  opinions . This makes identifying the set of documents that definitively answer a query difficult, and sometimes hallucinations occur for the simple reason that the system’s retrieval mechanism fails.

Second, even when retrieval occurs, the document that is retrieved can be an inapplicable authority. In the American legal system, rules and precedents differ across jurisdictions and time periods; documents that might be relevant on their face due to semantic similarity to a query may actually be inapposite for idiosyncratic reasons that are unique to the law. Thus, we also observe hallucinations occurring when these RAG systems fail to identify the truly binding authority. This is particularly problematic as areas where the law is in flux is precisely where legal research matters the most. One system, for instance, incorrectly recited the “undue burden” standard for abortion restrictions as good law, which was overturned in  Dobbs (see Figure 2). 

Third, sycophancy—the tendency of AI to agree with the user's incorrect assumptions—also poses unique risks in legal settings. One system, for instance, naively agreed with the question’s premise that Justice Ginsburg dissented in  Obergefell , the case establishing a right to same-sex marriage, and answered that she did so based on her views on international copyright. (Justice Ginsburg did not dissent in  Obergefell and, no, the case had nothing to do with copyright.) Notwithstanding that answer, here there are optimistic results. Our tests showed that both systems generally navigated queries based on false premises effectively. But when these systems do agree with erroneous user assertions, the implications can be severe—particularly for those hoping to use these tools to increase access to justice among  pro se and under-resourced litigants.

Responsible Integration of AI Into Law Requires Transparency

Ultimately, our results highlight the need for rigorous and transparent benchmarking of legal AI tools. Unlike other domains, the use of AI in law remains alarmingly opaque: the tools we study provide no systematic access, publish few details about their models, and report no evaluation results at all.

This opacity makes it exceedingly challenging for lawyers to procure and acquire AI products. The large law firm  Paul Weiss spent nearly a year and a half testing a product, and did not develop “hard metrics” because checking the AI system was so involved that it “makes any efficiency gains difficult to measure.” The absence of rigorous evaluation metrics makes responsible adoption difficult, especially for practitioners that are less resourced than Paul Weiss. 

The lack of transparency also threatens lawyers’ ability to comply with ethical and professional responsibility requirements. The bar associations of  California ,  New York , and  Florida have all recently released guidance on lawyers’ duty of supervision over work products created with AI tools. And as of May 2024,  more than 25 federal judges have issued standing orders instructing attorneys to disclose or monitor the use of AI in their courtrooms.

Without access to evaluations of the specific tools and transparency around their design, lawyers may find it impossible to comply with these responsibilities. Alternatively, given the high rate of hallucinations, lawyers may find themselves having to verify each and every proposition and citation provided by these tools, undercutting the stated efficiency gains that legal AI tools are supposed to provide.

Our study is meant in no way to single out LexisNexis and Thomson Reuters. Their products are far from the only legal AI tools that stand in need of transparency—a slew of startups offer similar products and have  made   similar   claims , but they are available on even more restricted bases, making it even more difficult to assess how they function. 

Based on what we know, legal hallucinations have not been solved.The legal profession should turn to public benchmarking and rigorous evaluations of AI tools. 

This story was updated on Thursday, May 30, 2024, to include analysis of a third AI tool, Westlaw’s AI-Assisted Research.

Paper authors: Varun Magesh is a research fellow at Stanford RegLab. Faiz Surani is a research fellow at Stanford RegLab. Matthew Dahl is a joint JD/PhD student in political science at Yale University and graduate student affiliate of Stanford RegLab. Mirac Suzgun is a joint JD/PhD student in computer science at Stanford University and a graduate student fellow at Stanford RegLab. Christopher D. Manning is Thomas M. Siebel Professor of Machine Learning, Professor of Linguistics and Computer Science, and Senior Fellow at HAI. Daniel E. Ho is the William Benjamin Scott and Luna M. Scott Professor of Law, Professor of Political Science, Professor of Computer Science (by courtesy), Senior Fellow at HAI, Senior Fellow at SIEPR, and Director of the RegLab at Stanford University. 

More News Topics

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• Choosing a Title
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The title summarizes the main idea or ideas of your study. A good title contains the fewest possible words needed to adequately describe the content and/or purpose of your research paper.

Importance of Choosing a Good Title

The title is the part of a paper that is read the most, and it is usually read first . It is, therefore, the most important element that defines the research study. With this in mind, avoid the following when creating a title:

• If the title is too long, this usually indicates there are too many unnecessary words. Avoid language, such as, "A Study to Investigate the...," or "An Examination of the...." These phrases are obvious and generally superfluous unless they are necessary to covey the scope, intent, or type of a study.
• On the other hand, a title which is too short often uses words which are too broad and, thus, does not tell the reader what is being studied. For example, a paper with the title, "African Politics" is so non-specific the title could be the title of a book and so ambiguous that it could refer to anything associated with politics in Africa. A good title should provide information about the focus and/or scope of your research study.
• In academic writing, catchy phrases or non-specific language may be used, but only if it's within the context of the study [e.g., "Fair and Impartial Jury--Catch as Catch Can"]. However, in most cases, you should avoid including words or phrases that do not help the reader understand the purpose of your paper.
• Academic writing is a serious and deliberate endeavor. Avoid using humorous or clever journalistic styles of phrasing when creating the title to your paper. Journalistic headlines often use emotional adjectives [e.g., incredible, amazing, effortless] to highlight a problem experienced by the reader or use "trigger words" or interrogative words like how, what, when, or why to persuade people to read the article or click on a link. These approaches are viewed as counter-productive in academic writing. A reader does not need clever or humorous titles to catch their attention because the act of reading research is assumed to be deliberate based on a desire to learn and improve understanding of the problem. In addition, a humorous title can merely detract from the seriousness and authority of your research. 
• Unlike everywhere else in a college-level social sciences research paper [except when using direct quotes in the text], titles do not have to adhere to rigid grammatical or stylistic standards. For example, it could be appropriate to begin a title with a coordinating conjunction [i.e., and, but, or, nor, for, so, yet] if it makes sense to do so and does not detract from the purpose of the study [e.g., "Yet Another Look at Mutual Fund Tournaments"] or beginning the title with an inflected form of a verb such as those ending in -ing [e.g., "Assessing the Political Landscape: Structure, Cognition, and Power in Organizations"].

Appiah, Kingsley Richard et al. “Structural Organisation of Research Article Titles: A Comparative Study of Titles of Business, Gynaecology and Law.” Advances in Language and Literary Studies 10 (2019); Hartley James. “To Attract or to Inform: What are Titles for?” Journal of Technical Writing and Communication 35 (2005): 203-213; Jaakkola, Maarit. “Journalistic Writing and Style.” In Oxford Research Encyclopedia of Communication . Jon F. Nussbaum, editor. (New York: Oxford University Press, 2018): https://oxfordre.com/communication.

Structure and Writing Style

The following parameters can be used to help you formulate a suitable research paper title:

• The purpose of the research
• The scope of the research
• The narrative tone of the paper [typically defined by the type of the research]
• The methods used to study the problem

The initial aim of a title is to capture the reader’s attention and to highlight the research problem under investigation.

Create a Working Title Typically, the final title you submit to your professor is created after the research is complete so that the title accurately captures what has been done . The working title should be developed early in the research process because it can help anchor the focus of the study in much the same way the research problem does. Referring back to the working title can help you reorient yourself back to the main purpose of the study if you find yourself drifting off on a tangent while writing. The Final Title Effective titles in research papers have several characteristics that reflect general principles of academic writing.

• Indicate accurately the subject and scope of the study,
• Rarely use abbreviations or acronyms unless they are commonly known,
• Use words that create a positive impression and stimulate reader interest,
• Use current nomenclature from the field of study,
• Identify key variables, both dependent and independent,
• Reveal how the paper will be organized,
• Suggest a relationship between variables which supports the major hypothesis,
• Is limited to 5 to 15 substantive words,
• Does not include redundant phrasing, such as, "A Study of," "An Analysis of" or similar constructions,
• Takes the form of a question or declarative statement,
• If you use a quote as part of the title, the source of the quote is cited [usually using an asterisk and footnote],
• Use correct grammar and capitalization with all first words and last words capitalized, including the first word of a subtitle. All nouns, pronouns, verbs, adjectives, and adverbs that appear between the first and last words of the title are also capitalized, and
• Rarely uses an exclamation mark at the end of the title.

The Subtitle Subtitles are frequently used in social sciences research papers because it helps the reader understand the scope of the study in relation to how it was designed to address the research problem. Think about what type of subtitle listed below reflects the overall approach to your study and whether you believe a subtitle is needed to emphasize the investigative parameters of your research.

1.  Explains or provides additional context , e.g., "Linguistic Ethnography and the Study of Welfare Institutions as a Flow of Social Practices: The Case of Residential Child Care Institutions as Paradoxical Institutions." [Palomares, Manuel and David Poveda.  Text & Talk: An Interdisciplinary Journal of Language, Discourse and Communication Studies 30 (January 2010): 193-212]

2.  Adds substance to a literary, provocative, or imaginative title or quote , e.g., "Listen to What I Say, Not How I Vote": Congressional Support for the President in Washington and at Home." [Grose, Christian R. and Keesha M. Middlemass. Social Science Quarterly 91 (March 2010): 143-167]

3.  Qualifies the geographic scope of the research , e.g., "The Geopolitics of the Eastern Border of the European Union: The Case of Romania-Moldova-Ukraine." [Marcu, Silvia. Geopolitics 14 (August 2009): 409-432]

4.  Qualifies the temporal scope of the research , e.g., "A Comparison of the Progressive Era and the Depression Years: Societal Influences on Predictions of the Future of the Library, 1895-1940." [Grossman, Hal B. Libraries & the Cultural Record 46 (2011): 102-128]

5.  Focuses on investigating the ideas, theories, or work of a particular individual , e.g., "A Deliberative Conception of Politics: How Francesco Saverio Merlino Related Anarchy and Democracy." [La Torre, Massimo. Sociologia del Diritto 28 (January 2001): 75 - 98]

6.  Identifies the methodology used , e.g. "Student Activism of the 1960s Revisited: A Multivariate Analysis Research Note." [Aron, William S. Social Forces 52 (March 1974): 408-414]

7.  Defines the overarching technique for analyzing the research problem , e.g., "Explaining Territorial Change in Federal Democracies: A Comparative Historical Institutionalist Approach." [ Tillin, Louise. Political Studies 63 (August 2015): 626-641.

With these examples in mind, think about what type of subtitle reflects the overall approach to your study. This will help the reader understand the scope of the study in relation to how it was designed to address the research problem.

Anstey, A. “Writing Style: What's in a Title?” British Journal of Dermatology 170 (May 2014): 1003-1004; Balch, Tucker. How to Compose a Title for Your Research Paper. Augmented Trader blog. School of Interactive Computing, Georgia Tech University; Bavdekar, Sandeep B. “Formulating the Right Title for a Research Article.” Journal of Association of Physicians of India 64 (February 2016); Choosing the Proper Research Paper Titles. AplusReports.com, 2007-2012; Eva, Kevin W. “Titles, Abstracts, and Authors.” In How to Write a Paper . George M. Hall, editor. 5th edition. (Oxford: John Wiley and Sons, 2013), pp. 33-41; Hartley James. “To Attract or to Inform: What are Titles for?” Journal of Technical Writing and Communication 35 (2005): 203-213; General Format. The Writing Lab and The OWL. Purdue University; Kerkut G.A. “Choosing a Title for a Paper.” Comparative Biochemistry and Physiology Part A: Physiology 74 (1983): 1; “Tempting Titles.” In Stylish Academic Writing . Helen Sword, editor. (Cambridge, MA: Harvard University Press, 2012), pp. 63-75; Nundy, Samiran, et al. “How to Choose a Title?” In How to Practice Academic Medicine and Publish from Developing Countries? A Practical Guide . Edited by Samiran Nundy, Atul Kakar, and Zulfiqar A. Bhutta. (Springer Singapore, 2022), pp. 185-192.

• << Previous: Applying Critical Thinking
• Next: Making an Outline >>
• Last Updated: Jun 12, 2024 8:53 AM
• URL: https://libguides.usc.edu/writingguide

May 30, 2024

A general passion

Image credit: pgen.1011291

research article

Conserved signalling functions for Mps1, Mad1 and Mad2 in the Cryptococcus neoformans spindle checkpoint

Mps1-dependent phosphorylation of C-terminal Mad1 residues is a critical step in Cryptococcus spindle checkpoint signalling. 

Image credit: pgen.1011302

Recently Published Articles

• Pitx2 and Pax7 temporally governs specification of extraocular muscle stem cells">Interplay between Pitx2 and Pax7 temporally governs specification of extraocular muscle stem cells
• Caenorhabditis elegans to a bacterial pathogen">TGF-β ligand cross-subfamily interactions in the response of Caenorhabditis elegans to a bacterial pathogen
• Single-step generation of homozygous knockout/knock-in individuals in an extremotolerant parthenogenetic tardigrade using DIPA-CRISPR

Current Issue May 2024

Adaptations to nitrogen availability drive ecological divergence of chemosynthetic symbionts

The importance of nitrogen availability in driving the ecological diversification of chemosynthetic symbiont species and the role that bacterial symbionts may play in the adaptation of marine organisms to changing environmental conditions.

Image credit: pgen.1011295

Paramutation at the maize pl1 locus is associated with RdDM activity at distal tandem repeats

pl1 paramutation depends on trans-chromosomal RNA-directed DNA methylation operating at a discrete cis-linked and copy-number-dependent transcriptional regulatory element.

Image credit: pgen.1011296

Research Article

Genomic analyses of Symbiomonas scintillans show no evidence for endosymbiotic bacteria but does reveal the presence of giant viruses

A multi-gene tree showed the three SsV genome types branched within highly supported clades with each of BpV2, OlVs, and MpVs, respectively.

Image credit: pgen.1011218

A natural bacterial pathogen of C . elegans uses a small RNA to induce transgenerational inheritance of learned avoidance

A mechanism of learning and remembering pathogen avoidance likely happens in the wild. 

Image credit: pgen.1011178

Spoink , a LTR retrotransposon, invaded D. melanogaster populations in the 1990s

Evidence of Spoink retrotransposon's horizontal transfer into D. melanogaster populations post-1993, suggesting its origin from D.willistoni .

Image credit: pgen.1011201

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions

Understanding AI, specifically Deep Learning, in facial diagnostics for genetic conditions can enhance the design and utilization of AI tools.

Image credit: pgen.1011168

Maintenance of proteostasis by Drosophila Rer1 is essential for competitive cell survival and Myc-driven overgrowth

Loss of Rer1 induces proteotoxic stress, leading to cell competition and elimination ...

Image credit: pgen.1011171

Anthracyclines induce cardiotoxicity through a shared gene expression response signature

TOP2i induce thousands of shared gene expression changes in cardiomyocytes.

Image credit: pgen.1011164

New PLOS journals accepting submissions

Five new journals unified in addressing global health and environmental challenges are now ready to receive submissions: PLOS Climate , PLOS Sustainability and Transformation , PLOS Water , PLOS Digital Health , and PLOS Global Public Health

COVID-19 Collection

The COVID-19 Collection highlights all content published across the PLOS journals relating to the COVID-19 pandemic.

Submit your Lab and Study Protocols to PLOS ONE !

PLOS ONE is now accepting submissions of Lab Protocols, a peer-reviewed article collaboration with protocols.io, and Study Protocols, an article that credits the work done prior to producing and publishing results.

PLOS Reviewer Center

A collection of free training and resources for peer reviewers of PLOS journals—and for the peer review community more broadly—drawn from research and interviews with staff editors, editorial board members, and experienced reviewers.

Ten Simple Rules

PLOS Computational Biology 's "Ten Simple Rules" articles provide quick, concentrated guides for mastering some of the professional challenges research scientists face in their careers.

Welcome New Associate Editors!

PLOS Genetics welcomes several new Associate Editors to our board: Nicolas Bierne, Julie Simpson, Yun Li, Hongbin Ji, Hongbing Zhang, Bertrand Servin, & Benjamin Schwessinger

Expanding human variation at PLOS Genetics

The former Natural Variation section at PLOS Genetics relaunches as Human Genetic Variation and Disease. Read the editors' reasoning behind this change.

PLOS Genetics welcomes new Section Editors

Quanjiang Ji (ShanghaiTech University) joined the editorial board and Xiaofeng Zhu (Case Western Reserve University) was promoted as new Section Editors for the PLOS Genetics Methods section.

PLOS Genetics editors elected to National Academy of Sciences

Congratulations to Associate Editor Michael Lichten and Consulting Editor Nicole King, who are newly elected members of the National Academy of Sciences.

Harmit Malik receives Novitski Prize

Congratulations to Associate Editor Harmit Malik, who was awarded the Edward Novitski Prize by the Genetics Society of America for his work on genetic conflict. Harmit has also been elected as a new member of the American Academy of Arts & Sciences.

Publish with PLOS

• Submission Instructions
• Submit Your Manuscript

Connect with Us

• PLOS Genetics on Twitter
• PLOS on Facebook

Get new content from PLOS Genetics in your inbox

Thank you you have successfully subscribed to the plos genetics newsletter., sorry, an error occurred while sending your subscription. please try again later..