comparative study case study

This website may not work correctly because your browser is out of date. Please update your browser .

• Comparative case studies
• Comparative case studies File type PDF File size 510.74 KB

This guide, written by Delwyn Goodrick for UNICEF, focuses on the use of comparative case studies in impact evaluation.

The paper gives a brief discussion of their use and then outlines when it is appropriate to use them. It then provides step by step guidance on their use for an impact evaluation.

"A case study is an in-depth examination, often undertaken over time, of a single case – such as a policy, programme, intervention site, implementation process or participant. Comparative case studies cover two or more cases in a way that produces more generalizable knowledge about causal questions – how and why particular programmes or policies work or fail to work.

Comparative case studies are undertaken over time and emphasize comparison within and across contexts. Comparative case studies may be selected when it is not feasible to undertake an experimental design and/or when there is a need to understand and explain how features within the context influence the success of programme or policy initiatives. This information is valuable in tailoring interventions to support the achievement of intended outcomes."

• Comparative case studies: a brief description
• When is it appropriate to use this method?
• How to conduct comparative case studies
• Ethical issues and practical limitations
• Which other methods work well with this one?
• Presentation of results and analysis
• Example of good practices
• Examples of challenges

Goodrick, D., (2014), Comparative Case Studies, UNICEF . Retrieved from: http://devinfolive.info/impact_evaluation/img/downloads/Comparative_Case_Studies_ENG.pdf

What does a non-experimental evaluation look like? How can we evaluate interventions implemented across multiple contexts, where constructing a control group is not feasible?

This is part of a series

• UNICEF Impact Evaluation series
• Overview of impact evaluation
• Overview: Strategies for causal attribution
• Overview: Data collection and analysis methods in impact evaluation
• Theory of change
• Evaluative criteria
• Evaluative reasoning
• Participatory approaches
• Randomized controlled trials (RCTs) video guide
• Quasi-experimental design and methods
• Developing and selecting measures of child well-being
• Interviewing
• UNICEF webinar: Overview of impact evaluation
• UNICEF webinar: Overview of data collection and analysis methods in Impact Evaluation
• UNICEF webinar: Theory of change
• UNICEF webinar: Overview: strategies for causal inference
• UNICEF webinar: Participatory approaches in impact evaluation
• UNICEF webinar: Randomized controlled trials
• UNICEF webinar: Comparative case studies
• UNICEF Webinar: Quasi-experimental design and methods

'Comparative case studies ' is referenced in:

• Developing a research agenda for impact evaluation
• Impact evaluation

Back to top

© 2022 BetterEvaluation. All right reserved.

• Tools and Resources
• Customer Services
• Original Language Spotlight
• Alternative and Non-formal Education 
• Cognition, Emotion, and Learning
• Curriculum and Pedagogy
• Education and Society
• Education, Change, and Development
• Education, Cultures, and Ethnicities
• Education, Gender, and Sexualities
• Education, Health, and Social Services
• Educational Administration and Leadership
• Educational History
• Educational Politics and Policy
• Educational Purposes and Ideals
• Educational Systems
• Educational Theories and Philosophies
• Globalization, Economics, and Education
• Languages and Literacies
• Professional Learning and Development
• Research and Assessment Methods
• Technology and Education
• Share This Facebook LinkedIn Twitter

Article contents

Comparative case study research.

• Lesley Bartlett Lesley Bartlett University of Wisconsin–Madison
•  and  Frances Vavrus Frances Vavrus University of Minnesota
• https://doi.org/10.1093/acrefore/9780190264093.013.343
• Published online: 26 March 2019

Case studies in the field of education often eschew comparison. However, when scholars forego comparison, they are missing an important opportunity to bolster case studies’ theoretical generalizability. Scholars must examine how disparate epistemologies lead to distinct kinds of qualitative research and different notions of comparison. Expanded notions of comparison include not only the usual logic of contrast or juxtaposition but also a logic of tracing, in order to embrace approaches to comparison that are coherent with critical, constructivist, and interpretive qualitative traditions. Finally, comparative case study researchers consider three axes of comparison : the vertical, which pays attention across levels or scales, from the local through the regional, state, federal, and global; the horizontal, which examines how similar phenomena or policies unfold in distinct locations that are socially produced; and the transversal, which compares over time.

• comparative case studies
• case study research
• comparative case study approach
• epistemology

You do not currently have access to this article

Please login to access the full content.

Access to the full content requires a subscription

Printed from Oxford Research Encyclopedias, Education. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 17 April 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [66.249.64.20|185.66.15.189]
• 185.66.15.189

Character limit 500 /500

• High contrast
• Our mandate
• Our history
• Annual report
• PRESS CENTRE

Search UNICEF

All reports, view and search all reports in our document library.

• View and search all
• Afghanistan (5)
• Bangladesh (3)
• Bosnia and Herzegovina (2)
• Botswana (1)
• Bulgaria (1)
• Burkina Faso (1)
• Cambodia (2)
• Comoros (1)
• Côte d'Ivoire (4)
• Croatia (1)
• Democratic Republic of the Congo (2)
• East Asia and the Pacific (5)
• Eastern and Southern Africa (8)
• Ecuador (1)
• Ethiopia (8)
• Europe and Central Asia (6)
• European Union (1)
• Global (170)
• Guinea-Bissau (1)
• Indonesia (2)
• Jamaica (1)
• Kosovo (under UNSC Resolution 1244) (2)
• Lao People's Democratic Republic (2)
• Latin America and the Caribbean (6)
• Lebanon (1)
• Lesotho (2)
• Liberia (3)
• Madagascar (6)
• Malaysia (2)
• Mauritania (2)
• Middle East and North Africa (2)
• Montenegro (2)
• Morocco (1)
• Mozambique (9)
• Namibia (3)
• Nigeria (2)
• North Macedonia (2)
• Pacific Islands (2)
• Paraguay (1)
• Philippines (1)
• Sierra Leone (1)
• Somalia (4)
• South Africa (2)
• South Asia (3)
• South Sudan (2)
• State of Palestine (1)
• Syrian Arab Republic (1)
• Thailand (2)
• Ukraine (2)
• United Republic of Tanzania (9)
• Viet Nam (1)
• West and Central Africa (4)
• Annual report (4)
• Document (41)
• Report (224)
• Accessibility and inclusivity (6)
• Access to information (2)
• Adolescent and youth participation (10)
• Adolescent health and development (8)
• Adolescents (2)
• Armed conflict (3)
• Assistive technology (2)
• Birth registration (1)
• Cash transfers (15)
• Child abuse (2)
• Child care and supervision (3)
• Child labour (7)
• Child marriage (2)
• Child protection (47)
• Children with disabilities (8)
• Child rights (8)
• Child survival (1)
• Climate change (17)
• Climate change and impacts (1)
• COVID-19 (67)
• Data analysis (1)
• Data and reports (17)
• Data collection (7)
• Digital technology (56)
• Discrimination (1)
• Displacement (7)
• Early childhood development (7)
• Early education (4)
• Education (99)
• Education in emergencies (2)
• Environment (3)
• Equitable access (6)
• Evaluation (1)
• Evidence for action (2)
• Female genital mutilation (1)
• Financial crisis (2)
• Food crisis and famine (3)
• Gender based violence (5)
• Gender equality (17)
• Gender equality in education (1)
• Gender rights (2)
• Girls education (1)
• Health (18)
• Humanitarian action and emergencies (1)
• Immunization (4)
• Mental health (10)
• Migrant and refugee crisis (3)
• Migration (11)
• Nutrition (6)
• Parenting (3)
• Partnerships (1)
• Poverty (28)
• Refugee and migrant children (13)
• Sexual abuse (3)
• Sexual exploitation (6)
• Social and behaviour change (11)
• Social and economic policy (18)
• Social inclusion (7)
• Social norms (1)
• Social policy (25)
• Social skills (2)
• Social well being (2)
• Sport for development (2)
• Sustainability (1)
• Sustainable Development Goals (1)
• Teachers (1)
• Unemployment (1)
• Violence against children (10)
• Water, sanitation and hygiene (1)
• Youth engagement (9)
• Youth media (1)

Sorry, we couldn't find any results

Please modify your search and try again.

Searching for an older publication?

If you are looking for a publication that is more than five years old, you may request it from our archive.

Spotlight projects

The state of the world’s children.

UNICEF’s flagship report – the most comprehensive analysis of global trends affecting children

Changing Childhood Project

What is childhood like today?

Prospects for Children: Global Outlook

An annual analysis of trends shaping the world and their impact on children

Report Card

Understanding child well-being everywhere

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 25 February 2020

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

• Bella Reichard   ORCID: orcid.org/0000-0001-5057-4019 1 ,
• Mark S Reed 1 ,
• Jenn Chubb 2 ,
• Ged Hall   ORCID: orcid.org/0000-0003-0815-2925 3 ,
• Lucy Jowett   ORCID: orcid.org/0000-0001-7536-3429 4 ,
• Alisha Peart 4 &
• Andrea Whittle 1  

Palgrave Communications volume  6 , Article number:  31 ( 2020 ) Cite this article

23k Accesses

14 Citations

84 Altmetric

Metrics details

• Language and linguistics

This paper reports on two studies that used qualitative thematic and quantitative linguistic analysis, respectively, to assess the content and language of the largest ever sample of graded research impact case studies, from the UK Research Excellence Framework 2014 (REF). The paper provides the first empirical evidence across disciplinary main panels of statistically significant linguistic differences between high- versus low-scoring case studies, suggesting that implicit rules linked to written style may have contributed to scores alongside the published criteria on the significance, reach and attribution of impact. High-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach than low-scoring cases. High-scoring case studies contained attributional phrases which were more likely to attribute research and/or pathways to impact, and they were written more coherently (containing more explicit causal connections between ideas and more logical connectives) than low-scoring cases. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, and often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than typically associated with academic writing. High-scoring case studies in two Main Panels were significantly easier to read than low-scoring cases on the Flesch Reading Ease measure, although both high-scoring and low-scoring cases tended to be of “graduate” reading difficulty. The findings of our work enable impact case study authors to better understand the genre and make content and language choices that communicate their impact as effectively as possible. While directly relevant to the assessment of impact in the UK’s Research Excellence Framework, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

Similar content being viewed by others

Worldwide divergence of values

Joshua Conrad Jackson & Danila Medvedev

Anger is eliminated with the disposal of a paper written because of provocation

Yuta Kanaya & Nobuyuki Kawai

An open source knowledge graph ecosystem for the life sciences

Tiffany J. Callahan, Ignacio J. Tripodi, … Lawrence E. Hunter

Introduction

Academics are under increasing pressure to engage with non-academic actors to generate “usable” knowledge that benefits society and addresses global challenges (Clark et al., 2016 ; Lemos, 2015 ; Rau et al., 2018 ). This is largely driven by funders and governments that seek to justify the societal value of public funding for research (Reed et al., 2020 ; Smith et al., 2011 ) often characterised as ‘impact’. While this has sometimes been defined narrowly as reflective of the need to demonstrate a return on public investment in research (Mårtensson et al., 2016 ; Tsey et al., 2016 ; Warry, 2006 ), there is also a growing interest in the evaluation of “broader impacts” from research (cf. Bozeman and Youtie, 2017 ; National Science Foundation, 2014 ), including less tangible but arguably equally relevant benefits for society and culture. This shift is exemplified by the assessment of impact in the UK’s Research Excellence Framework (REF) in 2014 and 2021, the system for assessing the quality of research in UK higher education institutions, and in the rise of similar policies and evaluation systems in Australia, Hong Kong, the United States, Horizon Europe, The Netherlands, Sweden, Italy, Spain and elsewhere (Reed et al., 2020 ).

The evaluation of research impact in the UK has been criticised by scholars largely for its association with a ‘market logic’ (Olssen and Peters, 2005 ; Rhoads and Torres, 2005 ). Critics argue that a focus of academic performativity can be seen to “destabilise” professional identities (Chubb and Watermeyer, 2017 ), which in the context of research impact evaluation can further “dehumanise and deprofessionalise” academic performance (Watermeyer, 2019 ), whilst leading to negative unintended consequences (which Derrick et al., 2018 , called “grimpact”). MacDonald ( 2017 ), Chubb and Reed ( 2018 ) and Weinstein et al. ( 2019 ) reported concerns from researchers that the impact agenda may be distorting research priorities, “encourag[ing] less discovery-led research” (Weinstein et al., 2019 , p. 94), though these concerns were questioned by University managers in the same study who were reported to “not have enough evidence to support that REF was driving specific research agendas in either direction” (p. 94), and further questioned by Hill ( 2016 ).

Responses to this critique have been varied. Some have called for civil disobedience (Watermeyer, 2019 ) and organised resistance (Back, 2015 ; MacDonald, 2017 ) against the impact agenda. In a review of Watermeyer ( 2019 ), Reed ( 2019 ) suggested that attitudes towards the neoliberal political roots of the impact agenda may vary according to the (political) values and beliefs of researchers, leading them to pursue impacts that either support or oppose neoliberal political and corporate interests. Some have defended the benefits of research impact evaluation. For example, Weinstein et al. ( 2019 ) found that “a focus on changing the culture outside of academia is broadly valued” by academics and managers. The impact agenda might enhance stakeholder engagement (Hill, 2016 ) and give “new currency” to applied research (Chubb, 2017 ; Watermeyer, 2019 ). Others have highlighted the long-term benefits for society of incentivising research impact, including increased public support and funding for a more accountable, outward-facing research system (Chubb and Reed, 2017 ; Hill, 2016 ; Nesta, 2018 ; Oancea, 2010 , 2014 ; Wilsdon et al., 2015 ).

In the UK REF, research outputs and impact are peer reviewed at disciplinary level in ‘Units of Assessment’ (36 in 2014, 34 in 2021), grouped into four ‘Main Panels’. Impact is assessed through case studies that describe the effects of academic research and are given a score between 1* (“recognised but modest”) and 4* (“outstanding”). The case studies follow a set structure of five sections: 1—Summary of the impact; 2—Underpinning research; 3—References to the research; 4—Details of the impact; 5—Sources to corroborate the impact (HEFCE, 2011 ). The publication of over 6000 impact case studies in 2014 Footnote 1 by Research England (formerly Higher Education Funding Council for England, HEFCE) was unique in terms of its size, and unlike the recent selective publication of high-scoring case studies from Australia’s 2018 Engagement and Impact Assessment, both high-scoring and low-scoring case studies were published. This provides a unique opportunity to evaluate the construction of case studies that were perceived by evaluation panels to have successfully demonstrated impact, as evidenced by a 4* rating, and to compare these to case studies that were judged as less successful.

The analysis of case studies included in this research is based on the definition of impact used in REF2014, as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (HEFCE, 2011 , p. 26). According to REF2014 guidance, the primary functions of an impact case study were to articulate and evidence the significance and reach of impacts arising from research beyond academia, clearly demonstrating the contribution that research from a given institution contributed to those impacts (HEFCE, 2011 ).

In addition to these explicit criteria driving the evaluation of impact in REF2014, a number of analyses have emphasised the role of implicit criteria and subjectivity in shaping the evaluation of impact. For example, Pidd and Broadbent ( 2015 ) emphasised the implicit role a “strong narrative” plays in high-scoring case studies (p. 575). This was echoed by the fears of one REF2014 panellist interviewed by Watermeyer and Chubb ( 2018 ) who said, “I think with impact it is literally so many words of persuasive narrative” as opposed to “giving any kind of substance” (p. 9). Similarly, Watermeyer and Hedgecoe ( 2016 ), reporting on an internal exercise at Cardiff University to evaluate case studies prior to submission, emphasised that “style and structure” were essential to “sell impact”, and that “case studies that best sold impact were those rewarded with the highest evaluative scores” (p. 651).

Recent research based on interviews with REF2014 panellists has also emphasised the subjectivity of the peer-review process used to evaluate impact. Derrick’s ( 2018 ) research findings based on panellist interviews and participant observation of REF2014 sub-panels argued that scores were strongly influenced by who the evaluators were and how the group assessed impact together. Indeed, a panellist interviewed by Watermeyer and Chubb ( 2018 ) concurred that “the panel had quite an influence on the criteria” (p. 7), including an admission that some types of (more intangible) evidence were more likely to be overlooked than other (more concrete) forms of evidence, “privileg[ing] certain kinds of impact”. Other panellists interviewed spoke of their emotional and intellectual vulnerability in making judgements about an impact criterion that they had little prior experience of assessing (Watermeyer and Chubb, 2018 ). Derrick ( 2018 ) argued that this led many evaluators to base their assessments on more familiar proxies for excellence linked to scientific excellence, which led to biased interpretations and shortcuts that mimicked “groupthink” (p. 193).

This paper will for the first time empirically assess the content and language of the largest possible sample of research impact case studies that received high versus low scores from assessment panels in REF2014. Combining qualitative thematic and quantitative linguistic analysis, we ask:

How do high-scoring versus low-scoring case studies articulate and evidence impacts linked to underpinning research?

Do high-scoring and low-scoring case studies have differences in their linguistic features or styles?

Do high-scoring and low-scoring case studies have lexical differences (words and phrases that are statistically more likely to occur in high- or low-scoring cases) or text-level differences (including reading ease, narrative clarity, use of cohesive devices)?

By answering these questions, our goal is to provide evidence for impact case study authors and their institutions to reflect on in order to optimally balance the content and to use language that communicates their impact as effectively as possible. While directly relevant to the assessment of impact in the UK’s REF, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

Research design and sample

The datasets were generated by using published institutional REF2014 impact scores to deduce the scores of some impact case studies themselves. Although scores for individual case studies were not made public, we were able to identify case studies that received the top mark of 4* based on the distribution of scores received by some institutions, where the whole submission by an institution in a given Unit of Assessment was awarded the same score. In those 20 Units of Assessment (henceforth UoA) where high-scoring case studies could be identified in this way, we also accessed all case studies known to have scored either 1* or 2* in order to compare the features of high-scoring case studies to those of low-scoring case studies.

We approached our research questions with two separate studies, using quantitative linguistic and qualitative thematic analysis respectively. The thematic analysis, explained in more detail in the section “Qualitative thematic analysis” below, allowed us to find answers to research question 1 (see above). The quantitative linguistic analysis was used to extract and compare typical word combinations for high-scoring and low-scoring case studies, as well as assessing their readability. It mainly addressed research questions 2 and 3.

The quantitative linguistic analysis was based on a sample of all identifiable high-scoring case studies in any UoA ( n  = 124) and all identifiable low-scoring impact case studies in those UoAs where high-scoring case studies could be identified ( n  = 93). As the linguistic analysis focused on identifying characteristic language choices in running text, only those sections designed to contain predominantly text were included (1—Summary of the impact; 2—Underpinning research; 4—Details of the impact). Figure 1 shows the distribution of case studies across Main Panels in the quantitative analysis. Table 1 summarises the number of words included in the analysis.

Distribution of case studies across Main Panels used for the linguistic analysis sample.

In order to detect patterns of content in high-scoring and low-scoring case studies across all four Main Panels, a sub-sample of case studies was selected for a qualitative thematic analysis. This included 60% of high-scoring case studies and 97% of low-scoring case studies from the quantitative analysis, such that only UoAs were included where both high-scoring and low-scoring case studies are available (as opposed to the quantitative sample, which includes all available high-scoring case studies). Further selection criteria were then designed to create a greater balance in the number of high-scoring and low-scoring case studies across Main Panels. Main Panels A (high) and C (low) were particularly over-represented, so a lower proportion of those case studies were selected and 10 additional high-scoring case studies were considered in Panel B, including institutions where at least 85% of the case studies scored 4* and the remaining scores were 3*. As this added a further UoA, we could also include 14 more low-scoring case studies in Main Panel B. This resulted in a total of 85 high-scoring and 90 low-scoring case studies. Figure 2 shows the distribution of case studies across Main Panels in the thematic analysis, illustrating the greater balance compared to the sample used in the quantitative analysis. The majority (75%) of the case studies analysed are included in both samples (Table 2 ).

Distribution of case studies across Main Panels used for the thematic analysis sample.

Quantitative linguistic analysis

Quantitative linguistic analysis can be used to make recurring patterns in language use visible and to assess their significance. We treated the dataset of impact case studies as a text collection (the ‘corpus’) divided into two sections, namely high-scoring and low-scoring case studies (the two ‘sub-corpora’), in order to explore the lexical profile and the readability of the case studies.

One way to explore the lexical profile of groups of texts is to generate frequency-based word lists and compare these to word lists from a reference corpus to determine which words are characteristic of the corpus of interest (“keywords”, cf. Scott, 1997 ). Another way is to extract word combinations that are particularly frequent. Such word combinations, called “lexical bundles”, are “extended collocations” (Hyland, 2008 , p. 41) that appear across a set range of texts (Esfandiari and Barbary, 2017 ). We merged these two approaches in order to uncover meanings that could not be made visible through the analysis of single-word frequencies, comparing lexical bundles from each sub-corpus to the other. Lexical bundles of 2–4 words were extracted with AntConc (specialist software developed by Anthony, 2014 ) firstly from the corpus of all high-scoring case studies and then separately from the sub-corpora of high-scoring case studies in Main Panel A, C and D. Footnote 2 The corresponding lists were extracted from low-scoring case studies overall and separated by panel. The lists of lexical bundles for each of the high-scoring corpus parts were then compared to the corresponding low-scoring parts (High-Overall vs. Low-Overall, High-Main Panel A vs. Low-Main Panel A, etc.) to detect statistically significant over-use and under-use in one set of texts relative to another.

Two statistical measures were used in the analysis of lexical bundles. Log Likelihood was used as a measure of the statistical significance of frequency differences (Rayson and Garside, 2000 ), with a value of >3.84 corresponding to p  < 0.05. This measure had the advantage, compared to the more frequently used chi-square test, of not assuming a normal distribution of data (McEnery et al., 2006 ). The Log Ratio (Hardie, 2014 ) was used as a measure of effect size, which quantifies the scale, rather than the statistical significance, of frequency differences between two datasets. The Log Ratio is technically the binary log of the relative risk, and a value of >0.5 or <−0.5 is considered meaningful in corpus linguistics (Hardie, 2014 ), with values further removed from 0 reflecting a bigger difference in the relative frequencies found in each corpus. There is currently no agreed standard effect size measure for keywords (Brezina, 2018 , p. 85) and the Log Ratio was chosen because it is straightforward to interpret. Each lexical bundle that met the ‘keyness’ threshold (Log Likelihood > 3.84 in the case of expected values > 12, with higher significance levels needed for expected values < 13—see Rayson et al., 2004 , p. 8) was then assigned a code according to its predominant meaning in the texts, as reflected in the contexts captured in the concordance lines extracted from the corpus.

In the thematic analysis, it appeared that high-scoring case studies were easier to read. In order to quantify the readability of the texts, we therefore analysed them using the Coh-Metrix online tool (www.cohmetrix.com, v3.0) developed by McNamara et al. ( 2014 ). This tool provides 106 descriptive indices of language features, including 8 principal component scores developed from combinations of the other indices (Graesser et al., 2011 ). We selected these principal component scores as comprehensive measures of “reading ease” because they assess multiple characteristics of the text, up to whole-text discourse level (McNamara et al., 2014 , p. 78). This was supplemented by the traditional and more wide-spread Flesch Reading Ease score of readability measuring the lengths of words and sentences, which are highly correlated with reading speed (Haberlandt and Graesser, 1985 ). The selected measures were compared across corpus sections using t -tests to evaluate significance. The effect size was measured using Cohen’s D , following Brezina ( 2018 , p. 190), where D  > 0.3 indicates a small, D  > 0.5 a medium, and D  > 0.8 a high effect size. As with the analysis of lexical bundles, comparisons were made between high- and low-scoring case studies in each of Main Panels A, C and D, as well as between all high-scoring and all low-scoring case studies across Main Panels.

Qualitative thematic analysis

While a quantitative analysis as described above can make differences in the use of certain words visible, it does not capture the narrative or content of the texts under investigation. In order to identify common features of high-scoring and low-scoring case studies, thematic analysis was chosen to complement the quantitative analysis by identifying patterns and inferring meaning from qualitative data (Auerbach and Silverstein, 2003 ; Braun and Clarke, 2006 ; Saldana, 2009 ). To familiarise themselves with the data and for inter-coder reliability, two research team members read a selection of REF2014 impact case studies from different Main Panels, before generating initial codes for each of the five sections of the impact case study template. These were discussed with the full research team, comprising three academic and three professional services staff who had all read multiple case studies themselves. They were piloted prior to defining a final set of themes and questions against which the data was coded (based on the six-step process outlined by Braun and Clarke, 2006 ) (Table 3 ). An additional category was used to code stylistic features, to triangulate elements of the quantitative analysis (e.g. readability) and to include additional stylistic features difficult to assess in quantitative terms (e.g. effective use of testimonials). In addition to this, 10 different types of impact were coded for, based on Reed’s ( 2018 ) typology: capacity and preparedness, awareness and understanding, policy, attitudinal change, behaviour change and other forms of decision-making, other social, economic, environmental, health and wellbeing, and cultural impacts. There was room for coders to include additional insights arising in each section of the case study that had not been captured in the coding system; and there was room to summarise other key factors they thought might account for high or low scores.

Coders summarised case study content pertaining to each code, for example by listing examples of effective or poor use of structure and formatting as they arose in each case study. Coders also quoted the original material next to their summaries so that their interpretation could be assessed during subsequent analysis. This initial coding of case study text was conducted by six coders, with intercoder reliability (based on 10% of the sample) assessed at over 90%. Subsequent thematic analysis within the codes was conducted by two of the co-authors. This involved categorising coded material into themes as a way of assigning meaning to features that occurred across multiple case studies (e.g. categorising types of corroborating evidence typically used in high-scoring versus low-scoring case studies).

Results and discussion

In this section, we integrate findings from the quantitative linguistic study and the qualitative analysis of low-scoring versus high-scoring case studies. The results are discussed under four headings based on the key findings that emerged from both analyses. Taken together, these findings provide the most comprehensive evidence to date of the characteristics of a top-rated (4*) impact case study in REF2014.

Highly-rated case studies provided specific, high-magnitude and well-evidenced articulations of significance and reach

One finding from our qualitative thematic analysis was that 84% of high-scoring cases articulated benefits to specific groups and provided evidence of their significance and reach, compared to 32% of low-scoring cases which typically focused instead on the pathway to impact, for example describing dissemination of research findings and engagement with stakeholders and publics without citing the benefits arising from dissemination or engagement. One way of conceptualising this difference is using the content/process distinction: whereas low-scoring cases tended to focus on the process through which impact was sought (i.e. the pathway used), the high-scoring cases tended to focus on the content of the impact itself (i.e. what change or improvement occurred as a result of the research).

Examples of global reach were evidenced across high-scoring case studies from all panels (including Panel D for Arts and Humanities research), but were less often claimed or evidenced in low-scoring case studies. Where reach was more limited geographically, many high-scoring case studies used context to create robust arguments that their reach was impressive in that context, describing reach for example in social or cultural terms or arguing for the importance of reaching a narrow but hard-to-reach or otherwise important target group.

Table 4 provides examples of evidence from high-scoring cases and low-scoring cases that were used to show significance and reach of impacts in REF2014.

Findings from the quantitative linguistic analysis in Table 5 show how high-scoring impact case studies contained more phrases that specified reach (e.g. “in England and”, “in the US”), compared to low-scoring case studies that used the more generic term “international”, leaving the reader in doubt about the actual reach. They also include more phrases that implicitly specified the significance of the impact (e.g. “the government’s” or “to the House of Commons”), compared to low-scoring cases which provided more generic phrases, such as “policy and practice”, rather than detailing specific policies or practices that had been changed.

The quantitative linguistics analysis also identified a number of words and phrases pertaining to engagement and pathways, which were intended to deliver impact but did not actually specify impact (Table 6 ). A number of phrases contained the word “dissemination”, and there were several words and phrases specifying types of engagement that could be considered more one-way dissemination than consultative or co-productive (cf. Reed et al.’s ( 2018 ) engagement typology), e.g. “the book” and “the event”. The focus on dissemination supports the finding from the qualitative thematic analysis that low-scoring case tended to focus more on pathways or routes than on impact. Although it is not possible to infer this directly from the data, it is possible that this may represent a deeper epistemological position underpinning some case studies, where impact generation was seen as one-way knowledge or technology transfer, and research findings were perceived as something that could be given unchanged to publics and stakeholders through dissemination activities, with the assumption that this would be understood as intended and lead to impact.

It is worth noting that none of the four UK countries appear significantly more often in either high-scoring or low-scoring case studies (outside of the phrase “in England and”). Wales ( n  = 50), Scotland ( n  = 71) and Northern Ireland ( n  = 32) appear slightly more often in high-scoring case studies, but the difference is not significant (England: n  = 162). An additional factor to take into account is that our dataset includes only submissions that are either high-scoring or low-scoring, and the geographical spread of the submitting institutions was not a factor in selecting texts. There was a balanced number of high-scoring and low-scoring case studies in the sample from English, Scottish and Welsh universities, but no guaranteed low-scoring submissions from Northern Irish institutions. The REF2014 guidance made it clear that impacts in each UK country would be evaluated equally in comparison to each other, the UK and other countries. While the quantitative analysis of case studies from our sample only found a statistically significant difference for the phrase “in England and”, this, combined with the slightly higher number of phrases containing the other countries of the UK in high-scoring case studies, might indicate that this panel guidance was implemented as instructed.

Figures 3 – 5 shows which types of impact could be identified in high-scoring or low-scoring case studies, respectively, in the qualitative thematic analysis (based on Reed’s ( 2018 ) typology of impacts). Note that percentages do not add up to 100% because it was possible for each case study to claim more than one type of impact (high-scoring impact case studies described on average 2.8 impacts, compared to an average of 1.8 impacts described by low-scoring case studies) Footnote 3 . Figure 3 shows the number of impacts per type as a percentage of the total number of impacts claimed in high-scoring versus low-scoring case studies. This shows that high-scoring case studies were more likely to claim health/wellbeing and policy impacts, whereas low-scoring case studies were more likely to claim understanding/awareness impacts. Looking at this by Main Panel, over 50% of high-scoring case studies in Main Panel A claimed health/wellbeing, policy and understanding/awareness impacts (Fig. 4 ), whereas over 50% of low-scoring case studies in Main Panel A claimed capacity building impacts (Fig. 5 ). There were relatively high numbers of economic and policy claimed in both high-scoring and low-scoring case studies in Main Panels B and C, respectively, with no impact type dominating strongly in Main Panel D (Figs. 4 and 5 ).

Number of impacts claimed in high- versus low-scoring case studies by impact type.

Percentage of high-scoring case studies that claimed different types of impact.

Percentage of low-scoring case studies that claimed different types of impact.

Highly-rated case studies used distinct features to establish links between research (cause) and impact (effect)

Findings from the quantitative linguistic analysis show that high-scoring case studies were significantly more likely to include attributional phrases like “cited in”, “used to” and “resulting in”, compared to low-scoring case studies (Table 7 provides examples for some of the 12 phrases more frequent in high-scoring case studies). However, there were some attributional phrases that were more likely to be found in low-scoring case studies (e.g. “from the”, “of the research” and “this work has”—total of 9 different phrases).

To investigate this further, all 564 and 601 instances Footnote 4 of attributional phrases in high-scoring and low-scoring case studies, respectively, were analysed to categorise the context in which they were used, to establish the extent to which these phrases in each corpus were being used to establish attribution to impacts. The first word or phrase preceding or succeeding the attributional content was coded. For example, if the attributional content was “used the”, followed by “research to generate impact”, the first word succeeding the attributional content (in this case “research”) was coded rather than the phrase it subsequently led to (“generate impact”). According to a Pearson Chi Square test, high-scoring case studies were significantly more likely to establish attribution to impact than low-scoring cases ( p  < 0.0001, but with a small effect size based on Cramer’s V  = 0.22; bold in Table 8 ). 18% ( n  = 106) of phrases in the low-scoring corpus established attribution to impact, compared to 37% ( n  = 210) in the high-scoring corpus, for example, stating that research, pathway or something else led to impact. Instead, low-scoring case studies were more likely to establish attribution to research (40%; n  = 241) compared to high-scoring cases (28%; n  = 156; p  < 0.0001, but with a small effect size based on Cramer’s V  = 0.135). Both high- and low-scoring case studies were similarly likely to establish attribution to pathways (low: 32%; n  = 194; high: 31% n  = 176).

Moreover, low-scoring case studies were more likely to include ambiguous or uncertain phrases. For example, the phrase “a number of” can be read to imply that it is not known how many instances there were. This occurred in all sections of the impact case studies, for example in the underpinning research section as “The research explores a number of themes” or in the summary or details of the impact section as “The work has also resulted in a number of other national and international impacts”, or “has influenced approaches and practices of a number of partner organisations”. Similarly, “an impact on” could give the impression that the nature of the impact is not known. This phrase occurred only in summary and details of the impact sections, for example, “These activities have had an impact on the professional development”, “the research has had an impact on the legal arguments”, or “there has also been an impact on the work of regional agency”.

In the qualitative thematic analysis, we found that only 50% of low-scoring case studies clearly linked the underpinning research to claimed impacts (compared to 97% of high-scoring cases). This gave the impression of over-claimed impacts in some low-scoring submissions. For example, one case study claimed “significant impacts on [a country’s] society” based on enhancing the security of a new IT system in the department responsible for publishing and archiving legislation. Another claimed “economic impact on a worldwide scale” based on billions of pounds of benefits, calculated using an undisclosed method by an undisclosed evaluator in an unpublished final report by the research team. One case study claimed attribution for impact based on similarities between a prototype developed by the researchers and a product subsequently launched by a major corporation, without any evidence that the product as launched was based on the prototype. Similar assumptions were made in a number of other case studies that appeared to conflate correlation with causation in their attempts to infer attribution between research and impact. Table 9 provides examples of different ways in which links between research and impact were evidenced in the details of the research section.

Table 10 shows how corroborating sources were used to support these claims. 82% of high-scoring case studies compared to 7% of low-scoring cases were identified in the qualitative thematic analysis as having generally high-quality corroborating evidence. In contrast, 11% of high-scoring case studies, compared to 71% of low-scoring cases, were identified as having corroborating evidence that was vague and/or poorly linked to claimed impacts. Looking at only case studies that claim policy impact, 11 out of 26 high-scoring case studies in the sample described both policy and implementation (42%), compared to just 5 out of 29 low-scoring case studies that included both policy and implementation (17%; the remainder described policy impacts only with no evidence of benefits arising from implementation). High- scoring case studies were more likely to cite evidence of impacts rather than just citing evidence pertaining to the pathway (which was more common in low-scoring cases). High-scoring policy case studies also provided evidence pertaining to the pathway, but because they typically also included evidence of policy change, this evidence helped attribute policy impacts to research.

Highly-rated case studies were easy to understand and well written

In preparation for the REF, many universities invested heavily in writing assistance (Coleman, 2019 ) to ensure that impact case studies were “easy to understand and evaluation-friendly” (Watermeyer and Chubb, 2018 ) for the assessment panels, which comprised academics and experts from other sectors (HEFCE, 2011 , p. 6). With this in mind, we investigated readability and style, both in the quantitative linguistic and in the qualitative thematic analysis.

High-scoring impact case studies scored more highly on the Flesch Reading Ease score, a readability measure based on the length of words and sentences. The scores in Table 11 are reported out of 100, with a higher score indicating that a text is easier to read. While the scores reveal a significant difference between 4* and 1*/2* impact case studies, they also indicate that impact case studies are generally on the verge of “graduate” difficulty (Hartley, 2016 , p. 1524). As such our analysis should not be understood as suggesting that these technical documents should be adjusted to the readability of a newspaper article, but they should be maintained at interested and educated non-specialist level.

Interestingly, there were differences between the main panels. Footnote 5 In Social Science and Humanities case studies (Main Panels C and D), high-scoring impact case studies scored significantly higher on reading ease than low-scoring ones. There was no significant difference in Main Panel A between 4* and 1*/2* cases. However, all Main Panel A case studies showed, on average, lower reading ease scores than the low-scoring cases in Main Panels C and D. This means that their authors used longer words and sentences, which may be explained in part by more and longer technical terms needed in Main Panel A disciplines; the difference between high- and low-scoring case studies in Main Panels C and D may be explained by the use of more technical jargon (confirmed in the qualitative analysis).

The Flesch Reading Ease measure assesses the sentence- and word-level, rather than capturing higher-level text-processing difficulty. While this is recognised as a reliable indicator of comparative reading ease, and the underlying measures of sentence-length and word-length are highly correlated with reading speed (Haberlandt and Graesser, 1985 ), Hartley ( 2016 ) is right in his criticism that the tool takes neither the meaning of the words nor the wider text into account. The Coh-Metrix tool (McNamara et al., 2014 ) provides further measures for reading ease based on textual cohesion in these texts compared to a set of general English texts. Of the eight principal component scores computed by the tool, most did not reveal a significant difference between high- and low-scoring case studies or between different Main Panels. Moreover, in most measures, impact case studies overall were fairly homogenous compared to the baseline of general English texts. However, there were significant differences between high- and low-scoring impact case studies in two of the measures: “deep cohesion” and “connectivity” (Table 12 ).

“Deep cohesion” shows whether a text makes causal connections between ideas explicit (e.g. “because”, “so”) or leaves them for the reader to infer. High-scoring case studies had a higher level of deep cohesion compared to general English texts (Graesser et al., 2011 ), while low-scoring case studies tended to sit below the general English average. In addition, Main Panel A case studies (Life Sciences), which received the lowest scores in Flesch Reading Ease, on average scored higher on deep cohesion than case studies in more discursive disciplines (Main Panel C—Social Sciences and Main Panel D—Arts and Humanities). “Connectivity” measures the level of explicit logical connectives (e.g. “and”, “or” and “but”) to show relations in the text. Impact case studies were low in connectivity compared to general English texts, but within each of the Main Panels, high-scoring case studies had more explicit connectivity than low-scoring case studies. This means that Main Panel A case studies, while using on average longer words and sentences as indicated by the Flesch Reading Ease scores, compensated for this by making causal and logical relationships more explicit in the texts. In Main Panels C and D, which on average scored lower on these measures, there was a clearer difference between high- and low-scoring case studies than in Main Panel A, with high-scoring case studies being easier to read.

Linked to this, low-scoring case studies across panels were more likely than high-scoring case studies to contain phrases linked to the research process (suggesting an over-emphasis on the research rather than the impact, and a focus on process over findings or quality; Table 18 ) and filler-phrases (Table 13 ).

High-scoring case studies were more likely to clearly identify individual impacts via subheadings and paragraph headings ( p  < 0.0001, with effect size measure Log Ratio 0.54). The difference is especially pronounced in Main Panel D (Log Ratio 1.53), with a small difference in Main Panel C and no significant difference in Main Panel A. In Units of Assessment combined in Main Panel D, a more discursive academic writing style is prevalent (see e.g. Hyland, 2002 ) using fewer visual/typographical distinctions such as headings. The difference in the number of headings used in case studies from those disciplines suggests that high-scoring case studies showed greater divergence from disciplinary norms than low-scoring case studies. This may have allowed them to adapt the presentation of their research impact to the audience of panel members to a greater extent than low-scoring case studies.

The qualitative thematic analysis of Impact Case Studies indicates that it is not simply the number of subheadings that matters, although this comparison is interesting especially in the context of the larger discrepancy in Main Panel D. Table 14 summarises formatting that was considered helpful and unhelpful from the qualitative analysis.

The observations in Tables 11 – 13 stem from quantitative linguistic analysis, which, while enabling statistical testing, does not show directly the effect of a text on the reader. When conducting the qualitative thematic analysis, we collected examples of formatting and stylistic features from the writing and presentation of high and low-scoring case studies that might have affected clarity of the texts (Tables 14 and 15 ). Specifically, 38% of low-scoring case studies made inappropriate use of adjectives to describe impacts (compared to 20% of high-scoring; Table 16 ). Inappropriate use of adjectives may have given an impression of over-claiming or created a less factual impression than case studies that used adjectives more sparingly to describe impacts. Some included adjectives to describe impacts in testimonial quotes, giving third-party endorsement to the claims rather than using these adjectives directly in the case study text.

Highly-rated case studies were more likely to describe underpinning research findings, rather than research processes

To be eligible, case studies in REF2014 had to be based on underpinning research that was “recognised internationally in terms of originality, significance and rigour” (denoted by a 2* quality profile, HEFCE, 2011 , p. 29). Ineligible case studies were excluded from our sample (i.e. those in the “unclassifiable” quality profile), so all the case studies should have been based on strong research. Once this research quality threshold had been passed, scores were based on the significance and reach of impact, so case studies with higher-rated research should not, in theory, get better scores on the basis of their underpinning research. However, there is evidence that units whose research outputs scored well in REF2014 also performed well on impact (unpublished Research England analysis cited in Hill, 2016 ). This observation only shows that high-quality research and impact were co-located, rather than demonstrating a causal relationship between high-quality research and highly rated impacts. However, our qualitative thematic analysis suggests that weaker descriptions of research (underpinning research was not evaluated directly) may have been more likely to be co-located with lower-rated impacts at the level of individual case studies. We know that the majority of underpinning research in the sample was graded 2* or above (because we excluded unclassifiable case studies from the analysis) but individual ratings for outputs in the underpinning research section are not provided in REF2014. Therefore, the qualitative analysis looked for a range of indicators of strong or weak research in four categories: (i) indicators of publication quality; (ii) quality of funding sources; (iii) narrative descriptions of research quality; and (iv) the extent to which the submitting unit (versus collaborators outside the institution) had contributed to the underpinning research. As would be expected (given that all cases had passed the 2* threshold), only a small minority of cases in the sample gave grounds to doubt the quality of the underpinning research. However, both our qualitative and quantitative analyses identified research-related differences between high- and low-scoring impact case studies.

Based on our qualitative thematic analysis of indicators of research quality, a number of low-scoring cases contained indications that underpinning research may have been weak. This was very rare in high-scoring cases. In the most extreme case, one case study was not able to submit any published research to underpin the impact, relying instead on having secured grant funding and having a manuscript under review. Table 17 describes indicators that underpinning research may have been weaker (presumably closer to the 2* quality threshold for eligibility). It also describes the indications of higher quality research (which were likely to have exceeded the 2* threshold) that were found in the rest of the sample. High-scoring case studies demonstrated the quality of the research using a range of direct and indirect approaches. Direct approaches included the construction of arguments that articulated the originality, significance and rigour of the research in the “underpinning research” section of the case study (sometimes with reference to outputs that were being assessed elsewhere in the exercise to provide a quick and robust check on quality ratings). In addition to this, a wide range of indirect proxies were used to infer quality, including publication venue, funding sources, reviews and awards.

These indicators are of particular interest given the stipulation in REF2021 that case studies must provide evidence of research quality, with the only official guidance suggesting that this is done via the use of indicators. The indicators identified in Table 17 overlap significantly with example indicators proposed by panels in the REF2021 guidance. However, there are also a number of additional indicators, which may be of use for demonstrating the quality of research in REF2021 case studies. In common with proposed REF2021 research quality indicators, many of the indicators in Table 17 are highly context dependent, based on subjective disciplinary norms that are used as short-cuts to assessments of quality by peers within a given context. Funding sources, publication venues and reviews that are considered prestigious in one disciplinary context are often perceived very differently in other disciplinary contexts. While REF2021 does not allow the use of certain indicators (e.g. journal impact factors), no comment is given on the appropriateness of the suggested indicators. While this may be problematic, given that an indicator by definition sign-posts, suggests or indicates by proxy rather than representing the outcome of any rigorous assessment, we make no comment on whether it is appropriate to judge research quality via such proxies. Instead, Table 17 presents a subjective, qualitative identification of indicators of high or low research quality, which were as far as possible considered within the context of disciplinary norms in the Units of Assessments to which the case studies belonged.

The quantitative linguistic analysis also found differences between the high-scoring and low-scoring case studies relating to underpinning research. There were significantly more words and phrases in low-scoring case studies compared to high-scoring cases relating to research outputs (e.g. “the paper”, “peer-reviewed”, “journal of”, “et al”), the research process (e.g. “research project”, “the research”, “his work”, “research team”) and descriptions of research (“relationship between”, “research into”, “the research”) (Table 18 ). The word “research” itself appears frequently in both (high: 91× per 10,000 words; low: 110× per 10,000 words), which is nevertheless a small but significant over-use in the low-scoring case studies (effect size measure log ratio = 0.27, p  < 0.0001).

There are two alternative ways to interpret these findings. First, the qualitative research appears to suggest a link between higher-quality underpinning research and higher impact scores. However, the causal mechanism is not clear. An independent review of REF2014 commissioned by the UK Government (Stern, 2016 ) proposed that underpinning research should only have to meet the 2* threshold for rigour, as the academic significance and novelty of the research is not in theory a necessary precursor to significant and far-reaching impact. However, a number of the indications of weaker research in Table 17 relate to academic significance and originality, and many of the indicators that suggested research exceeded the 2* threshold imply academic significance and originality (e.g. more prestigious publication venues often demand stronger evidence of academic significance and originality in addition to rigour). As such, it may be possible to posit two potential causal mechanisms related to the originality and/or significance of research. First, it may be argued that major new academic breakthroughs may be more likely to lead to impacts, whether directly in the case of applied research that addresses societal challenges in new and important ways leading to breakthrough impacts, or indirectly in the case of major new methodological or theoretical breakthroughs that make new work possible that addresses previously intractable challenges. Second, the highest quality research may have sub-consciously biased reviewers to view associated impacts more favourably. Further research would be necessary to test either mechanism.

However, these mechanisms do not explain the higher frequency of words and phrases relating to research outputs and process in low-scoring case studies. Both high-scoring and low-scoring cases described the underpinning research, and none of the phrases that emerged from the analysis imply higher or lower quality of research. We hypothesised that this may be explained by low-scoring case studies devoting more space to underpinning research at the expense of other sections that may have been more likely to contribute towards scores. Word limits were “indicative”, and the real limit of “four pages” in REF2014 (extended to five pages in REF2021) was operationalised in various way. However, a t -test found no significant difference between the underpinning research word counts (mean of 579 and 537 words in high and low-scoring case studies, respectively; p  = 0.11). Instead, we note that words and phrases relating to research in the low-scoring case studies focused more on descriptions of research outputs and processes rather than descriptions of research findings or the quality of research, as requested in REF2014 guidelines. Given that eligibility evidenced in this section is based on whether the research findings underpin the impacts and the quality of the research (HEFCE, 2011 ), we hypothesise that the focus of low-scoring case studies on research outputs and processes was unnecessary (at best) or replaced or obscured research findings (at worst). This could be conceptualised as another instance of the content/process distinction, whereby high-scoring case studies focused on what the research found and low-scoring case studies focused on the process through which the research was conducted and disseminated. It could be concluded that this tendency may have contributed towards lower scores if unnecessary descriptions of research outputs and process, which would not have contributed towards scores, used up space that could otherwise have been used for material that may have contributed towards scores.

Limitations

These findings may be useful in guiding the construction and writing of case studies for REF2021 but it is important to recognise that our analyses are retrospective, showing examples of what was judged to be ‘good’ and ‘poor’ practice in the authorship of case studies for REF2014. Importantly, the findings of this study should not be used to infer a causal relationship between the linguistic features we have identified and the judgements of the REF evaluation panel. Our quantitative analysis has identified similarities and differences in their linguistic features, but there are undoubtedly a range of considerations taken into account by evaluation panels. It is also not possible to anticipate how REF2021 panels will interpret guidance and evaluate case studies, and there is already evidence that practice is changing significantly across the sector. This shift in expectations regarding impact is especially likely to be the case in research concerned with public policy, which are increasingly including policy implementation as well as design in their requirements, and research involving public engagement, which is increasingly being expected to provide longitudinal evidence of benefits and provide evidence of cause and effect. We are unable to say anything conclusive from our sample about case studies that focused primarily on public engagement and pedagogy because neither of these types of impact were common enough in either the high-scoring or low-scoring sample to infer reliable findings. While this is the largest sample of known high-scoring versus low-scoring case studies ever analysed, it is important to note that this represents <3% of the total case studies submitted to REF2014. Although the number of case studies was fairly evenly balanced between Main Panels in the thematic analysis, the sample only included a selection of Units of Assessment from each Main Panel, where sufficient numbers of high and low-scoring cases could be identified (14 and 20 out of 36 Units of Assessment in the qualitative and quantitative studies, respectively). As such, caution should be taken when generalising from these findings.

This paper provides empirical insights into the linguistic differences in high-scoring and low-scoring impact case studies in REF2014. Higher-scoring case studies were more likely to have articulated evidence of significant and far-reaching impacts (rather than just presenting the activities used to reach intended future impacts), and they articulated clear evidence of causal links between the underpinning research and claimed impacts. While a cause and effect relationship between linguistic features, styles and the panel’s evaluation cannot be claimed, we have provided a granularity of analysis that shows how high-scoring versus low-scoring case studies attempted to meet REF criteria. Knowledge of these features may provide useful lessons for future case study authors, submitting institutions and others developing impact assessments internationally. Specifically, we show that high-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach, compared to low-scoring cases, which were more likely to provide less specific and lower-magnitude articulations of significance and reach. Lower-scoring case studies were more likely to focus on pathways to impact rather than articulating clear impact claims, with a particular focus on one-way modes of knowledge transfer. High-scoring case studies were more likely to provide clear links between underpinning research and impacts, supported by high-quality corroborating evidence, compared to low-scoring cases that often had missing links between research and impact and were more likely to be underpinned by corroborating evidence that was vague and/or not clearly linked to impact claims. Linked to this, high-scoring case studies were more likely to contain attributional phrases, and these phrases were more likely to attribute research and/or pathways to impact, compared to low-scoring cases, which contained fewer attributional phrases, which were more likely to provide attribution to pathways rather than impact. Furthermore, there is evidence that high-scoring case studies had more explicit causal connections between ideas and more logical connective words (and, or, but) than low-scoring cases.

However, in addition to the explicit REF2014 rules, which appear to have been enacted effectively by sub-panels, there is evidence that implicit rules, particularly linked to written style, may also have played a role. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than might be normally expected in academic writing (cf. e.g. Vold, 2006 ; Yang et al., 2015 ). Low-scoring case studies were more likely to contain filler phrases that could be described as “academese” (Biber and Gray, 2019 , p. 1), more likely to use unsubstantiated or vague adjectives to describe impacts, and were less likely to signpost readers to key points using sub-headings and paragraph headings. High-scoring case studies in two Main Panels (out of the three that could be analysed in this way) were significantly easier to read, although both high- and low-scoring case studies tended to be of “graduate” (Hartley, 2016 ) difficulty.

These findings suggest that aspects of written style may have contributed towards or compromised the scores of some case studies in REF2014, in line with previous research emphasising the role of implicit and subjective factors in determining the outcomes of impact evaluation (Derrick, 2018 ; Watermeyer and Chubb, 2018 ). If this were the case, it may raise questions about whether case studies are an appropriate way to evaluate impact. However, metric-based approaches have many other limitations and are widely regarded as inappropriate for evaluating societal impact (Bornmann et al., 2018 ; Pollitt et al., 2016 ; Ravenscroft et al., 2017 ; Wilsdon et al., 2015 ). Comparing research output evaluation systems across different countries, Sivertsen ( 2017 ) presents the peer-review-based UK REF as “best practice” compared to the metrics-based systems elsewhere. Comparing the evaluation of impact in the UK to impact evaluations in USA, the Netherlands, Italy and Finland, Derrick ( 2019 ) describes REF2014 and REF2021 as “the world’s most developed agenda for evaluating the wider benefits of research and its success has influenced the way many other countries define and approach the assessment of impact”.

We cannot be certain about the extent to which linguistic features or style shaped the judgement of REF evaluators, nor can such influences easily be identified or even consciously recognised when they are at work (cf. research on sub-conscious bias and tacit knowledge; the idea that “we know more than we can say”—Polanyi, 1958 cited in Goodman, 2003 , p. 142). Nonetheless, we hope that the granularity of our findings proves useful in informing decisions about presenting case studies, both for case study authors (in REF2021 and other research impact evaluations around the world) and those designing such evaluation processes. In publishing this evidence, we hope to create a more “level playing field” between institutions with and without significant resources available to hire dedicated staff or consultants to help write their impact case studies.

Data availability

The dataset analysed during the current study corresponds to the publicly available impact case studies defined through the method explained in Section “Research design and sample” and Table 2 . A full list of case studies included can be obtained from the corresponding author upon request.

https://impact.ref.ac.uk/casestudies/search1.aspx

For Main Panel B, only six high-scoring and two low-scoring case studies are clearly identifiable and available to the public (cf. Fig. 1 ). The Main Panel B dataset is therefore too small for separate statistical analysis, and no generalisations should be made on the basis of only one high-scoring and one low-scoring submission.

However, in the qualitative analysis, there were a similar number of high-scoring case studies that were considered to have reached this score due to a clear focus on one single, highly impressive impact, compared to those that were singled out for their impressive range of different impacts.

Note that there were more instances of the smaller number of attributional phrases in the low-scoring corpus.

For Main Panel B, only six high-scoring and two low-scoring case studies are clearly identifiable and available to the public. The Main Panel B dataset is therefore too small for separate statistical analysis, and no generalisations should be made on the basis of only one high-scoring and one low-scoring submission.

Anthony L (2014) AntConc, 3.4.4 edn. Waseda University, Tokyo

Google Scholar  

Auerbach CF, Silverstein LB (2003) Qualitative data: an introduction to coding and analyzing data in qualitative research. New York University Press, New York, NY

Back L (2015) On the side of the powerful: the ‘impact agenda’ and sociology in public. https://www.thesociologicalreview.com/on-the-side-of-the-powerful-the-impact-agenda-sociology-in-public/ . Last Accessed 24 Jan 2020

Biber D, Gray B (2019) Grammatical complexity in academic English: linguistic change in writing. Cambridge University Press, Cambridge

Bornmann L, Haunschild R, Adams J (2018) Do altmetrics assess societal impact in the same way as case studies? An empirical analysis testing the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF). J Informetr 13(1):325–340

Article   Google Scholar  

Bozeman B, Youtie J (2017) Socio-economic impacts and public value of government-funded research: lessons from four US National Science Foundation initiatives. Res Policy 46(8):1387–1398

Braun V, Clarke V (2006) Using thematic analysis in psychology. Quale Res Psychol 3(2):77–101

Brezina V (2018) Statistics in corpus linguistics: a practical guide. Cambridge University Press, Cambridge

Book   Google Scholar  

Chubb J (2017) Instrumentalism and epistemic responsibility: researchers and the impact agenda in the UK and Australia. University of York

Chubb J, Watermeyer R (2017) Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Stud High Educ 42(2):2360–2372

Chubb J, Reed MS (2017) Epistemic responsibility as an edifying force in academic research: investigating the moral challenges and opportunities of an impact agenda in the UK and Australia. Palgrave Commun 3:20

Chubb J, Reed MS (2018) The politics of research impact: academic perceptions of the implications for research funding, motivation and quality. Br Politics 13(3):295–311

Clark WC et al. (2016) Crafting usable knowledge for sustainable development. Proc Natl Acad Sci USA 113(17):4570–4578

Article   ADS   CAS   PubMed   Google Scholar  

Coleman I (2019) The evolution of impact support in UK universities. Cactus Communications Pvt. Ltd

Derrick G (2018) The evaluators’ eye: impact assessment and academic peer review. Palgrave Macmillan

Derrick G (2019) Cultural impact of the impact agenda: implications for social sciences and humanities (SSH) research. In: Bueno D et al. (eds.), Higher education in the world, vol. 7. Humanities and higher education: synergies between science, technology and humanities. Global University Network for Innovation (GUNi)

Derrick G et al. (2018) Towards characterising negative impact: introducing Grimpact. In: Proceedings of the 23rd international conference on Science and Technology Indicators (STI 2018). Centre for Science and Technology Studies (CWTS), Leiden, The Netherlands

Esfandiari R, Barbary F (2017) A contrastive corpus-driven study of lexical bundles between English writers and Persian writers in psychology research articles. J Engl Academic Purp 29:21–42

Goodman CP (2003) The tacit dimension. Polanyiana 2(1):133–157

Graesser AC, McNamara DS, Kulikowich J (2011) Coh-Metrix: providing multi-level analyses of text characteristics. Educ Res 40:223–234

Haberlandt KF, Graesser AC (1985) Component processes in text comprehension and some of their interactions. J Exp Psychol: Gen 114(3):357–374

Hardie A (2014) Statistical identification of keywords, lockwords and collocations as a two-step procedure. ICAME 35, Nottingham

Hartley J (2016) Is time up for the Flesch measure of reading ease? Scientometrics 107(3):1523–1526

HEFCE (2011) Assessment framework and guidance on submissions. Ref. 02.2011

Hill S (2016) Assessing (for) impact: future assessment of the societal impact of research. Palgrave Commun 2:16073

Hyland K (2002) Directives: argument and engagement in academic writing. Appl Linguist 23(2):215–238

Hyland K (2008) As can be seen: lexical bundles and disciplinary variation. Engl Specif Purp 27(1):4–21

Lemos MC (2015) Usable climate knowledge for adaptive and co-managed water governance. Curr Opin Environ Sustain 12:48–52

MacDonald R (2017) “Impact”, research and slaying Zombies: the pressures and possibilities of the REF. Int J Sociol Soc Policy 37(11–12):696–710

Mårtensson P et al. (2016) Evaluating research: a multidisciplinary approach to assessing research practice and quality. Res Policy 45(3):593–603

McEnery T, Xiao R, Tono Y (2006) Corpus-based language studies: an advanced resource book. Routledge, Abingdon

McNamara DS et al. (2014) Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press, New York, NY

National Science Foundation (2014) Perspectives on broader impacts

Nesta (2018) Seven principles for public engagement in research and innovation policymaking. https://www.nesta.org.uk/documents/955/Seven_principles_HlLwdow.pdf . Last Accessed 12 Dec 2019

Oancea A (2010) The BERA/UCET review of the impacts of RAE 2008 on education research in UK higher education institutions. ERA/UCET, Macclesfield

Oancea (2014) Research assessment as governance technology in the United Kingdom: findings from a survey of RAE 2008 impacts. Z Erziehungswis 17(S6):83–110

Olssen M, Peters MA (2005) Neoliberalism, higher education and the knowledge economy: from the free market to knowledge capitalism. J Educ Policy 20(3):313–345

Pidd M, Broadbent J (2015) Business and management studies in the 2014 Research Excellence Framework. Br J Manag 26:569–581

Pollitt A et al. (2016) Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers. BMJ Open 6(8):e010916

Article   PubMed   PubMed Central   Google Scholar  

Rau H, Goggins G, Fahy F (2018) From invisibility to impact: recognising the scientific and societal relevance of interdisciplinary sustainability research. Res Policy 47(1):266–276

Ravenscroft J et al. (2017) Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements. PLoS ONE 12(3):e0173152

Article   PubMed   PubMed Central   CAS   Google Scholar  

Rayson P, Garside R (2000) Comparing corpora using frequency profiling, Workshop on Comparing Corpora, held in conjunction with the 38th annual meeting of the Association for Computational Linguistics (ACL 2000), Hong Kong, pp. 1–6

Rayson P, Berridge D, Francis B (2004) Extending the Cochran rule for the comparison of word frequencies between corpora. In: Purnelle G, Fairon C, Dister A (eds.), Le poids des mots: Proceedings of the 7th international conference on statistical analysis of textual data (JADT 2004) (II). Presses universitaires de Louvain, Louvain-la-Neuve, Belgium, pp. 926–936

Reed MS (2018) The research impact handbook, 2nd edn. Fast Track Impact, Huntly, Aberdeenshire

Reed MS (2019) Book review: new book calls for civil disobedience to fight “dehumanising” impact agenda. Fast Track Impact

Reed MS et al. (under review) Evaluating research impact: a methodological framework. Res Policy

Rhoads R, Torres CA (2005) The University, State, and Market: The Political Economy of Globalization in the Americas. Stanford University Press, Stanford

Saldana J (2009) The Coding Manual for Qualitative Researchers. Sage, Thousand Oaks

Scott M (1997) PC analysis of key words—and key key words. System 25(2):233–245

Sivertsen G (2017) Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective. Palgrave Commun 3:17078

Smith S, Ward V, House A (2011) ‘Impact’ in the proposals for the UK’s Research Excellence Framework: shifting the boundaries of academic autonomy. Res Policy 40(10):1369–1379

Stern LN (2016) Building on success and learning from experience: an independent review of the Research Excellence Framework

Tsey K et al. (2016) Evaluating research impact: the development of a research for impact tool. Front Public Health 4:160

Vold ET (2006) Epistemic modality markers in research articles: a cross-linguistic and cross-disciplinary study. Int J Appl Linguist 16(1):61–87

Warry P (2006) Increasing the economic impact of the Research Councils (the Warry report). Research Council UK, Swindon

Watermeyer R (2019) Competitive accountability in academic life: the struggle for social impact and public legitimacy. Edward Elgar, Cheltenham

Watermeyer R, Hedgecoe A (2016) ‘Selling ‘impact’: peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. J Educ Policy 31:651–665

Watermeyer R, Chubb J (2018) Evaluating ‘impact’ in the UK’s Research Excellence Framework (REF): liminality, looseness and new modalities of scholarly distinction. Stud Higher Educ 44(9):1–13

Weinstein N et al. (2019) The real-time REF review: a pilot study to examine the feasibility of a longitudinal evaluation of perceptions and attitudes towards REF 2021

Wilsdon J et al. (2015) Metric tide: report of the independent review of the role of metrics in research assessment and management

Yang A, Zheng S, Ge G (2015) Epistemic modality in English-medium medical research articles: a systemic functional perspective. Engl Specif Purp 38:1–10

Download references

Acknowledgements

Thanks to Dr. Adam Mearns, School of English Literature, Language & Linguistics at Newcastle University for help with statistics and wider input to research design as a co-supervisor on the Ph.D. research upon which this article is based.

Author information

Authors and affiliations.

Newcastle University, Newcastle, UK

Bella Reichard, Mark S Reed & Andrea Whittle

University of York, York, UK

University of Leeds, Leeds, UK

Northumbria University, Newcastle, UK

Lucy Jowett & Alisha Peart

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mark S Reed .

Ethics declarations

Competing interests.

MR is CEO of Fast Track Impact Ltd, providing impact training to researchers internationally. JC worked with Research England as part of the Real-Time REF Review in parallel with the writing of this article. BR offers consultancy services reviewing REF impact case studies.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Reichard, B., Reed, M.S., Chubb, J. et al. Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Commun 6 , 31 (2020). https://doi.org/10.1057/s41599-020-0394-7

Download citation

Received : 10 July 2019

Accepted : 09 January 2020

Published : 25 February 2020

DOI : https://doi.org/10.1057/s41599-020-0394-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• AI Content Shield
• AI KW Research
• AI Assistant
• SEO Optimizer
• AI KW Clustering
• Customer reviews
• The NLO Revolution
• Press Center
• Help Center
• Content Resources
• Facebook Group

Writing a Comparative Case Study: Effective Guide

Table of Contents

As a researcher or student, you may be required to write a comparative case study at some point in your academic journey. A comparative study is an analysis of two or more cases. Where the aim is to compare and contrast them based on specific criteria. We created this guide to help you understand how to write a comparative case study . This article will discuss what a comparative study is, the elements of a comparative study, and how to write an effective one. We also include samples to help you get started.

What is a Comparative Case Study?

A comparative study is a research method that involves comparing two or more cases to analyze their similarities and differences . These cases can be individuals, organizations, events, or any other unit of analysis. A comparative study aims to gain a deeper understanding of the subject matter by exploring the differences and similarities between the cases.

Elements of a Comparative Study

Before diving into the writing process, it’s essential to understand the key elements that make up a comparative study. These elements include:

• Research Question : This is the central question the study seeks to answer. It should be specific and clear, and the basis of the comparison.
• Cases : The cases being compared should be chosen based on their significance to the research question. They should also be similar in some ways to facilitate comparison.
• Data Collection : Data collection should be comprehensive and systematic. Data collected can be qualitative, quantitative, or both.
• Analysis : The analysis should be based on the research question and collected data. The data should be analyzed for similarities and differences between the cases.
• Conclusion : The conclusion should summarize the findings and answer the research question. It should also provide recommendations for future research.

How to Write a Comparative Study

Now that we have established the elements of a comparative study, let’s dive into the writing process. Here is a detailed approach on how to write a comparative study:

Choose a Topic

The first step in writing a comparative study is to choose a topic relevant to your field of study. It should be a topic that you are familiar with and interested in.

Define the Research Question

Once you have chosen a topic, define your research question. The research question should be specific and clear.

Choose Cases

The next step is to choose the cases you will compare. The cases should be relevant to your research question and have similarities to facilitate comparison.

Collect Data

Collect data on each case using qualitative, quantitative, or both methods. The data collected should be comprehensive and systematic.

Analyze Data

Analyze the data collected for each case. Look for similarities and differences between the cases. The analysis should be based on the research question.

Write the Introduction

The introduction should provide background information on the topic and state the research question.

Write the Literature Review

The literature review should give a summary of the research that has been conducted on the topic.

Write the Methodology

The methodology should describe the data collection and analysis methods used.

Present Findings

Present the findings of the analysis. The results should be organized based on the research question.

Conclusion and Recommendations

Summarize the findings and answer the research question. Provide recommendations for future research.

Sample of Comparative Case Study

To provide a better understanding of how to write a comparative study , here is an example: Comparative Study of Two Leading Airlines: ABC and XYZ

Introduction

The airline industry is highly competitive, with companies constantly seeking new ways to improve customer experiences and increase profits. ABC and XYZ are two of the world’s leading airlines, each with a distinct approach to business. This comparative case study will examine the similarities and differences between the two airlines. And provide insights into what works well in the airline industry.

Research Questions

What are the similarities and differences between ABC and XYZ regarding their approach to business, customer experience, and profitability?

Data Collection and Analysis

To collect data for this comparative study, we will use a combination of primary and secondary sources. Primary sources will include interviews with customers and employees of both airlines, while secondary sources will include financial reports, marketing materials, and industry research. After collecting the data, we will use a systematic and comprehensive approach to data analysis. We will use a framework to compare and contrast the data, looking for similarities and differences between the two airlines. We will then organize the data into categories: customer experience, revenue streams, and operational efficiency.

After analyzing the data, we found several similarities and differences between ABC and XYZ. Similarities Both airlines offer a high level of customer service, with attentive flight attendants, comfortable seating, and in-flight entertainment. They also strongly focus on safety, with rigorous training and maintenance protocols in place. Differences ABC has a reputation for luxury, with features such as private suites and shower spas in first class. On the other hand, XYZ has a reputation for reliability and efficiency, with a strong emphasis on on-time departures and arrivals. In terms of revenue streams, ABC derives a significant portion of its revenue from international travel. At the same time, XYZ has a more diverse revenue stream, focusing on domestic and international travel. ABC also has a more centralized management structure, with decision-making authority concentrated at the top. On the other hand, XYZ has a more decentralized management structure, with decision-making authority distributed throughout the organization.

This comparative case study provides valuable insights into the airline industry and the approaches taken by two leading airlines, ABC and Delta. By comparing and contrasting the two airlines, we can see the strengths and weaknesses of each method. And identify potential strategies for improving the airline industry as a whole. Ultimately, this study shows that there is no one-size-fits-all approach to doing business in the airline industry. And that success depends on a combination of factors, including customer experience, operational efficiency, and revenue streams.

Wrapping Up

A comparative study is an effective research method for analyzing case similarities and differences. Writing a comparative study can be daunting, but proper planning and organization can be an effective research method. Define your research question, choose relevant cases, collect and analyze comprehensive data, and present the findings. The steps detailed in this blog post will help you create a compelling comparative study that provides valuable insights into your research topic . Remember to stay focused on your research question. And use the data collected to provide a clear and concise analysis of the cases being compared.

Abir Ghenaiet

Abir is a data analyst and researcher. Among her interests are artificial intelligence, machine learning, and natural language processing. As a humanitarian and educator, she actively supports women in tech and promotes diversity.

Explore All Write A Case Study Articles

How to write a leadership case study (sample) .

Writing a case study isn’t as straightforward as writing essays. But it has proven to be an effective way of…

• Write A Case Study

Top 5 Online Expert Case Study Writing Services 

It’s a few hours to your deadline — and your case study college assignment is still a mystery to you.…

Examples Of Business Case Study In Research

A business case study can prevent an imminent mistake in business. How? It’s an effective teaching technique that teaches students…

How to Write a Multiple Case Study Effectively

Have you ever been assigned to write a multiple case study but don’t know where to begin? Are you intimidated…

How to Write a Case Study Presentation: 6 Key Steps

Case studies are an essential element of the business world. Understanding how to write a case study presentation will give…

How to Write a Case Study for Your Portfolio

Are you ready to showcase your design skills and move your career to the next level? Crafting a compelling case…

The Biology Corner

Biology Teaching Resources

Case Study – Carrying Capacity and Rabbits

European settlers introduced rabbits to both England and Australia. In the absence of natural predators, rabbits were able to multiply rapidly and establish large populations. Rabbits are prolific breeders, with females capable of producing several litters of offspring each year. This high reproductive rate allows rabbit populations to grow exponentially under favorable conditions.

The case begins with the historic introduction of rabbits to England in the 12th century. Originally, humans imported rabbits for hunting as as a food source. The rabbit populations quickly grew out of control, damaging the ecosystem. For instance, native herbivores, like deer were impacted. Rabbits even caused erosion damage to soil, further limiting biodiversity.

Students learn that in most cases, animal populations have limiting factors that keep their numbers at a near stable level, the carrying capacity . Without limiting factors, the population can grow exponentially, as the rabbits did.

Next, students learn that the population of rabbits, having eaten all the vegetation, had a massive die off. Not all rabbits died, and their numbers recover quickly, known as a “Boom and Bust Cycle.” In both areas where rabbits were introduced, humans tried a number of measures to control the population. These included hunting and poisoning rabbits to reduce their numbers. However, none of the measures have worked well. In other areas, the myxoma virus reduced rabbit population numbers, though over time, the virus has become less effective. Go, evolution!

Finally, students examine a graph showing a logistic growth pattern. Limiting factors, such as food availability and predation, will keep the number of rabbits near the carrying capacity. In some areas of Europe, ecologists reintroduced the lynx, which is a natural predator of rabbits and helped to control the rabbit populations. A follow-up activity explores the lynx and conservation efforts.

Related Activities

Case Study – Loggerhead Turtles and Population Models  – explore the number of eggs hatched and survivorship

Ecology Case Study – The Wolves of Isle Royale  – population decline and reintroduction of wolves

Predator Prey Graph  – graph data on deer and wolf populations (growth curves)

Saving a Fussy Predator – reintroduction of the lynx to control rabbit populations

Rabbit Skeleton – model for use in comparative anatomy, showing teeth and skeletal features

Rabbit Digestive System – label the rabbit digestive system for comparative anatomy units

FDA officials offer advice on smooth BE studies, comparative analyses for combo products

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Case Study: How Aggressively Should a Bank Pursue AI?

• Thomas H. Davenport
• George Westerman

A Malaysia-based CEO weighs the risks and potential benefits of turning a traditional bank into an AI-first institution.

Siti Rahman, the CEO of Malaysia-based NVF Bank, faces a pivotal decision. Her head of AI innovation, a recent recruit from Google, has a bold plan. It requires a substantial investment but aims to transform the traditional bank into an AI-first institution, substantially reducing head count and the number of branches. The bank’s CFO worries they are chasing the next hype cycle and cautions against valuing efficiency above all else. Siti must weigh the bank’s mixed history with AI, the resistance to losing the human touch in banking services, and the risks of falling behind in technology against the need for a prudent, incremental approach to innovation.

Two experts offer advice: Noemie Ellezam-Danielo, the chief digital and AI strategy at Société Générale, and Sastry Durvasula, the chief information and client services officer at TIAA.

Siti Rahman, the CEO of Malaysia-headquartered NVF Bank, hurried through the corridors of the university’s computer engineering department. She had directed her driver to the wrong building—thinking of her usual talent-recruitment appearances in the finance department—and now she was running late. As she approached the room, she could hear her head of AI innovation, Michael Lim, who had joined NVF from Google 18 months earlier, breaking the ice with the students. “You know, NVF used to stand for Never Very Fast,” he said to a few giggles. “But the bank is crawling into the 21st century.”

• Thomas H. Davenport is the President’s Distinguished Professor of Information Technology and Management at Babson College, a visiting scholar at the MIT Initiative on the Digital Economy, and a senior adviser to Deloitte’s AI practice. He is a coauthor of All-in on AI: How Smart Companies Win Big with Artificial Intelligence (Harvard Business Review Press, 2023).
• George Westerman is a senior lecturer at MIT Sloan School of Management and a coauthor of Leading Digital (HBR Press, 2014).

Partner Center

Comprehensive evaluation of newly cultivated land sustainable utilization at project scale: A case study in Guangdong, China

• Research Articles
• Published: 17 April 2024
• Volume 34 , pages 745–762, ( 2024 )

Cite this article

• Chang Guo 1 , 2 ,
• Xiaobin Jin 1 , 2 , 3 ,
• Xuhong Yang 1 , 2 , 3 ,
• Weiyi Xu 1 , 2 ,
• Rui Sun 1 , 2 &
• Yinkang Zhou 1 , 2 , 3  

Cultivated land plays a pivotal role in ensuring national food security, particularly in populous nations like China, where substantial investments are made to develop cultivated land as a counterbalance to construction-occupied areas. Consequently, long-term, effective monitoring of the utilization of newly cultivated land becomes imperative. This study introduces a comprehensive monitoring framework, designed for refined scales, that leverages remote sensing data. The framework focuses on the sustainable utilization of newly cultivated land, emphasizing utilization sustainability, productivity stability, and landscape integration. Its effectiveness was validated through a case study in Guangdong province, China. The results revealed satisfactory utilization sustainability and improved productivity stability of newly cultivated land in Guangdong, though landscape integration showed sub-optimal results. Furthermore, the comprehensive evaluation categorized the newly cultivated land into three levels and eight types. The study recommends enhancing the site selection process for newly cultivated land and improving the long-term monitoring, as well as incentive and constraint mechanisms, for their utilization. This study can provide a scientific reference to bolster the implementation of cultivated land protection policies, thereby contributing significantly to high-quality economic and social development.

Article PDF

Download to read the full article text

Avoid common mistakes on your manuscript.

Alexander P, Prestele R, Verburg P H et al. , 2017. Assessing uncertainties in land cover projections. Global Change Biology , 23(2): 767–781.

Article   Google Scholar  

Bai Y, Liu Y S, Li Y H et al. , 2022. Land consolidation and eco-environmental sustainability in Loess Plateau: A study of Baota district, Shaanxi province, China. Journal of Geographical Sciences , 32(9): 1724–1744.

Bren d’Amour C, Reitsma F, Baiocchi G et al. , 2017. Future urban land expansion and implications for global croplands. Proceedings of the National Academy of Sciences of the United States of America , 114(34): 8939–8944.

Chen H, Tan Y Z, Xiao W et al. , 2022. Urbanization in China drives farmland uphill under the constraint of the requisition–compensation balance. Science of The Total Environment , 831: 154895.

Article   CAS   Google Scholar  

Chen W X, Ye X Y, Li J F et al. , 2019. Analyzing requisition–compensation balance of farmland policy in China through telecoupling: A case study in the middle reaches of Yangtze River Urban Agglomerations. Land Use Policy , 83: 134–146.

Dang Y X, Liao Y B, Kong X B et al. , 2022. Discussion on the top-level design and optimization of the system of cultivated land requisition–compensation balance. Natural Resource Economics of China , 35(6): 43–48, 88. (in Chinese)

Google Scholar  

Deng X Z, Huang J K, Rozelle S et al. , 2006. Cultivated land conversion and potential agricultural productivity in China. Land Use Policy , 23(4): 372–384.

Fan X J, Quan B, Deng Z W et al. , 2022. Study on land use changes in Changsha–Zhuzhou–Xiangtan under the background of cultivated land protection policy. Sustainability , 14(22): 15162.

Foley J A, DeFries R, Asner G P et al. , 2005. Global consequences of land use. Science , 309(5734): 570–574.

Guan X, Jin X B, Wei D Y et al. , 2014. Discussion on the construction of comprehensive monitoring system for land consolidation project. China Land Sciences , 28(4): 71–76. (in Chinese)

He J H, Guan X D, Yu Y, 2016. A modeling approach for farmland protection zoning considering spatial heterogeneity: A case study of E-zhou city, China. Sustainability , 8(10): 1052.

Huang H P, Chen W, Zhang Y et al. , 2021. Analysis of ecological quality in Lhasa Metropolitan Area during 1990–2017 based on remote sensing and Google Earth Engine platform. Journal of Geographical Sciences , 31(2): 265–280.

Kang L, Zhao R, Wu K N et al. , 2021. Impacts of farming layer constructions on cultivated land quality under the cultivated land balance policy. Agronomy , 11(12): 2403.

Kuang W H, Liu J Y, Tian H Q et al. , 2022a. Cropland redistribution to marginal lands undermines environmental sustainability. National Science Review , 9(1): 66–78.

Kuang W H, Zhang S W, Du G M et al. , 2022b. Monitoring periodically national land use changes and analyzing their spatiotemporal patterns in China during 2015–2020. Journal of Geographical Sciences , 32(9): 1705–1723.

Li H B, Jin X B, Liu J et al. , 2022. Analytical framework for integrating resources, morphology, and function of rural system resilience: An empirical study of 386 villages. Journal of Cleaner Production , 365: 132738.

Li J, Zhang S, Hu Y M et al. , 2021. Performance evaluation of cultivated land occupation compensation balance policy in Guangdong province from the perspective of time and space. Journal of Agricultural Resources and Environment , 38(6): 1064–1073. (in Chinese)

Liao W H, Jiang W G, Huang Z Q, 2022. Spatiotemporal variations of eco-environment in the Guangxi Beibu Gulf Economic Zone based on remote sensing ecological index and granular computing. Journal of Geographical Sciences , 32(9): 1813–1830.

Lichtenberg E, Ding C R, 2008. Assessing farmland protection policy in China. Land Use Policy , 25(1): 59–68.

Liu B, Zhou H T, Qi M J et al. , 2022. Requisition–compensation balance relief for hydraulic projects based on cultivated land quality improvement. Sustainability , 14(20): 13576.

Liu C Y, Song C Q, Ye S J et al. , 2023. Estimate provincial-level effectiveness of the arable land requisition-compensation balance policy in mainland of China in the last 20 years. Land Use Policy , 131: 106733.

Liu L, Liu Z J, Gong J Z et al. , 2019. Quantifying the amount, heterogeneity, and pattern of farmland: Implications for China’s requisition-compensation balance of farmland policy. Land Use Policy , 81: 256–266.

Liu L, Xu X L, Chen X, 2015. Assessing the impact of urban expansion on potential crop yield in China during 1990–2010. Food Security , 7: 33–43.

Liu X W, Zhao C L, Song W, 2017. Review of the evolution of cultivated land protection policies in the period following China’s reform and liberalization. Land Use Policy , 67: 660–669.

Liu Y S, Wang Y S, 2019. Rural land engineering and poverty alleviation: Lessons from typical regions in China. Journal of Geographical Sciences , 29(5): 643–657.

Lu X H, Zhang Y W, Zou Y C, 2021. Evaluation the effect of cultivated land protection policies based on the cloud model: A case study of Xingning, China. Ecological Indicators , 131: 108247.

Malakoff K L, Nolte C, 2023. Estimating the parcel-level impacts of agricultural conservation easements on farmland loss using satellite data in New England. Land Use Policy , 132: 106814.

Muruganantham P, Wibowo S, Grandhi S et al. , 2022. A systematic literature review on crop yield prediction with deep learning and remote sensing. Remote Sensing , 14(9): 1990.

Qie L, Pu L J, Tang P F et al. , 2023. Gains and losses of farmland associated with farmland protection policy and urbanization in China: An integrated perspective based on goal orientation. Land Use Policy , 129: 106643.

Seto K C, Ramankutty N, 2016. Hidden linkages between urbanization and food systems. Science , 352(6288): 943–945.

Shen X Q, Wang L P, Wu C F et al. , 2017. Local interests or centralized targets? How China’s local government implements the farmland policy of Requisition–Compensation Balance. Land Use Policy , 67: 716–724.

Shi W J, Tao F L, Liu J Y, 2013. Changes in quantity and quality of cropland and the implications for grain production in the Huang-Huai-Hai Plain of China. Food Security , 5(1): 69–82.

Skog K L, Steinnes M, 2016. How do centrality, population growth and urban sprawl impact farmland conversion in Norway? Land Use Policy , 59: 185–196.

Song W, Pijanowski B C, 2014. The effects of China’s cultivated land balance program on potential land productivity at a national scale. Applied Geography , 46: 158–170.

Song X Q, Ouyang Z, Li Y S et al. , 2012. Cultivated land use change in China, 1999–2007: Policy development perspectives. Journal of Geographical Sciences , 22(6): 1061–1078.

Su M, Guo R Z, Hong W Y, 2019. Institutional transition and implementation path for cultivated land protection in highly urbanized regions: A case study of Shenzhen, China. Land Use Policy , 81: 493–501.

Sun R, Sun P, Wu J X et al. , 2014. Effectiveness and limitations of cultivated land requisition-compensation balance policy in China. China Population, Resources and Environment , 24(3): 41–46. (in Chinese)

Tan M H, Li Y Y, 2019. Spatial and temporal variation of cropland at the global level from 1992 to 2015. Journal of Resources and Ecology , 10(3): 235.

Tan Y Z, Chen H, Lian K et al. , 2020. Comprehensive evaluation of cultivated land quality at county scale: A case study of Shengzhou, Zhejiang province, China. International Journal of Environmental Research and Public Health , 17(4): 1169.

van Vliet J, 2019. Direct and indirect loss of natural area from urban expansion. Nature Sustainability , 2(8): 755–763.

Wang J, Lin Y F, Zhai T L et al. , 2018. The role of human activity in decreasing ecologically sound land use in China. Land Degradation & Development , 29(3): 446–460.

Wang X M, Zhou D Y, Jiang G H et al. , 2023. How can the sustainable goal of cultivated land use in the Qinghai-Tibet Plateau be realized? Based on a research framework of cultivated land use patterns. Frontiers in Environmental Science , 11: 1134136.

Wang Y F, Cheng L L, Zheng Y, 2023. An adjusted landscape ecological security of cultivated land evaluation method based on the interaction between cultivated land and surrounding land types. Land , 12(4): 833.

Wei X, Liu L M, Zhang D X et al. , 2022. Performance evaluation on the policy of balancing paddy field occupation and reclamation in Hunan province. China Land Sciences , 36(1): 57–67. (in Chinese)

Yang H, Li X B, 2000. Cultivated land and food supply in China. Land Use Policy , 17(2): 73–88.

Yang J, Huang X, 2021. 30 m annual land cover and its dynamics in China from 1990 to 2019 (1.0.0) [Data set]. Zenodo.

Yang S L, Bai Y, Alatalo J M et al. , 2022. Spatial-temporal pattern of cultivated land productivity based on net primary productivity and analysis of influencing factors in the Songhua River basin. Land Degradation & Development , 33(11): 1917–1932.

Ye S J, Ren S Y, Song C Q et al. , 2022. Spatial patterns of county-level arable land productive-capacity and its coordination with land-use intensity in mainland of China. Agriculture, Ecosystems & Environment , 326: 107757.

Zhang B L, Sun P L, Jiang G H et al. , 2019. Rural land use transition of mountainous areas and policy implications for land consolidation in China. Journal of Geographical Sciences , 29(10): 1713–1730.

Zhang M Y, Chen Q X, Zhang K W, 2021. Influence of the variation in rural population on farmland preservation in the rapid urbanization area of China. Journal of Geographical Sciences , 31(9): 1365–1380.

Zhao C, Zhou Y, Li X G et al. , 2018. Assessment of cultivated land productivity and its spatial differentiation in Dongting Lake region: A case study of Yuanjiang city, Hunan province. Sustainability , 10(10): 3616.

Zheng W W, Li S M, Ke X L et al. , 2022. The impacts of cropland balance policy on habitat quality in China: A multiscale administrative perspective. Journal of Environmental Management , 323: 116182.

Zhou X, Chen W, Wang Y N et al. , 2019. Suitability evaluation of large-scale farmland transfer on the Loess Plateau of northern Shaanxi, China. Land Degradation & Development , 30(10): 1258–1269.

Zhou Y, Li X H, Liu Y S, 2021. Cultivated land protection and rational use in China. Land Use Policy , 106: 105454.

Zhu H Y, Li X B, 2003. Discussion on the index method of regional land use change. Journal of Geographical Sciences , 13(5): 643–650. (in Chinese)

Zhuang D F, Liu J Y, 1997. Study on the model of regional differentiation of land use degree in China. Journal of Natural Resources , 12(2): 10–16. (in Chinese)

Download references

Author information

Authors and affiliations.

School of Geography and Ocean Science, Nanjing University, Nanjing, 210023, China

Chang Guo, Xiaobin Jin, Xuhong Yang, Weiyi Xu, Rui Sun & Yinkang Zhou

Key Laboratory of Coastal Zone Exploitation and Protection, Ministry of Land and Resources, Nanjing, 210023, China

Jiangsu Land Development and Consolidation Technology Engineering Center, Nanjing, 210023, China

Xiaobin Jin, Xuhong Yang & Yinkang Zhou

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Xiaobin Jin .

Additional information

Foundation: National Natural Science Foundation of China, No.42271259

Author: Guo Chang, specialized in sustainable land use.

Rights and permissions

Reprints and permissions

About this article

Guo, C., Jin, X., Yang, X. et al. Comprehensive evaluation of newly cultivated land sustainable utilization at project scale: A case study in Guangdong, China. J. Geogr. Sci. 34 , 745–762 (2024). https://doi.org/10.1007/s11442-024-2225-z

Download citation

Received : 19 July 2023

Accepted : 04 January 2024

Published : 17 April 2024

Issue Date : April 2024

DOI : https://doi.org/10.1007/s11442-024-2225-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• land consolidation
• newly cultivated land
• utilization sustainability
• productivity stability
• landscape integration
• Find a journal
• Publish with us
• Track your research