literature review for academic success

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Academic achievement prediction in higher education through interpretable modeling

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Software, Supervision, Validation, Writing – original draft, Writing – review & editing

Affiliation School of Foreign Languages, Wuhan Business University, Wuhan, Hubei, People’s Republic of China

Roles Investigation, Software, Writing – review & editing

* E-mail: [email protected]

• Sixuan Wang, 

• Published: September 5, 2024
• https://doi.org/10.1371/journal.pone.0309838
• Reader Comments

Student academic achievement is an important indicator for evaluating the quality of education, especially, the achievement prediction empowers educators in tailoring their instructional approaches, thereby fostering advancements in both student performance and the overall educational quality. However, extracting valuable insights from vast educational data to develop effective strategies for evaluating student performance remains a significant challenge for higher education institutions. Traditional machine learning (ML) algorithms often struggle to clearly delineate the interplay between the factors that influence academic success and the resulting grades. To address these challenges, this paper introduces the XGB-SHAP model, a novel approach for predicting student achievement that combines Extreme Gradient Boosting (XGBoost) with SHapley Additive exPlanations (SHAP). The model was applied to a dataset from a public university in Wuhan, encompassing the academic records of 87 students who were enrolled in a Japanese course between September 2021 and June 2023. The findings indicate the model excels in accuracy, achieving a Mean absolute error (MAE) of approximately 6 and an R-squared value near 0.82, surpassing three other ML models. The model further uncovers how different instructional modes influence the factors that contribute to student achievement. This insight supports the need for a customized approach to feature selection that aligns with the specific characteristics of each teaching mode. Furthermore, the model highlights the importance of incorporating self-directed learning skills into student-related indicators when predicting academic performance.

Citation: Wang S, Luo B (2024) Academic achievement prediction in higher education through interpretable modeling. PLoS ONE 19(9): e0309838. https://doi.org/10.1371/journal.pone.0309838

Editor: Shahid Akbar, Abdul Wali Khan University Mardan, PAKISTAN

Received: May 30, 2024; Accepted: August 20, 2024; Published: September 5, 2024

Copyright: © 2024 Wang, Luo. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: Fund recepient:Sixuan Wang Funder name: Hubei Provincial Department of Education Grant No: 2022GB087 Project name: A Study on the Curriculum Connection between College Japanese and High School Japanese from the Perspective of Core Literacy. https://jyt.hubei.gov.cn/ The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Context and motivation.

Academic achievement is of paramount importance in educational contexts, serving as a key indicator of both learning ability and the effectiveness of school administration and teaching standards [ 1 ]. The prediction of academic achievement is a continuously evolving topic in educational management. The integration of predictive models in education empowers educators to make well-informed choices, offer specific support, and enhance teaching strategies, thereby improving student learning outcomes [ 2 ].

Previous research on achievement prediction primarily utilized statistical analysis methods to process data and forecast outcomes, with data mainly derived from educational management systems, student identification cards, or surveys [ 3 ]. ML techniques, known for their ability to tackle complex, nonlinear problems without presuppositions, are adept at identifying connections between various parameters [ 4 ]. The state-of-the-art ML techniques for prediction [ 5 ] include K-Nearest Neighbors (KNN), Decision Trees, Random Forests (RF), Support Vector Machines (SVM), Neural Networks, and Naive Bayes. Recent scholarly efforts, both domestically and internationally, have been geared towards increasing the precision of student achievement predictions through technological innovations in algorithms [ 6 – 8 ].

Despite these developments, challenges remain in the domain of achievement prediction. A primary issue is the limited alignment between the outcomes produced by ML algorithms and the foundational principles of education and instruction, leading to hesitancy among educators in relying on these models. Additionally, there is a gap in thorough data analysis, examination of relationships, and investigation into variables that impact student academic performance patterns.

Contribution of the study

In addressing these challenges, our study delivers distinctive contributions to the field of interpretable machine learning within the context of higher education. We delineate these contributions as follows:

• Theoretical contribution: this study introduces ML models coupled with game theory-based SHAP analysis which aims to develop and validate the XGB-SHAP model, a novel approach for interpreting machine learning-based predictions of student achievement, and explore its efficacy across various teaching modalities.
• Practical contributions: It evaluates the significance of different indicators and their positive or negative impacts on prediction outcomes, thus shedding light on the educational implications of achievement prediction models. The findings of this study provide empirical data support for teachers and educators, facilitating the refinement of their instructional strategies.
• Comparative analysis: It explores student achievement prediction models in three distinct educational settings: online, offline, and blended teachings. This exploration reveals variances in teaching patterns across these modes, yielding practical advice for educators in applying these prediction models.

Structure of the article

This paper is organized as follows: Section ‘Literature review’ presents a review of related literatures, providing a comprehensive review of the existing literature on student achievement prediction, examines the prevailing issues and identifies the gaps within the current body of research. Section ‘Methodology’ details the methodology employed in this study, introduces the interpretable performance prediction framework and the indicators system used in this paper and outlines the methodology used to conduct the data analysis for this paper. The findings and their implications are discussed in Sections ‘Case study’ and ‘Results’ respectively. The paper concludes with a summary of our key findings in the final Section ‘Discussion and Conclusions’. Table 1 illustrates the list of abbreviations.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0309838.t001

Literature review

Previous research, student achievement prediction indicators..

Prediction accuracy largely depends on the careful selection of indicators. The initial and most critical step is the selection of appropriate input data. Previous research has identified three key groups of student-related features as pertinent input parameters: historical student performance, student engagement, and demographic data (Tomasevic et al., 2020).

Historical student performance has been consistently identified as a reliable predictor. For instance, DeBerard et al. [ 9 ] demonstrated that high school GPA is a strong predictor of college academic success. Similarly, Shaw et al. [ 10 ] found that combined SAT scores explain about 28% of the variance in first-year college GPA. Moreover, test scores have been used to predict future academic performance in various studies [ 11 ].

Regarding student engagement, a notable correlation with academic achievement has been observed [ 12 ]. Hussain et al. [ 13 ] identified a moderately strong positive correlation between student engagement and academic achievement. With evolving teaching formats like Massive Open Online Courses and the flipped classrooms, several studies have developed predictive models by analyzing student behaviors in learning management systems, such as video interactions, assignment submissions, and forum discussions [ 14 ]. With the innovation of modern educational technology tools, including artificial intelligence tools (such as ChatGPT) and virtual reality, significant roles have been played in enhancing student learning outcomes by integrating with educational theories like constructivism, experiential learning, and collaborative learning. These technologies, by offering immersive and interactive learning experiences, have increased student engagement, motivation, and critical thinking skills, thereby positively impacting academic performance [ 15 , 16 ].

Studies have also considered demographic factors. Research indicates that demographic factors play a moderate role in predictive accuracy, with relevance around 60% in some studies, while others suggest that these variables have a limited impact on prediction precision [ 5 , 17 ]. Additional indicators, such as student collaboration, teacher-student communication, and psychological factors like motivation and attitude, have also been explored. Recent studies emphasize the importance of considering learners’ psychological well-being and cognitive processes in educational settings [ 18 , 19 ].These motivational and coping strategies remarkably influence students’ learning approaches and overall educational outcomes [ 20 ].

The above discussion shows that student achievement is a composite of cognitive, behavioral, skill-based, and emotional outcomes derived from educational experiences [ 21 ]. Although there is a consensus on the selection of certain important indicators, the selection of the dataset for student achievement prediction varies from study to study. Selecting the most suitable dataset depends largely on the specific goals and objectives of the researchers, with no universally accepted guidelines.

Student achievement prediction models.

Originally, conventional statistical methods such as Discriminant Analysis and Multiple Linear Regression were the predominant approaches in the early stages of educational research [ 22 ]. Furthermore, Structural Equation Modeling (SEM) has been widely adopted in the social sciences. However, these traditional methods have often fallen short of delivering consistent and precise predictions or classifications [ 23 ].

Recently, an array of machine learning algorithms has been employed, including Multiple Regression, Probabilistic and Logistic Regression, Neural Networks, Decision Trees, Random Forests (RF), Genetic Algorithms, and Bayesian algorithms. These have shown varied levels of success in achieving high predictive accuracy [ 24 ]. Comparative studies of machine learning methods have been conducted, with Caruana et al. [ 25 ] exploring the performance evaluation of these models. Their research underscores a fundamental point: no single model or method universally excels across all problems and metrics. Tomasevic et al. [ 5 ] used the Open University Learning Analytics Dataset for a regression problem, finding that Artificial Neural Networks (ANN) and Decision Trees were the most effective, while KNN, SVM, and Bayesian linear regression were less successful.

While previous approaches using machine learning models for predicting student achievement have focused on model optimization [ 26 ], there are growing concerns regarding the opaque nature of complex models, which may hinder their broader application [ 27 ].

Interpretable machine learning models.

Nowadays, with the rapid development of artificial intelligence (AI) technology, ML models are being applied in many critical fields, such as education [ 28 , 29 ], healthcare [ 30 – 32 ]. However, as the number of parameters soars, the ’black-box’ nature of neural networks has raised concerns. Interpretable machine learning is a promising tool to alleviate concerns regarding the opacity of machine learning models. It equips ML models with the capability to articulate their processes in a manner comprehensible to humans [ 33 ].

Broadly, interpretable machine learning methods are divided into two categories: self-interpretation models and post-hoc interpretation methods [ 34 ]. Self-interpreting models typically have a simpler structure and include Linear models, Logistic Regression, and Decision Trees. Post-hoc interpretation methods involve either model-independent or model-specific techniques, applicable to various models but may require additional computational resources and analytical expertise.

Post-hoc or model-independent interpretation methods are extensively used in different scenarios. These include Partial Dependence Plot [ 35 ], Individual Conditional Expectation [ 36 ], Permutation Feature Importance [ 37 ], Local Interpretable Model-agnostic Explanations, and the SHAP method. The survey in the field of information resource management revealed that 83.7% of explainable ML applications utilize post-hoc explanation methods, with SHAP (51.2%) and feature importance analysis (34.1%) being the most common. Unlike traditional feature importance which indicates the significance of features without clarifying their impact on predictions, SHAP offers detailed explanations on both sample and feature levels through various visualizations like waterfall diagrams and feature dependency diagrams.

These interpretative approaches have been applied in diverse fields such as medicine, policymaking, and science, aiding in auditing predictions under circumstances like regulatory pressures and the pursuit of fairness [ 35 ]. However, the critical aspect of interpretability in machine learning models within the domain of educational management research remains underexplored.

Research gap

Given the aforementioned limitations, the interpretability of ML is a contentious issue. The various ML algorithms employed often fail to effectively elucidate the relationship between factors influencing students’ academic performance and their grades. Additionally, they struggle to quantify the impact of each feature on the target value and to determine the positive or negative influence of each characteristic. To address these gaps in the literature, our study delves into the following areas:

• Feature Importance Analysis: Our research will quantify the influence of each feature on the prediction of student performance. This involves a detailed examination of the weight and significance of various factors in determining academic outcomes.
• Impact Assessment: We will assess the positive or negative impact of each feature on the target variable. This is crucial for understanding not only the magnitude of the influence but also its direction.
• Model Comparison: By comparing the interpretability and performance of different ML models, our study seeks to identify the most effective approaches for student achievement prediction.
• Practical Implications: We will discuss the practical implications of our findings, focusing on how increased interpretability can enhance educational practices and inform policy-making.

Through this comprehensive approach, our study seeks to bridge the gap in the current research by providing a clearer understanding of the mechanisms behind student achievement prediction models and their implications for educational stakeholders.

Methodology

Development of an interpretable performance prediction framework.

As shown in Fig 1 , we have developed an interpretable framework for performance prediction. The framework’s core involves extracting five key features: academic factors, student engagement, demographic factors, psychological aspects, and self-directed learning abilities. These features form an input vector that accurately represents factors relevant to achievement prediction. The data for this study is sourced from three main systems: the Education Administration System (EAS), the Chaoxing Xuexitong System, and various questionnaires.

https://doi.org/10.1371/journal.pone.0309838.g001

The methodology progresses in three phases. The initial phase involves creating an indicator system from these features. In the subsequent phase, we focus on constructing and elucidating performance prediction models. Four different ML algorithms are applied to our “learning” dataset. Their effectiveness is evaluated using two standard ML metrics: Mean Absolute Error (MAE) and R-squared ( R 2 ). The optimal model is then selected based on these evaluations. The final stage of our methodology is the model interpretability phase, which accounts for the educational significance of the model by analyzing the importance and directional influence of the indicators. This phase aims to provide educators with insights to refine their teaching strategies.

Development of the indicator system

As mentioned in ‘Literature review’ section, prior research insights advocate categorizing student-related features into historical student performance, engagement, and demographic data [ 5 ]. To capture a holistic view of learner characteristics, we have expanded this system to include psychological factors and self-directed learning capabilities to form a student achievement prediction indicator system, as shown in Table 2 . Considering the minimal variation in age, gender, and other demographic factors in our case study, we have chosen to focus solely on the major as the demographic data point.

https://doi.org/10.1371/journal.pone.0309838.t002

Model training

As SHAP is a model-agnostic interpretation framework, which enables it to be applied across a spectrum of common predictive models. This versatility allows SHAP to provide insights into the decision-making process of these models by quantifying the contribution of each feature to the prediction, thereby enhancing our understanding of the model’s behavior regardless of its underlying structure or algorithmic approach. Commonly used ML models for academic achievement prediction include RF, BPNN, SVM, and XGboost. The rationale for selecting these four models is their proficiency as data-driven prediction methods. RF, an ensemble learning technique, amalgamates numerous decision trees, thereby reducing variance relative to individual trees. It is known for its superior average prediction performance. BPNN, a supervised learning algorithm, builds multi-layer neural networks inspired by biological neurons and employs a back-propagation algorithm for training, excelling in handling non-linear relationships and high-dimensional data. SVM has gained recognition for its effectiveness in classification, regression, and time-series prediction. XGBoost, enhancing the Gradient Boosting Decision Tree algorithm, stands out for its accuracy and flexibility.

In this research, a 5-fold cross-validation approach was implemented to fine-tune the hyperparameter to avoid overfit, optimizing them according to the mean value derived from each test set.

Model interpretability

Addressing the opaque nature of ML models, our research employs the SHAP method for interpretability. Developed by Lundberg and Lee in 2017 [ 39 ], SHAP merges various existing approaches to provide a reliable and intuitive explanation of model predictions. It does so by illustrating how predictions shift when certain variables are omitted. The Python SHAP package ( https://github.com/slundberg/shap ), enables the calculation of SHAP values for any selected model, and it is extensively utilized due to its versatility.

SHAP is characterized by three fundamental properties: local accuracy (the sum of feature attributions equals the model output), missingness (zero attribution for non-present features), and consistency (no decrease in feature attribution despite an increased marginal contribution). A notable advantage of SHAP is its model-agnostic nature, making it applicable to any machine learning model.

Data for this study was obtained from the EAS of a Wuhan-based public university. This system provided access to students’ personal information, such as majors and academic grades. In addition, we gathered course-related learning data from the Chaoxing Xuexitong system, a widely used online education platform in China. To obtain data on self-study hours, learning attitudes, and self-directed learning indicators, we employed questionnaires as the methodological instrument. The learning attitude questionnaire adapted from the English-learning Motivation Scale developed by a Chinese scholar Meihua Liu [ 40 ] who is from Tsinghua University, a tool commonly utilized in in EFL teaching and learning in the Chinese context. For assessing self-directed learning capabilities, we used a questionnaire adapted from Jinfen Xu ‘s [ 41 ] self-directed learning capability scale. These questionnaires were administered in class under instructor supervision and lasted approximately 10 minutes each, aiming to evaluate students’ learning attitudes and their aptitude for independent learning. The surveys were conducted midway through each semester. Our dataset encompasses data from 87 students enrolled in the Japanese course for the class of 2021, spanning three different learning modes. It includes nine indicators linked to student grades, amounting to a total of 2349 data entries. Table 3 shows the types of nine indicators.

https://doi.org/10.1371/journal.pone.0309838.t003

While analyzing the datasets, an imbalanced data pattern was noted. To address this, we grouped students into three broad specialty categories: Arts, Science and Technology, and Arts and Sports. This categorization reduced data sparsity by assigning discrete values (1, 2, 3) to these groups.

Ethical considerations

The study was approved by the institutional review board, and the study runs from September 2021 to June 2023. All participants were not at risk if they chose or declined to participate. Parental consent is not required for undergraduate students participating in the study. Additionally, we explained the purpose of the study in the questionnaire, clarified that it was their right to participate or not to participate in the study, and informed all the participants that ‘submitting answers’ is considered informed consent for researchers to use their questionnaire responses and related data retrieved from EAS and Chaoxing platform in publications of the research.

Experimental setup

In this study, we conducted experiments employed PyCharm version 2022.3.3 as the compilation software, and implemented the algorithmic model using Python. The dataset was randomly partitioned into training and test sets in a 4:1 ratio for robust training and evaluation.

As state in the Methodology Section, we employ four classic ML models as our predictive model for academic performance. Table 4 presents the pseudo-code outlining the experimental procedures.

https://doi.org/10.1371/journal.pone.0309838.t004

Comparison of models

To obtain the optimal model parameters, the hyperparameters of the aforementioned four models were optimized separately. Table 5 displays the optimal hyperparameter combinations for the aforementioned four models.

https://doi.org/10.1371/journal.pone.0309838.t005

Table 6 presents the comparison of the task performance of four models. Both BPNN and XGBoost show higher task performance compared to RF, while SVM lags in terms of task performance. The comparison indicates that XGBoost slightly surpasses BPNN, establishing XGBoost as the model with the best predictive performance. Therefore, this study selects the XGBoost model to fit all the data. SHAP values are used for interpretation.

https://doi.org/10.1371/journal.pone.0309838.t006

Exploratory analysis utilizing XGBoost and SHAP

Given the effectiveness of the XGBoost model, it was selected for further analysis using SHAP to explore teaching patterns within the model across various teaching modes. SHAP offers insights into the influence of each indicator per sample, highlighting both positive and negative effects. In the associated figures, color coding is used to represent the magnitude of eigenvalues, with red indicating high values and blue representing low values.

Figs 2 and 3 shows the importance of indicators and a summary plot for offline teaching. The average SHAP value (horizontal axis) indicates the significance of each indicator, with their order of importance shown on the vertical axis in Fig 2 . Key findings include classroom performance, previous exam grades, and student major as the most influential indicators. The impact of eigenvalues on each sample is depicted in Fig 3 , where each row represents an indicator, each dot signifies a sample, and the SHAP value is plotted on the horizontal axis. Further analysis revealed a positive relationship between prior exam grades, self-directed learning ability, learning attitudes, and their effect on academic achievement predictions. Interestingly, occasional absences did not show a substantial negative influence on predicted grades, hinting at a divergence in the dynamics of college classrooms from high school settings. This might be attributed to the independent learning skills prevalent among college students. Moreover, it was noted that students majoring in Arts and Sports tend to have a slightly negative impact on predicted grades.

https://doi.org/10.1371/journal.pone.0309838.g002

https://doi.org/10.1371/journal.pone.0309838.g003

Analysis of online teaching using XGBoost and SHAP

Figs 4 and 5 presents the indicator importance and summary plot for online teaching. A key observation is the increased influence of previous exam grades on the predicted values in comparison to offline settings. This suggests that students with a strong academic foundation tend to be more self-directed, thereby enhancing their predicted performance more remarkably. The disparity in self-directed learning abilities is more evident in online courses, highlighting the detrimental effect of inadequate self-learning skills on performance. Students struggling with self-learning might not receive timely support, leading to poorer outcomes. In this context, classroom performance becomes a less critical predictor, and the influence of a student’s major on predicted scores also diminishes. Interestingly, self-study time shows a positive correlation with predicted grades, while the relationship between quiz scores and performance prediction remains insignificant.

https://doi.org/10.1371/journal.pone.0309838.g004

https://doi.org/10.1371/journal.pone.0309838.g005

Blended teaching: Insights from XGBoost and SHAP

Figs 6 and 7 examines the indicator importance and summary plot for blended teaching. In this teaching mode, the impact of self-directed learning skills is more notable compared to other teaching methods, possibly due to the adoption of flipped classroom techniques. Self-directed learning shows a stronger positive correlation with both previous exam grades and quiz scores. Furthermore, the relevance of attitude towards learning is accentuated, suggesting its growing importance in blended learning environments where independent study is emphasized.

https://doi.org/10.1371/journal.pone.0309838.g006

https://doi.org/10.1371/journal.pone.0309838.g007

Discussion and conclusions

The prediction of academic achievement in higher education has become an increasingly prominent topic within the field of education [ 42 ]. In today’s information age, the tremendous growth of educational institutions’ electronic data “…can be utilized for discovering unknown patterns and trends” [ 43 ].Recent researches on predicting student performance are frequently spearheaded by educators identifying as "AI" educators to identify features that can be used to make predictions [ 44 ], to identify algorithms that can improve predictions [ 45 ], and to quantify aspects of student performance. However, analyzing performance, providing high-quality education strategies for evaluating the students’ performance from these abundant resources are among the prevailing challenges universities face [ 46 ].

In this research, we have developed the XGB-SHAP model, integrating XGBoost with SHAP, to systematically explore the relationship between grade prediction and diverse indicators across various teaching methods. Focused on university Japanese language classes, our study demonstrated XGBoost’s superior performance over other models, as evidenced by R 2 and MAE metrics. The integration of SHAP offered a clear visual representation, highlighting the mode and directional influence of each indicator and sheds light on the educational implications of ML structures in pedagogy. The study also supported that the XGB-SHAP model can be effectively used in the field of educational management research.

The results reveal that, the study of student achievement prediction, using student-related features, such as student historical achievement, student engagement and demographic data, which have been used as important input features in the previous literature, is not sufficient. With the development of society and the diversification of teaching and learning modes, the importance of self-directed learning skills in the prediction of university students’ performance has been demonstrated in this study. Psychological factors such as attitude towards learning should also be taken into account. The impact of a student’s major on foreign language learning is considerable, which indicate differences in learning environments, cultural factors, motivation to learn foreign languages. While classroom response accuracy and attendance appeared less critical. This suggests a potential shift in focus within higher education classrooms, advocating for a tailored approach to characteristic selection based on teaching modes. This methodology provides educators with a quantitative view of how educational processes affect student achievement.

Our study also shows that the factors influencing student performance vary: offline teaching values classroom performance, while online teaching and blended teaching emphasize independent learning. In blended teaching, quiz scores have a remarkable positive impact, differing from the trends in other modes. This could be attributed to quizzes acting as formative assessments in blended learning, enhancing student participation and providing continual feedback. Consequently, teaching strategies and support systems should be adapted to meet the distinct needs of each teaching mode to optimize learning outcomes.

Acknowledging the formidable technical challenges associated with interpretable machine learning models in practical educational contexts, it is imperative to recognize their substantial contributions in enhancing our comprehension and utility of achievement prediction models. Additionally, they play a pivotal role in mitigating the skepticism harbored by educators towards machine learning models deployed for achievement prediction. Moving forward, there exist several promising avenues for exploration within the realm of interpretable machine models that merit thorough investigation: first, expand the dataset to cover more academic areas, different institutions, and varied student groups. This will test the model’s effectiveness in diverse settings. Second, the refinement and augmentation of existing interpretable models to enhance their accuracy and utility. These directions offer promising avenues for furthering the application and acceptance of interpretable machine learning in educational settings.

Supporting information

S1 file. original data..

https://doi.org/10.1371/journal.pone.0309838.s001

• View Article
• Google Scholar
• PubMed/NCBI
• 20. Boekaerts M, Niemivirta M. Self-regulated learning: Finding a balance between learning goals and ego-protective goals. Handbook of self-regulation. Academic Press, 2000.
• 23. Kyndt E, Musso M, Cascallar E, et al. Predicting academic performance: The role of cognition, motivation and learning approaches. A neural network analysis.Methodological challenges in research on student learning. Antwerp: Garant, 2015.
• 26. Shalev-Shwartz S, Ben-David S. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
• 42. Hellas A, Ihantola P, Petersen A, et al. Predicting academic performance: a systematic literature review[C]//Proceedings companion of the 23rd annual ACM conference on innovation and technology in computer science education. 2018: 175-199.doi:10.1145/3293881.3295783.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Defining and Measuring Academic Success

2015, Practical Assessment, Research and Evaluation

Despite, and perhaps because of its amorphous nature, the term ‘academic success’ is one of the most widely used constructs in educational research and assessment within higher education. This paper conducts an analytic literature review to examine the use and operationalization of the term in multiple academic fields. Dominant definitions of the term are conceptually evaluated using Astin’s I-E-O model resulting in the proposition of a revised definition and new conceptual model of academic success. Measurements of academic success found throughout the literature are presented in accordance with the presented model of academic success. These measurements are provided with details in a user-friendly table (Appendix B). Results also indicate that grades and GPA are the most commonly used measure of academic success. Finally, recommendations are given for future research and practice to increase effective assessment of academic success.

Related Papers

Zin Eddine Dadach

The final goals of this study is to investigate the impact of using a Programmable Logic Controller (PLC) as a research based learning tool on the performance and motivation of each student in an industrial automation course. For this purpose, ten PLC’s lab experiments are used to visualize the theories covered in the learning outcomes (CLO’S) of the course and investigate their applications in industrial automation. The success of the PLC as research-based tool are demonstrated by the fact the performance of 40 students (73%) is higher in this course than their corresponding average performance in the college. Moreover, using the Dadach Motivation Factor (DMF) as tool to measure motivation of students, our results also indicate that thirty-five students (63.63%) were motivated during the course. In overall, this investigation shows that the utilization of ten PLC lab experiments for the visualization of the opaque theory of industrial automation is successful.

Knowledge and Performance Management

Mercy Ogbeta

Academic resilience and emotional intelligence are considered important personal resources for furthering students’ academic performance. However, many educational organizations seem to trivialize the performance implications of these constructs in teachings and curriculum. Consequently, it can decrease not just their academic performance but also their employability, as they lack the generic competencies to adapt and survive in a stressful context. Even so, empirical evidence on integrating academic resilience, emotional intelligence, and academic performance remains unexplored in the Nigerian university context. Therefore, the study aimed to investigate the linkages between academic resilience, emotional intelligence, and academic performance in Nigeria. The partial least square (PLS) modeling method was utilized for testing the stated hypotheses with data collected from 179 final year undergraduate students in the regular B.Sc. Business Administration and B.Sc. Marketing program ...

International Journal of Higher Education

margie roma

Higher educational institutions (HEIs) play a substantial role in the development of knowledge and skills that can cope with the demands of industries in the fourth industrial revolution (4IR). This study examined the alignment between the current assessment practices used by HEIs and the competencies demanded by the hospitality and tourism industry. It also aimed to develop an assessment strategy typology that could specifically target the competencies required by the industry. In addition, the study was able to determine the three most and the three least preferred assessment methods as perceived by the hotel and restaurant management students in a private university in Mandaluyong City, Philippines. The findings revealed the common assessment methods employed by teachers in Hospitality and Tourism Managemnt (HTM) major courses. The study argues that the use of these identified assessment methods likewise contribute in developing the emerging skills in the 4IR such as sense-maki...

College & Research Libraries

Angela Rockwell

Practical Assessment, Research and Evaluation

Donna H Ziegenfuss

Turkish Journal of Computer and Mathematics Education

Saumya Kumar

The significance accorded to the amorphous concept of Academic Performance of students is immense, owing to the mere reason that better performances academically set the prospect for the better future of any nation. Likewise, understanding the integrities of the concept holds a great connotation. The current paper aims to present a review of the various definitions of "academic performance" along with the factors perceived to have an effect on the academic performance of students belonging to Higher Education Institutions (HEIs). Additionally, an effort has also been made to gain an insight into the reliability of "Grade Point Average" (GPA) as a method for measuring academic performance.

Wanda Robinson

Educating a sufficient nursing workforce to provide high quality, compassionate, and ethical care to an increasingly diverse population is an ongoing challenge and opportunity for nurse educators. Current literature highlights the importance of engaging students in learning to strengthen student achievements. Fostering student engagement within nursing courses is particularly important. Grit (consistency of interest and perseverance of effort) is a factor that may be associated with student course engagement. Demographic characteristics of age, gender, race/ethnicity, prior education, degree program, and self-reported grade point average (GPA) also may be factors associated with student course engagement. Guided by a conceptual model derived from the literature, the purpose of this study was to determine whether grit and demographic characteristics were associated with student course engagement (skills, emotion, participation/interaction, and performance) within a nursing course. Us...

Health and Academic Achievement

Marcela Verešová

Marilee Bresciani Ludvik

The clash of whether higher education should serve the public good or economic stimulation seems more alive than ever to some, and to others, it has come to an end. Not agreeing on the purpose of American higher education certainly makes it difficult to know whether educators are being responsible for delivering what is expected of them. Rather than reviewing the important debate that has already taken place, this chapter seeks to merge the two seemingly juxtaposed disagreements and discuss how bringing the two purposes together may influence how we examine accountability. As such, an inquiry model, including ways to gather and interpret institutional performance indicators for accountability is posited. Practical suggestions for implementation of this methodology are provided.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Community College Journal of Research and Practice

Juliann McBrayer

Lee O Farrell

Higher education's response to exponential societal shifts

Jillian Kinzie

Higher Education Studies

karma El Hassan

Matthew Gaertner

Monica Burnette

Ann Popkess

The International Review of Research in Open and Distributed Learning

Lynette Bryan

Liz Chinlund

Journal of College Student Development

Fred Newton

WHY SUPERVISORS MATTER FOR COLLEGE STUDENTS: THE RELATIONSHIP BETWEEN TYPE OF EMPLOYMENT AND STUDENT OUTCOMES

Dr. Justin Hultman

Heather Friesen

Meltem Ozmutlu

ANIMA Indonesian Psychological Journal

titik kristiyani

Shafeeka Dockrat

mohamed deen

International Journal of Learning, Teaching and Educational Research

Nahla Moussa

Richard Okundia

The Journal of Campus Activities, Practice, and Scholarship

Christy Moran Craft

MIRDEC-10th, International Academic Conference Global and Contemporary Trends in Social Science (Global Meeting of Social Science Community) 06-08 November 2018, Barcelona, Spain. ISBN: 978-605-81247-2-1. https://www.mirdec.com/barca2018proceedings.

Nazmi Xhomara

Journal of Hispanic Higher Education

Glai Martinez

The problem of college readiness, edited by William G. Tierney and Julia C. Duncheon

Julia Duncheon

Mohamed Tejan Tarawally Deen

Kristopher A Oliveira

David Edens

The Professional Counselor

Jeffrey Warren

Jefferson Chavez

International Journal of Knowledge and learning

David Cheng

Oscar Cervantes

Journal of Multidisciplinary Research

Cristina Lopez Vergara

Azucena Montoya

Journal for the Advancement of Educational Research

Dr. Justin Hultman , Dr. Daniel Eadens

Proceedings of the 2010 …

Hala Bayoumy

putri hazarianti

Teachers College Record: The Voice of Scholarship in Education

Dr. Regina Deil-Amen, Ph.D.

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Clinics (Sao Paulo)

Approaching literature review for academic purposes: The Literature Review Checklist

Debora f.b. leite.

I Departamento de Ginecologia e Obstetricia, Faculdade de Ciencias Medicas, Universidade Estadual de Campinas, Campinas, SP, BR

II Universidade Federal de Pernambuco, Pernambuco, PE, BR

III Hospital das Clinicas, Universidade Federal de Pernambuco, Pernambuco, PE, BR

Maria Auxiliadora Soares Padilha

Jose g. cecatti.

A sophisticated literature review (LR) can result in a robust dissertation/thesis by scrutinizing the main problem examined by the academic study; anticipating research hypotheses, methods and results; and maintaining the interest of the audience in how the dissertation/thesis will provide solutions for the current gaps in a particular field. Unfortunately, little guidance is available on elaborating LRs, and writing an LR chapter is not a linear process. An LR translates students’ abilities in information literacy, the language domain, and critical writing. Students in postgraduate programs should be systematically trained in these skills. Therefore, this paper discusses the purposes of LRs in dissertations and theses. Second, the paper considers five steps for developing a review: defining the main topic, searching the literature, analyzing the results, writing the review and reflecting on the writing. Ultimately, this study proposes a twelve-item LR checklist. By clearly stating the desired achievements, this checklist allows Masters and Ph.D. students to continuously assess their own progress in elaborating an LR. Institutions aiming to strengthen students’ necessary skills in critical academic writing should also use this tool.

INTRODUCTION

Writing the literature review (LR) is often viewed as a difficult task that can be a point of writer’s block and procrastination ( 1 ) in postgraduate life. Disagreements on the definitions or classifications of LRs ( 2 ) may confuse students about their purpose and scope, as well as how to perform an LR. Interestingly, at many universities, the LR is still an important element in any academic work, despite the more recent trend of producing scientific articles rather than classical theses.

The LR is not an isolated section of the thesis/dissertation or a copy of the background section of a research proposal. It identifies the state-of-the-art knowledge in a particular field, clarifies information that is already known, elucidates implications of the problem being analyzed, links theory and practice ( 3 - 5 ), highlights gaps in the current literature, and places the dissertation/thesis within the research agenda of that field. Additionally, by writing the LR, postgraduate students will comprehend the structure of the subject and elaborate on their cognitive connections ( 3 ) while analyzing and synthesizing data with increasing maturity.

At the same time, the LR transforms the student and hints at the contents of other chapters for the reader. First, the LR explains the research question; second, it supports the hypothesis, objectives, and methods of the research project; and finally, it facilitates a description of the student’s interpretation of the results and his/her conclusions. For scholars, the LR is an introductory chapter ( 6 ). If it is well written, it demonstrates the student’s understanding of and maturity in a particular topic. A sound and sophisticated LR can indicate a robust dissertation/thesis.

A consensus on the best method to elaborate a dissertation/thesis has not been achieved. The LR can be a distinct chapter or included in different sections; it can be part of the introduction chapter, part of each research topic, or part of each published paper ( 7 ). However, scholars view the LR as an integral part of the main body of an academic work because it is intrinsically connected to other sections ( Figure 1 ) and is frequently present. The structure of the LR depends on the conventions of a particular discipline, the rules of the department, and the student’s and supervisor’s areas of expertise, needs and interests.

Interestingly, many postgraduate students choose to submit their LR to peer-reviewed journals. As LRs are critical evaluations of current knowledge, they are indeed publishable material, even in the form of narrative or systematic reviews. However, systematic reviews have specific patterns 1 ( 8 ) that may not entirely fit with the questions posed in the dissertation/thesis. Additionally, the scope of a systematic review may be too narrow, and the strict criteria for study inclusion may omit important information from the dissertation/thesis. Therefore, this essay discusses the definition of an LR is and methods to develop an LR in the context of an academic dissertation/thesis. Finally, we suggest a checklist to evaluate an LR.

WHAT IS A LITERATURE REVIEW IN A THESIS?

Conducting research and writing a dissertation/thesis translates rational thinking and enthusiasm ( 9 ). While a strong body of literature that instructs students on research methodology, data analysis and writing scientific papers exists, little guidance on performing LRs is available. The LR is a unique opportunity to assess and contrast various arguments and theories, not just summarize them. The research results should not be discussed within the LR, but the postgraduate student tends to write a comprehensive LR while reflecting on his or her own findings ( 10 ).

Many people believe that writing an LR is a lonely and linear process. Supervisors or the institutions assume that the Ph.D. student has mastered the relevant techniques and vocabulary associated with his/her subject and conducts a self-reflection about previously published findings. Indeed, while elaborating the LR, the student should aggregate diverse skills, which mainly rely on his/her own commitment to mastering them. Thus, less supervision should be required ( 11 ). However, the parameters described above might not currently be the case for many students ( 11 , 12 ), and the lack of formal and systematic training on writing LRs is an important concern ( 11 ).

An institutional environment devoted to active learning will provide students the opportunity to continuously reflect on LRs, which will form a dialogue between the postgraduate student and the current literature in a particular field ( 13 ). Postgraduate students will be interpreting studies by other researchers, and, according to Hart (1998) ( 3 ), the outcomes of the LR in a dissertation/thesis include the following:

• To identify what research has been performed and what topics require further investigation in a particular field of knowledge;
• To determine the context of the problem;
• To recognize the main methodologies and techniques that have been used in the past;
• To place the current research project within the historical, methodological and theoretical context of a particular field;
• To identify significant aspects of the topic;
• To elucidate the implications of the topic;
• To offer an alternative perspective;
• To discern how the studied subject is structured;
• To improve the student’s subject vocabulary in a particular field; and
• To characterize the links between theory and practice.

A sound LR translates the postgraduate student’s expertise in academic and scientific writing: it expresses his/her level of comfort with synthesizing ideas ( 11 ). The LR reveals how well the postgraduate student has proceeded in three domains: an effective literature search, the language domain, and critical writing.

Effective literature search

All students should be trained in gathering appropriate data for specific purposes, and information literacy skills are a cornerstone. These skills are defined as “an individual’s ability to know when they need information, to identify information that can help them address the issue or problem at hand, and to locate, evaluate, and use that information effectively” ( 14 ). Librarian support is of vital importance in coaching the appropriate use of Boolean logic (AND, OR, NOT) and other tools for highly efficient literature searches (e.g., quotation marks and truncation), as is the appropriate management of electronic databases.

Language domain

Academic writing must be concise and precise: unnecessary words distract the reader from the essential content ( 15 ). In this context, reading about issues distant from the research topic ( 16 ) may increase students’ general vocabulary and familiarity with grammar. Ultimately, reading diverse materials facilitates and encourages the writing process itself.

Critical writing

Critical judgment includes critical reading, thinking and writing. It supposes a student’s analytical reflection about what he/she has read. The student should delineate the basic elements of the topic, characterize the most relevant claims, identify relationships, and finally contrast those relationships ( 17 ). Each scientific document highlights the perspective of the author, and students will become more confident in judging the supporting evidence and underlying premises of a study and constructing their own counterargument as they read more articles. A paucity of integration or contradictory perspectives indicates lower levels of cognitive complexity ( 12 ).

Thus, while elaborating an LR, the postgraduate student should achieve the highest category of Bloom’s cognitive skills: evaluation ( 12 ). The writer should not only summarize data and understand each topic but also be able to make judgments based on objective criteria, compare resources and findings, identify discrepancies due to methodology, and construct his/her own argument ( 12 ). As a result, the student will be sufficiently confident to show his/her own voice .

Writing a consistent LR is an intense and complex activity that reveals the training and long-lasting academic skills of a writer. It is not a lonely or linear process. However, students are unlikely to be prepared to write an LR if they have not mastered the aforementioned domains ( 10 ). An institutional environment that supports student learning is crucial.

Different institutions employ distinct methods to promote students’ learning processes. First, many universities propose modules to develop behind the scenes activities that enhance self-reflection about general skills (e.g., the skills we have mastered and the skills we need to develop further), behaviors that should be incorporated (e.g., self-criticism about one’s own thoughts), and each student’s role in the advancement of his/her field. Lectures or workshops about LRs themselves are useful because they describe the purposes of the LR and how it fits into the whole picture of a student’s work. These activities may explain what type of discussion an LR must involve, the importance of defining the correct scope, the reasons to include a particular resource, and the main role of critical reading.

Some pedagogic services that promote a continuous improvement in study and academic skills are equally important. Examples include workshops about time management, the accomplishment of personal objectives, active learning, and foreign languages for nonnative speakers. Additionally, opportunities to converse with other students promotes an awareness of others’ experiences and difficulties. Ultimately, the supervisor’s role in providing feedback and setting deadlines is crucial in developing students’ abilities and in strengthening students’ writing quality ( 12 ).

HOW SHOULD A LITERATURE REVIEW BE DEVELOPED?

A consensus on the appropriate method for elaborating an LR is not available, but four main steps are generally accepted: defining the main topic, searching the literature, analyzing the results, and writing ( 6 ). We suggest a fifth step: reflecting on the information that has been written in previous publications ( Figure 2 ).

First step: Defining the main topic

Planning an LR is directly linked to the research main question of the thesis and occurs in parallel to students’ training in the three domains discussed above. The planning stage helps organize ideas, delimit the scope of the LR ( 11 ), and avoid the wasting of time in the process. Planning includes the following steps:

• Reflecting on the scope of the LR: postgraduate students will have assumptions about what material must be addressed and what information is not essential to an LR ( 13 , 18 ). Cooper’s Taxonomy of Literature Reviews 2 systematizes the writing process through six characteristics and nonmutually exclusive categories. The focus refers to the reviewer’s most important points of interest, while the goals concern what students want to achieve with the LR. The perspective assumes answers to the student’s own view of the LR and how he/she presents a particular issue. The coverage defines how comprehensive the student is in presenting the literature, and the organization determines the sequence of arguments. The audience is defined as the group for whom the LR is written.
• Designating sections and subsections: Headings and subheadings should be specific, explanatory and have a coherent sequence throughout the text ( 4 ). They simulate an inverted pyramid, with an increasing level of reflection and depth of argument.
• Identifying keywords: The relevant keywords for each LR section should be listed to guide the literature search. This list should mirror what Hart (1998) ( 3 ) advocates as subject vocabulary . The keywords will also be useful when the student is writing the LR since they guide the reader through the text.
• Delineating the time interval and language of documents to be retrieved in the second step. The most recently published documents should be considered, but relevant texts published before a predefined cutoff year can be included if they are classic documents in that field. Extra care should be employed when translating documents.

Second step: Searching the literature

The ability to gather adequate information from the literature must be addressed in postgraduate programs. Librarian support is important, particularly for accessing difficult texts. This step comprises the following components:

• Searching the literature itself: This process consists of defining which databases (electronic or dissertation/thesis repositories), official documents, and books will be searched and then actively conducting the search. Information literacy skills have a central role in this stage. While searching electronic databases, controlled vocabulary (e.g., Medical Subject Headings, or MeSH, for the PubMed database) or specific standardized syntax rules may need to be applied.

In addition, two other approaches are suggested. First, a review of the reference list of each document might be useful for identifying relevant publications to be included and important opinions to be assessed. This step is also relevant for referencing the original studies and leading authors in that field. Moreover, students can directly contact the experts on a particular topic to consult with them regarding their experience or use them as a source of additional unpublished documents.

Before submitting a dissertation/thesis, the electronic search strategy should be repeated. This process will ensure that the most recently published papers will be considered in the LR.

• Selecting documents for inclusion: Generally, the most recent literature will be included in the form of published peer-reviewed papers. Assess books and unpublished material, such as conference abstracts, academic texts and government reports, are also important to assess since the gray literature also offers valuable information. However, since these materials are not peer-reviewed, we recommend that they are carefully added to the LR.

This task is an important exercise in time management. First, students should read the title and abstract to understand whether that document suits their purposes, addresses the research question, and helps develop the topic of interest. Then, they should scan the full text, determine how it is structured, group it with similar documents, and verify whether other arguments might be considered ( 5 ).

Third step: Analyzing the results

Critical reading and thinking skills are important in this step. This step consists of the following components:

• Reading documents: The student may read various texts in depth according to LR sections and subsections ( defining the main topic ), which is not a passive activity ( 1 ). Some questions should be asked to practice critical analysis skills, as listed below. Is the research question evident and articulated with previous knowledge? What are the authors’ research goals and theoretical orientations, and how do they interact? Are the authors’ claims related to other scholars’ research? Do the authors consider different perspectives? Was the research project designed and conducted properly? Are the results and discussion plausible, and are they consistent with the research objectives and methodology? What are the strengths and limitations of this work? How do the authors support their findings? How does this work contribute to the current research topic? ( 1 , 19 )
• Taking notes: Students who systematically take notes on each document are more readily able to establish similarities or differences with other documents and to highlight personal observations. This approach reinforces the student’s ideas about the next step and helps develop his/her own academic voice ( 1 , 13 ). Voice recognition software ( 16 ), mind maps ( 5 ), flowcharts, tables, spreadsheets, personal comments on the referenced texts, and note-taking apps are all available tools for managing these observations, and the student him/herself should use the tool that best improves his/her learning. Additionally, when a student is considering submitting an LR to a peer-reviewed journal, notes should be taken on the activities performed in all five steps to ensure that they are able to be replicated.

Fourth step: Writing

The recognition of when a student is able and ready to write after a sufficient period of reading and thinking is likely a difficult task. Some students can produce a review in a single long work session. However, as discussed above, writing is not a linear process, and students do not need to write LRs according to a specific sequence of sections. Writing an LR is a time-consuming task, and some scholars believe that a period of at least six months is sufficient ( 6 ). An LR, and academic writing in general, expresses the writer’s proper thoughts, conclusions about others’ work ( 6 , 10 , 13 , 16 ), and decisions about methods to progress in the chosen field of knowledge. Thus, each student is expected to present a different learning and writing trajectory.

In this step, writing methods should be considered; then, editing, citing and correct referencing should complete this stage, at least temporarily. Freewriting techniques may be a good starting point for brainstorming ideas and improving the understanding of the information that has been read ( 1 ). Students should consider the following parameters when creating an agenda for writing the LR: two-hour writing blocks (at minimum), with prespecified tasks that are possible to complete in one section; short (minutes) and long breaks (days or weeks) to allow sufficient time for mental rest and reflection; and short- and long-term goals to motivate the writing itself ( 20 ). With increasing experience, this scheme can vary widely, and it is not a straightforward rule. Importantly, each discipline has a different way of writing ( 1 ), and each department has its own preferred styles for citations and references.

Fifth step: Reflecting on the writing

In this step, the postgraduate student should ask him/herself the same questions as in the analyzing the results step, which can take more time than anticipated. Ambiguities, repeated ideas, and a lack of coherence may not be noted when the student is immersed in the writing task for long periods. The whole effort will likely be a work in progress, and continuous refinements in the written material will occur once the writing process has begun.

LITERATURE REVIEW CHECKLIST

In contrast to review papers, the LR of a dissertation/thesis should not be a standalone piece or work. Instead, it should present the student as a scholar and should maintain the interest of the audience in how that dissertation/thesis will provide solutions for the current gaps in a particular field.

A checklist for evaluating an LR is convenient for students’ continuous academic development and research transparency: it clearly states the desired achievements for the LR of a dissertation/thesis. Here, we present an LR checklist developed from an LR scoring rubric ( 11 ). For a critical analysis of an LR, we maintain the five categories but offer twelve criteria that are not scaled ( Figure 3 ). The criteria all have the same importance and are not mutually exclusive.

First category: Coverage

1. justified criteria exist for the inclusion and exclusion of literature in the review.

This criterion builds on the main topic and areas covered by the LR ( 18 ). While experts may be confident in retrieving and selecting literature, postgraduate students must convince their audience about the adequacy of their search strategy and their reasons for intentionally selecting what material to cover ( 11 ). References from different fields of knowledge provide distinct perspective, but narrowing the scope of coverage may be important in areas with a large body of existing knowledge.

Second category: Synthesis

2. a critical examination of the state of the field exists.

A critical examination is an assessment of distinct aspects in the field ( 1 ) along with a constructive argument. It is not a negative critique but an expression of the student’s understanding of how other scholars have added to the topic ( 1 ), and the student should analyze and contextualize contradictory statements. A writer’s personal bias (beliefs or political involvement) have been shown to influence the structure and writing of a document; therefore, the cultural and paradigmatic background guide how the theories are revised and presented ( 13 ). However, an honest judgment is important when considering different perspectives.

3. The topic or problem is clearly placed in the context of the broader scholarly literature

The broader scholarly literature should be related to the chosen main topic for the LR ( how to develop the literature review section). The LR can cover the literature from one or more disciplines, depending on its scope, but it should always offer a new perspective. In addition, students should be careful in citing and referencing previous publications. As a rule, original studies and primary references should generally be included. Systematic and narrative reviews present summarized data, and it may be important to cite them, particularly for issues that should be understood but do not require a detailed description. Similarly, quotations highlight the exact statement from another publication. However, excessive referencing may disclose lower levels of analysis and synthesis by the student.

4. The LR is critically placed in the historical context of the field

Situating the LR in its historical context shows the level of comfort of the student in addressing a particular topic. Instead of only presenting statements and theories in a temporal approach, which occasionally follows a linear timeline, the LR should authentically characterize the student’s academic work in the state-of-art techniques in their particular field of knowledge. Thus, the LR should reinforce why the dissertation/thesis represents original work in the chosen research field.

5. Ambiguities in definitions are considered and resolved

Distinct theories on the same topic may exist in different disciplines, and one discipline may consider multiple concepts to explain one topic. These misunderstandings should be addressed and contemplated. The LR should not synthesize all theories or concepts at the same time. Although this approach might demonstrate in-depth reading on a particular topic, it can reveal a student’s inability to comprehend and synthesize his/her research problem.

6. Important variables and phenomena relevant to the topic are articulated

The LR is a unique opportunity to articulate ideas and arguments and to purpose new relationships between them ( 10 , 11 ). More importantly, a sound LR will outline to the audience how these important variables and phenomena will be addressed in the current academic work. Indeed, the LR should build a bidirectional link with the remaining sections and ground the connections between all of the sections ( Figure 1 ).

7. A synthesized new perspective on the literature has been established

The LR is a ‘creative inquiry’ ( 13 ) in which the student elaborates his/her own discourse, builds on previous knowledge in the field, and describes his/her own perspective while interpreting others’ work ( 13 , 17 ). Thus, students should articulate the current knowledge, not accept the results at face value ( 11 , 13 , 17 ), and improve their own cognitive abilities ( 12 ).

Third category: Methodology

8. the main methodologies and research techniques that have been used in the field are identified and their advantages and disadvantages are discussed.

The LR is expected to distinguish the research that has been completed from investigations that remain to be performed, address the benefits and limitations of the main methods applied to date, and consider the strategies for addressing the expected limitations described above. While placing his/her research within the methodological context of a particular topic, the LR will justify the methodology of the study and substantiate the student’s interpretations.

9. Ideas and theories in the field are related to research methodologies

The audience expects the writer to analyze and synthesize methodological approaches in the field. The findings should be explained according to the strengths and limitations of previous research methods, and students must avoid interpretations that are not supported by the analyzed literature. This criterion translates to the student’s comprehension of the applicability and types of answers provided by different research methodologies, even those using a quantitative or qualitative research approach.

Fourth category: Significance

10. the scholarly significance of the research problem is rationalized.

The LR is an introductory section of a dissertation/thesis and will present the postgraduate student as a scholar in a particular field ( 11 ). Therefore, the LR should discuss how the research problem is currently addressed in the discipline being investigated or in different disciplines, depending on the scope of the LR. The LR explains the academic paradigms in the topic of interest ( 13 ) and methods to advance the field from these starting points. However, an excess number of personal citations—whether referencing the student’s research or studies by his/her research team—may reflect a narrow literature search and a lack of comprehensive synthesis of ideas and arguments.

11. The practical significance of the research problem is rationalized

The practical significance indicates a student’s comprehensive understanding of research terminology (e.g., risk versus associated factor), methodology (e.g., efficacy versus effectiveness) and plausible interpretations in the context of the field. Notably, the academic argument about a topic may not always reflect the debate in real life terms. For example, using a quantitative approach in epidemiology, statistically significant differences between groups do not explain all of the factors involved in a particular problem ( 21 ). Therefore, excessive faith in p -values may reflect lower levels of critical evaluation of the context and implications of a research problem by the student.

Fifth category: Rhetoric

12. the lr was written with a coherent, clear structure that supported the review.

This category strictly relates to the language domain: the text should be coherent and presented in a logical sequence, regardless of which organizational ( 18 ) approach is chosen. The beginning of each section/subsection should state what themes will be addressed, paragraphs should be carefully linked to each other ( 10 ), and the first sentence of each paragraph should generally summarize the content. Additionally, the student’s statements are clear, sound, and linked to other scholars’ works, and precise and concise language that follows standardized writing conventions (e.g., in terms of active/passive voice and verb tenses) is used. Attention to grammar, such as orthography and punctuation, indicates prudence and supports a robust dissertation/thesis. Ultimately, all of these strategies provide fluency and consistency for the text.

Although the scoring rubric was initially proposed for postgraduate programs in education research, we are convinced that this checklist is a valuable tool for all academic areas. It enables the monitoring of students’ learning curves and a concentrated effort on any criteria that are not yet achieved. For institutions, the checklist is a guide to support supervisors’ feedback, improve students’ writing skills, and highlight the learning goals of each program. These criteria do not form a linear sequence, but ideally, all twelve achievements should be perceived in the LR.

CONCLUSIONS

A single correct method to classify, evaluate and guide the elaboration of an LR has not been established. In this essay, we have suggested directions for planning, structuring and critically evaluating an LR. The planning of the scope of an LR and approaches to complete it is a valuable effort, and the five steps represent a rational starting point. An institutional environment devoted to active learning will support students in continuously reflecting on LRs, which will form a dialogue between the writer and the current literature in a particular field ( 13 ).

The completion of an LR is a challenging and necessary process for understanding one’s own field of expertise. Knowledge is always transitory, but our responsibility as scholars is to provide a critical contribution to our field, allowing others to think through our work. Good researchers are grounded in sophisticated LRs, which reveal a writer’s training and long-lasting academic skills. We recommend using the LR checklist as a tool for strengthening the skills necessary for critical academic writing.

AUTHOR CONTRIBUTIONS

Leite DFB has initially conceived the idea and has written the first draft of this review. Padilha MAS and Cecatti JG have supervised data interpretation and critically reviewed the manuscript. All authors have read the draft and agreed with this submission. Authors are responsible for all aspects of this academic piece.

ACKNOWLEDGMENTS

We are grateful to all of the professors of the ‘Getting Started with Graduate Research and Generic Skills’ module at University College Cork, Cork, Ireland, for suggesting and supporting this article. Funding: DFBL has granted scholarship from Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES) to take part of her Ph.D. studies in Ireland (process number 88881.134512/2016-01). There is no participation from sponsors on authors’ decision to write or to submit this manuscript.

No potential conflict of interest was reported.

1 The questions posed in systematic reviews usually follow the ‘PICOS’ acronym: Population, Intervention, Comparison, Outcomes, Study design.

2 In 1988, Cooper proposed a taxonomy that aims to facilitate students’ and institutions’ understanding of literature reviews. Six characteristics with specific categories are briefly described: Focus: research outcomes, research methodologies, theories, or practices and applications; Goals: integration (generalization, conflict resolution, and linguistic bridge-building), criticism, or identification of central issues; Perspective: neutral representation or espousal of a position; Coverage: exhaustive, exhaustive with selective citations, representative, central or pivotal; Organization: historical, conceptual, or methodological; and Audience: specialized scholars, general scholars, practitioners or policymakers, or the general public.

Literature Reviews

• What is a Literature Review?
• Steps for Creating a Literature Review
• Providing Evidence / Critical Analysis
• Challenges when writing a Literature Review
• Systematic Literature Reviews

Developing a Literature Review

1. Purpose and Scope

To help you develop a literature review, gather information on existing research, sub-topics, relevant research, and overlaps. Note initial thoughts on the topic - a mind map or list might be helpful - and avoid unfocused reading, collecting irrelevant content.  A literature review serves to place your research within the context of existing knowledge. It demonstrates your understanding of the field and identifies gaps that your research aims to fill. This helps in justifying the relevance and necessity of your study.

To avoid over-reading, set a target word count for each section and limit reading time. Plan backwards from the deadline and move on to other parts of the investigation. Read major texts and explore up-to-date research. Check reference lists and citation indexes for common standard texts. Be guided by research questions and refocus on your topic when needed. Stop reading if you find similar viewpoints or if you're going off topic.

You can use a "Synthesis Matrix" to keep track of your reading notes. This concept map helps you to provide a summary of the literature and its connections is produced as a result of this study. Utilizing referencing software like RefWorks to obtain citations, you can construct the framework for composing your literature evaluation.

2. Source Selection

Focus on searching for academically authoritative texts such as academic books, journals, research reports, and government publications. These sources are critical for ensuring the credibility and reliability of your review. 

• Academic Books: Provide comprehensive coverage of a topic.
• Journal Articles: Offer the most up-to-date research and are essential for a literature review.
• Research Reports: Detailed accounts of specific research projects.
• Government Publications: Official documents that provide reliable data and insights.

3. Thematic Analysis

Instead of merely summarizing sources, identify and discuss key themes that emerge from the literature. This involves interpreting and evaluating how different authors have tackled similar issues and how their findings relate to your research.

4. Critical Evaluation

Adopt a critical attitude towards the sources you review. Scrutinize, question, and dissect the material to ensure that your review is not just descriptive but analytical. This helps in highlighting the significance of various sources and their relevance to your research.

Each work's critical assessment should take into account:

Provenance:  What qualifications does the author have? Are the author's claims backed up by proof, such as first-hand accounts from history, case studies, stories, statistics, and current scientific discoveries? Methodology:  Were the strategies employed to locate, collect, and evaluate the data suitable for tackling the study question? Was the sample size suitable? Were the findings properly reported and interpreted? Objectivity : Is the author's viewpoint impartial or biased? Does the author's thesis get supported by evidence that refutes it, or does it ignore certain important facts? Persuasiveness:  Which of the author's arguments is the strongest or weakest in terms of persuasiveness? Value:  Are the author's claims and deductions believable? Does the study ultimately advance our understanding of the issue in any meaningful way?

5. Categorization

Organize your literature review by grouping sources into categories based on themes, relevance to research questions, theoretical paradigms, or chronology. This helps in presenting your findings in a structured manner.

6. Source Validity

Ensure that the sources you include are valid and reliable. Classic texts may retain their authority over time, but for fields that evolve rapidly, prioritize the most recent research. Always check the credibility of the authors and the impact of their work in the field.

7. Synthesis and Findings

Synthesize the information from various sources to draw conclusions about the current state of knowledge. Identify trends, controversies, and gaps in the literature. Relate your findings to your research questions and suggest future directions for research.

Practical Tips

• Use a variety of sources, including online databases, university libraries, and reference lists from relevant articles. This ensures a comprehensive coverage of the literature.
• Avoid listing sources without analysis. Use tables, bulk citations, and footnotes to manage references efficiently and make your review more readable.
• Writing a literature review is an ongoing process. Start writing early and revise as you read more. This iterative process helps in refining your arguments and identifying additional sources as needed.  

Brown University Library (2024) Organizing and Creating Information. Available at: https://libguides.brown.edu/organize/litreview (Accessed: 30 July 2024).

Pacheco-Vega, R. (2016) Synthesizing different bodies of work in your literature review: The Conceptual Synthesis Excel Dump (CSED) technique . Available at: http://www.raulpacheco.org/2016/06/synthesizing-different-bodies-of-work-in-your-literature-review-the-conceptual-synthesis-excel-dump-technique/ (Accessed: 30 July 2024).

Study Advice at the University of Reading (2024) Literature reviews . Available at: https://libguides.reading.ac.uk/literaturereview/developing (Accessed: 31 July 2024).

Further Reading

Frameworks for creating answerable (re)search questions  How to Guide

Literature Searching How to Guide

• << Previous: Steps for Creating a Literature Review
• Next: Providing Evidence / Critical Analysis >>
• Last Updated: Sep 4, 2024 11:43 AM
• URL: https://library.lsbu.ac.uk/literaturereviews

We Trust in Human Precision

20,000+ Professional Language Experts Ready to Help. Expertise in a variety of Niches.

API Solutions

• API Pricing
• Cost estimate
• Customer loyalty program
• Educational Discount
• Non-Profit Discount
• Green Initiative Discount1

Value-Driven Pricing

Unmatched expertise at affordable rates tailored for your needs. Our services empower you to boost your productivity.

• Special Discounts
• Enterprise transcription solutions
• Enterprise translation solutions
• Transcription/Caption API
• AI Transcription Proofreading API

Trusted by Global Leaders

GoTranscript is the chosen service for top media organizations, universities, and Fortune 50 companies.

GoTranscript

One of the Largest Online Transcription and Translation Agencies in the World. Founded in 2005.

Speaker 1: Welcome to the UCD Writing Centre. My name is Michael Pei and I'm a tutor here and today we're going to be running an online tutorial on how to write a literature review. I'm going to take you through the specifics on how to write the review and the point of this tutorial is to give you the tools and insights that you will need to be able to create a review that is right for you, your subject area, and can be completed in a timely and efficient manner. You will see as we go through the piece that there is no one-size-fits-all review type. As such, you can take, borrow from, or rejig anything you see here today as long as it suits your needs. So one of the first questions people generally ask is, what is a literature review? And there are many variations, but the most basic answer is that it is a piece of writing that maps the current state of knowledge in your field in a way that allows you to put your own research into perspective. So it allows you to gauge the field in a critical way by assessing major contributions. It ensures continuous development of analytical and critical skills, and this is particularly important as opposed to writing in reports or essay writings. The literature review encourages you to engage with the primary materials that you will be placing your own research in. You are able to position yourself in that regard. It also allows you to practice your writing. Literature reviews are sometimes written in a kind of rough and ready manner, but I would encourage you to focus on creating a narrative in your review. It just means that your ability to write academically and to think academically and analytically, it just means you will be constantly practicing that while you are also gauging the state of knowledge in the field. So it can be useful beyond just actually mapping the field itself. Finally, and the most obvious point, is you cannot make a contribution to knowledge if you don't have a clear grasp of the major points of the field. So the layout of this presentation is as follows. We'll begin with where to start, how to search, how to assess a source, active reading strategies, how to structure your review. We'll have a look at a sample review, some shortcuts you can use to make the process a little easier and a little faster. And finally, we'll have a list of other resources that you can use for further research into the review. So where to start. The first thing you have to think about are what are the major databases that have been recommended to me by my tutors and lecturers. A lot of people begin the review process by looking through Google Scholar with several quickly chosen keywords. The issue with this approach is that you'll either get far too many results or far too few. And they may not be specific enough to the questions that you need to answer in your review. So you need to speak to your lecturer, your colleagues, your tutor, whoever it is, to ensure that you're getting the correct portals to search through and you're looking at the correct magazines, books, et cetera. You need to consider central research questions and come up with keywords from these. I generally come up with three or four central research questions. What they do is they refer to the overall title of the review. And by answering those three or four questions, I can safely say that I have basically completed my objectives in mapping a particular section of the field, what I want to look at in the literature review. You should always talk to your colleagues and classmates when you're doing a literature review. It's very easy to do a literature review in isolation. And it's always better to talk to people, to point to the different things that you found, the different points that you'd like to address, see if they've come across them, see if they can recommend any other critiques or articles that might improve your position or where you're coming from. You should always title your literature review angled towards your objectives. As such, don't simply write a literature review called the Postcolonial Studies Literature Review, for example. You'll want a title that's particularly angled towards the objectives of the review. And we'll be going through this later in this tutorial. And also, you should draw up your bibliography. It's one of the first things you should do. A lot of people get slightly intimidated by this because there are obviously so many different sources to choose from. So we're going to have a look at some software that's available to us to make this process a little bit easier a little later in this tutorial. So how to search. Create your three or four central research questions according to what you want to achieve in the review. And pick your keywords very, very carefully. You want your keywords to be subject-specific. You want to make sure that you're picking words that are adequate to your area. If you forget a particular theoretical term or a term that's useful to addressing the issues that you want addressed, you won't be able to get the correct resources. So you need to make sure that your terms are carefully chosen. You can use the wildcard function in lots of search engines. It's usually a star. And this basically means that, for example, if you type in the word run, it will also search for words like running, et cetera. So it won't delimit your searches to the very precise parameters of the word. It'll open up the parameters slightly. This can be useful if you're getting far too few results. If you're getting far too many, your keywords are probably too general and you need to refine them. Ensure that your search portal is appropriate and that you have a list of journals that you need to look at. For example, again, if you are doing a post-colonial studies literature review and you don't have the Journal of Commonwealth Literature listed as one of the journals you need to look through, there is a good chance that you will miss a pivotal intervention or article that will allow you to map the state of the field. So you need to have a list of journals and portals, the Taylor & Francis online portal, JSTOR, Project News, whatever they are, that have access to these journals. Your lecturers, your tutors will generally know about these and they're usually accessible via library websites in your university or institution. You can also check if the article is cited a lot on a lot of search portals. This is really useful. If an article is quite recent and has a lot of citations, it might be quite important in terms of where the field is going. So it's good to make a note of that. And generally, you should make sure that the article is peer-reviewed. Oryxweb.com is a service that allows you to check for this, but it's not always available through your institution. If you go to the About section of a journal online, usually it will explain whether or not the article itself is peer-reviewed, which is important. The peer-review process, just very briefly, is basically a way of ensuring academic originality and integrity with articles that are written. Articles are submitted, checked by experts for veracity, whether they fit the journal's expectations, and if they're making an original contribution to knowledge, and then it goes through a whole editorial process before it's sent out. So that's basically what the peer-review process is. Assessing a source. Usually it's not enough to just read the title of an article. You should also have a look at the abstract and maybe the first few lines of the introduction. Run this against your keywords and central research questions, and if they match, make a note that you will possibly be using it in your literature review piece. At this stage, remember, we're just assessing what sources we want to use, so just to make a note of that. You should then check if the theoretical framework is adequately conceptualized. You can generally get a hint of this by reading the first few paragraphs of the piece. And if you feel that the framework doesn't quite live up to what it should in terms of the objectives of the article that you're reading, just make a note of this also. In qualitative studies in particular environments, check if there is any indication that the results may have been tampered with in order to fulfill a research expectation. A lot of the time, these sorts of things are unconscious. We all have our biases, and researchers and academics, of course, are not immune from this. Also make sure that the sample size is adequate. If you are citing results that have an inadequate sample size, obviously the conclusions of the experiment might over-determine its importance if the sample size is too small. Just to be aware of all of these things, make comments, make footnotes, just make sure you don't take something at its exact word. The point of the literature review is to be completely analytical, consistently analytical. You're always questioning the work as you're going through it. And again, is the article or book cited a lot? It might mean that it's a pivotal intervention. So we've spoken a lot about how to assess sources, how to find sources, but keeping track of them is a different story. Now we're lucky that there are some programs available to us that make this process a lot easier. So I have two examples here, Zotero and Mendley. I personally use Zotero. It is an add-on that you can use with Google Chrome, and basically it allows you to note all your sources with a click. Once you download Zotero and use it as an add-on, a new image will appear and you can click it, and it will save the bibliographical information of any article that you think might be useful. It means that you don't have to go to the trouble of switching between documents on your laptop or writing them down if you don't want to write them down. Just with the click of a mouse, you can log the article. The information stays logged forever. It's in the cloud, so you can access it from any location. It's very useful because it means you don't have to be overly discerning with the pieces that you're picking. If you feel like a piece might be relevant but you're not sure, just click it. If you don't use it, you don't use it. It's logged forever. It was only a click of a mouse. It's not too much trouble. So now we move on to picking the sources that we're going to use specifically for the literature review. So you found X number of key texts and X amount of other relevant works. I should point out now, there's no set amount of sources for a literature review. Generally, if you're using 40 or 50, you're probably a bit on the high side. Usually, something around 20 to 30 is adequate. Again, it's something you should discuss with your lecturer, your tutor. It depends if you're a master's student, PhD student. You'll get a feel for it the more reviews you do within your own subject area. You should look through Zotero and pick out key articles. So this is where Zotero becomes particularly useful. You'll have all the articles lined up. You can have a look again at the titles, any comments or notes that you've made about them, and you begin to discern which ones are particularly relevant for the review. Check that the articles feature your keywords. See if they're written by established scholars. Obviously, you'll need a certain amount of those to show that you're aware of the major contributors to the field. And you can also make use of already existing review articles. You'll often find these in peer-reviewed journals where someone will have written an article about three or four books, recently published research results, where they basically try and map the current state of the field or the contributions of particular authors. They can be very useful shortcuts to allow you to kind of gauge the state of the field without perhaps having to do a completely in the dark on your own. It's always worth thinking about who wrote the review. Is it a graduate student? Is it an established academic? What kind of things might affect the way they approach the piece? Is a framework of their analysis relevant to your objectives? All of these things are worth considering. So now that you've picked your sources, reading them adequately is obviously very important. So this is active reading. So you're going to want to read analytically. You don't want to read passively. You constantly want to think about what you're looking for and what your objectives are in the review. So have your keywords in mind and mark the margins with them. Write notes in the margins. Compare to other pieces as you read. So as you've read three or four different articles, you might begin to discern the different schools of thought in a subject area, particular academics who pair up better in the review piece. Write that in the margin. You're constantly trying to cross-reference. You're constantly exercising your brain to think in an analytical way to try and gauge what's going on in the field at a particular time. And the McGraw Centre recommends that you don't highlight. Make a note instead. Many times people bring literature reviews into the UCD Writing Centre and they might have a selection of articles that have many, many highlights on them but no notes. And when the students then try to take the information to create the review, they haven't written any notes, so they don't really know why they highlighted a section because they can't remember. So the trick is to write down, instead of highlighting, write down why it's worth highlighting. That way you'll know what you're doing when you return to look at the work a week or two later or whatever it is. And again, talk about it with someone. If you're looking at a particularly interesting article or report, talk about it with your lecturer, your tutor, your supervisor, your colleagues, anyone who might be interested. Constantly thinking about its validity, its relevance to your review piece. Structure. You should use subheadings. It allows you to break up the review and makes it easier to navigate. The essential research questions that you come up with can act as subheadings at the beginning. You'll know what bit of information you want to put in a particular section of paragraphs. If your instructor doesn't want you to use subheadings in your literature review, you can just delete them at the end and hand it up and it'll just flow as a piece of work with different paragraphs. You can do that kind of editorial stuff at the end. But particularly while you're doing the review, it's a good idea to have subheadings. You can also use a color code which can make it a lot easier to recall information. We'll see in the review sample that I'm going to show in a few minutes how a color code recalls particular research questions within the review itself. And cross-referencing. You should keep everything connected. Constantly bring up other critics, other positions. Try and gauge how critics rub off each other, the particular ways that they might look at a subject in comparison to other critics. That's the point of the review. You're comparing and contrasting in order to position yourself in the field. You should only use essential quotations. This is particularly important. A common mistake is that people use lots of block quotations and their argument is that the critic's work is too complex to be broken down into paragraphs. And that's understandable, but the issue is that you simply won't remember or really understand what the person is saying if you're just using block quotations. If you can't put it into your own words, the odds are you don't have a great understanding of it. So you need to just slow down, read a chapter, set yourself a challenge, give yourself three sentences to summarize the chapter in relation to the other work that you have done. That's the stuff that should go in the literature review. Particularly important quotations you should include. And I'll show you an example of this again in a couple of minutes when we move on to our sample pieces of literature reviews. You should use footnotes and comments. As I said earlier, comments and footnotes just allow you to kind of consider the sources outside of the literature review. If you feel that methodology isn't adequate, if you're concerned about the particular biases of a particular researcher, just a comment, just while you're doing the review. You can delete it at the end or you can keep it for yourself, just for your own little notes outside of the review parameters. You should have a bibliography as well as a works cited. Your works cited can be the works that you specifically analyzed in the review. Your bibliography are the larger works that you might come back to if you want to improve the review in any way, if you want to double check a source or read a source that you decided not to use in the review the first time, but that you think at this stage of your research career might actually be worth having a look at. So our first sample is from a postcolonial studies literature review. You should always do an abstract at the beginning of a literature review. The point of an abstract is to basically gauge what the review is about, what are the main points you're going to address, and what is the conclusion that you're going to demonstrate at the end. It's not like an introduction which lays out what you're going to do. It's much more complete than that. And it's usually about 100 to 150 words. So this is a sample abstract. This literature review seeks to clarify the different approaches to postcolonial studies and literature. Looking at the field through methodologies of Marxism, hybridity, subaltern studies, and eco-criticism, the review demonstrates how these different approaches have particular strengths and weaknesses in their historiography as well as their modes of aesthetic and formal analysis. Ultimately, the review concludes that the four approaches are distinct but share certain features and that particular areas are more suited to certain texts over others. This suggests that championing one particular approach might risk doing violence to a particular text. And then I follow this up with keywords that I used in my search parameters. Again, you may not be allowed to hand up a literature review piece with keywords. Just delete them. I mean, my approach to literature reviews is the piece that you hand up is all very well and good. But the more information you have when you're looking back over a literature review, whether you're traveling to a conference or about to discuss a paper with a colleague, the more information, the better. You can see here that the abstract uses the correct terminology. In the case of postcolonial studies, it tries to address what the author considers are the four main areas about the field. It uses vocabulary that may not be overly familiar to the non-expert but is assumed familiar to the person reading the review. Your introduction. So you should give an overview of the research topic. For example here, qualitative approaches to consumer psychology generally focus on, or political realism is principally informed by, it is used in order to, and then you just fill out those gaps. You should then point out the precise parameters within the topic that you are going to research. And then I would also list the subheadings and explain very briefly the general focus of each subheading. So those three or four subheadings that I discussed earlier, just list them under the introduction. You can delete this later if you want. It's just to keep you focused while you're actually doing the review piece. You can refer back to that when you're actually doing the writing in the paragraphs. And you should set forth your overall aims with regard to the central research questions, which will evolve while you're doing your research. So that aims section, you might want to return to it nearer the end. So here we have four sample subheadings with a colour code. We have Marxism in red, hybridity in blue, subaltern in purple, and eco-criticism in green. So you can see that the questions are very specific to what I want to address in a particular section of the review. You can see as well that I've listed the main names in particular subfields of post-colonial studies. All of this basically allows me to very rapidly recall information if I'm looking over the review again. It's all about making sure that you can recall things when you're glancing through this to make it as clear as possible to signpost it the whole way through. The colour code also just exercises other bits of your brain that reading in black and white don't. So it can be very useful for recalling information. So here we have a sample paragraph from a literature review. In this case, post-colonial approaches to Scottish literature. What I want you to notice is how some of the names are highlighted according to a colour code, and that two names are in bold to signal that they're from another literature review. So now what we're seeing is we're cross-referencing between different literature reviews. They're all speaking to our own overall understanding of subject areas. So you're beginning to increase your own pool of knowledge. You're getting more comfortable with talking about work in your field. So I'm just going to read through the piece now. In his introduction to Scottish literature and post-colonial literature, the first comparative collection of its kind, Michael Gardner claims that by learning the lessons of the post-colonial English literature departments teaching Scottish texts will prevent issues of Celtic lavism, the Scottish greats and canon formation in the English style. Yet investigation suggests that the collection is one of canon authors in the post-colonial framework, perhaps in an effort to call to mind a stability through familiarity, which does not really exist in post-colonial studies, still a hugely contentious field. As Liam Connell suggests, the academic capital of post-colonial theory resulted in Scottish literary academics applying post-colonial models to Scottish literature and culture to widen the interest globally, particularly in North American universities. Yet Scottish post-colonial interventions do recognise the importance of material history and provide provocative readings, particularly in refining the coloniser-colonised dualism in Scotland, as will now be demonstrated. So you can see this is a paragraph that sets up the next paragraph. You can see that there is an essential quotation on the third line, lessons of the post-colonial. The reasons you might want to leave a quotation in are because you think it particularly captures something in a book or article that you're reviewing. So you can see there that you can actually summarise positions in a couple of lines over a paragraph. That's how the review should work. It should be that quick. So now concluding the review, what are the major trends and who are the major contributors? Just make a note of that. And most importantly, what gaps do you feel can be exploited or require more analysis? Can you contribute? You need to establish this in your concluding section, just to show what was the point of the review in the first place. And finally, you need to ask yourself, is the review worth continuing and updating or are you happy that any further research would require a wholly new approach? If by the end of your review and you've concluded it, say you've taken three or four weeks of it, you're still not happy about it, you need to talk to your supervisor. Don't just try and fix it on your own if you've already spent a substantial amount of time on it. You might need a bit of guidance. They might recommend that you get rid of some critics, incorporate others, maybe a new subject area to look at, that sort of thing. So you should just talk to your supervisor if that happens. So some useful shortcuts. Don't read everything. Take a short note of an abstract with a footnote. That's going back to how we take notes using Zotero. Take little notes while you're using Zotero of particularly useful points. Only list the principal authors below your subheadings. It's just very useful for looking over notes quickly before going into meetings, before going into conferences. You just want to refresh everything. Set a time limit, generally no more than four weeks, a couple of hours most days. If it's getting out of control, talk to someone. And you should also look for reviews in major journals, book reviews or review articles that take on a couple of articles or a couple of books, chart a trajectory in a particular subfield of a topic, that sort of thing. I just want to show an example of that now in this book review, Ireland and Eco-Criticism Towards a Trajectory. Now, whether or not you're in this subject doesn't necessarily matter. You'll see reviews like this in peer review journals. As you can see, four books are listed at the beginning. And if you read the opening paragraph, it's a review piece that maps a trajectory. So even if you disagree with the findings, even if you disagree with the methodology, you might find that this particular piece is useful. It probably has other articles that might be of interest to your own work, and it will show you particular directions that the topic is going in that you can look at. So now we just have some other resources that you can have a look at to improve your own literature reviews. Some of these are subject specific, but they're all very useful in their own ways. And the last resource has sample literature reviews for subject specific areas. So you might find your own subject in that particular website. So thank you very much for watching the UCD Writing Centre presentation, How to Write a Literature Review Without Melting Down. I wish you the very best of luck in your writing.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Literature review on collaborative project delivery for sustainable construction: bibliometric analysis.

1. Introduction

2. literature review, 2.1. collaborative project delivery, 2.2. design build (db), 2.3. construction manager at risk (cmar), 2.4. integrated project delivery method (ipd), 2.5. sustainability, 2.6. sustainable construction, 2.7. benefits of eci comparing case studies, 2.8. collaborative delivery models, 3. methodology, 3.1. research methods, 3.2. database research, 4.1. ipd, design-build, and cmar overview, 4.1.1. yearly publication distribution of db cmar and ipd, 4.1.2. major country analysis, 4.1.3. most relevant and influential journals, 4.1.4. corresponding author countries, 4.2. keyword analysis, 4.2.1. high-frequency keyword analysis, 4.2.2. co-occurrence network analysis, 4.2.3. analysis of keywords’ frequency over time, 5. discussion, 5.1. findings of advantages and disadvantages of ipd, db, and cmar for sustainable construction, 5.1.1. advantages of ipd, 5.1.2. advantages of design-build, 5.1.3. advantages of construction manager at risk, 5.1.4. disadvantages of ipd, 5.1.5. disadvantages of design-build, 5.1.6. disadvantages of construction manager at risk, 5.2. most suitable cpd technique for sustainable construction based on literature review, 5.2.1. limitations, 5.2.2. recommendations for future research, 6. future trend, 6.1. enhancing innovation through collaborative project delivery, 6.2. open communication and block chain technology, 6.3. multi-party agreement, 6.4. utilizing artificial intelligence in decision support systems, 7. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Giachino, J.; Cecil, M.; Husselbee, B.; Matthews, C. Alternative Project Delivery: Construction Management at Risk, Design-Build and Public-Private Partnerships. In Proceedings of the Utility Management Conference 2016, San Diego, CA, USA, 24–26 February 2016. [ Google Scholar ]
• Shrestha, P.P.; Maharjan, R.; Batista, J.R. Performance of Design-Build and Construction Manager-at-Risk Methods in Water and Wastewater Projects. Pract. Period. Struct. Des. Constr. 2019 , 24 , 04018029. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Batista, J. Lessons Learned in Design-Build and Construction-Manager-at-Risk Water and Wastewater Project. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520002. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, A.P.C. Identification of Selection Criteria for Operational Variations of The Design-Build System: A Delphi Study in China. J. Civ. Eng. Manag. 2012 , 18 , 173–183. [ Google Scholar ] [ CrossRef ]
• Shane, J.S.; Bogus, S.M.; Molenaar, K.R. Municipal Water/Wastewater Project Delivery Performance Comparison. J. Manag. Eng. 2013 , 29 , 251–258. [ Google Scholar ] [ CrossRef ]
• Sullivan, J.; El Asmar, M.; Chalhoub, J.; Obeid, H. Two Decades of Performance Comparisons for Design-Build, Construction Manager at Risk, and Design-Bid-Build: Quantitative Analysis of the State of Knowledge on Project Cost, Schedule, and Quality. J. Constr. Eng. Manag. 2017 , 143 , 04017009. [ Google Scholar ] [ CrossRef ]
• Raouf, A.M.; Al-Ghamdi, S. Effectiveness of Project Delivery Systems in Executing Green Buildings. J. Constr. Eng. Manag. 2019 , 145 , 03119005. [ Google Scholar ] [ CrossRef ]
• Francom, T.; El Asmar, M.; Ariaratnam, S.T. Performance Analysis of Construction Manager at Risk on Pipeline Engineering and Construction Projects. J. Manag. Eng. 2016 , 32 , 04016016. [ Google Scholar ] [ CrossRef ]
• Gransberg, D.D.; Shane, J.S.; Transportation Research Board. Construction Manager-at-Risk Project Delivery for Highway Programs ; The National Academies Press: Washington, DC, USA, 2010. [ Google Scholar ]
• Rahman, M.M.; Kumaraswamy, M.M. Potential for Implementing Relational Contracting and Joint Risk Management. J. Manag. Eng. 2004 , 20 , 178–189. [ Google Scholar ] [ CrossRef ]
• Feghaly, J.; El Asmar, M.; Ariaratnam, S.; Bearup, W. Selecting project delivery methods for water treatment plants. Eng. Constr. Archit. Manag. 2019 , 27 , 936–951. [ Google Scholar ] [ CrossRef ]
• Park, H.-S.; Lee, D.; Kim, S.; Kim, J.-L. Comparing Project Performance of Design-Build and Design-Bid-Build Methods for Large-sized Public Apartment Housing Projects in Korea. J. Asian Archit. Build. Eng. 2015 , 14 , 323–330. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Batista, J.; Maharajan, R. Risks involved in using alternative project delivery (APD) methods in water and wastewater projects. Procedia Eng. 2016 , 145 , 219–223. [ Google Scholar ] [ CrossRef ]
• Hettiaarachchige, N.; Rathnasinghe, A.; Ranadewa, K.; Thurairajah, N. Thurairajah, Lean Integrated Project Delivery for Construction Procurement: The Case of Sri Lanka. Buildings 2022 , 12 , 524. [ Google Scholar ] [ CrossRef ]
• Kent, D.C.; Becerik-Gerber, B. Understanding Construction Industry Experience and Attitudes toward Integrated Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 815–825. [ Google Scholar ] [ CrossRef ]
• Franz, B.; Leicht, R.; Molenaar, K.; Messner, J. Impact of Team Integration and Group Cohesion on Project Delivery Performance. J. Constr. Eng. Manag. 2017 , 143 , 04016088. [ Google Scholar ] [ CrossRef ]
• Engebø, A.; Klakegg, O.J.; Lohne, J.; Lædre, O. A collaborative project delivery method for design of a high-performance building. Int. J. Manag. Proj. Bus. 2020 , 13 , 1141–1165. [ Google Scholar ] [ CrossRef ]
• Ahmed, S.; El-Sayegh, S. Critical Review of the Evolution of Project Delivery Methods in the Construction Industry. Buildings 2020 , 11 , 11. [ Google Scholar ] [ CrossRef ]
• Bond-Barnard, T.J.; Fletcher, L.; Steyn, H. Linking trust and collaboration in project teams to project management success. Int. J. Manag. Proj. Bus. 2018 , 11 , 432–457. [ Google Scholar ] [ CrossRef ]
• Rodrigues, M.R.; Lindhard, S.M. Lindhard, Benefits and challenges to applying IPD: Experiences from a Norwegian mega-project. Constr. Innov. 2021 , 23 , 287–305. [ Google Scholar ] [ CrossRef ]
• Kaminsky, J. The fourth pillar of infrastructure sustainability: Tailoring civil infrastructure to social context. Constr. Manag. Econ. 2015 , 33 , 299–309. [ Google Scholar ] [ CrossRef ]
• Al Khalil, M.I. Selecting the appropriate project delivery method using AHP. Int. J. Proj. Manag. 2002 , 20 , 469–474. [ Google Scholar ] [ CrossRef ]
• Ibbs, C.W.; Kwak, Y.H.; Ng, T.; Odabasi, A.M. Project Delivery Systems and Project Change: Quantitative Analysis. J. Constr. Eng. Manag. 2003 , 129 , 382–387. [ Google Scholar ] [ CrossRef ]
• Jansen, J.; Beck, A. Overcoming the Challenges of Large Diameter Water Project in North Texas via CMAR Delivery Method. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 264–271. [ Google Scholar ] [ CrossRef ]
• Bingham, E.; Gibson, G.E.; Asmar, M.E. Measuring User Perceptions of Popular Transportation Project Delivery Methods Using Least Significant Difference Intervals and Multiple Range Tests. J. Constr. Eng. Manag. 2018 , 144 , 04018033. [ Google Scholar ] [ CrossRef ]
• Cho, Y.J. A review of construction delivery systems: Focus on the construction management at risk system in the Korean public construction market. KSCE J. Civ. Eng. 2016 , 20 , 530–537. [ Google Scholar ] [ CrossRef ]
• Rosayuru, H.D.R.R.; Waidyasekara, K.G.A.S.; Wijewickrama, M.K.C.S. Sustainable BIM based integrated project delivery system for construction industry in Sri Lanka. Int. J. Constr. Manag. 2022 , 22 , 769–783. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P.; Beliveau, Y.J. Symbiotic Relationships between Integrated Project Delivery (IPD) and Trust. Int. J. Constr. Educ. Res. 2016 , 12 , 179–192. [ Google Scholar ] [ CrossRef ]
• Sherif, M.; Abotaleb, I.; Alqahtani, F.K. Alqahtani, Application of Integrated Project Delivery (IPD) in the Middle East: Implementation and Challenges. Buildings 2022 , 12 , 467. [ Google Scholar ] [ CrossRef ]
• Manata, B.; Garcia, A.J.; Mollaoglu, S.; Miller, V.D. The effect of commitment differentiation on integrated project delivery team dynamics: The critical roles of goal alignment, communication behaviors, and decision quality. Int. J. Proj. Manag. 2021 , 39 , 259–269. [ Google Scholar ] [ CrossRef ]
• Kraatz, J.A.; Sanchez, A.X.; Hampson, K.D. Hampson, Digital Modeling, Integrated Project Delivery and Industry Transformation: An Australian Case Study. Buildings 2014 , 4 , 453–466. [ Google Scholar ] [ CrossRef ]
• Zhang, L.; He, J.; Zhou, S. Sharing Tacit Knowledge for Integrated Project Team Flexibility: Case Study of Integrated Project Delivery. J. Constr. Eng. Manag. 2013 , 139 , 795–804. [ Google Scholar ] [ CrossRef ]
• El Asmar, M.; Hanna, A.S.; Loh, W.-Y. Quantifying Performance for the Integrated Project Delivery System as Compared to Established Delivery Systems. J. Constr. Eng. Manag. 2013 , 139 , 04013012. [ Google Scholar ] [ CrossRef ]
• Ghassemi, R.; Becerik-Gerber, B. Transitioning to integrated project delivery: Potential barriers and lessons learned. Lean Constr. J. 2011 , 32–52. Available online: https://leanconstruction.org/resources/lean-construction-journal/lcj-back-issues/2011-issue/ (accessed on 11 August 2024).
• Mei, T.; Guo, Z.; Li, P.; Fang, K.; Zhong, S. Influence of Integrated Project Delivery Principles on Project Performance in China: An SEM-Based Approach. Sustainability 2022 , 14 , 4381. [ Google Scholar ] [ CrossRef ]
• Ilozor, B.D.; Kelly, D.J. Building information modeling and integrated project delivery in the commercial construction industry: A conceptual study. J. Eng. Proj. Prod. Manag. 2012 , 2 , 23–36. [ Google Scholar ] [ CrossRef ]
• Zabihi, H.; Habib, F.; Mirsaeedie, L. Sustainability in Building and Construction: Revising Definitions and Concepts. Int. J. Emerg. Sci. 2012 , 2 , 570–578. [ Google Scholar ]
• Young, J.W.S. A Framework for the Ultimate Environmental Index—Putting Atmospheric Change Into Context With Sustainability. Environ. Monit. Assess. 1997 , 46 , 135–149. [ Google Scholar ] [ CrossRef ]
• Ding, G.K.C. Sustainable construction—The role of environmental assessment tools. J. Environ. Manag. 2008 , 86 , 451–464. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Conte, E. The Era of Sustainability: Promises, Pitfalls and Prospects for Sustainable Buildings and the Built Environment. Sustainability 2018 , 10 , 2092. [ Google Scholar ] [ CrossRef ]
• Standardized Method of Life Cycle Costing for Construction Procurement. A Supplement to BS ISO 15686-5. Buildings and Constructed Assets. Service Life Planning. Life Cycle Costing ; BSI British Standards: London, UK, 2008. [ CrossRef ]
• Sustainability|Free Full-Text|A Hybrid Multi-Criteria Decision Support System for Selecting the Most Sustainable Structural Material for a Multistory Building Construction. Available online: https://www.mdpi.com/2071-1050/15/4/3128 (accessed on 2 April 2024).
• Korkmaz, S.; Riley, D.; Horman, M. Piloting Evaluation Metrics for Sustainable High-Performance Building Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 877–885. [ Google Scholar ] [ CrossRef ]
• Ng, M.S.; Graser, K.; Hall, D.M. Digital fabrication, BIM and early contractor involvement in design in construction projects: A comparative case study. Archit. Eng. Des. Manag. 2021 , 19 , 39–55. [ Google Scholar ] [ CrossRef ]
• Moradi, S.; Kähkönen, K.; Sormunen, P. Analytical and Conceptual Perspectives toward Behavioral Elements of Collaborative Delivery Models in Construction Projects. Buildings 2022 , 12 , 316. [ Google Scholar ] [ CrossRef ]
• Zupic, I.; Čater, T. Bibliometric Methods in Management and Organization. 2015. Available online: https://journals.sagepub.com/doi/abs/10.1177/1094428114562629 (accessed on 3 April 2024).
• Rozas, L.W.; Klein, W.C. The Value and Purpose of the Traditional Qualitative Literature Review. J. Evid.-Based Soc. Work 2010 , 7 , 387–399. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Cobo, M.J.; López-Herrera, A.G.; Herrera-Viedma, E.; Herrera, F. Science mapping software tools: Review, analysis, and cooperative study among tools. J. Am. Soc. Inf. Sci. Technol. 2011 , 62 , 1382–1402. [ Google Scholar ] [ CrossRef ]
• Cancino, C.A.; Merigó, J.M.; Coronado, F.C. A bibliometric analysis of leading universities in innovation research. J. Innov. Knowl. 2017 , 2 , 106–124. [ Google Scholar ] [ CrossRef ]
• Pedro, L.F.M.G.; Barbosa, C.M.M.d.O.; Santos, C.M.d.N. A critical review of mobile learning integration in formal educational contexts. Int. J. Educ. Technol. High. Educ. 2018 , 15 , 10. [ Google Scholar ] [ CrossRef ]
• Wen, S.; Tang, H.; Ying, F.; Wu, G. Exploring the Global Research Trends of Supply Chain Management of Construction Projects Based on a Bibliometric Analysis: Current Status and Future Prospects. Buildings 2023 , 13 , 373. [ Google Scholar ] [ CrossRef ]
• Hosseini, M.R.; Martek, I.; Zavadskas, E.K.; Aibinu, A.A.; Arashpour, M.; Chileshe, N. Critical evaluation of off-site construction research: A Scientometric analysis. Autom. Constr. 2018 , 87 , 235–247. [ Google Scholar ] [ CrossRef ]
• Toyin, J.O.; Mewomo, M.C. Mewomo, Overview of BIM contributions in the construction phase: Review and bibliometric analysis. J. Inf. Technol. Constr. 2023 , 28 , 500–514. [ Google Scholar ] [ CrossRef ]
• Kahvandi, Z.; Saghatforoush, E.; Alinezhad, M.; Noghli, F. Integrated Project Delivery (IPD) Research Trends. J. Eng. 2017 , 7 , 99–114. [ Google Scholar ] [ CrossRef ]
• Hale, D.R.; Shrestha, P.P.; Gibson, G.E.; Migliaccio, G.C. Empirical Comparison of Design/Build and Design/Bid/Build Project Delivery Methods. J. Constr. Eng. Manag. 2009 , 135 , 579–587. [ Google Scholar ] [ CrossRef ]
• Mollaoglu-Korkmaz, S.; Swarup, L.; Riley, D. Delivering Sustainable, High-Performance Buildings: Influence of Project Delivery Methods on Integration and Project Outcomes. J. Manag. Eng. 2013 , 29 , 71–78. [ Google Scholar ] [ CrossRef ]
• Ugwu, O.O.; Haupt, T.C. Key performance indicators and assessment methods for infrastructure sustainability—a South African construction industry perspective. Build. Environ. 2007 , 42 , 665–680. [ Google Scholar ] [ CrossRef ]
• Kines, P.; Andersen, L.P.S.; Spangenberg, S.; Mikkelsen, K.L.; Dyreborg, J.; Zohar, D. Improving construction site safety through leader-based verbal safety communication. J. Safety Res. 2010 , 41 , 399–406. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ballard, G. The Lean Project Delivery System: An Update. 2008. [ Google Scholar ]
• Bynum, P.; Issa, R.R.A.; Olbina, S. Building information modeling in support of sustainable design and construction. J. Constr. Eng. Manag. 2013 , 139 , 24–34. [ Google Scholar ] [ CrossRef ]
• Choudhry, R.M.; Fang, D.; Lingard, H. Measuring Safety Climate of a Construction Company. J. Constr. Eng. Manag. 2009 , 135 , 890–899. [ Google Scholar ] [ CrossRef ]
• Wardani, M.A.E.; Messner, J.I.; Horman, M.J. Comparing procurement methods for Design-Build projects. J. Constr. Eng. Manag. 2006 , 132 , 230–238. [ Google Scholar ] [ CrossRef ]
• Liu, J.; Zhao, X.; Yan, P. Risk Paths in International Construction Projects: Case Study from Chinese Contractors. J. Constr. Eng. Manag. 2016 , 142 . [ Google Scholar ] [ CrossRef ]
• El-Sayegh, S. Evaluating the effectiveness of project delivery methods. J. Constr. Manag. Econ. 2008 , 23 , 457–465. [ Google Scholar ]
• Fang, C.; Marle, F.; Zio, E.; Bocquet, J.-C. Network theory-based analysis of risk interactions in large engineering projects. Reliability Eng. Syst. Safety 2012 , 106 , 1–10. [ Google Scholar ] [ CrossRef ]
• Franz, B.; Leicht, R.M. Initiating IPD Concepts on Campus Facilities with a ‘Collaboration Addendum’. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012; pp. 61–70. [ Google Scholar ] [ CrossRef ]
• Kim, H.; Kim, K.; Kim, H. Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects. J. Comput. Civil Eng. 2016 , 30 . [ Google Scholar ] [ CrossRef ]
• Zhou, Y.; Ding, L.Y.; Chen, L.J. Application of 4D visualization technology for safety management in metro construction. Automation Constr. 2013 , 34 , 25–36. [ Google Scholar ] [ CrossRef ]
• Wanberg, J.; Harper, C.; Hallowell, M.R.; Rajendran, S. Relationship between Construction Safety and Quality Performance. J. Constr. Eng. Manag. 2013 , 139 . [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; O’Connor, J.T.; Gibson, G.E. Performance comparison of large Design-Build and Design-Bid-Build highway projects. J. Constr. Eng. Manag. 2012 , 138 , 1–13. [ Google Scholar ] [ CrossRef ]
• Torabi, S.A.; Hassini, E. Multi-site production planning integrating procurement and distribution plans in multi-echelon supply chains: An interactive fuzzy goal programming approach. Int. J. Prod. Res. 2009 , 47 , 5475–5499. [ Google Scholar ] [ CrossRef ]
• Baradan, S.; Usmen, M. Comparative Injury and Fatality Risk Analysis of Building Trades. J. Constr. Eng. Manag.-ASCE 2006 , 132 . [ Google Scholar ] [ CrossRef ]
• Levitt, R.E. CEM Research for the Next 50 Years: Maximizing Economic, Environmental, and Societal Value of the Built Environment1. J. Constr. Eng. Manag. 2007 , 133 , 619–628. [ Google Scholar ] [ CrossRef ]
• Araya, F. Modeling the spread of COVID-19 on construction workers: An agent-based approach. Saf. Sci. 2021 , 133 , 105022. [ Google Scholar ] [ CrossRef ]
• Zheng, X.; Le, Y.; Chan, A.P.; Hu, Y.; Li, Y. Review of the application of social network analysis (SNA) in construction project management research. Int. J. Proj. Manag. 2016 , 34 , 1214–1225. [ Google Scholar ] [ CrossRef ]
• Elghaish, F.; Abrishami, S. A centralised cost management system: Exploiting EVM and ABC within IPD. Eng. Constr. Archit. Manag. 2021 , 28 , 549–569. [ Google Scholar ] [ CrossRef ]
• Smith, R.E.; Mossman, A.; Emmitt, S. Lean and integrated project delivery. Lean Constr. J. 2011 , 1–16. [ Google Scholar ]
• Bröchner, J.; Badenfelt, U. Changes and change management in construction and IT projects. Autom. Constr. 2011 , 20 , 767–775. [ Google Scholar ] [ CrossRef ]
• Monteiro, A.; Mêda, P.; Martins, J.P. Framework for the coordinated application of two different integrated project delivery platforms. Autom. Constr. 2014 , 38 , 87–99. [ Google Scholar ] [ CrossRef ]
• Azhar, N.; Kang, Y.; Ahmad, I.U. Factors influencing integrated project delivery in publicly owned construction projects: An information modelling perspective. Procedia Eng. 2014 , 77 , 213–221. [ Google Scholar ] [ CrossRef ]
• Mihic, M.; Sertic, J.; Zavrski, I. Integrated Project Delivery as Integration between Solution Development and Solution Implementation. Procedia Soc. Behav. Sci. 2014 , 119 , 557–565. [ Google Scholar ] [ CrossRef ]
• Nawi, M.N.M.; Haron, A.T.; Hamid, Z.A.; Kamar, K.A.M.; Baharuddin, Y. Improving integrated practice through building information modeling-integrated project delivery (BIM-IPD) for Malaysian industrialised building system (IBS) Construction Projects. Malays. Constr. Res. J. 2014 , 15 , 29–38. Available online: https://dsgate.uum.edu.my/jspui/handle/123456789/1651 (accessed on 24 April 2024).
• Ma, Z.; Zhang, D.; Li, J. A dedicated collaboration platform for Integrated Project Delivery. Autom. Constr. 2018 , 86 , 199–209. [ Google Scholar ] [ CrossRef ]
• Yadav, S.; Kanade, G. Application of Revit as Building Information Modeling (BIM) for Integrated Project Delivery (IPD) to Building Construction Project—A Review. Int. Res. J. Eng. Technol. 2018 , 5 , 11–14. [ Google Scholar ]
• Salim, M.S.; Mahjoob, A.M.R. Integrated project delivery (IPD) method with BIM to improve the project performance: A case study in the Republic of Iraq. Asian J. Civ. Eng. 2020 , 21 , 947–957. [ Google Scholar ] [ CrossRef ]
• Ling, Y.Y.; Lau, B.S.Y. A case study on the management of the development of a large-scale power plant project in East Asia based on design-build arrangement. Int. J. Proj. Manag. 2002 , 20 , 413–423. [ Google Scholar ] [ CrossRef ]
• Dalui, P.; Elghaish, F.; Brooks, T.; McIlwaine, S. Integrated Project Delivery with BIM: A Methodical Approach Within the UK Consulting Sector. J. Inf. Technol. Constr. 2021 , 26 , 922–935. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P. Case Studies on the Role of Integrated Project Delivery (IPD) Approach on the Establishment and Promotion of Trust. Int. J. Constr. Educ. Res. 2017 , 13 , 102–124. [ Google Scholar ] [ CrossRef ]
• Singleton, M.S.; Hamzeh, F.R. Implementing integrated project delivery on department of the navy construction projects: Lean Construction Journal. Lean Constr. J. 2011 , 17–31. [ Google Scholar ]
• Tran, D.Q.; Nguyen, L.D.; Faught, A. Examination of communication processes in design-build project delivery in building construction. Eng. Constr. Archit. Manag. 2017 , 24 , 1319–1336. [ Google Scholar ] [ CrossRef ]
• Park, J.; Kwak, Y.H. Design-Bid-Build (DBB) vs. Design-Build (DB) in the U.S. public transportation projects: The choice and consequences. Int. J. Proj. Manag. 2017 , 35 , 280–295. [ Google Scholar ] [ CrossRef ]
• Wiss, R.A.; Roberts, R.T.; Phraner, S.D. Beyond Design-Build-Operate-Maintain: New Partnership Approach Toward Fixed Guideway Transit Projects. Transp. Res. Rec. J. Transp. Res. Board 2000 , 1704 , 13–18. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, A.P. Key competences of design-build clients in China. J. Facil. Manag. 2010 , 8 , 114–129. [ Google Scholar ] [ CrossRef ]
• DeBernard, D.M. Beyond Collaboration—The Benefits of Integrated Project Delivery ; AIA Soloso Website: Washington, DC, USA, 2008. [ Google Scholar ]
• Chen, Q.; Jin, Z.; Xia, B.; Wu, P.; Skitmore, M. Time and Cost Performance of Design–Build Projects. J. Constr. Eng. Manag. 2016 , 142 , 04015074. [ Google Scholar ] [ CrossRef ]
• Xia, B.; Chan, P. Review of the design-build market in the People’s Republic of China. J. Constr. Procure. 2008 , 14 , 108–117. [ Google Scholar ]
• Mcwhirt, D.; Ahn, J.; Shane, J.S.; Strong, K.C. Military construction projects: Comparison of project delivery methods. J. Facil. Manag. 2011 , 9 , 157–169. [ Google Scholar ] [ CrossRef ]
• Minchin, R.E.; Li, X.; Issa, R.R.; Vargas, G.G. Comparison of Cost and Time Performance of Design-Build and Design-Bid-Build Delivery Systems in Florida. J. Constr. Eng. Manag. 2013 , 139 , 04013007. [ Google Scholar ] [ CrossRef ]
• Adamtey, S.; Onsarigo, L. Effective tools for projects delivered by progressive design-build method. In Proceedings of the CSCE Annual Conference 2019, Laval, QC, Canada, 12–15 June 2019; pp. 1–10. [ Google Scholar ]
• Adamtey, S.A. A Case Study Performance Analysis of Design-Build and Integrated Project Delivery Methods. Int. J. Constr. Educ. Res. 2021 , 17 , 68–84. [ Google Scholar ] [ CrossRef ]
• Gad, G.M.; Adamtey, S.A.; Gransberg, D.D. Gransberg, Trends in Quality Management Approaches to Design–Build Transportation Projects. Transp. Res. Rec. J. Transp. Res. Board. 2015 , 2504 , 87–92. [ Google Scholar ] [ CrossRef ]
• Sari, E.M.; Irawan, A.P.; Wibowo, M.A.; Siregar, J.P.; Praja, A.K.A. Project delivery systems: The partnering concept in integrated and non-integrated construction projects. Sustainability 2022 , 15 , 86. [ Google Scholar ] [ CrossRef ]
• Chakra, H.A.; Ashi, A. Comparative analysis of design/build and design/bid/build project delivery systems in Lebanon. J. Ind. Eng. Int. 2019 , 15 , 147–152. [ Google Scholar ] [ CrossRef ]
• Perkins, R.A. Sources of Changes in Design–Build Contracts for a Governmental Owner. J. Constr. Eng. Manag. 2009 , 135 , 588–593. [ Google Scholar ] [ CrossRef ]
• Palaneeswaran, E.; Kumaraswamy, M.M. Contractor Selection for Design/Build Projects. J. Constr. Eng. Manag. 2000 , 126 , 331–339. [ Google Scholar ] [ CrossRef ]
• Chan, A.P.C. Evaluation of enhanced design and build system a case study of a hospital project. Constr. Manag. Econ. 2000 , 18 , 863–871. [ Google Scholar ] [ CrossRef ]
• Shrestha, P.P.; Davis, B.; Gad, G.M. Investigation of Legal Issues in Construction-Manager-at-Risk Projects: Case Study of Airport Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520022. [ Google Scholar ] [ CrossRef ]
• Marston, S. CMAR Project Delivery Method Generates Team Orientated Project Management with Win/Win Mentality. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; pp. 167–170. [ Google Scholar ] [ CrossRef ]
• Francom, T.; El Asmar, M.; Ariaratnam, S.T. Ariaratnam, Longitudinal Study of Construction Manager at Risk for Pipeline Rehabilitation. J. Pipeline Syst. Eng. Pract. 2017 , 8 , 04017001. [ Google Scholar ] [ CrossRef ]
• Peña-Mora, F.; Tamaki, T. Effect of Delivery Systems on Collaborative Negotiations for Large-Scale Infrastructure Projects. J. Manag. Eng. 2001 , 17 , 105–121. [ Google Scholar ] [ CrossRef ]
• Mahdi, I.M.; Alreshaid, K. Decision support system for selecting the proper project delivery method using analytical hierarchy process (AHP). Int. J. Proj. Manag. 2005 , 23 , 564–572. [ Google Scholar ] [ CrossRef ]
• Randall, T.; Pool, S.; Limke, J.; Bradney, A. CMaR Delivery of Critical Water and Wastewater Pipelines. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 280–289. [ Google Scholar ] [ CrossRef ]
• Perrenoud, A.; Reyes, M.; Ghosh, S.; Coetzee, M. Collaborative Risk Management of the Approval Process of Building Envelope Materials. In Proceedings of the AEI 2017, Oklahoma City, OK, USA, 11–13 April 2017; pp. 806–816. [ Google Scholar ] [ CrossRef ]
• Parrott, B.C.; Bomba, M.B. Integrated Project Delivery and Building Information Modeling: A New Breed of Contract. 2010. Available online: https://content.aia.org/sites/default/files/2017-03/Integrated%20project%20delivery%20and%20BIM-%20A%20new%20breed%20of%20contract.pdf (accessed on 18 November 2023).
• Cheng, R. IPD Case Studies. Report. March 2012. Available online: http://conservancy.umn.edu/handle/11299/201408 (accessed on 1 May 2024).
• Lee, H.W.; Anderson, S.M.; Kim, Y.-W.; Ballard, G. Ballard, Advancing Impact of Education, Training, and Professional Experience on Integrated Project Delivery. Pract. Period. Struct. Des. Constr. 2014 , 19 , 8–14. [ Google Scholar ] [ CrossRef ]
• Hoseingholi, M.; Jalal, M.P. Jalal, Identification and Analysis of Owner-Induced Problems in Design–Build Project Lifecycle. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2017 , 9 , 04516013. [ Google Scholar ] [ CrossRef ]
• Öztaş, A.; Ökmen, Ö. Risk analysis in fixed-price design–build construction projects. Build. Environ. 2004 , 39 , 229–237. [ Google Scholar ] [ CrossRef ]
• Lee, D.-E.; Arditi, D. Total Quality Performance of Design/Build Firms Using Quality Function Deployment. J. Constr. Eng. Manag. 2006 , 132 , 49–57. [ Google Scholar ] [ CrossRef ]
• Garner, B.; Richardson, K.; Castro-Lacouture, D. Design-Build Project Delivery in Military Construction: Approach to Best Value Procurement. J. Adv. Perform. Inf. Value 2008 , 1 , 35–50. [ Google Scholar ] [ CrossRef ]
• Graham, P. Evaluation of Design-Build Practice in Colorado Project IR IM(CX) 025-3(113) ; Colorado Department of Transportation: Denver, CO, USA, 2001. [ Google Scholar ]
• Parami Dewi, A.; Too, E.; Trigunarsyah, B. Implementing design build project delivery system in Indonesian road infrastructure projects. In Innovation and Sustainable Construction in Developing Countries (CIB W107 Conference 2011) ; Uwakweh, B.O., Ed.; Construction Publishing House/International Council for Research and Innovation in Building and C: Hanoi, Vietnam, 2011; pp. 108–117. [ Google Scholar ]
• Arditi, D.; Lee, D.-E. Assessing the corporate service quality performance of design-build contractors using quality function deployment. Constr. Manag. Econ. 2003 , 21 , 175–185. [ Google Scholar ] [ CrossRef ]
• Rao, T. . Is Design-Build Right for Your Next WWW Project? presented at the WEFTEC 2009, Water Environment Federation. January 2009, pp. 6444–6458. Available online: https://www.accesswater.org/publications/proceedings/-297075/is-design-build-right-for-your-next-www-project- (accessed on 3 April 2024).
• Touran, A.; Molenaar, K.R.; Gransberg, D.D.; Ghavamifar, K. Decision Support System for Selection of Project Delivery Method in Transit. Transp. Res. Rec. 2009 , 2111 , 148–157. [ Google Scholar ] [ CrossRef ]
• Culp, G. Alternative Project Delivery Methods for Water and Wastewater Projects: Do They Save Time and Money? Leadersh. Manag. Eng. 2011 , 11 , 231–240. [ Google Scholar ] [ CrossRef ]
• Ling, F.Y.Y.; Poh, B.H.M. Problems encountered by owners of design–build projects in Singapore. Int. J. Proj. Manag. 2008 , 26 , 164–173. [ Google Scholar ] [ CrossRef ]
• Pishdad-Bozorgi, P.; de la Garza, J.M. Comparative Analysis of Design-Bid-Build and Design-Build from the Standpoint of Claims. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012. [ Google Scholar ] [ CrossRef ]
• Walewski, J.; Gibson, G.E., Jr.; Jasper, J. Project Delivery Methods and Contracting Approaches Available for Implementation by the Texas Department of Transportation. University of Texas at Austin. Center for Transportation Research. 2001. Available online: https://rosap.ntl.bts.gov/view/dot/14863 (accessed on 3 April 2024).
• Alleman, D.; Antoine, A.; Gransberg, D.D.; Molenaar, K.R. Comparison of Qualifications-Based Selection and Best-Value Procurement for Construction Manager–General Contractor Highway Construction. 2017. Available online: https://journals.sagepub.com/doi/abs/10.3141/2630-08 (accessed on 2 April 2024).
• Gransberg, N.J.; Gransberg, D.D. Public Project Construction Manager-at-Risk Contracts: Lessons Learned from a Comparison of Commercial and Infrastructure Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04519039. [ Google Scholar ] [ CrossRef ]
• Anderson, S.D.; Damnjanovic, I. Selection and Evaluation of Alternative Contracting Methods to Accelerate Project Completion ; The National Academies Press: Washington, DC, USA, 2008; Available online: http://elibrary.pcu.edu.ph:9000/digi/NA02/2008/23075.pdf (accessed on 26 April 2024).
• Shrestha, P.P.; Batista, J.; Maharjan, R. Impediments in Using Design-Build or Construction Management-at-Risk Delivery Methods for Water and Wastewater Projects. In Proceedings of the Construction Research Congress 2016, San Juan, PR, USA, 31 May–2 June 2016; pp. 380–387. [ Google Scholar ] [ CrossRef ]
• Chateau, L. Environmental acceptability of beneficial use of waste as construction material—State of knowledge, current practices and future developments in Europe and in France. J. Hazard. Mater. 2007 , 139 , 556–562. [ Google Scholar ] [ CrossRef ]
• Lam, T.I.; Chan, H.W.E.; Chau, C.K.; Poon, C.S. An Overview of the Development of Green Specifications in the Construction Industry. In Proceedings of the International Conference on Urban Sustainability [ICONUS], 1 January 2008; pp. 295–301. Available online: https://research.polyu.edu.hk/en/publications/an-overview-of-the-development-of-green-specifications-in-the-con (accessed on 2 May 2024).
• Tabish, S.Z.S.; Jha, K.N. Success Traits for a Construction Project. J. Constr. Eng. Manag. 2012 , 138 , 1131–1138. [ Google Scholar ] [ CrossRef ]
• Niroumand, H.; Zain, M.; Jamil, M. A guideline for assessing of critical parameters on Earth architecture and Earth buildings as a sustainable architecture in various countries. Renew. Sustain. Energy Rev. 2013 , 28 , 130–165. [ Google Scholar ] [ CrossRef ]
• Rogulj, K.; Jajac, N. Achieving a Construction Barrier–Free Environment: Decision Support to Policy Selection. J. Manag. Eng. 2018 , 34 , 04018020. [ Google Scholar ] [ CrossRef ]
• Sackey, S.; Kim, B.-S. Environmental and Economic Performance of Asphalt Shingle and Clay Tile Roofing Sheets Using Life Cycle Assessment Approach and TOPSIS. J. Constr. Eng. Manag. 2018 , 144 , 04018104. [ Google Scholar ] [ CrossRef ]
• Carretero-Ayuso, M.J.; García-Sanz-Calcedo, J.; Rodríguez-Jiménez, C.E. Rodríguez-Jiménez, Characterization and Appraisal of Technical Specifications in Brick Façade Projects in Spain. J. Perform. Constr. Facil. 2018 , 32 , 04018012. [ Google Scholar ] [ CrossRef ]
• Golabchi, A.; Guo, X.; Liu, M.; Han, S.; Lee, S.; AbouRizk, S. An integrated ergonomics framework for evaluation and design of construction operations. Autom. Constr. 2018 , 95 , 72–85. [ Google Scholar ] [ CrossRef ]
• Jha, K.; Iyer, K. Commitment, coordination, competence and the iron triangle. Int. J. Proj. Manag. 2007 , 25 , 527–540. [ Google Scholar ] [ CrossRef ]
• Tabassi, A.A.; Ramli, M.; Roufechaei, K.M.; Tabasi, A.A. Team development and performance in construction design teams: An assessment of a hierarchical model with mediating effect of compensation. Constr. Manag. Econ. 2014 , 32 , 932–949. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Okudan, G.E.; Riley, D.R. Sustainable performance criteria for construction method selection in concrete buildings. Autom. Constr. 2010 , 19 , 235–244. [ Google Scholar ] [ CrossRef ]
• Doloi, H.; Sawhney, A.; Iyer, K.; Rentala, S. Analysing factors affecting delays in Indian construction projects. Int. J. Proj. Manag. 2012 , 30 , 479–489. [ Google Scholar ] [ CrossRef ]
• Kog, Y.C.; Loh, P.K. Critical Success Factors for Different Components of Construction Projects. J. Constr. Eng. Manag. 2012 , 138 , 520–528. [ Google Scholar ] [ CrossRef ]
• Gunduz, M.; Almuajebh, M. Critical success factors for sustainable construction project management. Sustainability 2020 , 12 , 1990. [ Google Scholar ] [ CrossRef ]
• Cao, D.; Li, H.; Wang, G.; Luo, X.; Tan, D. Relationship Network Structure and Organizational Competitiveness: Evidence from BIM Implementation Practices in the Construction Industry. J. Manag. Eng. 2018 , 34 , 04018005. [ Google Scholar ] [ CrossRef ]
• Clevenger, C.M. Development of a Project Management Certification Plan for a DOT. J. Manag. Eng. 2018 , 34 , 06018002. [ Google Scholar ] [ CrossRef ]
• Bygballe, L.E.; Swärd, A. Collaborative Project Delivery Models and the Role of Routines in Institutionalizing Partnering. Proj. Manag. J. 2019 , 50 , 161–176. [ Google Scholar ] [ CrossRef ]
• Collins, W.; Parrish, K. The Need for Integrated Project Delivery in the Public Sector. In Proceedings of the Construction Research Congress 2014, Atlanta, GA, USA, 19–21 May 2014; pp. 719–728. [ Google Scholar ] [ CrossRef ]
• Turk, Ž.; Klinc, R. Potentials of Blockchain Technology for Construction Management. Procedia Eng. 2017 , 196 , 638–645. [ Google Scholar ] [ CrossRef ]
• Elghaish, F.; Abrishami, S.; Hosseini, M.R. Integrated project delivery with blockchain: An automated financial system. Autom. Constr. 2020 , 114 , 103182. [ Google Scholar ] [ CrossRef ]
• Fish, A. Integrated Project Delivery: The Obstacles of Implementation. May 2011. Available online: http://hdl.handle.net/2097/8554 (accessed on 3 April 2024).
• Pan, Y.; Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 2020 , 122 , 103517. [ Google Scholar ] [ CrossRef ]
• Mellit, A.; Kalogirou, S.A. Artificial intelligence techniques for photovoltaic applications: A review. Prog. Energy Combust. Sci. 2008 , 34 , 574–632. [ Google Scholar ] [ CrossRef ]
• Smith, C.J.; Wong, A.T.C. Advancements in Artificial Intelligence-Based Decision Support Systems for Improving Construction Project Sustainability: A Systematic Literature Review. Informatics 2022 , 9 , 43. [ Google Scholar ] [ CrossRef ]
• Villa, F. Semantically driven meta-modelling: Automating model construction in an environmental decision support system for the assessment of ecosystem services flows. In Information Technologies in Environmental Engineering ; Athanasiadis, I.N., Rizzoli, A.E., Mitkas, P.A., Gómez, J.M., Eds.; Springer: Berlin, Heidelberg, 2009; pp. 23–36. [ Google Scholar ]
• Minhas, M.R.; Potdar, V. Decision Support Systems in Construction: A Bibliometric Analysis. Buildings 2020 , 10 , 108. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Babalola, O.G.; Alam Bhuiyan, M.M.; Hammad, A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability 2024 , 16 , 7707. https://doi.org/10.3390/su16177707

Babalola OG, Alam Bhuiyan MM, Hammad A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability . 2024; 16(17):7707. https://doi.org/10.3390/su16177707

Babalola, Olabode Gafar, Mohammad Masfiqul Alam Bhuiyan, and Ahmed Hammad. 2024. "Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis" Sustainability 16, no. 17: 7707. https://doi.org/10.3390/su16177707

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Pardon Our Interruption

As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:

• You've disabled JavaScript in your web browser.
• You're a power user moving through this website with super-human speed.
• You've disabled cookies in your web browser.
• A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this support article .

To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.

Effects of a State Pre-kindergarten Program on the Kindergarten Readiness and Attendance of At-Risk Four-Year-Olds

• Open access
• Published: 04 September 2024

Cite this article

You have full access to this open access article

• Jamie Heng-Chieh Wu 1 , 2 ,
• Hope Onyinye Akaeze 2 &
• Laurie A. Van Egeren 3  

The effect of public pre-kindergarten (pre-K) on the short-term outcomes of children from disadvantaged backgrounds is well established; however, the mechanisms for this effect are not well understood. Of the many factors that influence how pre-K participants progress during and after kindergarten, one understudied factor is the effect of pre-K participation on kindergarten attendance. The effects of absenteeism are cumulative, and habits established early in the school years are likely to affect later school outcomes. Thus, if pre-K improves kindergarten attendance, participants may be poised for later school success. To begin to test this hypothesis, we conducted a quasi-experimental study to examine the kindergarten readiness of 19,490 children and attendance records of 39,113 children who either were enrolled in Michigan’s Great Start Readiness Program (GSRP) or were placed on waitlists because their GSRP sites were full. Using variants of multilevel modeling, we found, as expected, that GSRP children performed better than waitlisted children on the Kindergarten Readiness Assessment. Examination of kindergarten attendance records found that waitlisted children were more likely to be absent than their counterparts who participated in GSRP, with particularly strong effects for children who were Black, economically disadvantaged, or English Language Learners.

Similar content being viewed by others

Direct and Indirect Impacts of Voluntary Pre-Kindergarten on Kindergarten Readiness and Achievement

Preparing Students for Success: Differential Outcomes by Preschool Experience in Baltimore City, Maryland

Comparing full-day and half-day kindergarten then and now: what data from two ecls-k cohorts reveal about changes in demographics and child outcomes.

Avoid common mistakes on your manuscript.

Publicly funded pre-kindergarten (pre-K) programs aim to improve the academic and social-emotional outcomes of young children, particularly those from disadvantaged backgrounds. The ongoing question in the literature and in the public square is whether and to what extent they do so. That center-based (as opposed to home-based) preschools help make children ready for kindergarten is well established (see meta-analyses and research compilations by Duncan & Magnuson, 2013 ; Fischer et al., 2020 ; Murano et al., 2020 ; Yoshikawa et al., 2016 ), though exceptions do exist (e.g., Allee et al., 2024 ). Effects on readiness skills have sometimes been found to be more pronounced for children from economically disadvantaged backgrounds and particularly for English Language Learners (ELLs, e.g., Bassok et al., 2019 ; Duncan & Magnuson, 2013 ; Fischer et al., 2020 ; Phillips et al., 2017 ; Watts et al., 2023 ).

Less clear is how preschool participation affects medium- and long-term outcomes. The few randomized control trials of publicly funded pre-K programs (Lipsey et al., 2018 ; Puma et al., 2010 , 2012 ; Weiland et al., 2020 ) and many quasi-experimental program evaluations (Duncan & Magnuson, 2013 ; Yoshikawa et al., 2016 ) have found that gains in children’s cognitive outcomes at the end of pre-K faded by third grade. The most recent results of the randomized control trial of Tennessee’s state-funded pre-K program show that program participants were significantly behind non-participants on several measures by grade 6 (Durkin et al., 2022 ); however, a long-term study of North Carolina’s program found lingering positive effects through grade 5 (Watts et al., 2023 ). Meanwhile, longer-term studies have often found positive effects of preschool participation on academic, social, and even economic outcomes in adolescence and young adulthood (e.g., Amadon et al., 2022 ; Duncan & Magnuson, 2013 ; McCoy et al., 2017 ).

The mechanisms by which preschool participation may affect later academic and social outcomes are not fully understood (Bassok et al., 2019 ; Heckman et al., 2013 ; Phillips et al., 2017 ). Certainly, the mechanisms are not limited to the effect of instruction on narrowly defined cognitive skills (Hustedt et al., 2018 ; Reynolds & Ou, 2016 ). For example, one study found that play-based preschool experiences improved kindergarten readiness by teaching children to learn, explore, communicate, and empathize (Fyffe et al., 2014 ). In addition to program content and approach, a wide variety of factors influence preschool and subsequent outcomes (Watts et al., 2023 ), including the quality of the preschool (e.g., Allee et al., 2024 ; Sylva et al., 2011 ; Yoshikawa et al., 2016 ), the quality of later education (e.g., Bailey et al., 2020 ; Phillips et al., 2017 ; Yoshikawa et al., 2016 ), and family and societal influences (e.g., Bivens et al., 2016 ; Burger, 2010 ). The influences on children’s development are many and complex.

One of the challenges in understanding whether and how pre-K participation affects students’ subsequent academic and social-emotional development is a relative dearth of information about what happens to children between kindergarten entry, when the effects of preschool on school readiness are assessed, and grade 3, when the first standardized tests of academic achievement are typically administered. To help fill in this gap, we studied how participation in a state-funded pre-K program affected both kindergarten readiness and kindergarten attendance. Our hypothesis was that pre-K participation would improve not only children’s kindergarten readiness but also their kindergarten attendance. Research suggests that pre-K impacts on school attendance may have lasting and cumulative effects on children’s subsequent academic and social-emotional development.

Literature Review

Kindergarten readiness.

Numerous studies have documented the positive effect of children’s academic readiness at kindergarten entry on later academic and socioeconomic performance (e.g., Davoudzadeh et al., 2015 ; Fitzpatrick et al., 2020 ; Quirk et al., 2017 ). Examinations of social-emotional readiness at kindergarten entry have found positive outcomes as much as 30 years later (Vergunst et al., 2019 ).

Definitions of kindergarten readiness are evolving, as kindergarten itself has evolved since the passage of the No Child Left Behind Act of ( 2001 ), to emphasize academic skills, acquired through paper-and-pencil tasks, over social and behavioral skills, acquired through play and child-directed activities (Bassok et al., 2016 ; Brown et al., 2023 ). Many believe, with Bassok et al. ( 2016 ), that kindergarten has become “the new first grade.” Meanwhile, qualitative studies have found that many preschool and some kindergarten teachers still tend to emphasize play, child-directed activities, and the development of social-emotional skills (Akaba et al., 2020 ; Brown et al., 2023 ; Hustedt et al., 2018 ). Such discrepancies between preschool and school programs and environments make the transition difficult for many children, perhaps particularly for children from low-income backgrounds (Cook & Coley, 2021 ; Pears & Peterson, 2018 ). Most researchers and educators (including those cited here and the Michigan Department of Education) include both academic and social-emotional milestones in their definition of kindergarten readiness.

Families’ involvement in young children’s learning and development is an important factor in kindergarten readiness and in child development generally. Home learning activities have been associated with the kindergarten readiness, defined in both academic and social-emotional terms, of low-income children (Barnett et al., 2020 ; Puccioni, 2018 ; Sheridan et al., 2020 ; Welsh et al., 2014 ). The quantity of learning activities parents provide at home can be influenced by preschool centers’ family engagement efforts (Barnett et al., 2020 ). Head Start parent-focused interventions have shown effects on kindergarten readiness that last at least into early elementary school (e.g., Bierman et al., 2018 , 2019 ). One family engagement tactic shown to influence academic learning and socioemotional development is home visits (Bierman et al., 2018 ; Loughlin-Presnal & Bierman, 2017 ; Nix et al., 2018 ).

Kindergarten Absenteeism

The prevalence of absenteeism in kindergarten is well established. Every year, about 10% of kindergarten children are chronically absent, defined as missing 18 or more days per year (Chang et al., 2015 ). The definition is equivalent to about 10% of the school year, and reflects the most common definition of chronic absenteeism selected by states under the Every Student Succeeds Act of 2015 (Jordan & Miller, 2017 ). Elementary school absence rates tend to be highest in kindergarten, decreasing year by year until at least grade 5 (Balfanz & Byrnes, 2012 ; Chang et al., 2015 ).

Attendance and Academic Outcomes

The link between attendance and academic outcomes is well established, though the phenomenon is less thoroughly studied among children in early elementary school than those in later grades. Children with better attendance rates in elementary grades have better academic outcomes in elementary school (Aucejo & Romano, 2016 ; Gottfried, 2009 , 2011 , 2019 ; Morrissey et al., 2014 ). Ansari and colleagues have found benefits of good elementary attendance that last into adolescence (Ansari & Pianta, 2019 ) and even young adulthood (Ansari et al., 2020 ). One study found that chronic absenteeism among students in grades 3 and 4 was associated with lower test scores not only for those students but among their classmates as well (Gottfried, 2019 ).

Though the impact of absenteeism specifically on test scores is greater in later grades (Gershenson et al., 2017 ), attendance in kindergarten matters in subtle ways. For one thing, some studies have shown that better attendance in kindergarten leads to better academic outcomes in kindergarten (Gershenson et al., 2017 ; Morrissey et al., 2014 ; Ready, 2010 ). Perhaps more importantly, children with better kindergarten attendance tend to have better attendance in first grade and beyond (Ansari & Pianta, 2019 ; Chang et al., 2015 ; Connolly & Olson, 2012 ). Conversely, chronic absenteeism in kindergarten has been linked to poor academic outcomes in first grade and beyond (Chang et al., 2015 ; Gottfried, 2014 ). These findings are particularly important in light of research suggesting that the effects of absence in early elementary school, starting in kindergarten, are cumulative (Ansari & Gottfried, 2021 ; Ansari et al., 2020 ): the more absences year over year, the greater the effect on academic and other outcomes.

Studies that focus, as ours does, on the effects of pre-K program participation on kindergarten attendance are fewer in number. A consensus for a positive effect is emerging, but with notable exceptions, which may result from methodological choices. The Tennessee randomized control trial, for example, found that the treatment group of children who attended the state pre-K program had a high average attendance rate of 95%, identical to that of the control group (Lipsey et al., 2018 ). However, focusing on aggregate attendance rates, as the Tennessee study does, can easily obscure the impact of absenteeism for individual students and at-risk groups (Chang et al., 2015 ). Furthermore, Balfanz and Byrnes ( 2012 ) point out that a district can have a 90% average daily attendance even though up to 40% of students miss 10% or more of school days.

Many analyses therefore examine the effects on individual children of chronic absenteeism. Gottfried ( 2015 ) found that enrollment in center-based preschool had a significant effect in reducing chronic absenteeism in kindergarten, regardless of family socioeconomic status (SES). He speculates that the mechanism by which preschool enrollment improves academic and socioemotional outcomes may be precisely that participation reduces chronic absenteeism in the early grades. Analysis of city-funded pre-K and Head Start programs in Baltimore (Connolly & Olson, 2012 ) showed that chronic absenteeism in pre-K was often but not consistently repeated in K–3; children who began to attend more consistently in kindergarten were likely to continue to do so in grades 1–3. Ehrlich et al. ( 2018 ) found that chronic absenteeism in pre-K predicted chronic absenteeism in K–2, with corresponding lack of progress in academic skills. Amadon et al. ( 2022 ) found that Tulsa participants in Oklahoma’s universal pre-K program had better attendance records and less chronic absenteeism than matched nonparticipants during kindergarten, elementary school, and high school (but not middle school).

Reasons for Absenteeism

Illness is the most common reason for school absence (Balfanz & Byrnes, 2012 ; Chang et al., 2015 ). However, childhood illnesses rarely add up to chronic absenteeism unless some other factor is at play, such as a chronic illness that is poorly managed due to lack of access to affordable health care (Balfanz & Byrnes, 2012 ). Health, thus, is one of many factors in absenteeism that is profoundly influenced by socioeconomic status, discussed in the next section. Also, every parent and teacher is familiar with the phenomenon of children feigning illnesses to avoid school. Absentee rates spike in years when children attend a new building, including kindergarten (Balfanz & Byrnes, 2012 ; Gottfried, 2015 ), indicating that the stress of a new experience is part of the phenomenon (Balfanz & Byrnes, 2012 ). Another factor is that some caregivers do not understand that attendance is as important in the early grades as in later years (Balfanz & Byrnes, 2012 ; Chang et al., 2015 ).

Attendance researchers, most notably Gottfried ( 2015 ; Gottfried & Kirksey, 2021 ), explain that family routines, such as regular bedtimes and morning routines, support attendance. This observation may help to explain why pre-K participation can improve kindergarten attendance: Pre-K families already have routines in place before kindergarten.

The Role of Race/Ethnicity and Socioeconomic Status

Race/ethnicity and SES have profound associations with academic achievement. Gaps between the test scores of White children on the one hand and Black and Hispanic children on the other have been thoroughly documented (as described in, for example, Bond & Lang, 2018 ; Reardon et al., 2015 ), though some have challenged the practice of presenting White children’s aggregate achievement as a norm against which children of other races are measured (e.g., Howard, 2010 ). Much recent research has explored the intersections of race/ethnicity with SES and other environmental factors (e.g., Bond & Lang, 2018 ; Rothstein & Wozny, 2013 ). Researchers have increasingly taken nuanced approaches focusing on the intersections among race/ethnicity, SES measures including not only income but also parental education and family structure, and the consequences of poverty, including poor health, frequent mobility, and lack of educational resources in the home, all of which have documented effects on academic achievement and school readiness (e.g., Bradbury et al., 2015 ; Henry et al., 2020 ; Reardon, 2011 ; Reardon & Portilla, 2016 ; Rhoades Cooper & Lanza, 2014 ). One widely accepted proposition is that racial/ethnic and SES-related academic gaps begin before kindergarten (e.g., Duncan & Magnuson, 2011 ; Reardon et al., 2015 ). Many studies have found that pre-K programs have stronger effects on the academic readiness of children placed at risk by such factors as SES, racial/ethnic background, or home language other than English (e.g., Burger, 2010 ; Phillips et al., 2017 ; Rhoades Cooper & Lanza, 2014 ; Yoshikawa et al., 2016 ).

Similarly, race/ethnicity and SES are related to kindergarten attendance patterns. Black and Hispanic children tend to have higher kindergarten absenteeism than White and Asian children (Chang et al., 2015 ). Children from homes with low-SES are more likely to be absent from kindergarten than higher-income children (Balfanz & Byrnes, 2012 ; Chang et al., 2015 ; Gershenson et al., 2017 ). Reasons cited in the research include homelessness and housing instability, poor health on the part of children or caregivers, parents’ work hours, unstable family structures, issues with transportation, and others (Chang et al., 2015 ; Gottfried, 2015 ). The comparatively greater prevalence of absenteeism among low-SES children is particularly concerning in light of evidence suggesting that schooling makes more difference in the academic growth of low-SES children than of high-SES children (Aucejo & Romano, 2016 ; Balfanz & Byrnes, 2012 ; Gershenson et al., 2017 ).

The Current Study

Our study adds to the evidence on how publicly funded pre-K education affects kindergarten readiness and attendance. As part of an ongoing longitudinal study, we examined data from Michigan’s Great Start Readiness Program (GSRP), which targets children from disadvantaged backgrounds. Taking advantage of the natural experiment arising from the fact that some GSRP sites do not have the capacity to serve all interested and eligible families, we compared a control group of children placed on waitlists, because the local GSRP sites did not have space, with the GSRP treatment group. Though we understand the interactions among risk factors and potentially multifaceted experiences associated with economic disadvantage, racial discrimination, and social determinants of health, we were constrained by our data source to simple definitions of independent variables related to gender, race/ethnicity, disability, economic status, and ELL status. We compared the scores of the GSRP treatment group and the Waitlisted Control group on the Kindergarten Readiness Assessment (WestEd, 2014 ). Next, we compared the kindergarten attendance of the GSRP and Waitlisted Control groups.

Research Questions

Our examination of short-term outcomes for GSRP participants and nonparticipants was guided by the following research questions.

The pre-K effects on kindergarten readiness:

To what extent does GSRP participation improve children’s kindergarten readiness, as measured by the assessment chosen by the Michigan Department of Education, in comparison to nonparticipants?

To what extent does GSRP participation have differential effects on kindergarten readiness for children of different subgroups, categorized by gender, race/ethnicity, disability status, ELL status, and economic disadvantage?

The pre-K effects on kindergarten attendance:

To what extent does GSRP participation improve children’s kindergarten attendance, in comparison to nonparticipants?

To what extent does GSRP participation have differential effects on kindergarten attendance for children of different subgroups, categorized by gender, race/ethnicity, disability status, ELL status, and economic disadvantage?

This study employed a quasi-experimental model to compare the kindergarten readiness and subsequent attendance of eligible children who attended GSRP with the readiness and attendance of eligible children who applied but were placed on a waitlist.

Program Context

GSRP is Michigan’s state-funded pre-K program for four-year-old children, currently administered by the Michigan Department of Lifelong Education, Advancement, and Potential (MiLEAP; formerly administered by Michigan Department of Education in 1985–2023) to improve developmental outcomes for children who are at risk of educational failure.

Eligibility for GSRP Enrollment

Children are eligible for GSRP if their families have incomes below 250% of the federal poverty level (FPL) or if they have any of several risk factors such as experiencing developmental delay or disability, living in a single-parent family, or suffering from abuse or neglect (Wu et al., 2020 ). Family income is the most important factor: The lowest-income families get priority for enrollment. If two families have the same percentage of FPL, the one with more eligibility factors is admitted first.

Crucially for this study, enrollment is based not only on children’s eligibility but also on the availability of open slots at GSRP sites. Children whose parents apply for enrollment but whose sites are full are placed on a waitlist. This fact enabled us to choose a design that compensates for the ethical and logistical difficulties of random assignment: We assigned children who enrolled to the treatment condition and children placed on the waitlist to the control condition. The two groups thus are generally comparable in terms of the families’ interest and commitment to enroll their children in pre-K and their proximity to a GSRP site. However, the GSRP participants in the treatment group are at higher risk than the waitlisted children in the control group. Children with lower incomes and more risk factors were admitted first, so waitlisted children at a given site had higher incomes and fewer risk factors than those who were admitted. We used statistical methods to control for and examine the effects of differences in income and risk factors.

GSRP Curriculum

The GSRP Implementation Manual (MiLEAP, 2023 ) specifies that curricula must be child-focused and play-based; classrooms are to be set up as interest-based play areas with minimal space for large-group or desk-based activities. Adult–child interactions and relationships are acknowledged as key to children’s growth (MiLEAP, 2023 ).

GSRP Family Engagement

GSRP has a strong family engagement component. The GSRP Implementation Manual (MiLEAP, 2023 ) requires children’s lead or associate teachers to conduct two home visits and two conferences per year with each family. Teachers are also required to complete monthly logs in which they track the parent involvement strategies they used with the families of each child. Program guidance stresses ways in which teachers and administrators can encourage family involvement in program activities and mitigate barriers including transportation, scheduling, and language differences. Administrators are required to complete an inventory of involvement strategies used with families based on Epstein’s ( 2002 ) framework for school, family, and community partnership (MiLEAP, 2023 ).

Participants

The sample for this analysis consists of 40,713 kindergarten children who were either enrolled in Michigan’s GSRP or placed on the GSRP waitlist in the 2018–2019 school year. This is the latest year for which data were available in their present form; in 2019, the state education department stopped the requirement of administering a kindergarten readiness assessment. At the time of testing, the children were between 57 and 71 months old (mean = 67; SD = 3.6), or from 4 years, 9 months to 5 years, 11 months. Approximately 48.2% of the children in the sample were female, 8.7% were ELLs, 82.2% were economically disadvantaged, 25.6% were Black, 10.5% were Hispanic or Latino, 55.6% were White, and 8.3% were other races. Table 1 outlines the demographic characteristics of the treatment and control groups.

Kindergarten Readiness Assessment

The level of academic development of the children in the study sample as they transitioned into kindergarten was measured using the Kindergarten Readiness Assessment (KRA; WestEd, 2014 ), selected by the Michigan Department of Education. KRA, which is administered mostly through teacher observation, has been proven to be a reliable and valid assessment of children’s readiness for school (Maryland State Department of Education, 2017 ; WestEd, 2014 ). Kindergarten teachers’ observations of children’s skills have been found to predict academic success in elementary school (Gullo & Impellizeri, 2022 ). KRA yields one overall score and four domain scores. The four domains are Social foundations , Physical development and wellbeing , Language and literacy, and Mathematics . This article reports on the overall score rather than domain scores.

Kindergarten Absence Rate

Children’s levels of kindergarten attendance were determined from attendance records obtained from Michigan’s Center for Educational Performance and Information. In alignment with the federal Every Student Succeeds Act ( ESSA ) of 2015, the Michigan Department of Education defines chronic absenteeism as missing 10% or more of total possible school days in a school year (MI School Data, 2020 ). However, we used a more nuanced definition: We defined the absence rate for each child as the ratio of school days absent to the total number of school days in the school year. Thus, we used the proportion of days absent from school along a continuum rather than separating children into two groups, chronically absent and not chronically absent.

Independent Variables

To control for the effects of demographic characteristics and to examine whether and to what extent these characteristics moderated the relationship between GSRP participation and the outcome variables, we included the following characteristics as independent variables: gender, disability status, ELL status, economic disadvantage, and race/ethnicity. The race/ethnicity categories are White, Black, Hispanic/Latino, and “other.” This last category includes Native Hawaiian or Pacific Islander, American Indian or Alaskan, Asian, and multiracial; we collapsed these categories into “other” to avoid data sparsity. The Black category was treated as the reference group for race/ethnicity. We considered use of school locality, that is, urban, suburban, or rural, as another variable. However, the small size of the Waitlisted Control group for both the readiness and attendance analyses (see Table  1 ) restricted the number of variables whose interactions could be meaningfully analyzed.

KRA is administered to only a fraction of all kindergarten-aged children in Michigan. Of the 40,713 children in our original sample, only 19,495 children took part in KRA for the 2019–2020 school year; hence, our analysis of kindergarten readiness used this subsample. Of the 19,495 children, five (0.02%) were missing entries for their gender. Following Enders ( 2010 ), we excluded cases with missing data from the analysis. Thus, we were left with 19,490 records as the analytic sample for kindergarten readiness.

The analysis for absenteeism considered all 40,713 children. However, 234 children were missing attendance records for the 2019–2020 school year, and another 1,366 were missing values for their gender or race. Thus, a total of 1600 (3.9%) of records were missing data. After excluding these records, we had 39,113 records for the kindergarten absenteeism analysis (Fig.  1 ).

Flow diagram of samples for analysis

To assess the impact of educational interventions, researchers must account for the fact that children are nested in classrooms and schools. We therefore adopted a multilevel modeling approach, with children as the first level of data and schools as the second level. This approach accounts for dependency in outcomes within schools by partitioning and modeling variation at both levels of the data hierarchy (Dunn et al., 2015 ; Raudenbush & Bryk, 2002 ; Snijders & Bosker, 2011 ). Although classrooms might have been an even more salient cluster level, we did not have sufficient data to identify classrooms for many children in our sample; nearly 50% of the 40,713 children had no entries for their classroom IDs. We therefore nested children within schools only. Students who attended the same school presumably had similar experiences that could affect both KRA administration and school attendance. We chose schools rather than GSRP sites because the Waitlist Control group could not be clustered into GSRP sites and because kindergarten sites influence kindergarten behaviors such as attendance. Furthermore, 73% of GSRP children stayed in the school district in which their GSRP site was located, so GSRP sites and school sites had high overlap.

Assessing the Immediate Impact of GSRP on Kindergarten Readiness

Using the multilevel modeling approach, we assessed the immediate effect of GSRP participation by examining its effect on the KRA score, comparing the average performance of the treatment and control groups. We also examined the scores of the four KRA domains. The model is presented in Appendix A. Model estimations were conducted in MLwiN, version 3.05 (Charlton et al., 2020 ).

Assessing the Short-Term Impact of GSRP on Kindergarten Attendance

In the absence of data on children’s academic progress in kindergarten, which is maintained in classrooms and schools but not statewide, kindergarten attendance is a potent indicator of the short-term behavioral impact of GSRP. Whereas the KRA is administered near the beginning of the kindergarten year, children’s attendance records cover the full kindergarten school year, ending about 15 months after students finish their GSRP year.

For the attendance records, children’s schools serve as well-defined clusters with meaningful environmental impacts on children’s attendance. However, some children in our sample changed schools during their kindergarten year. Of the 40,479 children in the sample with attendance records, 38,414 attended only one school, 1,940 attended two, 110 attended three, 13 attended four, and two attended five schools during their kindergarten year. Thus, children are not perfectly nested within schools. To properly account for the impact of every school a child attended, we employed the multiple membership multilevel model (MMMM; Leckie & Owen, 2013 ), with a binomial link function, to examine the effect of GSRP participation on proportion of days absent from school. The model is outlined in Appendix A. MMMMs use membership weights to define the effect of the cluster variable (in this case, school) on the outcome as a function of all clusters to which a subject belongs (Beretvas, 2011 ). For our study, we weighted the influence of each school by the proportion of total attendance in that school, regardless of enrollment status. For example, say a child was enrolled in school A for 90 days and school B for 90 days. She attended school A for 20 days and school B for 80 days, for a total of 100 days. In that case, we weighted school A at 0.2 and school B at 0.8. The school the child attended the most is assumed to have the most influence on the child’s attendance pattern.

All models used the following child demographic variables as covariates: gender, race/ethnicity, disability status, ELL status, and economic disadvantage. We used these covariates as moderators to determine whether the effects of GSRP participation on kindergarten readiness or attendance differed with respect to any of these variables. Due to sample size limitations, particularly with the Waitlisted Control group, we kept all slope parameters fixed at the school level and restricted the examination of interaction effects to two-way interactions only. Table 1 provides a descriptive summary for the kindergarten readiness and attendance subsamples in this study and compares GSRP and Waitlisted Control group children.

Sample Balance and Standard Deviations

KRA overall test scores ranged from 202 to 298 for both groups. The score showed a significant difference between the Waitlisted Control and GSRP groups. The Waitlisted Control group had slightly higher standard deviation (13.34) than the GSRP group (12.57), suggesting more variation in scores for the control group than for the treatment group. The demographic variables reveal significant imbalances between the two groups in the distribution of disability status, economically disadvantaged status, and race. Because state policy prioritizes children from low-income families, the Waitlisted Control group has higher proportions than the GSRP group of children who are White, who are not economically disadvantaged, and who have no disability.

The sample for the absenteeism analysis shows similar values for the average proportion of days absent in both groups, and the standard deviations were also equal across groups. However, as in the sample for kindergarten readiness, the GSRP and Waitlisted Control groups were balanced with respect to gender and ELL status, but significantly unbalanced with respect to disability, economic disadvantage, and race/ethnicity due to the state enrollment priority policy.

The Impact of GSRP Participation on Kindergarten Readiness

To assess the impact of GSRP participation on children’s kindergarten readiness, we examined the results of multilevel linear regression of each child’s overall KRA score based on GSRP participation, using waitlisted children as the comparison group and controlling for child demographic characteristics. This analysis was based on the sample of 19,490 children who participated in the KRA assessment and had complete demographic data. Table 2 shows that, as expected, economic disadvantage and disability are negatively associated with kindergarten readiness. Black children had significantly lower KRA scores than did children from other racial/ethnic groups, and male children had lower scores than female children.

The average kindergarten readiness of children in the Waitlisted Control group was lower than that of children in the GSRP group. However, the differences are significant only in relation to economic disadvantage and ELL status. Holding all other factors constant, economically disadvantaged GSRP participants outperformed their disadvantaged waitlisted peers on the KRA by 6 points, on average. The 95% credible interval ranges are shown in Fig.  2 . Similarly, holding all other factors constant, GSRP participants whose native language was not English outperformed their ELL waitlisted peers by 10 points on average (see Fig.  3 ). None of the other demographic variables had any significant moderating effect on the relationship between GSRP participation and kindergarten readiness.

Predicted average KRA scores and confidence intervals by economically disadvantaged status and GSRP participation

Predicted Average KRA scores and confidence intervals by ELL status and GSRP participation

We also analyzed children’s scores on the four KRA domains, controlling for the demographic variables. The results did not significantly add to our understanding of the effect of GSRP participation; domain scores basically tracked to the overall KRA scores. For all four domains, GSRP participation had a significant effect on the scores of children from economically disadvantaged backgrounds. Other significant effects emerged in the Mathematics domain for race, disability, and ELL status and in the Physical development domain for race and disability. Implications for research or practice are difficult to discern, so we chose not to publish the results of each domain. Contact the lead author for a full description of our methods and findings.

The Impact of GSRP Participation on Kindergarten Attendance

Table 3 shows the results for the effect of GSRP participation on children’s attendance, controlling for demographic variables and the random effect of schools. The outcome variable is the likelihood of being absent from kindergarten. Analysis was conducted with data from 39,113 children who had attendance records for the 2019–2020 school year and had complete demographic data. As in the analysis of kindergarten readiness, we included all demographic variables as covariates and moderators of the effect of GSRP participation on absenteeism.

As Table  3 shows, the odds of being absent from school were generally higher for the Waitlisted Control group than for the GSRP group. However, the gaps between the two groups differ significantly by ELL status, economic disadvantage, and race. ELL Waitlisted Control children had 29% higher odds of being absent than ELL GSRP students, while the difference among native English speakers is not statistically significant (Fig.  4 ). Among economically disadvantaged children, those on the waitlist had 14% higher odds of being absent than their GSRP peers, holding all other factors constant. Again, the differences for less economically disadvantaged children are not significant (Fig.  5 ). While the results in Table  3 show a significant interaction effect for gender and waitlist, the wide confidence interval suggests a low precision for this estimate.

Predicted likelihood and confidence intervals for the probability of kindergarten absence by ELL status and GSRP participation

Predicted likelihood and confidence intervals for the probability of kindergarten absence by economically disadvantaged status and GSRP participation

Among both treatment and control groups, Black students were significantly more likely to be absent than members of other racial/ethnic groups. However, the difference between Waitlisted Control and GSRP groups is much higher among Black children than any other racial/ethnic group, suggesting that Black children benefitted significantly more from GSRP participation than their peers. Figure  6 shows the confidence intervals and the average likelihood of absence by racial/ethnic group.

Predicted likelihood and confidence intervals for the probability of kindergarten absence by race and GSRP participation

This study employed a quasi-experimental model to compare the kindergarten readiness and attendance records of program-eligible children who attended the Great Start Readiness Program (GSRP) with those of students who were eligible for the program but were waitlisted. We employed a multilevel model to examine kindergarten readiness and a multiple membership multilevel model to examine kindergarten attendance.

Effects on Kindergarten Readiness and Attendance

In keeping with decades of research, our study found that GSRP participation had significant effects on participants’ kindergarten readiness. Our finding that GSRP had a positive and significant effect on kindergarten attendance also aligns with previous findings, though the literature in this area is less extensive.

The findings that the effects of GSRP participation on both kindergarten readiness and attendance were significant for economically disadvantaged children suggests that the program is functioning as intended to improve outcomes for the neediest children. Because both treatment and control families in our quasi-experimental design sought to enroll their children in GSRP, the differences in outcomes are not likely to be due to differences in families’ views on the importance of pre-K education or their motivation levels. The fact that waitlisted children had fewer risk factors than the children who were prioritized for admission to oversubscribed GSRP sites makes the improved outcomes for the treatment group even more remarkable. Furthermore, we could not control for the possibility that children in the Waitlisted Control group attended a center-based preschool or licensed child care program other than GSRP or Head Start. This possibility makes it likely that our study underestimates the effect of GSRP on readiness and attendance.

Differential attendance results for ELLs and Black children suggest that GSRP may be improving the prospects of children in these groups for better school performance. This finding adds to the body of evidence suggesting that state-funded pre-K programs improve short-term outcomes for children in groups that otherwise tend to start school at a disadvantage (Duncan & Magnuson, 2013 ; Yoshikawa et al., 2016 ).

The significance of kindergarten attendance lies in its effect on later attendance and then, in turn, on elementary school academic outcomes. Although these effects have been described by previous researchers, the literature on the effects of attendance is smaller than that on academic readiness and subsequent performance.

Limitations

One limitation of this study is the choice of the cluster unit. Although children’s learning experience occurred within classrooms, information about the classrooms, such as teacher demographics and years of experience, was not available. As a result, the cluster unit was set at the school level; differences across classrooms within a school were not accounted for. Also, we clustered children by kindergarten school rather than by GSRP site. The choice was motivated by clear links between the outcomes of interest and the school site, including the fact that children’s kindergarten attendance is likely to be influenced by factors related to the kindergarten site, and by the fact that waitlisted children could not be clustered into GSRP sites.

Another limitation of this study is the exclusion of some pre-K factors that can affect child outcomes. Factors such as pre-K attendance and pre-K program quality have been shown to affect child outcomes (e.g., Allee et al., 2024 ; Yoshikawa et al., 2016 ). We did not have access to data on children’s GSRP attendance and did not include program quality in our analysis. Average daily attendance levels were likely to have been high because consistent attendance was a requirement imposed by the state. However, we could not obtain GSRP attendance data for individual participants. On the Michigan Program Quality Assessment, virtually all GSRP classrooms in 2018–2019 had a rating of at least 3 out of 5; most were rated at 4 or above (Wu et al., 2020 ). This lack of variation would have limited our ability to examine the effects of program quality statistically. Moreover, because we clustered children at the school rather than the program level, none of these program-level variables could be included in the model.

An alternative to assessing program-level variables would have been to include some school-level contextual variables like urban, rural, or suburban locality. However, the relatively small number of waitlisted children in our sample limited the value of this analysis. The cross-tabulation of our model variables with whether school was rural, suburban, or urban revealed several sparse cells. For instance, there was only one waitlisted child in an urban school who met the condition of having a disability and belonging to the “other” race group. We therefore decided to exclude the locality variable to avoid generating unreliable model estimates.

We also had no information about school attendance initiatives during the GSRP participants’ kindergarten year. Though Michigan began reporting attendance and absenteeism under the Every Student Succeeds Act in 2015–2016 (MI School Data, 2020 ), we find no evidence of any statewide attendance initiative. Because of the ESSA reporting, regional grantees and local districts are likely paying more attention to attendance and chronic absenteeism. However, any district attendance initiative would affect treatment and control children equally, so it is not likely to be a mechanism for the effect of GSRP on attendance.

Finally, some children in the Waitlisted Control group may have attended private preschools or participated in other educational alternatives. Our data show only that these children did not attend any publicly funded pre-K program. If waitlisted children participated in some other preschool education program, our models would have underestimated the effect of GSRP.

Implications

Implications for public policy.

One reason for continuing to examine the effects of publicly funded pre-K programs is to justify the enormous public expenditure already invested. Meanwhile, many states and the federal government are considering expanding access to pre-K, whether to serve all four-year-olds in universal programs or to provide enough seats to serve all low-income children in targeted programs. In 2018–2019, the year of this study’s cohort of GSRP children, Michigan invested $244.6 million in GSRP. Annual spending has increased steadily since then. Michigan’s governor has called to expand GSRP to serve children from all backgrounds regardless of income (MiLEAP, 2024 ). Considering this investment, the people and policymakers of Michigan deserve to know whether and to what extent the program is working.

Meanwhile, policymakers and voters both in and outside of Michigan have a stake in the results of our study, which joins a vast body of literature on the effects of pre-K on later child outcomes. The fact that these studies do not add up to a single obvious conclusion is understandable and even expected in light of the multitude of factors that affect child development. Some of these factors pertain to the pre-K environment, such as the populations served, program content and format, and program and teacher quality. Far more factors are outside the purview of the pre-K program because they have to do with families, communities, public health, schools, and the other myriad influences on children’s development.

One reason to continue to examine how participation in a publicly funded pre-K program affects short-term academic readiness, beyond the enormous public investment, is that policies, practices, and programming change over time. So do environmental factors such as families’ perception of pre-K education and the accessibility of other center-based programs. All of these factors can affect child outcomes.

Our study provides one glimpse into what a consortium of education researchers called the “black box” question (Phillips et al., 2017 , p. 2); that is, the mechanisms by which pre-K education affects both short-term and long-term outcomes. Effects on kindergarten attendance could be one of those mechanisms (Gottfried, 2015 ). Children who attend kindergarten consistently are likely to continue to have good attendance in first grade and beyond (Ansari & Pianta, 2019 ; Chang et al., 2015 ; Connolly & Olson, 2012 ). Children with good attendance in elementary school have better academic outcomes in elementary school (Aucejo & Romano, 2016 ; Gottfried, 2009 , 2011 , 2019 ; Morrissey et al., 2014 ) and into high school (Ansari & Pianta, 2019 ; Ansari et al., 2020 ). One policy implication of this research is that government funders should invest in pre-K and kindergarten attendance. Policies can encourage pre-K providers to track not just average daily attendance but also individual students’ absence rates. Funders can then provide resources to help pre-K administrators and teachers intervene when children have high absence rates.

Implications for Practice

One mechanism for the attendance effect may be GSRP’s family engagement component, required of all GSRP grantees. GSRP staff conduct home visits and family conferences, show families how to engage in learning activities at home, and involve caregivers in children’s activities in the classroom (MiLEAP, 2023 ). Researchers have noted that pre-K participation can build family habits that facilitate their children’s pre-K attendance (Chang et al., 2015 ; Gottfried, 2015 ). In light of this research, pre-K programs could, with family input, investigate ways to support families in getting their children to class every day.

Future Research Steps

The next steps for our own research are to track academic, behavioral, and attendance outcomes in third grade for this 2018–2019 cohort of GSRP and waitlisted children. Studies will examine the extent to which kindergarten readiness differences continue as academic and behavioral advantage in grade 3 and whether GSRP impacts on kindergarten attendance yields later positive effects on elementary attendance and academic or behavioral outcomes.

Though we would like to be able to correlate GSRP attendance with school attendance rates, we have access to school attendance but not GSRP attendance data. GSRP implemented an attendance policy in which children with excessive absences, determined by site administrators, lose their slots. Thus, the GSRP participants in the study were assumed to have good pre-K attendance. Researchers who have access to data on individual students’ pre-K attendance can add to the scant, but policy-relevant, literature on the correlation between pre-K and school attendance. If research confirms that pre-K individual attendance predicts school attendance, this finding would have significant implications for policy and practice.

Access to data on pre-K attendance at the individual level would also facilitate studies on how pre-K programs with low rates of chronic absenteeism and high kindergarten attendance achieve these results. Qualitative data from administrators, teachers, and families could further open the “black box” (Phillips et al., 2017 , p. 2) to show what programs can do to help families understand the importance of attendance in pre-K and support them in acting on that understanding. Pre-K programs like GSRP are already working with families to overcome barriers such as health, work schedule, and transportation issues. The opinions of families and front-line staff on possible additional efforts could be particularly useful in policy decisions and program planning.

Thus, future research should continue to investigate which publicly funded pre-K efforts are having effects on kindergarten readiness and pre-K and kindergarten attendance. No doubt, children and families can benefit from practices based on the findings of the current study showing that GSRP enrollment and attendance had significant positive effects on kindergarten attendance, with differential effects by race/ethnicity, SES, and ELL status. The nation’s youngest learners, particularly those most at risk, deserve researchers’ best efforts to discover which policies and practices are most likely to enable them to thrive.

Akaba, S., Peters, L. E., Liang, E., & Graves, S. B. (2020). “That’s the whole idea of college readiness”: A critical examination of universal Pre-K teachers’ understandings around kindergarten readiness. Teaching and Teacher Education, 96 , 1–12. https://doi.org/10.1016/j.tate.2020.103172

Article   Google Scholar  

Allee, K. A., Clark, M. H., Bai, H., & Roberts, S. K. (2024). Direct and indirect impacts of voluntary pre-kindergarten on kindergarten readiness and achievement. Early Childhood Education Journal, 52 , 319–331. https://doi.org/10.1007/s10643-022-01436-w

Amadon, S., Gormley, W. T., Claessens, A., Magnuson, K., Hummel-Price, D., & Romm, K. (2022). Does early childhood education help to improve high school outcomes? Results from Tulsa. Child Development , 93 (4), e379–e395. https://doi.org/10.1111/cdev.13752

Ansari, A., & Gottfried, M. (2021). The grade-level and cumulative outcomes of absenteeism. Child Development, 92 (4), e548–e564. https://doi.org/10.1111/cdev.13555

Ansari, A., & Pianta, R. (2019). School absenteeism in the first decade of education and outcomes in adolescence. Journal of School Psychology, 76 , 48–61. https://doi.org/10.1016/j.jsp.2019.07.010

Ansari, A., Hofkens, T., & Pianta, R. (2020). Absenteeism in the first decade of education forecasts civic engagement and educational and socioeconomic prospects in young adulthood. Journal of Youth and Adolescence, 49 , 1835–1848. https://doi.org/10.1007/s10964-020-01272-4

Aucejo, E., & Romano, T. (2016). Assessing the effect of school days and absences on test score performance. Economics of Education Review, 55 , 70–87. https://doi.org/10.1016/j.econedurev.2016.08.007

Bailey, D. H., Jenkins, J. M., & Alvarez-Vargas, D. (2020). Complementarities between early educational intervention and later educational quality? A systematic review of the sustaining environments yypothesis. Developmental Review, 56 , 100910. https://doi.org/10.1016/j.dr.2020.100910

Balfanz, R., & Byrnes, V. (2012). The importance of being in school: A report on absenteeism in the nation’s public schools. Johns Hopkins University Center for Social Organization of Schools. https://ies.ed.gov/ncee/edlabs/regions/west/relwestFiles/pdf/508_ChronicAbsenteeism_NatlSummary_Balfanz_Byrnes_2012.pdf

Barnett, M. A., Paschall, K. W., Mastergeorge, A. M., Cutshaw, C. A., & Warren, S. M. (2020). Influences of parent engagement in early childhood education centers and the home on kindergarten school readiness. Early Childhood Research Quarterly, 53 , 260–273. https://doi.org/10.1016/j.ecresq.2020.05.005

Bassok, D., Gibbs, C. R., & Latham, S. (2019). Preschool and children’s outcomes in elementary school: Have patterns changed nationwide between 1998 and 2010? Child Development, 90 (6), 1875–1897. https://doi.org/10.1111/cdev.13067

Bassok, D., Latham, S., & Rorem, A. (2016). Is kindergarten the new first grade? AERA Open, 4 , 1–31. https://doi.org/10.1177/23328.58415616358

Beretvas, S. N. (2011). Cross-classified and multiple-membership models. Handbook of advanced multilevel analysis (pp. 321–342). Routledge.

Google Scholar  

Bierman, K. L., McDoniel, M. E., & Loughlin-Presnal, J. E. (2019). How a preschool parent intervention produced later benefits: A longitudinal mediation analysis. Journal of Applied Developmental Psychology . https://doi.org/10.1016/j.appdev.2019.101058

Bierman, K. L., Welsh, J., Heinrichs, B. S., & Nix, R. L. (2018). Effect of preschool home visiting on school readiness and need for services in elementary school: A randomized clinical trial. JAMA Pediatrics, 172 (8), e181029. https://doi.org/10.1001/jamapediatrics.2018.1029

Bivens, J., Garcia, E., Gould, E., Weiss, E., & Wilson, V. (2016). It’s time for an ambitious national investment in America’s children . Economic Policy Institute.

Bond, T. N., & Lang, K. (2018). The Black-White education scaled test-score gap in grades K-7. Journal of Human Resources, 53 (4), 891–917. https://doi.org/10.3368/jhr.53.4.0916.8242R

Bradbury, B., Corak, M., Waldfogel, J., & Washbrook, E. (2015). Too many children left behind: The US achievement gap in comparative perspective . Russell Sage Foundation.

Brown, C. P., Ku, D. H., & Englehardt, J. (2023). Mixed understandings: A case study of how a sample of preschool stakeholders made sense of the changed kindergarten. Early Childhood Education Journal, 51 , 545–557. https://doi.org/10.1007/s10643-022-01315-4

Burger, K. (2010). How does early childhood care and education affect cognitive development? An international review of the effects of early interventions for children from different social backgrounds. Early Childhood Research Quarterly, 25 , 140–165. https://doi.org/10.1016/j.ecresq.2009.11.001

Chang, H. N., Davis, R., & Smith, R. (2015). Mapping the early attendance gap . Attendance Works & Healthy Schools Campaign. https://www.attendanceworks.org/wp-content/uploads/2017/05/Mapping-the-Early-Attendance-Gap_Final-4.pdf

Charlton, C., Rasbash, J., Browne, W.J., Healy, M., & Cameron, B. (2020). MLwiN Version 3.05 . Centre for Multilevel Modelling, University of Bristol.

Connolly, F., & Olson, L. (2012). Early elementary performance and attendance in Baltimore City Schools’ pre-kindergarten and kindergarten. Baltimore Education Research Consortium. https://files.eric.ed.gov/fulltext/ED535768.pdf

Cook, K. D., & Coley, R. L. (2021). Using multiple methods to describe supports for the transition from Head Start to kindergarten. Early Childhood Education Journal, 49 (3), 425–438. https://doi.org/10.1007/s10643-020-01085-x

Davoudzadeh, P., McTernan, M. L., & Grimm, K. J. (2015). Early school readiness predictors of grade retention from kindergarten through eighth grade: A multilevel discrete-time survival analysis approach. Early Childhood Research Quarterly, 32 , 183–192. https://doi.org/10.1016/j.ecresq.2015.04.005

Duncan, G. J., & Magnuson, K. (2011). The nature and impact of early achievement skills, attention skills, and behavior problems. In G. J. Duncan & R. Murnane (Eds.), Whither opportunity? Rising inequality, schools, and children’s life chances (pp. 47–69). Russell Sage Foundation.

Duncan, G., & Magnuson, K. (2013). Investing in preschool programs. Journal of Economic Perspectives, 27 (2), 109–132. https://doi.org/10.1257/jep.27.2.109

Dunn, E. C., Masyn, K. E., Johnston, W. R., & Subramanian, S. V. (2015). Modeling contextual effects using individual-level data and without aggregation: An illustration of multilevel factor analysis (MLFA) with collective efficacy. Population Health Metrics, 13 (1), 1–11.

Durkin, K., Lipsey, M., Farran, D., & Wiesen, S. (2022). Effects of a statewide pre-kindergarten program on children’s achievement and behavior through sixth grade. Developmental Psychology, 58 (3), 470–484. https://doi.org/10.1037/dev0001301

Ehrlich, S., Gwynne, J., & Allensworth, E. (2018). Pre-kindergarten attendance matters: Early chronic absence patterns and relationships to learning outcomes. Early Childhood Research Quarterly, 44 , 136–151. https://doi.org/10.1016/j.ecresq.2018.02.012

Enders, C. K. (2010). Applied missing data analysis . Guilford Press.

Epstein, J. (2002). School, family, and community partnerships: Preparing educators and improving schools . Westview Press.

Every Student Succeeds Act, 20 U.S.C. § 6301 (2015).

Fischer, A., Keily, T., & Weyer, M. (2020, May). Exploring new research on pre-K outcomes. Education Commission of the States. https://www.ecs.org/wp-content/uploads/Exploring_New_Research_on_Pre-K_Outcomes.pdf

Fitzpatrick, C., Boers, E., & Pagani, L. S. (2020). Kindergarten readiness, later health, and social costs. Pediatrics, 146 (6), e20200978. https://doi.org/10.1542/peds.2020-0978

Fyffe, L., Sample, P. L., Lewis, A., Rattenborg, K., & Bundy, A. C. (2014). Entering kindergarten after years of play: A cross-case analysis of school readiness following play-based education. Early Childhood Education Journal, 52 , 167–179. https://doi.org/10.1007/s10643-022-01428-w

Gershenson, S., Jacknowitz, A., & Brannegan, A. (2017). Are student absences worth the worry in U.S. primary schools? Education Finance and Policy, 12 (2), 137–165. https://doi.org/10.1162/EDFP_a_00207

Gottfried, M. (2009). Excused versus unexcused: How student absences in elementary school affect academic achievement. Educational Evaluation and Policy Analysis, 31 , 392–415. https://doi.org/10.3102/0162373709342467

Gottfried, M. A. (2011). The detrimental effects of missing school: Evidence from urban siblings. American Journal of Education, 117 , 147–182. https://doi.org/10.1086/657886

Gottfried, M. A. (2014). Chronic absenteeism and its effects on students’ academic and socioemotional outcomes. Journal of Education for Students Placed at Risk, 19 , 53–75. https://doi.org/10.1080/10824669.2014.962696

Gottfried, M. A. (2015). Can center-based childcare reduce the odds of early chronic absenteeism? Early Childhood Research Quarterly, 32 , 160–173. https://doi.org/10.1016/j.ecresq.2015.04.002

Gottfried, M. A. (2019). Chronic absenteeism in the classroom context: Effects on achievement. Urban Education . https://doi.org/10.1177/0042085915618709

Gottfried, M. A., & Kirksey, J. J. (2021). Going to sleep and going to school: Linking bedtime to student absenteeism. Journal of Sleep Research, 30 , e13396. https://doi.org/10.1111/jsr.13396

Gullo, D. F., & Impellizeri, W. E. (2022). Kindergarten teachers’ ratings of children’s behavioral and learning competencies: Predictive impact on children’s third and fourth grade achievement trajectories. Early Childhood Education Journal, 50 , 301–314. https://doi.org/10.1007/s10643-021-01157-6

Heckman, J., Pinto, R., & Savelyev, P. (2013). Understand the mechanisms through which an influential early childhood program boosted adult outcomes. American Economic Review, 103 (6), 2052–2086. https://doi.org/10.1257/aer.103.6.2052

Henry, D. A., Betancur Cortés, L., & Votruba-Drzal, E. (2020). Black-White achievement gaps differ by family socioeconomic status from early childhood through early adolescence. Journal of Educational Psychology, 112 (8), 1471–1489. https://doi.org/10.1037/edu0000439

Howard, T. C. (2010). Why race and culture matter in schools: Closing the achievement gap in America’s classrooms . Teachers College Press.

Hustedt, J. T., Buell, M. J., Hallam, R. A., & Pinder, W. M. (2018). While kindergarten has changed, some beliefs stay the same: Kindergarten teachers’ beliefs about readiness. Journal of Research in Childhood Education, 32 (1), 52–66. https://doi.org/10.1080/02568543.2017.1393031

Jordan, P. W., & Miller, R. (2017, September). Who’s in: Chronic absenteeism under the Every Student Succeeds Act. FutureEd, Georgetown University. https://www.future-ed.org/wp-content/uploads/2017/09/REPORT_Chronic_Absenteeism_final_v5.pdf

Leckie, G., & Owen, D. (2013). Multiple membership multilevel models – MLwiN practical. LEMMA VLE Module 13, 1–48. Centre for Multilevel Modelling. http://www.bristol.ac.uk/cmm/learning/course.html

Lipsey, M. W., Farran, D. C., & Durkin, K. (2018). Effects of the Tennesse prekindergarten program on children’s achievement and behavior through third grade. Early Childhood Research Quarterly, 45 , 155–176. https://doi.org/10.1016/j.ecresq.2018.03.005

Loughlin-Presnal, J. E., & Bierman, K. L. (2017). Promoting parent academic expectations predicts improved school outcomes for low-income children entering kindergarten. Journal of School Psychology, 62 , 67–80. https://doi.org/10.1016/j.jsp.2017.03.007

Maryland State Department of Education. (2017). The 2016–2017 Kindergarten Readiness Assessment technical report. https://earlychildhood.marylandpublicschools.org/system/files/filedepot/4/kra_technical_report_2016-17.pdf

McCoy, D., Yoshikawa, H., Ziol-Guest, K., Duncan, G., Schindler, H., Magnuson, K., & Shonkoff, J. (2017). Impacts of early childhood education on medium- and long-term educational outcomes. Educational Researcher, 46 (8), 474–487. https://doi.org/10.3102/0013189X17737739

MiLEAP. (2023). GSRP implementation manual. https://www.michigan.gov/mileap/early-childhood-education/early-learners-and-care/gsrp/resources/implementation-manual

MiLEAP. (2024). PreK for all. https://www.michigan.gov/mileap/early-childhood-education/prek-for-all

MiLEAP & Michigan Department of Education. (2020). MiFamily: Michigan’s family engagement framework. https://www.michigan.gov/mileap/-/media/Project/Websites/mileap/Documents/Early-Childhood-Education/Family-Engagement/mifamily_family_engagement_framework.pdf

MI School Data. (2020). Student attendance. https://www.mischooldata.org/student-attendance

Morrissey, T., Hutchison, L., & Winsler, A. (2014). Family income, school attendance, and academic achievement in elementary school. Developmental Psychology, 50 (3), 741–753. https://doi.org/10.1037/a0033848

Murano, D., Sawyer, J. E., & Lipnevich, A. A. (2020). A meta-analytic review of preschool social and emotional learning interventions. Review of Educational Research, 90 (2), 227–263. https://doi.org/10.3102/0034654320914743

Nix, R. L., Bierman, K. L., Motamedi, M., Heinrichs, B. S., & Gill, S. (2018). Parent engagement in a Head Start home visiting program predicts sustained growth in children’s school readiness. Early Childhood Research Quarterly, 45 , 106–114. https://doi.org/10.1016/j.ecresq.2018.06.006

No Child Left Behind Act of 2001, 20 U.S.C. § 6319 (2008).

Pears, K. C., & Peterson, E. (2018). Recognizing and addressing the effects of early adversity on children’s transitions to kindergarten. In A. J. Mashburn, J. LoCasale-Crouch, & K. C. Pears (Eds.), Kindergarten transition and readiness (pp. 163–183). Springer.

Chapter   Google Scholar  

Phillips, D., Lipsey, M., Dodge, K., Haskins, R., Bassok, D., Burchinal, M., . . . Weiland, C. (2017). Puzzling it out: The current state of scientific knowledge on pre-kindergarten effects. Brookings. https://www.brookings.edu/research/puzzling-it-out-the-current-state-of-scientific-knowledge-on-pre-kindergarten-effects/

Puccioni, J. (2018). Parental beliefs about school readiness, home and school-based involvement, and children’s academic achievement. Journal of Research in Childhood Education, 32 (4), 435–454. https://doi.org/10.1080/02568543.2018.1494065

Puma, M., Bell, S., Cook, R., & Heid, C. (2010). Head Start impact study final report. U.S. Department of Health & Human Services, Administration for Children and Families. https://files.eric.ed.gov/fulltext/ED507845.pdf

Puma, M., Bell, S., Cook, R., Heid, C., Broene, P., Jenkins, F., . . . Downer, J. (2012). Third grade follow-up to the Head Start impact study. U.S. Department of Health and Human Services, Administration for Children and Families. https://www.acf.hhs.gov/sites/default/files/documents/opre/head_start_report_0.pdf

Quirk, M., Dowdy, E., Goldstein, A., & Carnazzo, K. (2017). School readiness as a longitudinal predictor of social-emotional and reading performance across the elementary grades. Assessment for Effective Intervention, 42 (4), 248–253. https://doi.org/10.1177/1534508417719680

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models applications and data analysis methods . Sage.

Ready, D. D. (2010). Socioeconomic disadvantage, school attendance, and early cognitive development: The differential effects of school exposure. Sociology of Education, 83 (4), 271–286. https://doi.org/10.1177/0038040710383520

Reardon, S. F. (2011). The widening socioeconomic status achievement gap: New evidence and possible explanations. In R. J. Murnane & G. J. Duncan (Eds.), Whither opportunity? Rising inequality and the uncertain life chances of low-income children. Russell Sage Foundation.

Reardon, S. F., Robinson-Cimpian, J. P., & Weathers, E. S. (2015). Patterns and trends in racial/ethnic and socioeconomic academic achievement baps”. In H. F. Ladd & M. E. Goertz (Eds.), Handbook of Research in Education Finance and Policy (pp. 499–518). Routledge.

Reardon, S. F., & Portilla, X. A. (2016). Recent trends in income, racial, and ethnic school readiness gaps at kindergarten entry. AERA Open, 2 (3), 1–18. https://doi.org/10.1177/2332858416657343

Reynolds, A., & Ou, S. (2016). Generative mechanisms in early childhood interventions: A confirmatory research framework for prevention. Prevention Science, 17 , 794–805. https://doi.org/10.1007/s11121-015-0611-6

Rhoades Cooper, B., & Lanza, S. T. (2014). Who benefits most from Head Start? Using latent class moderation to examine differential treatment effects. Child Development, 85 (6), 2317–2338. https://doi.org/10.1111/cdev.12278

Rothstein, J., & Nathan Wozny, N. (2013). Permanent income and the Black-White test score gap. Journal of Human Resources, 48 (3), 510–544. https://doi.org/10.3368/jhr.48.3.510

Sheridan, S. M., Koziol, N., Witte, A. L., Iruka, I., & Knoche, L. L. (2020). Longitudinal and geographic trends in family engagement during the pre-kindergarten to kindergarten transition. Early Childhood Education Journal, 48 (3), 365–377. https://doi.org/10.1007/s10643-019-01008-5

Snijders, T. A., & Bosker, R. J. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling . Sage.

Sylva, K., Melhuish, E., Sammons, P., Siraj-Blatchford, I., & Taggart, B. (2011). Pre-school quality and educational outcomes at age 11: Low quality has little benefit. Journal of Early Childhood Research, 9 (2), 109–124. https://doi.org/10.1177/1476718X10387900

Vergunst, F., Tremblay, R. E., Nagin, D., Algan, Y., Beasley, E., Park, J., Galera, C., Vitaro, F., & Côté, S. M. (2019). Association between childhood behaviors and adult employment earnings in Canada. JAMA Psychiatry, 76 (10), 1044–1051. https://doi.org/10.1001/jamapsychiatry.2019.1326

Watts, T. W., Jenkins, J. M., Dodge, K. A., Carr, R. C., Sauval, M., Bai, Y., Escueta, M., Duer, J., Ladd, H., Muschkin, C., Peisner-Feinberg, E., & Ananat, E. (2023). Understanding heterogeneity in the impact of public preschool programs. Monographs of the Society for Research in Child Development, 88 (1), 7–182. https://doi.org/10.1111/mono.12463

Weiland, C., Unterman, R., Shapiro, A., Staszak, S., Rochester, S., & Martin, E. (2020). The effects of enrolling in oversubscribed pre-kindergarten programs through third grade. Child Development, 91 (5), 1401–1422. https://doi.org/10.1111/cdev.13308

Welsh, J. A., Bierman, K. L., & Mathis, E. T. (2014). Parenting programs that promote school readiness. In M. Boivin & K. Bierman (Eds.), Promoting school readiness and early learning: The implications of developmental research for practice (pp. 253–278). Guilford Press.

WestEd. (2014). Ready for kindergarten: Kindergarten Readiness Assessment technical report. https://ed.sc.gov/tests/tests-files/pre-k-and-kindergarten-readiness-assessments/kra-technical-report-2014/

Wu, J. H., Weber, E., Van Egeren, L. A., & Akaeze, H. O. (2020). Great Start Readiness Program state evaluation 2018–2019 annual report. Michigan State University Community Engagement Programs. https://cep.msu.edu/upload/Great%20Start%20Readiness%20Program%20State%20Evaluation%202018-19%20Annual%20Report.pdf

Yoshikawa, H., Weiland, C., & Brooks-Gunn, J. (2016). When does preschool matter? The Future of Children, 26 (2), 21–35. https://doi.org/10.1353/foc.2016.0010

Download references

Funding was supported by Michigan Department of Education.

Author information

Authors and affiliations.

Department of Human Development and Family Studies, Michigan State University, 219 S. Harrison Rd., Room 32, East Lansing, MI, 48824, USA

Jamie Heng-Chieh Wu

Community Evaluation Programs at University Outreach and Engagement, Michigan State University, 219 S. Harrison Rd., Room 31, East Lansing, MI, 48824, USA

Jamie Heng-Chieh Wu & Hope Onyinye Akaeze

Office for Public Engagement, University of Minnesota, 231 Pillsbury Dr. SE, Minneapolis, MN, 55455, USA

Laurie A. Van Egeren

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jamie Heng-Chieh Wu .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Model Estimation Formulae

Multilevel Model for KRA Scores

Combined model:

where \({Y}_{ij}\) is the overall KRA score for student i, who attended school j , \({\varepsilon }_{ij}\) is the residual for student i in school j and \({u}_{0j}\) is the residual for school j .

Multiple Membership Multilevel Binary Logistic Model for Absenteeism

where \({Y}_{i\{j\}}\) is the probability that student i, who attended set \(\{j\}\) of schools will be absent from school, \({w}_{ih}\) is the weight (in this case, proportion) assigned to student i ’s association with school h of the j schools for that student; and \({u}_{0h}\) is the random effect of school h .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Wu, J.HC., Akaeze, H. & Van Egeren, L.A. Effects of a State Pre-kindergarten Program on the Kindergarten Readiness and Attendance of At-Risk Four-Year-Olds. Early Childhood Educ J (2024). https://doi.org/10.1007/s10643-024-01736-3

Download citation

Accepted : 29 July 2024

Published : 04 September 2024

DOI : https://doi.org/10.1007/s10643-024-01736-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Kindergarten readiness
• School attendance
• Great Start Readiness Program
• Pre-kindergarten
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 02 September 2024

Using mixed methods and partnership to develop a program evaluation toolkit for organizations that provide physical activity programs for persons with disabilities

• Sarah V. C. Lawrason 1 , 2 ,
• Pinder DaSilva 3 ,
• Emilie Michalovic 3 ,
• Amy Latimer-Cheung 4 , 5 ,
• Jennifer R. Tomasone 4 , 5 ,
• Shane Sweet 6 , 7 ,
• Tanya Forneris 1 ,
• Jennifer Leo 8 ,
• Matthew Greenwood 9 ,
• Janine Giles 10 ,
• Jane Arkell 11 ,
• Jackie Patatas 12 ,
• Nick Boyle 10 ,
• Nathan Adams 1 , 2 &
• Kathleen A. Martin Ginis 1 , 2 , 13  

Research Involvement and Engagement volume  10 , Article number:  91 ( 2024 ) Cite this article

Metrics details

The purpose of this paper is to report on the process for developing an online RE-AIM evaluation toolkit in partnership with organizations that provide physical activity programming for persons with disabilities.

A community-university partnership was established and guided by an integrated knowledge translation approach. The four-step development process included: (1) identify, review, and select knowledge (literature review and two rounds of Delphi consensus-building), (2) adapt knowledge to local context (rating feasibility of outcomes and integration into online platform), (3) assess barriers and facilitators (think-aloud interviews), and (4) select, tailor, implement (collaborative dissemination plan).

Step 1: Fifteen RE-AIM papers relevant to community programming were identified during the literature review. Two rounds of Delphi refined indicators for the toolkit related to reach, effectiveness, adoption, implementation, and maintenance. Step 2: At least one measure was linked with each indicator. Ten research and community partners participated in assessing the feasibility of measures, resulting in a total of 85 measures. Step 3: Interviews resulted in several recommendations for the online platform and toolkit. Step 4: Project partners developed a dissemination plan, including an information package, webinars, and publications.

This project demonstrates that community and university partners can collaborate to develop a useful, evidence-informed evaluation resource for both audiences. We identified several strategies for partnership when creating a toolkit, including using a set of expectations, engaging research users from the outset, using consensus methods, recruiting users through networks, and mentorship of trainees. The toolkit can be found at et.cdpp.ca. Next steps include disseminating (e.g., through webinars, conferences) and evaluating the toolkit to improve its use for diverse contexts (e.g., universal PA programming).

Plain English summary

Organizations that provide sport and exercise programming for people with disabilities need to evaluate their programs to understand what works, secure funding, and make improvements. However, these programs can be difficult to evaluate due to lack of evidence-informed tools, low capacity, and few resources (e.g., money, time). For this project, we aimed to close the evaluation gap by creating an online, evidence-informed toolkit that helps organizations evaluate physical activity programs for individuals with disabilities. The toolkit development process was guided by a community-university partnership and used a systematic four-step approach. Step one included reviewing the literature and building consensus among partners and potential users about indicators related to the success of community-based programs. Step two involved linking indicators with at least one measure for assessment. Step three involved interviews with partners who provided several recommendations for the online toolkit. Step four included the co-creation of a collaborative plan to distribute the toolkit for academic and non-academic audiences. Our comprehensive toolkit includes indicators for the reach, effectiveness, adoption, implementation, and maintenance of physical activity programs for individuals with disabilities. This paper provides a template for making toolkits in partnership with research users, offers strategies for community-university partnerships, and resulted in the co-creation of an evidence-informed evaluation resource to physical activity organizations. Users can find the toolkit at et.cdpp.ca.

Peer Review reports

Disability and physical activity

The United Nations Convention on the Rights of Persons with a Disability protects the rights of people living with disabilities to access full and effective participation in all aspects of life, including sports and other recreational forms of physical activity (PA) such as exercise and active play. But because of countless environmental, attitudinal and policy barriers [ 1 ], children, youth and adults with disabilities are the most physically inactive segment of society [ 2 , 3 ]. Physical inactivity increases the risk that people with disabilities will experience physical and mental health conditions, social isolation, and stigma [ 4 ]. Systematic reviews have evaluated the effects of participation in PA programs among children, youth, and adults with physical, intellectual, mental, or sensory disabilities. Many, but not all, of these reviews have reported significant improvements in physical health, mental health, and social inclusion [ 2 ]. One reason for the inconsistent outcomes is that the PA participation experiences of people with disabilities are not universally positive [ 5 ].

Qualitative and quantitative research shows that people with disabilities often report negative PA experiences; for instance, being marginalized, excluded, and receiving sub-standard equipment, access, instruction, and opportunities to fully participate in PA [ 6 , 7 , 8 ]. Research and theorizing on quality PA participation and disability indicate that these low-quality PA experiences deter ongoing participation and undermine the potential physical and psychosocial benefits of PA for children and adults [ 5 , 9 ]. These findings attest to the need for evaluation of existing PA programs to identify what is working, and where improvements are needed to achieve optimal participation and impact.

Evaluating community-based programs

Persons with disabilities increasingly participate in disability sport to be physically active, and disability sport is often delivered by community organizations [ 2 ]. Like many community-based and non-profit organizations, organizations that provide PA programming for persons with disabilities (herein referred to as ‘this sector’) are often expected to conduct evaluations. These evaluations are done to secure and maintain external funding, demonstrate impact to board members and collaborators, and understand capacity for growth [ 10 ]. Even though program evaluations are often required, real-world programs are difficult to evaluate [ 11 ] and organizations often lack capacity and resources to conduct evaluations effectively [ 12 ]. Programs may be difficult to evaluate due to program complexity (e.g., setting, target population, intended outcomes; [ 11 ], and evaluation priorities (e.g., differing partner needs and resources; [ 13 ]. Organizations may lack capacity in understanding and using appropriate evaluation methods and tools [ 14 ], determining what counts as evidence and its application [ 15 ], and the roles of researchers and practitioners in supporting real-world program evaluations [ 16 ].

Evaluation frameworks can be used to facilitate a guided, systematic approach to evaluation. A framework involves an overview or structure with descriptive categories, meaning they focus on describing phenomena and how they fit into a set of categories rather than providing explanations of how something is working or not working [ 17 ]. One evaluation framework that is commonly applied in PA and disability settings is the RE-AIM framework [ 18 ]. RE-AIM is comprised of five evaluation dimensions or categories: (a) Reach: the number, proportion, and representativeness of individuals who engage in a program, (b) Effectiveness: the positive and negative outcomes derived from a program, (c) Adoption, the number, proportion, and representativeness of possible settings and staff participating in the program, (d) Implementation: the cost and extent to which the program was intended to be delivered, and (e) Maintenance: the assessment beyond six months at the individual and organizational levels. The RE-AIM framework is appropriate in this sector because it aligns with organizations’ need to understand factors that influence PA participation at both individual and organizational levels and for process (formative) and outcome (summative) evaluations [ 19 , 20 , 21 , 22 , 23 ]. Additionally, the RE-AIM framework has demonstrated feasibility to evaluate programs in this sector [ 19 , 21 , 22 ]. The RE-AIM framework was developed to address the failures and delays of getting scientific research evidence into practice and policy [ 18 ].

Gaps between evaluation research and practice

There has been a growing body of evidence to suggest that one of the most effective ways to bridge the gap between research and practice is through integrated knowledge translation (IKT; [ 24 ]). IKT means that the right research users are meaningfully engaged at the right time throughout the research process [ 25 ]. IKT involves a paradigmatic shift from recognizing researchers as ‘experts’ to valuing the expertise of individuals with lived experience, programmers, and policymakers through their inclusion in the development of the research questions, methods, execution, and dissemination to ensure that the research is relevant, useful, and usable [ 25 ]. A commitment to IKT aligns with the “nothing about us without us” philosophy of the disability rights movement [ 26 ] and is therefore ideal for a toolkit development process for this sector.

To address the gaps of lack of evidence-informed resources and reduced organizational capacity to conduct program evaluations [ 12 ], our community partners (leaders from seven Canadian organizations in this sector) identified that a toolkit is needed. An evaluation toolkit is a collection of tools that includes materials that may be used individually or collectively, such as educational material, timelines, and assessment tools, and the tools may often be customized based on context, thus helping to bridge the translation gap between evidence and practice [ 27 ]. Toolkit development can be a multi-step process including literature reviews, interviewing partners, and using a Delphi approach [ 27 ]. Previous research with community-based disability PA organizations suggests that digital platforms can be an efficient way for participants and staff to provide evaluation access to evaluation tools [ 19 , 23 ]. Together, this research culminated in our decision to (1) use RE-AIM for the toolkit’s framework, meaning the toolkit was organized using the five evaluation dimensions, and (2) to deliver the toolkit through interactive technology. The purpose of this paper is to report on a systematic, IKT-focused process for the design, development, and formulation of implementation considerations for an online RE-AIM evaluation toolkit for organizations that provide PA programming for persons with disabilities.

Research approach

A community-university partnership was established between seven Canadian disability PA organizations and three universities. A technology partner guided the back-end development of the online toolkit. Using an IKT approach [ 25 ], community partners were engaged before the research grant was written and submitted to ensure that the project was meaningful and focused on the appropriate tasks and outcomes. To guide our partnership, we agreed to adopt the IKT guiding principles for SCI research [ 25 ] which aim to provide a foundation for meaningful engagement between partners. An example of a guiding principle is partners share in decision-making [ 25 ]. The principles were presented at each bi-monthly team meeting and participants had the opportunity to share concerns if certain principles were not upheld. Partners had regular opportunities for sharing in decision making, provided financial contributions to accelerate the project, and benefitted from developing the toolkit to tailor indicators and measures relevant for disability PA organizations. Two community partner leaders also provided mentorship to academic trainees on community engagement in research, employment in non-academia, and project management, emphasizing the multi-directional nature of the partnership. To see the entire IKT process, see Appendix A in the supplemental file.

To maximize the likelihood that our toolkit is used in practice, our development process was guided by the Knowledge-to-Action (KTA) framework (see Fig.  1 ; [ 28 ]). The KTA framework was developed to help researchers with knowledge translation by identifying the steps in moving knowledge into action [ 28 ]. The KTA framework has two components: (a) knowledge creation and (b) action cycle. Our toolkit development process followed the steps of the action cycle, whereby existing knowledge is synthesized, applied, and mobilized. The problem to be addressed is a need for a program evaluation toolkit. To solve the problem, as shown with the yellow boxes in Fig.  1 , the steps for developing the RE-AIM evaluation toolkit included: (1) identify, review, and select knowledge; (2) adapt the knowledge to the local context and users; (3) assess the barriers and facilitators to knowledge use; and (4) select, tailor, and implement the toolkit.

Knowledge to action framework (adapted from [ 28 ])

To guide toolkit development, we ensured the methods aligned with recommendations from the COnsensus-based Standards for the selection of health Measurement Instruments/ Core Outcome Measures in Effectiveness Trials (COSMIN/COMET) groups for generating a set of core outcomes to be included in health intervention studies [ 29 ]. These guidelines state that developing a core outcome set requires finding existing outcome measurement instruments (see Step 1), quality assessment of instruments (see Step 2), and a consensus procedure to agree on the core outcome set (see Step 2) [ 29 ].

Step 1: Identify, review, and select knowledge

Literature review.

The first step in identifying, reviewing, and selecting knowledge was to conduct a literature review. The literature review examined research using the RE-AIM framework to evaluate community-based and health-related programs. This was completed through a search of www.re-aim.org (which lists all RE-AIM evaluations) to identify indicators for each RE-AIM dimension within community-based and health-related contexts. Studies were included if they: used the RE-AIM framework to evaluate a community-based health program or involved persons with disabilities, were published in English, and were peer reviewed. All study designs were included. The review also examined qualitative and quantitative studies of outcomes of community-based PA programs for people with disabilities (e.g., [ 9 ]) and outcomes our own partners have used in their own program evaluations. These papers and outcomes were discussed and chosen during early partnership meetings to initiate a list of indicators. Examples of community-based programs included peer support programs for individuals with spinal cord injuries in Quebec. Data extracted from papers included indicators (and their definitions) and associated measures used for evaluations.

Delphi process

The second part in identifying, reviewing, and selecting knowledge involves critically appraising the relevant literature identified, to determine its usefulness and validity for addressing the problem [ 28 ]. To determine usefulness and validity, a consensus-building outreach activity was used—an online Delphi method. Briefly, the Delphi method is used to arrive at a group decision by surveying a panel of experts [ 30 , 31 ]. The experts each respond to several rounds of surveys. Survey responses are synthesized and shared with the group after each round. The experts can adjust their responses in the next round based on their interpretations of the “group response.” The final response is considered a true consensus of the group’s opinion [ 30 , 31 ]. Delphi was ideal for our partnership approach because it eliminates power dynamics from the consensus-building process and ensures every expert’s opinion is heard and equally valued. Previous research has demonstrated the utility of Delphi methods to generate consensus among disability organizations regarding the most important outcomes to measure in a peer-support program evaluation tool [ 32 ].

Delphi methodologies are considered a reliable means for achieving consensus when a minimum of six experts are included [ 33 ]. Therefore, we aimed to recruit a minimum of six participants from each target group (i.e., members of disability PA organizations and researchers). Partners were encouraged to invite members who may qualify and be interested in completing the Delphi process. Participants completed a two-round Delphi process and were asked to rate each RE-AIM indicator on a scale of 1 (not at all important) to 10 (one of the most important). An indicator was included if at least 70% of participants agreed it was “very important” (8 or above) [ 31 ]. Indicators that did not meet these criteria were removed from the list.

Retained indicators were then paired with at least one possible measure of that indicator (e.g., the ‘Positive Youth Development’ indicator was paired with the Out-of-School Time Observation instrument [ 34 ]). The partnership’s goal was to develop a toolkit comprised of valid and reliable measures. Therefore, the validity and reliability of each measure were critically appraised by the academic team-members using COSMIN/COMET criteria [ 29 ]. For some ‘Effectiveness’ indicators, published questionnaires were identified from the scientific literature. Measures were retained if they had high quality evidence of good content validity and internal consistency reliability [ 29 ] and were used in PA contexts and/or contexts involving participants with disabilities. The measures of all other indicators (where no published questionnaire measure was identified) were assessed by nine partners and modified to ensure that the measure was accurate and reliable for evaluation use in this sector.

Step 2: Adapt knowledge to local context

In the KTA framework, this phase involves groups making decisions about the value, usefulness, and appropriateness of knowledge for their settings and circumstances and customizing the knowledge to their particular situation [ 28 ]. Using Microsoft Excel, partners were sent a list of the selected indicators and measures in two phases (Phase 1: “RE” indicators and Phase 2: “AIM” indicators). Partners were asked to rate, on a scale of 0 to 2 the following categories for each measure: feasibility-time (not at all feasible to feasible), feasibility-complexity (not at all feasible to feasible), accuracy (not at all accurate to accurate), and unintended consequences (no, maybe, yes). They were also asked to provide additional feedback. This step only involved partners on the project with experience administering questionnaires (in research or evaluation settings) because the process required knowledge of how to administer measures to respondents. The median and mean of each category were calculated with community partner responses given double weighting/value relative to academic partner responses. Double weighting was given to community partner responses as the toolkit is anticipated to be used more frequently in community settings. The feedback was summarized. Results were presented to all partners during an online meeting, and team members discussed feedback to establish agreement on measures. The measures were sent out to partners again to provide any final feedback on included indicators and measures. The selected indicators and measures were compiled in an online program evaluation toolkit compliant with accessibility standards.

Step 3: Assess barriers and facilitators

In the KTA framework, this step involves identifying potential barriers that may limit knowledge uptake and supports or facilitators that can be leveraged to enhance uptake [ 28 ]. In Step 3, partners were invited to participate in an unstructured, think-aloud interview while they used the online program evaluation toolkit [ 35 ]. Interviews were conducted to collect detailed data about how users reacted to different parts of the toolkit content, format, and structure. Each interview was conducted over Zoom with one participant and two interviewers. The two-to-one interview format [ 36 ] supported the ability to take notes during the interview, ask questions from different perspectives, and reflect on common experiences to the two interviewers [ 36 ] with the website. Participants were also asked how the toolkit was used and any barriers to its use, and identified features of the toolkit that may need to be changed. In a separate group meeting, team members were asked for ideas on how to overcome potential barriers to using the toolkit and tips for its implementation. Data were analyzed using a content analysis approach [ 37 ] and recommendations were prioritized by the lead and senior authors using the MoSCoW method [ 38 ]. The MoSCoW method is a prioritization technique that has authors categorize recommendations using the following criteria: (a) “Must Have” (Mo), (b) “Should Have” (S), (c) “Could Have” (Co), and (d) “Won't Have This Time” (W). These recommendations were presented to all partners for further discussion. Based on the feedback, the toolkit content and technology were further iterated as needed. Information from this step was used to write brief user guides for toolkit users.

Step 4: Select, tailor, implement

In the KTA framework, this step involves planning and executing interventions to promote awareness and implementation of knowledge, and tailoring interventions to barriers and audiences [ 28 ]. In Step 4, during an online partnership meeting, a brainstorming activity was completed to discuss target audiences for the toolkit, barriers and facilitators to outreach, and dissemination ideas. Team members formulated a dissemination plan and identified promotional resources they need to tailor the dissemination of the toolkit to their sector networks.

Literature Review

The initial searching process on the re-aim.org database identified 15 papers with relevant indicators for a RE-AIM toolkit. These papers and their citations are in Appendix B in the supplemental file. Additional resources identified by partners included: [ 2 , 9 , 39 , 40 ], and partners’ previous experiences with evaluations to inform potential indicator choices. In total, 62 indicators were identified across all RE-AIM domains.

In round 1, 32 people participated in the exercise (two participants did not provide demographic information). In round 2, 28 people completed the questionnaire (four participants did not provide demographic information). Detailed participant demographics are presented in Table  1 . The adaptation of indicators through the Delphi process can be found in Fig.  2 . Given that nearly all indicators were deemed important from round 2, we agreed that a third round of the Delphi process was not needed. Based on the literature review, measures for each indicator were identified.

Adaptation process for indicators and measures from the Delphi process and partner feedback during COSMIN/COMET rating

Eight partners ( n  = 3 academic, n  = 5 community) completed the rating process for the “RE” domains and 10 partners ( n  = 3 academic, n  = 7 community) completed the rating process for the “AIM” domains (rating feasibility, complexity, accuracy, and unintended consequences; see Table  2 ). Respondent feedback was used to adapt and improve the measures to make them more feasible, less complex, and more accurate to reflect the indicators properly. Respondents also suggested that each measure should also include information boxes about the respondents, administrators, type of data collection, and time to complete data collection. The adaptation of indicators and measures from this process can be found in Fig.  2 . The final list of indicators and measures can be found in Table  3 .

Six partners (community and academic partners) participated in unstructured think-aloud interviews, one of which was conducted jointly with two partners ( M time  = 43.37, SD  ± 13.50 min). Across interviews, 45 unique recommendations were identified for improving the usability of the toolkit. These recommendations were sorted using the MoSCoW method, and prioritized based on budgetary constraints, team skillsets, and competing needs. Of the 45 recommendations, 30 were identified as ‘Must haves’, 6 as ‘Should haves’, 4 as ‘Could haves’, and 5 as ‘Won’t haves’ (see Appendix C in the supplemental file). All 30 ‘Must have’ recommendations were implemented in collaboration with the technology partner, along with 2 ‘Should have’ recommendations.

After all recommendations were executed by the technology partner, a final project meeting was held to discuss project updates, barriers and facilitators to outreach, and ideas for dissemination. Barriers to outreach included lack of research or evaluation knowledge to use the toolkit, lack of funding to conduct evaluations, poor turnover from reaching users (i.e., users becoming aware of the toolkit) to receiving (i.e., users browse the toolkit website) to using the toolkit (i.e., users use the toolkit for an evaluation), and challenges connecting with hard-to-reach organizations. Facilitators to outreach included providing resources for evaluation support, connecting with trainees to support evaluations, having positive self-efficacy and attitude for conducting evaluation, building awareness on the benefits of the toolkit through a dissemination campaign, credibility in the toolkit development process, and reaching out to key funders for administration of toolkit as guidance.

The toolkit can be found at et.cdpp.ca and is intended to be used by community organizations and academic institutions that conduct program evaluations involving PA and disability (and inclusive integrated programming). This interactive toolkit allows users to customize to their program evaluation situation by selecting a) which RE-AIM dimensions they want to evaluate, and b) which indicators they want to measure within a particular RE-AIM dimension (e.g., self-efficacy and quality participation within the Effectiveness dimension). Based on users’ selections, the toolkit program compiles the corresponding measures for each indicator into a customized, downloadable document that the user can then put in the format of their choosing (e.g., online survey, paper questionnaire) for their program evaluation. This design aligns with partner requests for a simple online interface that provides flexibility and tailoring to their program evaluation needs. The toolkit and user guides are made freely available (i.e., open access), to maximize accessibility to community organization and academic audiences.

A plan with dissemination and capacity building activities was created to ensure the supported uptake of the evaluation toolkit. Our priority was to create a knowledge translation and communications package (e.g., newsletter article, social media content) for community partner organizations to disseminate through their channels. This included disseminating information to other community organizations within their network and funding partners (e.g., Sport Canada, Canadian Tire Jumpstart, ParticipACTION, provincial ministries, and the Canadian Paralympic Committee). This package served as the official ‘launch’ of the evaluation toolkit on July 20, 2023. Through this package, other activities were listed as potential ‘services’ interested parties can use. These services include bookable time for ‘office hours’ whereby a one-on-one meeting on how to use the toolkit and conduct program evaluation can be arranged and a 1-h ‘frequently asked questions’ webinar/workshop. Other activities included publishing an open-access manuscript, writing knowledge translation and media blogs about the manuscript, and delivering academic and community conference presentations.

The purpose of this paper was to report on the process of developing an evaluation toolkit in partnership with organizations that provide PA programming for persons with disabilities. Informed by the RE-AIM framework [ 18 ] and the knowledge-to-action framework [ 28 ], the toolkit development process involved a literature review, Delphi process, and interviews to adapt indicators and measures. Recommendations from partners were implemented, and the final toolkit can be found at et.cdpp.ca. Partners collaborated to create a dissemination and capacity building plan to support the uptake of the toolkit across the target audience.

Community organizations struggle to conduct program evaluations and to use existing evaluation frameworks. A recent scoping review identified 71 frameworks used to evaluate PA and dietary change programs [ 41 ]. Despite access to many frameworks, Fynn et al. [ 41 ] found limited guidance and resources for using the frameworks. In response to these concerns, the toolkit acts as a resource for using the RE-AIM framework by facilitating the uptake of evidence-informed evaluation practices. The toolkit will help organizations overcome barriers to evaluation identified by previous research by increasing capacity to use appropriate methods and tools [ 14 ] and providing education on determining what counts as evidence and data [ 15 ]. This can facilitate better organizational direction, improved programming, and importantly, better quality PA experiences for individuals with disabilities. The toolkit also complied with accessibility standards, an important benchmark for our partnership and a necessary step when creating a product for organizations that serve persons with disabilities. Accessibility standards were relatively easy to achieve and should be customary in all IKT activities.

To the best of our ability, the toolkit was developed specifically for organizations that provide programming for people with disabilities by focussing the literature review, having program partners in the disability community participate in the Delphi process, and ensuring the validity and reliability of indicators in disability contexts. However, there is an enormous shortage of data related to PA and disability as most national health surveillance systems exclude or do not measure disability [ 2 ]. While this general limitation may affect the toolkit, it also means that the toolkit may be useful for universal PA organizations that are interested in evaluating programs with non-disabled individuals. Additional research is needed to examine the effectiveness of the toolkit in diverse contexts.

This project provides a template for developing open-access, online evidence-informed toolkits using an IKT approach with community partners. There are few resources on how to develop toolkits for the health and well-being field informed by knowledge translation frameworks or that include perspectives of end-users (e.g., [ 42 , 43 ]). The four-step mixed-methods approach was guided by the systematic use of frameworks to inform toolkit development. Our project utilized a rigorous, step-by-step process for creating toolkits and resources for this sector that centres the knowledge and expertise of research users. To centre the knowledge and expertise of research users, we employed several strategies identified by Hoekstra et al. [ 44 ] for building strong disability research partnerships. Important strategies for partnership when developing a toolkit include (1) using a set of norms, rules, and expectations, (2) engagement of research users in the planning of research, (3) using consensus methods (i.e., Delphi), and (4) recruiting research users via professional or community networks [ 44 ].

First, we used the IKT Guiding Principles [ 25 ] as the set of norms, rules, and expectations to guide our partnership. These principles were addressed throughout the partnership and provided criteria to understand the success of the partnership. Second, we engaged with community partners from the beginning of the research process. Working with community partners who were committed to developing a high-quality product was integral to the success of this project. Community partners were committed and highly engaged as the toolkit stemmed from a community-identified need, rather than solely a ‘research gap’. Third, using consensus methods is an excellent strategy to avoid decision-making that is dominated by certain voices or interests in the partnership [ 45 ]. One way that our project allowed for multiple voices to be heard was through our anonymous Delphi processes, which encouraged partners to share their input in a non-confrontational and data-driven manner. Fourth, in our partnership, many individuals and organizations had longstanding working relationships and aligned priorities for the project. Building our partnership based on previous trusting, respectful relationships was essential and using the IKT guiding principles [ 25 ] ensured that we maintained similar values and priorities throughout the partnership.

We used an additional strategy that has not been previously mentioned in the IKT literature: mentorship of research trainees by community partners. Through monthly meetings, two community partners provided mentorship sessions to three trainees. These sessions focused on how to close the research-to-practice gap and helped to facilitate strong relationships between researchers and research users. Mentorship was an important step for training the next generation of researchers to use IKT.

Limitations

This project has some limitations. First, an exhaustive systematic scoping review was not conducted to identify evaluation indicators. This may have limited the number of relevant evaluation indicators included in the Delphi surveys. However, given that only five indicators were removed, and none were added after two rounds of Delphi, we are confident that our search returned relevant indicators. In the future, it may be worthwhile to consider an in-person or video-conference-facilitated Delphi process to encourage discussion and differentiation of indicators. Second, we identified several barriers and facilitators for using the toolkit, but addressing these barriers meaningfully was beyond the scope of this paper. We are currently in the process of disseminating (e.g., social media campaigns, blogs, discussions with funders) and evaluating the toolkit (e.g., surveys, using data analytics). This data will be reported in a future paper. Third, the interviews revealed 45 unique recommendations for the website and toolkit, but only some of these recommendations could be implemented due to budgetary constraints (e.g., adding a search function and filtering indicators to the website could not be completed).

Conclusions

In summary, this paper reports on the development of an online, open-access program evaluation toolkit for the disability and PA sector. The toolkit is informed by the RE-AIM framework [ 18 ] and available at et.cdpp.ca. Our paper describes a four-step process guided by the KTA framework [ 28 ] and IKT principles [ 25 ] to work with community partners to ensure the toolkit is relevant, useful, and usable. The process included reviewing the literature, building consensus through two rounds of Delphi surveys, rating the feasibility and complexity of measures, assessing barriers and facilitators through think-aloud interviews, and crafting a dissemination and capacity-building plan. This paper provides a template for creating toolkits in partnership with research users, demonstrates strategies to enable successful community-university partnerships, and offers an evidence-informed evaluation resource to organizations that provide PA programming for persons with disabilities.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Integrated knowledge translation

Knowledge-to-action

Physical activity

Reach, effectiveness, adoption, implementation, maintenance

Martin Ginis KA, Ma JK, Latimer-Cheung AE, Rimmer JH. A systematic review of review articles addressing factors related to physical activity participation among children and adults with physical disabilities. Health Psychol Rev. 2016;10(4):478–94. https://doi.org/10.1080/17437199.2016.1198240 .

Article   PubMed   Google Scholar  

Martin Ginis KA, van der Ploeg HP, Foster C, Lai B, McBride CB, Ng K, et al. Participation of people living with disabilities in physical activity: a global perspective. Lancet. 2021;398(10298):443–55.

van den Berg-Emons RJ, Bussmann JB, Stam HJ. Accelerometry-based activity spectrum in persons with chronic physical conditions. Arch Phys Med Rehabil. 2010;91(12):1856–61.

World Health Organization. World report on disability. Geneva; 2011.

Evans MB, Shirazipour CH, Allan V, Zanhour M, Sweet SN, Martin Ginis KA, et al. Integrating insights from the parasport community to understand optimal experiences: the quality parasport participation framework. Psychol Sport Exerc. 2018;37:79–90.

Article   Google Scholar  

Martin Ginis KA, Gee CM, Sinden AR, Tomasone JR, Latimer-Cheung AE. Relationships between sport and exercise participation and subjective well-being among adults with physical disabilities: Is participation quality more important than participation quantity? Psychol Sport Exerc. 2024;70:102535.

Allan V, Smith B, Côté J, Martin Ginis KA, Latimer-Cheung AE. Narratives of participation among individuals with physical disabilities: a life-course analysis of athletes’ experiences and development in parasport. Psychol Sport Exerc. 2018;37:170–8.

Orr K, Tamminen KA, Sweet SN, Tomasone JR, Arbour-Nicitopoulos KP. “I’ve had bad experiences with team sport”: sport participation, peer need-thwarting, and need-supporting behaviors among youth identifying with physical disability. Adapt Phys Activ Q. 2018;35(1):36–56.

Shirazipour CH, Latimer-Cheung AE. Understanding quality participation: exploring ideal physical activity outcomes for military veterans with a physical disability. Qual Res Sport Exerc Health. 2020;12(4):563–78. https://doi.org/10.1080/2159676X.2019.1645037 .

Patton M. Qualitative research and evaluation methods. London: Sage Publications; 2015.

Google Scholar  

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015 Mar 19 [cited 2022 Apr 18];350. Available from: https://www.bmj.com/content/350/bmj.h1258

Lawrason S, Turnnidge J, Tomasone J, Allan V, Côté J, Dawson K, et al. Employing the RE-AIM framework to evaluate multisport service organization initiatives. J Sport Psychol Action. 2021;12(2):87–100.

Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 1999;28(1):10–8. https://doi.org/10.1093/ije/28.1.10 .

Article   CAS   PubMed   Google Scholar  

Milstein B, Wetterhall S. A framework featuring steps and standards for program evaluation. Health Promot Pract. 2000;1(3):221–8. https://doi.org/10.1177/152483990000100304 .

Li V, Carter SM, Rychetnik L. Evidence valued and used by health promotion practitioners. Health Educ Res. 2015;30(2):193–205. https://doi.org/10.1093/her/cyu071 .

Lobo R, Petrich M, Burns SK. Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health. 2014;14(1):1315. https://doi.org/10.1186/1471-2458-14-1315 .

Article   PubMed   PubMed Central   Google Scholar  

Nilsen P (2015) Making sense of implementation theories, models and frameworks. Implementation Sci 10:53. https://doi.org/10.1186/s13012-015-0242-0

Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Shaw RB, Sweet SN, McBride CB, Adair WK, Martin Ginis KA. Operationalizing the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework to evaluate the collective impact of autonomous community programs that promote health and well-being. BMC Public Health. 2019;19(1):803. https://doi.org/10.1186/s12889-019-7131-4 .

Bean CN, Kendellen K, Halsall T, Forneris T. Putting program evaluation into practice: enhancing the Girls Just Wanna Have Fun program. Eval Program Plann. 2015;49:31–40.

Gainforth HL, Latimer-Cheung AE, Athanasopoulos P, Martin Ginis KA. Examining the feasibility and effectiveness of a community-based organization implementing an event-based knowledge mobilization initiative to promote physical activity guidelines for people with spinal cord injury among support personnel. Health Promot Pract. 2014;16(1):55–62. https://doi.org/10.1177/1524839914528210 .

Sweet SN, Ginis KAM, Estabrooks PA, Latimer-Cheung AE. Operationalizing the RE-AIM framework to evaluate the impact of multi-sector partnerships. Implement Sci. 2014;9(1):74. https://doi.org/10.1186/1748-5908-9-74 .

Whitley MA, Forneris T, Barker B. The reality of evaluating community-based sport and physical activity programs to enhance the development of underserved youth: challenges and potential strategies. Quest. 2014;66(2):218–32. https://doi.org/10.1080/00336297.2013.872043 .

Graham ID, Kothari A, McCutcheon C, Angus D, Banner D, Bucknall T, et al. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13(1):22. https://doi.org/10.1186/s13012-017-0700-y .

Gainforth HL, Hoekstra F, McKay R, McBride CB, Sweet SN, Martin Ginis KA, et al. Integrated knowledge translation guiding principles for conducting and disseminating spinal cord injury research in partnership. Arch Phys Med Rehabil. 2021;102(4):656–63.

Charlton JI. Nothing about us without us: disability oppression and empowerment. Berkeley: University of California Press; 1998.

Book   Google Scholar  

Thoele K, Ferren M, Moffat L, Keen A, Newhouse R. Development and use of a toolkit to facilitate implementation of an evidence-based intervention: a descriptive case study. Implement Sci Commun. 2020;1(1):86. https://doi.org/10.1186/s43058-020-00081-x .

Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26(1):13–24. https://doi.org/10.1002/chp.47 .

Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set”—a practical guideline. Trials. 2016;17(1):449. https://doi.org/10.1186/s13063-016-1555-2 .

Hsu CC, Sandford BA. Minimizing non-response in the Delphi Process: how to respond to non-response. Practical Assessment, Research and Evaluation. 2007;12(17).

Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

Shi Z, Michalovic E, McKay R, Gainforth HL, McBride CB, Clarke T, et al. Outcomes of spinal cord injury peer mentorship: a community-based Delphi consensus approach. Ann Phys Rehabil Med. 2023;66(1):101678.

Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48. https://doi.org/10.1177/135581969900400410 .

Birmingham J, Pechman EM, Russell CA, Mielke M. Shared features of high-performing after-school programs: a follow-up to the TASC evaluation. Washington, DC; 2005.

van den Haak MJ, de Jong MDT, Schellens PJ. Evaluation of an informational Web site: three variants of the think-aloud method compared. Tech Commun. 2007;54(1):58–71.

Monforte J, Úbeda-Colomer J. Tinkering with the two-to-one interview: reflections on the use of two interviewers in qualitative constructionist inquiry. Methods Psychol. 2021;5:100082.

Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. https://doi.org/10.1177/1049732305276687 .

Hatton S. Early Prioritisation of Goals. In: Hainaut JL, Rundensteiner EA, Kirchberg M, Bertolotto M, Brochhausen M, Chen YPP, et al., editors. Advances in conceptual modeling—foundations and applications. Berlin, Heidelberg: Springer; 2007. p. 235–44.

Williams TL, Ma JK, Martin Ginis KA. Participant experiences and perceptions of physical activity-enhancing interventions for people with physical impairments and mobility limitations: a meta-synthesis of qualitative research evidence. Health Psychol Rev. 2017;11(2):179–96. https://doi.org/10.1080/17437199.2017.1299027 .

Shirazipour CH, Tennant EM, Aiken AB, Latimer-Cheung AE. Psychosocial aspects of physical activity participation for military personnel with illness and injury: a scoping review. Mil Behav Health. 2019;7(4):459–76. https://doi.org/10.1080/21635781.2019.1611508 .

Fynn JF, Hardeman W, Milton K, Jones AP. A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health. 2020;20(1):1000. https://doi.org/10.1186/s12889-020-09062-0 .

Hildebrand J, Lobo R, Hallett J, Brown G, Maycock B. My-peer toolkit [1.0]: developing an online resource for planning and evaluating peer-based youth programs. Youth Studies Australia. 2012;31(2):53–61.

Buckingham S, Anil K, Demain S, Gunn H, Jones RB, Kent B, et al. Telerehabilitation for people with physical disabilities and movement impairment: development and evaluation of an online toolkit for practitioners and patients. Disabil Rehabil. 2023;45(11):1885–92. https://doi.org/10.1080/09638288.2022.2074549 .

Hoekstra F, Trigo F, Sibley KM, Graham ID, Kennefick M, Mrklas KJ, et al. Systematic overviews of partnership principles and strategies identified from health research about spinal cord injury and related health conditions: a scoping review. J Spinal Cord Med. 2023;46(4):614–31. https://doi.org/10.1080/10790268.2022.2033578 .

Oliver K, Kothari A, Mays N. The dark side of coproduction: do the costs outweigh the benefits for health research? Health Res Policy Syst. 2019;17(1):33. https://doi.org/10.1186/s12961-019-0432-3 .

Salsman JM, Lai JS, Hendrie HC, Butt Z, Zill N, Pilkonis PA, et al. Assessing psychological well-being: self-report instruments for the NIH Toolbox. Qual Life Res. 2014;23(1):205–15. https://doi.org/10.1007/s11136-013-0452-3 .

Bandura A. Self-Efficacy Beliefs of Adolescents. Pajares F, Urdan T, editors. Greenwich: information age publishing; 2006.

Bandura A. Social foundations of thought and action: a social cognitive theory. Englewood Cliffs: Prentice Hall; 1986.

Salsman JM, Schalet BD, Merluzzi TV, Park CL, Hahn EA, Snyder MA, et al. Calibration and initial validation of a general self-efficacy item bank and short form for the NIH PROMIS®. Qual Life Res. 2019;28(9):2513–23. https://doi.org/10.1007/s11136-019-02198-6 .

Smith BW, Dalen J, Wiggins K, Tooley E, Christopher P, Bernard J. The brief resilience scale: assessing the ability to bounce back. Int J Behav Med. 2008;15(3):194–200. https://doi.org/10.1080/10705500802222972 .

The WHO QoL Group. Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol Med. 1998/05/01. 1998;28(3):551–8.

Sallis JF, Grossman RM, Pinski RB, Patterson TL, Nader PR. The development of scales to measure social support for diet and exercise behaviors. Prev Med (Baltimore). 1987;16(6):825–36.

Article   CAS   Google Scholar  

Asunta P, Rintala P, Pochstein F, Lyyra N, McConkey R. The development and initial validation of a short, self-report measure on social inclusion for people with intellectual disabilities: a transnational study. Int J Environ Res Public Health. 2021;18(5):2540.

Craig CL, Marshall AL, Sjöström M, Bauman AE, Booth ML, Ainsworth BE, et al. International physical activity questionnaire: 12-country reliability and validity. Med Sci Sports Exerc. 2003;35(8):1381–95.

Forrest CB, Bevans KB, Pratiwadi R, Moon J, Teneralli RE, Minton JM, et al. Development of the PROMIS® pediatric global health (PGH-7) measure. Qual Life Res. 2014;23(4):1221–31. https://doi.org/10.1007/s11136-013-0581-8 .

Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res. 2009;18(7):873–80. https://doi.org/10.1007/s11136-009-9496-9 .

Marsh HW, Martin AJ, Jackson S. Introducing a short version of the physical self description questionnaire: new strategies, short-form evaluative criteria, and applications of factor analyses. J Sport Exerc Psychol. 2010;32(4):438–82.

Wicker P, Breuer C. Exploring the organizational capacity and organizational problems of disability sport clubs in Germany using matched pairs analysis. Sport Manag Rev. 2014;17(1):23–34.

Thrash TM, Elliot AJ. Inspiration: core characteristics, component processes, antecedents, and function. J Pers Soc Psychol. 2004;87(6):957–73.

Bakhsh JT, Lachance EL, Thompson A, Parent MM. Outcomes of the sport event volunteer experience: examining demonstration effects on first-time and returning volunteers. Int J Event Festiv Manag. 2021;12(2):168–83. https://doi.org/10.1108/IJEFM-09-2020-0057 .

Caron JG, Martin Ginis KA, Rocchi M, Sweet SN. Development of the measure of experiential aspects of participation for people with physical disabilities. Arch Phys Med Rehabil. 2019;100(1):67–77.

Bean C, Kramers S, Camiré M, Fraser-Thomas J, Forneris T. Development of an observational measure assessing program quality processes in youth sport. Cogent Soc Sci. 2018;4(1):1467304. https://doi.org/10.1080/23311886.2018.1467304 .

Download references

Acknowledgements

We would like to acknowledge Ava Neely and Kenedy Olsen for their contributions in assisting with this project. In memoriam of Jane Arkell who played an important role on this project and dedicated herself to improving the lives of individuals with disabilities.

This work was supported by a Social Sciences and Humanities Research Council Connection Grant.

Author information

Authors and affiliations.

School of Health and Exercise Sciences, University of British Columbia, Kelowna, BC, Canada

Sarah V. C. Lawrason, Tanya Forneris, Nathan Adams & Kathleen A. Martin Ginis

International Collaboration on Repair Discoveries, Vancouver, BC, Canada

Sarah V. C. Lawrason, Nathan Adams & Kathleen A. Martin Ginis

Abilities Centre, Whitby, ON, Canada

Pinder DaSilva & Emilie Michalovic

School of Kinesiology and Health Studies, Queen’s University, Kingston, ON, Canada

Amy Latimer-Cheung & Jennifer R. Tomasone

Revved Up, Kingston, ON, Canada

Department of Kinesiology and Physical Education, McGill University, Montreal, QC, Canada

Shane Sweet

Center for Interdisciplinary Research in Rehabilitation of Greater Montreal (CRIR), Montreal, Canada

The Steadward Centre for Personal and Physical Achievement, University of Alberta, Edmonton, AB, Canada

Jennifer Leo

Pickering Football Club, Pickering, ON, Canada

Matthew Greenwood

Rocky Mountain Adaptive, Canmore, AB, Canada

Janine Giles & Nick Boyle

Active Living Alliance, Ottawa, ON, Canada

Jane Arkell

BC Wheelchair Sports Association, Vancouver, BC, Canada

Jackie Patatas

Department of Medicine, University of British Columbia, Vancouver, BC, Canada

Kathleen A. Martin Ginis

You can also search for this author in PubMed   Google Scholar

Contributions

KMG, PDS, EM, TF, JL, MG, JG, JA, JP, & NB made substantial contributions to the conception of the project. SVCL, PDS, EM, ALC, JRT, SS, TF, JL, and KMG designed the project. All authors were involved in acquiring the data through recruitment. SVCL, NA, and KMG analyzed the data. All authors were involved in interpreting the data. SVCL, NA, and KMG drafted the paper or substantively revised it. All authors have approved the submitted version of this paper. All authors have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Sarah V. C. Lawrason .

Ethics declarations

Ethics approval and consent to participate.

Approval was waived as the project conducted under ‘program evaluation’ requirements for University of British Columbia.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1, additional file 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lawrason, S.V.C., DaSilva, P., Michalovic, E. et al. Using mixed methods and partnership to develop a program evaluation toolkit for organizations that provide physical activity programs for persons with disabilities. Res Involv Engagem 10 , 91 (2024). https://doi.org/10.1186/s40900-024-00618-7

Download citation

Received : 02 February 2024

Accepted : 23 July 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s40900-024-00618-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation science
• Knowledge translation
• Delphi technique

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]