what are the four stages of literature review

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

• University of Texas Libraries

Literature Reviews

Steps in the literature review process.

• What is a literature review?
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• Artificial Intelligence (AI) Tools
• You may need to some exploratory searching of the literature to get a sense of scope, to determine whether you need to narrow or broaden your focus
• Identify databases that provide the most relevant sources, and identify relevant terms (controlled vocabularies) to add to your search strategy
• Finalize your research question
• Think about relevant dates, geographies (and languages), methods, and conflicting points of view
• Conduct searches in the published literature via the identified databases
• Check to see if this topic has been covered in other discipline's databases
• Examine the citations of on-point articles for keywords, authors, and previous research (via references) and cited reference searching.
• Save your search results in a citation management tool (such as Zotero, Mendeley or EndNote)
• De-duplicate your search results
• Make sure that you've found the seminal pieces -- they have been cited many times, and their work is considered foundational 
• Check with your professor or a librarian to make sure your search has been comprehensive
• Evaluate the strengths and weaknesses of individual sources and evaluate for bias, methodologies, and thoroughness
• Group your results in to an organizational structure that will support why your research needs to be done, or that provides the answer to your research question  
• Develop your conclusions
• Are there gaps in the literature?
• Where has significant research taken place, and who has done it?
• Is there consensus or debate on this topic?
• Which methodological approaches work best?
• For example: Background, Current Practices, Critics and Proponents, Where/How this study will fit in 
• Organize your citations and focus on your research question and pertinent studies
• Compile your bibliography

Note: The first four steps are the best points at which to contact a librarian. Your librarian can help you determine the best databases to use for your topic, assess scope, and formulate a search strategy.

Videos Tutorials about Literature Reviews

This 4.5 minute video from Academic Education Materials has a Creative Commons License and a British narrator.

Recommended Reading

• Last Updated: Aug 13, 2024 1:52 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

• Franklin University |
• Help & Support |
• Locations & Maps |

• | Research Guides

To access Safari eBooks,

• Select not listed in the Select Your Institution drop down menu.
• Enter your Franklin email address and click Go
• click "Already a user? Click here" link
• Enter your Franklin email and the password you used to create your Safari account.

Continue Close

Literature Review

• Getting Started
• Framing the Literature Review

Literature Review Process

• Mistakes to Avoid & Additional Help

The structure of a literature review should include the following :

• An overview of the subject, issue or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories (e.g. works that support of a particular position, those against, and those offering alternative approaches entirely),
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research

The critical evaluation of each work should consider :

• Provenance  -- what are the author's credentials? Are the author's arguments supported by evidence (e.g. primary historical material, case studies, narratives, statistics, recent scientific findings)?
• Objectivity  -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness  -- which of the author's theses are most/least convincing?
• Value  -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

Development of the Literature Review

Four stages:.

• Introduce the reader to the importance of the topic being studied . The reader is oriented to the significance of the study and the research questions or hypotheses to follow.
• Places the problem into a particular context  that defines the parameters of what is to be investigated.
• Provides the framework for reporting the results  and indicates what is probably necessary to conduct the study and explain how the findings will present this information.
• Literature search -- finding materials relevant to the subject being explored.
• Evaluation of resources  -- determining which literature makes a significant contribution to the understanding of the topic.
• Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review:

Sources and expectations.  if your assignment is not very specific about what form your literature review should take, seek clarification from your professor by asking these questions:.

• Roughly how many sources should I include?
• What types of sources should I review (books, journal articles, websites)?
• Should I summarize, synthesize, or critique your sources by discussing a common theme or issue?
• Should I evaluate the sources?
• Should I provide subheadings and other background information, such as definitions and/or a history?

Find Models.   When reviewing the current literature, examine how authors in your discipline or area of interest have organized their literature reviews. Read not only for information, but also to get a sense of the types of themes you might want to look for in your own research review.

Narrow the topic.  the narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources., consider whether your sources are current and applicable.  s ome disciplines require that you use information that is as current as possible. this is very common in the sciences where research conducted only two years ago could be obsolete. however, when writing a review in the social sciences, a survey of the history of the literature may be what is needed because what is important is how perspectives have changed over the years or within a certain time period. try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. you can also use this method to consider what is consider by scholars to be a "hot topic" and what is not., follow the bread crumb trail.  the bibliography or reference section of sources you read are excellent entry points for further exploration. you might find resourced listed in a bibliography that points you in the direction you wish to take your own research., ways to organize your literature review, chronologically:  .

If your review follows the chronological method, you could write about the materials according to when they were published or the time period they cover.

By Publication:  

Order your sources chronologically by publication date, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies.

Conceptual Categories:

The literature review is organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it will still be organized chronologically reflecting technological developments in media. The only difference here between a "chronological" and a "thematic" approach is what is emphasized the most.

Methodological:  

A methodological approach focuses on the methods utilized by the researcher.  A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Sections of Your Literature Review:  

Once you've decided on the organizational method for your literature review, the sections you need to include should be easy to figure out because they arise from your organizational strategy.

Here are examples of other sections you may need to include depending on the type of review you write:

• Current Situation : information necessary to understand the topic or focus of the literature review.
• History : the chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : the criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed articles and journals.
• Standards : the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence:

A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be Selective:  

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological.

Use Quotes Sparingly:  

Some short quotes are okay if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Sometimes you may need to quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute your own summary and interpretation of the literature.

Summarize and Synthesize:  

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to their own work.

Keep Your Own Voice:  

While the literature review presents others' ideas, your voice (the writer's) should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording.

Use Caution When Paraphrasing:  

When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

• << Previous: Getting Started
• Next: Mistakes to Avoid & Additional Help >>
• Last Updated: Oct 3, 2023 2:44 PM
• URL: https://guides.franklin.edu/LITREVIEW

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved August 12, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Research Process :: Step by Step

• Introduction
• Select Topic
• Identify Keywords
• Background Information
• Develop Research Questions
• Refine Topic
• Search Strategy
• Popular Databases
• Evaluate Sources
• Types of Periodicals
• Reading Scholarly Articles
• Primary & Secondary Sources
• Organize / Take Notes
• Writing & Grammar Resources
• Annotated Bibliography
• Literature Review
• Citation Styles
• Paraphrasing
• Privacy / Confidentiality
• Research Process
• Selecting Your Topic
• Identifying Keywords
• Gathering Background Info
• Evaluating Sources

Organize the literature review into sections that present themes or identify trends, including relevant theory. You are not trying to list all the material published, but to synthesize and evaluate it according to the guiding concept of your thesis or research question.  

What is a literature review?

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment, but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries

A literature review must do these things:

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

Ask yourself questions like these:

• What is the specific thesis, problem, or research question that my literature review helps to define?
• What type of literature review am I conducting? Am I looking at issues of theory? methodology? policy? quantitative research (e.g. on the effectiveness of a new procedure)? qualitative research (e.g., studies of loneliness among migrant workers)?
• What is the scope of my literature review? What types of publications am I using (e.g., journals, books, government documents, popular media)? What discipline am I working in (e.g., nursing psychology, sociology, medicine)?
• How good was my information seeking? Has my search been wide enough to ensure I've found all the relevant material? Has it been narrow enough to exclude irrelevant material? Is the number of sources I've used appropriate for the length of my paper?
• Have I critically analyzed the literature I use? Do I follow through a set of concepts and questions, comparing items to each other in the ways they deal with them? Instead of just listing and summarizing items, do I assess them, discussing strengths and weaknesses?
• Have I cited and discussed studies contrary to my perspective?
• Will the reader find my literature review relevant, appropriate, and useful?

Ask yourself questions like these about each book or article you include:

• Has the author formulated a problem/issue?
• Is it clearly defined? Is its significance (scope, severity, relevance) clearly established?
• Could the problem have been approached more effectively from another perspective?
• What is the author's research orientation (e.g., interpretive, critical science, combination)?
• What is the author's theoretical framework (e.g., psychological, developmental, feminist)?
• What is the relationship between the theoretical and research perspectives?
• Has the author evaluated the literature relevant to the problem/issue? Does the author include literature taking positions she or he does not agree with?
• In a research study, how good are the basic components of the study design (e.g., population, intervention, outcome)? How accurate and valid are the measurements? Is the analysis of the data accurate and relevant to the research question? Are the conclusions validly based upon the data and analysis?
• In material written for a popular readership, does the author use appeals to emotion, one-sided examples, or rhetorically-charged language and tone? Is there an objective basis to the reasoning, or is the author merely "proving" what he or she already believes?
• How does the author structure the argument? Can you "deconstruct" the flow of the argument to see whether or where it breaks down logically (e.g., in establishing cause-effect relationships)?
• In what ways does this book or article contribute to our understanding of the problem under study, and in what ways is it useful for practice? What are the strengths and limitations?
• How does this book or article relate to the specific thesis or question I am developing?

Text written by Dena Taylor, Health Sciences Writing Centre, University of Toronto

http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review

• << Previous: Annotated Bibliography
• Next: Step 5: Cite Sources >>
• Last Updated: Jun 13, 2024 4:27 PM
• URL: https://libguides.uta.edu/researchprocess

University of Texas Arlington Libraries 702 Planetarium Place · Arlington, TX 76019 · 817-272-3000

• Internet Privacy
• Accessibility
• Problems with a guide? Contact Us.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE: Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Aug 13, 2024 12:57 PM
• URL: https://libguides.usc.edu/writingguide

Writing Literature Reviews

Literature review steps.

• Types of Literature Reviews
• Research Question Format
• Managing Your Sources
• Library Resources
• Infographic
• Infographic Text

4 Step Guide to Writing a Literature Review

(as transcribed by Clark College Libraries )

1. What is a Literature Review?

A literature review is a description of the literature relevant to a particular field or topic.

It gives an overview of:

• what has been said
• key writers
• prevailing theories and hypotheses
• questions being asked
• appropriate and useful methods and methodologies

It make take two forms

• Purely descriptive - as in an annotated bibliography. A descriptive review should not just list and paraphrase, but should add comment and bring out themes and trends.
• A critical assessment of the literature in a particular field, stating where tje weaknesses amd gaps are, contrasting the views of particular authors, or raising questions. It will evaluate and show relationships, so that key themes emerge.
• A whole paper, which annotates and/or critiques the literature in a particular subject area.
• Part of a thesis or dissertation, forming an early context-setting chapter.
• A useful background outlining a piece of research, or putting forward a hypothesis.

DO look at the relationships between the views and draw out themes

DON'T just write a list of quote authors without citing them.

2. The Stages of a Literature Review.

Define the problem.

• It is important to define the problem or area which you wish to address.
• Have a purpose for your literature review to narrow the scope of what you need to look out for when you read.

Carry out a search for relevant materials.

• Peer reviewed journal articles
• Newspaper articles
• Historical records
• Commercial/government reports and statistical information
• Theses and dissertations
• Other relevant information
• Search the university or academic library with a good collection in your subject area.
• Search using the internet--but be sure to avoid the pitfalls.
• Use specific rather than general keywords and phrases for your search strategy.

Evaluate the materials.

Points to consider when evaluating material:

• Author credentials--are they an expert in the field? Are they affiliated to a reputable organization?
• Date of publication--is it sufficiently current or has knowledge moved on?
• If a book--is it the latest edition?
• If a journal--is it a peer reviewed, scholarly journal?
• Is the publisher reputable and scholarly?
• Is it addressing a scholarly audience?
• Does it review relevant literature?
• Is it an objective fact-based viewpoint?  Is it logically organized and clear to follow?
• Does it follow a particular theoretical viewpoint, e.g.feminist?
• What is the relationship of this work to other material on the same topic--does it substantiate it or add a different perspective?
• If using research, is the design sound?  Is it primary or secondary material? 
• If it is from a practice-based perspective, what are the implications for practice?

Analyse the findings.

• What themes emerge, and what conclusions can be drawn?
• What are the major similarities and differences between various writers?
• Are there any significant questions which emerge and which could form a basis for further investigation?

3. How to Organize a Literature Review

Introduction: Define the topic and state reasons for choice.  You could also point out overall trends, gaps and themes that emerge.

Body: Discuss your sources.  You can organize your discussion chronologically, thematically or methidologically.  

Conclusion: Summarize the major contributions, evaluating the current position, and pointing out flaws in methodology, gaps in the research, contradictions and areas for further study. 

You are now at the stage when you can write up your literature review.  

4. Further Information.

These universities have good information on how to write a literature review: 

• Deakin University--http://www.deakin.edu.au/library/research/index.php
• University of Wisconsin-Madison--http://www.wisc.edu/writing/Handbook/ReviewofLiterature.html
• University of North Carolina--http://www.writingcenter.unc.edu/handouts/literature-reviews/
• University of California, Santa Cruz--http://www.library.ucsc.edu/ref/howto  (Follow links to "Write a Literature Review".)

Brought to you by Emerald Group Publishing www.emeraldgrouppublishing.com 

• << Previous: Get Started
• Next: Types of Literature Reviews >>

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

University Library

Write a literature review.

• Examples and Further Information

1. Introduction

Not to be confused with a book review, a literature review surveys scholarly articles, books and other sources (e.g. dissertations, conference proceedings) relevant to a particular issue, area of research, or theory, providing a description, summary, and critical evaluation of each work. The purpose is to offer an overview of significant literature published on a topic.

2. Components

Similar to primary research, development of the literature review requires four stages:

• Problem formulation—which topic or field is being examined and what are its component issues?
• Literature search—finding materials relevant to the subject being explored
• Data evaluation—determining which literature makes a significant contribution to the understanding of the topic
• Analysis and interpretation—discussing the findings and conclusions of pertinent literature

Literature reviews should comprise the following elements:

• An overview of the subject, issue or theory under consideration, along with the objectives of the literature review
• Division of works under review into categories (e.g. those in support of a particular position, those against, and those offering alternative theses entirely)
• Explanation of how each work is similar to and how it varies from the others
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research

In assessing each piece, consideration should be given to:

• Provenance—What are the author's credentials? Are the author's arguments supported by evidence (e.g. primary historical material, case studies, narratives, statistics, recent scientific findings)?
• Objectivity—Is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness—Which of the author's theses are most/least convincing?
• Value—Are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

3. Definition and Use/Purpose

A literature review may constitute an essential chapter of a thesis or dissertation, or may be a self-contained review of writings on a subject. In either case, its purpose is to:

• Place each work in the context of its contribution to the understanding of the subject under review
• Describe the relationship of each work to the others under consideration
• Identify new ways to interpret, and shed light on any gaps in, previous research
• Resolve conflicts amongst seemingly contradictory previous studies
• Identify areas of prior scholarship to prevent duplication of effort
• Point the way forward for further research
• Place one's original work (in the case of theses or dissertations) in the context of existing literature

The literature review itself, however, does not present new primary scholarship.

• Next: Examples and Further Information >>

Creative Commons Attribution 3.0 License except where otherwise noted.

Land Acknowledgement

The land on which we gather is the unceded territory of the Awaswas-speaking Uypi Tribe. The Amah Mutsun Tribal Band, comprised of the descendants of indigenous people taken to missions Santa Cruz and San Juan Bautista during Spanish colonization of the Central Coast, is today working hard to restore traditional stewardship practices on these lands and heal from historical trauma.

The land acknowledgement used at UC Santa Cruz was developed in partnership with the Amah Mutsun Tribal Band Chairman and the Amah Mutsun Relearning Program at the UCSC Arboretum .

• UWF Libraries

Literature Review: Conducting & Writing

• Steps for Conducting a Lit Review

1. Choose a topic. Define your research question.

2. decide on the scope of your review., 3. select the databases you will use to conduct your searches., 4. conduct your searches and find the literature. keep track of your searches, 5. review the literature..

• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window
• Sample Literature Reviews

Disclaimer!!

Conducting a literature review is usually recursive, meaning that somewhere along the way, you'll find yourself repeating steps out-of-order.

That is actually a good sign.  

Reviewing the research should lead to more research questions and those questions will likely lead you to either revise your initial research question or go back and find more literature related to a more specific aspect of your research question.

Your literature review should be guided by a central research question.  Remember, it is not a collection of loosely related studies in a field but instead represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor.

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

Tip: This may depend on your assignment.  How many sources does the assignment require?

Make a list of the databases you will search.  Remember to include comprehensive databases such as WorldCat and Dissertations & Theses, if you need to.

Where to find databases:

• Find Databases by Subject UWF Databases categorized by discipline
• Find Databases via Research Guides Librarians create research guides for all of the disciplines on campus! Take advantage of their expertise and see what discipline-specific search strategies they recommend!
• Review the abstracts of research studies carefully. This will save you time.
• Write down the searches you conduct in each database so that you may duplicate them if you need to later (or avoid dead-end searches   that you'd forgotten you'd already tried).
• Use the bibliographies and references of research studies you find to locate others.
• Ask your professor or a scholar in the field if you are missing any key works in the field.
• Use RefWorks to keep track of your research citations. See the RefWorks Tutorial if you need help.

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions. Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited?; if so, how has it been analyzed?

Tips: 

• Again, review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• << Previous: Home
• Next: Finding "The Literature" >>
• Last Updated: Aug 8, 2024 11:00 AM
• URL: https://libguides.uwf.edu/litreview

• Learn How to Use the Library
• Providers & Employees
• Research Help
• All Research Guides

Conducting Literature Reviews

• About Literature Reviews

Process Overview

Step 1: the research question, step 2: search the literature, step 3: manage results, step 4: synthesize information, step 5: write the review.

• Additional Resources
• APA Style (7th ed.) This link opens in a new window

Like research, writing a literature review is an iterative process. Here is a very broad example of the process:

• Frame the research question and determine the scope of the literature review
• Search relevant bodies of literature
• Manage and organize search results
• Synthesize the literature
• Write an assessment of the literature

The initial steps should already be familiar to you, as they parallel steps of the research process you have used before.

Research questions, like topics, must be specific and focused so that you can 1) search for materials to address the question, and 2) write a literature review that is manageable in scope and purpose.

Developing a research question is the next logical step after selecting and then narrowing a topic. It is important to have a research question because it focuses your next step in the literature review process: searching. As Booth (2008) explains in  The Craft of Research : "If a writer asks no specific  question  worth asking, he can offer no specific  answer  worth supporting. And without an answer to support, he cannot select from all the data he  could  find on a topic to just those relevant to his answer" (p. 41).

Once you have selected and narrowed your topic, ask yourself questions about the topic's:

• History (Is is part of a large context? What is its own internal history? How has it changed over time?)
• Structure and composition (Is it part of a larger system/structure? How do its parts fit together?)
• Categorization (Can you compare/contrast it with similar topics? Does it belong to a group of similar kinds?)

You can also:

• Turn positive questions into negative ones by focusing on "nots" (why didn't this happen? why isn't this significant in context?) or by contrasting differences
• Ask "what if" speculative questions (what if your topic disappeared? Was put in a different context?)
• Ask questions suggested by your initial background research, such as those that build on agreement (Author X made a persuasive point...) or reflect disagreement (Author Y's conclusion doesn't account for this contextual element...)

You may find that you need to reframe or revise your question as you continue through the literature process. That's ok! Remember, the literature review process is iterative.

For more detailed information on forming and evaluating research questions, see these books available to order through ILL from OhioLINK.

• OhioLINK Library Catalog This link opens in a new window Catalog of books and other materials held in Ohio college and university libraries.

More Resources

• The Research Process Get help with selecting and narrowing a topic.

General guidance on where to search for sources:

• Where to Find Sources

Subject-specific guidance on where to search for sources:

• Evidence-Based Practice by Mike Jundi Last Updated Jan 26, 2023 18 views this year
• Finding Legislation, Data, & Statistics by Mike Jundi Last Updated Dec 5, 2023 22 views this year
• Nursing Research by Mike Jundi Last Updated Jan 18, 2024 57 views this year
• RAD 112 - Introduction to Radiography by Mike Jundi Last Updated Jan 18, 2024 14 views this year
• RAD 246 - Radiographic Pathology by Mike Jundi Last Updated Mar 14, 2024 209 views this year
• Social Work Resources by Mike Jundi Last Updated Dec 5, 2023 15 views this year

How to search for sources by developing a search strategy:

• How to Search for Sources

General guidance on using catalogs and databases:

• Basic Library Tutorials by Mike Jundi Last Updated Jan 18, 2024 443 views this year

Research management involves collecting, organizing, and citing.

Research management is also based largely on personal preference. Do you have a system that works for you? Great! If you aren't used to research management and/or don't have an effective system in place, you have options.

• Do-it-yourself: maintain your resources on your computer's hard drive or on the cloud (Microsoft OneDrive, Google Drive, DropBox)
• Use a free research management software (Mendeley, Zotero, EndNote)

Regardless of what system you use, it is necessary to keep track of the these elements:

• The literature you found (Did you find full text in a PDF? Save it. Did you find a record in a database, but need to request the article? Save the permalink to the record.)
• The full APA citation for the literature
• An easy way to track results you've found in databases is to create folders

Finally, you will need a note-taking system that will help you record the key concepts from the literature when you read and synthesize it. If you already have one, great! If you struggle with note-taking, see the links below.

What is synthesis?

Synthesizing information is much the opposite of analyzing information. When you read an article or book, you have to pull out specific concepts from the larger document in order to understand it. This is analyzing.

When you synthesize information, you take specific concepts and consider them together to understand how they compare/contrast and how they relate to one another. In other terms, synthesis involves combining multiple elements to create a whole. In regard to literature reviews, the  elements  refer to the findings from the literature you've gathered. The  whole  then becomes your conclusion(s) about those findings.

How do I synthesize information?

Note: This stage in the literature review process is as iterative and personal as any other. These steps offer a guideline, but do what works for you best.

• This is where you really decide if you want to read specific materials
• If you have gathered a substantial amount of literature and reading all of it would prove overwhelming, read the abstracts to get a better idea of the content, then select the materials that would best support your review
• Describe and analyze the findings (What were the results? How did the authors get these results? What are the impacts? Etc.)
• Identify the key concepts
• Compare and contrast findings, concepts, conclusions, methods, etc.
• Evaluate the quality and significance of findings, concepts, conclusions, methods, etc.
• Interpret the findings, concepts, conclusions, methods, etc. in the context of your research question
• This is the step where your synthesis of the information will lead to logical conclusions about that information
• These conclusions should speak directly to your research question (i.e. your question should have an answer)

Visit the link below for helpful resources on note-taking: 

• Other Helpful Tips: Note-Taking & Proofreading

Writing style

You are expected to follow APA Style in your writing. Visit this guide for an introduction, tips, and tutorials:

• APA Style Resources (7th ed.) by Mike Jundi Last Updated Jan 13, 2023 260 views this year

The structure and flow of your literature review should be logical and should reflect the synthesis you have done.

A common pitfall for students is using an  author-driven structure , which might look something like this:

• Introduction
• Author 1 says x
• Author 2 says y
• Author ∞ says...

Why doesn't the author-driven structure work?

• Leans toward listing or summarizing information
• Doesn't illustrate synthesis of information (all of the findings are listed based on where they came from, not their meaning, impact, or significance)

What structures do work? The APA suggests three structures for literature reviews:

• Theme-based (group studies based on common themes or concepts present)
• Methodology-based (group studies based on the methodologies used)
• Chronological (group studies based on the historical developments in the field)

Theme-based structure 

The theme-based structure is applicable to most bodies of literature you might gather. It may look like this:

• Concept x from author 1
• Concept a from author 5
• Concept y from author 2
• Concepts…

Why does the them-based structure work better?

• It avoids listing information
• It clearly shows the synthesis that occurred
• It illustrates the connections between concepts and the significance of particular concepts
• << Previous: About Literature Reviews
• Next: Additional Resources >>
• Last Updated: Oct 13, 2022 3:08 PM
• URL: https://aultman.libguides.com/literaturereviews

Aultman Health Sciences Library

Aultman Education Center, C2-230, 2600 Sixth St SW, Canton, OH 44710  |  330-363-5000   |  [email protected]

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 

How to write a good literature review 

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 

Find academic papers related to your research topic faster. Try Research on Paperpal  

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!    

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as you go with Paperpal Research. Start now for free.   

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface with the option to save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 
• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide), five things authors need to know when using..., 7 best referencing tools and citation management software..., maintaining academic integrity with paperpal’s generative ai writing....

Literature Reviews

• Tools & Visualizations
• Literature Review Examples
• Videos, Books & Links

Business & Econ Librarian

Click to Chat with a Librarian

Text: (571) 248-7542

What is a literature review?

A literature review discusses published information in a particular subject area. Often part of the introduction to an essay, research report or thesis, the literature review is literally a "re" view or "look again" at what has already been written about the topic, wherein the author analyzes a segment of a published body of knowledge through summary, classification, and comparison of prior research studies, reviews of literature, and theoretical articles. Literature reviews provide the reader with a bibliographic history of the scholarly research in any given field of study. As such,  as new information becomes available, literature reviews grow in length or become focused on one specific aspect of the topic.

A literature review can be just a simple summary of the sources, but usually contains an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, whereas a synthesis is a re-organization, or a reshuffling, of that information. The literature review might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. Depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

A literature review is NOT:

• An annotated bibliography – a list of citations to books, articles and documents that includes a brief description and evaluation for each citation. The annotations inform the reader of the relevance, accuracy and quality of the sources cited.
• A literary review – a critical discussion of the merits and weaknesses of a literary work.
• A book review – a critical discussion of the merits and weaknesses of a particular book.
• Teaching Information Literacy Reframed: 50+ Framework-Based Exercises for Creating Information-Literate Learners
• The UNC Writing Center – Literature Reviews
• The UW-Madison Writing Center: The Writer’s Handbook – Academic and Professional Writing – Learn How to Write a Literature Review

What is the difference between a literature review and a research paper?

The focus of a literature review is to summarize and synthesize the arguments and ideas of others without adding new contributions, whereas academic research papers present and develop new arguments that build upon the previously available body of literature.

How do I write a literature review?

There are many resources that offer step-by-step guidance for writing a literature review, and you can find some of them under Other Resources in the menu to the left. Writing the Literature Review: A Practical Guide suggests these steps:

• Chose a review topic and develop a research question
• Locate and organize research sources
• Select, analyze and annotate sources
• Evaluate research articles and other documents
• Structure and organize the literature review
• Develop arguments and supporting claims
• Synthesize and interpret the literature
• Put it all together

What is the purpose of writing a literature review?

Literature reviews serve as a guide to a particular topic: professionals can use literature reviews to keep current on their field; scholars can determine credibility of the writer in his or her field by analyzing the literature review.

As a writer, you will use the literature review to:

• See what has, and what has not, been investigated about your topic
• Identify data sources that other researches have used
• Learn how others in the field have defined and measured key concepts
• Establish context, or background, for the argument explored in the rest of a paper
• Explain what the strengths and weaknesses of that knowledge and ideas might be
• Contribute to the field by moving research forward
• To keep the writer/reader up to date with current developments in a particular field of study
• Develop alternative research projects
• Put your work in perspective
• Demonstrate your understanding and your ability to critically evaluate research in the field
• Provide evidence that may support your own findings
• Next: Tools & Visualizations >>
• Last Updated: Jul 30, 2024 3:43 PM
• URL: https://subjectguides.library.american.edu/literaturereview

University Libraries

Literature review.

• What is a Literature Review?
• What is Its Purpose?
• 1. Select a Topic
• 2. Set the Topic in Context
• 3. Types of Information Sources
• 4. Use Information Sources
• 5. Get the Information
• 6. Organize / Manage the Information
• 7. Position the Literature Review
• 8. Write the Literature Review

A literature review is a comprehensive summary of previous research on a topic. The literature review surveys scholarly articles, books, and other sources relevant to a particular area of research.  The review should enumerate, describe, summarize, objectively evaluate and clarify this previous research.  It should give a theoretical base for the research and help you (the author) determine the nature of your research.  The literature review acknowledges the work of previous researchers, and in so doing, assures the reader that your work has been well conceived.  It is assumed that by mentioning a previous work in the field of study, that the author has read, evaluated, and assimiliated that work into the work at hand.

A literature review creates a "landscape" for the reader, giving her or him a full understanding of the developments in the field.  This landscape informs the reader that the author has indeed assimilated all (or the vast majority of) previous, significant works in the field into her or his research. 

 "In writing the literature review, the purpose is to convey to the reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. The literature review must be defined by a guiding concept (eg. your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries.( http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review )

Recommended Reading

• Next: What is Its Purpose? >>
• Last Updated: Oct 2, 2023 12:34 PM

Planning and carrying out a literature review

http://prezi.com/w9f6pi15hyyb/?utm_campaign=share&utm_medium=copy Draft Text

4. Stages of a literature review

Before you start your literature review, you should already have  defined   your research topic . Familiarise yourself with your topic's main themes, and  be aware of your limits .  

There are  four main steps  you’ll need to take to complete your literature review:

1.  You will firstly need to  find  literature which is  relevant  to your research topic. Before starting your search, make sure that you have  defined your subject . Your literature search should be led by the main themes and limits of your research.  

2.  Evaluating  your results is the next step. You will need to make a decision as to whether the literature you have found makes a significant contribution to the understanding of your topic. The Prezi ( You've finished your literature search... What next? ) at the bottom of the page will guide you through the  process of deciding  what literature needs to be included in your literature review.

  3.  You will then need to  analyse and interpret  the literature you have decided to include in your review. Identify themes/ideas/theories/approaches that have emerged from reading the literature.

  4.  Once you are satisfied that you have reviewed enough literature relevant to your research topic, you can get down to  writing . You will need to introduce and explain each theme (or theory/approach), provide evidence from the literature, comment critically on the literature and  relate it to your own research .

Prezi: You've finished your literature search... What next? 

Whilst this process is usually followed by medical students undertaking a systematic review, the framework could easily be used by students from any discipline.     

The Teaching & Learning Support Team are always ready to assist:  Contact the T&LS Team   

Graduate Research: Guide to the Literature Review

• "Literature review" defined
• Research Communication Graphic
• Literature Review Steps
• Search techniques
• Finding Additional "Items
• Evaluating information
• Citing Styles
• Ethical Use of Information
• Research Databases This link opens in a new window
• Get Full Text
• Reading a Scholarly Article
• Author Rights
• Selecting a publisher

Introduction to Research Process: Literature Review Steps

When seeking information for a literature review or for any purpose, it helps to understand information-seeking as a process that you can follow. 5 Each of the six (6) steps has its own section in this web page with more detail. Do (and re-do) the following six steps:

1. Define your topic. The first step is defining your task -- choosing a topic and noting the questions you have about the topic. This will provide a focus that guides your strategy in step II and will provide potential words to use in searches in step III.

2. Develop a strategy. Strategy involves figuring out where the information might be and identifying the best tools for finding those types of sources. The strategy section identifies specific types of research databases to use for specific purposes.

3. Locate the information . In this step, you implement the strategy developed in II in order to actually locate specific articles, books, technical reports, etc.

4. Use and Evaluate the information. Having located relevant and useful material, in step IV you read and analyze the items to determine whether they have value for your project and credibility as sources.

5. Synthesize. In step V, you will make sense of what you've learned and demonstrate your knowledge. You will thoroughly understand, organize and integrate the information --become knowledgeable-- so that you are able to use your own words to support and explain your research project and its relationship to existing research by others.

6. Evaluate your work. At every step along the way, you should evaluate your work. However, this final step is a last check to make sure your work is complete and of high quality.

Continue below to begin working through the process.

5. Eisenberg, M. B., & Berkowitz, R. E. (1990). Information Problem-Solving: the Big Six Skills Approach to Library & Information Skills Instruction . Norwood, NJ: Ablex Publishing.

1. Define your topic.

I. Define your topic

A.  Many students have difficulty selecting a topic. You want to find a topic you find interesting and will enjoy learning more about.

B.   Students often select a topic that is too broad.  You may have a broad topic in mind initially and will need to narrow it.

1. To help narrow a broad topic :

a. Brainstorm.  

1). Try this technique for brainstorming to narrow your focus.   

a) Step 1.  Write down your broad topic.

b) Step 2. Write down a "specific kind" or "specific aspect" of the topic you identified in step 1.  

c) Step 3. Write down an aspect  --such as an attribute or behavior-- of the "specific kind" you identified in step 2.  

d) Step 4.  Continue to add  levels of specificity as needed to get to a focus that is manageable. However, you may want to begin researching the literature before narrowing further to give yourself the opportunity to explore what others are doing and how that might impact the direction that you take for your own research.                     

2) Three examples of using the narrowing technique. These examples start with very, very broad topics, so the topic at step 3 or 4 in these examples would be used for a preliminary search in the literature in order to identify a more specific focus.  Greater specificity than level 3 or 4 will ultimately be necessary for developing a specific research question. And we may discover in our preliminary research that we need to alter the direction that we originally were taking.

a) Example 1.      

             Step 1. information security

                      Step  2. protocols

                              Step 3.  handshake protocol

            Brainstorming has brought us to focus on the handshake protocol.

b) Example 2.  

            Step 1. information security

                     Step 2. single sign-on authentication

                              Step 3.  analyzing

                                       Step 4. methods

            Brainstorming has brought us to focus on methods for analyzing the security of single sign-on authentication

c) Example 3.  The diagram below is an example using the broad topic of "software" to show two potential ways to begin to narrow the topic. 

C. Once you have completed the brainstorming process and your topic is more focused, you can do preliminary research to help you identify a specific research question . 

1) Examine overview sources such as subject-specific encyclopedias and textbooks that are likely to break down your specific topic into sub-topics and to highlight core issues that could serve as possible research questions. [See section II. below on developing a strategy to learn how to find these encyclopedias]

2). Search the broad topic in a research database that includes scholarly journals and professional magazines (to find technical and scholarly articles) and scan recent article titles for ideas. [See section II. below on developing a strategy to learn how to find trade and scholarly journal articles]

D. Once you have identified a research question or questions, ask yourself what you need to know to answer the questions. For example,

1. What new knowledge do I need to gain?

2. What has already been answered by prior research of other scholars?

E.  Use the answers to the questions in C. to identify what words  to use to describe the topic when you are doing searches.

1. Identify key words

a.  For example , if you are investigating "security audits in banking", key terms to combine in your searches would be: security, audits, banking.

2.  Create a list of alternative ways of referring to a key word or phrase

a.For example , "information assurance" may be referred to in various ways such as: "information assurance," "information security," and "computer security."

b. Use these alternatives when doing searches.

3. As you are searching, pay attention to how others are writing about the topic and add new words or phrases to your searches if appropriate.

2. Develop a strategy.

II. Develop a strategy for finding the information. 

A. Start by considering what types of source might contain the information you need .  Do you need a dictionary for definitions? a directory for an address? the history of a concept or technique that might be in a book or specialized encyclopedia? today's tech news in an online tech magazine or newspaper?  current research in a journal article? background information that might be in a specialized encyclopedia? data or statistics from a specific organization or website?  Note that you will typically have online access to these source types.

B. This section provides a description of some of the common types of information needed for research.  

1. For technical and business analysis , look for articles in technical and trade magazines . These articles are written by information technology professionals to help other IT professionals do their jobs better. Content might include news on new developments in hardware or software, techniques, tools, and practical advice. Technical journals are also likely to have product ads relevant to information technology workers and to have job ads. Examples iof technical magazines include Network Computing and IEEE Spectrum .

2. To read original research studies , look for articles in scholarly journals and conference proceedings . They will provide articles written by  information technology professionals who are reporting original research; that is, research that has been done by the authors and is being reported for the first time. The audience for original research articles is other information technology scholars and professionals. Examples of scholarly journals include Journal of Applied Security Research , Journal of Management Information Systems , IEEE Transactions on Computers , and ACM Transactions on Information and System Security .

3. For original research being reported to funding agencies , look for technical reports on agency websites. Technical reports are researcher reports to funding agencies about progress on or completion of research funded by the agency.

4. For in-depth, comprehensive information on a topic , look for book-length volumes . All chapters in the book might be written by the same author(s) or might be a collection of separate papers written by different authors.

5. To learn about an unfamiliar topic , use textbooks ,  specialized encyclopedias and handbooks to get get overviews of topics, history/background, and key issues explained.

6. For instructions for hardware, software, networking, etc., look for manuals  that provide step-by-step instructions.

7. For technical details about inventions (devices, instruments, machines), look for patent documents .

C.   NOTE -  In order to search for and find original research studies,  it will help if you  understand  how information is produced, packaged  and  communicated  within your profession. This is explained in the tab  "Research Communication: Graphic."

3. Locate the information.

III. Locate the information

A. Use search tools designed to find the sources you want.  Types of sources were described in section II. above. 

Always feel free to Ask a librarian for assistance when you have questions about where and how locate the information you need.

B. Evaluate the search results (no matter where you find the information)

1. Evaluate the items you find using at least these 5 criteria:

a. accuracy -- is the information reliable and error free?

1) Is there an editor or someone who verifies/checks the information?

2) Is there adequate documentation: bibliography, footnotes, credits?

3) Are the conclusions justified by the information presented?

b. authority -- is the source of the information reputable?

1) How did you find the source of information: an index to edited/peer-reviewed material, in a bibliography from a published article, etc.?

2) What type of source is it: sensationalistic, popular, scholarly?

c. objectivity -- does the information show bias?

1) What is the purpose of the information: to inform, persuade, explain, sway opinion, advertise?

2) Does the source show political or cultural biases?

d. currency -- is the information current? does it cover the time period you need?

e. coverage -- does it provide the evidence or information you need?

2. Is the search producing the material you need? -- the right content? the right quality? right time period? right geographical location? etc. If not, are you using

a. the right sources?

b. the right tools to get to the sources?

c. are you using the right words to describe the topic?

3. Have you discovered additional terms that should be searched? If so, search those terms.

4. Have you discovered additional questions you need to answer? If so, return to section A above to begin to answer new questions.

4. Use and evaluate the information.

IV. Use the information.

A. Read, hear or view the source

1. Evaluate: Does the material answer your question(s)? -- right content? If not, return to B.

2. Evaluate: Is the material appropriate? -- right quality? If not, return to B.

B. Extract the information from the source : copy/download information, take notes, record citation, keep track of items using a citation manager.

1. Note taking (these steps will help you when you begin to write your thesis and/or document your project.):

a. Write the keywords you use in your searches to avoid duplicating previous searches if you return to search a research database again. Keeping track of keywords used will also save you time if your search is interrupted or you need return and do the search again for some other reason. It will help you remember which search terms worked successfully in which databases

b. Write the citations or record the information needed to cite each article/document you plan to read and use, or make sure that any saved a copy of the article includes all the information needed to cite it. Some article pdf files may not include all of the information needed to cite, and it's a waste of your valuable time to have to go back to search and find the items again in order to be able to cite them. Using citation management software such as EndNote will help keep track of citations and help create bibliographies for your research papers.

c. Write a summary of each article you read and/or why you want to use it.

5. Synthesize.

V. Synthesize.

A. Organize and integrate information from multiple sources

B. Present the information (create report, speech, etc. that communicates)

C. Cite material using the style required by your professor or by the venue (conference, publication, etc.). For help with citation styles, see  Guide to Citing Sources .  A link to the citing guide is also available in the "Get Help" section on the left side of the Library home page

6. Evaluate your work.

VI. Evaluate the paper, speech, or whatever you are using to communicate your research.

A. Is it effective?

B. Does it meet the requirements?

C. Ask another student or colleague to provide constructive criticism of your paper/project.

• << Previous: Research Communication Graphic
• Next: Search techniques >>
• Last Updated: Apr 15, 2024 3:27 PM
• URL: https://library.dsu.edu/graduate-research

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard

• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Jul 11, 2024 6:38 AM
• URL: https://lib.guides.umd.edu/SR

• My Library Account
• Articles, Books & More
• Course Reserves
• Site Search
• Advanced Search
• Sac State Library
• Research Guides
• Writing a Literature Review
• 6 Stages to Writing a Literature Review
• What is a Literature Review?
• Literature Review Examples
• Organizing Your Literature Review
• Managing your Citations
• Further Reading on Lit Reviews

Academic Phrasebank

• Last Updated: Jun 6, 2024 9:36 AM
• URL: https://csus.libguides.com/litreview

• Systematic review
• Open access
• Published: 07 August 2024

Models and frameworks for assessing the implementation of clinical practice guidelines: a systematic review

• Nicole Freitas de Mello   ORCID: orcid.org/0000-0002-5228-6691 1 , 2 ,
• Sarah Nascimento Silva   ORCID: orcid.org/0000-0002-1087-9819 3 ,
• Dalila Fernandes Gomes   ORCID: orcid.org/0000-0002-2864-0806 1 , 2 ,
• Juliana da Motta Girardi   ORCID: orcid.org/0000-0002-7547-7722 4 &
• Jorge Otávio Maia Barreto   ORCID: orcid.org/0000-0002-7648-0472 2 , 4  

Implementation Science volume  19 , Article number:  59 ( 2024 ) Cite this article

499 Accesses

6 Altmetric

Metrics details

The implementation of clinical practice guidelines (CPGs) is a cyclical process in which the evaluation stage can facilitate continuous improvement. Implementation science has utilized theoretical approaches, such as models and frameworks, to understand and address this process. This article aims to provide a comprehensive overview of the models and frameworks used to assess the implementation of CPGs.

A systematic review was conducted following the Cochrane methodology, with adaptations to the "selection process" due to the unique nature of this review. The findings were reported following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) reporting guidelines. Electronic databases were searched from their inception until May 15, 2023. A predetermined strategy and manual searches were conducted to identify relevant documents from health institutions worldwide. Eligible studies presented models and frameworks for assessing the implementation of CPGs. Information on the characteristics of the documents, the context in which the models were used (specific objectives, level of use, type of health service, target group), and the characteristics of each model or framework (name, domain evaluated, and model limitations) were extracted. The domains of the models were analyzed according to the key constructs: strategies, context, outcomes, fidelity, adaptation, sustainability, process, and intervention. A subgroup analysis was performed grouping models and frameworks according to their levels of use (clinical, organizational, and policy) and type of health service (community, ambulatorial, hospital, institutional). The JBI’s critical appraisal tools were utilized by two independent researchers to assess the trustworthiness, relevance, and results of the included studies.

Database searches yielded 14,395 studies, of which 80 full texts were reviewed. Eight studies were included in the data analysis and four methodological guidelines were additionally included from the manual search. The risk of bias in the studies was considered non-critical for the results of this systematic review. A total of ten models/frameworks for assessing the implementation of CPGs were found. The level of use was mainly policy, the most common type of health service was institutional, and the major target group was professionals directly involved in clinical practice. The evaluated domains differed between the models and there were also differences in their conceptualization. All the models addressed the domain "Context", especially at the micro level (8/12), followed by the multilevel (7/12). The domains "Outcome" (9/12), "Intervention" (8/12), "Strategies" (7/12), and "Process" (5/12) were frequently addressed, while "Sustainability" was found only in one study, and "Fidelity/Adaptation" was not observed.

Conclusions

The use of models and frameworks for assessing the implementation of CPGs is still incipient. This systematic review may help stakeholders choose or adapt the most appropriate model or framework to assess CPGs implementation based on their specific health context.

Trial registration

PROSPERO (International Prospective Register of Systematic Reviews) registration number: CRD42022335884. Registered on June 7, 2022.

Peer Review reports

Contributions to the literature

Although the number of theoretical approaches has grown in recent years, there are still important gaps to be explored in the use of models and frameworks to assess the implementation of clinical practice guidelines (CPGs). This systematic review aims to contribute knowledge to overcome these gaps.

Despite the great advances in implementation science, evaluating the implementation of CPGs remains a challenge, and models and frameworks could support improvements in this field.

This study demonstrates that the available models and frameworks do not cover all characteristics and domains necessary for a complete evaluation of CPGs implementation.

The presented findings contribute to the field of implementation science, encouraging debate on choices and adaptations of models and frameworks for implementation research and evaluation.

Substantial investments have been made in clinical research and development in recent decades, increasing the medical knowledge base and the availability of health technologies [ 1 ]. The use of clinical practice guidelines (CPGs) has increased worldwide to guide best health practices and to maximize healthcare investments. A CPG can be defined as "any formal statements systematically developed to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" [ 2 ] and has the potential to improve patient care by promoting interventions of proven benefit and discouraging ineffective interventions. Furthermore, they can promote efficiency in resource allocation and provide support for managers and health professionals in decision-making [ 3 , 4 ].

However, having a quality CPG does not guarantee that the expected health benefits will be obtained. In fact, putting these devices to use still presents a challenge for most health services across distinct levels of government. In addition to the development of guidelines with high methodological rigor, those recommendations need to be available to their users; these recommendations involve the diffusion and dissemination stages, and they need to be used in clinical practice (implemented), which usually requires behavioral changes and appropriate resources and infrastructure. All these stages involve an iterative and complex process called implementation, which is defined as the process of putting new practices within a setting into use [ 5 , 6 ].

Implementation is a cyclical process, and the evaluation is one of its key stages, which allows continuous improvement of CPGs development and implementation strategies. It consists of verifying whether clinical practice is being performed as recommended (process evaluation or formative evaluation) and whether the expected results and impact are being reached (summative evaluation) [ 7 , 8 , 9 ]. Although the importance of the implementation evaluation stage has been recognized, research on how these guidelines are implemented is scarce [ 10 ]. This paper focused on the process of assessing CPGs implementation.

To understand and improve this complex process, implementation science provides a systematic set of principles and methods to integrate research findings and other evidence-based practices into routine practice and improve the quality and effectiveness of health services and care [ 11 ]. The field of implementation science uses theoretical approaches that have varying degrees of specificity based on the current state of knowledge and are structured based on theories, models, and frameworks [ 5 , 12 , 13 ]. A "Model" is defined as "a simplified depiction of a more complex world with relatively precise assumptions about cause and effect", and a "framework" is defined as "a broad set of constructs that organize concepts and data descriptively without specifying causal relationships" [ 9 ]. Although these concepts are distinct, in this paper, their use will be interchangeable, as they are typically like checklists of factors relevant to various aspects of implementation.

There are a variety of theoretical approaches available in implementation science [ 5 , 14 ], which can make choosing the most appropriate challenging [ 5 ]. Some models and frameworks have been categorized as "evaluation models" by providing a structure for evaluating implementation endeavors [ 15 ], even though theoretical approaches from other categories can also be applied for evaluation purposes because they specify concepts and constructs that may be operationalized and measured [ 13 ]. Two frameworks that can specify implementation aspects that should be evaluated as part of intervention studies are RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) [ 16 ] and PRECEDE-PROCEED (Predisposing, Reinforcing and Enabling Constructs in Educational Diagnosis and Evaluation-Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development) [ 17 ]. Although the number of theoretical approaches has grown in recent years, the use of models and frameworks to evaluate the implementation of guidelines still seems to be a challenge.

This article aims to provide a complete map of the models and frameworks applied to assess the implementation of CPGs. The aim is also to subside debate and choices on models and frameworks for the research and evaluation of the implementation processes of CPGs and thus to facilitate the continued development of the field of implementation as well as to contribute to healthcare policy and practice.

A systematic review was conducted following the Cochrane methodology [ 18 ], with adaptations to the "selection process" due to the unique nature of this review (details can be found in the respective section). The review protocol was registered in PROSPERO (registration number: CRD42022335884) on June 7, 2022. This report adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 19 ] and a completed checklist is provided in Additional File 1.

Eligibility criteria

The SDMO approach (Types of Studies, Types of Data, Types of Methods, Outcomes) [ 20 ] was utilized in this systematic review, outlined as follows:

Types of studies

All types of studies were considered for inclusion, as the assessment of CPG implementation can benefit from a diverse range of study designs, including randomized clinical trials/experimental studies, scale/tool development, systematic reviews, opinion pieces, qualitative studies, peer-reviewed articles, books, reports, and unpublished theses.

Studies were categorized based on their methodological designs, which guided the synthesis, risk of bias assessment, and presentation of results.

Study protocols and conference abstracts were excluded due to insufficient information for this review.

Types of data

Studies that evaluated the implementation of CPGs either independently or as part of a multifaceted intervention.

Guidelines for evaluating CPG implementation.

Inclusion of CPGs related to any context, clinical area, intervention, and patient characteristics.

No restrictions were placed on publication date or language.

Exclusion criteria

General guidelines were excluded, as this review focused on 'models for evaluating clinical practice guidelines implementation' rather than the guidelines themselves.

Studies that focused solely on implementation determinants as barriers and enablers were excluded, as this review aimed to explore comprehensive models/frameworks.

Studies evaluating programs and policies were excluded.

Studies that only assessed implementation strategies (isolated actions) rather than the implementation process itself were excluded.

Studies that focused solely on the impact or results of implementation (summative evaluation) were excluded.

Types of methods

Not applicable.

All potential models or frameworks for assessing the implementation of CPG (evaluation models/frameworks), as well as their characteristics: name; specific objectives; levels of use (clinical, organizational, and policy); health system (public, private, or both); type of health service (community, ambulatorial, hospital, institutional, homecare); domains or outcomes evaluated; type of recommendation evaluated; context; limitations of the model.

Model was defined as "a deliberated simplification of a phenomenon on a specific aspect" [ 21 ].

Framework was defined as "structure, overview outline, system, or plan consisting of various descriptive categories" [ 21 ].

Models or frameworks used solely for the CPG development, dissemination, or implementation phase.

Models/frameworks used solely for assessment processes other than implementation, such as for the development or dissemination phase.

Data sources and literature search

The systematic search was conducted on July 31, 2022 (and updated on May 15, 2023) in the following electronic databases: MEDLINE/PubMed, Centre for Reviews and Dissemination (CRD), the Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), EMBASE, Epistemonikos, Global Health, Health Systems Evidence, PDQ-Evidence, PsycINFO, Rx for Change (Canadian Agency for Drugs and Technologies in Health, CADTH), Scopus, Web of Science and Virtual Health Library (VHL). The Google Scholar database was used for the manual selection of studies (first 10 pages).

Additionally, hand searches were performed on the lists of references included in the systematic reviews and citations of the included studies, as well as on the websites of institutions working on CPGs development and implementation: Guidelines International Networks (GIN), National Institute for Health and Care Excellence (NICE; United Kingdom), World Health Organization (WHO), Centers for Disease Control and Prevention (CDC; USA), Institute of Medicine (IOM; USA), Australian Department of Health and Aged Care (ADH), Healthcare Improvement Scotland (SIGN), National Health and Medical Research Council (NHMRC; Australia), Queensland Health, The Joanna Briggs Institute (JBI), Ministry of Health and Social Policy of Spain, Ministry of Health of Brazil and Capes Theses and Dissertations Catalog.

The search strategy combined terms related to "clinical practice guidelines" (practice guidelines, practice guidelines as topic, clinical protocols), "implementation", "assessment" (assessment, evaluation), and "models, framework". The free term "monitoring" was not used because it was regularly related to clinical monitoring and not to implementation monitoring. The search strategies adapted for the electronic databases are presented in an additional file (see Additional file 2).

Study selection process

The results of the literature search from scientific databases, excluding the CRD database, were imported into Mendeley Reference Management software to remove duplicates. They were then transferred to the Rayyan platform ( https://rayyan.qcri.org ) [ 22 ] for the screening process. Initially, studies related to the "assessment of implementation of the CPG" were selected. The titles were first screened independently by two pairs of reviewers (first selection: four reviewers, NM, JB, SS, and JG; update: a pair of reviewers, NM and DG). The title screening was broad, including all potentially relevant studies on CPG and the implementation process. Following that, the abstracts were independently screened by the same group of reviewers. The abstract screening was more focused, specifically selecting studies that addressed CPG and the evaluation of the implementation process. In the next step, full-text articles were reviewed independently by a pair of reviewers (NM, DG) to identify those that explicitly presented "models" or "frameworks" for assessing the implementation of the CPG. Disagreements regarding the eligibility of studies were resolved through discussion and consensus, and by a third reviewer (JB) when necessary. One reviewer (NM) conducted manual searches, and the inclusion of documents was discussed with the other reviewers.

Risk of bias assessment of studies

The selected studies were independently classified and evaluated according to their methodological designs by two investigators (NM and JG). This review employed JBI’s critical appraisal tools to assess the trustworthiness, relevance and results of the included studies [ 23 ] and these tools are presented in additional files (see Additional file 3 and Additional file 4). Disagreements were resolved by consensus or consultation with the other reviewers. Methodological guidelines and noncomparative and before–after studies were not evaluated because JBI does not have specific tools for assessing these types of documents. Although the studies were assessed for quality, they were not excluded on this basis.

Data extraction

The data was independently extracted by two reviewers (NM, DG) using a Microsoft Excel spreadsheet. Discrepancies were discussed and resolved by consensus. The following information was extracted:

Document characteristics : author; year of publication; title; study design; instrument of evaluation; country; guideline context;

Usage context of the models : specific objectives; level of use (clinical, organizational, and policy); type of health service (community, ambulatorial, hospital, institutional); target group (guideline developers, clinicians; health professionals; health-policy decision-makers; health-care organizations; service managers);

Model and framework characteristics : name, domain evaluated, and model limitations.

The set of information to be extracted, shown in the systematic review protocol, was adjusted to improve the organization of the analysis.

The "level of use" refers to the scope of the model used. "Clinical" was considered when the evaluation focused on individual practices, "organizational" when practices were within a health service institution, and "policy" when the evaluation was more systemic and covered different health services or institutions.

The "type of health service" indicated the category of health service where the model/framework was used (or can be used) to assess the implementation of the CPG, related to the complexity of healthcare. "Community" is related to primary health care; "ambulatorial" is related to secondary health care; "hospital" is related to tertiary health care; and "institutional" represented models/frameworks not specific to a particular type of health service.

The "target group" included stakeholders related to the use of the model/framework for evaluating the implementation of the CPG, such as clinicians, health professionals, guideline developers, health policy-makers, health organizations, and service managers.

The category "health system" (public, private, or both) mentioned in the systematic review protocol was not found in the literature obtained and was removed as an extraction variable. Similarly, the variables "type of recommendation evaluated" and "context" were grouped because the same information was included in the "guideline context" section of the study.

Some selected documents presented models or frameworks recognized by the scientific field, including some that were validated. However, some studies adapted the model to this context. Therefore, the domain analysis covered all models or frameworks domains evaluated by (or suggested for evaluation by) the document analyzed.

Data analysis and synthesis

The results were tabulated using narrative synthesis with an aggregative approach, without meta-analysis, aiming to summarize the documents descriptively for the organization, description, interpretation and explanation of the study findings [ 24 , 25 ].

The model/framework domains evaluated in each document were studied according to Nilsen et al.’s constructs: "strategies", "context", "outcomes", "fidelity", "adaptation" and "sustainability". For this study, "strategies" were described as structured and planned initiatives used to enhance the implementation of clinical practice [ 26 ].

The definition of "context" varies in the literature. Despite that, this review considered it as the set of circumstances or factors surrounding a particular implementation effort, such as organizational support, financial resources, social relations and support, leadership, and organizational culture [ 26 , 27 ]. The domain "context" was subdivided according to the level of health care into "micro" (individual perspective), "meso" (organizational perspective), "macro" (systemic perspective), and "multiple" (when there is an issue involving more than one level of health care).

The "outcomes" domain was related to the results of the implementation process (unlike clinical outcomes) and was stratified according to the following constructs: acceptability, appropriateness, feasibility, adoption, cost, and penetration. All these concepts align with the definitions of Proctor et al. (2011), although we decided to separate "fidelity" and "sustainability" as independent domains similar to Nilsen [ 26 , 28 ].

"Fidelity" and "adaptation" were considered the same domain, as they are complementary pieces of the same issue. In this study, implementation fidelity refers to how closely guidelines are followed as intended by their developers or designers. On the other hand, adaptation involves making changes to the content or delivery of a guideline to better fit the needs of a specific context. The "sustainability" domain was defined as evaluations about the continuation or permanence over time of the CPG implementation.

Additionally, the domain "process" was utilized to address issues related to the implementation process itself, rather than focusing solely on the outcomes of the implementation process, as done by Wang et al. [ 14 ]. Furthermore, the "intervention" domain was introduced to distinguish aspects related to the CPG characteristics that can impact its implementation, such as the complexity of the recommendation.

A subgroup analysis was performed with models and frameworks categorized based on their levels of use (clinical, organizational, and policy) and the type of health service (community, ambulatorial, hospital, institutional) associated with the CPG. The goal is to assist stakeholders (politicians, clinicians, researchers, or others) in selecting the most suitable model for evaluating CPG implementation based on their specific health context.

Search results

Database searches yielded 26,011 studies, of which 107 full texts were reviewed. During the full-text review, 99 articles were excluded: 41 studies did not mention a model or framework for assessing the implementation of the CPG, 31 studies evaluated only implementation strategies (isolated actions) rather than the implementation process itself, and 27 articles were not related to the implementation assessment. Therefore, eight studies were included in the data analysis. The updated search did not reveal additional relevant studies. The main reason for study exclusion was that they did not use models or frameworks to assess CPG implementation. Additionally, four methodological guidelines were included from the manual search (Fig.  1 ).

PRISMA diagram. Acronyms: ADH—Australian Department of Health, CINAHL—Cumulative Index to Nursing and Allied Health Literature, CDC—Centers for Disease Control and Prevention, CRD—Centre for Reviews and Dissemination, GIN—Guidelines International Networks, HSE—Health Systems Evidence, IOM—Institute of Medicine, JBI—The Joanna Briggs Institute, MHB—Ministry of Health of Brazil, NICE—National Institute for Health and Care Excellence, NHMRC—National Health and Medical Research Council, MSPS – Ministerio de Sanidad Y Política Social (Spain), SIGN—Scottish Intercollegiate Guidelines Network, VHL – Virtual Health Library, WHO—World Health Organization. Legend: Reason A –The study evaluated only implementation strategies (isolated actions) rather than the implementation process itself. Reason B – The study did not mention a model or framework for assessing the implementation of the intervention. Reason C – The study was not related to the implementation assessment. Adapted from Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. https://doi.org/10.1136/bmj.n71 . For more information, visit:

According to the JBI’s critical appraisal tools, the overall assessment of the studies indicates their acceptance for the systematic review.

The cross-sectional studies lacked clear information regarding "confounding factors" or "strategies to address confounding factors". This was understandable given the nature of the study, where such details are not typically included. However, the reviewers did not find this lack of information to be critical, allowing the studies to be included in the review. The results of this methodological quality assessment can be found in an additional file (see Additional file 5).

In the qualitative studies, there was some ambiguity regarding the questions: "Is there a statement locating the researcher culturally or theoretically?" and "Is the influence of the researcher on the research, and vice versa, addressed?". However, the reviewers decided to include the studies and deemed the methodological quality sufficient for the analysis in this article, based on the other information analyzed. The results of this methodological quality assessment can be found in an additional file (see Additional file 6).

Documents characteristics (Table  1 )

The documents were directed to several continents: Australia/Oceania (4/12) [ 31 , 33 , 36 , 37 ], North America (4/12 [ 30 , 32 , 38 , 39 ], Europe (2/12 [ 29 , 35 ] and Asia (2/12) [ 34 , 40 ]. The types of documents were classified as cross-sectional studies (4/12) [ 29 , 32 , 34 , 38 ], methodological guidelines (4/12) [ 33 , 35 , 36 , 37 ], mixed methods studies (3/12) [ 30 , 31 , 39 ] or noncomparative studies (1/12) [ 40 ]. In terms of the instrument of evaluation, most of the documents used a survey/questionnaire (6/12) [ 29 , 30 , 31 , 32 , 34 , 38 ], while three (3/12) used qualitative instruments (interviews, group discussions) [ 30 , 31 , 39 ], one used a checklist [ 37 ], one used an audit [ 33 ] and three (3/12) did not define a specific instrument to measure [ 35 , 36 , 40 ].

Considering the clinical areas covered, most studies evaluated the implementation of nonspecific (general) clinical areas [ 29 , 33 , 35 , 36 , 37 , 40 ]. However, some studies focused on specific clinical contexts, such as mental health [ 32 , 38 ], oncology [ 39 ], fall prevention [ 31 ], spinal cord injury [ 30 ], and sexually transmitted infections [ 34 ].

Usage context of the models (Table  1 )

Specific objectives.

All the studies highlighted the purpose of guiding the process of evaluating the implementation of CPGs, even if they evaluated CPGs from generic or different clinical areas.

Levels of use

The most common level of use of the models/frameworks identified to assess the implementation of CPGs was policy (6/12) [ 33 , 35 , 36 , 37 , 39 , 40 ]. In this level, the model is used in a systematic way to evaluate all the processes involved in CPGs implementation and is primarily related to methodological guidelines. This was followed by the organizational level of use (5/12) [ 30 , 31 , 32 , 38 , 39 ], where the model is used to evaluate the implementation of CPGs in a specific institution, considering its specific environment. Finally, the clinical level of use (2/12) [ 29 , 34 ] focuses on individual practice and the factors that can influence the implementation of CPGs by professionals.

Type of health service

Institutional services were predominant (5/12) [ 33 , 35 , 36 , 37 , 40 ] and included methodological guidelines and a study of model development and validation. Hospitals were the second most common type of health service (4/12) [ 29 , 30 , 31 , 34 ], followed by ambulatorial (2/12) [ 32 , 34 ] and community health services (1/12) [ 32 ]. Two studies did not specify which type of health service the assessment addressed [ 38 , 39 ].

Target group

The focus of the target group was professionals directly involved in clinical practice (6/12) [ 29 , 31 , 32 , 34 , 38 , 40 ], namely, health professionals and clinicians. Other less related stakeholders included guideline developers (2/12) [ 39 , 40 ], health policy decision makers (1/12) [ 39 ], and healthcare organizations (1/12) [ 39 ]. The target group was not defined in the methodological guidelines, although all the mentioned stakeholders could be related to these documents.

Model and framework characteristics

Models and frameworks for assessing the implementation of cpgs.

The Consolidated Framework for Implementation Research (CFIR) [ 31 , 38 ] and the Promoting Action on Research Implementation in Health Systems (PARiHS) framework [ 29 , 30 ] were the most commonly employed frameworks within the selected documents. The other models mentioned were: Goal commitment and implementation of practice guidelines framework [ 32 ]; Guideline to identify key indicators [ 35 ]; Guideline implementation checklist [ 37 ]; Guideline implementation evaluation tool [ 40 ]; JBI Implementation Framework [ 33 ]; Reach, effectiveness, adoption, implementation and maintenance (RE-AIM) framework [ 34 ]; The Guideline Implementability Framework [ 39 ] and an unnamed model [ 36 ].

Domains evaluated

The number of domains evaluated (or suggested for evaluation) by the documents varied between three and five, with the majority focusing on three domains. All the models addressed the domain "context", with a particular emphasis on the micro level of the health care context (8/12) [ 29 , 31 , 34 , 35 , 36 , 37 , 38 , 39 ], followed by the multilevel (7/12) [ 29 , 31 , 32 , 33 , 38 , 39 , 40 ], meso level (4/12) [ 30 , 35 , 39 , 40 ] and macro level (2/12) [ 37 , 39 ]. The "Outcome" domain was evaluated in nine models. Within this domain, the most frequently evaluated subdomain was "adoption" (6/12) [ 29 , 32 , 34 , 35 , 36 , 37 ], followed by "acceptability" (4/12) [ 30 , 32 , 35 , 39 ], "appropriateness" (3/12) [ 32 , 34 , 36 ], "feasibility" (3/12) [ 29 , 32 , 36 ], "cost" (1/12) [ 35 ] and "penetration" (1/12) [ 34 ]. Regarding the other domains, "Intervention" (8/12) [ 29 , 31 , 34 , 35 , 36 , 38 , 39 , 40 ], "Strategies" (7/12) [ 29 , 30 , 33 , 35 , 36 , 37 , 40 ] and "Process" (5/12) [ 29 , 31 , 32 , 33 , 38 ] were frequently addressed in the models, while "Sustainability" (1/12) [ 34 ] was only found in one model, and "Fidelity/Adaptation" was not observed. The domains presented by the models and frameworks and evaluated in the documents are shown in Table  2 .

Limitations of the models

Only two documents mentioned limitations in the use of the model or frameworks. These two studies reported limitations in the use of CFIR: "is complex and cumbersome and requires tailoring of the key variables to the specific context", and "this framework should be supplemented with other important factors and local features to achieve a sound basis for the planning and realization of an ongoing project" [ 31 , 38 ]. Limitations in the use of other models or frameworks are not reported.

Subgroup analysis

Following the subgroup analysis (Table  3 ), five different models/frameworks were utilized at the policy level by institutional health services. These included the Guideline Implementation Evaluation Tool [ 40 ], the NHMRC tool (model name not defined) [ 36 ], the JBI Implementation Framework + GRiP [ 33 ], Guideline to identify key indicators [ 35 ], and the Guideline implementation checklist [ 37 ]. Additionally, the "Guideline Implementability Framework" [ 39 ] was implemented at the policy level without restrictions based on the type of health service. Regarding the organizational level, the models used varied depending on the type of service. The "Goal commitment and implementation of practice guidelines framework" [ 32 ] was applied in community and ambulatory health services, while "PARiHS" [ 29 , 30 ] and "CFIR" [ 31 , 38 ] were utilized in hospitals. In contexts where the type of health service was not defined, "CFIR" [ 31 , 38 ] and "The Guideline Implementability Framework" [ 39 ] were employed. Lastly, at the clinical level, "RE-AIM" [ 34 ] was utilized in ambulatory and hospital services, and PARiHS [ 29 , 30 ] was specifically used in hospital services.

Key findings

This systematic review identified 10 models/ frameworks used to assess the implementation of CPGs in various health system contexts. These documents shared similar objectives in utilizing models and frameworks for assessment. The primary level of use was policy, the most common type of health service was institutional, and the main target group of the documents was professionals directly involved in clinical practice. The models and frameworks presented varied analytical domains, with sometimes divergent concepts used in these domains. This study is innovative in its emphasis on the evaluation stage of CPG implementation and in summarizing aspects and domains aimed at the practical application of these models.

The small number of documents contrasts with studies that present an extensive range of models and frameworks available in implementation science. The findings suggest that the use of models and frameworks to evaluate the implementation of CPGs is still in its early stages. Among the selected documents, there was a predominance of cross-sectional studies and methodological guidelines, which strongly influenced how the implementation evaluation was conducted. This was primarily done through surveys/questionnaires, qualitative methods (interviews, group discussions), and non-specific measurement instruments. Regarding the subject areas evaluated, most studies focused on a general clinical area, while others explored different clinical areas. This suggests that the evaluation of CPG implementation has been carried out in various contexts.

The models were chosen independently of the categories proposed in the literature, with their usage categorized for purposes other than implementation evaluation, as is the case with CFIR and PARiHS. This practice was described by Nilsen et al. who suggested that models and frameworks from other categories can also be applied for evaluation purposes because they specify concepts and constructs that may be operationalized and measured [ 14 , 15 , 42 , 43 ].

The results highlight the increased use of models and frameworks in evaluation processes at the policy level and institutional environments, followed by the organizational level in hospital settings. This finding contradicts a review that reported the policy level as an area that was not as well studied [ 44 ]. The use of different models at the institutional level is also emphasized in the subgroup analysis. This may suggest that the greater the impact (social, financial/economic, and organizational) of implementing CPGs, the greater the interest and need to establish well-defined and robust processes. In this context, the evaluation stage stands out as crucial, and the investment of resources and efforts to structure this stage becomes even more advantageous [ 10 , 45 ]. Two studies (16,7%) evaluated the implementation of CPGs at the individual level (clinical level). These studies stand out for their potential to analyze variations in clinical practice in greater depth.

In contrast to the level of use and type of health service most strongly indicated in the documents, with systemic approaches, the target group most observed was professionals directly involved in clinical practice. This suggests an emphasis on evaluating individual behaviors. This same emphasis is observed in the analysis of the models, in which there is a predominance of evaluating the micro level of the health context and the "adoption" subdomain, in contrast with the sub-use of domains such as "cost" and "process". Cassetti et al. observed the same phenomenon in their review, in which studies evaluating the implementation of CPGs mainly adopted a behavioral change approach to tackle those issues, without considering the influence of wider social determinants of health [ 10 ]. However, the literature widely reiterates that multiple factors impact the implementation of CPGs, and different actions are required to make them effective [ 6 , 46 , 47 ]. As a result, there is enormous potential for the development and adaptation of models and frameworks aimed at more systemic evaluation processes that consider institutional and organizational aspects.

In analyzing the model domains, most models focused on evaluating only some aspects of implementation (three domains). All models evaluated the "context", highlighting its significant influence on implementation [ 9 , 26 ]. Context is an essential effect modifier for providing research evidence to guide decisions on implementation strategies [ 48 ]. Contextualizing a guideline involves integrating research or other evidence into a specific circumstance [ 49 ]. The analysis of this domain was adjusted to include all possible contextual aspects, even if they were initially allocated to other domains. Some contextual aspects presented by the models vary in comprehensiveness, such as the assessment of the "timing and nature of stakeholder engagement" [ 39 ], which includes individual engagement by healthcare professionals and organizational involvement in CPG implementation. While the importance of context is universally recognized, its conceptualization and interpretation differ across studies and models. This divergence is also evident in other domains, consistent with existing literature [ 14 ]. Efforts to address this conceptual divergence in implementation science are ongoing, but further research and development are needed in this field [ 26 ].

The main subdomain evaluated was "adoption" within the outcome domain. This may be attributed to the ease of accessing information on the adoption of the CPG, whether through computerized system records, patient records, or self-reports from healthcare professionals or patients themselves. The "acceptability" subdomain pertains to the perception among implementation stakeholders that a particular CPG is agreeable, palatable or satisfactory. On the other hand, "appropriateness" encompasses the perceived fit, relevance or compatibility of the CPG for a specific practice setting, provider, or consumer, or its perceived fit to address a particular issue or problem [ 26 ]. Both subdomains are subjective and rely on stakeholders' interpretations and perceptions of the issue being analyzed, making them susceptible to reporting biases. Moreover, obtaining this information requires direct consultation with stakeholders, which can be challenging for some evaluation processes, particularly in institutional contexts.

The evaluation of the subdomains "feasibility" (the extent to which a CPG can be successfully used or carried out within a given agency or setting), "cost" (the cost impact of an implementation effort), and "penetration" (the extent to which an intervention or treatment is integrated within a service setting and its subsystems) [ 26 ] was rarely observed in the documents. This may be related to the greater complexity of obtaining information on these aspects, as they involve cross-cutting and multifactorial issues. In other words, it would be difficult to gather this information during evaluations with health practitioners as the target group. This highlights the need for evaluation processes of CPGs implementation involving multiple stakeholders, even if the evaluation is adjusted for each of these groups.

Although the models do not establish the "intervention" domain, we thought it pertinent in this study to delimit the issues that are intrinsic to CPGs, such as methodological quality or clarity in establishing recommendations. These issues were quite common in the models evaluated but were considered in other domains (e.g., in "context"). Studies have reported the importance of evaluating these issues intrinsic to CPGs [ 47 , 50 ] and their influence on the implementation process [ 51 ].

The models explicitly present the "strategies" domain, and its evaluation was usually included in the assessments. This is likely due to the expansion of scientific and practical studies in implementation science that involve theoretical approaches to the development and application of interventions to improve the implementation of evidence-based practices. However, these interventions themselves are not guaranteed to be effective, as reported in a previous review that showed unclear results indicating that the strategies had affected successful implementation [ 52 ]. Furthermore, model domains end up not covering all the complexity surrounding the strategies and their development and implementation process. For example, the ‘Guideline implementation evaluation tool’ evaluates whether guideline developers have designed and provided auxiliary tools to promote the implementation of guidelines [ 40 ], but this does not mean that these tools would work as expected.

The "process" domain was identified in the CFIR [ 31 , 38 ], JBI/GRiP [ 33 ], and PARiHS [ 29 ] frameworks. While it may be included in other domains of analysis, its distinct separation is crucial for defining operational issues when assessing the implementation process, such as determining if and how the use of the mentioned CPG was evaluated [ 3 ]. Despite its presence in multiple models, there is still limited detail in the evaluation guidelines, which makes it difficult to operationalize the concept. Further research is needed to better define the "process" domain and its connections and boundaries with other domains.

The domain of "sustainability" was only observed in the RE-AIM framework, which is categorized as an evaluation framework [ 34 ]. In its acronym, the letter M stands for "maintenance" and corresponds to the assessment of whether the user maintains use, typically longer than 6 months. The presence of this domain highlights the need for continuous evaluation of CPGs implementation in the short, medium, and long term. Although the RE-AIM framework includes this domain, it was not used in the questionnaire developed in the study. One probable reason is that the evaluation of CPGs implementation is still conducted on a one-off basis and not as a continuous improvement process. Considering that changes in clinical practices are inherent over time, evaluating and monitoring changes throughout the duration of the CPG could be an important strategy for ensuring its implementation. This is an emerging field that requires additional investment and research.

The "Fidelity/Adaptation" domain was not observed in the models. These emerging concepts involve the extent to which a CPG is being conducted exactly as planned or whether it is undergoing adjustments and adaptations. Whether or not there is fidelity or adaptation in the implementation of CPGs does not presuppose greater or lesser effectiveness; after all, some adaptations may be necessary to implement general CPGs in specific contexts. The absence of this domain in all the models and frameworks may suggest that they are not relevant aspects for evaluating implementation or that there is a lack of knowledge of these complex concepts. This may suggest difficulty in expressing concepts in specific evaluative questions. However, further studies are warranted to determine the comprehensiveness of these concepts.

It is important to note the customization of the domains of analysis, with some domains presented in the models not being evaluated in the studies, while others were complementarily included. This can be seen in Jeong et al. [ 34 ], where the "intervention" domain in the evaluation with the RE-AIM framework reinforced the aim of theoretical approaches such as guiding the process and not determining norms. Despite this, few limitations were reported for the models, suggesting that the use of models in these studies reflects the application of these models to defined contexts without a deep critical analysis of their domains.

Limitations

This review has several limitations. First, only a few studies and methodological guidelines that explicitly present models and frameworks for assessing the implementation of CPGs have been found. This means that few alternative models could be analyzed and presented in this review. Second, this review adopted multiple analytical categories (e.g., level of use, health service, target group, and domains evaluated), whose terminology has varied enormously in the studies and documents selected, especially for the "domains evaluated" category. This difficulty in harmonizing the taxonomy used in the area has already been reported [ 26 ] and has significant potential to confuse. For this reason, studies and initiatives are needed to align understandings between concepts and, as far as possible, standardize them. Third, in some studies/documents, the information extracted was not clear about the analytical category. This required an in-depth interpretative process of the studies, which was conducted in pairs to avoid inappropriate interpretations.

Implications

This study contributes to the literature and clinical practice management by describing models and frameworks specifically used to assess the implementation of CPGs based on their level of use, type of health service, target group related to the CPG, and the evaluated domains. While there are existing reviews on the theories, frameworks, and models used in implementation science, this review addresses aspects not previously covered in the literature. This valuable information can assist stakeholders (such as politicians, clinicians, researchers, etc.) in selecting or adapting the most appropriate model to assess CPG implementation based on their health context. Furthermore, this study is expected to guide future research on developing or adapting models to assess the implementation of CPGs in various contexts.

The use of models and frameworks to evaluate the implementation remains a challenge. Studies should clearly state the level of model use, the type of health service evaluated, and the target group. The domains evaluated in these models may need adaptation to specific contexts. Nevertheless, utilizing models to assess CPGs implementation is crucial as they can guide a more thorough and systematic evaluation process, aiding in the continuous improvement of CPGs implementation. The findings of this systematic review offer valuable insights for stakeholders in selecting or adjusting models and frameworks for CPGs evaluation, supporting future theoretical advancements and research.

Availability of data and materials

Abbreviations.

Australian Department of Health and Aged Care

Canadian Agency for Drugs and Technologies in Health

Centers for Disease Control and

Consolidated Framework for Implementation Research

Cumulative Index to Nursing and Allied Health Literature

Clinical practice guideline

Centre for Reviews and Dissemination

Guidelines International Networks

Getting Research into Practice

Health Systems Evidence

Institute of Medicine

The Joanna Briggs Institute

Ministry of Health of Brazil

Ministerio de Sanidad y Política Social

National Health and Medical Research Council

National Institute for Health and Care Excellence

Promoting action on research implementation in health systems framework

Predisposing, Reinforcing and Enabling Constructs in Educational Diagnosis and Evaluation-Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

International Prospective Register of Systematic Reviews

Reach, effectiveness, adoption, implementation, and maintenance framework

Healthcare Improvement Scotland

United States of America

Virtual Health Library

World Health Organization

Medicine I of. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001. Available from: http://www.nap.edu/catalog/10027 . Cited 2022 Sep 29.

Field MJ, Lohr KN. Clinical Practice Guidelines: Directions for a New Program. Washington DC: National Academy Press. 1990. Available from: https://www.nap.edu/read/1626/chapter/8 Cited 2020 Sep 2.

Dawson A, Henriksen B, Cortvriend P. Guideline Implementation in Standardized Office Workflows and Exam Types. J Prim Care Community Heal. 2019;10. Available from: https://pubmed.ncbi.nlm.nih.gov/30900500/ . Cited 2020 Jul 15.

Unverzagt S, Oemler M, Braun K, Klement A. Strategies for guideline implementation in primary care focusing on patients with cardiovascular disease: a systematic review. Fam Pract. 2014;31(3):247–66. Available from: https://academic.oup.com/fampra/article/31/3/247/608680 . Cited 2020 Nov 5.

Article   PubMed   Google Scholar  

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):1–13. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0242-0 . Cited 2022 May 1.

Article   Google Scholar  

Mangana F, Massaquoi LD, Moudachirou R, Harrison R, Kaluangila T, Mucinya G, et al. Impact of the implementation of new guidelines on the management of patients with HIV infection at an advanced HIV clinic in Kinshasa, Democratic Republic of Congo (DRC). BMC Infect Dis. 2020;20(1):N.PAG-N.PAG. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=146325052&amp .

Browman GP, Levine MN, Mohide EA, Hayward RSA, Pritchard KI, Gafni A, et al. The practice guidelines development cycle: a conceptual tool for practice guidelines development and implementation. 2016;13(2):502–12. https://doi.org/10.1200/JCO.1995.13.2.502 .

Killeen SL, Donnellan N, O’Reilly SL, Hanson MA, Rosser ML, Medina VP, et al. Using FIGO Nutrition Checklist counselling in pregnancy: A review to support healthcare professionals. Int J Gynecol Obstet. 2023;160(S1):10–21. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85146194829&doi=10.1002%2Fijgo.14539&partnerID=40&md5=d0f14e1f6d77d53e719986e6f434498f .

Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3(1):1–12. Available from: https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-015-0089-9 . Cited 2020 Nov 5.

Cassetti V, M VLR, Pola-Garcia M, AM G, J JPC, L APDT, et al. An integrative review of the implementation of public health guidelines. Prev Med reports. 2022;29:101867. Available from: http://www.epistemonikos.org/documents/7ad499d8f0eecb964fc1e2c86b11450cbe792a39 .

Eccles MP, Mittman BS. Welcome to implementation science. Implementation Science BioMed Central. 2006. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-1-1 .

Damschroder LJ. Clarity out of chaos: Use of theory in implementation research. Psychiatry Res. 2020;1(283):112461.

Handley MA, Gorukanti A, Cattamanchi A. Strategies for implementing implementation science: a methodological overview. Emerg Med J. 2016;33(9):660–4. Available from: https://pubmed.ncbi.nlm.nih.gov/26893401/ . Cited 2022 Mar 7.

Wang Y, Wong ELY, Nilsen P, Chung VC ho, Tian Y, Yeoh EK. A scoping review of implementation science theories, models, and frameworks — an appraisal of purpose, characteristics, usability, applicability, and testability. Implement Sci. 2023;18(1):1–15. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-023-01296-x . Cited 2024 Jan 22.

Moullin JC, Dickson KS, Stadnick NA, Albers B, Nilsen P, Broder-Fingert S, et al. Ten recommendations for using implementation frameworks in research and practice. Implement Sci Commun. 2020;1(1):1–12. Available from: https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-020-00023-7 . Cited 2022 May 20.

Glasgow RE, Vogt TM, Boles SM. *Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322. Available from: /pmc/articles/PMC1508772/?report=abstract. Cited 2022 May 22.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Asada Y, Lin S, Siegel L, Kong A. Facilitators and Barriers to Implementation and Sustainability of Nutrition and Physical Activity Interventions in Early Childcare Settings: a Systematic Review. Prev Sci. 2023;24(1):64–83. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85139519721&doi=10.1007%2Fs11121-022-01436-7&partnerID=40&md5=b3c395fdd2b8235182eee518542ebf2b .

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. version 6. Cochrane; 2022. Available from: https://training.cochrane.org/handbook. Cited 2022 May 23.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372. Available from: https://www.bmj.com/content/372/bmj.n71 . Cited 2021 Nov 18.

M C, AD O, E P, JP H, S G. Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Higgins JP, Green S, eds Cochrane Handb Syst Rev Interv. 2011;Version 5.

Kislov R, Pope C, Martin GP, Wilson PM. Harnessing the power of theorising in implementation science. Implement Sci. 2019;14(1):1–8. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-019-0957-4 . Cited 2024 Jan 22.

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):1–10. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0384-4 . Cited 2022 May 20.

JBI. JBI’s Tools Assess Trust, Relevance & Results of Published Papers: Enhancing Evidence Synthesis. Available from: https://jbi.global/critical-appraisal-tools . Cited 2023 Jun 13.

Drisko JW. Qualitative research synthesis: An appreciative and critical introduction. Qual Soc Work. 2020;19(4):736–53.

Pope C, Mays N, Popay J. Synthesising qualitative and quantitative health evidence: A guide to methods. 2007. Available from: https://books.google.com.br/books?hl=pt-PT&lr=&id=L3fbE6oio8kC&oi=fnd&pg=PR6&dq=synthesizing+qualitative+and+quantitative+health+evidence&ots=sfELNUoZGq&sig=bQt5wt7sPKkf7hwKUvxq2Ek-p2Q#v=onepage&q=synthesizing=qualitative=and=quantitative=health=evidence& . Cited 2022 May 22.

Nilsen P, Birken SA, Edward Elgar Publishing. Handbook on implementation science. 542. Available from: https://www.e-elgar.com/shop/gbp/handbook-on-implementation-science-9781788975988.html . Cited 2023 Apr 15.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):1–15. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-4-50 . Cited 2023 Jun 13.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. Available from: https://pubmed.ncbi.nlm.nih.gov/20957426/ . Cited 2023 Jun 11.

Bahtsevani C, Willman A, Khalaf A, Östman M, Ostman M. Developing an instrument for evaluating implementation of clinical practice guidelines: a test-retest study. J Eval Clin Pract. 2008;14(5):839–46. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=105569473&amp . Cited 2023 Jan 18.

Balbale SN, Hill JN, Guihan M, Hogan TP, Cameron KA, Goldstein B, et al. Evaluating implementation of methicillin-resistant Staphylococcus aureus (MRSA) prevention guidelines in spinal cord injury centers using the PARIHS framework: a mixed methods study. Implement Sci. 2015;10(1):130. Available from: https://pubmed.ncbi.nlm.nih.gov/26353798/ . Cited 2023 Apr 3.

Article   PubMed   PubMed Central   Google Scholar  

Breimaier HE, Heckemann B, Halfens RJGG, Lohrmann C. The Consolidated Framework for Implementation Research (CFIR): a useful theoretical framework for guiding and evaluating a guideline implementation process in a hospital-based nursing practice. BMC Nurs. 2015;14(1):43. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=109221169&amp . Cited 2023 Apr 3.

Chou AF, Vaughn TE, McCoy KD, Doebbeling BN. Implementation of evidence-based practices: Applying a goal commitment framework. Health Care Manage Rev. 2011;36(1):4–17. Available from: https://pubmed.ncbi.nlm.nih.gov/21157225/ . Cited 2023 Apr 30.

Porritt K, McArthur A, Lockwood C, Munn Z. JBI Manual for Evidence Implementation. JBI Handbook for Evidence Implementation. JBI; 2020. Available from: https://jbi-global-wiki.refined.site/space/JHEI . Cited 2023 Apr 3.

Jeong HJJ, Jo HSS, Oh MKK, Oh HWW. Applying the RE-AIM Framework to Evaluate the Dissemination and Implementation of Clinical Practice Guidelines for Sexually Transmitted Infections. J Korean Med Sci. 2015;30(7):847–52. Available from: https://pubmed.ncbi.nlm.nih.gov/26130944/ . Cited 2023 Apr 3.

GPC G de trabajo sobre implementación de. Implementación de Guías de Práctica Clínica en el Sistema Nacional de Salud. Manual Metodológico. 2009. Available from: https://portal.guiasalud.es/wp-content/uploads/2019/01/manual_implementacion.pdf . Cited 2023 Apr 3.

Australia C of. A guide to the development, implementation and evaluation of clinical practice guidelines. National Health and Medical Research Council; 1998. Available from: https://www.health.qld.gov.au/__data/assets/pdf_file/0029/143696/nhmrc_clinprgde.pdf .

Health Q. Guideline implementation checklist Translating evidence into best clinical practice. 2022.

Google Scholar  

Quittner AL, Abbott J, Hussain S, Ong T, Uluer A, Hempstead S, et al. Integration of mental health screening and treatment into cystic fibrosis clinics: Evaluation of initial implementation in 84 programs across the United States. Pediatr Pulmonol. 2020;55(11):2995–3004. Available from: https://www.embase.com/search/results?subaction=viewrecord&id=L2005630887&from=export . Cited 2023 Apr 3.

Urquhart R, Woodside H, Kendell C, Porter GA. Examining the implementation of clinical practice guidelines for the management of adult cancers: A mixed methods study. J Eval Clin Pract. 2019;25(4):656–63. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=137375535&amp . Cited 2023 Apr 3.

Yinghui J, Zhihui Z, Canran H, Flute Y, Yunyun W, Siyu Y, et al. Development and validation for evaluation of an evaluation tool for guideline implementation. Chinese J Evidence-Based Med. 2022;22(1):111–9. Available from: https://www.embase.com/search/results?subaction=viewrecord&id=L2016924877&from=export .

Breimaier HE, Halfens RJG, Lohrmann C. Effectiveness of multifaceted and tailored strategies to implement a fall-prevention guideline into acute care nursing practice: a before-and-after, mixed-method study using a participatory action research approach. BMC Nurs. 2015;14(1):18. Available from: https://search.ebscohost.com/login.aspx?direct=true&db=c8h&AN=103220991&amp .

Lai J, Maher L, Li C, Zhou C, Alelayan H, Fu J, et al. Translation and cross-cultural adaptation of the National Health Service Sustainability Model to the Chinese healthcare context. BMC Nurs. 2023;22(1). Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85153237164&doi=10.1186%2Fs12912-023-01293-x&partnerID=40&md5=0857c3163d25ce85e01363fc3a668654 .

Zhao J, Li X, Yan L, Yu Y, Hu J, Li SA, et al. The use of theories, frameworks, or models in knowledge translation studies in healthcare settings in China: a scoping review protocol. Syst Rev. 2021;10(1):13. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7792291 .

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50. Available from: https://pubmed.ncbi.nlm.nih.gov/22898128/ . Cited 2023 Apr 4.

Phulkerd S, Lawrence M, Vandevijvere S, Sacks G, Worsley A, Tangcharoensathien V. A review of methods and tools to assess the implementation of government policies to create healthy food environments for preventing obesity and diet-related non-communicable diseases. Implement Sci. 2016;11(1):1–13. Available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-016-0379-5 . Cited 2022 May 1.

Buss PM, Pellegrini FA. A Saúde e seus Determinantes Sociais. PHYSIS Rev Saúde Coletiva. 2007;17(1):77–93.

Pereira VC, Silva SN, Carvalho VKSS, Zanghelini F, Barreto JOMM. Strategies for the implementation of clinical practice guidelines in public health: an overview of systematic reviews. Heal Res Policy Syst. 2022;20(1):13. Available from: https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-022-00815-4 . Cited 2022 Feb 21.

Grimshaw J, Eccles M, Tetroe J. Implementing clinical guidelines: current evidence and future implications. J Contin Educ Health Prof. 2004;24 Suppl 1:S31-7. Available from: https://pubmed.ncbi.nlm.nih.gov/15712775/ . Cited 2021 Nov 9.

Lotfi T, Stevens A, Akl EA, Falavigna M, Kredo T, Mathew JL, et al. Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines. J Clin Epidemiol. 2021;135:182–6. Available from: https://pubmed.ncbi.nlm.nih.gov/33836255/ . Cited 2024 Jan 25.

Lenzer J. Why we can’t trust clinical guidelines. BMJ. 2013;346(7913). Available from: https://pubmed.ncbi.nlm.nih.gov/23771225/ . Cited 2024 Jan 25.

Molino C de GRC, Ribeiro E, Romano-Lieber NS, Stein AT, de Melo DO. Methodological quality and transparency of clinical practice guidelines for the pharmacological treatment of non-communicable diseases using the AGREE II instrument: A systematic review protocol. Syst Rev. 2017;6(1):1–6. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-017-0621-5 . Cited 2024 Jan 25.

Albers B, Mildon R, Lyon AR, Shlonsky A. Implementation frameworks in child, youth and family services – Results from a scoping review. Child Youth Serv Rev. 2017;1(81):101–16.

Download references

Acknowledgements

Not applicable

This study is supported by the Fundação de Apoio à Pesquisa do Distrito Federal (FAPDF). FAPDF Award Term (TOA) nº 44/2024—FAPDF/SUCTI/COOBE (SEI/GDF – Process 00193–00000404/2024–22). The content in this article is solely the responsibility of the authors and does not necessarily represent the official views of the FAPDF.

Author information

Authors and affiliations.

Department of Management and Incorporation of Health Technologies, Ministry of Health of Brazil, Brasília, Federal District, 70058-900, Brazil

Nicole Freitas de Mello & Dalila Fernandes Gomes

Postgraduate Program in Public Health, FS, University of Brasília (UnB), Brasília, Federal District, 70910-900, Brazil

Nicole Freitas de Mello, Dalila Fernandes Gomes & Jorge Otávio Maia Barreto

René Rachou Institute, Oswaldo Cruz Foundation, Belo Horizonte, Minas Gerais, 30190-002, Brazil

Sarah Nascimento Silva

Oswaldo Cruz Foundation - Brasília, Brasília, Federal District, 70904-130, Brazil

Juliana da Motta Girardi & Jorge Otávio Maia Barreto

You can also search for this author in PubMed   Google Scholar

Contributions

NFM and JOMB conceived the idea and the protocol for this study. NFM conducted the literature search. NFM, SNS, JMG and JOMB conducted the data collection with advice and consensus gathering from JOMB. The NFM and JMG assessed the quality of the studies. NFM and DFG conducted the data extraction. NFM performed the analysis and synthesis of the results with advice and consensus gathering from JOMB. NFM drafted the manuscript. JOMB critically revised the first version of the manuscript. All the authors revised and approved the submitted version.

Corresponding author

Correspondence to Nicole Freitas de Mello .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

13012_2024_1389_moesm1_esm.docx.

Additional file 1: PRISMA checklist. Description of data: Completed PRISMA checklist used for reporting the results of this systematic review.

Additional file 2: Literature search. Description of data: The search strategies adapted for the electronic databases.

13012_2024_1389_moesm3_esm.doc.

Additional file 3: JBI’s critical appraisal tools for cross-sectional studies. Description of data: JBI’s critical appraisal tools to assess the trustworthiness, relevance, and results of the included studies. This is specific for cross-sectional studies.

13012_2024_1389_MOESM4_ESM.doc

Additional file 4: JBI’s critical appraisal tools for qualitative studies. Description of data: JBI’s critical appraisal tools to assess the trustworthiness, relevance, and results of the included studies. This is specific for qualitative studies.

13012_2024_1389_MOESM5_ESM.doc

Additional file 5: Methodological quality assessment results for cross-sectional studies. Description of data: Methodological quality assessment results for cross-sectional studies using JBI’s critical appraisal tools.

13012_2024_1389_MOESM6_ESM.doc

Additional file 6: Methodological quality assessment results for the qualitative studies. Description of data: Methodological quality assessment results for qualitative studies using JBI’s critical appraisal tools.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Freitas de Mello, N., Nascimento Silva, S., Gomes, D.F. et al. Models and frameworks for assessing the implementation of clinical practice guidelines: a systematic review. Implementation Sci 19 , 59 (2024). https://doi.org/10.1186/s13012-024-01389-1

Download citation

Received : 06 February 2024

Accepted : 01 August 2024

Published : 07 August 2024

DOI : https://doi.org/10.1186/s13012-024-01389-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation
• Practice guideline
• Evidence-Based Practice
• Implementation science

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 07 August 2024

Relationship between patient safety culture and patient experience in hospital settings: a scoping review

• Adel Alabdaly   ORCID: orcid.org/0000-0003-0914-5225 1 , 2 ,
• Reece Hinchcliff   ORCID: orcid.org/0000-0001-9920-4211 3 , 4 ,
• Deborah Debono   ORCID: orcid.org/0000-0003-2095-156X 5 &
• Su-Yin Hor   ORCID: orcid.org/0000-0002-6498-9722 5  

BMC Health Services Research volume  24 , Article number:  906 ( 2024 ) Cite this article

108 Accesses

Metrics details

Measures of patient safety culture and patient experience are both commonly utilised to evaluate the quality of healthcare services, including hospitals, but the relationship between these two domains remains uncertain. In this study, we aimed to explore and synthesise published literature regarding the relationships between these topics in hospital settings.

This study was performed using the five stages of Arksey and O’Malley’s Framework, refined by the Joanna Briggs Institute. Searches were conducted in the CINAHL, Cochrane Library, ProQuest, MEDLINE, PsycINFO, SciELO and Scopus databases. Further online search on the websites of pertinent organisations in Australia and globally was conducted. Data were extracted against predetermined criteria.

4512 studies were initially identified; 15 studies met the inclusion criteria. Several positive statistical relationships between patient safety culture and patient experience domains were identified. Communication and teamwork were the most influential factors in the relationship between patient safety culture and patient experience. Managers and clinicians had a positive view of safety and a positive relationship with patient experience, but this was not the case when managers alone held such views. Qualitative methods offered further insights into patient safety culture from patients’ and families’ perspectives.

The findings indicate that the patient can recognise safety-related issues that the hospital team may miss. However, studies mostly measured staff perspectives on patient safety culture and did not always include patient experiences of patient safety culture. Further, the relationship between patient safety culture and patient experience is generally identified as a statistical relationship, using quantitative methods. Further research assessing patient safety culture alongside patient experience is essential for providing a more comprehensive picture of safety. This will help to uncover issues and other factors that may have an indirect effect on patient safety culture and patient experience.

Peer Review reports

Introduction

Patient safety is a pressing challenge for health systems, globally. The importance of promoting and sustaining a robust safety culture is widely recognised [ 1 ]. The importance of the patient’s role in supporting patient safety is also increasingly recognised [ 2 ]. Despite the prominence of the concepts of patient safety culture and patient experience in academia and industry, the relationship between them remains underexplored and diffuse.

The concept of patient safety culture was defined as a collective of beliefs, attitudes, values, and norms that influence behaviours and attitudes, concerning patient safety [ 3 ]. Patient perspectives are often neglected when measuring safety culture [ 4 ]. Patient experience has been defined as patients’ perspectives of services, recognising that patients are the most valuable sources of information about their experiences [ 5 ].

It is essential to put the patient at the centre of healthcare services [ 6 ], and to do this requires nurturing caring cultures through the assurance that health professionals feel esteemed, involved and supported [ 7 ]. Patients pay attention to staff performance and other issues and can identify safety problems that hospital staff may miss, such as problems entering and exiting the healthcare system, systemic (multiple and distributed) problems that are cumulative, and errors of omission, especially the failure to attend to patients’ concerns [ 2 , 8 , 9 , 10 ]. A cultural change from the conventional approach that considered patients as care recipients, to seeing patients as partners in their care, is essential to provide patient-centred care that is informed by patient experience.

There has been considerable knowledge gained about patient safety, but it persists as a worldwide challenge in healthcare [ 11 ], with serious incidents and iatrogenic harm continuing to occur across health care settings, including within hospital settings. There has been a focus on reducing iatrogenic harm by enhancing safety culture in hospitals.

Understanding patient safety from the staff perspective alone is not enough. It is essential to also understand what factors might link safety culture and patient experience, as concepts often measured separately, but both important indicators of safety and quality. In examining this link, we hope to better understand what facets of care might contribute to both safety culture, as experienced by staff, and the safety and quality of care, as experienced by patients. The aim of this review is to explore and synthesise existing research literature to find out what is known regarding the relationship between patient safety culture and patient experience (of safety and quality) in hospital settings. We sought to achieve this aim through the following objectives: (a) to identify how these concepts have been defined or described in the literature; (b) to identify how these concepts are measured; and (c) to identify the links between the concepts.

This study followed a published protocol [ 12 ]. The methodology of this scoping review was developed using the Arksey and O’Malley [ 13 ] framework for a scoping review (Arksey & O’Malley, 2005), refined by the Joanna Briggs Institute [ 14 ]. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMAScR) [ 15 ] guidelines were followed. The study does not critically appraise the included papers’ quality and risk of bias. The aim in our scoping review is not to evaluate the quality of the evidence found, but rather to explore what research has been done in this field, and what approaches were undertaken.

The processes of searching, applying inclusion and exclusion criteria, screening, data extraction, and reporting of the findings followed a published protocol for this study [ 12 ]. The search terms and strategies appear in the protocol, and searches were completed on 18 June 2022.

The inclusion and exclusion criteria

This review followed the Population, Concept and Context (PCC) framework for the inclusion criteria recommended by the Joanna Briggs Institute for scoping reviews [ 14 ]. In addition to the PCC criteria noted in Table  1 , included studies must have been conducted in the hospital context and reported in English or Arabic languages.

We searched journals from seven electronic databases relevant to the scope of the study (CINAHL, Cochrane Library, ProQuest, MEDLINE, PsycINFO, SciELO and Scopus); web search engine Google Scholar (first 30 results); and four organisations in Australia and globally: the Agency for Healthcare Research and Quality (AHRQ), the Australian Commission for Safety and Quality in Healthcare (ACSQHC), the Agency for Clinical Innovation (ACI), and National Institutes of Health (NIH). We supplemented these searches with hand-searching the reference lists of the final included papers for additional studies of relevance.

Study selection

As indicated in the protocol for this study [ 12 ], retrieved papers were screened and selected in two phases. In the first phase, one reviewer (AA) evaluated all titles and abstracts to determine whether each paper met the eligibility criteria, including categorising screened studies as ‘included’, ‘excluded’ or ‘not sure’. All papers screened as ‘included’ and ‘not sure’ in the first phase were considered for full-text review by the reviewer (AA). In the second phase, three reviewers (RH, DD, SH) screened ten per cent of titles and abstracts of studies screened as ‘included’, ‘excluded’ or ‘not sure’ against selection criteria. All authors (AA, RH, DD, SH) independently reviewed the full text of the included studies. The authors discussed the included papers in a meeting and reached a consensus on the included papers, with no disagreement between the authors.

Charting the data

One reviewer (AA) extracted relevant data from the included studies to address the scoping review question using the template provided in the published protocol [ 12 ]. Three reviewers (RH, DD and SH) verified the accuracy of the data extraction exercise. The data extracted included the following:

Aims/objective(s).

Methodology/methods.

Inclusion/exclusion criteria (e.g., PCC).

Types of intervention (if applicable).

Measurement of outcomes (if applicable).

Key results that relate to the review question.

Reporting the findings

Other concepts related to patient safety culture and patient experience, such as safety climate and patient satisfaction, were used in literature that measured safety culture or patient experience. The nuances of these terms were illustrated in the published protocol. The decision was taken to incorporate findings about safety climate alongside those about patient safety culture, and to incorporate findings about both patient satisfaction and patient experience. We noticed that the ‘patient experience’ and ‘patient satisfaction’ terms are often used interchangeably. For example, a study conducted by Mazurenko et al. [ 16 ] used the term ‘patient satisfaction’ in the paper title but measured patient satisfaction using the HCAHPS tool, which is a well-known tool for measuring ‘patient experience’. In fact, the terms, as operationalised in the instruments, overlap more than they should.

According to Bull [ 17 ], ‘patient satisfaction’ involves an evaluation and hence is subjective, suggesting that ‘patient experience’ is the more objective measure. However, considering the questions in the HCAHPS tool (commonly used for measuring ‘patient experience’ as mentioned above), we see that several questions involve an element of subjectivity and evaluation from the patient’s perspective. For instance, questions like: “During this hospital stay, how often did nurses treat you with courtesy and respect?” or “How often did you get help in getting to the bathroom or in using a bedpan as soon as you wanted?”. The point made by Bull [ 17 ] reflects a tension between the recognised importance of finding out what care is like, from patients’ perspectives (which is subjective and evaluative), and the desire for objective measurements of care delivery for the purposes of comparison and evaluation of health services [ 18 ]. Due to these concepts being so intertwined in how they are understood and measured, and not wanting to limit the understanding of the patient experience only to objective measures devoid of patients’ subjective judgements, papers on patient satisfaction from the review were included, based on the inclusion criteria.

The study sought to review a wide range of literature in relation to the study aim and inclusion criteria. Rather than being a systematic review or meta-analysis, the study aims to offer the reader an overview of the research carried out regarding the relationship between safety culture and patient experience. The characteristics and findings of the included papers were analysed initially by (AA), performing a content analysis, using a framework of categories aligned with the research questions. Within these categories, study features and findings were discussed among all the authors (AA, RH, DD, SH), and descriptively summarised. All authors agreed upon the findings and categories. This descriptive content analysis was found to be sufficient to address the study objectives. Thus, deviating from the published protocol [ 12 ], no further thematic analysis was conducted. The results are presented according to the categories as follows:

Conceptualisations of patient safety culture and patient experience.

Measurement of patient safety culture and patient experience.

Relationship between patient safety culture and patient experience.

As depicted in Fig.  1 , the initial search yielded 4512 articles. After removing duplicates, 3833 articles remained, and 3793 were excluded at the first stage of screening (title and abstract). Following full-text screening, 15 articles remained that met the inclusion criteria. The included studies were conducted in different countries, including Australia (one study) [ 19 ], Canada (two studies) [ 8 , 20 ], Germany (one study) [ 4 ], Indonesia (one study) [ 21 ], Iran (one study) [ 22 ], Israel (two studies) [ 10 , 23 ], Nigeria (one study) [ 24 ], United Kingdom (one study) [ 2 ] and United States (five studies) [ 16 , 25 , 26 , 27 , 28 ]. A summary of the characteristics of the included studies is presented in Table  2 .

PRISMA flowchart of search process and results

Conceptualisations of patient safety culture and patient experience

Patient safety culture.

In the studies reviewed, patient safety culture was commonly conceptualised as relating to the attitudes, beliefs, perceptions, norms and values that workers share about safety [ 8 , 10 , 24 , 27 ]. These shared characteristics shape healthcare professionals’ understandings of what is essential in a healthcare institution, how they should act, what attitudes or actions are acceptable, and what approaches are rewarded or punished concerning patient safety [ 8 , 10 , 27 ]. Patient safety culture has been identified within the included studies as being central to the behaviour of the individuals, and influences staff proficiency, attitudes and behaviours concerning their safety performance [ 8 , 10 , 27 ].

The reviewed literature also identified patient safety culture as one element of a broader organisational culture, related to preventing and detecting shortfalls in patient safety, and managing patient safety in healthcare settings [ 16 , 20 , 21 ]. The concept of ‘safety climate’ was also prevalent in the literature, and was often used in studies that also described ‘safety culture’ [ 10 , 16 , 19 , 26 , 27 ] without distinguishing between the two concepts.

• Patient experience

From our review of the studies, the concept of patient satisfaction was more commonly used than patient experience, and defined as a subjective assessment of the ways those receiving healthcare react to particular relevant elements of treatment, including the process, environment, and outcomes, and this was quantified as representing the degree to which patients believe that their requirements and aspirations were fulfilled by their experiences [ 24 , 26 ]. Although the research that examined patient experience, did not offer specific definitions of the concept, patient experience was conceptualised as a resource for understanding patients’ perceptions, which helps promote the quality and safety of healthcare services [ 2 , 8 , 25 , 27 , 28 ].

The reviewed research frequently refered to the concept of patient satisfaction and ways of measuring it, regarding patient satisfaction as indicative of the effectiveness of organisational performance with regard to patient safety [ 2 , 8 , 25 , 26 , 27 ]. Review of the included studies identified another related concept, customer satisfaction, which is defined as how the individual feels when making a comparison between what they expected and how they regarded what they received; this is regarded as a high-performance target for the delivery of public services [ 21 ]. The variation in the concepts also reflected variation in the measurement tools currently used.

Measuring patient safety culture and patient experience

In the research reviewed, patient safety culture was most commonly measured by the deployment of questionnaires. Included studies also presented assessments of the validity of deployed instruments. The most common patient safety culture tool used in the reviewed studies was the Hospital Survey on Patient Safety Culture (HSOPS) [ 2 , 16 , 20 , 22 , 24 , 25 , 27 , 28 ]. The next most common tool used was the Safety Attitudes Questionnaire (SAQ) [ 19 , 26 ]. The SAQ was also combined with the Leadership Effectiveness Survey (LES) to construct a new tool named the Safety Culture and Leadership Questionnaire to assess clinician perceptions of safety, teamwork and leadership [ 19 ].

The HSOPS tool developed by the Agency of Healthcare Research and Quality was employed in included studies to assess clinician and staff perceptions of the culture of safety at the hospital’s macro level [ 16 , 22 , 27 , 28 ]. HSOPS is also used in individual departments within a hospital [ 2 , 20 , 24 , 25 ], and regarded as a reliable and valid tool. The SAQ is another reliable and valid tool employed for the evaluation of patient safety culture [ 26 ]. The safety culture domains in HSOPS and SAQ tools are different but overlapping (Table  3 ).

The use of HSOPS and SAQ tools reflected the overlap in use of the concepts of safety culture and safety climate. For example, HSOPS includes more dimensions of patient safety culture than the SAQ, and both tools were employed to measure ‘patient safety culture’ [ 2 , 16 , 20 , 21 , 24 , 25 , 26 , 27 , 28 ], although the HSOPS was also employed for the measurement of ‘safety climate’ [ 16 ]. In addition, the SAQ includes two dimensions referring to climate: teamwork climate and safety climate [ 29 ]. Importantly however, both the HSOPS and SAQ offer a quantitative measure of patient safety culture from the point of view of staff alone [ 2 , 16 , 20 , 24 , 25 , 26 , 27 , 28 ].

Patient-reported measures of safety were limited and mentioned more frequently in more recent literature. The Patient Measure of Safety (PMOS), Patients’ Perceptions of Safety Culture (PaPSC) and narratives were used in the research reviewed to identify safety concerns from the patient’s perspective and provide data regarding safety matters, including patient safety culture [ 2 , 4 , 8 , 19 ]. Lawton et al. [ 2 ] noted that the PMOS has undergone considerable testing and is generally recognised as having both validity and reliability; it is also popular with patients and allows researchers to assess how patients perceive the ways in which organisational elements influence patient safety within a hospital by collecting patient feedback about contributing factors to safety incidents [ 2 ].

With regard to measuring patient experience, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) was the most frequently used tool in studies reviewed, and is regarded as a valid and reliable instrument for measuring the ways in which patients perceive their interactions with the hospital, and can be used by government as a tool for assessing hospital funding [ 16 , 25 , 26 , 28 ]. HCAHPS (also referred to as Hospital CAHPS) asks the patient to report on their recent experiences with inpatient care [ 16 , 25 , 26 , 28 ]. The HCAHPS tool measures the following domains: nurse communication, doctor communication, pain management, staff responsiveness, hospital environment, communication about medicine, discharge information, and overall patient perception [ 16 , 25 , 26 , 28 ]. Similarly to the overlapping concepts described with the safety culture surveys earlier, the HCAHPS has been employed for the measurement of both patient satisfaction [ 16 , 26 ] and patient experience [ 25 , 28 ]. Other feedback tools such as the Patient Satisfaction Questionnaire Short Form (PSQ) [ 24 ], the Friends and Family Test (FFT) [ 2 ] and Family Satisfaction in the Intensive Care Unit questionnaire (FS-ICU-24) [ 20 ] were used for measuring patient feedback and perception of care in our reviewed studies.

Finally, only one study in our review used a qualitative method to examine patient experience; drawing on pre-recorded video narratives published on the Canadian Patient Safety Institute website [ 2 ].

Relationship between patient safety culture and patient experience

In the research reviewed, the relationship between patient safety culture and patient experience was generally identified and presented as a statistical correlation [ 2 , 16 , 24 , 25 , 26 , 27 , 28 ]. Positive correlations were found between some domains of patient safety culture and patient experience (Table  4 ) [ 2 , 8 , 20 , 21 , 23 , 25 , 28 ]. The teamwork and communication domains seem to be central to positive correlations between patient safety culture and patient experience [ 8 , 16 , 25 , 26 , 27 ]. Other studies reviewed demonstrated no correlation between patient safety culture and patient experience overall scores [ 2 , 24 , 26 ].

Staff responsibilities, including direct contact with patients, may affect the relationship between patient safety culture and patient experience. For instance, no significant correlation was found between patient satisfaction and safety climate when management alone had a highly positive view of the safety climate [ 16 ]. However, when management and clinicians both had a positive view of the safety climate, there was a positive correlation. The FFT tool that measured patient experience was correlated with the ways patients perceived safety issues but was not correlated with either the staff safety culture or publicly available safety data [ 2 ]. From the sole qualitative study, we find that structuring safety and quality based on teamwork among healthcare professionals, patients, and family members is a more effective approach than relying on the individual healthcare practitioner alone [ 8 ]. Also, patients’ and families’ involvement is essential for creating a trusting relationship, which helps create an inviting environment that facilitates and encourages open communication and coordination among staff and patients [ 8 ]. Finally, conversation between staff, patients and families is crucial to capture different views of safety and better understand safety culture, particularly from the patient’s perspective.

The research under review also frequently examined how patient safety culture and patient experience, either individually or in combination, were related to other quality measures such as hospital performance, however this is outside of the scope of our review.

Patient safety culture and patient experience overlapped with other concepts

The concepts “safety culture” and “safety climate” were used interchangeably in the reviewed literature, which reflects their overlap in the broader literature, although these concepts are also sometimes differentiated. Patient safety culture tends to refer more broadly to the complex set of shared perceptions about safety that form over time in an organisation, while safety climate is considered ‘a snapshot’ of these shared perceptions, that can be measured at a specific time point using survey studies [ 29 , 30 ].

In the reviewed studies, the use of the terms patient experience and patient satisfaction also significantly overlapped. The two terms are recognised quality indicators for assessing healthcare quality, and while both concepts are related, they have also been differentiated [ 31 ]. Although the reviewed studies did not offer specific definitions, patient experience has been described elsewhere as patient “perceptions of phenomena for which they are the best or only sources of information, such as personal comfort or effectiveness of discharge planning” [5 p1]. While patient experience is viewed as the sum of all interactions that influence patient perceptions over the entire experience [ 32 ], as noted earlier, patient satisfaction is more about whether patients’ expectations are met [ 33 ]. In this regard, patient satisfaction is viewed as evaluating the patient experience of health services. Therefore, patients’ perception of what they actually experienced in healthcare organisations (patient experience) has an influential impact on how they evaluate healthcare services (patient satisfaction).

Measuring the relationship between patient safety culture and patient experience

The relationship identified between patient safety culture and patient experience in the reviewed literature is mostly measured by quantitative approaches/surveys, and thus little is known about causality or the underlying reasons (or mechanisms) for any relationship identified between these concepts. The availability, validity and reliability of the surveys such as HSOPS and HCAHPS may facilitate and encourage the use of questionnaires in busy working environments such as hospitals. However, the significant differences and variations in methodologies/tools (including dimensions captured by the instruments) employed to measure safety culture and patient experience, makes it difficult to compare the different items of research, and results in variations in the findings.

Patient involvement in the measurement of patient safety culture

Our review findings support research arguing that patients can provide useful feedback on safety [ 34 ]. Patient voice is increasingly included in other aspects of patient safety, but we need to include it more in the measurement of safety culture. In fact, some measures of patient experience pay attention to safety, for instance, in terms of physical comfort and a safe environment, which are also domains of patient safety culture. It was recognised in the included studies that instruments for assessing patient perceptions could be adapted to incorporate questions regarding patient safety, such as PMOS and PaPSC. This would enable patient perceptions and experience of safety to be assessed and the findings employed to effect enhancements in safety culture.

The PMOS and PaPSC scales were developed specifically to capture patients’ feedback on the safety of their care. The PMOS is based on the Yorkshire Contributory Factors Framework (YCFF) to capture patient feedback regarding the contributing factors to patient safety incidents [ 35 ]. However, the YCFF was developed based on input from healthcare professionals alone [ 36 ]. Likewise, the PaPSC scale was also initially developed based on staff perceptions. Although these scales are administered to patients, they may not fully reflect the patients’ perceptions of safety culture, if patients identify other aspects. In addition, the PMOS data was collected from one hospital in northern England; as such, the outcomes of the survey are not reflective of the perceptions of the general global population.

Another measurement approach for capturing patient perceptions of safety culture is to consider patients’ and families’ pre-recorded narratives as a qualitative assessment method [ 8 ]. This approach was limited in terms of inability to ask questions or follow-up with the participants, and the analysis was based on a revised or edited perspective that could carry certain biases. However, this study demonstrated the value of patient narratives and interviews in understanding the interrelationships between different aspects of patient safety culture. In contrast to surveys, qualitative interviews aim to understand participants’ attitudes, behaviours, experiences and perceptions. Qualitative research methods are common in healthcare research, but are largely missing in research into the association between safety culture and patient perceptions of safety culture.

No consensus exists as to the best method to be employed for the measurement of the concepts in question. Different measurements have been employed for each concept for various purposes, resulting in variations in data sources, and variations in results. Consequently, to create useful and usable data, there is a need to adopt measurement methods that are reliable, comparable and valid, for examining the relationship between patient safety culture and patient experience, such as the HSOPS and HCAHPS. It is also useful to consider qualitative investigation when exploring the relationship between these concepts.

Several relationships between patient experience and safety culture subdomains were identified in the included studies (Table  4 ). This suggests that staff and patient views on aspects of safety can be usefully incorporated and examined together. For example, the communication between staff and patients, and the coordination within and across hospital departments. According to Doyle, Lennox, and Bell [ 37 ], the smooth coordination (integration) of care is a key and valued aspect of the patient experience.

In this review, we found that the conceptual relationship between patient safety culture and the patient experience was not clearly described. The differences and overlaps between concepts, results, or measurement tools makes it difficult to understand the relationship between patient safety culture (among health professionals and managers) and patient experience. Future investigations may benefit from the development of a conceptual framework that allows researchers to test and develop their understandings of how patients’ experiences intersect with safety culture. We know that patient experience and safety culture are both valuable quality indicators. Better understanding how they are associated will enable healthcare staff to comprehend patient needs and create an effective strategy for enhancing patient safety culture that aligns with patients’ needs.

This scoping review has offered an overview of extant research regarding the association between patient experience and patient safety culture within the hospital context, and identified potential associations between the two concepts. However, the included studies have been conducted in limited countries, and generally assessed the relationship between these two concepts using quantitative methods. It may be the case that in other countries or cultures, the type of relationship could vary. Differences in ethnicity and national cultures could play an important role in patient experience. For instance, it was recognised in the reviewed literature, that Arab patients reported lower patient satisfaction levels compared with other ethnic groups within the same setting [ 10 ]. Therefore, it is important to consider other elements that may have an indirect effect on patient safety culture and patient experience, particularly in ethnic or national cultures where this relationship has not yet been investigated. Likewise, other factors related to the organisation could impact the relationship between the concepts. For example, the accreditation status of a facility has been shown to have a significant positive relationship with patient satisfaction [ 21 ].

It has been demonstrated that the terms “safety culture” and “safety climate,” as well as “patient experience” and “patient satisfaction” are not always consistently applied across research, with the concepts not often being clearly defined, lacking a theoretical basis for the relationship, not being widely investigated with qualitative methodologies and with considerable diversity in terms of the tools and methodologies employed. The outcomes of this review suggest that research into the association between patient safety culture and patient experience needs to be investigated by using a suitable theoretical framework, in combination with validated methods, and supported by qualitative inquiry, in order to investigate this relationship more comprehensively, particularly in contexts where such investigations have not taken place.

Limitations

While the literature search was conducted in major electronic databases without restrictions on date of publication or country of origin, additional relevant resources not in English or Arabic languages are likely to have been missed. This may lead to a language bias and limit the chance of capturing different perspectives from diverse communities to obtain a comprehensive understanding of the research phenomena, impacting the findings’ generalisability. Further, in accordance with the scoping review methodology of Arksey and O’Malley, a quality assessment was not conducted. Thus, it would be challenging to determine the validity of the reported findings due to the lack of quality assessment. These limitations are common in scoping reviews.

Data availability

Not applicable.

Abbreviations

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews

Population, Concept and Context

The Agency for Healthcare Research and Quality

The Australian Commission for Safety and Quality in Healthcare

The Agency for Clinical Innovation

National Institutes of Health

The Hospital Survey on Patient Safety Culture

The Safety Attitudes Questionnaire

The Patient Measure of Safety

The Hospital Consumer Assessment of Healthcare Providers and Systems

The Patient Satisfaction Questionnaire Short Form

The Friends and Family Test

Family Satisfaction in the Intensive Care Unit questionnaire

The Yorkshire Contributory Factors Framework

Donaldson MS, Corrigan JM, Kohn LT. To err is human: building a safer health system. National Academies; 2000.

Lawton R, O’Hara JK, Sheard L, Reynolds C, Cocks K, Armitage G, et al. Can staff and patient perspectives on hospital safety predict harm-free care? An analysis of staff and patient survey data and routinely collected outcomes. BMJ Qual Saf. 2015;24:369–76. https://doi.org/10.1136/bmjqs-2014-003691 .

Article   PubMed   PubMed Central   Google Scholar  

Zohar D, Livne Y, Tenne-Gazit O, Admi H, Donchin Y. Healthcare climate: a framework for measuring and improving patient safety. Crit Care Med. 2007;35(5):1312–7.

Article   PubMed   Google Scholar  

Monaca C, Bestmann B, Kattein M, Langner D, Müller H, Manser T. Assessing patients’ perceptions of safety culture in the hospital setting: development and initial evaluation of the patients’ perceptions of safety culture scale. J Patient Saf. 2020;16:90–7. https://doi.org/10.1097/pts.0000000000000436 .

Hagerty TA, Samuels W, Norcini-Pala A, Gigliotti E. Peplau’s theory of interpersonal relations: an alternate factor structure for patient experience data? Nurs Sci Quaterly. 2017;30(2):160–7.

Article   Google Scholar  

Dixon-Woods M, Baker R, Charles K, Dawson J, Jerzembek G, Martin G, McCarthy I, McKee L, Minion J, Ozieranski P, et al. Culture and behaviour in the English National Health Service: overview of lessons from a large multimethod study. BMJ Qual Saf. 2013;23:106–15.

Zimlichman E, Rozenblum R, Millenson ML. The road to patient experience of care measurement: lessons from the United States. Isr J Health Policy Res. 2013;2:35. https://doi.org/10.1186/2045-4015-2-35 .

Bishop AC, Cregan BR. Patient safety culture: finding meaning in patient experiences. Int J Health Care Qual Assur. 2015;28:595–610. https://doi.org/10.1108/IJHCQA-03-2014-0029 .

Gillespie A, Reader TW. Patient-centered insights: using health care complaints to reveal hot spots and blind spots in quality and safety. Milbank Q. 2018;96:530–67. https://doi.org/10.1111/1468-0009.12338 .

Kagan I, Porat N, Barnoy S. The quality and safety culture in general hospitals: patients’, physicians’ and nurses’ evaluation of its effect on patient satisfaction. Int J Qual Health Care. 2019;31:261–8. https://doi.org/10.1093/intqhc/mzy138 .

Ayorinde MO, Alabi PI. Perception and contributing factors to medication administration errors among nurses in Nigeria. Int J Afr Nurs Sci. 2019;11:100153–60.

Google Scholar  

Alabdaly A, Debono D, Hinchcliff R, Hor SY. Relationship between patient safety culture and patient experience in hospital settings: a scoping review protocol. BMJ Open. 2021;11:e049873. https://doi.org/10.1136/bmjopen-2021-049873c .

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32. https://doi.org/10.1080/1364557032000119616 .

Joanna Briggs Institute. Joanna Briggs Institute reviewers’ manual. 2015. https://nursing.lsuhsc.edu/JBI/docs/ReviewersManuals/Scoping-.pdf . Accessed 18 Jun 2022.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73. https://doi.org/10.7326/M18-0850 .

Mazurenko O, Richter J, Kazley AS, Ford E. Examination of the relationship between management and clinician perception of patient safety climate and patient satisfaction. Health Care Manage Rev. 2019;44(1):79–89. https://doi.org/10.1097/HMR.0000000000000156 .

Bull C. Patient satisfaction and patient experience are not interchangeable concepts. Int J Qual Health Care. 2021;33(1). https://doi.org/10.1093/intqhc/mzab023 .

Lord L, Gale N. Subjective experience or objective process: understanding the gap between values and practice for involving patients in designing patient-centred care. J Health Organ Manag. 2014;28(6):714–30. https://doi.org/10.1108/JHOM-08-2013-0160 .

Do VQ, Mitchell R, Clay-Williams R, Taylor N, Ting HP, Arnolda G, Braithwaite J. Safety climate, leadership and patient views associated with hip fracture care quality and clinician perceptions of hip fracture care performance. Int J Qual Health Care. 2021;33:mzab152. https://doi.org/10.1093/intqhc/mzab152 .

Dodek PM, Wong H, Heyland DK, Cook DJ, Rocker GM, Kutsogiannis DJ, et al. The relationship between organizational culture and family satisfaction in critical care. Crit Care Med. 2012;40(5):1506–12. https://doi.org/10.1097/CCM.0b013e318241e368 .

Sembodo T, Hadi C, Purnomo W. Service quality model with cultural perspective in effect opatient satisfaction in hospitals with different accreditation status. Medico-Legal Update. 2019;19:204–9. https://doi.org/10.5958/0974-1283.2019.00041.0 .

Afshar PJ, Karbasi BJ, Moghadam MN. The relationship between patient safety culture with patient satisfaction and hospital performance in Shafa Hospital of Kerman in 2020. J Educ Health Promotion. 2021;10:455.

Burlakov N, Rozani V, Bluvstein I, Kagan I. The Association between quality and safety climate of a hospital ward, family members’ empowerment, and satisfaction with provided care. J Nurs Scholarsh. 2021;53:727–36. https://doi.org/10.1111/jnu.12682 .

Okafor CH, Ugwu AC, Okon IE. Effects of patient safety culture on patient satisfaction with radiological services in Nigerian radiodiagnostic practice. J Patient Experience. 2018;5:267–71. https://doi.org/10.1177/2374373518755500 .

Abrahamson K, Hass Z, Morgan K, Fulton B, Ramanujam R. The relationship between nurse-reported safety culture and the patient experience. J Nurs Adm. 2016;46:662–8. https://doi.org/10.1097/NNA.0000000000000423 .

Lyu H, Wick EC, Housman M, Freischlag JA, Makary MA. Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg. 2013;148:362–7. https://doi.org/10.1001/2013.jamasurg.270 .

Smith SA, Yount N, Sorra J. Exploring relationships between hospital patient safety culture and consumer reports safety scores. BMC Health Serv Res. 2017;17:143. https://doi.org/10.1186/s12913-017-2078-6 .

Sorra J, Khanna K, Dyer N, Mardon R, Famolaro T. Exploring relationships between patient safety culture and patients’ assessments of hospital care. J Patient Saf. 2012;8:131–9. https://doi.org/10.1097/PTS.0b013e318258ca46 .

Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, Roberts PR, Thomas EJ. The safety attitudes questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44.

Weaver SJ, Lubomksi LH, Wilson RF, Pfoh ER, Martinez KA, Dy SM. Promoting a culture of safety as a patient safety strategy. Ann Intern Med. 2013;158(5Part2):369–74. https://doi.org/10.7326/0003-4819-158-5-201303051-00002 .

Kumah E. Patient experience and satisfaction with a healthcare system: connecting the dots. Int J Healthc Manag. 2019;12:173–9. https://doi.org/10.1080/20479700.2017.1353776 .

The Beryl Institute, Defining. patient experience. 2018. https://www.theberylinstitute.org/page/DefiningPatientExp . Accessed 18 Jun 2022.

The Agency for Healthcare Research and Quality. What Is Patient Experience?. 2016. https://www.ahrq.gov/cahps/about-cahps/patient-experience/index.html . Accessed 18 Jun 2022.

Hor SY, Godbold N, Collier A, Iedema R. Finding the patient in patient safety. Health. 2013;17:567–83. https://doi.org/10.1177/1363459312472082 .

Giles SJ, Lawton RJ, Din I, McEachan RR. Developing a patient measure of safety (PMOS). BMJ Qual Saf Sci. 2013;22:554–62.

Lawton R, McEachan RR, Giles SJ, Sirriyeh R, Watt IS, Wright J. Development of an evidence-based framework of factors contributing to patient safety incidents in hospital settings: a systematic review. BMJ Qual Saf. 2012;21:369–80.

Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:1–18. https://doi.org/10.1136/bmjopen-2012-001570 .

Download references

Acknowledgements

The authors wish to acknowledge the librarians at the University of Technology Sydney for providing support in developing the search strategy for this study. The authors acknowledge the Gadigal of the Eora Nation, the traditional custodians of the land on which this study was conducted, and pay our respects to the Elders both past and present.

The first author is funded for a PhD scholarship from Imam Abdulrahman Bin Faisal University, Saudi Arabia.

Author information

Authors and affiliations.

Faculty of Health, University of Technology Sydney, Sydney, NSW, Australia

Adel Alabdaly

College of Nursing, Imam Abdulrahman Bin Faisal University, Dammam, Eastern Province, Saudi Arabia

School of Applied Psychology, Griffith Health Group, Griffith University, Brisbane, QLD, Australia

Reece Hinchcliff

School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, Brisbane, QLD, Australia

School of Public Health, University of Technology Sydney, Sydney, NSW, Australia

Deborah Debono & Su-Yin Hor

You can also search for this author in PubMed   Google Scholar

Contributions

A.A conceived and wrote the original manuscript. R.H, D.D and S.H reviewed and edited the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Adel Alabdaly .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Alabdaly, A., Hinchcliff, R., Debono, D. et al. Relationship between patient safety culture and patient experience in hospital settings: a scoping review. BMC Health Serv Res 24 , 906 (2024). https://doi.org/10.1186/s12913-024-11329-w

Download citation

Received : 24 August 2023

Accepted : 18 July 2024

Published : 07 August 2024

DOI : https://doi.org/10.1186/s12913-024-11329-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Safety culture
• Safety climate
• Patient satisfaction
• Customer satisfaction
• Healthcare quality
• Health services
• Quality indicators
• Patient safety

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Loading metrics

Open Access

Peer-reviewed

Research Article

Proportion of active tuberculosis among HIV-infected children after antiretroviral therapy in Ethiopia: A systematic review and meta-analysis

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected] , [email protected]

Affiliation College of Health Sciences, Debre Markos University, Debre Markos, Ethiopia

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization

Roles Resources, Software, Supervision, Validation, Visualization, Writing – review & editing

Affiliation College of Health Sciences, Debre Berhan University, Debre Berhan, Ethiopia

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliation College of Social Science Bahir Dare University, Bahir Dare, Ethiopia

• Fassikaw Kebede Bizuneh, 
• Dejen Tsegaye, 
• Belete Negese Gemeda, 
• Tsehay Kebede Bizuneh

• Published: August 2, 2024
• https://doi.org/10.1371/journal.pgph.0003528
• Peer Review
• Reader Comments

Despite effectiveness of antiretroviral therapy in reducing mortality of opportunistic infections among HIV infected children, however tuberculosis (TB) remains a significant cause for morbidity and attributed for one in every three deaths. HIV-infected children face disproportionate death risk during co-infection of TB due to their young age and miniatures immunity makes them more vulnerable. In Ethiopia, there is lack of aggregated data TB and HIV mortality in HIV infected children. We conducted an extensive systematic review of literature using Preferred Reporting of Systematic Review and Meta-Analysis (PRISMA) guideline. Five electronic databases were used mainly Scopus, PubMed, Medline, Web of Science, and Google scholar for articles searching. The pooled proportion of TB was estimated using a weighted inverse variance random-effects meta-regression using STATA version-17. Heterogeneity of the articles was evaluated using Cochran’s Q test and I2 statistic. Subgroup analysis, sensitivity test, and Egger’s regression were conducted for publication bias. This met-analysis is registered in Prospero-CRD42024502038. In the final met-analysis report, 13 out of 1221 articles were included and presented. During screening of 6668 HIV-infected children for active TB occurrence, 834 cases were reported after ART was initiated. The pooled proportion of active TB among HIV infected children was found 12.07% (95% CI: 10.71–13.41). In subgroup analysis, the Oromia region had 15.6% (95%CI: 10.2–20.6) TB burden, followed by southern Ethiopia 12.8% (95%CI: 10.03–15.67). During meta-regression, missed isoniazid Preventive therapy (IPT) (OR: 2.28), missed contrimoxazole preventive therapy (OR: 4.26), WHO stage III&IV (OR: 2.27), and level of Hgb ≤ 10gm/dl (OR = 3.11.7) were predictors for active TB. The systematic review found a higher proportion of active TB in HIV-infected children in Ethiopia compared to estimated rates in end TB strategy. To prevent premature death during co-infection, implement effective TB screening and cases tracing strategies in each follow up is needed.

Citation: Kebede Bizuneh F, Tsegaye D, Negese Gemeda B, Kebede Bizuneh T (2024) Proportion of active tuberculosis among HIV-infected children after antiretroviral therapy in Ethiopia: A systematic review and meta-analysis. PLOS Glob Public Health 4(8): e0003528. https://doi.org/10.1371/journal.pgph.0003528

Editor: Sanghyuk S. Shin, University of California Irvine, UNITED STATES OF AMERICA

Received: November 28, 2023; Accepted: July 3, 2024; Published: August 2, 2024

Copyright: © 2024 Kebede Bizuneh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: TB, uberculosis; WHO, World Health Organization; FMOH, Federal Ministry of Health; HIV, human immune deficiency virus; HAAR, highly active antiretroviral therapy; IPT, isoniazid preventive Therapy; CPT, co-trimoxazole preventive therapy

Introduction

People living with the Human Immune deficiency virus (PLHIV) are more susceptible to tuberculosis (TB), which is a leading cause of mortality [ 1 , 2 ]. There is a strong synergy between HIV infection and TB, while PLHIV is at high risk of dying from TB and HIV infection is the biggest risk factor for active TB incidence through declining cellular immunity and increased endogenous reactivation of latent TB bacilli in the lungs [ 3 , 4 ]. HIV infected children are at increased risk of acquiring active TB. HIV-infected persons are sixteen times more likely to be co-infected by TB disease as compared to HIV-negative person [ 5 ].

Tuberculosis continued to be the leading cause of morbidity and mortality for people living with HIV (PLHIV) worldwide [ 6 ]. Globally, in 2022, an estimated 1.3 million children (aged 0–14 years) were diagnosed with TB, accounting for approximately 12% of the total TB cases of 10.6 million [ 5 , 7 ]. The co-infection of HIV and TB is particularly dangerous, with around 214,000 children dying from TB disease in 2022 where 31,000 of those were attributed to children TB and HIV infections [ 5 ]. The burden of TB infection varies significantly across each continents, African and Southeast Asian regions attributed for 81% of global TB deaths in 2022 [ 8 ]. In Sub-Saharan African countries, 10% to 15% of HIV-infected children suffer from the dual burden of HIV and TB, with a lifetime risk of 21% and two-thirds of cases remain undiagnosed [ 5 , 9 – 11 ].

By the end of 2022, only 46% of children (aged 0–14 years) who were receiving antiretroviral therapy (ART) were able to achieve viral load suppression, which is a crucial factor in reducing the occurrence of new opportunistic infections [ 5 , 8 , 11 ]. However, TB infection remains responsible for one in every third deaths of HIV infected children in source-limited setting [ 7 , 12 , 13 ]. Ethiopia is one of the top 30 countries burdened by tuberculosis (TB) and experiences a significant distribution of TB and HIV co-infection across all regions. The incidence rates was estimated as 0.17 cases per 1000 population for HIV and 1.64 per 1000 for TB [ 1 , 2 ]. Previous national level study finding among 1,830,880 HIV and 192,359 TB patients reported,7.34% of TB patient had HIV infection with a significant regional variation across regions [ 14 ]. The prevalence of TB/HIV co-infection varies considerably in across each regions including 7.2% in Amhara region (Northern Ethiopia) [ 15 ] to 23.6% southern Ethiopian (SNNR) [ 16 ]. The differences in healthcare accessibility and socio-demographic factors including wealth index and literacy rate contribute to variations in TB/HIV co-infection prevalence [ 14 ]. HIV-infected children face a higher risk of morbidity and mortality during co-infection due to their young age and immature immune makes them more vulnerable [ 4 , 17 ].

Previous studies finding in Ethiopia [ 7 , 18 , 19 ] reported that multifactorial causative factors were attributed for active TB occurrence among HIV infected children including underweight, advanced WHO clinical stages, missed IPT and CPT [ 5 , 7 , 13 , 20 ]. However, CD4 count being ≤200 cells/ml serves as a proxy indicator for incidence of active TB [ 13 ]. Concomitant administration of ART with isoniazid preventive Therapy (IPT) had significantly effect of reducing active TB cases by over 80% HIV infected Children [ 21 ]. However, IPT completion rate and adherence of ART has affected by caregivers and regimen characteristic [ 18 ]. Although several small-scale studies have reported on the epidemiology of TB/HIV co-infection among HIV-infected children in various parts of Ethiopia [ 5 , 12 , 13 ]; however, there is a lack of aggregated data on co-infection after HIV-infected children started antiretroviral therapy. Therefore, this systematic review and meta-analysis aimed to estimate the pooled burden of active TB among HIV-infected children.

Study area and setting

This study was conducted in Ethiopia from January 1, 2013, to December 30, 2022, spanning a period of 10 years. In Ethiopia, there are nine regions including Tigray, Afar, Amhara, Oromia, SNNR, Somalia, Gambella, Benishangul Gumuz, Harari and two city administrative [ 13 ].

Searching strategy and protocol.

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guideline was followed to report the findings of the selected articles presented clearly described in ( S1 Checklist ) [ 22 ].

Additionally, this systematic review and meta-analysis have registered in the Prospero protocol with CRD42024502038 ( https://www.crd.york.ac.uk/prospero/#recordDetails ).

Furthermore, this systematic review was used five international electronic databases were mainly used including Scopus, PubMed, Medline, Web of Science, and Google scholar. The searching was focused on English language published articles and the searching was done. We employed controlled vocabulary terms (MeSH) and free text to extract articles ( S1 Text ) .

The search included topics such as active tuberculosis, pulmonary TB, extra pulmonary TB, HIV infection, individuals, children, pediatrics, neonates, lymphadenitis, disseminated TB, and Ethiopia. The search terms used to identify relevant studies included "Epidemiology" OR "Incidence" OR "Case fatality" "Tuberculosis" OR "Pulmonary Tuberculosis" OR "Disseminated Tuberculosis" OR "Lymphadenitis" AND "HIV" OR "AIDS" AND "Children" OR "Pediatrics" OR "Infant" AND "Ethiopia". Furthermore, this systematic review and meta-analysis employed the PICO (Population, Intervention, Comparison, and Outcomes) framework to assess the eligibility of the articles and enhance evidence-based medicine and research by facilitating the structuring of clinical or research questions. This included as follows (P) Population of interest: Children living with HIV on anti-retroviral therapy in Ethiopia,(I) Intervention; all children HIV infected children started Anti-retroviral therapy, (C) Comparison; children without active TB with stand on HIV cohort (O) Outcome of interest: active TB in HIV-infected children found in Ethiopia were used for PICO frameworks.

Eligibility criteria

Inclusion criteria..

This systematic review and meta-analysis report had included a given articles with defined outcome of any TB types in HIV infected children with the following inclusion criteria. 1) scientific papers reporting co-infections of TB and HIV in HIV-infected children in Ethiopia, 2) articles containing burden or incidence reports of active TB in HIV-infected children, 3) studies published within the past ten years with cross-sectional or cohort designs and published in English, and 4) study subjects limited to children aged ≤15 years.

Exclusion criteria.

Studies that reported lacking abstracts and/or full-text, anonymous reports, editorials, and qualitative studies were excluded from the analysis. Furthermore, prior to the analysis, unfitted articles without a journal name and/or author, lacked the year of publication, and citations without abstracts and/or full-text were removed.

Outcome ascertainment.

The first outcome was the proportion of active TB cases (including all types of TB) among HIV infected children after anti-retroviral therapy. The proportions of TB burden was calculated by the number of children who developed active TB during on ART treatment divided by the total children from thirteen study and multiplied it by 100. Identifying independent predictors for active TB occurrence in HIV infected children on ART was the second objective. Accordingly, we collected significant predictors reported from included articles with their adjusted odd ratio with its 95% confidence interval was extracted from original studies and to computed the pooled odds ratio for final predictors.

Operational words

Advanced hiv disease..

Defined as WHO clinical stages III and IV in children older than five years. However, in children younger than five years living with HIV, they are considered to have advanced HIV disease regardless of their clinical stages. Mild WHO clinical stages refer to stages II and I in HIV-positive children. ART adherence for children; is categorized as follows: Good (>95%) if ≤2 doses are missed out of 30 doses or ≤3 doses out of 60 doses and Fair (85–94%) if 3–4 doses are missed out of 30 doses and poor (<85%) if >5 doses are missed out of 30 doses of ART drug [ 23 ].

Data extraction.

Four Authors (FK, BN, DT, and TK) extracted articles and evaluated the quality of each study by determining the eligibility based on given criteria for selection of studies. The discussion was used to settle any disagreement or uncertainty that arose during the article extraction and removing duplication process. These reviewers assessed the full-text articles; if one or more of them believed an article could be significant, it qualified after the article was carefully examined its titles, abstracts, and full text by three authors (FK, TK, and DT) used a Microsoft Excel spreadsheet to extract the specifics of each article. Three independent reviewers assessed each included article’s quality using the JBI checklists given for all articles as described in ( S1 Table ) [ 24 , 25 ]. All eligible studies approved by all authors’ agreements about principal investigators, year of publication, study period, study setting, study population, and sample size retrieved from the identified articles. The biases of primary studies checked, assessed and screened by three authors (FK, BN and TK), evaluated, and screened ( S2 Table ). Any disagreements among reviewers regarding the critical appraisal were settled through discussion and building consensus for submission.

Software and statistical-analysis.

Using End-Note Aversion 8.1, all detected and potentially suitable published article citations were exported and gathered; duplications were eliminated during the selection and screening processes. Two independent reviewers (FK, and TK) first reviewed the abstracts of the publications before moving on to the full-text articles, which they then evaluated following the particular standards for ultimate inclusion and modifying the data on a Microsoft Excel spreadsheet, and employed the STATA version 17 for further analysis. Descriptive statistics, and weighted inverse variance random-effects meta-regression were used to present the review’s results to estimate pooled burden of active TB in HIV infected children [ 26 ]. The eligible articles were extracted using Meta-XL Excel version 5.3sheet [ 27 ] using identified risk factors from each selected studies and made combined each categorical variables and estimated risk factors for active TB [ 26 ]. The Higgs I 2 statistics were utilized to detect heterogeneity among studies and elaborated using Cochran’s Q test [ 28 ]. The degrees of statistical heterogeneity between the studies were assessed using I2 statistics; values of 25%, 50%, and 75% were thought to indicate modest, medium, and high levels of heterogeneity, respectively [ 38 ]. The source of heterogeneity among studies was examined using the subgroup and sensitivity analysis. The random effect regression model was used for the data-identified heterogeneous analysis [ 26 ]. The publication biases were assessed by visual inspection of funnel plots of the graph and quantitative using Egger’s weighted regression at p <0.1 [ 29 , 30 ].

Descriptive characteristics of the studies

A total of 1221 primary studies were identified including 43 from Web of Science, 631 from PubMed, 352 from Medline, 15 from Scopus, and 162 articles from Google Scholar. After care full screening throughout the articles titles and abstracts, 1208 articles excluded. Thirteen (N = 13) individual studies that met inclusion criteria were included for the final meta-analysis reported [ 4 , 15 – 17 , 31 – 39 ] as presented and described in PRISMA diagram ( Fig 1 ) .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pgph.0003528.g001

Regarding to include articles description all are published in scientific journals from December 30, 2012 to January 1 st , 2023. Regionally seven(N = 7)of articles among eligible articles were from Amhara region (Northern parts of Ethiopia) [ 15 , 31 – 33 , 38 , 39 ] and three articles were from southern nation nationalities region (SNNRs) of Ethiopia [ 16 , 34 ], two of the remaining articles were from Benishangul Gumuz (North west) part of Ethiopia [ 4 , 17 ], and one were from Oromia [ 37 ] regions which is clearly described in ( Table 1 ) .

https://doi.org/10.1371/journal.pgph.0003528.t001

Description reports of included studies

From included 13 individual studies, 6668 HIV infected children were participated and 834 cases of TB among HIV, infected children reported. The mean (±SD age of the participants was reneged from 8.2(±3.6) years. Of the total, nine(9/13) included articles were employed cohort design [ 16 , 17 , 32 , 34 , 38 , 39 ], whereas four of the included articles data were collected by correctional recorded review of follow up design [ 4 , 15 , 31 , 35 , 38 ] were used. The highest number of active TB cases (23.6%) was reported from the SNNPR region (Southern Ethiopia) [ 16 ] and the lowest number (7.2%) of active TB cases was from Amhara region (North West Ethiopia) [ 15 ] regions respectively.

Pooled prevalence of TB in HIV infected children on ART

In the final meta-analysis report, utilizing 13 published studies, we discovered that the estimated pooled burden of active TB among HIV-infected children in Ethiopia was 12.1% (95% CI: 10.7–13.4; I 2 = 63.4%, P = 0.001) as described in ( Fig 2 ) .

https://doi.org/10.1371/journal.pgph.0003528.g002

Factors associated with active TB in HIV infected children on ART

In our final report, there was significant heterogeneity observed among the studies included in the meta-analysis (I2 = 63.4%, P <0.001 as depicted in pooled proportion of active TB in HIV infected children. Accordingly, the pooled TB prevalence was slightly lower in hospital setups at 11.05% (95%CI: 9.4–12.3) compared to health center studies, which reported 14.1% (95%CI: 11.74–16.33) ( Fig 3 ). Likewise, the pooled TB burden among HIV-infected children was significantly higher in studies conducted in the Oromia region at 15.6% (95%CI: 10.2–20.6) compared to studies included from the SNNR, which had a result of 12.8% (95%CI: 10.03–15.67) as described in ( Fig 3 ) .

https://doi.org/10.1371/journal.pgph.0003528.g003

In this report, the duration of follow-up periods was found to be significantly associated with the occurrence of active TB. Sub-group analysis revealed that the pooled burden of TB among HIV-infected children with a follow-up period of ≤10 years was significantly higher at 13.67% (95%CI: 11.24–15.1) compared to those with a follow-up period of >10 years, which had estimation of 10.9% (95%CI: 9.1–12.8) as described in ( Fig 4 ) .

https://doi.org/10.1371/journal.pgph.0003528.g004

In this systematic review, to identify factors associated with active TB we analyzed adjusted odds ratios from primary studies and made grouped significant categorical variables from previous studies by themes, including WHO advanced clinical stages (III&IV), baseline CD4 count, missed isoniazid preventive therapy (IPT), missed cotrimoxazole preventive therapy (CPT), level of hemoglobin, antiretroviral therapy (ART) adherence status, and functional status of children. But, it is noted that only missed IPT, missed CPT, WHO advanced clinical stages (III&IV) and level of hemoglobin were found predictors for TB as shown in ( Table 2 ) .

https://doi.org/10.1371/journal.pgph.0003528.t002

Accordingly, studies containing HIV infected children who missed IPT was double fold increase the odds of active TB occurrence compared with ever given children (OR: 2.28; 95% CI: 1.99–3.08) and also the likelihood of active TB occurrence in HIV infected children who are on advanced WHO clinical stage (III&IV) was 2.27 times (OR: 2.27; 95% CI: 1.81–2.73) higher than with children were on WHO clinical stage II and I. Furthermore, the probability of TB co-infection for HIV-infected children was 3.11 times higher (OR = 3.11, 95% CI: 1.57–4.7) for cases having hemoglobin≤10 mg/dl compared to children with a Hgb >10 mg/dl as described in ( Fig 5 ) . Furthermore HIV-infected children who missed CPT had 4 time higher odds of TB co-infection than counter group (OR: 4.26, 95% CI: 3.47–5.28) clearly depicted in ( Fig 6 ) .

https://doi.org/10.1371/journal.pgph.0003528.g005

https://doi.org/10.1371/journal.pgph.0003528.g006

Publication bias assessment

The publication bias was assessed graphically using funnel plots, and the findings revealed no systematic deviation as depicted in ( Fig 7 ) .

https://doi.org/10.1371/journal.pgph.0003528.g007

In addition, Quantitative analysis we had conducted and assessed using Begg’s and Egger’s tests for biases. Egger’s regression was performed, and the report indicated the absence of publication bias for using two factors sample size and follow up periods as elaborated in ( Table 3 ) .

https://doi.org/10.1371/journal.pgph.0003528.t003

This systematic review and meta-analysis revealed the pooled burden of active (TB) among HIV-infected children in Ethiopia and further identified predictors associated with active TB.

In the final report of 13 individual studies with including 5834 participants, 834 TB and HIV co-infected cases were found at national level. This made the pooled estimated prevalence of active TB was 12.07% (95%CI: 10.73–13.4). This finding is higher than previously reported 0.78% in Ethiopia [ 40 ], 43% in SSA countries [ 41 ], and 1.03% in Portugal [ 42 ]. The findings indicate a significantly high burden of active and need for immediate attention to meet the targets set by the End TB Strategy to achieved the goal of a 90% reduction should to be (≤ 10 TB cases per 100,000 population) by 3035 requires urgent action [ 1 , 2 ]. Conversely, this report is lower than the previous meta-analysis finding 15% in middle-income countries [ 43 , 44 ] and 32% meta-analysis reported in Nigeria [ 45 ]. The variation in the pooled estimate of active TB among the included studies may be attributed to differences in study time TB control measures, and variations among the studies in Nigeria and in Ethiopia [ 46 ].Also the variation in healthcare infrastructure, treatment practices, and regional differences also influence pooled prevalence rates.

This systematic review and meta-analysis revealed that HIV-infected children with advanced WHO clinical stage (III&IV) have a twofold increased likelihood of experiencing active TB compared with mild advanced WHO clinical stage. This finding is supported by previous meta-analyses conducted in Ethiopia [ 47 – 50 ]. This possible justification for this finding is that children with advanced HIV disease may have compromised immune systems due to their clinical stage III&IV, which is associated with low CD4 counts and could lead to an increased risk of developing opportunistic infections, including TB [ 51 ].

The report of this meta-analysis revealed that, the risk of developing active TB was four fold increased for HIV infected children with Hgb ≤10 gm/dl than in those with Hgb >10 gm /dl. This is consistent with the previous study finding [ 52 – 54 ]. This could be attributed to the fact that anemia can indeed impair the immune response and the bactericidal activity of leukocytes, making individuals more vulnerable to infections, including tuberculosis.

The odds of developing active TB among HIV infected children who missed CPT had a four-fold risk as compared ever given children. This is consistent with previously reported meta-analysis in finding in Ethiopia [ 53 , 55 , 56 ]. This might be due to cotrimoxazole, is prescribed to HIV-infected children to prevent lethal opportunistic by preventing production of nucleic acids and proteins essential for the growth of opportunistic infections including PCP, and toxoplasmosis thus helping to counteract immunosuppression and disease progression.

Consistent with previous studies finding in [ 12 , 17 , 32 , 33 , 38 , 57 – 60 ],concurrent administering of IPT after ruled-out of active TB symptoms with ART demoted more than 90% of active TB-associated incidence of morbidity [ 3 , 61 , 62 ]. In the final report of this systematic review and meta-analysis, it was found that HIV-infected children who did not receive IPT (preventive therapy) were at a twofold higher risk compared to the control group. This might be IPT (Isoniazid Preventive Therapy) has the potential to reduce the burden of latent mycobacteria in the lungs. This is because Isoniazid preventive therapy can effectively stop the progression of latent TB infection from developing into active TB disease [ 17 , 63 ].

In contrast to previous systematic review findings [ 53 , 55 ] and primary studies reported [ 12 , 17 , 32 , 33 , 38 , 57 – 60 ] this meta-regression found no significant association between declined CD4 count (≤200 cells/cml), age of patients, duration of follow-up, comorbidity status, and functional status with the risk developing active TB in HIV co-infection children. This might be related to the methodological differences, heterogeneity of included study populations, sample size limitations, publication bias, unaccounted factors, and further experimental studies are highly needed to better understand this relationship.

Strengths and limitations of the study

The strengths of this study include an extensive search strategy, clear inclusion criteria, and the involvement of five independent authors in the quality, inclusion and analysis for this systematic review and meta-analysis. However, there are several methodological limitations including focusing on articles published only on English were included and the extracted articles were from four Ethiopian regions were included in the analysis, such that some of the region may not be represented. In addition, limitations such as reliance on clinical stratification or non-laboratory-supported staging, sub-standard diagnostic capacities in health facilities, a small number of included studies, and the use of retrospective data may potentially affect the validity of the results.

Conclusion and recommendation

This systematic review found a higher prevalence of active TB in HIV-infected children in Ethiopia compared to the estimated rates in the end TB strategy. Risk factors for active TB were identified included WHO clinical stages IV and III, low hemoglobin, missed IPT, and missed CPT were predictors. To reduce the risk of active TB, it is crucial to implement effective strategies such as regular IPT mission and addressing the gaps in treatment, and routine screening for active TB during follow-ups to prevent premature death.

Supporting information

S1 checklist. prisma 2020 chiecklist..

https://doi.org/10.1371/journal.pgph.0003528.s001

S1 Text. Article searching strategy for one of PubMed date base.

https://doi.org/10.1371/journal.pgph.0003528.s002

S1 Table. The JBI quality assessment check list for included studies.

https://doi.org/10.1371/journal.pgph.0003528.s003

S2 Table. Minimal data set for this met-analysis.

https://doi.org/10.1371/journal.pgph.0003528.s004

Acknowledgments

Author contributions.

Fassikaw Kebede Bizune’s ; Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing–original draft, Writing–review & editing.

Dejen Tsegaye’s ; Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization.

Belete Negese; Resources, Software, Supervision, Validation, Visualization, Writing -review & editing.

Tsehay Kebede’s Bizueneh: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Validation, Visualization, Writing–original draft, Writing–review & editing.

• View Article
• Google Scholar
• PubMed/NCBI
• 23. Ethiopia National Guidelines For Comprehensive Hiv Prevention, Care And Treatment. Guideline 2017;Art Management Gudeline Accessed 27 / 05/2024( https://www.childrenandaids.org/sites/default/files/2018-05/Ethiopia_Nat%20Guidelines%20ART_2017.pdf )

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals

You are here

• Online First
• The road to a world-unified approach to the management of patients with gastric intestinal metaplasia: a review of current guidelines
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-0121-6850 Mario Dinis-Ribeiro 1 , 2 ,
• http://orcid.org/0000-0002-2049-9959 Shailja Shah 3 ,
• http://orcid.org/0000-0001-5964-7579 Hashem El-Serag 4 ,
• http://orcid.org/0000-0002-9137-2779 Matthew Banks 5 ,
• http://orcid.org/0000-0002-3029-9272 Noriya Uedo 6 ,
• Hisao Tajiri 7 ,
• http://orcid.org/0000-0002-8721-7696 Luiz Gonzaga Coelho 8 ,
• http://orcid.org/0000-0003-2691-7522 Diogo Libanio 1 , 2 ,
• http://orcid.org/0000-0002-9503-8639 Edith Lahner 9 ,
• http://orcid.org/0000-0002-4997-4098 Antonio Rollan 10 ,
• http://orcid.org/0000-0003-2282-0248 Jing-Yuan Fang 11 ,
• http://orcid.org/0000-0002-4518-8591 Leticia Moreira 12 , 13 ,
• Jan Bornschein 14 ,
• http://orcid.org/0000-0001-8439-9036 Peter Malfertheiner 15 ,
• Ernst J Kuipers 15 ,
• http://orcid.org/0000-0002-0011-3924 Emad M El-Omar 16
• 1 Department of Gastroenterology , Porto Comprehensive Cancer Center & RISE@CI-IPO, University of Porto , Porto , Portugal
• 2 MEDCIDS (Department of Community Medicine, Health Information, and Decision) , University of Porto , Porto , Portugal
• 3 Division of Gastroenterology , University of California and Jennifer Moreno Veterans Affairs San Diego Healthcare System , San Diego , California , USA
• 4 Gastroenterology and Hepatology , Baylor College of Medicine , Houston , Texas , USA
• 5 University College London Hospital , University College London Hospitals NHS Foundation Trust , London , UK
• 6 Gastrointestinal Oncology , Osaka International Cancer Institute , Osaka , Japan
• 7 Endoscopy , The Jikei University School of Medicine , Tokyo , Japan
• 8 Instituto Alfa de Gastrenterologia , Hospital das Clínicas, Universidade Federal de Minas Gerais , Belo Horizonte , Brazil
• 9 Department of Medical-Surgical Sciences and Translational Medicine , Sant'Andrea Hospital , Rome , Italy
• 10 Facultad de Medicina Clinica Alemana-Universidad del Desarrollo , Santiago , Chile
• 11 Division of Gastroenterology and Hepatology , Shanghai Institute of Digestive Disease , Shanghai , China
• 12 Gastroenterology , Hospital Clinic de Barcelona , Barcelona , Spain
• 13 Centro de Investigación Biomédica en Red de Enfermedades Hepáticas y Digestivas (CIBEREHD) , Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS) , Barcelona , Spain
• 14 MRC Translational Immune Discovery Unit , MRC Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford , Oxford , UK
• 15 Medical Department II , LMU University Clinic , München , Germany
• 16 UNSW Microbiome Research Centre , University of New South Wales , Sydney , New South Wales , Australia
• Correspondence to Professor Mario Dinis-Ribeiro, Department of Gastroenterology, RISE—Health Research Network, Porto, 4200-072, Portugal; mario.ribeiro{at}ipoporto.min-saude.pt

Objective During the last decade, the management of gastric intestinal metaplasia (GIM) has been addressed by several distinct international evidence-based guidelines. In this review, we aimed to synthesise these guidelines and provide clinicians with a global perspective of the current recommendations for managing patients with GIM, as well as highlight evidence gaps that need to be addressed with future research.

Design We conducted a systematic review of the literature for guidelines and consensus statements published between January 2010 and February 2023 that address the diagnosis and management of GIM.

Results From 426 manuscripts identified, 15 guidelines were assessed. There was consistency across guidelines regarding the purpose of endoscopic surveillance of GIM, which is to identify prevalent neoplastic lesions and stage gastric preneoplastic conditions. The guidelines also agreed that only patients with high-risk GIM phenotypes (eg, corpus-extended GIM, OLGIM stages III/IV, incomplete GIM subtype), persistent refractory Helicobacter pylori infection or first-degree family history of gastric cancer should undergo regular-interval endoscopic surveillance. In contrast, low-risk phenotypes, which comprise most patients with GIM, do not require surveillance. Not all guidelines are aligned on histological staging systems. If surveillance is indicated, most guidelines recommend a 3-year interval, but there is some variability. All guidelines recommend H. pylori eradication as the only non-endoscopic intervention for gastric cancer prevention, while some offer additional recommendations regarding lifestyle modifications. While most guidelines allude to the importance of high-quality endoscopy for endoscopic surveillance, few detail important metrics apart from stating that a systematic gastric biopsy protocol should be followed. Notably, most guidelines comment on the role of endoscopy for gastric cancer screening and detection of gastric precancerous conditions, but with high heterogeneity, limited guidance regarding implementation, and lack of robust evidence.

Conclusion Despite heterogeneous populations and practices, international guidelines are generally aligned on the importance of GIM as a precancerous condition and the need for a risk-stratified approach to endoscopic surveillance, as well as H. pylori eradication when present. There is room for harmonisation of guidelines regarding (1) which populations merit index endoscopic screening for gastric cancer and GIM detection/staging; (2) objective metrics for high-quality endoscopy; (3) consensus on the need for histological staging and (4) non-endoscopic interventions for gastric cancer prevention apart from H. pylori eradication alone. Robust studies, ideally in the form of randomised trials, are needed to bridge the ample evidence gaps that exist.

• surveillance
• gastric carcinoma

Data availability statement

Data are available on reasonable request. All data analysed are available in proper databases depending on publisher.

https://doi.org/10.1136/gutjnl-2024-333029

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

X @mdinisribeiro, @ShailjaShahMD, @diogolibanio, @MoreiraR_Leti, @emadelomar

MD-R and SS contributed equally.

Contributors MD-R, SS and EME-O developed the protocol, conducted the review and revised the final draft of the manuscript. All the authors collected data, provided input for the protocol and revised the draft of the manuscript, approving the final version. MD-R, SS and EME-O are the guarantors.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. JB is supported by the UK Medical Research Council in the context of the Clinical Academic Research Partnership (MRC CARP) scheme (Grant ref.: MR/W029960/1).

Competing interests No conflicts of interest are declared by MD-R, SS, HE-S, MB, NU, HT, LGC, DL, EL, AR, J-YF, LM, JB, EJK, EME-O. PM is a member of advisory board/lecturer of Aboca, Alfasigma, Allergosan, Bayer, Biocodex, Menarini advisory boards/lectures.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Assessment of green innovation efficiency in chinese industrial enterprises based on an improved relational two-stage dea approach: regional disparities and convergence analysis.

1. Introduction

2. literature review, 3. methodology and data, 3.1. measurement of green innovation efficiency, 3.1.1. green innovation process of industrial enterprises, 3.1.2. design of green innovation efficiency evaluation index, 3.1.3. an improved relational two-stage dea model, 3.2. research methods on regional disparities of green innovation efficiency, 3.2.1. dagum gini coefficient and its decomposition, 3.2.2. kernel density estimation, 3.3. methodology for assessing the convergence of green innovation efficiency, 3.4. sample selection and data sources, 4.1. results of green innovation efficiency measurement, 4.1.1. national-level analysis, 4.1.2. regional-level analysis, 4.2. regional disparities in green innovation efficiency and its decomposition, 4.2.1. regional disparities in r&d efficiency and its decomposition, 4.2.2. regional disparities in achievement conversion efficiency and its decomposition, 4.3. dynamic evolution of green innovation efficiency, 4.3.1. dynamic evolution of r&d efficiency, 4.3.2. dynamic evolution of achievement conversion efficiency, 4.4. convergence analysis of green innovation efficiency, 4.4.1. convergence analysis of r&d efficiency, 4.4.2. convergence analysis of achievement conversion efficiency, 5. discussion, 6. conclusions and policy recommendations, 6.1. conclusions, 6.2. policy recommendations, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Li, D.; Cao, C.; Zhang, L.; Chen, X.; Ren, S.; Zhao, Y. Effects of corporate environmental responsibility on financial performance: The moderating role of government regulation and organizational slack. J. Clean. Prod. 2017 , 166 , 1323–1334. [ Google Scholar ] [ CrossRef ]
• Carayannis, E.G.; Grigoroudis, E.; Goletsis, Y. A multilevel and multistage efficiency evaluation of innovation systems: A multiobjective DEA approach. Expert Syst. Appl. 2016 , 62 , 63–80. [ Google Scholar ] [ CrossRef ]
• Aytekin, A.; Ecer, F.; Korucuk, S.; Karamaşa, Ç. Global innovation efficiency assessment of EU member and candidate countries via DEA-EATWIOS multi-criteria methodology. Technol. Soc. 2022 , 68 , 101896. [ Google Scholar ] [ CrossRef ]
• Zhao, P.; Lu, Z.; Kou, J.; Du, J. Regional differences and convergence of green innovation efficiency in China. J. Environ. Manag. 2023 , 325 , 116618. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhao, N.; Liu, X.; Pan, C.; Wang, C. The performance of green innovation: From an efficiency perspective. Socio-Econ. Plan. Sci. 2021 , 78 , 101062. [ Google Scholar ] [ CrossRef ]
• Guan, J.; Chen, K. Measuring the innovation production process: A cross-region empirical study of China’s high-tech innovations. Technovation 2010 , 30 , 348–358. [ Google Scholar ] [ CrossRef ]
• Li, L.-B.; Liu, B.-L.; Liu, W.-L.; Chiu, Y.H. Efficiency evaluation of the regional high-tech industry in China: A new framework based on meta-frontier dynamic DEA analysis. Socio-Econ. Plan. Sci. 2017 , 60 , 24–33. [ Google Scholar ] [ CrossRef ]
• Bi, K.; Huang, P.; Wang, X. Innovation performance and influencing factors of low-carbon technological innovation under the global value chain: A case of Chinese manufacturing industry. Technol. Forecast. Soc. Change 2016 , 111 , 275–284. [ Google Scholar ] [ CrossRef ]
• Tang, K.; Qiu, Y.; Zhou, D. Does command-and-control regulation promote green innovation performance? Evidence from China’s industrial enterprises. Sci. Total Environ. 2020 , 712 , 136362. [ Google Scholar ] [ CrossRef ]
• Miao, C.-L.; Meng, X.-N.; Duan, M.-M.; Wu, X.-Y. Energy consumption, environmental pollution, and technological innovation efficiency: Taking industrial enterprises in China as empirical analysis object. Environ. Sci. Pollut. Res. 2020 , 27 , 34147–34157. [ Google Scholar ] [ CrossRef ]
• Xu, X.L.; Chen, H.H. Exploring the innovation efficiency of new energy vehicle enterprises in China. Clean Technol. Environ. Policy 2020 , 22 , 1671–1685. [ Google Scholar ] [ CrossRef ]
• Wang, Q.; Hang, Y.; Sun, L.; Zhao, Z. Two-stage innovation efficiency of new energy enterprises in China: A non-radial DEA approach. Technol. Forecast. Soc. Change 2016 , 112 , 254–261. [ Google Scholar ] [ CrossRef ]
• Aigner, D.; Lovell, C.K.; Schmidt, P. Formulation and estimation of stochastic frontier production function models. J. Econom. 1977 , 6 , 21–37. [ Google Scholar ] [ CrossRef ]
• Lin, B.; Luan, R. Are government subsidies effective in improving innovation efficiency? Based on the research of China’s wind power industry. Sci. Total Environ. 2020 , 710 , 136339. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, G.; Wang, J.; Liu, F.; Wang, T.; Zhou, Y.; Tian, A. Regional Differences and Convergence of Technical Efficiency in China’s Marine Economy under Carbon Emission Constraints. Sustainability 2023 , 15 , 7632. [ Google Scholar ] [ CrossRef ]
• Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978 , 2 , 429–444. [ Google Scholar ] [ CrossRef ]
• Hashimoto, A.; Haneda, S. Measuring the change in R&D efficiency of the Japanese pharmaceutical industry. Res. Policy 2008 , 37 , 1829–1836. [ Google Scholar ]
• Fang, Z.; Razzaq, A.; Mohsin, M.; Irfan, M. Spatial spillovers and threshold effects of internet development and entrepreneurship on green innovation efficiency in China. Technol. Soc. 2022 , 68 , 101844. [ Google Scholar ] [ CrossRef ]
• Luo, Q.; Miao, C.; Sun, L.; Meng, X.; Duan, M. Efficiency evaluation of green technology innovation of China’s strategic emerging industries: An empirical analysis based on Malmquist-data envelopment analysis index. J. Clean. Prod. 2019 , 238 , 117782. [ Google Scholar ] [ CrossRef ]
• Namazi, M.; Mohammadi, E. Natural resource dependence and economic growth: A TOPSIS/DEA analysis of innovation efficiency. Resour. Policy 2018 , 59 , 544–552. [ Google Scholar ] [ CrossRef ]
• Qiu, L.; Yu, R.; Hu, F.; Zhou, H.; Hu, H. How can China’s medical manufacturing listed firms improve their technological innovation efficiency? An analysis based on a three-stage DEA model and corporate governance configurations. Technol. Forecast. Soc. Change 2023 , 194 , 122684. [ Google Scholar ] [ CrossRef ]
• Min, S.; Kim, J.; Sawng, Y.-W. The effect of innovation network size and public R&D investment on regional innovation efficiency. Technol. Forecast. Soc. Change 2020 , 155 , 119998. [ Google Scholar ]
• Yang, H.; Zhang, X.; Liu, B.; Huang, Y. Evaluation of green low-carbon innovation development efficiency: An improved two-stage non-cooperative DEA model. J. Clean. Prod. 2023 , 400 , 136662. [ Google Scholar ] [ CrossRef ]
• An, Q.; Meng, F.; Xiong, B.; Wang, Z.; Chen, X. Assessing the relative efficiency of Chinese high-tech industries: A dynamic network data envelopment analysis approach. Ann. Oper. Res. 2020 , 290 , 707–729. [ Google Scholar ] [ CrossRef ]
• Tidd, J.; Bessant, J.R. Managing Innovation: Integrating Technological, Market and Organizational Change ; John Wiley & Sons: Hoboken, NJ, USA, 2020. [ Google Scholar ]
• Paradi, J.C.; Rouatt, S.; Zhu, H. Two-stage evaluation of bank branch efficiency using data envelopment analysis. Omega 2011 , 39 , 99–109. [ Google Scholar ] [ CrossRef ]
• Zhang, Y.; Wang, S. Influence of marine industrial agglomeration and environmental regulation on marine innovation efficiency—From an innovation value chain perspective. Mar. Policy 2021 , 134 , 104807. [ Google Scholar ] [ CrossRef ]
• Wang, K.; Huang, W.; Wu, J.; Liu, Y.-N. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega 2014 , 44 , 5–20. [ Google Scholar ] [ CrossRef ]
• Kao, C.; Hwang, S.-N. Efficiency decomposition in two-stage data envelopment analysis: An application to non-life insurance companies in Taiwan. Eur. J. Oper. Res. 2008 , 185 , 418–429. [ Google Scholar ] [ CrossRef ]
• Dagum, C. A new approach to the decomposition of the Gini income inequality ratio. In Income Inequality, Poverty, and Economic Welfare ; Springer: Berlin/Heidelberg, Germany, 1998. [ Google Scholar ]
• Liu, F.; Nan, T.; Wang, X. Causes and evolution characteristics of green innovation efficiency loss: The perspective of factor mismatch under local government competition. Sustainability 2022 , 14 , 8338. [ Google Scholar ] [ CrossRef ]
• Tang, K.; Xiong, C.; Wang, Y.; Zhou, D. Carbon emissions performance trend across Chinese cities: Evidence from efficiency and convergence evaluation. Environ. Sci. Pollut. Res. 2021 , 28 , 1533–1544. [ Google Scholar ] [ CrossRef ]
• Gao, K.; Yuan, Y.-J. Spatiotemporal pattern assessment of China’s industrial green productivity and its spatial drivers: Evidence from city-level data over 2000–2017. Appl. Energy 2022 , 307 , 118248. [ Google Scholar ] [ CrossRef ]
• Mingran, W. Measurement and spatial statistical analysis of green science and technology innovation efficiency among Chinese Provinces. Environ. Ecol. Stat. 2021 , 28 , 423–444. [ Google Scholar ] [ CrossRef ]
• Dong, F.; Li, Y.; Qin, C.; Sun, J. How industrial convergence affects regional green development efficiency: A spatial conditional process analysis. J. Environ. Manag. 2021 , 300 , 113738. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Hall, L.A.; Bagchi-Sen, S. A study of R&D, innovation, and business performance in the Canadian biotechnology industry. Technovation 2002 , 22 , 231–244. [ Google Scholar ]
• Chen, X.; Liu, Z.; Zhu, Q. Reprint of “Performance evaluation of China’s high-tech innovation process: Analysis based on the innovation value chain”. Technovation 2020 , 94 , 102094. [ Google Scholar ] [ CrossRef ]
• Ganotakis, P.; Love, J.H. The innovation value chain in new technology-based firms: Evidence from the UK. J. Prod. Innov. Manag. 2012 , 29 , 839–860. [ Google Scholar ] [ CrossRef ]
• Wang, C.-H.; Lu, Y.-H.; Huang, C.-W.; Lee, J.-Y. R&D, productivity, and market value: An empirical study from high-technology firms. Omega 2013 , 41 , 143–155. [ Google Scholar ]
• Zhu, L.; Luo, J.; Dong, Q.; Zhao, Y.; Wang, Y.; Wang, Y. Green technology innovation efficiency of energy-intensive industries in China from the perspective of shared resources: Dynamic change and improvement path. Technol. Forecast. Soc. Change 2021 , 170 , 120890. [ Google Scholar ] [ CrossRef ]
• Hansen, M.T.; Birkinshaw, J. The innovation value chain. Harv. Bus. Rev. 2007 , 85 , 121–130,142. [ Google Scholar ]
• Furman, J.L.; Porter, M.E.; Stern, S. The determinants of national innovative capacity. Res. Policy 2002 , 31 , 899–933. [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Pan, J.-F.; Pei, R.-M.; Yi, B.-W.; Yang, G.-L. Assessing the technological innovation efficiency of China’s high-tech industries with a two-stage network DEA approach. Socio-Econ. Plan. Sci. 2020 , 71 , 100810. [ Google Scholar ] [ CrossRef ]
• Zuo, Z.; Guo, H.; Li, Y.; Cheng, J. A two-stage DEA evaluation of Chinese mining industry technological innovation efficiency and eco-efficiency. Environ. Impact Assess. Rev. 2022 , 94 , 106762. [ Google Scholar ] [ CrossRef ]
• Li, J.; Du, Y. Spatial effect of environmental regulation on green innovation efficiency: Evidence from prefectural-level cities in China. J. Clean. Prod. 2021 , 286 , 125032. [ Google Scholar ] [ CrossRef ]
• Song, M.; Zhu, S.; Wang, J.; Zhao, J. Share green growth: Regional evaluation of green output performance in China. Int. J. Prod. Econ. 2020 , 219 , 152–163. [ Google Scholar ] [ CrossRef ]
• Wu, Y. R&D stock, knowledge function and productive efficiency. China Econ. Q. Beijing 2006 , 5 , 1129. [ Google Scholar ]
• Hong, J.; Feng, B.; Wu, Y.; Wang, L. Do government grants promote innovation efficiency in China’s high-tech industries? Technovation 2016 , 57 , 4–13. [ Google Scholar ] [ CrossRef ]
• Bronzini, R.; Piselli, P. The impact of R&D subsidies on firm innovation. Res. Policy 2016 , 45 , 442–457. [ Google Scholar ]
• Li, D.; Zhao, Y.; Zhang, L.; Chen, X.; Cao, C. Impact of quality management on green innovation. J. Clean. Prod. 2018 , 170 , 462–470. [ Google Scholar ] [ CrossRef ]
• Ren, S.; He, D.; Yan, J.; Zeng, H.; Tan, J. Environmental labeling certification and corporate environmental innovation: The moderating role of corporate ownership and local government intervention. J. Bus. Res. 2022 , 140 , 556–571. [ Google Scholar ] [ CrossRef ]
• Griliches, Z. Patent statistics as economic indicators: A survey. part I. In R&D and Productivity: The Econometric Evidence ; National Bureau of Economic Reserch: Cambridge, MA, USA, 1990. [ Google Scholar ]
• Seiford, L.M.; Zhu, J. Modeling undesirable factors in efficiency evaluation. Eur. J. Oper. Res. 2002 , 142 , 16–20. [ Google Scholar ] [ CrossRef ]
• Liang, L.; Cook, W.D.; Zhu, J. DEA models for two-stage processes: Game approach and efficiency decomposition. Nav. Res. Logist. 2008 , 55 , 643–653. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Liang, L.; Zhu, J. Equivalence in two-stage DEA approaches. Eur. J. Oper. Res. 2009 , 193 , 600–604. [ Google Scholar ] [ CrossRef ]
• Li, Y.; Chen, Y.; Liang, L.; Xie, J. DEA models for extended two-stage network structures. Omega 2012 , 40 , 611–618. [ Google Scholar ] [ CrossRef ]
• Davis, R.A.; Lii, K.-S.; Politis, D.N. Remarks on some nonparametric estimates of a density function. In Selected Works of Murray Rosenblatt ; Springer: New York, NY, USA, 2011; pp. 95–100. [ Google Scholar ]
• Shen, W.; Shi, J.; Meng, Q.; Chen, X.; Liu, Y.; Cheng, K.; Liu, W. Influences of environmental regulations on industrial green technology innovation efficiency in China. Sustainability 2022 , 14 , 4717. [ Google Scholar ] [ CrossRef ]
• Xiao, Q.; Lu, Z. Heterogeneous Environment Regulation, FDI and the Efficiency of China’s Green Technology Innovation. Mod. Econ. Res. 2020 , 4 , 29–40. [ Google Scholar ]
• Li, T.; Li, Y.; An, D.; Han, Y.; Xu, S.; Lu, Z.; Crittenden, J. Mining of the association rules between industrialization level and air quality to inform high-quality development in China. J. Environ. Manag. 2019 , 246 , 564–574. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhang, Y.; Hu, H.; Zhu, G.; You, D. The impact of environmental regulation on enterprises’ green innovation under the constraint of external financing: Evidence from China’s industrial firms. Environ. Sci. Pollut. Res. 2023 , 30 , 42943–42964. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kleer, R. Government R&D subsidies as a signal for private investors. Res. Policy 2010 , 39 , 1361–1374. [ Google Scholar ]
• Change, E.T. Endogenous Technological Change. J. Political Econ. 1990 , 98 , 2. [ Google Scholar ]
• Zhou, H.; Wang, R. Exploring the impact of energy factor prices and environmental regulation on China’s green innovation efficiency. Environ. Sci. Pollut. Res. 2022 , 29 , 78973–78988. [ Google Scholar ] [ CrossRef ]
• Fujii, H.; Managi, S. Decomposition analysis of sustainable green technology inventions in China. Technol. Forecast. Soc. Change 2019 , 139 , 10–16. [ Google Scholar ] [ CrossRef ]
• Du, J.-L.; Liu, Y.; Diao, W.-X. Assessing regional differences in green innovation efficiency of industrial enterprises in China. Int. J. Environ. Res. Public Health 2019 , 16 , 940. [ Google Scholar ] [ CrossRef ]
• Yao, M.; Duan, J.; Wang, Q. Spatial and temporal evolution analysis of industrial green technology innovation efficiency in the Yangtze River Economic Belt. Int. J. Environ. Res. Public Health 2022 , 19 , 6361. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Xu, S.; Wu, T.; Zhang, Y. The spatial-temporal variation and convergence of green innovation efficiency in the Yangtze River Economic Belt in China. Environ. Sci. Pollut. Res. 2020 , 27 , 26868–26881. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhao, T.; Zhou, H.; Jiang, J.; Yan, W. Impact of green finance and environmental regulations on the green innovation efficiency in China. Sustainability 2022 , 14 , 3206. [ Google Scholar ] [ CrossRef ]
• Li, D.; Zheng, M.; Cao, C.; Chen, X.; Ren, S.; Huang, M. The impact of legitimacy pressure and corporate profitability on green innovation: Evidence from China top 100. J. Clean. Prod. 2017 , 141 , 41–49. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Chen, X.; Xu, R. Assessment of Green Innovation Efficiency in Chinese Industrial Enterprises Based on an Improved Relational Two-Stage DEA Approach: Regional Disparities and Convergence Analysis. Sustainability 2024 , 16 , 6908. https://doi.org/10.3390/su16166908

Chen X, Xu R. Assessment of Green Innovation Efficiency in Chinese Industrial Enterprises Based on an Improved Relational Two-Stage DEA Approach: Regional Disparities and Convergence Analysis. Sustainability . 2024; 16(16):6908. https://doi.org/10.3390/su16166908

Chen, Xiaohong, and Ruochen Xu. 2024. "Assessment of Green Innovation Efficiency in Chinese Industrial Enterprises Based on an Improved Relational Two-Stage DEA Approach: Regional Disparities and Convergence Analysis" Sustainability 16, no. 16: 6908. https://doi.org/10.3390/su16166908

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Publication Home

• About the Journal
• Editorial Team
• Article Processing Charges
• Privacy Statement
• Submissions

Systematic Literature Review and Meta-Analysis of Microcontroller Learning Development in the Industry 4.0

##plugins.themes.bootstrap3.article.sidebar##, ##plugins.themes.bootstrap3.article.main##.

Sigit Candra Setya

Krismadinata

Jusmita Weriza

The ability to design and implement microcontroller-based systems is one of the skills and soft skills needed in Industry 4.0. Many researchers have identified various problems and have made solutions related to the learning process using microcontrollers, even though the names of the courses taught in the study programs at their respective universities are different. So a map of microcontroller learning problems and solutions that have been carried out by previous researchers is needed, then summarize and analyze them so that the core problems and solutions related to microcontroller learning in the Industry 4.0 era can be known. So in this study, we conducted a systematic literature review and meta-analysis of problems and solutions in microcontroller learning from research works on the Scopus database from 2019 to 2023, using the Watase UAKE application in which there is Prisma. The first stage carried out is the identification of studies on the Scopus database using several keyword variables that have been determined in advance, resulting in 173 articles. Then the first stage screening resulted in 78 articles, and continued with the second stage screening resulting in 36 articles. Retrieve articles that have DOI obtained 29 articles to be uploaded into the Watase database. Based on the results of data extraction and reading the contents of the article carefully, 20 articles were obtained that discussed the industry in the article, consisting of 13 qualitative research articles and 7 quantitative research articles. Only 7 quantitative research articles can proceed to the classification process stage of the meta-analysis. The results showed that the most popular learning method related to the learning process using microcontrollers in the Industry 4.0 era is the Project Based Learning (PBL) learning method.

##plugins.themes.bootstrap3.article.details##

This work is licensed under a Creative Commons Attribution 4.0 International License .