An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
- Publications
- Account settings
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
- Advanced Search
- Journal List
How-to conduct a systematic literature review: A quick guide for computer science research
Angela carrera-rivera.
a Faculty of Engineering, Mondragon University
William Ochoa
Felix larrinaga.
b Design Innovation Center(DBZ), Mondragon University
Associated Data
- No data was used for the research described in the article.
Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:
- • Clearly defined strategies to follow for a systematic literature review in computer science research, and
- • Algorithmic method to tackle a systematic literature review.
Graphical abstract
Specifications table
| Subject area: | Computer-science |
| More specific subject area: | Software engineering |
| Name of your method: | Systematic literature review |
| Name and reference of original method: | |
| Resource availability: | Resources referred to in this article: ) ) |
Method details
A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.
Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.
Define PICOC and synonyms
The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.
Planning Step 1 “Defining PICOC keywords and synonyms”.
| Description | Example (PICOC) | Example (Synonyms) | |
|---|---|---|---|
| Population | Can be a specific role, an application area, or an industry domain. | Smart Manufacturing | • Digital Factory • Digital Manufacturing • Smart Factory |
| Intervention | The methodology, tool, or technology that addresses a specific issue. | Semantic Web | • Ontology • Semantic Reasoning |
| Comparison | The methodology, tool, or technology in which the is being compared (if appropriate). | Machine Learning | • Supervised Learning • Unsupervised Learning |
| Outcome | Factors of importance to practitioners and/or the results that could produce. | Context-Awareness | • Context-Aware • Context-Reasoning |
| Context | The context in which the comparison takes place. Some systematic reviews might choose to exclude this element. | Business Process Management | • BPM • Business Process Modeling |
Formulate research questions
Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).
Research questions examples.
| Research Questions examples |
|---|
| • : What are the current challenges of context-aware systems that support the decision-making of business processes in smart manufacturing? • : Which technique is most appropriate to support decision-making for business process management in smart factories? • : In which scenarios are semantic web and machine learning used to provide context-awareness in business process management for smart manufacturing? |
Select digital library sources
The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.
Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.
| Database | Description | URL | Area | Advanced Search Y/N |
|---|---|---|---|---|
| Scopus | From Elsevier. sOne of the largest databases. Very user-friendly interface | Interdisciplinary | Y | |
| Web of Science | From Clarivate. Multidisciplinary database with wide ranging content. | Interdisciplinary | Y | |
| EI Compendex | From Elsevier. Focused on engineering literature. | Engineering | Y (Query view not available) | |
| IEEE Digital Library | Contains scientific and technical articles published by IEEE and its publishing partners. | Engineering and Technology | Y | |
| ACM Digital Library | Complete collection of ACM publications. | Computing and information technology | Y |
Define inclusion and exclusion criteria
Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.
Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.
| Criteria Type | Description | Example |
|---|---|---|
| Period | Articles can be selected based on the time period to review, e.g., reviewing the technology under study from the year it emerged, or reviewing progress in the field since the publication of a prior literature review. | : From 2015 to 2021 Articles prior 2015 |
| Language | Articles can be excluded based on language. | : Articles not in English |
| Type of Literature | Articles can be excluded if they are fall into the category of grey literature. | Reports, policy literature, working papers, newsletters, government documents, speeches |
| Type of source | Articles can be included or excluded by the type of origin, i.e., conference or journal articles or books. | : Articles from Conferences or Journals Articles from books |
| Impact Source | Articles can be excluded if the author limits the impact factor or quartile of the source. | Articles from Q1, and Q2 sources : Articles with a Journal Impact Score (JIS) lower than |
| Accessibility | Not accessible in specific databases. | : Not accessible |
| Relevance to research questions | Articles can be excluded if they are not relevant to a particular question or to “ ” number of research questions. | Not relevant to at least 2 research questions |
Define the Quality Assessment (QA) checklist
Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.
Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.
| Do the researchers discuss any problems (limitations, threats) with the validity of their results (reliability)? | 1 – No, and not considered (Score: 0) 2 – Partially (Score: 0.5) 3 – Yes (Score: 1) |
| Is there a clear definition/ description/ statement of the aims/ goals/ purposes/ motivations/ objectives/ questions of the research? | 1 – Disagree (Score: 1) 2 – Somewhat disagree (Score: 2) 3 – Neither agree nor disagree (Score: 3) 4 – Somewhat agree (Score: 4) 5 – Agree (Score: 5) |
Define the “Data Extraction” form
The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .
Planning Step 6 “Define data extraction form”. Examples of fields.
| Classification and fields to consider for data extraction | Description and examples |
|---|---|
| Research type | • focuses on abstract ideas, concepts, and theories built on literature reviews . • uses scientific data or case studies for explorative, descriptive, explanatory, or measurable findings . an SLR on context-awareness for S-PSS and categorized the articles in theoretical and empirical research. |
| By process phases, stages | When analyzing a process or series of processes, an effective way to structure the data is to find a well-established framework of reference or architecture. : • an SLR on self-adaptive systems uses the MAPE-K model to understand how the authors tackle each module stage. • presented a context-awareness survey using the stages of context-aware lifecycle to review different methods. |
| By technology, framework, or platform | When analyzing a computer science topic, it is important to know the technology currently employed to understand trends, benefits, or limitations. : • an SLR on the big data ecosystem in the manufacturing field that includes frameworks, tools, and platforms for each stage of the big data ecosystem. |
| By application field and/or industry domain | If the review is not limited to a specific “Context” or “Population" (industry domain), it can be useful to identify the field of application : • an SLR on adaptive training using virtual reality (VR). The review presents an extensive description of multiple application domains and examines related work. |
| Gaps and challenges | Identifying gaps and challenges is important in reviews to determine the research needs and further establish research directions that can help scholars act on the topic. |
| Findings in research | Research in computer science can deliver multiple types of findings, e.g.: |
| Evaluation method | Case studies, experiments, surveys, mathematical demonstrations, and performance indicators. |
The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.
After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.
Build digital library search strings
Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:
(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”
Gather studies
Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.
Example of Advanced search on Scopus.
In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.
Study Selection and Refinement
The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.
In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.
Performing quality assessment (QA) in Parsif.al.
Data extraction
Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.
Example of data extraction form using Parsif.al.
The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.
Analysis and Report
The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.
The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.
Techniques for bibliometric analysis and examples.
| Publication-related analysis | Description | Example |
|---|---|---|
| Years of publications | Determine interest in the research topic by years or the period established by the SLR, by quantifying the number of papers published. Using this information, it is also possible to forecast the growth rate of research interest. | [ ] identified the growth rate of research interest and the yearly publication trend. |
| Top contribution journals/conferences | Identify the leading journals and conferences in which authors can share their current and future work. | , |
| Top countries' or affiliation contributions | Examine the impacts of countries or affiliations leading the research topic. | [ , ] identified the most influential countries. |
| Leading authors | Identify the most significant authors in a research field. | - |
| Keyword correlation analysis | Explore existing relationships between topics in a research field based on the written content of the publication or related keywords established in the articles. | using keyword clustering analysis ( ). using frequency analysis. |
| Total and average citation | Identify the most relevant publications in a research field. | Scatter plot citation scores and journal factor impact |
Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.
[1] Keyword co-relationship analysis using clusterization in vos viewer.
This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.
Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.
The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].
SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.
CRediT author statement
Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.
Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.
1 https://parsif.al/
Data Availability
Writing a Systematic Literature Review Dissertation
This is an interactive pdf document guiding students through the planning process of a literature based dissertation for our MEd in Academic Practice. It might be useful for anyone wanting to write a systematic literature review, or a literature based dissertation.
Links from the document: Ohio Library Guide and University of Glasgow Guide
Advance HE Fellowship status
- Senior Fellowship
Author's role
- Educational developer
Accessibility status
- Not accessible, or has not been checked
Usage metrics
- Other education not elsewhere classified
A systematic literature review of time series methods applied to epidemic prediction
- August 2024
- Informatics in Medicine Unlocked
- CC BY-NC 4.0
- University of Ngaoundere
- Tilburg University
Discover the world's research
- 25+ million members
- 160+ million publication pages
- 2.3+ billion citations
- Yong-Chao Jin
- Xi-Yin Wang
- Andrew Grizzle
- Esther Ososonya
- Ke-Nan Wang
- MULTIMED TOOLS APPL
- Yeongha Kim
- Chang-Reung Park
- Jae-Pyoung Ahn
- Beakcheol Jang
- PLOS NEGLECT TROP D
- Mahdi Shafiee Kamalabad
- SOC NETWORKS
- Recruit researchers
- Join for free
- Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up
COMMENTS
systematic review research. Systematic literature review differs from structured literature reviews and primary research projects, and students need to understand the difference. Therefore, they must comprehend how to plan and prepare their systematic review dissertation project. When should you not use a systematic review?
ecent years, it constitutes an interesting and comprehensive research topic.This thesis presents a systematic literature rev. ew (SLR) of published research articles concerning agile project management. Based on a predefined search strategy. 273 such articles were identified, of which 44 were included in the.
fined criteria in order to answer a research question. The quantitative combination and statistical synthesis of the systema. cally-collected data is what defines a meta-analysis. Here, we first attempt to delineate the basic steps for conducting a systematic review: initial planning, conduc.
Example for a Systematic Literature Review: In references 5 example for paper that use Systematic Literature Review (SlR) example: ( Event-Driven Process Chain for Modeling and Verification of ...
Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...
at each of these in turn.IntroductionThe first part of any literature review is a way of inviting your read. into the topic and orientating them. A good introduction tells the reader what the review is about - its s. pe—and what you are going to cover. It may also specifically tell you.
The guide I present here has eight major steps (see Figure 1) that one needs to take to conduct a systematic literature review. Indeed, these steps are very valuable for any kind of literature review; however, for a review to be scientifically rigorous, all of the following steps are essential.
rapid review, scoping review; Grant & Booth, 2009), we believe that systematic literature review (SLR) is especially approachable for novice researchers as it follows a rigorous methodological ... especially in connection with their thesis or dissertation work, we provide a 13-step approach accompanied by a simplified checklist that graduate ...
The first stage in conducting a systematic. review is to develop a protocol that clearly defines: 1) the aims. and objectives of the review; 2) the inclusion and exclusion. criteria for studies ...
A systematic review is a special type of literature review that confers added advantages. It is "A review of a clearly formulated question that uses systematic and explicit
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
A preliminary review, which can often result in a full systematic review, to understand the available research literature, is usually time or scope limited. Complies evidence from multiple reviews and does not search for primary studies. 3. Identifying a topic and developing inclusion/exclusion criteria.
A Complete Dissertation 7 purpose, or it does not stand alone as a document. Chapter 2: Literature Review This chapter situates the study in the con-text of previous research and scholarly mate - rial pertaining to the topic, presents a critical synthesis of empirical literature according to relevant themes or variables, justifies how
Systematic Review Components. Starts with a clearly articulated question. Uses explicit, rigorous methods to identify, critically appraise, and synthesize relevant studies. Appraises relevant published and unpublished evidence for validity before combining and analyzing data. Reports methodology, studies included in the review, and conclusions ...
guide writers of literature reviews, the labor intensive. process of writing one compounds the problem. Gall, Borg, and Gall (1996) estima te that completion of an. acceptable dissertation ...
The present study is a systematic review and meta-analysis that explores the effectiveness of child-parent interventions for childhood anxiety disorders. The research located during the literature search was coded for inclusionary criteria and resulted in eight qualifying individual randomized controlled trials (RCT) with a total of 710
A Systematic Literature Review of Project Management Tools and Their ...
Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure .An SLR updates the reader with current literature about a subject .The goal is to review critical points of current knowledge on a topic about research ...
Gall, Borg, and Gall (1996) estimate that completion of an acceptable dissertation literature review will take between three and six months of effort. The purpose of this guide is to collect and summarize the most relevant information on how to write a dissertation literature review.
This is an interactive pdf document guiding students through the planning process of a literature based dissertation for our MEd in Academic Practice. It might be useful for anyone wanting to write a systematic literature review, or a literature based dissertation. Links from the document: Ohio Library Guide and University of Glasgow Guide.
Systematic literature reviews (SRs) are a way of synt hesising scientific evidence to answer a particular. research question in a way that is transparent and reproducible, while seeking to include ...
Peterson R A (2017) A case study of grade 12 international school students' perceptions of the impact of a bible curriculum and the teachers' delivery of that bible curriculum in a Christian international school (Order No. 10269987) [Doctoral dissertation, Liberty University, USA]. ProQuest Dissertations & Thesis Global.
This Systematic Literature Review aims to assess, based on a sample of relevant papers, the use of Time Series Methods (TSM) in epidemic prediction, with a special focus on African issues and the ...