literature review collaborative learning

REVIEW article

Virtual reality and collaborative learning: a systematic literature review.

• 1 Centre for Education and Learning, Delft University of Technology, Delft, Netherlands
• 2 Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands
• 3 Interactive Intelligence, Delft University of Technology, Delft, Netherlands

Background: While research on Virtual Reality’s potential for education continues to advance, research on its support for Collaborative Learning is small in scope. With remote collaboration and distance learning becoming increasingly relevant for education (especially since the COVID-19 pandemic), an understanding of Virtual Reality’s potential for Collaborative Learning is of importance. To establish how this immersive technology can support and enhance collaboration between learners, this systematic literature review analyses scientific research on Virtual Reality for Collaborative Learning with the intention to identify 1) skills and competences trained, 2) domains and disciplines addressed, 3) systems used and 4) empirical knowledge established.

Method: Two scientific databases—Scopus and Web of Science—were used for this review. Following the PRISMA method, a total of 139 articles were analyzed. Reliability of this selection process was assessed using five additional coders. A taxonomy was used to classify these articles. Another coder was used to assess the reliability of the primary coder before this taxonomy was applied to the selected articles

Results: Based on the literature reviewed, skills and competences developed are divided into five categories. Educational fields and domains seem interested in Virtual Reality for Collaborative Learning because of a need for innovation, communities and remote socialization and collaboration between learners. Systems primarily use monitor-based Virtual Reality and mouse-and-keyboard controls. A general optimism is visible regarding the use of Virtual Reality to support and enhance Collaborative Learning

Conclusion: Five distinct affordances of Virtual Reality for Collaborative Learning are identified: it 1) is an efficient tool to engage and motivate learners, 2) supports distance learning and remote collaboration, 3) provides multi- and interdisciplinary spaces for both learning and collaborating, 4) helps develop social skills and 5) suits Collaborative Learning-related paradigms and approaches. Overall, the reviewed literature suggests Virtual Reality to be an effective tool for the support and enhancement of Collaborative Learning, though further research is necessary to establish pedagogies.

1 Introduction

Beginning in the 1980s, academia has studied how to support and enhance Collaborative Learning (CL) in educational settings using technology. Referred to as Computer-Supported Collaborative Learning (CSCL), this pedagogical approach stems from social learning, an educational theory revolving around the idea that “new behavior can be acquired through the observation of other people’s behaviors” ( Shi et al., 2019 ) and focusing on social interaction between learners. CSCL’s strength appears to lie in its flexibility: by using characteristics of technology, both distant and face-to-face collaboration, as well as synchronous and asynchronous collaboration between learners, can be supported ( Stahl et al., 2006 ). As such, CSCL has been attributed numerous affordances, including joint information processing, sharing resources and co-construction of knowledge ( Shawky et al., 2014 ; Jeong and Hmelo-Silver, 2016 ).

An on-going development in the field of CSCL is the use of Virtual Reality, a technology that ‘[transports] a person to a reality (i.e., a virtual environment) which he or she is not physically present but feels like he or she is there’ ( Rebelo et al., 2012 ). These virtual environments (VEs) are shared, simulated spaces that allow distributed users to communicate with each other, as well as to participate in joint activities, making them an effective tool for remote collaboration ( Daphne et al., 2000 ). VEs tend to be highly customizable; their visual representation can be realistic (i.e., similar to reality or containing recognizable elements from reality) or abstract (e.g., three-dimensional representations of abstract concepts) depending on their purpose, making VEs adaptable for many different fields and disciplines ( Jackson et al., 1999 ; Joyner et al., 2021 ). Virtual Reality (VR), then, functions as a human-computer interface, allowing users to access these VEs through a variety of hardware, including flat-surface monitors and displays connected to desktop computers, room-sized devices called CAVE systems that project the VE onto its walls and Head-Mounted Displays (HMDs), helmets or headpieces that visualize the VE individually for each eye. In some cases, users inhabit avatars, virtual embodiments that represent their place inside the VE, though in other cases (such as the aforementioned CAVE systems, where users do not have to wear HMDs), no avatars are required for users to detect each other. Like VEs, the visual representation of avatars can be diverse: avatars can provide realistic depictions of users’ real-life appearances, but can also be visualized as something abstract, such as geometric objects or animals. Using these avatars to mediate interactions with each other, users progressively construct a shared understanding of the VE together ( Girvan, 2018 ). Of particular interest is VR’s ability to “immerse” users, providing them a sense of being inside the VE despite its non-physical, digital nature ( Freina and Ott, 2015 ). This immersion may lead to a state of presence, wherein users begin to behave inside the VE as they would in the physical world ( Jensen and Konradsen, 2018 ). Affordances of VR in education include enhancement of experiential learning ( Le et al., 2015 ; Kwon, 2019 ), spatial learning ( Dalgarno and Lee, 2010 ; de Back et al., 2020 ) and motivation and engagement among different types of learners ( Merchant et al., 2014 ; Chavez and Bayona, 2018 ). While research on VR has generally revolved around discovering its potential to support and enhance education, academics appear to agree that the field of educational use of VR lacks pedagogical practices or strategies, with little focus on how the technology should be implemented to reap its benefits ( Cook et al., 2019 ; Smith, 2019 ; Scavarelli et al., 2021 ).

VR technology has already shown potential for the field of CSCL, improving the effectiveness of team behavior, enhancing communication between group members and increasing learning outcome gains ( Le et al., 2015 ; Godin and Pridmore, 2019 ; Zheng et al., 2019 ). What makes the use of Virtual Reality for Collaborative Learning (VRCL) even more appealing for education is its diversity in hardware and, as a result, the different forms it can take depending on the setting. Whether learners interact with the VEs via display monitors, CAVE systems or HMDs, they all seem to produce positive effects such as positive learning gains and outcomes, as well as engagement and motivation for CL ( Abdullah et al., 2019 ; Zheng et al., 2019 ; de Back et al., 2020 ; Tovar et al., 2020 ).

To advance the field of VRCL, as well as to establish its benefits and affordances, several literature reviews have examined research on VRCL. For example, Muhammad Nur Affendy and Ajune Wanis (2019) , aiming to provide an overview of the capabilities of CL through the adoption of collaborative system in AR and VR, review how VEs are used for different types of collaboration (e.g., remote and co-located collaboration), with different VR hardware (e.g., eye tracking) and multiple intended uses (e.g., increasing social engagement and supporting awareness of collaboration among learners). In comparison, Zheng et al. (2019) evaluate VRCL technology affordances by conducting a meta-analysis as well as a qualitative analysis of VRCL prototypes to explore potential learning benefits; Scavarelli et al. (2021) explore a more theoretical side with the intention to produce educational frameworks for future VRCL-related research, discussing how several learning theories (e.g., constructivism, social cognitive theory and connectivism) are reflected in prior research on the potential of VR as well as Augmented Reality (AR) for social learning spaces.

Together, the literature reviews of Muhammad Nur Affendy and Ajune Wanis (2019) , Zheng et al. (2019) ; Scavarelli et al. (2021) describe a general optimism towards VR in educational settings to support collaboration. The reviews outline VRCL’s strengths as 1) its ability to enhance learning outcomes, 2) its potential to facilitate learning, 3) its effectiveness in supporting remote collaboration between learners, as well as experts and novices, 4) its support for interpersonal awareness between collaborating learners and 5) its diversity, both in terms of its customizability (allowing VEs to better suit objectives) as well as its technology. Affordances of VRCL are identified as 1) social interaction (strengthened by VR’s affordances of immersion and presence), 2) resource sharing (strengthened by VR’s ability to present imaginary elements) and 3) knowledge construction (supported by the two prior affordances of VRCL). Furthermore, challenges and gaps related to (research on) VRCL are outlined. First, accessibility should be considered a primary concern according to Scavarelli et al.,; this does not just relate to the technical accessibility of VR when used in education, but more so to the accessibility of social engagement between learners sharing these virtual learning spaces. Second, they recommend to explore the interplay and connectivity between VEs and the real world, as doing so could reveal new learning theories that innovate VRCL. Third, Zheng et al., suggest that research focus on pedagogical strategies involving VRCL, including how to apply VR to educational settings involving collaboration. Fourth, they propose a focus on finding a balance between using VRCL to recreate (or simulate) existing (“real”) situations and creating new situations that would normally be impossible, considering that prior work has primarily been centered on the former and as such misses out on VR’s potential to do the latter.

Considering that remote collaboration and distance learning, especially since the COVID-19 pandemic, are becoming increasingly important for learners, an understanding of VR’s potential for CL could prove beneficial for the field of education. While research on the topic is apparent, studies focusing on VR’s ability to support and enhance CL are still small in scale ( Zheng et al., 2019 ; Scavarelli et al., 2021 ), accentuating the scarcity of knowledge on the topic. This systematic review specifically centers on scientific research on VRCL, with a particular focus on the empirical knowledge that such literature has established. The aim of this paper is to examine in what ways VR supports and enhances CL according to prior research on these topics; to achieve this, it reports on what VRCL is used for in different fields of education, discusses what research has stated regarding VRCL in terms of affordances and benefits for education, describes the characteristics of VRCL that allow these benefits to come to fruition and provides an insight into the technology behind VRCL, as well as how this compares to the state-of-the-art of VR. In doing so, this study intends to identify possible gaps in the field of VRCL research for possible future studies, in addition to highlighting VRCL’s strengths to support current research. To the best of the authors’ knowledge, this study is the first systematic review on the topic of VRCL. As a means to provide the relevant information, this review addresses the following four research questions.

1. What skills and competences have been trained with use of VRCL (and what should a VRCL environment provide to train these)?

2. What domains and disciplines have been addressed (and why)?

3. What systems have been developed and/or established?

4. What empirical knowledge has been established (and with what methods and/or study designs)?

This section discusses the process of collecting the relevant studies for this literature review. In particular, the inclusion and exclusion criteria, databases and methods used are described.

2.1 Identification

The systematic review used two databases: Scopus and Web of Science. The search query contained the following key elements: 1) collaborative interaction, 2) VR, 3) education, training and learning, 4) simulations of a three-dimensional nature, 5) empirical data and 6) the use of a system (application or prototype). As such, the following search string was used in both databases:

[collaboration OR cooperation OR collaborative OR cooperative OR collaborate OR cooperate] [AND] ["virtual reality” OR “mixed reality” OR “extended reality"] [AND] ["3D” OR 3d OR 3-D OR 3-d OR threedimension* OR three-dimension* OR “three dimension*" OR CGI OR “computer generated” OR “computer-generated” OR model* OR construct*] [AND] [evaluat* OR data OR result* OR observ* OR empiric* OR trial* OR experiment* OR significan* OR participant* OR subject*] [AND] [education OR training OR learning OR university OR school OR vocational] [AND] [system* OR prototyp* OR application* OR program*]

To be considered suitable, papers had to meet five specific inclusion criteria. Firstly, an article had to discuss collaborative or cooperative interaction between human users of a virtual, three-dimensional simulation. Secondly, the article had to include and discuss Virtual-, Augmented-, Mixed Reality (MR) or Extended Reality (XR) as a three-dimensional simulation of a physical space or object(s). While this review focuses on VR for CL, mediums such as AR, MR and XR were included in this search for two reasons. On the one hand, definitions for these mediums appear to overlap to such an extent (with some even considering them too vague and ambiguous ( Tovar et al., 2020 )) that ‘pedagogical advantages of either technologies are [considered] comparable’ ( Sims et al., 2022 ). On the other hand, the mediums in question do not always get defined as separate ones, but rather as different points on one spectrum, commonly referred to as the virtuality continuum, in which ‘“reality” lies at one end, and “virtuality” […] at the other, with Mixed Reality […] placed between’ ( Scavarelli et al., 2021 ). As such, the decision was made to include these mediums, so as to ensure that no pedagogical advantages of VR would be excluded. The third inclusion criterium required an article to include an empirical study (i.e., containing qualitative or quantitative data) for it to be considered suitable. For the fourth and fifth criteria, an article had to contain an educational objective or goal (for human entities) and discuss a system used for educational purposes (for human entities) in order to be eligible.

Additionally, studies would be disqualified from the literature review if they 1) only described a patent, 2) only contained a summary (review) of a conference, 3) only consisted of a literature review, 4) were not accessible to the authors of this study, 5) were not available in English, 6) were a duplicate or a version, edition or release of an older study that already had been included or 7) did not specifically state the number of participants of any experiment involved in the study.

The search query resulted in 1,058 publications for Scopus and 845 studies for Web of Science, resulting in a total of 1,608 studies after duplicates were removed. Using the inclusion and exclusion criteria to filter out ineligible articles (initially based on title and abstract, then on full text), this review resulted in 139 articles analyzed. Results and details of the process (which followed the guidelines of the PRISMA method ( Moher et al., 2009 )) can be seen in Figure 1 . Appendix A shows the complete list of all 139 articles.

FIGURE 1 . Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram of the screening process.

To examine reliability of the selection process, five additional coders screened a random sample of 50 studies individually (10 per coder) using the inclusion and exclusion criteria. After comparing and discussing results, inter-rater reliability (between the first coder and the five coders) was calculated using a Kappa-metric, resulting in a moderate level of agreement of 0.77 ( McHugh, 2012 ) (results can be found in Supplementary Table B1 ).

A taxonomy ( Figure 2 ) was created to help classify all 139 articles. With this review’s research questions in mind, three vital topics were established to function as main categories for the coding process: education, system and evaluation (illustrated in column C1 in Figure 2 ). For RQ1 and RQ2, the first category, education, was established to extract information from the articles, concentrating on six classes. Similarly, information necessary to answer RQ3 was collected by coding attributes related to the second category, system, which included eight classes. Focusing the coding on elements related to the third category, evaluation (with five classes), allowed for extraction of relevant information required to answer RQ4. After the relevant categories, classes (visible in column C2 in Figure 2 ) and attributes (visible in column C3 in Figure 2 ) were decided upon, the classification hierarchy in Figure 2 was constructed, partially based on scientific literature ( Bloom et al., 1956 ; Schreiber and Asnerly-Self, 2011 ; Motejlek and Alpay, 2019 ), to provide assistance during the coding process. For an in-depth description of the motivation behind this classification hierarchy, please see Supplementary Appendix C . While the required information for some of these attributes could easily be inferred directly from each study, other attributes required the first coder to deduce which attributes were applicable.

FIGURE 2 . Classification hierarchy used for coding, including percent agreement ( p a ) and Cohen’s kappa (K) between first and second coder on the right.

To assess reliability of the first coder, a second coder classified articles with the taxonomy ( Supplementary Table D1, D2, D3 ). Inter-rater reliability between the two coders for 30 randomly selected studies was 0.60 (with a percent agreement of 0.85), considered a moderate level of agreement ( McHugh, 2012 ). Additionally, Figure 2 shows the inter-rater reliability for each individual class.

3 Descriptive results

In this section, discussion of descriptive results is divided into three sections according to the structure of the taxonomy. An overview of all results (according to the taxonomy) can be found in Figure 3 .

FIGURE 3 . Results of coding of data found in the literature, according to the taxonomy.

3.1 Education

As a first dimension, elements related to education were analyzed. A majority of the selected articles focused on VRCL in tertiary education (i.e., university), discussing possible uses for students. Educators providing support (e.g., scaffolding) for learners proved most prominent, though not all studies discussed this topic. While a wide selection of educational domains were discussed, computer sciences and social sciences were the most popular fields. Most studies specifically focused on synchronous collaboration. Prevalent among learning paradigms and educational approaches were problem-based learning (PBL) and constructivism. The specific results related to this dimension are found in Table 1 .

TABLE 1 . Distribution of Education-related attributes.

In contrast to the high number of articles focusing on tertiary education (64.0%), primary education was central in 10.8% while 5.0% discussed VRCL in secondary education. A small percentage of studies (6.5%) focused on types of learners outside of formal education (e.g., on-the-job training). In relation to the educators, a little over half of the studies reported on educators supporting the learners by providing varying degrees of scaffolding (55.4%). For 20.9% of cases, educators provided presentations and lectures inside the VE, providing a more passive learning experience. On a broader scope, the studies showed a wide variety of educational domains and fields of expertise to which VR was applied. While approximately a quarter of studies reviewed (25.9%) reported use of VRCL for education, specific domains that were often discussed included computer science, robotics, ICT and informatics (12.2%), social sciences (11.5%) medical fields (9.4%) and engineering (8.6%).

Also shown in Table 1 is the appearance of different types of social learning: 62.6% of studies reviewed discussed synchronous (collaborative) interaction, while in comparison a much lower 18.0% discussed asynchronous (cooperative) interaction. For a 10th of the studies, an expert-novice type of social learning was apparent (9.4%). On the topic of educational approaches and learning paradigms, 29.5% of articles did not seem to discuss any specific approaches. Among those that did, constructivism and PBL were featured substantially (33.1% and 41.0%, respectively), while paradigms such as experientialism, situated learning and distributed cognition were discussed less frequently. Other educational approaches, discussed in 35.3% of articles, included self-regulation and shared regulation (e.g., Al-Hatem et al., 2018 ) as well as cognitive apprenticeship (e.g., Bouta and Retalis, 2013 ). Looking at the learning goals and outcomes, the cognitive domain proved to be popular (50.4%), whereas affective and psychomotor domains were featured much less (7.9% and 5.0%, respectively). Other goals and outcomes included general student engagement (discussed in 31.7%) and support of collaboration amongst learners (60.4%).

The second dimension took a closer look at systems used in the studies, including aspects related to the hardware used (e.g., devices, types of control) as well as users’ interaction with VEs (e.g., degree of virtuality, virtual embodiment). A majority of the studies reviewed did not use VR technologies such as HMD-based VR (HMD VR), but instead focused on monitors and displays when discussing VRCL. Most studies chose general purpose controls (e.g., mouse and keyboard) over more advanced hardware such as positional tracking. A majority of studies provided their participants with full-body embodiment (e.g., avatars) and the ability to manipulate virtual objects while inside the VEs. Approximately a quarter of studies used systems for edutainment purposes (i.e., learning by having fun), while system use for training or therapeutic purposes was less common. Table 2 shows these results in detail.

TABLE 2 . Distribution of System-related attributes.

Results showed a clear preference for 3D (non-HMD) simulations, i.e., a virtual simulation of a (physical) environment projected on a surface or display that is not a Head-Mounted Display (and, as such, is considered less immersive): this degree of virtuality was far more prominent in the reviewed studies (78.4%) compared to the lesser implemented AR/MR (16.5%) and HMD VR (7.2%). The hardware used in these studies reflected this: a large amount (89.2%) implemented flat-surface monitors and displays to present VRCL environments. These studies commonly used desktop computer set-ups that included a keyboard, mouse and monitor, though in the case of AR and MR, surface-based mobile devices were often used. When using the system in a larger setting (i.e., larger group size), studies utilized projector-based (but still flat) surfaces to display the VE (e.g., Bower et al., 2017 ). In some cases, several types of these flat-surface displays were being used in different phases of a study (e.g., Nuñez et al., 2008 ). Cases that used CAVE systems (3.6%) included ImmersaDesks, CAVE-like devices that derive from the original CAVE systems. Studies that involved HMD VR used devices like the Oculus Rift and HTC Vive, while studies revolving around AR and MR implemented devices like the HoloLens. Some studies involved multiple devices to compare effects based on the difference (e.g., monitor-based vs HMD VR, as discussed in Vallance et al., 2015 ) while others discussed implementation of HMD VR and AR-related devices as possible future directions without using these in their experiments. With regard to user interaction, studies that implemented general purpose controls used simple computer keyboard and mouse, though some cases also involved video game controllers such as the Nintendo Wiimote and Nunchuck ( Li et al., 2012 ). Apart from the more default specialized controls such as 3DoF and 6DoF controllers or mobile device-based touch screens, studies also discussed a wide variety of other tools in this category, including multi-touch tabletops, haptic feedback devices, Xbox Kinect and gesture-sensing data gloves. While scarce, gaze control and positional tracking (15.1% and 11.5%, respectively) was primarily found in studies that used (mobile-based) AR and HMD VR, though some studies also provided these through devices such as the HoloLens or as part of a CAVE system.

Of the studies examined for this review, 55.4% discussed (self-developed) prototypes, while 44.6% used (pre-existing) applications. The most prominently-mentioned engine for prototypes was Unity, with % (of 77 studies) using it. Concerning the ones that used applications (62 of 139), more than half discussed VE application Second Life (%), while open-source VEs OpenSimulator and Open Wonderland were used in smaller numbers (% and %, respectively). In regard to the intended function of systems used, the majority of articles described a strictly educational one (58.3%) and revolved around implementing these systems in educational contexts as well as using them to facilitate collaborative learning. Studies that used systems to both educate and entertain (22.3%) tended to focus on game-based learning and serious games, though some cases also discussed video games originally not intended for educational purposes (e.g., World of Warcraft ( Kong and Kwok, 2013 ), Minecraft ( Mørch et al., 2019 )). When training purposes were mentioned (17.3%), this often indicated the use of VEs to train specific expertises, such as liver surgery or aircraft inspection. Rare cases where a system was used for therapeutic purposes (just 2.2%) included use of VRCL to teach social skills to patients with autism ( Ke and Lee, 2016 ) or to train physical activities amongst elderly ( Arlati et al., 2019 ).

Motivation behind studies’ choices for the size of collaboration differed between experimental reasons (e.g., a limited number of participants), pedagogical reasons (e.g., using pairs to better stimulate personal social interaction between members compared to larger groups) and reasons related to the systems (e.g., limited hardware availability). Small groups proved to be the most used group size, with 37.4% describing groups of between three and nine members. Pairs were used in 22.3% of studies. Motivations behind pairs included focus on expert-novice interaction and system capabilities (e.g., support for two users maximum). Articles that described larger groups (ten or more members) generally had entire classes of learners interact with system (15.1%).

Apart from a small number of studies that did not provide sufficient information on the matter, virtual embodiment of the users was featured prominently. In cases where physical attributes were virtually represented by (imagery of) tools (18.0%), the VRCL environment was often implemented for specific training of certain expertises. In general, partial virtual embodiment appears in first person, HMD VR (for example, when only the user’s hands are made visible); while scarce (3.6%), studies that displayed partial virtual embodiment provided some interesting examples outside of HMD VR. Examples of partial embodiment included a detailed 3D face to focus on emotional and social expressions ( Cheng and Ye, 2010 ) and using controllable, flat-surfaced rectangles in a 3D environment on which users’ real-life faces were projected via webcam ( Nikolic and Nicholls, 2018 ). Full-body embodiment proved to be the most popular, with 67.6% of studies using systems that provide users complete (full-body) virtual representation. To a degree, the relatively high number of studies that present full-body embodiment can be explained by the systems that were implemented; applications such as Open Simulator and Second Life provide users with customizable avatars, making a full-body virtual embodiment a default feature. In some cases, however, studies specifically examined the effects of virtual embodiment, such as Gerhard et al. (2001) examining possible influences of different avatars on users’ sense of presence. On the topic of user influence on VEs, a little more than half the studies (53.2%) used systems that allowed (some degree of) virtual object manipulation, whereas approximately a quarter of the studies (26.6%) also provided users the tools to manipulate actual content of the VRCL environment. In 16.5% of studies, the system only allowed users to be visibly present inside the VE, while only 3.6% did not provide sufficient information on the matter.

3.3 Evaluation

For the third dimension, the selected articles were analyzed on how they evaluated applying VRCL. Articles frequently concentrated on evaluation of the system(s), with a higher number of them using self-report evaluation methods. Study design of the studies shows a similar result: pre-experimental study design (typically used for preliminary testing of systems) was regularly implemented, with surveys being a popular method of collecting data. While the number of participants was diverse, roughly half of studies reviewed used a sample size between 1 and 25 participants. The majority of articles discussed positive outcomes, whereas only a small amount featured negative results. Detailed results are displayed in Table 3 .

TABLE 3 . Distribution of Evaluation-related attributes.

The majority of studies focused on evaluating a system’s effectiveness when using it in educational settings (71.2%). These studies concentrated on the system’s capacity to support collaboration between learners. Other topics of discussion were student interest in the system and how the system can facilitate learning. Whenever studies examined processes (34.5%), evaluation would be centered around attempts to understand how group interaction materializes in these environments. This included how learners resolve social conflicts ( Cheong et al., 2015 ) and examining how co-presence (e.g., Kong and Kwok, 2013 ) and PBL take shape in VRCL environments. 35.3% of articles discussed learning outcomes after participants interacted with the system. The few situations where the above three attributes did not apply (3.6%) included a study that aimed to develop design guidelines ( Economou et al., 2001 ) and a study primarily interested in the teacher’s role when learners interact with VEs ( Lattemann and Stieglitz, 2012 ).

Most studies collected self-reported data from their participants (85.6%), while over half used behavioral methods to obtain tracking and observational data (59.0%). Articles that reported on knowledge- and/or performance-based assessments (20.9% of studies) often used pre- and post-tests to acquire their data, while only one appeared to use physiological data, tracking participants’ heart rate (0.7%). A notable number of articles (79.9%) implemented pre-experimental design in their studies. Some of these were case studies, applying VEs to educational settings (e.g., Terzidou et al., 2012 ), while others performed pilot studies to establish a first impression of the effects of a system on specific pedagogical situations (e.g., examining how VE-based application OpenSimulator influences Transactive Memory Systems amongst learners ( Kleanthous et al., 2016 )). Quasi-experimental- (13.7%) and true experimental designs (5.8%) were used scarcely, while only 2 out of 139 studies (1.4%) performed an experiment with single-subject design. With respect to non-experimental and descriptive designs, 84.9% of studies implemented a survey-based design, whereas a little over half used observational designs to collect data (56.1%). In some cases, comparative and correlation designs were implemented (7.9% and 15.8%, respectively).

Table 3 also reveals that approximately half of the studies sampled between 1 and 25 participants (53.2%), while around a quarter (26.6%) used a sample size between 26 and 50 participants. For 13.7% of articles, between 51 and 100 participants were used, whereas only 6.5% discussed using more than 100 participants for collecting data. In terms of outcomes, around half of the studies concluded that their system(s) seemed positive and promising (53.2%), while 17.3% draw positive conclusions based on significant outcomes from statistical hypothesis testing. Negative outcomes were scarce, with only 2.2% of the studies reporting negative results. Mixed outcomes were reported for 7.2% of the studies, whereas 20.1% discussed results that were inconclusive, showed no effect or reported outcomes on which positive and negative effects are not applicable.

4 Qualitative results

In general, the literature reviewed for this paper shows a positive attitude towards the use of VR to support and enhance CL. However, the results quickly make it apparent that the methods of applying VR to educational fields to support and enhance CL can vary greatly amongst the studies examined here. In order to acquire a general understanding how these studies have attempted to support and enhance CL using VR, this section will discuss qualitative results established. The rest of this section will be divided into sub-sections, each focusing on discussing results related to one of the four research questions of this literature review.

4.1 Skills and competences trained with VRCL

A number of elements can be identified regarding skills and competences trained with VRCL. Based on the skills and competences discussed in the reviewed literature, five categories were established for this study with the intention to provide a concise overview. These categories, including examples of each category, can be viewed in Table 4 .

TABLE 4 . Skills focused on in the reviewed literature.

For the types of skills and competences shown in Table 4 to be trained effectively, a VRCL environment requires a number of features that support the learners in learning these abilities. Based on the information provided by the reviewed literature, nine required features and design parameters of VRCL can be identified. First, virtual embodiment plays an important role in how learners view themselves and each other inside the VE, impacting learning outcomes and collaborative behavior by providing a sense of awareness and belonging ( Edirisingha et al., 2009 ; McArdle and Bertolotto, 2012 ). Second, efficient communicational tools are essential for effective collaboration: verbal (audio) communication is crucial ( Economou et al., 2001 ; De Pace et al., 2019 ), though additional modalities such as haptic technology can further enhance collaboration ( Moll and Pysander, 2013 ). Third, usability and accessibility should be taken into consideration: VRCL systems should be accessible to all levels of technical skills as differences negatively affect group cohesion and learning between group members (Y. Chang et al., 2016 ; Denoyelles and Kyeong-Ju Seo, 2012 ). Fourth, learners’ perceived usefulness of the VE also affects group cohesion; factors such as awareness, presence and social presence appear to significantly influence this perceived usefulness ( Denoyelles and Kyeong-Ju Seo, 2012 ; Yeh et al., 2012 ). Fifth, the ability to interact with elements inside the VE are considered key: to optimize learning outcomes, learners must have the option to manipulate elements inside the VE (e.g., virtual objects or virtual tools) in a seemingly natural and intuitive way ( Vrellis et al., 2010 ; Bower et al., 2017 ). Sixth, academic efficacy can be achieved if tasks inside the VE are designed around its educational, collaborative objectives, especially when designed for equal input from all learners in a group ( Wang et al., 2014 ; Nisiotis and Kleanthous, 2019 ). Seventh, educators should be ready to provide support, motivation and moderation of collaboration while learners interact inside the VE ( Lattemann and Stieglitz, 2012 ; Bower et al., 2017 ). However, the eighth feature, a level of autonomy, is equally important for each individual learner, not just in terms of independence from the educators, but more importantly from each other, as this allows them to provide different points of views as well as to explore multiple representations, thus improving CL ( Hwang and Hu, 2013 ). Ninth, implementation of VRCL should make sure to primarily support socialization inside the VE, as underestimating the importance of socialization might lead to features of VR obstructing rather than facilitating CL ( Chang et al., 2009 ).

Surprisingly, only a small number of the literature reviewed focused on goals related to the affective domain (7.9%). With some calling VR the “ultimate empathy machine” ( Rueda and Lara, 2020 , p.6), the medium’s ability to induce emotions has been prominently discussed and studied. Not only has VR been shown to indeed be capable of enhancing empathy amongst users ( Herrera et al., 2018 ), with some even arguing it to be more effective than traditional empathy-shaping methods ( Liu, 2020 ), studies have also suggested it to be an effective tool to offer a uniquely different level of understanding ( de la Peña et al., 2010 ). This would suggest that VR’s ability to create a better understanding of different group members’ points of view could in turn support collaboration between learners.

Similarly, even less literature reviewed focused on goals related to the psychomotor domain (5.0%). Prior studies have been positive and hopeful regarding VR to expand the possibilities of physical training ( Pastel et al., 2020 ). Interestingly, technical features such as positional tracking even seem to be effective in predicting psychomotor outcomes ( Moore et al., 2021 ), which could prove useful for domains that specifically focus on expert-novice training in primarily physical tasks (e.g., certain types of engineering). However, positional tracking, not unlike psychomotor outcomes, is only discussed sparingly (11.5%) in the literature reviewed.

An interesting observation in relation to the evaluation methods used in the scientific literature is that only 1 out of 139 articles used physiological measures. As suggested by research, physiological synchrony between group members can serve as an effective indicator for the quality of interpersonal interaction between them (with a higher physiological synchrony correlating with a higher interaction level) ( Liu et al., 2021 ). Furthermore, physiological measurements can be used to identify multiple predictors related to education and training, including the quality of collaboration between group members ( Dich et al., 2018 ). Additionally, visualizing physiological results of each member of a group to the others in real-time during collaboration has shown to have a positive effect on the empathy levels and cohesion of the group, further suggesting how collaboration between learners could benefit from physiological measures ( Tan et al., 2014 ). Considering VR’s visual characteristics as well as research arguing that physical signals such as electroencephalogram (EEG) can conveniently and unobtrusively be tracked during use of HMD VR ( Tremmel et al., 2019 ), future research on VRCL could prove fruitful in terms of training collaborative skills and competences via use of physiological-based information.

4.2 Disciplines focused on regarding VRCL

When looking at the most prominently-featured domains in the literature reviewed (as shown in Figure 4 ), examining what motivated researchers to study VRCL in the field of 1) education, 2) computer science, robotics and informatics and 3) social sciences can provide an understanding of VRCL’s role in these different disciplines.

FIGURE 4 . Results of the educational disciplines focused on in the reviewed literature.

For the field of education, some studies focus on the potential behind VRCL, intending to discover what it can mean for the development of cognitive and technical skills ( Franco and De Deus Lopes, 2009 ). Other studies focus on possible learning gains, examining how knowledge gained in VEs transfers to the real world (i.e., how learners apply outcomes in VEs to situations in actual reality) or attempting to facilitate this transfer by implementing elements of both ( Carron et al., 2013 ). In certain cases, articles specifically examine VEs’ effects on collaboration and how VR can be used to reinforce CL (e.g., Tüzün et al., 2019 ), whereas others aim to determine if existing educational paradigms such as constructivism can be applied to VRCL environments and, if so, how that affects group knowledge gain between learners ( Girvan and Savage, 2010 ). Together, these studies present a general motivation to discover what VRCL can mean for education and where its potential may lie.

For computer science, robotics and informatics, use of VRCL can be summarized in two motivations: 1) innovate these domains and 2) create a learning community. In the first case, researchers intend to utilize the affordances VRCL environments have to offer to further advance fields such as computer science, which have been criticized in the past for using two-dimensional learning platforms and oral-based teaching methods ( Pellas, 2014 ). With VEs, educators can provide learners realistic yet illusionary worlds that are flexible, customizable and even allow for detailed statistics on learners’ performance ( Champsas et al., 2012 ). In the second case, reviewed articles vocalize a desire to use VRCL to provide learners purposeful collaborative activities that create a sense of belonging to a learning community, using aspects such as awareness, presence and different methods of communication to motivate learners in these fields to work together closely ( De Lucia et al., 2009 ).

In similar fashion, social studies appears to be interested in how socialization between learners is manifested inside VRCL (e.g., Edirisingha et al., 2009 ). Some articles go further, studying how VRCL can support socialization: Molka-Danielsen and Brask (2014) suggest that presence, awareness and belonging allow for communication, negotiation and trust between learners, elements deemed necessary for completing collaborative tasks. Other studies focus on specific characteristics of socialization, such as how gender could affect social interaction and group cohesion inside VEs ( Denoyelles and Kyeong-Ju Seo, 2012 ). Collectively, these articles show a desire to understand how elements related to socialization transfer to VRCL, as well as how these environments can sustain and even enhance those elements.

4.3 Systems developed and/or established for VRCL

The results related to systems used show that there is quite a disparity between use of HMD VR and that of non-HMD VR. Almost 80% of systems implemented non-HMD VR, with AR/MR and HMD VR implemented far less frequently (16.5% and 7.2%, respectively, as illustrated in Figure 5 ). Almost 90% of studies described the use of flat-surface monitors and displays, which, when compared to the 10.8% of studies that used HMD devices, further highlights the low use of HMD VR in the literature reviewed (see Figure 6 ).

FIGURE 5 . Results of the degree of virtuality of systems discussed in the reviewed literature.

FIGURE 6 . Results of the hardware used in the reviewed literature.

The lack of representation of HMD VR in these articles is somewhat surprising, considering this type of virtuality and hardware is commonly associated with the medium of VR ( Dixon, 2006 ; Bonner and Reinders, 2018 ; Jing et al., 2018 ). The statement that research into application of VR to the field of education lacks a focus on HMD VR, however, is not uncommon ( Sousa Santos et al., 2009 ; Scavarelli et al., 2021 ), thus begging the question: why is it underrepresented in the reviewed literature?

One possible explanation could be that HMD VR is known to be difficult to apply to educational settings because of its high costs ( Olmos et al., 2018 ). Some of the articles analyzed for this review were published in the late 90s; while HMD VR technology was already available in those times, devices were more expensive and less technologically advanced compared to the technology that is available now ( Mehrfard et al., 2019 ; Wang et al., 2022 ). Furthermore, the technical skills necessary to implement VR properly in educational settings can prove challenging ( Jensen and Konradsen, 2018 ). Since collaboration involves multiple people, difficulties related to accessibility could be more severe when applying VR to a larger group of learners. Another possible reason is the health risks associated with the technology: HMD VR is often connected to motion sickness and cybersickness ( Olmos et al., 2018 ; Yoon et al., 2020 ). A third reason refers to the general lack of pedagogy on the topic of HMD VR: while the medium’s potential for education is often discussed, general guidelines as to how it should be applied efficiently to educational settings ( Cook et al., 2019 ; Zheng et al., 2019 ) as well as an understanding of how learning mechanisms operate inside VR environments ( Smith, 2019 ) are missing. Naturally, the small size of research done on VR and CL exacerbates this lack even further when specifically discussing VRCL. A possible fourth reason that is more closely tied to this particular literature review is that, despite its popularity in research, HMD VR appears to still lack empirical evidence of its educational value ( Sousa Santos et al., 2009 ; Makransky et al., 2019 ; Radianti et al., 2020 ), which, considering this review’s focus on empirically-based knowledge, could explain its scarcity.

The low representation of HMD VR and high representation of non-HMD VR could be related to the ongoing discussion about what defines VR and how it differs from VEs, as discussed in-depth by Girvan, (2018) . Girvan argues that some use terms synonymously with VR and/or VEs, while others use these same terms to classify different types of VEs, thus creating a fragmented understanding of what these are (and what they are not). Girvan’s point is reflected in the reviewed literature of this paper: while some studies identify Second Life as a “virtual environment” or “virtual world” (e.g., Terzidou et al., 2012 ), others refer to it as “virtual reality” (e.g., Sulbaran and Jones, 2012 ). To prevent further confusion with technologies with similar technical features, Girvan suggests to conceptualize VEs as ‘shared, simulated spaces which are inhabited and shaped by their inhabitants, who are represented as avatars [that] mediate our experience of this space as (…) we interact with others, with whom we construct a shared understanding of the world at that time’. VR, then, should be defined as ‘a technical system through which a user or multiple users can experience [such] a simulated environment’ ( Girvan, 2018 ).

Apart from causing a fragmented understanding of the terms in the literature, different interpretations of VR and VEs also lead to HMD VR and non-HMD VR being described as one and the same thing under the moniker of “virtual reality”. Though this may seem a trivial dispute about labels, treating these two types as identical will lead to misconceptions regarding both, as HMD and non-HMD VR contain different benefits and limitations when applied to education. While some studies showed no differences between the two in terms of specific learning outcomes (e.g., spatial- ( Srivastava et al., 2019 ) and language learning (J. Y. Jeong et al., 2018 )), other research highlighted several differences between HMD and non-HMD. Compared to non-HMD, HMD VR has shown to provide a much higher sense of embodiment, which in turn is hypothesized to lead to higher performances, in particular in psychomotor skills ( Juliano et al., 2020 ; Saldana et al., 2020 ). Similarly, HMD VR appeared superior to computer screens in terms of arousal, engagement and motivation in learners ( Makransky and Lilleholt, 2018 ). In contrast, however, Makransky et al. (2019) reported overloads and distractions caused by HMD VR, leading to poorer learning outcomes compared to non-HMD, a sentiment shared by Parong and Mayer (2021) , who described HMD VR to cause high affective and cognitive distractions. Amati and McNeill (2012) even argue that the difference between HMD and non-HMD VR (and in particular how the two are interacted with by users) have severe implications for teaching and practice.

With all of the above in mind, the low representation of HMD VR in the literature examined for this review can be interpreted in two ways. On the one hand, the underutilization underlines that HMD VR is not being used to its full potential and could very well hold much more promise for the field of education and CL. On the other hand, the low use of HMD VR could suggest that implementation of HMD VR in education and/or CL is, in fact, not worth the trouble it brings with it. Whether HMD VR is a benefit or a burden, then, arguably depends on three important elements: 1) the goals (i.e., what skills and/or competences are supposed to be trained), 2) the setting (i.e., the disciplines and fields to which it is applied), and 3) the affordances of VRCL (and to what degree these conform to the goals and setting).

4.4 Empirical knowledge established regarding VRCL

When summarizing the outcomes of the 139 articles, 70% of the studies reviewed displayed a positive attitude towards the application of VRCL to education. While a relatively low number (approximately 25%) presented statistically significant outcomes, this does illustrate a strong optimism amongst those studying VRCL environments in different fields of education as described in prior literature reviews on the topic. This could also explain the high number of studies that deployed pre-experimental study designs: with VRCL being a relatively new addition to the world of CSCL, as well as one that continues to rapidly advance because of the technology behind it, many seem enthusiastic and eager to see what promises VRCL holds when used in different fields and with different types of learners.

Regarding affordances discussed in the reviewed literature, several features are identified. First, VRCL appears an efficient tool to engage learners and to motivate them to study and learn. The ability to customize VRCL environments and their content provides learners more personalized experiences that better suit their personalities and attitudes, thereby enhancing the motivation to learn on both an individual and group level ( Arlati et al., 2019 ). Furthermore, VRCL’s immersive qualities tend to make the experiences more engaging for learners, encouraging them to engage in presentations and demonstrations as well as to communicate and collaborate with each other ( Avanzato, 2018 ).

The second affordance identified VRCL as a great tool for distance learning and remote collaboration. VEs provide a method for learners and educators to work together and collaborate despite distances. In comparison to other media, however, VRCL brings with it a high sense of immediacy (i.e., ‘verbal and non-verbal behaviors that give a sense of reduction of physical and psychological distance between the communicators’), which in turn presents an increased perception of learning ( Edirisingha et al., 2009 ). Additionally, VRCL’s immersive qualities and high presence allow for environments capable of simulating training as preparation for real-life experiences ( Al-Hatem et al., 2018 ) that simultaneously promote active participation and social interaction ( Mystakidis et al., 2017 ) in a setting that feels personal despite distances between learners ( Desai et al., 2017 ). In certain cases, such as education for learners with physical disabilities, learners and educators even considered connectivity to be more accessible and easier than real-life equivalents ( Aydogan and Aras, 2019 ), illustrating that VRCL environments can potentially go beyond simply being a replacement. To effectively support the distance learning and remote collaboration, however, design of the VEs should focus on providing learners a sense of 1) presence, 2) awareness and 3) belonging to the group ( Molka-Danielsen and Brask, 2014 ).

Thirdly, the literature reviewed suggests that VRCL environments are effective spaces to support multi- and interdisciplinary learning and collaboration. The ability to customize VEs, adapting to suit users’ needs, prevents them from being restricted to just a single specific subject field. This in turn allows educators to change the environments to accommodate many different subject fields and topics so as to make sure that learners from different backgrounds can collaborate with each other undisturbed ( Bilyatdinova et al., 2016 ). Moreover, it seems that VRCL environments made some of the literature studies reviewed realize the importance of interdisciplinary collaboration in the learning process ( Franco et al., 2006 ; Nadolny et al., 2013 ).

The fourth affordance identified might be an unsurprising but nonetheless important one: VRCL seems to be a tool for the development of social skills. While identity construction and projection through virtual embodiments can be complex for learners (depending on their technical skills), VRCL is found to facilitate social presence and foster socialization ( Edirisingha et al., 2009 ). VRCL’s customizability allows learners to integrate personal preferences and identity expressions into processes inside the environment (e.g., through their virtual embodiments), in turn mediating identity and norm construction for real-life social settings ( Ke and Lee, 2016 ). Vital social skills, such as the ability to identify and manipulate basic emotional states, can be taught and trained using VEs, improving learners’ socialization, communication skills and emotional intelligence ( López-Faican and Jaen, 2020 ). Learners’ prior experience with VEs, however, should not be underestimated, as a difference in familiarity with VRCL environments has been shown to impact collaboration ( Bluemink et al., 2010 ).

Fifth, VEs appear fitting for CL-related learning paradigms and educational approaches. Some studies specifically focus on examining to what degree VRCL environments are applicable to paradigms such as constructivism, socio-constructivism and constructionism (e.g., Girvan and Savage, 2010 ; Pellas et al., 2013 ; Abdullah et al., 2019 ), concluding that these indeed go well together. Other studies, however, focus on theories and methods commonly associated with these paradigms. In particular, experiential learning and PBL seem appropriate for VRCL environments. VEs allow for safe, consequence-free learning for exploring, experiencing and practicing without any real-life risks ( Cheong et al., 2015 ; Le et al., 2015 ), making it suitable for experiential learning. Moreover, VRCL’s immersive qualities seem to support and even elevate experiential learning strategies such as roleplay and improvisation, providing learners close to real-world experiences in a controlled environment ( Jarmon et al., 2008 ; Ashley et al., 2014 ). In the case of PBL, each individual learner can use different tools inside VRCL environments to illustrate and represent ideas and suggestions to the rest of the group. Considering that VEs seem great tools for conceptual learning because of their customizability and visual nature ( Brna and Aspin, 1998 ; Griol et al., 2014 ), learners can use these features to explain their point of view in ways that they otherwise could not. As a result, learners appear to become more active and effective in sharing ideas, joint problem solving and the co-construction of mental models when working in groups inside VRCL environments ( Rogers, 2011 ; Hwang and Hu, 2013 ).

Returning to the topic of disparity between HMD and non-HMD VR represented in the reviewed literature, as well as both being discussed as one and the same “Virtual Reality”, an important question to ask is whether the affordances identified here are transferable between the two. HMD and non-HMD VR differ in several ways: they are interacted with differently, face different obstacles when applied to education and appear to have different learning outcomes based on different educational settings.

With the definitions of VEs and VR as given by Girvan (2018) as a frame of reference, however, an answer can be given regarding the transferability of these affordances between HMD and non-HMD VR. Both HMD and non-HMD VR should be considered tools, technical systems through which users can virtually enter VEs, i.e., shared simulated spaces in which they can interact with the environment as well as each other. As such, the affordances described in this paper do not revolve around the tools used, but that which they provide access to: the VRCL environments. Simultaneously, which tool is used to access these VRCL environments can in turn affect both the interaction and the outcome of users’ experiences with VEs. For example, HMD VR might offer more effective development of social skills compared to non-HMD VR, considering the former provides a higher sense of embodiment and, in extension, more intuitive and expansive methods of expression. If, however, cognitive learning outcomes are the most important educational objective, non-HMD VR could be a better option, considering HMD VR’s tendency to cause affective and cognitive distractions. This, then, reflects the aforementioned statement regarding HMD VR being a benefit or a burden. While affordances of VRCL environments apply to both HMD and non-HMD VR, the effect of these affordances depend on 1) the goals, 2) the setting and 3) which affordances of VRCL are most vital to the first two elements. As such, the choice between non-HMD VR and HMD VR should be made depending on those three elements.

5 Conclusion and future research

With current research on the topic being scarce while the demand for remote collaboration and distance learning keeps increasing, this literature review intends to study how VR has been (and can be) used to support and enhance CL. To achieve this, it attempts to answer four research questions regarding prior research on VRCL: what skills and competences have been trained with VRCL and what does VRCL provide in these scenarios? To what educational domains has VRCL been applied and why? What systems have been used for VRCL? And what empirical knowledge has been established regarding VRCL?

This paper identifies five types of skills and competences commonly trained with the use of VRCL. Furthermore, a number of features and design principles are identified in terms of what these environments should offer for these skills to be developed. Educational fields and domains appear to be interested in VRCL because of a desire to innovate, to form communities, to support remote collaboration and to enhance socialization skills of learners. In terms of technology, systems used for VRCL-related purposes appear to predominantly focus on monitor-based (non-HMD) VR and mouse-and-keyboard controls, contrasting what VR is commonly associated with (e.g., HMD VR, specialized controls involving gaze control and positional tracking). This study perceives a general optimism present in the literature reviewed regarding the use of VR to support and enhance CL in learners. Additionally, a number of affordances of VRCL are described, though it is of importance to note that these affordances could differ in strength depending on which type of VR (i.e., non-HMD or HMD) is used.

While the literature on VRCL reviewed for this paper is diverse, it suggests that Virtual Reality can be an effective tool for supporting and enhancing Collaborative Learning. This diversity, however, also highlights that pedagogies of VRCL are lacking, with studies showing many different and contrasting approaches to applying VR to their respective fields for the support of CL. In order to see VR become more adopted as an educational tool for collaborative purposes, pedagogies should be clearly structured, highlighting similarities and differences in regards to both the technologies used and the domains they are used in. As such, this paper proposes a number of suggestions for future research. First, the difference between hardware used in the literature reviewed and the state-of-the-art of VR suggests that further examination of differences between non-HMD and HMD VRCL, both in terms of affordances as well as challenges and obstacles, could lead to a better understanding of VRCL’s potential. Second, despite the advantages VR has for development in affective and psychomotor skills, the scientific literature on VRCL shows only minor focus on these domains. This study argues that CL would benefit from both these domains being featured more prominently and as such encourages more research into these matters. Third, this paper suggests that research into VRCL focuses on using study designs and evaluation methods that are less frequently (or barely) featured in the reviewed literature. While the repeated and dominant use of pre-experimental study design is understandably meant to identify the potential behind the technology, the domain of VRCL (and, in extension, research on VR in education) would benefit from more true experimental design. Additionally, considering that the use of physiological data for evaluation methods appears to be unexplored terrain, this paper suggests that future research into VRCL implements these types of methods.

Author contributions

NvM: Main author VvW: Co-author and coder W-PB: Co-author and supervisor MS: Co-author and supervisor. All authors contributed to the article and approved the submitted version.

This project has been funded by the Leiden-Delft-Erasmus Centre for Education and Learning (LDE-CEL).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2023.1159905/full#supplementary-material

Supplementary Appendix A | List of all articles included.

Supplementary Appendix B1 | Results of agreement between first author and five additional coders on in- and exclusion criteria.

Supplementary Appendix C | Explanation/motivation behind Taxonomy.

Supplementary Appendix D1 | Results of agreement between first author and coder on use of taxonomy’s first category (Education).

Supplementary Appendix D2 | Results of agreement between first author and coder on use of taxonomy’s second category (System).

Supplementary Appendix D3 | Results of agreement between first author and coder on use of taxonomy’s third category (Evaluation).

Abdullah, J., Mohd-Isa, W. N., and Samsudin, M. A. (2019). Virtual reality to improve group work skill and self-directed learning in problem-based learning narratives. Virtual Real. 23 (4), 461–471. doi:10.1007/s10055-019-00381-1

CrossRef Full Text | Google Scholar

Al-Hatem, A. I., Masood, M., and Al-Samarraie, H. (2018). Fostering student nurses’ self-regulated learning with the Second Life environment: An empirical study. J. Inf. Technol. Educ. Res. 17, 285–307. doi:10.28945/4110

Amati, M., and McNeill, M. (2012). Learning from and through virtual worlds: A pilot study of second life. J. Educ. Built Environ. 7 (1), 39–55. doi:10.11120/jebe.2012.07010039

Arlati, S., Colombo, V., Spoladore, D., Greci, L., Pedroli, E., Serino, S., et al. (2019). A social virtual reality-based application for the physical and cognitive training of the elderly at home. Sensors Switz. 19 (2), 261. doi:10.3390/s19020261

Ashley, C., Kibbe, S., and Thornton, S. (2014). Experiential learning in second life: An application in retail management. Atl. Mark. J. 3 (2), 94–112. http://conf.vwbpe.org/index.php/VWBPE/11/paper/view/132 .

Google Scholar

Avanzato, R. (2018). Virtual technology to support student engagement and collaboration in online engineering courses. Comput. Educ. J. 9 (4). https://www.scopus.com/inward/record.uri?eid=2-s2.0-85058935867&partnerID=40&md5=c38a848de46d17a869b0307c6046d06e .

Aydogan, H., and Aras, F. (2019). Design, simulation and virtual implementation of a novel fundamental programmable logic controllers laboratory in a 3D virtual world. Int. J. Electr. Eng. Educ. 59, 266–281. doi:10.1177/0020720919856249

Bilyatdinova, A., Karsakov, A., Bezgodov, A., and Dukhanov, A. (2016). Virtual environment for creative and collaborative learning. Adv. Intelligent Syst. Comput. 416, 371–381. doi:10.1007/978-3-319-27478-2_26

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). “The classification of educational goals,” in Taxonomy of educational objectives (New York, NY: David McKay ), 62–197.

Bluemink, J., Hämäläinen, R., Manninen, T., and Järvelä, S. (2010). Group-level analysis on multiplayer game collaboration: How do the individuals shape the group interaction? Interact. Learn. Environ. 18 (4), 365–383. doi:10.1080/10494820802602444

Bonner, E., and Reinders, H. (2018). Augmented and virtual reality in the language classroom: Practical ideas. Teach. Engl. Technol. 18 (3), 33–53. http://www.tewtjournal.org .

Bouta, H., and Retalis, S. (2013). Enhancing primary school children collaborative learning experiences in maths via a 3D virtual environment. Educ. Inf. Technol. 18 (4), 571–596. doi:10.1007/s10639-012-9198-8

Bower, M., Lee, M. J. W., and Dalgarno, B. (2017). Collaborative learning across physical and virtual worlds: Factors supporting and constraining learners in a blended reality environment. Br. J. Educ. Technol. 48 (2), 407–430. doi:10.1111/bjet.12435

Brna, P., and Aspin, R. (1998). Collaboration in a virtual world: Support for conceptual learning? Educ. Inf. Technol. 3 (3–4), 247–259. doi:10.1023/a:1009649631868

Carron, T., Pernelle, P., and Talbot, S. (2013). “Issues of learning games: From virtual to real,” in IADIS international conference on cognition and exploratory learning in digital age, CELDA (Texas, TX: International Assn for Development of the Information Society (IADIS) ), 133–140.

Champsas, I., Leftheris, I., Tsiatsos, T., Terzidou, T., and Mavridis, A. (2012). “Open games: A framework for implementing 3D collaborative educational games in opensim,” in Proceedings of the European conference on games-based learning (Cork, Ireland: European Conference on Games Based Learning (ECGBL) ).

Chang, V., Gütl, C., Kopeinik, S., and Williams, R. (2009). Evaluation of collaborative learning settings in 3D virtual worlds. Int. J. Emerg. Technol. Learn. 4, 6–17. doi:10.3991/ijet.v4s3.1112

Chang, Y., Aziz, E. S. S., Zhang, Z., Zhang, M., and Esche, S. K. (2016). Usability evaluation of a virtual educational laboratory platform. Comput. Educ. J. 16 (1), 24–36. doi:10.18260/p.24973

Chavez, B., and Bayona, S. (2018). “Virtual reality in the learning process,” in Advances in intelligent systems and computing (Midtown Manhattan, New York City: Springer International Publishing ), 746. doi:10.1007/978-3-319-77712-2_129

Cheng, Y., and Ye, J. (2010). Exploring the social competence of students with autism spectrum conditions in a collaborative virtual learning environment - the pilot study. Comput. Educ. 54 (4), 1068–1077. doi:10.1016/j.compedu.2009.10.011

Cheong, Y. G., Khaled, R., HolmgȦrd, C., and Yannakakis, G. N. (2015). “Serious games for teaching conflict resolution: Modeling conflict dynamics,” in Conflict and multimodal communication: Social research and machine intelligence (Cham, Switzerland: Springer ), 449–475. doi:10.1007/978-3-319-14081-0_21

Cook, M., Lischer-Katz, Z., Hall, N., Hardesty, J., Johnson, J., McDonald, R., et al. (2019). Challenges and strategies for educational virtual reality: Results of an expert-led forum on 3D/VR technologies across academic institutions. Inf. Technol. Libr. 38 (4), 25–48. doi:10.6017/ital.v38i4.11075

Dalgarno, B., and Lee, M. J. W. (2010). What are the learning affordances of 3-D virtual environments? Br. J. Educ. Technol. 41 (1), 10–32. doi:10.1111/j.1467-8535.2009.01038.x

Daphne, E., William, M. L., Steve, P. R., and Adrian, W. J. (2000). “CVE technology development based an real world application and user needs,” in Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000) , Gaithersburg, MD, USA , 14-16 June 2000 . doi:10.1109/ENABL.2000.883698

de Back, T. T., Tinga, A. M., Nguyen, P., and Louwerse, M. M. (2020). Benefits of immersive collaborative learning in CAVE-based virtual reality. Int. J. Educ. Technol. High. Educ. 17 (1), 51. doi:10.1186/s41239-020-00228-9

de la Peña, N., Weil, P., Llobera, J., Giannopoulos, E., Pomés, A., Spanlang, B., et al. (2010). Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence Teleoperators Virtual Environ. 19 (4), 291–301. doi:10.1162/PRES_a_00005

De Lucia, A., Francese, R., Passero, I., and Tortora, G. (2009). Development and evaluation of a system enhancing second Life to support synchronous role-based collaborative learning. Softw. - Pract. Exp. 39 (12), 1025–1054. doi:10.1002/spe.926

De Pace, F., Manuri, F., Sanna, A., and Zappia, D. (2019). A comparison between two different approaches for a collaborative mixed-virtual environment in industrial maintenance. Front. Robotics AI 6, 18–14. doi:10.3389/frobt.2019.00018

Denoyelles, A., and Kyeong-Ju Seo, K. (2012). Inspiring equal contribution and opportunity in a 3d multi-user virtual environment: Bringing together men gamers and women non-gamers in Second Life®. Comput. Educ. , 58(1), 21–29. doi:10.1016/j.compedu.2011.07.007

Desai, K., Belmonte, U. H. H., Jin, R., Prabhakaran, B., Diehl, P., Ramirez, V. A., et al. (2017). “Experiences with multi-modal collaborative virtual laboratory (MMCVL),” in Proceedings - 2017 IEEE 3rd International Conference on Multimedia Big Data, BigMM , Laguna Hills, CA, USA , 19-21 April 2017 , 376–383. doi:10.1109/BigMM.2017.62

Dich, Y., Reilly, J., and Schneider, B. (2018). “Using physiological synchrony as an indicator of collaboration quality, task performance and learning,” in Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Midtown Manhattan, New York City: Springer International Publishing ). doi:10.1007/978-3-319-93843-1_8

Dixon, S. (2006). A history of virtual reality in performance. Int. J. Perform. Arts Digital Media 2 (1), 23–54. doi:10.1386/padm.2.1.23/1

Economou, D., Mitchell, W., Pettifer, S., Cook, J., and Marsh, J. (2001). “User centred virtual actor technology,” in Proceedings VAST 2001 virtual reality, archeology, and cultural heritage (New York, NY: Association for Computing Machinery ), 323–331. doi:10.1145/584993.585052

Edirisingha, P., Nie, M., Pluciennik, M., and Young, R. (2009). Socialisation for learning at a distance in a 3-D multi-user virtual environment. Br. J. Educ. Technol. 40 (3), 458–479. doi:10.1111/j.1467-8535.2009.00962.x

Franco, J. F., Da Cruz, S. R., and Lopes, R. D. D. (2006). “Computer graphics, interactive technologies and collaborative learning synergy supporting individuals’ skills development,” in ACM SIGGRAPH 2006 educators program, SIGGRAPH ’06 (New York, NY: Association for Computing Machinery ). doi:10.1145/1179295.1179338

Franco, J. F., and De Deus Lopes, R. (2009). “Three-dimensional digital enviroments and computer graphics influencing K-12 individuals’ digital literacy development and interdisciplinary lifelong learning,” in ACM SIGGRAPH ASIA 2009 educators program, SIGGRAPH ASIA ’09 (New York, NY: Association for Computing Machinery ). doi:10.1145/1666611.1666626

Freina, L., and Ott, M. (2015). “A literature review on immersive virtual reality in education: State of the art and perspectives,” in Proceedings of ELearning and Software for Education (ELSE) , Bucharest, Romania , April 23--24, 2015 .

Gerhard, M., Moore, D. J., and Hobbs, D. J. (2001). Continuous presence in collaborative virtual environments: Towards a hybrid avatar-agent model for user representation. Lect. Notes Comput. Sci. 2190, 137–155. doi:10.1007/3-540-44812-8_12

Girvan, C., and Savage, T. (2010). Identifying an appropriate pedagogy for virtual worlds: A communal constructivism case study. Comput. Educ. 55 (1), 342–349. doi:10.1016/j.compedu.2010.01.020

Girvan, C. (2018). What is a virtual world? Definition and classification. Educ. Technol. Res. Dev. 66 (5), 1087–1100. doi:10.1007/s11423-018-9577-y

Godin, J. J., and Pridmore, J. (2019). Using design thinking strategies and virtual reality in global virtual teams. Issues Inf. Syst. 20 (4), 104–106. doi:10.48009/4_iis_2019_104-106

Griol, D., Molina, J. M., and Callejas, Z. (2014). An approach to develop intelligent learning environments by means of immersive virtual worlds. J. Ambient Intell. Smart Environ. 6 (2), 237–255. doi:10.3233/AIS-140255

Herrera, F., Bailenson, J., Weisz, E., Ogle, E., and Zak, J. (2018). Building long-term empathy: A large-scale comparison of traditional and virtual reality perspective-taking. PLoS ONE 13, e0204494. doi:10.1371/journal.pone.0204494

PubMed Abstract | CrossRef Full Text | Google Scholar

Hwang, W. Y., and Hu, S. S. (2013). Analysis of peer learning behaviors using multiple representations in virtual reality and their impacts on geometry problem solving. Comput. Educ. 62, 308–319. doi:10.1016/j.compedu.2012.10.005

Jackson, R. L., Taylor, W., and Winn, W. (1999). “Peer collaboration and virtual environments: A preliminary investigation of multi-participant virtual reality applied in science education,” in Proceedings of the ACM symposium on applied computing (New York, NY: Association for Computing Machinery ), 121–125.

Jarmon, L., Traphagan, T., and Mayrath, M. (2008). Understanding project-based learning in Second Life with a pedagogy, training, and assessment trio. Educ. Media Int. 45 (3), 157–176. doi:10.1080/09523980802283889

Jensen, L., and Konradsen, F. (2018). A review of the use of virtual reality head-mounted displays in education and training. Educ. Inf. Technol. 23 (4), 1515–1529. doi:10.1007/s10639-017-9676-0

Jeong, H., and Hmelo-Silver, C. E. (2016). Seven affordances of computer-supported collaborative learning: How to support collaborative learning? How can technologies help? Educ. Psychol. 51 (2), 247–265. doi:10.1080/00461520.2016.1158654

Jeong, J. Y., Seo, S. J., Han, Y. J., Kim, S. J., and Jeong, H. (2018). “Learning English conversations in an immersive virtual reality environment,” in Icce 2018 - 26th international conference on computers in education, workshop proceedings (Philippines: Asia-Pacific Society for Computers in Education ), 638–643.

Jing, Y., Behzad, M., Jeff, H. P., Sarah Farukhi, A., S Khizer, K., Hinton, R. J., et al. (2018). Vital roles of β-catenin in trans-differentiation of chondrocytes to bone cells. Int. J. Ophthalmol. Clin. Res. 5 (2), 1–9. doi:10.7150/ijbs.23165

Joyner, D. A., Mavilakandy, A., Mittal, I., Kutnick, D., and Macintyre, B. (2021). Content - neutral immersive environments for cultivating scalable camaraderie . New York, NY: Association for Computing Machinery , 47–58.

Juliano, J. M., Spicer, R. P., Vourvopoulos, A., Lefebvre, S., Jann, K., Ard, T., et al. (2020). Embodiment is related to better performance on a brain–computer interface in immersive virtual reality: A pilot study. Sensors Switz. 20 (4), 1204. doi:10.3390/s20041204

Ke, F., and Lee, S. (2016). Virtual reality based collaborative design by children with high-functioning autism: Design-based flexibility, identity, and norm construction. Interact. Learn. Environ. 24 (7), 1511–1533. doi:10.1080/10494820.2015.1040421

Kleanthous, S., Michael, M., Samaras, G., and Christodoulou, E. (2016). “Transactive memory in task-driven 3d virtual world teams,” in ACM international conference proceeding series (New York, NY: Association for Computing Machinery ), 23–27. doi:10.1145/2971485.2996728

Kong, J. S. L., and Kwok, R. C. W. (2013). “Knowledge conversion in massive peers: A preliminary study on mutualistic co-presence,” in Proceedings - pacific asia conference on information systems, PACIS 2013 (Korea: Pacific Asia Conference on Information Systems (PACIS) ). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84928485362&partnerID=40&md5=9a1b86ea4622e938ec393f4b968a7fc6 .

Kwon, C. (2019). Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies. Virtual Real. 23 (1), 101–118. doi:10.1007/s10055-018-0364-1

Lattemann, C., and Stieglitz, S. (2012). “Challenges for lecturers in virtual worlds,” in Ecis 2012 - proceedings of the 20th European conference on information systems (Barcelona, Spain: European Conference on Information Systems (ECIS) ). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84905717967&partnerID=40&md5=3fda0e424f60827726a346d39465e313 .

Le, Q. T., Pedro, A., and Park, C. S. (2015). A social virtual reality based construction safety education system for experiential learning. J. Intelligent Robotic Syst. Theory Appl. 79 (3–4), 487–506. doi:10.1007/s10846-014-0112-z

Li, H., Chan, G., and Skitmore, M. (2012). Multiuser virtual safety training system for tower crane dismantlement. J. Comput. Civ. Eng. 26 (5), 638–647. doi:10.1061/(asce)cp.1943-5487.0000170(asce)cp.1943-5487.0000170

Liu, Y., Wang, T., Wang, K., and Zhang, Y. (2021). Collaborative learning quality classification through physiological synchrony recorded by wearable biosensors. Front. Psychol. 12, 674369. doi:10.3389/fpsyg.2021.674369

Liu, Y. (2020). The application of virtual reality in empathy establishment: Foresee the future. Proc. Int. Conf. Comput. Intell. Appl. ICCIA 2020, 188–193. doi:10.1109/ICCIA49625.2020.00043

López-Faican, L., and Jaen, J. (2020). EmoFindAR: Evaluation of a mobile multiplayer augmented reality game for primary school children. Comput. Educ. 149, 103814. doi:10.1016/j.compedu.2020.103814

Makransky, G., and Lilleholt, L. (2018). A structural equation modeling investigation of the emotional value of immersive virtual reality in education. Educ. Technol. Res. Dev. 66 (5), 1141–1164. doi:10.1007/s11423-018-9581-2

Makransky, G., Terkildsen, T. S., and Mayer, R. E. (2019). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 60, 225–236. doi:10.1016/j.learninstruc.2017.12.007

McArdle, G., and Bertolotto, M. (2012). Assessing the application of three-dimensional collaborative technologies within an e-learning environment. Interact. Learn. Environ. 20 (1), 57–75. doi:10.1080/10494821003714749

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochem. Medica 22 (3), 276–282. https://hrcak.srce.hr/89395 . doi:10.11613/bm.2012.031

Mehrfard, A., Fotouhi, J., Taylor, G., Forster, T., Navab, N., and Fuerst, B. (2019). A comparative analysis of virtual reality head-mounted display systems . arXiv . http://arxiv.org/abs/1912.02913 .

Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., and Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Comput. Educ. 70, 29–40. doi:10.1016/j.compedu.2013.07.033

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., and Group, T. P. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 6 (7), e1000097. doi:10.1371/journal.pmed.1000097

Molka-Danielsen, J., and Brask, O. D. (2014). Designing virtual collaboration environment for distance learners: Exploring socialization as a basis for collaboration. Lect. Notes Bus. Inf. Process. 186, 74–89. doi:10.1007/978-3-319-09546-2_6

Moll, J., and Pysander, E. L. S. (2013). A haptic tool for group work on geometrical concepts engaging blind and sighted pupils. ACM Trans. Accessible Comput. 4 (4), 1–37. doi:10.1145/2493171.2493172

Moore, A. G., McMahan, R. P., and Ruozzi, N. (2021). Exploration of feature representations for predicting learning and retention outcomes in a VR training scenario. Big Data Cognitive Comput. 5 (3), 29. doi:10.3390/bdcc5030029

Mørch, A. I., Mifsud, L., and Eie, S. (2019). Developing a model of collaborative learning with minecraft for social studies classrooms using role-play theory and practice. Computer-Supported Collab. Learn. Conf. CSCL 1, 272–279.

Motejlek, J., and Alpay, E. (2019). “A taxonomy for virtual and augmented reality in education,” in Proceedings of the 46th SEFI annual conference 2018: Creativity, innovation and entrepreneurship for engineering education excellence ( arXiv ), 1089–1100.

Muhammad Nur Affendy, N., and Ajune Wanis, I. (2019). “A review on collaborative learning environment across virtual and augmented reality technology,” in IOP Conference Series: Materials Science and Engineering 5(1). doi:10.1088/1757-899X/551/1/012050

Mystakidis, S., Berki, E., and Valtanen, J. (2017). Toward successfully integrating mini learning games into social virtual reality environments – recommendations for improving open and distance learning. Edulearn17 Proc. 1, 968–977. doi:10.21125/edulearn.2017.1203

Nadolny, L., Woolfrey, J., Pierlott, M., and Kahn, S. (2013). SciEthics Interactive: Science and ethics learning in a virtual environment. Educ. Technol. Res. Dev. 61 (6), 979–999. doi:10.1007/s11423-013-9319-0

Nikolic, S., and Nicholls, B. (2018). Exploring student interest of online peer assisted learning using mixed-reality technology . Cham, Switzerland: Springer , 715, 48–54. doi:10.1007/978-3-319-73210-7_6

Nisiotis, L., and Kleanthous, S. (2019). “The relationship between students’ engagement and the development of transactive memory systems in MuvE: An experience report,” in Annual conference on innovation and technology in computer science education, ITiCSE, 71–77 (New York, NY: Association for Computing Machinery ). doi:10.1145/3304221.3319743

Nuñez, M., Quirós, R., Nuñez, I., Carda, J. B., and Camahort, E. (2008). Collaborative augmented reality for inorganic chemistry education. WSEAS Int. Conf. Proc. Math. Comput. Sci. Eng. 5, 271–277.

Olmos, E., Cavalcanti, J. F., Soler, J.-L., Contero, M., and Alcañiz, M. (2018). Mobile virtual reality: A promising technology to change the way we learn and teach (Singapore: Springer ), 95–106. doi:10.1007/978-981-10-6144-8_6

Parong, J., and Mayer, R. E. (2021). Cognitive and affective processes for learning science in immersive virtual reality. J. Comput. Assisted Learn. 37 (1), 226–241. doi:10.1111/jcal.12482

Pastel, S., Chen, C. H., Petri, K., and Witte, K. (2020). Effects of body visualization on performance in head-mounted display virtual reality. PLoS ONE 15, 0239226. doi:10.1371/journal.pone.0239226

Pellas, N. (2014). Harnessing and evaluating open sim for the implementation of an Inquiry-Based Collaborative Learning (Ib[C]L) script in computer science: Preliminary findings from a case study in higher education. Open Source Technol. Concepts, Methodol. Tools, Appl. 3–4, 1223–1246. doi:10.4018/978-1-4666-7230-7.ch059

Pellas, N., Peroutseas, E., and Kazanidis, I. (2013). “Virtual communities of inquiry (VCoI) for learning basic algorithmic structures with open simulator and Scratch4OS: A case study from the secondary education in Greece,” in ACM international conference proceeding series (New York, NY: Association for Computing Machinery ), 187–194. doi:10.1145/2490257.2490278

Radianti, J., Majchrzak, T. A., Fromm, J., and Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comput. Educ. 147, 103778. doi:10.1016/j.compedu.2019.103778

Rebelo, F., Noriega, P., Duarte, E., and Soares, M. (2012). Using virtual reality to assess user experience. Hum. Factors 54 (6), 964–982. doi:10.1177/0018720812465006

Rogers, L. (2011). Developing simulations in multi-user virtual environments to enhance healthcare education. Br. J. Educ. Technol. 42 (4), 608–615. doi:10.1111/j.1467-8535.2010.01057.x

Rueda, J., and Lara, F. (2020). Virtual reality and empathy enhancement: Ethical aspects. Front. Robotics AI 7, 506984. doi:10.3389/frobt.2020.506984

Saldana, D., Neureither, M., Schmiesing, A., Jahng, E., Kysh, L., Roll, S. C., et al. (2020). Applications of head-mounted displays for virtual reality in adult physical rehabilitation: A scoping review. Am. J. Occup. Ther. 74 (5), 7405205060p1–7405205060p15. doi:10.5014/ajot.2020.041442

Scavarelli, A., Arya, A., and Teather, R. J. (2021). Virtual reality and augmented reality in social learning spaces: A literature review. Virtual Real. 25 (1), 257–277. doi:10.1007/s10055-020-00444-8

Schreiber, J., and Asnerly-Self, K. (2011). Educational research. The interrelationship of questions, sampling, design and analysis . Hoboken, New Jersey: John Wiley & Sons, Inc .

Shawky, D., Badawi, A., Said, T., and Hozayin, R. (2014). “Affordances of computer-supported collaborative learning platforms: A systematic review,” in Proceedings of 2014 international conference on interactive collaborative learning, ICL (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 633–651. doi:10.1109/ICL.2014.7017846

Shi, Y., Dua, J., Ahn, C. R., and Ragan, E. (2019). Impact assessment of reinforced learning methods on construction workers’ fall risk behavior using virtual reality. Automation Constr. 104, 197–214. doi:10.1016/j.autcon.2019.04.015

Sims, R., Chang, B., Bennett, V., Krishnan, A., Aboubakar, A., Coman, G., et al. (2022). “Step into my mind palace: Exploration of a collaborative paragogy tool in VR,” in Proceedings of 2022 8th international conference of the immersive learning research network, iLRN 2022. Immersive learning research network (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ). doi:10.23919/iLRN55037.2022.9815936

Smith, S. A. (2019). Virtual reality in episodic memory research: A review. Psychonomic Bull. Rev. 26 (4), 1213–1237. doi:10.3758/s13423-019-01605-w

Sousa Santos, B., Dias, P., Pimentel, A., Baggerman, J. W., Ferreira, C., Silva, S., et al. (2009). Head-mounted display versus desktop for 3D navigation in virtual reality: A user study. Multimedia Tools Appl. 41 (1), 161–181. doi:10.1007/s11042-008-0223-2

Srivastava, P., Rimzhim, A., Vijay, P., Singh, S., and Chandra, S. (2019). Desktop VR is better than non-ambulatory HMD VR for spatial learning. Front. Robotics AI 6, 50–15. doi:10.3389/frobt.2019.00050

Stahl, G., Koschmann, T., and Suthers, D. (2006). “Computer-supported collaborative learning: An historical perspective,” in Cambridge handbook of the learning sciences . Editor R. K. Sawyer (Cambridge: Cambridge University Press ), 409–426. http://gerrystahl.net/cscl/CSCL_Chinese_traditional.pdf .

Sulbaran, T., and Jones, L. F. (2012). “Utilizing a collaborative virtual reality environment as a training tool for construction students,” in ASEE annual conference and exposition, conference proceedings (Texas, TX: American Society for Engineering Education (ASEE) ). doi:10.18260/1-2--22208

Tan, C. S. S., Luyten, K., van Den Bergh, J., Schoning, J., and Coninx, K. (2014). The role of physiological cues during remote collaboration. Presence Teleoperators Virtual Environ. 23 (1), 90–107. doi:10.1162/PRES_a_00168

Terzidou, T., Tsiatsos, T., Dae, A., Samaras, O., and Chasanidou, A. (2012). “Utilizing virtual worlds for game based learning: Grafica, a 3D educational game in second life,” in Proceedings of the 12th IEEE international conference on advanced learning technologies, ICALT (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 624–628. doi:10.1109/ICALT.2012.55

Tovar, D. F., Jonker, V., and Hürst, W. (2020). Virtual reality and augmented reality in education: A review . Utrecht, Netherlands: Utrecht University .

Tremmel, C., Herff, C., Sato, T., Rechowicz, K., Yamani, Y., and Krusienski, D. J. (2019). Estimating cognitive workload in an interactive virtual reality environment using EEG. Front. Hum. Neurosci. 13, 401–412. doi:10.3389/fnhum.2019.00401

Tüzün, H., Bilgiç, H. G., and Elçi, S. Y. (2019). The effects of 3D multi-user virtual environments on collaborative learning and social presence. Int. Electron. J. Elem. Educ. 11 (3), 221–231. doi:10.26822/iejee.2019349247

Vallance, M., Ibayashi, K., and Goto, Y. (2015). Engineering active learning in 3D virtual worlds. Commun. Comput. Inf. Sci. 533, 268–282. doi:10.1007/978-3-319-22629-3_22

Vrellis, I., Papachristos, N. M., Bellou, J., Avouris, N., and Mikropoulos, T. A. (2010). Designing a collaborative learning activity in second life: An exploratory study in physics. Proc. - 10th IEEE Int. Conf. Adv. Learn. Technol. ICALT 2010, 210–214. doi:10.1109/ICALT.2010.65

Wang, C. J., Low, A. S. L., and Woo, C. S. (2014). “3D system for learning gains in the lower secondary normal (technical) stream,” in 3D Immersive and Interactive Learning (Singapore: Springer ), 93–103. doi:10.1007/978-981-4021-90-6_7

Wang, A., Thompson, M., Roy, D., Pan, K., Perry, J., Tan, P., et al. (2022). Iterative user and expert feedback in the design of an educational virtual reality biology game. Interact. Learn. Environ. 30 (4), 677–694. doi:10.1080/10494820.2019.1678489

Yeh, S. C., Hwang, W. Y., Wang, J. L., and Chen, Y. R. (2012). “Effects of multi-symbols on enhancing virtual reality based collaborative task,” in Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Berlin, Germany: Springer Berlin Heidelberg ), Vol. 7220, 101–111. doi:10.1007/978-3-642-31439-1_10

Yoon, H. J., Kim, J., Park, S. W., and Heo, H. (2020). Influence of virtual reality on visual parameters: Immersive versus non-immersive mode. BMC Ophthalmol. 20 (1), 200–208. doi:10.1186/s12886-020-01471-4

Zheng, L., Xie, T., and Liu, G. (2019). “Affordances of virtual reality for collaborative learning,” in Proceedings - international joint conference on information, media and engineering, ICIME 2018 (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 6–10. doi:10.1109/ICIME.2018.00011

Keywords: virtual reality, collaborative learning, virtual reality education, collaborative virtual environment, virtual reality and collaborative learning, collaborative virtual reality, collaborative virtual reality systems, educational technologies

Citation: van der Meer N, van der Werf V, Brinkman W-P and Specht M (2023) Virtual reality and collaborative learning: a systematic literature review. Front. Virtual Real. 4:1159905. doi: 10.3389/frvir.2023.1159905

Received: 06 February 2023; Accepted: 02 May 2023; Published: 19 May 2023.

Reviewed by:

Copyright © 2023 van der Meer, van der Werf, Brinkman and Specht. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Nesse van der Meer, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Corpus ID: 48805979

Collaboration: a Literature Review Research Report

• H. Phan , Jolana Rivas , Tian Song
• Published 2011
• Education, Psychology

46 Citations

The impact of technology on collaborative learning, students’ participation and collaboration skills through radec learning model and the influencing factors, using collaborative formative assessments in enhancing students' understanding of concepts in grade 9 electron configuration, collaborative feedback in a blended learning environment: a case study of an efl writing class, effect of service and collaborative learning strategies on learners’ environmental attitude, quantitative approach to collaborative learning: performance prediction, individual assessment, and group composition, university academics’ experiences of learning through mentoring, getting radical: using design thinking to tackle collaboration issues, why 6 cs the urgency of learning at elementary school, collaboration in east africa: a contextualised approach to defining the construct, 42 references, the evolution of research on collaborative learning.

• Highly Influential
• 15 Excerpts

When teams do not function the way they ought to

Cooperative activities in the classroom, group discussion and large-scale language arts assessment: effects on students' comprehension, the construction of shared knowledge in collaborative problem solving, collaborative learning tasks and the elaboration of conceptual knowledge., processes and consequences of peer collaboration: a vygotskian analysis, learning by collaborating: convergent conceptual change, group collaboration in assessment: multiple objectives, processes, and outcomes, collaborative group versus individual assessment in mathematics: processes and outcomes, related papers.

Showing 1 through 3 of 0 Related Papers

Online Collaborative Learning in Higher Education: A Review of the Literature

As an instructor in a fully online program, I often use group work as a means to increase engagement and facilitate a connection in the online classroom. In some classes, I ask students to work in groups on individual assignments, but for the purpose of giving and receiving feedback on their respective projects. For example, in a course on Nutrition Education Methods, students work to develop individual lessons that they ultimately deliver at the end of the term. In this case, peer feedback is used to strengthen their work. In other classes, I ask students to work together in groups where they all contribute to a larger, shared project that they submit at the end of the term. In a course on Health Communication, for example, students work collaboratively to develop and implement a social marketing campaign that addresses a health-related issue of their choosing.

In course evaluations, the group assignments established for giving and receiving peer feedback are generally well-received and students note their appreciation for their groups’ remarks. In the second example, student evaluations about their experience are often mixed. Some report a positive group experience, while others are disappointed with the final outcome.

The ability to work collaboratively with a team is a skill that serves students well beyond their college years. A  recent article  on LinkedIn Learning (Pace, 2020) outlines the “soft” skills that companies are seeking in prospective employees in 2020. These skills include creativity, persuasion, collaboration, adaptability, and emotional intelligence – all skills that “demonstrate how we work with others and bring new ideas to the table” (Pace, 2020, para. 2). As an instructor, I see the larger benefits of collaborative learning, but recognize how these assignments translate in the online classroom isn’t always successful.

In this review of the literature, my aim is to share the results of my research on collaborative learning and its applications in the online environment in higher education, as well as the circumstances that make collaborative learning a positive experience for students and teachers alike.

What is Collaborative Learning?

The word collaboration “suggests a way of dealing with people which respects and highlights individual group members’ abilities and contributions. There is a sharing of authority and acceptance of responsibility among group members for the groups’ actions” (Laal & Ghodsi, 2012, p. 486). Collaborative learning requires that learners work together to make connections and uncover new ways of understanding concepts (Laal & Laal, 2012 as cited by Falcione et al., 2019). Falcione et al. (2019) add to this definition by explaining that collaborative learning is a way for students to intertwine their independent work in order to achieve a shared goal. The results of these efforts are a “product or a learning experience that is more than the summation of individual contributions” (Falcione et al., 2019, p. 1).

The foundation of collaborative learning is the idea that learning with others is better than learning alone (Nokes-Malach et al., 2015). In fact, the primary goal of team-centered, collaborative environments is to apply the unique backgrounds and skills that individuals bring to a group and accomplish something together that they may otherwise be unable to accomplish individually (Roberts, 2004).

Online learning naturally lends itself to student-centered instructional strategies and assessments, and collaborative learning most certainly fits this category (Muller et al., 2019). Given the physical distance that separates online students, collaborative learning efforts may also help students connect in an effort to dissolve any feelings of isolation they may be experiencing (Writers, 2018).

Harasim (2012) as cited by Bates (2015), offers the following definition of Online Collaborative Learning (OCL):

Online collaborative learning theory provides a model of learning in which students are encouraged and supported to work together to create knowledge: to invent, to explore ways to innovate, and, by so doing, to seek the conceptual knowledge needed to solve problems rather than recite what they think is the right answer (Harasim, 2012 as cited by Bates, 2015, para 1).

The image shown below depicts the core principles of Online Collaborative Learning and how these principles are operationalized through online discussions. Discussion forums often serve as the backbone for learning in online environments. Bates (2015) argues that online discussion forums are not meant to supplement course content (typically delivered through lectures and textbooks), but should be the central means for content delivery. Here, students identify readings and resources  to support the discussion  as opposed to allowing the readings and resources to be the driver. It is through this discourse that students are able to generate and organize ideas and ultimately achieve “intellectual convergence” by synthesizing the ideas presented (Bates, 2015). Because the discussion happens asynchronously, students have time to ruminate over the ideas presented and respond in a more thoughtful manner (Roberts, 2004).

(Bates, 2015)

Another important element of the Online Collaborative Learning model depicted above is the role of the teacher. Here, the teacher serves as a facilitator of the discussion in an effort to move students through the process of generating, organizing, and synthesizing ideas (Bates, 2015). The concept of “teacher” as “facilitator” is a hallmark of student-centered, online learning.

Related Terms and Theories

Collaborative learning is sometimes used interchangeably with the term “cooperative learning” (Writers, 2018). Balkcom (1992) defines  cooperative learning  as an instructional strategy that uses groups made up of diverse learners. Groups are engaged in a variety of activities to enhance their understanding of lesson concepts, and members have a shared responsibility to help one another learn and grow. A key component of cooperative learning is  positive interdependence  ( What is Cooperative Learning , n.d.). Positive interdependence is established when students perceive that the contribution of each group member is essential to the success of the group. Scager et al. (2016) found positive interdependence to be a critical factor in successful collaboration.

While the two concepts share many of the same characteristics, Falcione et al. (2019), argues that cooperative learning is, in fact, different from collaborative learning. The primary factor that differentiates collaborative learning from cooperative learning is the independent work that group members do in order to contribute to the task at hand. This work is done at different times and is often developed alone. However, the individual’s work is later combined with the work of other group members in order to synthesize ideas. The following video expands on this idea and identifies additional factors that differentiate collaborative learning from cooperative learning:

(Wufei87, 2018)

Another related concept evident in the literature is a Community of Inquiry (CoI). In the image below, Garrison, Anderson, and Archer (2000) depict the CoI framework that they argue is integral to the online learning experience in higher education.

(Garrison, Anderson, & Archer, 2000, p. 88)

Here, the educational experience is at the center of the CoI model, and learning takes place through the interaction of three vital components: social presence, cognitive presence, and teaching presence. Social presence represents the idea that individuals within the community are able to interject elements of their personality into the group so that they are seen as “real people” (Garrison, Anderson, & Archer, 2000, p. 89). Cognitive presence is deemed the most important of the three and refers to the ability of learners to “construct meaning through sustained communication” (Garrison, Anderson, & Archer, 2000, p. 89). The authors argue that cognitive presence is critical to developing higher-order thinking skills (which is necessary in postsecondary education). Finally, teaching presence is defined by two key functions: 1) course design and 2) facilitation. Essentially, the goal of teaching presence is to facilitate cognitive presence and social presence within the community (Garrison, Anderson, & Archer, 2000). Bates (2015) concludes that CoI and OCL are more “complementary rather than competing” (section 4.4.3) ideas and are, therefore, not mutually exclusive models for learning.

Online collaborative learning may be classified as a constructivist approach to learning (Bates, 2015). Constructivism is a theory that posits that learners actively construct knowledge as opposed to passively receiving it. This knowledge is further developed through life experiences allowing learners to develop mental models as a way to make sense of new information ( Constructivism , n.d.). The table below outlines the differences between traditional learning and constructivist learning:

(Constructivism, n.d.)

Examples of Collaborative Learning

In the traditional classroom setting, collaborative learning can take on many forms. Problem-based learning, jigsaw activities, think-pair-share, and peer review are just a few common examples (Nokes-Malach et al., 2015). These strategies are defined in more detail below:

Problem-based learning:  In this strategy, students work in groups to collaboratively solve a larger problem. The group work takes place over an extended period of time and often requires some deliverable at the end of the project (Active and Collaborative Learning | University of Maryland—Teaching and Learning Transformation Center, n.d.).

Jigsaw:  This strategy takes a problem or task and divides it into smaller components. Each component is assigned to a group in order to gain a deeper understanding of the topic, who ultimately reports out in an effort to contribute their understanding as a piece of the larger puzzle (Active and Collaborative Learning | University of Maryland—Teaching and Learning Transformation Center, n.d.).

Think-pair-share:  This strategy starts by dividing students into pairs. The instructor then provides students with a discussion prompt or question to consider. Individual learners reflect on the problem independently before sharing their thoughts or ideas with a peer. Once both students have had a chance to discuss, they may share a summary of their discussion with the rest of the group (Active and Collaborative Learning | University of Maryland—Teaching and Learning Transformation Center, n.d.).

Peer review:  This strategy allows students to review one another’s work and provide positive and constructive feedback to facilitate improvement. The strategy teaches students as writers to receive, evaluate, and choose whether or not to incorporate the feedback into their work. As editors, it teaches students to analyze and clearly communicate feedback with their peers. As an instructor, it is critical to provide guidance and structure to best facilitate the process (Active and Collaborative Learning | University of Maryland—Teaching and Learning Transformation Center, n.d.).

All of these strategies can be adapted for the online learning environment, however, online collaboration tools, i.e.  Zoom ,  Google Docs ,  S lack , or  T rello , are often used to facilitate the transition (Writers, 2018). Tarun (2019) defines online collaboration tools as “web-based tools that allow individuals to do things together online like messaging, file sharing, and assessment” (p. 276). Integrating technology tools like these in the classroom fosters “authentic and meaningful learning experiences” (Boundless, 2015, sec 2) and also supports differentiated learning efforts (Boundless, 2015).

A basic search online for “online collaboration tools for education” yielded a variety of sites ranking the top-rated tools for digital collaboration (EDsmart, 2015; TeachThought, 2019). In the 2019 article, the tools ranked covered broad categories like tools for communication, project management, peer review, and game-based learning. Listed below are some examples that the authors highlighted in this post:

Diigo : Diigo is a social bookmarking tool that allows learners to collect, annotate, organize, and share online resources.

Flipgrid : Flipgrid is a tool that allows learners to create and share short videos and can be used for reflections, discussions, or short presentations. Additionally, peers can respond to posts in video form. The “grade book” feature within the tool allows instructors to track and monitor participation.

VideoAnt : VideoAnt is a tool that allows students and teachers to annotate YouTube videos. Here, students can ask questions or add critiques at various spots throughout the video.

Padlet : Padlet is a multimodal group collaboration tool. Here, students can collect videos, articles, or images and post them to a virtual corkboard. Students can also comment on posts in a threaded discussion format.

Appavoo, Sukon, Gokhool, and Gooria (2019) add that tools like WhatsApp, Skype, and Moodle are popular tools for online collaborative learning in higher education. These tools offer learners a way to discuss and share ideas and gain instant feedback. Furthermore, some students report that they prefer to learn on tools like these as they feel more open to discussing any academic-related issues they may be experiencing (Preston, Phillips, Gosper, McNeil, Woo, and Green, 2010, as cited by Appavoo et al., 2019).

Benefits of Collaborative Learning

Scager et al. (2016) note that there are decades of literature that demonstrate the positive effects of collaborative learning on academic success. In one such article, Laal and Ghodsi (2012) compiled and categorized the benefits of collaborative learning found in the literature between 1964-2010. The noted benefits were divided into four overarching categories to include social, psychological, academic, and assessment.

Social : Collaborative learning creates a support system for students as they work through challenges together. The group work also facilitates learning communities while improving student’s understanding of diverse viewpoints and strengthening cooperation.

Psychological : Learner-centered instruction improves self-confidence in the learner and working on problems together can help lessen feelings of anxiety for students. Affectively, collaborative learning efforts may lead to more “positive attitudes towards teachers”.

Academic:  Collaborative learning creates a student-centered approach to learning, fosters higher-order thinking and facilitates problem-solving skills.

Assessment:  Collaborative learning efforts use a multitude of assessment techniques.

Falcione et al. (2019) add that collaborative learning leads to a mastery of course content and the cultivation of interpersonal skills that benefit the student outside of the classroom environment.

In collaborative learning, the metacognitive ability of participants is improved due to the absence of a professor’s help throughout the process; learners must turn to each other, or outside sources, to overcome barriers, encouraging recognition of their own misunderstandings. (Davidson & Major, 2014 as cited by Falcione et al, 2019).

Benefits Specific to Online Collaborative Learning Roberts (2004) describes additional benefits specific to collaborative learning in the online environment. Examples include:

• Quiet students may open up.
• Little off-task behavior.
• The asynchronous nature of discussions fosters deeper responses.
• Students can use technology tools to access additional information.
• Few student disruptions.
• The content of online discussions can be retrieved at a later time.
• Discussions can extend across the term.
• Online learning creates an environment that supports the instructor’s role as facilitator.

Challenges with Collaborative Learning

While there are many documented benefits of collaborative learning, this strategy also comes with its fair share of challenges. One such challenge includes the “cognitive costs of coordinating and collaborating with others” (Nokes-Malach et al., 2015, p. 647). In other words, if an individual member can solve the problem independently, then they are not likely to benefit from collaborative efforts and may even perform worse as a result of trying to coordinate many varied ideas (Nokes-Malach et al, 2015 as cited by Nokes-Malach et al., 2012). This notion also applies to less complex activities where little is gained from group collaboration. Group members benefit when the task is complex, i.e. “high cognitive load”, and parts can be distributed among the group.

Other potential challenges described by Nokes-Malach et al. (2015) include “retrieval strategy disruption” and “production blocking”. The former concept occurs when one person loses their train of thought because they are paying attention to other group members, while the latter refers to the practice of allowing others to finish speaking before attempting to speak. This example can lead to “missed retrieval opportunities”.

A third example includes “social loafing” which describes the phenomenon where one group member may not contribute at the same level because they believe other group members may help “pick up the slack”.

A fourth, and final challenge is of collaborative learning is “fear of evaluation”. Here, students may avoid sharing ideas out of fear of judgment from their group members (Nokes-Malach et al., 2015).

Johnson and Johnson (2009, as cited by Nokes-Malach et al., 2015) propose that the latter two examples may occur when there is a lack of individual accountability or positive interdependence among group members as described earlier in this review.

It’s important to note that there are also drawbacks related more specifically to online collaborative learning efforts. One such drawback involves some of the online collaboration tools used. Tarun (2019) discusses the inadequacies of such tools to include a lack of features that may improve usability as well as the inability to customize some tools to meet classroom, instructor, or school needs.

Appavoo et al. (2019) add that collaborative learning efforts in online courses can be difficult to coordinate for learners, as some are also balancing professional and family-related commitments.

Implications for Future Work

In order to overcome some of the common challenges of collaborative learning and maximize benefits, it is important to adhere to the recommendations that have emerged from the research on collaborative learning efforts (Scager et al., 2016).

The first factor that instructors should keep in mind when implementing collaborative learning efforts is to use a small group size (Scager et al., 2016). Three to five students per group is recommended to maximize efficacy ( Cooperative learning classroom.research , n.d.).

Another factor to consider is group composition. Groups comprised of members with diverse perspectives have been shown to increase learning in group work (Kozhevnikov et al., 2014). It is interesting to note that mixed ability groups tend to benefit lower ability students and may not benefit higher ability students (Webb et al, 2002). What may be even more important when it comes to learning, however, is equal participation among group members regardless of “ability”. When all students participate equally, they are more likely to fully utilize each member’s unique skillsets and contributions (Woolley et al., 2015).

A third factor to consider is the nature of the task itself. For collaborative learning efforts to be most successful, tasks should be both complex and appropriate for the topic at hand. The task should also allow students to create unique work with autonomy and self-regulation (Scager et al., 2016), but within a structure or framework to guide collaborative learning efforts (Appavoo et al., 2019).

Last, but not least, for collaboration to be successful, social interaction is imperative (Volet et al., 2009). The process of discussing, debating, and explaining ideas to one another, as well as building off of others’ ideas helps to facilitate metacognition (Scager et al., 2016).

Gaps and Conclusions

While the concept of collaborative learning certainly isn’t new, collaborative learning in a digital environment is for many teachers and students. With the advances in technology as well as the increase in quantity and quality of digital tools available, there is great potential for the future of online learning. To get to that point, however, it will be important to address some of the gaps in the existing literature.

Research efforts for this review uncovered fewer articles related specifically to online collaborative learning when compared to collaborative learning in the traditional classroom setting. Chang and Hannafin (2015) add that it will be important to consider the unique traits of adult-learners and the impact that online collaboration tools may have on learning for this group.

Tarun (2019) notes that research on educational technology tools most often includes tests of quality, to include “functionality and usability”, but fail to evaluate the effects of integration into the online classroom. In future research, it will be important to consider if and how the technology tools used for collaboration are actually accomplishing what educators believe they are accomplishing.

Active and Collaborative Learning | University of Maryland — Teaching and Learning   Transformation Center . (n.d.). Retrieved April 4, 2020, from https://tltc.umd.edu/active-and-collaborative-learning

Bates, A. W. (Tony). (2015). 4.4 Online collaborative learning. In  Teaching in a Digital Age . Tony Bates Associates Ltd. https://opentextbc.ca/teachinginadigitalage/chapter/6-5-online-collaborative-learning/

Boundless (2015, July 21). Advantages of using technology in the classroom.  Boundless Education . Retrieved from http://oer2go.org/mods/en-boundless/www.boundless.com/education/textbooks/boundless-education- textbook/technology-in-the-classroom-6/edtech-25/advantages-of-using-technology-in-the-classroom-77- 13007/index.html

Chang, Eunice & Hannafin, M. J. (2015). The uses (and misuses) of collaborative distance education technologies: Implications for the debate on transience in technology. Quarterly Review of Distance Education ,  16 (2), 77–92.

Constructivism . (n.d.). Retrieved April 4, 2020, from http://www.buffalo.edu/ubcei/enhance/learning/constructivism.html Cooperative learning classroom.research . (n.d.). Retrieved April 4, 2020, from http://alumni.media.mit.edu/~andyd/mindset/design/clc_rsch.html

EDsmart. (2015, December 29).  50 free online collaboration tools for educators .  https://www.edsmart.org/50-free-online-collaboration-tools-for-educators/

Falcione, S., Campbell, E., McCollum, B., Chamberlain, J., Macias, M., Morsch, L., & Pinder, C. (2019). Emergence of different perspectives of success in collaborative learning.  Canadian Journal for the Scholarship of Teaching and Learning ,  10 (2). https://eric.ed.gov/?id=EJ1227390

Kozhevnikov, M., Evans, C., & Kosslyn, S. M. (2014). Cognitive style as environmentally sensitive individual differences in cognition: A modern synthesis and applications in education, business, and management.  Psychological Science in the Public Interest ,  15 (1), 3–33. https://doi.org/10.1177/1529100614525555

Laal, M., & Ghodsi, S. M. (2012). Benefits of collaborative learning.  Procedia – Social and Behavioral Sciences , 31, 486–490. https://doi.org/10.1016/j.sbspro.2011.12.091

Muller, K., Gradel, K., Forte, M., McCabe, R., Pickett, A. M., Piorkowski, R., Scalzo, K., & Sullivan, R. (n.d.).  Assessing Student Learning in the Online Modality . 32.

Nokes-Malach, T. J., Richey, J. E., & Gadgil, S. (2015). When is it better to learn together? Insights from research on collaborative learning.  Educational Psychology Review ,  27 (4), 645–656. https://doi.org/10.1007/s10648-015-9312-8

Roberts, T. S. (2004).  Online Collaborative Learning: Theory and Practice . Idea Group Inc (IGI).

Scager, K., Boonstra, J., Peeters, T., Vulperhorst, J., & Wiegant, F. (2016). Collaborative learning in higher education: Evoking positive interdependence.  CBE Life Sciences Education ,  15 (4). https://doi.org/10.1187/cbe.16-07-0219

Tarun, I. M. (2019). The Effectiveness of a Customized Online Collaboration Tool for Teaching and Learning.  Journal of Information Technology Education: Research , 18, 275–292.

TeachThought. (2019, June 9).  30 Of the best digital collaboration tools for students .  https://www.teachthought.com/technology/12-tech-tools-for-student-to-student-digital-collaboration/

Volet, S., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: How does it emerge and how is it sustained?  Learning and Instruction ,  19 (2), 128–143. https://doi.org/10.1016/j.learninstruc.2008.03.001

What is Cooperative Learning?  (n.d.). Cooperative Learning. Retrieved April 3, 2020, from https://serc.carleton.edu/introgeo/cooperative/whatis.html

Writers, S. (2018, February 14).  Current Trends in Online Education . TheBestSchools.Org. https://thebestschools.org/magazine/current-trends-online-education/

May 7, 2020

Strategies for Online Learning

collaborative learning , cooperative learning

July 29, 2020 at 8:32 pm

This is a great article for the times. How to make sure you get social interaction while operating a digital classroom is so important.

December 14, 2020 at 9:53 am

After going over a few of the articles on your website, I seriously like your way of blogging. I saved as a favorite it to my bookmark site list and will be checking back soon. Please visit my web site as well and tell me your opinion.|

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

© 2024 — Powered by WordPress

Theme by Anders Noren — Up ↑

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Learning Collaboratives: a Strategy for Quality Improvement and Implementation in Behavioral Health

Heather j. gotham.

1 Mental Health Technology Transfer Center Network Coordinating Office, Stanford University School of Medicine, 1520 Page Mill Road, Palo Alto, CA 94304 USA

Manuel Paris, Jr.

2 The Annapolis Coalition on the Behavioral Health Workforce & Yale University School of Medicine, 34 Park Street, New Haven, CT 06511 USA

Michael A. Hoge

3 The Annapolis Coalition On the Behavioral Health Workforce, & Yale University School of Medicine, 300 George Street, Suite 901, New Haven, CT 06511 USA

Associated Data

The de-identified interview data that support this commentary are available from the corresponding author upon reasonable request.

Learning collaboratives are increasingly used in behavioral health. They generally involve bringing together teams from different organizations and using experts to educate and coach the teams in quality improvement, implementing evidence-based practices, and measuring the effects. Although learning collaboratives have demonstrated some effectiveness in general health care, the evidence is less clear in behavioral health and more rigorous studies are needed. Learning collaboratives may contain a range of elements, and which elements are included in any one learning collaborative varies widely; the unique contribution of each element has not been established. This commentary seeks to clarify the concept of a learning collaborative, highlight its common elements, review evidence of its effectiveness, identify its application in behavioral health, and highlight recommendations to guide technical assistance purveyors and behavioral health providers as they employ learning collaboratives to improve behavioral health access and quality.

Introduction

The last several decades have been marked by growing concerns about the quality of health care. A seminal report at the turn of the century by the Institute of Medicine highlighted that as many as 98,000 people were dying annually from medical errors. 1 This caught public attention and stimulated a national agenda for improving patient safety. What followed were efforts throughout health care, and behavioral health care, to further develop and disseminate evidence-based practices to enhance the quality and effectiveness of care. 2 , 3 Developing an evidence-base on safe and effective practices is an essential first step. However, translating science to practice and improving the quality of care delivered in real world settings has proved to be a daunting challenge. 4

While first developed in general health care, learning collaboratives, also called quality improvement collaboratives, have been used frequently in behavioral health by providers of training, technical assistance, and services as a strategy to promote quality improvement and implementation of a range of evidence-based practices (e.g., cognitive behavioral therapy for psychosis, integrated services for co-occurring mental health and substance use disorders, school mental health, supported employment, trauma informed care). 5 – 12 These collaboratives generally involve bringing together teams from different organizations and using experts to educate and coach them in a quality or implementation project and measure the effects. 13 Sharing of strategies, data, successes, and obstacles among participating teams is central to the approach. 14

With the proliferation of learning collaboratives has come wide variability in how they are conducted. 15 – 17 Technical assistance providers may be unclear about which components of a learning collaborative are most important, how long it should last, and when to offer a learning collaborative rather than an alternative improvement or implementation strategy (e.g., training workshop, community of practice). Behavioral health care providers may struggle with whether to invest significant staff, time, and financial resources in a learning collaborative, whether a collaborative being offered is adequately designed, and what outcomes to expect.

Learning collaboratives have been the subject of many publications and a more modest number of research studies. 6 , 7 , 16 This commentary stems from a project led by the Annapolis Coalition on the Behavioral Health Workforce for the Mental Health Technology Transfer Center Network to improve knowledge about collaboratives within the behavioral health community and to promote, if warranted, an increase in their effective use in the USA. The specific aims were to clarify the concept of a learning collaborative, highlight its common elements, review evidence of its effectiveness, identify its application in behavioral health, and highlight recommendations to guide technical assistance purveyors and behavioral health providers as they employ learning collaboratives to improve behavioral health access and quality.

This commentary integrates the findings from a traditional literature review of the published and gray literature and key informant interviews of experts in the field.

Literature search and review

A literature search of 16 terms related to learning collaboratives was conducted using Ovid MEDLINE, PsycINFO, AMED, PsycARTICLES, PsycEXTRA, Embase, CINAHL, Google Scholar, and Google with filters set for publications in English from 2010 to 2020. The search included the following terms: learning collaborative(s), quality improvement collaborative(s), quality improvement learning collaborative(s), quality collaborative(s), mental health service collaborative care, mental health service quality, behavioral health learning collaborative(s), behavioral health quality improvement collaborative(s), quality improvement collaborative(s) in behavioral health, evidence-based engagement strategy(ies), multidisciplinary quality improvement team(s), community based quality improvement, learning laboratory(ies), breakthrough series, and Network for the Improvement of Addiction Treatment (NIATx). Reference lists in the initial documents identified were reviewed to find additional resources. One member of the study team (MP) conducted the search and reviewed reference lists, which yielded 640 documents. He performed the initial analysis of this literature, identifying 151 relevant documents and then categorized them by focus (mental health, substance use, general health care, or integration of general health and behavioral health). A second member of the team (MH) conducted a detailed analysis of the relevant literature, identifying the major categories of findings and the national surveys, reviews, and individual studies that offered specific findings within those categories. Sources on learning collaboratives in general health care were examined first, providing a context for review of the more limited resources in behavioral health. The other members of the team then vetted the categorization scheme and the distillation of findings into those categories.

Key informant interviews

Semi-structured interview questions were developed after the review of the literature clarified topics needing further exploration. Fifteen interviews were conducted involving 17 individuals (see Acknowledgements for names) with substantial knowledge related to learning collaboratives in health or behavioral health, including those focused on diverse populations. Experience within this expert group included founding the learning collaborative movement; managing technical assistance and quality improvement networks; funding and shaping federal policy regarding technical assistance; and conducting research on learning collaboratives, quality improvement, and implementation. One member of the study team (MH) conducted, recorded, and analyzed the recordings, using open coding, which was then reviewed by another author (HG). 18

The first author (MH) integrated information from the literature review and interview findings. Another author (HG) shaped a conceptual framework that further organized the information into a set of distinct findings and recommendations. National leaders in the provision of behavioral health technical assistance, who comprised a Workforce Development Working Group for the Mental Health Technology Transfer Center Network, then reviewed and commented on a draft study report before its finalization.

Learning collaboratives: current knowledge

The following points describe and characterize current knowledge about learning collaboratives based on their extensive use in health care and increasing use in behavioral health, as well as evaluation of those efforts. They are organized by the specific aims of clarifying the concept of a learning collaborative, highlighting common elements and structure, reviewing evidence of effectiveness, and identifying applications in behavioral health.

Clarifying the concept of a learning collaborative

The traditional learning collaborative model has been reasonably well-defined, broadly disseminated, and widely adopted. Launched in 1995, the Institute for Healthcare Improvement (IHI) Breakthrough Series (BTS) model is considered the origin of the learning collaborative. 13 Key elements of a BTS collaborative, as identified by IHI in its early publications, included: selecting a specific improvement topic, recruiting expert faculty, enrolling organizations, providing face-to-face learning sessions, Plan-Do-Study-Act (PDSA) cycles of change, offering technical assistance to organization teams, inter-agency sharing and learning, and summation of results and lessons learned. 13 , 19 , 20 The IHI model became extremely popular and was widely disseminated and adopted nationally and internationally. 13 , 17

Currently, the term learning collaborative is often used loosely as a label for efforts at quality improvement and implementation that are often not well-defined or described. As learning collaboratives have proliferated, the term has been used for projects that vary widely from the original BTS model, including different elements, lengths, and frequency of meetings, raising questions about the use of the term and expected efficacy of the project. 15 , 16 Confusion and variation stem from a number of sources, including a lack of detail in published studies and descriptions of learning collaboratives. 6 Variation also occurs as technical assistance providers experiment with collaboratives that are shorter, less intensive, and more virtual to reduce overall cost and burden (Lang, Venkatesh, personal communication). 21  Technical assistance providers depart from the BTS model when the evidence-base for improving quality is not strong, funding is inadequate, the capacity of participating organizations is a concern, or the project goal is limited to provider skills (Lang, Reid, Venkatesh, personal communication). Finally, it is important to note that a BTS learning collaborative is distinct from learning communities and communities of practice, which tend to be more about individual provider development than organizational change, and more about learning than the collection and analysis of data. 25 – 27

Common elements and functions

Experts have highlighted the elements of learning collaboratives they deem most important, though limited research has yet to consistently demonstrate the relative impact of any specific element. 20 , 28 – 30 A learning collaborative is a bundle of specific elements, such as the composition of the participating teams, number and format of learning sessions (in-person and/or virtual), use of different types of pre-work, and collection of and feedback about data. A systematic review of 20 studies of learning collaboratives in general health care identified 14 process and structural elements (see Table ​ Table1 1 ). 28 Although some elements were used more consistently than others (i.e., in-person learning sessions, PDSA cycles, and collection of data for quality improvement were each used by at least 15 of the 20 studies), research has not disentangled the effectiveness of each element.

Learning collaborative process and structural elements

* Adapted from Nadeem and colleagues 28

The length of learning collaboratives varies greatly, and their optimal length and intensity have not been demonstrated. This issue is controversial, particularly as shortening the experience saves costs and reduces burden on participating agencies and their teams. IHI BTS collaboratives currently last between 18 and 24 months, whereas the average for behavioral health collaboratives, based on a systematic review by Nadeem and colleagues, was 14 months (Laderman, personal communication). 6  Factors to be considered in setting the length include complexity of the intervention, resources available to the sponsor and participating organizations, and goals to be achieved (Everett, Hoover, personal communication). On the one hand, organizational level change takes time and brief learning collaboratives may limit effectiveness and sustainability (Amaya-Jackson, Becker, Laderman, Reid, personal communication). On the other hand, traditional, lengthy collaboratives may outstrip the resources and attention span of many behavioral health agencies (Gustafson, personal communication).

Evidence-base for learning collaboratives

In general health care, research on traditional learning collaboratives, albeit incomplete, suggests some effectiveness in improving provider practices and health outcomes. 5 , 7 , 15 – 17 A systematic review of studies of BTS-style collaboratives from 1995 through 2014 found that most studies reported significant improvement on at least one clinical process or patient outcome measure with a wide range in the magnitude of changes. 7 More successful collaboratives addressed straightforward aspects of treatment and the large gap between current practice and the evidence-based practice being implemented. However, the authors noted that many studies were not included in the review due to flaws in design, lack of fidelity to a specific collaborative model, and sparse descriptions of collaborative elements. When other reviews have included a broader variety of learning collaboratives and quality improvement processes, they have found more mixed results and suggest that effectiveness varies by contextual factors; for example, learning collaboratives may be more effective in changing provider rather than patient outcomes, and effectiveness increases when a learning collaborative is paired with other quality improvement or implementation strategies. 5 , 15 – 17 The more that learning collaboratives depart from the fidelity of the traditional BTS model, the less assured one can be of their effectiveness.

Organizational and system change is a complex, difficult process. Multiple reviews in health care and the field of dissemination and implementation science have shown that passive implementation strategies (e.g., distributing published or printed recommendations) and even traditional training strategies (e.g., workshops, lectures) are not effective in changing provider behavior and/or implementing evidence-based practices.  37 – 39 Moreover, using a single implementation strategy to foster practice change is less effective, in general, than multi-component strategies. 40 – 42 As is highlighted in Table ​ Table1, 1 , a learning collaborative is a multi-component strategy. Some research and strong consensus among experts suggest that learning collaboratives are likely more effective in improving provider practices and patient outcomes than low intensity interventions such as lectures, workshops, and webinars (Amaya-Jackson, Becker, Gustafson, personal communication). 30 , 43 , 44 A few studies comparing learning collaboratives to less intensive or single strategies have found promising results, and several larger scale, controlled trials are in process. 30 , 43 – 46

Learning collaboratives in behavioral health

In behavioral health, learning collaboratives have been used frequently to address a broad range of topics, including large scale, high-quality applications of the traditional BTS style model. Behavioral health learning collaboratives are described in over 60 publications which report on their perceived value in improving quality and implementing evidence-based practices, with several prominent examples. Over 50 collaboratives on Trauma Focused Cognitive Behavioral Therapy have been conducted by the National Center for Child Traumatic Stress. 4 The National Center for School Mental Health adapted the BTS model to improve school mental health systems across 14 states. 9 The Child Health and Development Institute used the BTS model in 15 collaboratives to disseminate and sustain various evidence-based mental health treatments. 47 Others have focused on behavioral health and primary care integration, supported employment, addiction treatment, and emergency department care. 48 – 52 Generally, learning collaboratives used for behavioral health topics, compared to health settings, are not markedly different, although those included in Nadeem and colleagues’ systematic review tended to be shorter in length. 6

Few controlled studies have evaluated the effectiveness of learning collaboratives in behavioral health, though the body of research is growing. 51 , 53 – 55 Nadeem and colleagues’ systematic review found that among the 20 studies that met review criteria, numerous positive trends were reported on provider and patient outcomes, sustainability, and acceptability of collaboratives to providers. 6 However, the absence of detailed descriptions of studies and adequate comparison data made it impossible to draw firm conclusions about the effects of learning collaboratives in behavioral health. It appears that the amount and quality of research on behavioral health learning collaboratives is increasing, offering the prospect of improved evidence about their effectiveness.

Health equity has received little attention among learning collaboratives in either general health care or behavioral health. 56 , 57 Serious inequities in access and quality of services related to race/ethnicity, gender identity, sexual orientation, income, and other social determinants of health are widely recognized in behavioral health care. 58 – 61 Several SAMHSA-funded, disparities-focused technical assistance centers offer a range of training and resources on services for diverse populations, including the National Hispanic and Latino and National American Indian and Alaska Native Mental Health Technology Transfer Centers, Behavioral Health Centers of Excellence on African-American, Aging, and LGBTQ populations, and National Network to Eliminate Disparities in Behavioral Health. 62 It is becoming seen as an imperative that organizational and policy changes be enacted to achieve behavioral health equity. 63 , 64

Recommendations

Drawing on the appraisal of literature and key informant interviews, the following recommendations are offered for organizations that provide or receive technical assistance. The recommendations focus on the underlying rationale for choosing and using a learning collaborative for any given improvement or implementation initiative, the specifics of how a learning collaborative is conducted, including elements and structure, and issues of evaluation. General recommendations that apply to any improvement or implementation initiative appear first, followed by recommendations specific to learning collaboratives. In addition, Table ​ Table2 2 presents selected best practices in planning and conducting collaboratives summarized from the appraisal of the literature and interviews of experts.

Learning collaborative best practices

For any quality improvement or implementation initiative

Adopt an explicit framework for clarifying goals and selecting the quality improvement or implementation strategy to accomplish those goals. The goals of a quality improvement or implementation project vary, and can include changes in attitudes, knowledge, skills, clinical practice, quality, and/or patient outcomes. The providers and recipients of technical assistance should use an explicit planning framework to clarify their goals (Laderman, personal communication). As one example, the Kirkpatrick Model was developed in the 1950s to promote effective training and has been used extensively in health care education. 72 It identifies four possible goals: Reaction—whether participants find the experience favorable, engaging, and relevant; Learning—whether participants acquire the intended knowledge, skills, attitudes, confidence, and commitment; Behavior—whether participants apply what they learned; and Results—whether targeted outcomes occur as a result of the intervention. 72 After clarifying goals, decisions can be made about which process improvement or implementation strategy should be used to achieve those goals (i.e., a training workshop may be appropriate for goals of reaction or learning, whereas a learning collaborative may be more appropriate if the goals are behavior or results). Other frameworks include concept mapping and group model building. 73

Ensure that actual improvements in health care quality and health outcomes are among the goals. Although lectures, workshops, and webinars can increase awareness, facilitate learning, and shift attitudes, more intensive, multi-component interventions, like learning collaboratives, are likely necessary to change provider practice and improve patient outcomes. 30 , 43 , 44 Improving health care and patient health are the ultimate goals of quality improvement and implementation, and should be well represented among the project goals, especially as the evidence regarding whether learning collaboratives affect change at the patient level is mixed. 5 , 15 – 17

Adopt a model for improvement or logic model that explicitly identifies how planned changes will lead to desired outcomes and how those outcomes will be measured. While the selection of goals occurs at a broad conceptual level, detailed planning is needed to map out the process of change. One example is IHI’s Model for Improvement that guides the technical assistance or service provider through this process. 69 Other models include Lean, Six Sigma, and NIATx (developed specifically for behavioral health). 74 – 76 Another example is a logic model, which is a widely recognized method for visually mapping the link between resources applied, activities planned, and desired outcomes and impact. The W.K. Kellogg Foundation offers a Logic Model Development Guide as a resource for non-profit organizations. 77

Be guided by the evidence on how to promote improvement and implementation, but do not be paralyzed by imperfections in the evidence. Professional perspectives about the effectiveness of learning collaboratives differ, which came to light during expert interviews. Technical assistance providers highlighted the practical and immediate need to choose among the best available strategies to improve quality and accelerate the adoption of evidence-based practices. Researchers emphasized the imperative for more clarity and fidelity to learning collaborative models and more sophisticated evaluations of their effects. In its current, imperfect state, the body of evidence points technical assistance and service providers toward change strategies, such as learning collaboratives, that are more likely to be effective, though still must be tailored to the unique organizational contexts in which they are applied. 30 , 43 , 44

Make health equity the lens through which all work on improvement and implementation is conducted. Adoption of this recommendation is long overdue. In the words of one expert, “Improvement is not occurring if whole groups of people are being left behind.” 25 Learning collaboratives can be organized and focused on health equity projects and outcomes, as well as include a health equity perspective in all aspects of their development and evaluation. 63 – 65  Multiple frameworks and guides exist to assist health and behavioral health agencies to improve health equity. 56 , 65 , 66  Other specific recommendations related to health equity are listed in Table ​ Table2 2 .

Measure the impact of initiatives by examining participant satisfaction, practice change, patient outcomes, cost benefit, and sustainability. A thorough evaluation should measure a variety of outcomes to understand what was done and whether it worked. Relevant outcomes in learning collaborative studies include patient and provider variables, acceptability of the collaborative model to providers, sustainability of the changes achieved, impact of specific elements of the collaborative, and estimates of the cost of the collaborative. 6 In addition, the Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM) framework is frequently used to organize outcomes at both the patient/intervention level (e.g., Reach refers to how many patients receive the practice) and organization/implementation level (e.g., Adoption refers to how many providers/organizations use the practice). 67

For learning collaboratives specifically

Offer or participate in a traditional IHI BTS learning collaborative for improvement or implementation if there are clear evidence-based practices to be adopted, and sufficient resources, provider interest, and capacity to make this approach viable. While traditional learning collaboratives involve significant investments of money, time, and effort, meaningful change generally requires significant investment (Nadeem, personal communication). If the pre-conditions of an evidence base, resources, interest, and capacity are not met, then a learning collaborative, in its traditional format, may not be the best option.

Make sure that the learning collaborative includes key elements identified by IHI and in systematic reviews of research on collaboratives. Individual elements of learning collaboratives have been identified in theoretical and empirical analyses (see Table ​ Table1 1 ). 15 , 28 , 30 While the unique impact of these elements has yet to be established, some elements are frequently used, and are well-described in related materials and resources. 13 , 68 , 69 When planning and considering participation in a learning collaborative, purveyors and providers should carefully consider which elements are included. 30

Consider offering or participating in variants of a learning collaborative, like learning networks or learning communities, if the evidence-base is less clear or resources and provider interest are less robust. Learning networks, learning communities, and communities of practice are generally less rigorous, intensive, and data-driven, with a more uncertain link to patient outcomes. 25 – 27 , 70  However, they are widely used, particularly to heighten awareness, promote shared learning, and increase professional development among individual practitioners or agencies with a common interest or focus. The impact of such activities on practice change and patient outcomes is less clear.

Pair time-limited collaboratives with participation in ongoing networks to enhance sustainability. Learning collaboratives promote organization and system change to help sustain improved outcomes over time. A key contribution in the field of behavioral health has involved combining time-limited learning collaboratives to catalyze change with ongoing strategies and structures, such as learning communities, peer networks, and technical assistance centers, to sustain change. These typically function to provide continuous training and technical support, centralized data collection, benchmarking, and peer learning among participating organizations. Exceptional examples are found in the areas of child trauma (The National Child Traumatic Stress Network), evidence-based practices in children’s mental health (Child Health and Development Institute), school-based mental health (National Center for School Mental Health), and supported employment (The Individual Placement and Support [IPS] Employment Center). 4 , 9 , 47 , 50

Build capacity to offer or participate in collaboratives through internal staff development and partnerships with other organizations. Organizations can be strengthened by developing the competencies of their staff to offer or participate in learning collaboratives. Such staff development builds organizational capacity to set goals, develop strategic plans, organize change management teams, create logic models, conduct tests of change, implement evidence-based practices, improve quality, and measure outcomes. Several organizations offer guides to conducting collaboratives, compilations of change strategies, and PDSA resources. For example, IHI developed the Breakthrough Series College, IHI Open School, conferences, Improvement Coach professional development program, and a quality improvement toolkit. Other examples include the TOOLCIT Curriculum for Learning Collaborative Facilitators by the National Child Traumatic Stress Network, which includes an e-learning course, and Planning and Implementing a Successful Learning Collaborative guide by the New York Department of Health AIDS Institute. 68 , 69  Educational and training programs (such as the IHI College) are accessible ways for staff to enhance their knowledge and skills, as are the many other guides, resources, and experiential opportunities related to learning collaboratives. Partnering with other organizations to create and offer collaboratives is often an efficient way to build on internal capacities.

Document learning collaboratives and lessons learned from them in detail to maximize the value of these efforts to the field. For the knowledge base about learning collaboratives to grow, especially in behavioral health, it is imperative that those who offer and participate in quality improvement and implementation efforts document the experience and disseminate that information. This includes providing detail about the collaborative elements used, the processes implemented, and the outcomes achieved. Several sets of guidelines for such documentation exist, including “Name it, Define it, Specify it” in the field of implementation science, and the Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) from the field of health care improvement. 71 , 72  Two learning collaborative-specific strategies include the Agency for Healthcare Research and Quality’s learning collaborative taxonomy and the template for describing any learning collaborative across 14 common dimensions developed by Nadeem et al. 6 , 73 , 74

Implications for Behavioral Health

The behavioral health field has yet to adequately bridge the enormous gaps in access to high-quality, effective treatment for mental health and substance use disorders. 75 , 76  This is especially true for diverse populations. 59 – 61 Quality improvement and implementation of evidence-based practices are two related approaches being used to close these gaps. Learning collaboratives have emerged as a frequently employed strategy to assist with these efforts.

As learning collaboratives have proliferated, deviations from the traditional model have become commonplace, introducing confusion about the components of collaboratives and their effects. This commentary summarized information from the literature with wisdom drawn from interviews of experts to support technical assistance providers and behavioral health agencies in better understanding what learning collaboratives are and how to use them most effectively.

While some evidence suggests that the traditional model can change provider practice and improve patient outcomes, more research is needed to determine learning collaboratives’ effects in behavioral health care, their essential elements, and optimal length. However, technical assistance providers and behavioral health agencies should not shy away from learning collaboratives because of imperfections in the evolving research. They can draw inspiration and direction from the many high-quality examples of learning collaboratives that have been conducted in behavioral health and the wealth of available educational and training resources. Best practices, as drawn from the literature review and expert interviews, focus on the importance of maximizing interpersonal interaction (in person and virtually), carefully selecting participating organizations and their change teams, conducting pre-work, and assisting teams in implementing interventions and measuring change. Adopting a health equity lens that shapes every aspect of a collaborative is considered essential.

Learning collaboratives are viewed as a heavy lift, involving significant costs, time, and effort. 66 In an article on engaging behavioral health providers in learning collaboratives, Jensen-Doss et al. suggested that we must “…reconcile the fact that less intensive training methods, such as one-time workshops, are feasible for participants, yet generally ineffective for creating sustained practice change, whereas more intensive, more effective training methods are challenging for many providers to complete.” 77 (p.288–289)

The role of a technical assistance provider is to select an effective improvement and implementation strategy, as well as the agencies and individuals that can participate effectively in that change effort. At the same time, leaving behind other provider agencies because they have less resources and capacity is not justifiable. Such agencies are more likely serving diverse communities that already suffer from serious health inequities. While creative models and numerous resources inform the use of learning collaboratives and other comprehensive strategies to improve practice and health, more intensive advocacy is crucial to ensuring that our communities and the providers that serve them have the resources and supports necessary to be part of these solutions.

Acknowledgements

This work was supported by cooperative agreement SM081726 from the Substance Abuse and Mental Health Services Administration (SAMHSA) of the US Department of Health and Human Services (HHS). The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by SAMHSA/HHS, or the US Government. The authors acknowledge the following key informants who were interviewed for this this project: Lisa Amaya-Jackson, MD, MPH, DFAACAP, UCLA-Duke National Center for Child Traumatic Stress; Sean A. Bear, BA, National American Indian & Alaska Native Mental Health Technology Transfer Center; Deborah Becker, Med, CRC, Westat; Larry Davidson, PhD, New England Mental Health Technology Transfer Center; Lisa Dixon, MD, MPH, Columbia University Vagelos College of Physicians and Surgeons; Anita Everett, MD, Substance Abuse and Mental Health Services Administration; Dave Gustafson, PhD, University of Wisconsin; Sharon Hoover, PhD, University of Maryland School of Medicine; Mara Laderman, MSPH, Institute for Healthcare Improvement; Jason Lang, PhD, UCONN Health & Yale School of Medicine; Erum Nadeem, PhD, Rutgers Graduate School of Applied & Professional Psychology; Darice Orobitg, PhD, National Hispanic and Latino Mental Health Technology Transfer Center; Amy Reid, MPH, Institute for Healthcare Improvement; Sonja K. Schoenwald, PhD, Oregon Social Learning Center; Anne Helene Skinstad, PsyD, PhD, National American Indian & Alaska Native Mental Health Technology Transfer Center; Janis Tondora, PsyD, Yale School of Medicine; Mohini Venkatesh, MPH, National Council for Mental Well-Being.

Data Availability

Declarations.

The authors declare no competing interests.

The original online version of this article was revised: The first name of a co-author is misspelled. “Manual” should be “Manuel”.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Change history

A Correction to this paper has been published: 10.1007/s11414-023-09830-x

Contributor Information

Heather J. Gotham, Email: ude.drofnats@hmahtog .

Manuel Paris, Jr., Email: [email protected] .

Michael A. Hoge, Email: [email protected] .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Collaborative learning in practice: A systematic review and narrative synthesis of the research evidence in nurse education

Affiliations.

• 1 The Exeter School of Nursing, University of Plymouth, Topsham Rd, Exeter, Devon, EX26HA, UK. Electronic address: [email protected].
• 2 Royal Cornwall Hospitals NHS Trust, Treliske, Truro, Cornwall, TR1 3LJ, UK. Electronic address: [email protected].
• 3 School of Nursing and Midwifery, University of Plymouth, Drake Circus, Plymouth, Devon, PL48AA, UK. Electronic address: [email protected].
• 4 Health Education England, Plumer House, Tailyour Rd, Plymouth, Devon, PL6 5DH, UK. Electronic address: [email protected].
• 5 The Exeter School of Nursing, University of Plymouth, Topsham Rd, Exeter, Devon, EX26HA, UK. Electronic address: [email protected].
• 6 School of Nursing and Midwifery, University of Plymouth, The Knowledge Spa, Truro, Cornwall, TR1 3HD, UK. Electronic address: [email protected].
• PMID: 32001428
• DOI: 10.1016/j.nepr.2020.102706

Collaborative Learning in Practice is a model of placement learning for student nurses that is currently being implemented in the United Kingdom, apparently originating in Amsterdam. Potential benefits are reported to be increased placement capacity, reduced burdens on mentors as practice assessors, improvements in qualified nurses' job satisfaction, recruitment and retention, and better-developed preparedness for registrant practice amongst student nurses. We conducted a thorough, rigorous systematic review between October and December 2018 of the literature on Collaborative Learning in Practice to discover whether there was a research evidence base for these claims. We found nothing published in English in peer reviewed journals. We found 14 related papers, although these were about the Dedicated Education Unit concept, and we have conducted a narrative synthesis of them. Key findings support the assertions related to Collaborative Learning in Practice, albeit in different models of placement learning. Further research is necessary with Collaborative Learning in Practice stakeholders including staff and students, and regarding patient care metrics, to demonstrate benefits or otherwise and until that research takes place potential gains remain unproven.

Keywords: Collaborative learning in practice; Narrative synthesis; Nurse education; Nursing; Systematic review.

Copyright © 2020 Elsevier Ltd. All rights reserved.

PubMed Disclaimer

Conflict of interest statement

Declaration of competing interest The authors declare no conflicts of interest with respect to the research, authorship, and/or publication of this article. We acknowledge that JB is Quality Manager with Health Education England with a responsibility for CLIP implementation, and that AK held a Fellowship with Health Education England (SW) and also had a responsibility for CLIP implementation. HP had a responsibility for CLIP implementation in her clinical area and other authors have been involved in similar projects in their local areas.

Similar articles

• Student nurses as a future general practice nursing workforce. Implementing collaborative learning in practice: implications for placement learning and patient access. A mixed methods study. Williamson GR, Kane A, Evans S, Attrill L, Cook F, Nash K. Williamson GR, et al. BMC Nurs. 2023 Sep 21;22(1):326. doi: 10.1186/s12912-023-01501-8. BMC Nurs. 2023. PMID: 37735375 Free PMC article.
• Feasibility and outcomes of paid undergraduate student nurse positions. Gamroth L, Budgen C, Lougheed M. Gamroth L, et al. Nurs Leadersh (Tor Ont). 2006 Sep;19(3):e1-14. doi: 10.12927/cjnl.2006.19032. Nurs Leadersh (Tor Ont). 2006. PMID: 19830923
• Student and educator experiences of maternal-child simulation-based learning: a systematic review of qualitative evidence protocol. MacKinnon K, Marcellus L, Rivers J, Gordon C, Ryan M, Butcher D. MacKinnon K, et al. JBI Database System Rev Implement Rep. 2015 Jan;13(1):14-26. doi: 10.11124/jbisrir-2015-1694. JBI Database System Rev Implement Rep. 2015. PMID: 26447004
• Peer learning and collaborative placement models in health care: a systematic review and qualitative synthesis of the literature. Markowski M, Bower H, Essex R, Yearley C. Markowski M, et al. J Clin Nurs. 2021 Jun;30(11-12):1519-1541. doi: 10.1111/jocn.15661. Epub 2021 Feb 9. J Clin Nurs. 2021. PMID: 33461240 Review.
• Evidence Brief: The Quality of Care Provided by Advanced Practice Nurses [Internet]. McCleery E, Christensen V, Peterson K, Humphrey L, Helfand M. McCleery E, et al. Washington (DC): Department of Veterans Affairs (US); 2014 Sep. Washington (DC): Department of Veterans Affairs (US); 2014 Sep. PMID: 27606392 Free Books & Documents. Review.
• A concept analysis on the transfer climate in health sciences education. Hugo-Van Dyk L, Botma Y, Ndhlovu M, Nyoni CN. Hugo-Van Dyk L, et al. Heliyon. 2023 Mar 8;9(3):e14299. doi: 10.1016/j.heliyon.2023.e14299. eCollection 2023 Mar. Heliyon. 2023. PMID: 36967964 Free PMC article.
• FabNet: A Features Agglomeration-Based Convolutional Neural Network for Multiscale Breast Cancer Histopathology Images Classification. Amin MS, Ahn H. Amin MS, et al. Cancers (Basel). 2023 Feb 5;15(4):1013. doi: 10.3390/cancers15041013. Cancers (Basel). 2023. PMID: 36831359 Free PMC article.
• Learning in and across communities of practice: health professions education students' learning from boundary crossing. Stoffels M, van der Burgt SME, Bronkhorst LH, Daelmans HEM, Peerdeman SM, Kusurkar RA. Stoffels M, et al. Adv Health Sci Educ Theory Pract. 2022 Dec;27(5):1423-1441. doi: 10.1007/s10459-022-10135-5. Epub 2022 Jul 12. Adv Health Sci Educ Theory Pract. 2022. PMID: 35819568 Free PMC article.
• A phenomenographic study exploring the conceptions of stakeholders on their teaching and learning roles in nursing education. Munangatire T, McInerney P. Munangatire T, et al. BMC Med Educ. 2022 May 26;22(1):404. doi: 10.1186/s12909-022-03392-w. BMC Med Educ. 2022. PMID: 35619092 Free PMC article.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Elsevier Science
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Systematic literature review as a digital collaborative research-like learning activity: a case study

• Open access
• Published: 15 July 2023
• Volume 29 , pages 5243–5257, ( 2024 )

Cite this article

You have full access to this open access article

• Anne Wally Ryan   ORCID: orcid.org/0000-0002-3710-8416 1 ,
• Line Kolås 1 ,
• Anders Grov Nilsen 2 &
• Aslaug Grov Almås 2  

1637 Accesses

Explore all metrics

Within the higher education sector, the principle of student-active and research-based education are established in strategy documents and action plans, but at the same time there is an ongoing debate about what is meant by research-based learning and how it can be applied in practical teaching contexts. The aim of this empirical study is to explore inclusion of research elements in higher education. The study introduces the concept of using systematic literature review (SLR) and digital collaboration as a learning method, and addresses how to succeed with digital collaborative systematic literature review as a research-like learning activity in higher education. An exploratory multiple case design is used, with participatory observation technique and thematic analysis. A practical contribution of this study is an example of how SLR is well-suited to do collective research-like learning activities. The main contribution is that the higher education teacher needs integrated knowledge, including research competence in addition to the traditional link between professional, didactic and technological competence. A model for research-like learning is proposed, which illustrates the need for research knowledge in relation to the technological pedagogical content knowledge.

Similar content being viewed by others

Linking Research to Practice: Teachers as Key Stakeholders in Mathematics Education Research

Moving Research into the Classroom: Synergy in Collaboration

Digital ethnography in higher education teaching and learning—a methodological review

Explore related subjects.

• Artificial Intelligence
• Digital Education and Educational Technology

Avoid common mistakes on your manuscript.

1 Introduction

In recent years, student-active learning activities with the teacher as facilitator has become important (Damşa & de Lange, 2019 ). Robertson ( 2017 ) points out the need to rethink the roles of students and teachers in student-active learning. The interaction between students, and between students and teachers, no longer solely takes place analogously in a physical learning environment. Buzzard et al. ( 2011 ) claim that digitalization in society provides a framework for understanding whether, how, and why digital tools are used in a learning process. Student-active learning in digital environments can be challenging and change an already established work culture related to the practice of teaching, perspective on student and teacher roles, introduction and updating of new digital systems, and requirements for one’s own research (Lillejord et al., 2018 ; Bovill, 2019 ). Nerland and Prøitz ( 2018 , p. 205) call for more research on collaboration and communication in higher education.

Since the 1990s, discussion of the role of research in teaching has been increasing, and the goal of reconnecting the core activities of universities – research and teaching – in a way that consolidates and validates the values of academic life, has led to a view of ‘student as producers’, rather than ‘student as consumers’ (Neary & Winn, 2009 ; Valter & Akerlind, 2010 ; Hynes, 2018 ). A Norwegian teacher survey concludes that higher education teachers use knowledge from recent research in teaching, and that the curriculum is updated and adapted to developments in society and working life. Students are less exposed to research-like work and participate even less in research and development work (Lid et al., 2018 ). In this study, the concept “research-like learning activity” is used and defined as exploratory learning activities that activate parts of a research process within any university course, e.g., formulating researchable questions, collecting data, analysing data or performing a literature review.

The aim of this empirical study is to explore inclusion of research elements in digital collaborative student-active learning activities in higher education. Generally, any research project is based on a knowledge base within a research field. This means that one must prepare an overview of the knowledge, e.g. in the form of a systematic literature review (SLR). Students need to learn how to conduct SLRs, and this can be a learning objective in a research method course, but it is also possible to learn this through a “learning by doing” approach, as a student-active learning activity in any course. The research question of the study is: How to succeed with digital collaborative systematic literature review as a research-like learning activity in higher education?

According to Khan et al. ( 2003 ), a literature review is systematic if it is based on clearly formulated questions, identifies relevant studies, assesses the quality of the studies, and summarizes evidence using explicit methodology. The use of SLR as a learning activity is explored through three different cases at two Norwegian higher education institutions, with students from different subject areas and grade levels of higher education. The idea is to activate the students, let students collaborate and make them responsible in the learning process, and therefore the study is rooted in a social constructivist pedagogical perspective. Findings are discussed in a theoretical framework related to research-based learning, student-active learning, and collaboration technology within the scope of the teacher’s role in the process.

2 Research-based and student-active teaching in a digital learning design

The study is based on a social constructivist view of learning that differs from a behavioural pedagogical approach where the student is often more dependent on the teacher and instructions given (Adams, 2006 ). Based on the thinking of Vygotsky (1896–1934) and a broad literature study, Adams ( 2006 , p. 247) summarizes the following key characteristics of a social constructivist approach to student-active learning activities: Emphasis on learning not performance; A perspective on learners as co-constructors of meaning and knowledge; Establish a teacher-student relationship based on the idea of guidance, not instruction; Seek to engage learners in tasks seen as ends in themselves and consequently as having implicit worth; and Promote assessment as an active process of uncovering and acknowledging shared understanding. In connection with an SLR as a learning activity, the teacher will introduce a framework so that the students understand the task, but even more, will engage the students to take an active part in a process where one is collaborating to create the content.

2.1 Research-based learning and teaching

Increased emphasis on research-based learning in higher education institutions is emphasized in the Norwegian Report to the Storting (White Paper) no. 16, which emphasizes that “active participation in research among students has a clear connection with students’ ability for critical thinking, investigation and lifelong learning” (Ministry of Education & Research, 2017, p. 54). Within the higher education sector, this principle is established in strategy documents and action plans, but at the same time there is uncertainty and an ongoing debate about what is meant by research-based learning and how it can be applied in practical teaching contexts. Norwegian teaching practice, as briefly described in the introduction section, corresponds to model one of the four that Griffiths ( 2004 , p. 722) has defined in the link between research and teaching: (1) Research-led teaching, with curriculum based on research and focus on understanding research findings, not the research process itself. (2) Research-oriented teaching, where the goal is to understand the research process behind the findings. (3) Research-based teaching with exploratory learning activities. (4) Research-informed teaching where systematic exploration of teaching and learning are emphasized. The use of SLR as a research-like learning activity in this study corresponds with Griffiths’ third model. Complementary is the work of Pedaste et al. ( 2015 ) who refers to a pedagogical perspective where students are engaged in a framework of a scientific research process, where the complex scientific process is divided into research phases, and during the work the students are supervised, and important features of scientific thinking are emphasized. Willison and O’Regan ( 2007 ) proposed a framework for students’ research skill development, illustrating the process of students becoming researchers, emphasising that undergraduate students must move from a low degree of autonomy to a high degree of autonomy. In this article, digital student collaboration related to SLR is based on Khan et al.’s ( 2003 ) 5 steps for systematic literature reviews; (1) Specification of issues. (2) Identification of relevant literature and selection criteria are specified based on the issue. (3) Assessment of the quality of the studies. (4) Summary of the findings. (5) Interpretation of the findings.

We did not find any studies combining SLR as a research-like and digital student-active learning activity. However, issues as how to engage undergraduate students in research-like learning activities as SLR have been addressed by Brereton ( 2010 ) who explored the effectiveness of second-year undergraduate computing students in carrying out an SLR, and identified the elements of the process that the students found most difficult. For example, the conduct phase was more problematic than the planning phase. It can be concluded that undergraduates can do SLRs, but the task is clearly quite challenging and time-consuming. SLRs are well suited to being undertaken by groups (Brereton, 2010 ).

2.2 Collaboration technology

In this study, we take as our starting point that research is a collective process, and we further explore how digital tools can be used to increase interaction among students in a research process. Aagaard et al. ( 2018 ) conclude that digitalisation promotes relevance as well as making education accessible, flexible and open to more student-active forms of learning. For technology to function as a collaboration tool, everyone involved should experience an immediate understanding of others’ interaction in a shared work area (Gutwin & Greenberg, 2002 ). This is useful for coordination, initiation of collaboration and communication about the task at hand. Digital tools for collaborative learning are related to the research field CSCL, Computer-Supported Collaborative Learning (Stahl et al., 2006 ). Jeong and Hmelo-Silver ( 2016 ) present 7 opportunities within CSCL, which address specific needs and challenges that students experience through collaboration: opportunities to engage in a common task; communicate; share resources; participate in productive collaborative learning; co-create; monitor and regulate collaborative learning; and find and build groups and communities. Collaboration technology represents potential in the development of student-active forms of learning where students can also be a resource for each other. In this study we elaborate on both opportunities and challenges when such tools are to be used in connection with an SLR.

Koehler and Mishra ( 2009 ) illustrate the interrelated knowledge and skills that teachers need in order to use technology meaningfully into their teaching through their TPACK-model that highlights a need for technological pedagogical content knowledge. Teaching and learning with technology exist in a dynamic transactional relationship between the three components technology, pedagogic and content, and a change in any one of the factors has to be ‘‘compensated’’ by changes in the other two (Mishra & Koehler, 2006 ).

This study was conducted as an exploratory multiple case study inspired by Stake ( 1995 ), and based on an understanding that knowledge is constructed socially. In order to achieve enough in-depth information about what characterizes a research-like learning activity using collaboration technology, a qualitative approach was used. Data was collected using participatory observation, and the empirical data is thematically analysed and interpreted in a hermeneutic interpretive tradition.

3.1 Selection of cases and data collection

To obtain characteristics of the learning activity that did not depend only on a specific level of education or academic affiliation, we selected three cases based on the criteria of a variation of subject areas and study level. The study programs involved were Games and Entertainment Technology with the bachelor courses “Game Lab 2” and “Game Design 2” (anonymous University), Geography with the master course “Experience-based value creation, the environment and passion” (anonymous University), and Teacher Education with the master course “MASIKT-TEK02 Technological profile subject 2 ”(anonymous University of Applied Sciences). The two bachelor courses involved the same student group. The researchers were also teachers, who planned and facilitated the use of SLR as a learning activity, and adapted SLR and digital tools to their subject area and study level. Data was collected through participatory observation, where the teachers/researchers had the role of “participant as observer” (Cohen et al., 2018 , p. 543). Predetermined topics including the use of collaboration technology, learning resources, student autonomy, student engagement, collaboration between students, and the role of the teacher were used to narrow the field of observation. These topics were based on the research teams’ preconceptions and constituted the structure of a digital observation form. The observation form also included an open-ended question for unexpected occurrences. After each learning session, the teacher/researcher made notes in the digital observation form. All the researchers reported their observations as textual data, and these notes were later assembled in one common document, available for the whole research team. The following section describes the three cases structured with the five steps for SLR from Khan et al. ( 2003 ).

3.1.1 Case 1: Games and Experience Technology

In the first week of the semester, 22 students participated in conducting an SLR within the theme “Educational games”. In step 1, the teacher formulated 3 research questions (RQ) with different perspectives; game developers’ (RQ1), educators’ (RQ2) and researchers’ (RQ3). Steps 2 and 3 were also completed by the teacher who selected 29 articles for RQ1, 24 articles for RQ2 and 8 articles for RQ3. With the 2nd year bachelor students, it was challenging to find relevant databases and articles. The teacher took articles from academic journals, but they were too specific in the topic for RQ1, so more popular science articles were chosen for the game developer perspective (RQ1). The articles were made available digitally, and according to the “first-come-first-served” principle, the students chose 3 articles each, which they analysed (step 4). The articles for RQ1 and RQ2 were analysed individually, while the articles for RQ3 were analysed in groups. The students analysed the same points in all articles, based on an analysis table initially made by the teacher, and adjusted through a discussion in class. The analysis table was first available in a digital co-authoring document, but this worked poorly, and the teacher made a digital questionnaire where the students filled out one form per article. This did not work optimally either, as the students had to be added manually to access the answers from other students. Interpretation of findings (step 5) was first done in a plenary session in the classroom, and then each student group had to use relevant findings when presenting their game ideas.

3.1.2 Case 2: Geography

The students from the master’s program in social sciences with specialization in Geography took the course “Experience based value creation, the environment and passion” in their 2nd semester. The study is a hybrid study programme, predominantly online-based with occasional physical meetings. Eight students, divided into two groups, participated. The students were introduced to an SLR and were briefed on organization, progress, and group composition. The university’s collaboration tool OneDrive (with associated editing software) was chosen. The co-authoring document was a table with a framework for the literature review, based on a revised version of the five steps of Khan et al. ( 2003 ). The teacher formulated an overarching research question and the students had to search for scientific articles in Google Scholar to show the large number of hits they get with a wide research question and the need for their own refinement (step 1). In step 2, search engines, keywords, and the number of hits, as well as 3–4 relevant articles were noted. In step 3, the student selected an article, wrote a summary, and noted a reference. In step 4, the article was analysed with a focus on relevant arguments and a geographical perspective. Step 5 should have been a more critical interpretation in a real research situation but was modified to identify further research needs discussed in the article. In the end, everyone had a table with 8 relevant articles.

3.1.3 Case 3: MASIKT-TEK

In the master’s program ICT in learning, 23 students participated in three activities connected to systematic literature review. The learning activities were part of the assessment basis from a course in the second semester. The students had a first draft of the problem ready (step 1). This was further developed through a systematized literature search. The literature search was required to produce an overview of which databases, time, focus, type of activity, language, keywords, method, and result were included/excluded. The overview served as a scaffolding for step 2. In steps 3, 4 and 5, the students reflected by writing on previous research. The literature review was an individual piece of work, but before completion, the students presented their work in online meetings to the class for guidance and input. The students worked in groups, and despite the individual focus, the teacher held the group with common milestones in the research process. In the second learning activity “Writing summaries”, all students had to read and write a summary of the same scientific article (selected by the teacher). The summaries were collected in a digital booklet, accessible to all, and were discussed in groups and a plenary session, with the aim of learning to extract the most important things in articles and formulate this in a summary. The learning activity provided an opportunity to learn from each other and increase the quality of the work through collaboration. In the final learning activity, the students worked in groups with an argumentation analysis of the same article, based on several points prepared by the teacher. The groups presented the analyses in a plenary session afterwards. In the last two learning activities, the focus was on step 3 (analysis) in an SLR, and the teacher focused the work by selecting the article text (steps 1 and 2).

3.2 Ethical reflections on the study

An exploratory study divided into three different cases regarding study level, learning outcomes, organization and scope can be challenging methodologically, however can also be a strength in that more researchers encourage characteristics enabling the relevance of the study to increase. The responsible teacher anonymised the students before the data was processed together after the learning activities. The study follows the national research ethics guidelines in Norway.

3.3 Thematic analysis

Data from the three cases were dealt with as a comprehensive text. We used thematic analysis to identify, analyse and report patterns (themes) in the data material, cf. Braun and Clarke, 2006 . Based on the research question and elaboration of this, as well as the observation form, we initially defined codes such as student activity, and opportunities and challenges in the use of digital tools. Later in the thematic analysis, we identified patterns in the data such as functionality of collaboration technology, relevance, and usefulness of SLR, and adaption to educational level. Four main themes were defined: (1) Structure/coordination, (2) Professional focus, (3) Relevance and motivation, and (4) Challenges. These were systematized in a cross-linking table (Table  1 ) to further analyse the themes in relation to four key parts of an SLR learning design, related to SLR as a digital collaborative research activity: Learning Activity, Collaboration Technology, Activating students and The Role of the Teacher .

Overall, Table  1 constitutes a systematization of the main themes found in the thematic analysis of the use of SLR as a research-like learning activity. The analysis clarified the importance of the teacher role as a facilitator in a digital collaborative research-like learning activity, both related to the competence with regards to digital collaborative tools, and experience using SLR in a research context. Further, SLR with the use of collaboration technology provides scaffolding opportunities for exploratory learning and co-creation of knowledge, more specifically through coordination of communication and interaction among students. However, using SLR and digital collaboration tools may provide challenges for students to analyse from in-depth understanding to a larger whole. It is important that the teachers prepare the teaching sessions thoroughly and that the teacher functions as an active facilitator in the process. A main characteristic is that the role of the teacher, in addition to exercising a traditional link between content, pedagogical and technological competence, also include research competence.

4 Findings and discussion

Research-like learning activity such as SLR using collaboration technology requires that the teacher has a complex competence, which corresponds to and expand Koehler and Mishra’s (2009) TPACK model where they emphasise that the teacher must have technological knowledge, pedagogical knowledge, and content knowledge, and be able to connect these competencies. A main finding is that research competence, including knowledge about digital tools which can be used in a collaborative research process, knowledge about research designs relevant for the content area and knowledge about explorative learning activities, e.g., SLR is useful in research-like learning activities. Based on this main finding, we propose an expanded TPACK-model, called “TPACK and Research-like learning” (see Fig.  1 ), which illustrates the need for research knowledge in relation to the technological pedagogical content knowledge.

TPACK and Research-like learning (an expanded TPACK model based on Koehler and Mishra, 2009 )

The model “TPACK and research-like learning” will in the following paragraphs be further elaborated and discussed based on SLR as a digital collaborative research-like learning activity. First, with an individual focus on Explorative learning activity knowledge, then focus on Research design knowledge and Digital research tool knowledge, and finally a focus on the integrated parts, also referring to the analysis (Table  1 ).

4.1 Explorative learning activity knowledge

Integration of research methods in learning activities is relevant in higher education for students to learn how research-based knowledge is produced, cf. Report to the Norwegian Storting, no. 16 (Ministry of Education & Research, 2017, p. 45): “… Higher education is in a unique situation when it comes to educating candidates who can read and use research, ask critical questions and use scientific methods to solve problems during their studies and in working life.” Implementation of SLR as a research-like learning activity in this study’s cases support that SLR is well-suited, partly because the method has defined procedures, and in addition was perceived as useful because students could apply the methodology in their own bachelor or masters’ theses, cf. Table  1 . By using SLR as a learning method, the students in the three cases gained training in interaction with others and insight into research as a collective process.

In this study, the five steps of Khan et al. ( 2003 ) acted as scaffolding in students’ explorative learning and contributed to the teacher’s planning. There are several similar methodologies for SLR with clear steps and procedures that can work in a learning process (Badger et al., 2000 ; Tranfield et al., 2003 ), and an assessment and adaptation must be made for use in a learning process, requiring an explorative learning activity knowledge among teachers. Pedaste et al. ( 2015 ) also had the goal of structuring learning in a scientific research process divided into phases, and in addition highlighting and integrating scientific thinking. As previously mentioned, higher education in Norway has mainly used research-led teaching, just one of Griffiths’ ( 2004 ) four models of research-based teaching. SLR using digital collaboration tools is an exploratory learning activity that exemplifies Griffiths’ ( 2004 ) third model in the link between research and education. It is not a traditional learning method, but the experience from the cases shows that the method was not so complex as to decrease focus from the intended learning outcomes. Explorative learning activity knowledge means that teachers in higher education need pedagogical knowledge about how to use research methods, e.g., SLR, as frameworks for their teaching practices to succeed with research-like learning activities.

4.2 Research design knowledge

As mentioned earlier, the Norwegian Ministry of Education and Research ( 2017 , p.54) emphasizes that “active participation in research among students has a clear connection with students’ ability for critical thinking, investigation and lifelong learning”. This corresponds with research in other contexts, e.g., Valter and Akerlind’s ( 2010 ) study in Australia involving introducing students to ways of thinking and acting like researchers, with a focus on how discovery- and research-led education can be introduced into mainstream curriculum in an affordable way.

TPACKs “content knowledge” must in research-like learning be regarded as the teacher’s knowledge of relevant and discipline-specific research topics and methods. Even though there are several, but quite similar, methodologies for SLR (Badger et al., 2000 ; Khan et al., 2003 ; Tranfield et al., 2003 ), SLR is a general methodology useable in most study programs. This study underlines the notion that teachers must adapt the SLR-methodology to the course content. This relates to the students’ motivation and engagement to participate in the learning activity. Further, the teacher must make adaptions of the learning content to fit the educational level. The teacher in the bachelor course prepared several of the Khan et al.’s ( 2003 ) five steps, while the teachers of the master courses would leave more of the work to the students. This corresponds to Willison and O’Regan’s ( 2007 ) framework for students’ research skill development, where they divided the skill development into five levels of student autonomy. The student collaboration at bachelor and master’s level each had a somewhat different character and function, but a common feature was the usefulness of each individual student having to contribute to the production of knowledge and assembling that knowledge into a system that gave a greater overview and learning outcomes.

4.3 Digital research tool knowledge

Digital research tools can include both specific research tools e.g. NVIVO for qualitative analysis, SPSS for quantitative analysis, and general tools, e.g. collaboration tools, text editors, spreadsheets, which also are useful tools in a research process. This study highlights research as a collective activity, which requires a focus on digital collaborative tools. Collaboration technology was used as a coordination tool where, for example, the division of labour in the student group was handled. In addition, the collaboration tools contributed to co-creation and the analysis process. An example is using the analysis table or the five steps in SLR as the structure of a shared document. This simplifies the compilation and synthesis of findings. Using OneDrive or Google enabled co-creation in shared documents (Jeong & Hmelo-Silver, 2016 ), both synchronously and asynchronously in the learning process. Getting an overview of the research field in the start-up of a research project can be overwhelming and challenging. By facilitating collaboration and co-creation of knowledge, the students may experience an added value, both in terms of academic and social benefits. The use of collaboration technology made digital tools visible as a means and added value for the learning process, including the interaction between students and between students and the teacher, cf. Blau and Shamir-Inbal, 2017 , Bovill ( 2019 ) and Buzzard et al. ( 2011 ). In line with Jeong and Hmelo-Silver ( 2016 ), findings from the study showed that the students were engaged in solving the task together, and digital sharing of the documents made progress in the process visible and obliged everyone in the group to contribute. In this way, “free riders”, who can be a challenge in group work, were avoided.

In this study, various digital collaboration tools were used, such as co-writing in word processors and spreadsheets, as well as the use of digital questionnaires. The advantage of co-writing in a word processor is that all participants have access to a common document for reading and editing, and all analyses are collected in the same document. This gave the students a common understanding of the co-creation process, cf. Gutwin and Greenberg, 2002 . Some of the cases experienced challenges in handling large joint documents, and therefore chose other tools along the way. Shared spreadsheets allow for co-creation by letting students fill in tables together. Each student filled in the findings in either “their” column or “their” row. This can be experienced systematically when registering but can also provide an overwhelming amount of data when comparing findings afterwards. With the questionnaire tool, each student can enter their analyses by answering specific questions. The advantages are that the students make a structured analysis, and that presentation of the results can be structured in a spreadsheet format and diagrams afterwards. The challenge is that access to each other’s analyses is limited during the process and thus not optimal in a co-creation process, cf. Gutwin and Greenberg, 2002 , as the individual student contributed without seeing their fellow students’ contributions along the way. These experiences indicate that teachers need knowledge of both the usefulness of, and barriers to using collaborative digital research tools, in order to succeed with research-like learning activities in higher education.

4.4 TPACK and research-like learning – integrated knowledge need

The analysis of the study indicates that all three cases cover four of five characteristics of the student-active approach to teaching highlighted by Adams ( 2006 ); exploratory learning activities, students as co-producers, the teacher as supervisor, and emphasis on work tasks where the process is the goal. Findings in the study underline that the teachers need to prepare thoroughly, facilitate the process, and provide good information about SLR as method and its relevance, and digital tools for collaboration. Digitalisation enhance the student-active forms of learning, cf. Aagaard et al., 2018 , and also in research-like learning activities. Overall, it is essential to clarify information to the students about the purpose, implementation, and expectations of participation in a research-like learning activity. The purpose is for students to emphasize the academic learning process as an end in itself (Damşa & de Lange, 2019 ; Blikstad-Balas, 2019 ).

The core of Fig.  1 illustrates the teachers’ holistic integrated knowledge need, connecting respectively explorative learning activity knowledge, research design knowledge and digital research tool knowledge. To succeed with research-like learning activities as SLR, the teacher must consider the disciplinary learning outcome when selecting collaborative digital research tools.

In all three cases, an SLR using digital collaboration technology was tested for the first time. The first implementation involved, among other things, revising the framework offered by Khan et al. ( 2003 ) for SLR with the 5 steps so that it was adapted to a relevant professional context, and in addition select relevant and available digital collaboration tools. In carrying out the cases, we were aware of if the tool itself was receiving too much focus at the expense of the academic content and learning. By starting with relatively simple and “familiar” tools, no issues in the use of these tools were discovered in this context. Bower ( 2017 ) emphasizes that it is the teacher with a pedagogical drive who ensures that technology-supported learning takes place, and that this is central to whether the learning experience is positive or not. To check the quality of the student groups’ understanding of the task and to ensure progress in the learning process, the study showed the need for a thorough review of the five steps, differentiated for different levels. For example, greater demands were made on students at master’s level concerning responsibility and independence in solving the tasks.

After the learning activity started, the teachers in the cases became active supervisors and facilitators in the process. Bower ( 2017 , p. 417) emphasises that in such a perspective, the role of the teacher is not only to be part of the social learning environment, answer questions and provide intellectual guidance, but also structure activities so that students can learn from and with each other.

5 Conclusion

The exploratory multiple case study introduces the concept of using systematic literature review (SLR) and digital collaboration as learning method in higher education. Using participatory observation technique and thematic analysis, we identified characteristics on how to succeed with digital collaborative systematic literature review as a research-like learning activity in higher education.

The literature review refers to recent studies concluding that students seldom are exposed to research-like work. A practical contribution of this study is an example of how SLR is well-suited to do collective research-like learning activities. Findings underline that the teachers need to provide good information about SLR as method and its relevance in a research process. Selecting familiar digital tools for collaboration can ensure that the students are focusing on the co-creation of content, and hopefully avoid technological obstacles. Adapting the SLR-methodology to the course content is important for the engagement and motivation of the students.

The main contribution of the study is that the higher education teacher needs integrated knowledge, including research competence in addition to the traditional link between professional, didactic, and technological competence. The study adds to the existing body of knowledge a model for research-like learning, called “TPACK and Research-like learning – an expanded TPACK model”. Based on the existing TPACK model, our proposed model illustrates the need for research knowledge in relation to the technological pedagogical content knowledge. The expanded model is discussed focusing on the need for explorative learning activity knowledge, research design knowledge and digital research tool knowledge. Finally, an integrated view of these needs must be seen in relation to each other. The model illustrates one way of achieving both student-active learning, where students are encouraged to engage with course content, digital collaborative learning, and research-based education.

Our findings suggest that the hiring of new teachers in higher education should emphasize the applicants’ research background, in addition to teaching qualifications to strengthen student-active learning and research-based teaching with exploratory learning activities. In addition, newly hired teachers should be offered/required to take pedagogical courses for university levels, which increasingly should include learning goals on ways of teaching research-like learning activities and using digital tools to enhance student collaboration to reach the aim of research-based higher education. However, it must also be remembered, that research-like learning activities are only one of Griffiths’ ( 2004 ) four models of research-based teaching.

Looking ahead, there is a need for more research on how to perform summative assessment when using research-like learning method in higher education, to obtain a goal of constructive alignment, with a clear connection between learning objectives, learning activities and assessment. There is also need for more knowledge on SLR as a learning activity, from the students’ perspective.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to national regulation with regards to individual privacy.

Aagaard, T., Lund, A., Lanestedt, J., Ramberg, K. R., & Swanberg, A. B. (2018). Sammenhenger mellom digitalisering og utdanningskvalitet – innspill og utspill. Uniped , 41 (3), 289–303. https://doi.org/10.18261/issn.1893-8981-2018-03-09 .

Article   Google Scholar  

Adams, P. (2006). Exploring social constructivism: Theories and practicalities. Education 3–13 , 34 (3), 243–257. https://doi.org/10.1080/03004270600898893 .

Badger, D., Nursten, J., Williams, P., & Woodward, M. (2000). Should all literature reviews be systematic? Evaluation & Research in Education , 14 (3–4), 220–230. https://doi.org/10.1080/09500790008666974 .

Blau, I., & Shamir-Inbal, T. (2017). Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation. Computers & Education , 115 , 69–81. https://doi.org/10.1016/j.compedu.2017.07.014 .

Blikstad-Balas, M. (2019). Digital teknologi i klasserommet – noen sentrale utfordringer. I T. A. Wølner, K. Kverndokken, M. Moe, & H. H. Siljan (Red.), 101 digitale grep – en didaktikk for profesjonsfaglig digital kompetanse (pp. 51–64). Fagbokforlaget.

Bovill, C. (2019). A co-creation of learning and teaching typology: What kind of co-creation are you planning or doing? International Journal for Students as Partners , 3 (2), 91–98. https://doi.org/10.15173/ijsap.v3i2.3953 .

Bower, M. (2017). Design of technology-enhanced learning: Integrating research and practice . Emerald Publishing.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology , 3 (2), 77–101. https://doi.org/10.1191/1478088706qp063oa .

Brereton, P. (2010). A study of computing undergraduates undertaking a systematic literature review. IEEE Transactions on Education , 54 (4), 558–563. https://doi.org/10.1109/TE.2010.2090662 .

Buzzard, C., Crittenden, V. L., Crittenden, W. F., & McCarty, P. (2011). The Use of Digital Technologies in the Classroom: A teaching and learning perspective. Journal of Marketing Education , 33 (2), 131–139. https://doi.org/10.1177/0273475311410845 .

Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8nd ed.). Routledge.

Damşa, C., & de Lange, T. (2019). Student-centred learning environments in higher education: From conceptualization to design. Uniped , 42 (1), 9–26. https://doi.org/10.18261/issn.1893-8981-2019-01-02 .

Griffiths, R. (2004). Knowledge production and the research–teaching nexus: The case of the built environment disciplines. Studies in Higher Education , 29 (6), 709–726. https://doi.org/10.1080/0307507042000287212 .

Gutwin, C., & Greenberg, S. (2002). A descriptive Framework of Workspace Awareness for Real-Time Groupware. Computer Supported Cooperative Work (CSCW) , 11 (3–4), 411–446. https://doi.org/10.1023/A:1021271517844 .

Hynes, M. (2018). Students-as-producers: Developing valuable student-centered research and learning opportunities. International Journal of Research Studies in Education , 7 (4), 1–13. https://doi.org/10.5861/ijrse.2017.1858 .

Jeong, H., & Hmelo-Silver, C. E. (2016). Seven Affordances of Computer-Supported collaborative learning: How to support collaborative learning? How can Technologies help? Educational Psychologist , 51 (2), 247–265. https://doi.org/10.1080/00461520.2016.1158654 .

Khan, K. S., Kunz, R., Kleijnen, J., & Antes, G. (2003). Five steps to conducting a systematic review. Journal of the Royal Society of Medicine , 96 (3), 118–121. https://doi.org/10.1177/014107680309600304 .

Koehler, M., & Mishra, P. (2009). What is Technological Pedagogical Content Knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education , 9 (1), 60–70. Retrieved from https://www.learntechlib.org/primary/p/29544 .

Lid, S. E., Pedersen, L. F., & Damen, M. L. (2018). Underviserundersøkelsen 2017. Hovedtendenser . Oslo: NOKUT. Retrieved from https://www.nokut.no/globalassets/studiebarometeret/underviserundersokelsen/lid_pedersen_damen_underviserundersokelsen-2017_hovedtendenser_2-2018.pdf .

Lillejord, S., Børte, K., Nesje, K., & Ruud, E. (2018). Learning and teaching with technology in higher education – a systematic review . Knowledge Centre for Education. Retrieved from https://www.forskningsradet.no/siteassets/publikasjoner/1254035532334.pdf .

Ministry of Education and Research (2017). Meld. St. 16 (2016–2017). Quality Culture in Higher Education. Ministry of Education and Research. https://www.regjeringen.no/en/dokumenter/meld.-st.-16-20162017/id2536007/

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers college record , 108 (6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x .

Neary, M., & Winn, J. (2009). The student as producer: reinventing the student experience in higher education. In Bell, L., Neary, M., & Stevenson, H. (Eds.). (2009). The future of higher education: policy, pedagogy and the student experience, 192–210 . A&C Black.

Nerland, M., & Prøitz, T. S. (2018). Pathways to quality in higher education: Case studies of educational practices in eight courses. Oslo: NIFU. Retrieved from https://nifu.brage.unit.no/nifu-xmlui/bitstream/handle/11250/2478911/NIFUreport2018-3 .pdf?sequence=1&isAllowed=y.

Pedaste, M., Mäeots, M., Siiman, L. A., de Jong, T., van Riesen, S. A., Kamp, E. T., Manolic, C. C., Zachariac, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review , 14 , 47–61. https://doi.org/10.1016/j.edurev.2015.02.003 .

Robertson, J. (2017). Rethinking learner and teacher roles: Incorporating Student Voice and Agency into Teaching Practice. Journal of Initial Teacher Inquiry , 3 , 41–44. https://doi.org/10.26021/802 .

Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported Collaborative Learning: An Historical Perspective. I R. K. Sawyer (Red.), The Cambridge Handbook of The Learning Sciences (pp. 409–426). Cambridge: Cambridge University Press. Retrieved from http://gerrystahl.net/cscl/CSCL_English.pdf .

Stake, R. E. (1995). The art of case study research . SAGE.

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management , 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375 .

Valter, K., & Akerlind, G. (2010). Introducing students to Ways of thinking and acting like a researcher: A Case Study of Research-led education in the Sciences. International Journal of Teaching and Learning in Higher Education , 22 (1), 89–97.

Google Scholar  

Willison, J., & O’Regan, K. (2007). Commonly known, commonly not known, totally unknown: A framework for students becoming researchers. Higher Education Research & Development , 26 (4), 393–409. https://doi.org/10.1080/07294360701658609 .

Download references

Acknowledgements

Not applicable.

Open access funding provided by Nord University. The research was funded by Norway Opening Universities (NOU) and the authors declare that they have no competing interests.

Author information

Authors and affiliations.

Faculty of Social Sciences, Nord University, Postboks 93, Levanger, 7601, Norway

Anne Wally Ryan & Line Kolås

Department of Pedagogy, Religion and Social Studies, Western Norway University of Applied Sciences, Klingenbergvegen 8, Stord, 5414, Norway

Anders Grov Nilsen & Aslaug Grov Almås

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Anne Wally Ryan .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Ryan, A.W., Kolås, L., Nilsen, A.G. et al. Systematic literature review as a digital collaborative research-like learning activity: a case study. Educ Inf Technol 29 , 5243–5257 (2024). https://doi.org/10.1007/s10639-023-11997-x

Download citation

Received : 29 December 2022

Accepted : 20 June 2023

Published : 15 July 2023

Issue Date : April 2024

DOI : https://doi.org/10.1007/s10639-023-11997-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Collaboration technology
• Research-like learning
• Student-active learning
• Systematic literature review
• Find a journal
• Publish with us
• Track your research

Literature Review: Effect of Cooperative Learning on Intrinsic Motivation

• August 2021
• 08(07):01-07

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Mindo Uly Sinaga
• Pesta Simamora
• Samsul Mustika

• Weixuan Shi
• Tarek Ghodbane
• Hafida Hamzaoui El Achachi

• Roberto Llamedo

• Siti Ermawati

• PSYCHOL SCHOOLS

• EDUC LEADERSHIP

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up