electronic health record research journal

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 25 September 2019

The future of electronic health records

• Jeff Hecht 0

Jeff Hecht is a science writer based in Newton, Massachusetts.

You can also search for this author in PubMed   Google Scholar

Credit: Totto Renna

Advances in medical imaging and the proliferation of diagnostic and screening tests have generated mountains of data on patient health. Digital information technology has seemed poised to revolutionize health care in the United States since 2009, when the Obama administration made the technology part of plans to revive a sinking economy. The US government has now spent tens of billions of dollars on putting patient information at doctors’ fingertips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Nature 573 , S114-S116 (2019)

doi: https://doi.org/10.1038/d41586-019-02876-y

This article is part of Nature Outlook: Digital health , an editorially independent supplement produced with the financial support of third parties. About this content .

Dinov, I, D. J. Med. Stat. Inform. https://dx.doi.org/10.7243/2053-7662-4-3 (2016).

PubMed   Google Scholar  

Liberatore, K. Pa. Patient Saf. Advis. 15 (Suppl.), 16–24 (2018).

Google Scholar  

Arndt, B. G. et al. Ann. Fam. Med. 15 , 419–426 (2017).

Article   PubMed   Google Scholar  

Shanafelt, T. D. et al. Mayo Clin. Proc. 90 , 1600–1613 (2015).

Download references

Related Articles

• Information technology
• Health care

The dream of electronic newspapers becomes a reality — in 1974

News & Views 07 MAY 24

How scientists are making the most of Reddit

Career Feature 01 APR 24

A global timekeeping problem postponed by global warming

Article 27 MAR 24

AI’s keen diagnostic eye

Outlook 18 APR 24

So … you’ve been hacked

Technology Feature 19 MAR 24

No installation required: how WebAssembly is changing scientific computing

Technology Feature 11 MAR 24

Bird flu in US cows: where will it end?

News 08 MAY 24

US funders to tighten oversight of controversial ‘gain-of-function’ research

News 07 MAY 24

Microbubble ultrasound maps hidden signs of heart disease

News & Views 06 MAY 24

Faculty Positions in School of Engineering, Westlake University

The School of Engineering (SOE) at Westlake University is seeking to fill multiple tenured or tenure-track faculty positions in all ranks.

Hangzhou, Zhejiang, China

Westlake University

High-Level Talents at the First Affiliated Hospital of Nanchang University

For clinical medicine and basic medicine; basic research of emerging inter-disciplines and medical big data.

Nanchang, Jiangxi, China

The First Affiliated Hospital of Nanchang University

Technician / Senior Technician in Structural Biology of Membrane-Less Organelles

Job description APPLICATION CLOSING DATE: June 15th, 2024. Human Technopole (HT) is a distinguished life science research institute founded and sup...

Human Technopole

Research Associate (part-time) / Ph.D. candidate in Surface Science

The University of Bonn is an international research university with a wide education and research profile. With a 200-year history, approximately...

Bonn, Nordrhein-Westfalen (DE)

Rheinische Friedrich-Wilhelms-Universität

Research assistant (praedoc) (m/f/d) - Department of Physics

Department of Physics - Institute of Experimental Physics   Research assistant (praedoc) (m/f/d) with 75 %part-time job limited up to 4 years salar...

Berlin (DE)

Freie Universität Berlin

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open access
• Published: 27 October 2021

A narrative review on the validity of electronic health record-based research in epidemiology

• Milena A. Gianfrancesco 1 &
• Neal D. Goldstein   ORCID: orcid.org/0000-0002-9597-5251 2  

BMC Medical Research Methodology volume  21 , Article number:  234 ( 2021 ) Cite this article

11k Accesses

46 Citations

5 Altmetric

Metrics details

Electronic health records (EHRs) are widely used in epidemiological research, but the validity of the results is dependent upon the assumptions made about the healthcare system, the patient, and the provider. In this review, we identify four overarching challenges in using EHR-based data for epidemiological analysis, with a particular emphasis on threats to validity. These challenges include representativeness of the EHR to a target population, the availability and interpretability of clinical and non-clinical data, and missing data at both the variable and observation levels. Each challenge reveals layers of assumptions that the epidemiologist is required to make, from the point of patient entry into the healthcare system, to the provider documenting the results of the clinical exam and follow-up of the patient longitudinally; all with the potential to bias the results of analysis of these data. Understanding the extent of as well as remediating potential biases requires a variety of methodological approaches, from traditional sensitivity analyses and validation studies, to newer techniques such as natural language processing. Beyond methods to address these challenges, it will remain crucial for epidemiologists to engage with clinicians and informaticians at their institutions to ensure data quality and accessibility by forming multidisciplinary teams around specific research projects.

Peer Review reports

The proliferation of electronic health records (EHRs) spurred on by federal government incentives over the past few decades has resulted in greater than an 80% adoption-rate at hospitals [ 1 ] and close to 90% in office-based practices [ 2 ] in the United States. A natural consequence of the availability of electronic health data is the conduct of research with these data, both observational and experimental [ 3 ], due to lower overhead costs and lower burden of study recruitment [ 4 ]. Indeed, a search on PubMed for publications indexed by the MeSH term “electronic health records” reveals an exponential growth in biomedical literature, especially over the last 10 years with an excess of 50,000 publications.

An emerging literature is beginning to recognize the many challenges that still lay ahead in using EHR data for epidemiological investigations. Researchers in Europe identified 13 potential sources of “bias” (bias was defined as a contamination of the data) in EHR-based data covering almost every aspect of care delivery, from selective entrance into the healthcare system, to variation in care and documentation practices, to identification and extraction of the right data for analysis [ 5 ]. Many of the identified contaminants are directly relevant to traditional epidemiological threats to validity [ 4 ]. Data quality has consistently been invoked as a central challenge in EHRs. From a qualitative perspective, healthcare workers have described challenges in the healthcare environment (e.g., heavy workload), imperfect clinical documentation practices, and concerns over data extraction and reporting tools, all of which would impact the quality of data in the EHR [ 6 ]. From a quantitative perspective, researchers have noted limited sensitivity of diagnostic codes in the EHR when relying on discrete codings, noting that upon a manual chart review free text fields often capture the missed information, motivating such techniques as natural language processing (NLP) [ 7 ]. A systematic review of EHR-based studies also identified data quality as an overarching barrier to the use of EHRs in managing the health of the community, i.e. “population health” [ 8 ]. Encouragingly this same review also identified more facilitators than barriers to the use of EHRs in public health, suggesting that opportunities outweigh the challenges. Shortreed et al. further explored these opportunities discussing how EHRs can enhance pragmatic trials, bring additional sophistication to observational studies, aid in predictive modeling, and be linked together to create more comprehensive views of patients’ health [ 9 ]. Yet, as Shortreed and others have noted, significant challenges still remain.

It is our intention with this narrative review to discuss some of these challenges in further detail. In particular, we focus on specific epidemiological threats to validity -- internal and external -- and how EHR-based epidemiological research in particular can exacerbate some of these threats. We note that while there is some overlap in the challenges we discuss with traditional paper-based medical record research that has occurred for decades, the scale and scope of an EHR-based study is often well beyond what was traditionally possible in the manual chart review era and our applied examples attempt to reflect this. We also describe existing and emerging approaches for remediating these potential biases as they arise. A summary of these challenges may be found in Table 1 . Our review is grounded in the healthcare system in the United States, although we expect many of the issues we describe to be applicable regardless of locale; where necessary, we have flagged our comments as specific to the U.S.

Challenge #1: Representativeness

The selection process for how patients are captured in the EHR is complex and a function of geographic, social, demographic, and economic determinants [ 10 ]. This can be termed the catchment of the EHR. For a patient record to appear in the EHR the patient must have been registered in the system, typically to capture their demographic and billing information, and upon a clinical visit, their health details. While this process is not new to clinical epidemiology, what tends to separate EHR-based records from traditional paper-based records is the scale and scope of the data. Patient data may be available for longer periods of time longitudinally, as well as have data corresponding to interactions with multiple, potentially disparate, healthcare systems [ 11 ]. Given the consolidation of healthcare [ 12 ] and aggregated views of multiple EHRs through health information networks or exchanges [ 11 ] the ability to have a complete view of the patients’ total health is increasing. Importantly, the epidemiologist must ascertain whether the population captured within the EHR or EHR-derived data is representative of the population targeted for inference. This is particularly true under the paradigm of population health and inferring the health status of a community from EHR-based records [ 13 ]. For example, a study of Clostridium difficile infection at an urban safety net hospital in Philadelphia, Pennsylvania demonstrated notable differences in risk factors in the hospital’s EHR compared to national surveillance data, suggesting how catchment can influence epidemiologic measures [ 14 ]. Even health-related data captured through health information exchanges may be incomplete [ 15 ].

Several hypothetical study settings can further help the epidemiologist appreciate the relationship between representativeness and validity in EHR research. In the first hypothetical, an EHR-based study is conducted from a single-location federally qualified health center, and in the second hypothetical, an EHR-based study is conducted from a large academic health system. Suppose both studies occur in the same geographic area. It is reasonable to believe the patient populations captured in both EHRs will be quite different and the catchment process could lead to divergent estimates of disease or risk factor prevalence. The large academic health system may be less likely to capture primary care visits, as specialty care may drive the preponderance of patient encounters. However, this is not a bias per se : if the target of inference from these two hypothetical EHR-based studies is the local community, then selection bias becomes a distinct possibility. The epidemiologist must also consider the potential for generalizability and transportability -- two facets of external validity that respectively relate to the extrapolation of study findings to the source population or a different population altogether -- if there are unmeasured effect modifiers, treatment interference, or compound treatments in the community targeted for inference [ 16 ].

There are several approaches for ascertaining representativeness of EHR-based data. Comparing the EHR-derived sample to Census estimates of demography is straightforward but has several important limitations. First, as previously described, the catchment process may be driven by discordant geographical areas, especially for specialty care settings. Second and third, the EHR may have limited or inaccurate information on socioeconomic status, race, and ethnicity that one may wish to compare [ 17 , 18 ], and conversely the Census has limited estimates of health, chiefly disability, fertility, and insurance and payments [ 19 ]. If selection bias is suspected as a result of missing visits in a longitudinal study [ 20 ] or the catchment process in a cross-sectional study [ 21 ], using inverse probability weighting may remediate its influence. Comparing the weighted estimates to the original, non-weighted estimates provides insight into differences in the study participants. In the population health paradigm whereby the EHR is used as a surveillance tool to identify community health disparities [ 13 ], one also needs to be concerned about representativeness. There are emerging approaches for producing such small area community estimates from large observational datasets [ 22 , 23 ]. Conceivably, these approaches may also be useful for identifying issues of representativeness, for example by comparing stratified estimates across sociodemographic or other factors that may relate to catchment. Approaches for issues concerning representativeness specifically as it applies to external validity may be found in these references [ 24 , 25 ].

Challenge #2: Data availability and interpretation

Sub-challenge #2.1: billing versus clinical versus epidemiological needs.

There is an inherent tension in the use of EHR-based data for research purposes: the EHR was never originally designed for research. In the U.S., the Health Information Technology for Economic and Clinical Health Act, which promoted EHRs as a platform for comparative effectiveness research, was an attempt to address this deficiency [ 26 ]. A brief history of the evolution of the modern EHR reveals a technology that was optimized for capturing health details relevant for billing, scheduling, and clinical record keeping [ 27 ]. As such, the availability of data for fundamental markers of upstream health that are important for identifying inequities, such as socioeconomic status, race, ethnicity, and other social determinants of health (SDOH), may be insufficiently captured in the EHR [ 17 , 18 ]. Similarly, behavioral risk factors, such as being a sexual minority person, have historically been insufficiently recorded as discrete variables. It is only recently that such data are beginning to be captured in the EHR [ 28 , 29 ], or techniques such as NLP have made it possible to extract these details when stored in free text notes (described further in “ Unstructured data: clinical notes and reports ” section).

As an example, assessing clinical morbidities in the EHR may be done on the basis of extracting appropriate International Classification of Diseases (ICD) codes, used for billing and reimbursement in the U.S. These codes are known to have low sensitivity despite high specificity for accurate diagnostic status [ 30 , 31 ]. Expressed as predictive values, which depend upon prevalence, presence of a diagnostic code is a likely indicator of a disease state, whereas absence of a diagnostic code is a less reliable indicator of the absence of that morbidity. There may further be variation by clinical domain in that ICD codes may exist but not be used in some specialties [ 32 ], variation by coding vocabulary such as the use of SNOMED for clinical documentation versus ICD for billing necessitating an ontology mapper [ 33 ], and variation by the use of “rule-out” diagnostic codes resulting in false-positive diagnoses [ 34 , 35 , 36 ]. Relatedly is the notion of upcoding, or the billing of tests, procedures, or diagnoses to receive inflated reimbursement, which, although posited to be problematic in EHRs [ 37 ] in at least one study, has not been shown to have occurred [ 38 ]. In the U.S., the billing and reimbursement model, such as fee-for-service versus managed care, may result in varying diagnostic code sensitivities and specificities, especially if upcoding is occurring [ 39 ]. In short, there is potential for misclassification of key health data in the EHR.

Misclassification can potentially be addressed through a validation study (resources permitting) or application of quantitative bias analysis, and there is a rich literature regarding the treatment of misclassified data in statistics and epidemiology. Readers are referred to these texts as a starting point [ 40 , 41 ]. Duda et al. and Shepherd et al. have described an innovative data audit approach applicable to secondary analysis of observational data, such as EHR-derived data, that incorporates the audit error rate directly in the regression analysis to reduce information bias [ 42 , 43 ]. Outside of methodological tricks in the face of imperfect data, researchers must proactively engage with clinical and informatics colleagues to ensure that the right data for the research interests are available and accessible.

Sub-challenge #2.2: Consistency in data and interpretation

For the epidemiologist, abstracting data from the EHR into a research-ready analytic dataset presents a host of complications surrounding data availability, consistency and interpretation. It is easy to conflate the total volume of data in the EHR with data that are usable for research, however expectations should be tempered. Weiskopf et al. have noted such challenges for the researcher: in their study, less than 50% of patient records had “complete” data for research purposes per their four definitions of completeness [ 44 ]. Decisions made about the treatment of incomplete data can induce selection bias or impact precision of estimates (see Challenges #1 , #3 , and #4 ). The COVID-19 pandemic has further demonstrated the challenge of obtaining research data from EHRs across multiple health systems [ 45 ]. On the other hand, EHRs have a key advantage of providing near real-time data as opposed to many epidemiological studies that have a specific endpoint or are retrospective in nature. Such real-time data availability was leveraged during COVID-19 to help healthcare systems manage their pandemic response [ 46 , 47 ]. Logistical and technical issues aside, healthcare and documentation practices are nuanced to their local environments. In fact, researchers have demonstrated how the same research question analyzed in distinct clinical databases can yield different results [ 48 ].

Once the data are obtained, choices regarding operationalization of variables have the potential to induce information bias. Several hypothetical examples can help demonstrate this point. As a first example, differences in laboratory reporting may result in measurement error or misclassification. While the order for a particular laboratory assay is likely consistent within the healthcare system, patients frequently have a choice where to have that order fulfilled. Given the breadth of assays and reporting differences that may differ lab to lab [ 49 ], it is possible that the researcher working with the raw data may not consider all possible permutations. In other words, there may be lack of consistency in the reporting of the assay results. As a second example, raw clinical data requires interpretation to become actionable. A researcher interested in capturing a patient’s Charlson comorbidity index, which is based on 16 potential diagnoses plus the patient’s age [ 50 ], may never find such a variable in the EHR. Rather, this would require operationalization based on the raw data, each of which may be misclassified. Use of such composite measures introduces the notion of “differential item functioning”, whereby a summary indicator of a complexly measured health phenomenon may differ from group to group [ 51 ]. In this case, as opposed to a measurement error bias, this is one of residual confounding in that a key (unmeasured) variable is driving the differences. Remediation of these threats to validity may involve validation studies to determine the accuracy of a particular classifier, sensitivity analysis employing alternative interpretations when the raw data are available, and omitting or imputing biased or latent variables [ 40 , 41 , 52 ]. Importantly, in all cases, the epidemiologists should work with the various health care providers and personnel who have measured and recorded the data present in the EHR, as they likely understand it best.

Furthermore and related to “Billing versus Clinical versus Epidemiological Needs” section, the healthcare system in the U.S. is fragmented with multiple payers, both public and private, potentially exacerbating the data quality issues we describe, especially when linking data across healthcare systems. Single payer systems have enabled large and near-complete population-based studies due to data availability and consistency [ 53 , 54 , 55 ]. Data may also be inconsistent for retrospective longitudinal studies spanning many years if there have been changes to coding standards or practices over time, for example due to the transition from ICD-9 to ICD-10 largely occurring in the mid 2010s or the adoption of the Patient Protection and Affordable Care Act in the U.S. in 2010 with its accompanying changes in billing. Exploratory data analysis may reveal unexpected differences in key variables, by place or time, and recoding, when possible, can enforce consistency.

Sub-challenge #2.3: Unstructured data: clinical notes and reports

There may also be scenarios where structured data fields, while available, are not traditionally or consistently used within a given medical center or by a given provider. For example, reporting of adverse events of medications, disease symptoms, and vaccinations or hospitalizations occurring at different facility/health networks may not always be entered by providers in structured EHR fields. Instead, these types of patient experiences may be more likely to be documented in an unstructured clinical note, report (e.g. pathology or radiology report), or scanned document. Therefore, reliance on structured data to identify and study such issues may result in underestimation and potentially biased results.

Advances in NLP currently allow for information to be extracted from unstructured clinical notes and text fields in a reliable and accurate manner using computational methods. NLP utilizes a range of different statistical, machine learning, and linguistic techniques, and when applied to EHR data, has the potential to facilitate more accurate detection of events not traditionally located or consistently used in structured fields. Various NLP methods can be implemented in medical text analysis, ranging from simplistic and fast term recognition systems to more advanced, commercial NLP systems [ 56 ]. Several studies have successfully utilized text mining to extract information on a variety of health-related issues within clinical notes, such as opioid use [ 57 ], adverse events [ 58 , 59 ], symptoms (e.g., shortness of breath, depression, pain) [ 60 ], and disease phenotype information documented in pathology or radiology reports, including cancer stage, histology, and tumor grade [ 61 ], and lupus nephritis [ 32 ]. It is worth noting that scanned documents involve an additional layer of computation, relying on techniques such as optical character recognition, before NLP can be applied.

Hybrid approaches that combine both narrative and structured data, such as ICD codes, to improve accuracy of detecting phenotypes have also demonstrated high performance. Banerji et al. found that using ICD-9 codes to identify allergic drug reactions in the EHR had a positive predictive value of 46%, while an NLP algorithm in conjunction with ICD-9 codes resulted in a positive predictive value of 86%; negative predictive value also increased in the combined algorithm (76%) compared to ICD-9 codes alone (39%) [ 62 ]. In another example, researchers found that the combination of unstructured clinical notes with structured data for prediction tasks involving in-hospital mortality and 30-day hospital readmission outperformed models using either clinical notes or structured data alone [ 63 ]. As we move forward in analyzing EHR data, it will be important to take advantage of the wealth of information buried in unstructured data to assist in phenotyping patient characteristics and outcomes, capture missing confounders used in multivariate analyses, and develop prediction models.

Challenge #3: Missing measurements

While clinical notes may be useful to recover incomplete information from structured data fields, it may be the case that certain variables are not collected within the EHR at all. As mentioned above, it is important to remember that EHRs were not developed as a research tool (see “ Billing versus clinical versus epidemiological needs ” section), and important variables often used in epidemiologic research may not be typically included in EHRs including socioeconomic status (education, income, occupation) and SDOH [ 17 , 18 ]. Depending upon the interest of the provider or clinical importance placed upon a given variable, this information may be included in clinical notes. While NLP could be used to capture these variables, because they may not be consistently captured, there may be bias in identifying those with a positive mention as a positive case and those with no mention as a negative case. For example, if a given provider inquires about homelessness of a patient based on knowledge of the patient’s situation or other external factors and documents this in the clinical note, we have greater assurance that this is a true positive case. However, lack of mention of homelessness in a clinical note should not be assumed as a true negative case for several reasons: not all providers may feel comfortable asking about and/or documenting homelessness, they may not deem this variable worth noting, or implicit bias among clinicians may affect what is captured. As a result, such cases (i.e. no mention of homelessness) may be incorrectly identified as “not homeless,” leading to selection bias should a researcher form a cohort exclusively of patients who are identified as homeless in the EHR.

Not adjusting for certain measurements missing from EHR data can also lead to biased results if the measurement is an important confounder. Consider the example of distinguishing between prevalent and incident cases of disease when examining associations between disease treatments and patient outcomes [ 64 ]. The first date of an ICD code entered for a given patient may not necessarily be the true date of diagnosis, but rather documentation of an existing diagnosis. This limits the ability to adjust for disease duration, which may be an important confounder in studies comparing various treatments with patient outcomes over time, and may also lead to reverse causality if disease sequalae are assumed to be risk factors.

Methods to supplement EHR data with external data have been used to capture missing information. These methods may include imputation if information (e.g. race, lab values) is collected on a subset of patients within the EHR. It is important to examine whether missingness occurs completely at random or at random (“ignorable”), or not at random (“non-ignorable”), using the data available to determine factors associated with missingness, which will also inform the best imputation strategy to pursue, if any [ 65 , 66 ]. As an example, suppose we are interested in ascertaining a patient's BMI from the EHR. If men were less likely to have BMI measured than women, the probability of missing data (BMI) depends on the observed data (gender) and may therefore be predictable and imputable. On the other hand, suppose underweight individuals were less likely to have BMI measured; the probability of missing data depends on its own value, and as such is non-predictable and may require a validation study to confirm. Alternatively to imputing missing data, surrogate measures may be used, such as inferring area-based SES indicators, including median household income, percent poverty, or area deprivation index, by zip code [ 67 , 68 ]. Lastly, validation studies utilizing external datasets may prove helpful, such as supplementing EHR data with claims data that may be available for a subset of patients (see Challenge #4 ).

As EHRs are increasingly being used for research, there are active pushes to include more structured data fields that are important to population health research, such as SDOH [ 69 ]. Inclusion of such factors are likely to result in improved patient care and outcomes, through increased precision in disease diagnosis, more effective shared decision making, identification of risk factors, and tailoring services to a given population’s needs [ 70 ]. In fact, a recent review found that when individual level SDOH were included in predictive modeling, they overwhelmingly improved performance in medication adherence, risk of hospitalization, 30-day rehospitalizations, suicide attempts, and other healthcare services [ 71 ]. Whether or not these fields will be utilized after their inclusion in the EHR may ultimately depend upon federal and state incentives, as well as support from local stakeholders, and this does not address historic, retrospective analyses of these data.

Challenge #4: Missing visits

Beyond missing variable data that may not be captured during a clinical encounter, either through structured data or clinical notes, there also may be missing information for a patient as a whole. This can occur in a variety of ways; for example, a patient may have one or two documented visits in the EHR and then is never seen again (i.e. right censoring due to lost to follow-up), or a patient is referred from elsewhere to seek specialty care, with no information captured regarding other external issues (i.e. left censoring). This may be especially common in circumstances where a given EHR is more likely to capture specialty clinics versus primary care (see Challenge #1 ). A third scenario may include patients who appear, then are not observed for a long period of time, and then reappear: this case is particularly problematic as it may appear the patient was never lost to follow up but simply had fewer visits. In any of these scenarios, a researcher will lack a holistic view of the patient’s experiences, diagnoses, results, and more. As discussed above, assuming absence of a diagnostic code as absence of disease may lead to information and/or selection bias. Further, it has been demonstrated that one key source of bias in EHRs is “informed presence” bias, where those with more medical encounters are more likely to be diagnosed with various conditions (similar to Berkson’s bias) [ 72 ].

Several solutions to these issues have been proposed. For example, it is common for EHR studies to condition on observation time (i.e. ≥n visits required to be eligible into cohort); however, this may exclude a substantial amount of patients with certain characteristics, incurring a selection bias or limiting the generalizability of study findings (see Challenge #1 ). Other strategies attempt to account for missing visit biases through longitudinal imputation approaches; for example, if a patient missed a visit, a disease activity score can be imputed for that point in time, given other data points [ 73 , 74 ]. Surrogate measures may also be used to infer patient outcomes, such as controlling for “informative” missingness as an indicator variable or using actual number of missed visits that were scheduled as a proxy for external circumstances influencing care [ 20 ]. To address “informed presence” bias described above, conditioning on the number of health-care encounters may be appropriate [ 72 ]. Understanding the reason for the missing visit may help identify the best course of action and before imputing, one should be able to identify the type of missingness, whether “informative” or not [ 65 , 66 ]. For example, if distance to a healthcare location is related to appointment attendance, being able to account for this in analysis would be important: researchers have shown how the catchment of a healthcare facility can induce selection bias [ 21 ]. Relatedly, as telehealth becomes more common fueled by the COVID-19 pandemic [ 75 , 76 ], virtual visits may generate missingness of data recorded in the presence of a provider (e.g., blood pressure if the patient does not have access to a sphygmomanometer; see Challenge #3 ), or necessitate a stratified analysis by visit type to assess for effect modification.

Another common approach is to supplement EHR information with external data sources, such as insurance claims data, when available. Unlike a given EHR, claims data are able to capture a patient’s interaction with the health care system across organizations, and additionally includes pharmacy data such as if a prescription was filled or refilled. Often researchers examine a subset of patients eligible for Medicaid/Medicare and compare what is documented in claims with information available in the EHR [ 77 ]. That is, are there additional medications, diagnoses, hospitalizations found in the claims dataset that were not present in the EHR. In a study by Franklin et al., researchers utilized a linked database of Medicare Advantage claims and comprehensive EHR data from a multi-specialty outpatient practice to determine which dataset would be more accurate in predicting medication adherence [ 77 ]. They found that both datasets were comparable in identifying those with poor adherence, though each dataset incorporated different variables.

While validation studies such as those using claims data allow researchers to gain an understanding as to how accurate and complete a given EHR is, this may only be limited to the specific subpopulation examined (i.e. those eligible for Medicaid, or those over 65 years for Medicare). One study examined congruence between EHR of a community health center and Medicaid claims with respect to diabetes [ 78 ]. They found that patients who were older, male, Spanish-speaking, above the federal poverty level, or who had discontinuous insurance were more likely to have services documented in the EHR as compared to Medicaid claims data. Therefore, while claims data may help supplement and validate information in the EHR, on their own they may underestimate care in certain populations.

Research utilizing EHR data has undoubtedly positively impacted the field of public health through its ability to provide large-scale, longitudinal data on a diverse set of patients, and will continue to do so in the future as more epidemiologists take advantage of this data source. EHR data’s ability to capture individuals that traditionally aren’t included in clinical trials, cohort studies, and even claims datasets allows researchers to measure longitudinal outcomes in patients and perhaps change the understanding of potential risk factors.

However, as outlined in this review, there are important caveats to EHR analysis that need to be taken into account; failure to do so may threaten study validity. The representativeness of EHR data depends on the catchment area of the center and corresponding target population. Tools are available to evaluate and remedy these issues, which are critical to study validity as well as extrapolation of study findings. Data availability and interpretation, missing measurements, and missing visits are also key challenges, as EHRs were not specifically developed for research purposes, despite their common use for such. Taking advantage of all available EHR data, whether it be structured or unstructured fields through NLP, will be important in understanding the patient experience and identifying key phenotypes. Beyond methods to address these concerns, it will remain crucial for epidemiologists and data analysts to engage with clinicians and informaticians at their institutions to ensure data quality and accessibility by forming multidisciplinary teams around specific research projects. Lastly, integration across multiple EHRs, or datasets that encompass multi-institutional EHR records, add an additional layer of data quality and validity issues, with the potential to exacerbate the above-stated challenges found within a single EHR. At minimum, such studies should account for correlated errors [ 79 , 80 ], and investigate whether modularization, or submechanisms that determine whether data are observed or missing in each EHR, exist [ 65 ].

The identified challenges may also apply to secondary analysis of other large healthcare databases, such as claims data, although it is important not to conflate the two types of data. EHR data are driven by clinical care and claims data are driven by the reimbursement process where there is a financial incentive to capture diagnoses, procedures, and medications [ 48 ]. The source of data likely influences the availability, accuracy, and completeness of data. The fundamental representation of data may also differ as a record in a claims database corresponds to a “claim” as opposed to an “encounter” in the EHR. As such, the representativeness of the database populations, the sensitivity and specificity of variables, as well as the mechanisms of missingness in claims data may differ from EHR data. One study that evaluated pediatric quality care measures, such as BMI, noted inferior sensitivity based on claims data alone [ 81 ]. Linking claims data to EHR data has been proposed to enhance study validity, but many of the caveats raised in herein still apply [ 82 ].

Although we focused on epidemiological challenges related to study validity, there are other important considerations for researchers working with EHR data. Privacy and security of data as well as institutional review board (IRB) or ethics board oversight of EHR-based studies should not be taken for granted. For researchers in the U.S., Goldstein and Sarwate described Health Insurance Portability and Accountability Act (HIPAA)-compliant approaches to ensure the privacy and security of EHR data used in epidemiological research, and presented emerging approaches to analyses that separate the data from analysis [ 83 ]. The IRB oversees the data collection process for EHR-based research and through the HIPAA Privacy Rule these data typically do not require informed consent provided they are retrospective and reside at the EHR’s institution [ 84 ]. Such research will also likely receive an exempt IRB review provided subjects are non-identifiable.

Conclusions

As EHRs are increasingly being used for research, epidemiologists can take advantage of the many tools and methods that already exist and apply them to the key challenges described above. By being aware of the limitations that the data present and proactively addressing them, EHR studies will be more robust, informative, and important to the understanding of health and disease in the population.

Availability of data and materials

All data and materials used in this review are described herein.

Abbreviations

Body Mass Index

Electronic Health Record

International Classification of Diseases

Institutional review board/ethics board

Health Insurance Portability and Accountability Act

Natural Language Processing

Social Determinants of Health

Socioeconomic Status

Adler-Milstein J, Holmgren AJ, Kralovec P, et al. Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divide. J Am Med Inform Assoc. 2017;24(6):1142–8.

Article   PubMed   PubMed Central   Google Scholar  

Office of the National Coordinator for Health Information Technology. ‘Office-based physician electronic health record adoption’, Health IT quick-stat #50. dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php . Accessed 15 Jan 2019.

Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017;106(1):1–9.

Article   PubMed   Google Scholar  

Casey JA, Schwartz BS, Stewart WF, et al. Using electronic health records for population health research: a review of methods and applications. Annu Rev Public Health. 2016;37:61–81.

Verheij RA, Curcin V, Delaney BC, et al. Possible sources of bias in primary care electronic health record data use and reuse. J Med Internet Res. 2018;20(5):e185.

Ni K, Chu H, Zeng L, et al. Barriers and facilitators to data quality of electronic health records used for clinical research in China: a qualitative study. BMJ Open. 2019;9(7):e029314.

Coleman N, Halas G, Peeler W, et al. From patient care to research: a validation study examining the factors contributing to data quality in a primary care electronic medical record database. BMC Fam Pract. 2015;16:11.

Kruse CS, Stein A, Thomas H, et al. The use of electronic health records to support population health: a systematic review of the literature. J Med Syst. 2018;42(11):214.

Shortreed SM, Cook AJ, Coley RY, et al. Challenges and opportunities for using big health care data to advance medical science and public health. Am J Epidemiol. 2019;188(5):851–61.

In: Smedley BD, Stith AY, Nelson AR, editors. Unequal treatment: confronting racial and ethnic disparities in health care. Washington (DC) 2003.

Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–52.

Cutler DM, Scott Morton F. Hospitals, market share, and consolidation. JAMA. 2013;310(18):1964–70.

Article   CAS   PubMed   Google Scholar  

Cocoros NM, Kirby C, Zambarano B, et al. RiskScape: a data visualization and aggregation platform for public health surveillance using routine electronic health record data. Am J Public Health. 2021;111(2):269–76.

Vader DT, Weldie C, Welles SL, et al. Hospital-acquired Clostridioides difficile infection among patients at an urban safety-net hospital in Philadelphia: demographics, neighborhood deprivation, and the transferability of national statistics. Infect Control Hosp Epidemiol. 2020;42:1–7.

Google Scholar  

Dixon BE, Gibson PJ, Frederickson Comer K, et al. Measuring population health using electronic health records: exploring biases and representativeness in a community health information exchange. Stud Health Technol Inform. 2015;216:1009.

PubMed   Google Scholar  

Hernán MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. 2011;22(3):368–77.

Casey JA, Pollak J, Glymour MM, et al. Measures of SES for electronic health record-based research. Am J Prev Med. 2018;54(3):430–9.

Polubriaginof FCG, Ryan P, Salmasian H, et al. Challenges with quality of race and ethnicity data in observational databases. J Am Med Inform Assoc. 2019;26(8-9):730–6.

U.S. Census Bureau. Health. Available at: https://www.census.gov/topics/health.html . Accessed 19 Jan 2021.

Gianfrancesco MA, McCulloch CE, Trupin L, et al. Reweighting to address nonparticipation and missing data bias in a longitudinal electronic health record study. Ann Epidemiol. 2020;50:48–51 e2.

Goldstein ND, Kahal D, Testa K, Burstyn I. Inverse probability weighting for selection bias in a Delaware community health center electronic medical record study of community deprivation and hepatitis C prevalence. Ann Epidemiol. 2021;60:1–7.

Gelman A, Lax J, Phillips J, et al. Using multilevel regression and poststratification to estimate dynamic public opinion. Unpublished manuscript, Columbia University. 2016 Sep 11. Available at: http://www.stat.columbia.edu/~gelman/research/unpublished/MRT(1).pdf . Accessed 22 Jan 2021.

Quick H, Terloyeva D, Wu Y, et al. Trends in tract-level prevalence of obesity in philadelphia by race-ethnicity, space, and time. Epidemiology. 2020;31(1):15–21.

Lesko CR, Buchanan AL, Westreich D, Edwards JK, Hudgens MG, Cole SR. Generalizing study results: a potential outcomes perspective. Epidemiology. 2017;28(4):553–61.

Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am J Epidemiol. 2017;186(8):1010–4.

Congressional Research Services (CRS). The Health Information Technology for Economic and Clinical Health (HITECH) Act. 2009. Available at: https://crsreports.congress.gov/product/pdf/R/R40161/9 . Accessed Jan 22 2021.

Hersh WR. The electronic medical record: Promises and problems. Journal of the American Society for Information Science. 1995;46(10):772–6.

Article   Google Scholar  

Collecting sexual orientation and gender identity data in electronic health records: workshop summary. Washington (DC) 2013.

Committee on the Recommended Social and Behavioral Domains and Measures for Electronic Health Records; Board on Population Health and Public Health Practice; Institute of Medicine. Capturing social and behavioral domains and measures in electronic health records: phase 2. Washington (DC): National Academies Press (US); 2015.

Goff SL, Pekow PS, Markenson G, et al. Validity of using ICD-9-CM codes to identify selected categories of obstetric complications, procedures and co-morbidities. Paediatr Perinat Epidemiol. 2012;26(5):421–9.

Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 2005;58(4):323–37.

Gianfrancesco MA. Application of text mining methods to identify lupus nephritis from electronic health records. Lupus Science & Medicine. 2019;6:A142.

National Library of Medicine. SNOMED CT to ICD-10-CM Map. Available at: https://www.nlm.nih.gov/research/umls/mapping_projects/snomedct_to_icd10cm.html . Accessed 2 Jul 2021.

Klabunde CN, Harlan LC, Warren JL. Data sources for measuring comorbidity: a comparison of hospital records and medicare claims for cancer patients. Med Care. 2006;44(10):921–8.

Burles K, Innes G, Senior K, Lang E, McRae A. Limitations of pulmonary embolism ICD-10 codes in emergency department administrative data: let the buyer beware. BMC Med Res Methodol. 2017;17(1):89.

Asgari MM, Wu JJ, Gelfand JM, Salman C, Curtis JR, Harrold LR, et al. Validity of diagnostic codes and prevalence of psoriasis and psoriatic arthritis in a managed care population, 1996-2009. Pharmacoepidemiol Drug Saf. 2013;22(8):842–9.

Hoffman S, Podgurski A. Big bad data: law, public health, and biomedical databases. J Law Med Ethics. 2013;41(Suppl 1):56–60.

Adler-Milstein J, Jha AK. Electronic health records: the authors reply. Health Aff. 2014;33(10):1877.

Geruso M, Layton T. Upcoding: evidence from medicare on squishy risk adjustment. J Polit Econ. 2020;12(3):984–1026.

Lash TL, Fox MP, Fink AK. Applying quantitative bias analysis to epidemiologic data. New York: Springer-Verlag New York; 2009.

Book   Google Scholar  

Gustafson P. Measurement error and misclassification in statistics and epidemiology: impacts and Bayesian adjustments. Boca Raton: Chapman and Hall/CRC; 2004.

Duda SN, Shepherd BE, Gadd CS, et al. Measuring the quality of observational study data in an international HIV research network. PLoS One. 2012;7(4):e33908.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Shepherd BE, Yu C. Accounting for data errors discovered from an audit in multiple linear regression. Biometrics. 2011;67(3):1083–91.

Weiskopf NG, Hripcsak G, Swaminathan S, et al. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform. 2013;46(5):830–6.

Kaiser Health News. As coronavirus strikes, crucial data in electronic health records hard to harvest. Available at: https://khn.org/news/as-coronavirus-strikes-crucial-data-in-electronic-health-records-hard-to-harvest/ . Accessed 15 Jan 2021.

Reeves JJ, Hollandsworth HM, Torriani FJ, Taplitz R, Abeles S, Tai-Seale M, et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc. 2020;27(6):853–9.

Grange ES, Neil EJ, Stoffel M, Singh AP, Tseng E, Resco-Summers K, et al. Responding to COVID-19: The UW medicine information technology services experience. Appl Clin Inform. 2020;11(2):265–75.

Madigan D, Ryan PB, Schuemie M, et al. Evaluating the impact of database heterogeneity on observational study results. Am J Epidemiol. 2013;178(4):645–51.

Lippi G, Mattiuzzi C. Critical laboratory values communication: summary recommendations from available guidelines. Ann Transl Med. 2016;4(20):400.

Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.

Jones RN. Differential item functioning and its relevance to epidemiology. Curr Epidemiol Rep. 2019;6:174–83.

Edwards JK, Cole SR, Troester MA, Richardson DB. Accounting for misclassified outcomes in binary regression models using multiple imputation with internal validation data. Am J Epidemiol. 2013;177(9):904–12.

Satkunasivam R, Klaassen Z, Ravi B, Fok KH, Menser T, Kash B, et al. Relation between surgeon age and postoperative outcomes: a population-based cohort study. CMAJ. 2020;192(15):E385–92.

Melamed N, Asztalos E, Murphy K, Zaltz A, Redelmeier D, Shah BR, et al. Neurodevelopmental disorders among term infants exposed to antenatal corticosteroids during pregnancy: a population-based study. BMJ Open. 2019;9(9):e031197.

Kao LT, Lee HC, Lin HC, Tsai MC, Chung SD. Healthcare service utilization by patients with obstructive sleep apnea: a population-based study. PLoS One. 2015;10(9):e0137459.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Jung K, LePendu P, Iyer S, Bauer-Mehren A, Percha B, Shah NH. Functional evaluation of out-of-the-box text-mining tools for data-mining tasks. J Am Med Inform Assoc. 2015;22(1):121–31.

Canan C, Polinski JM, Alexander GC, et al. Automatable algorithms to identify nonmedical opioid use using electronic data: a systematic review. J Am Med Inform Assoc. 2017;24(6):1204–10.

Iqbal E, Mallah R, Jackson RG, et al. Identification of adverse drug events from free text electronic patient records and information in a large mental health case register. PLoS One. 2015;10(8):e0134208.

Rochefort CM, Verma AD, Eguale T, et al. A novel method of adverse event detection can accurately identify venous thromboembolisms (VTEs) from narrative electronic health record data. J Am Med Inform Assoc. 2015;22(1):155–65.

Koleck TA, Dreisbach C, Bourne PE, et al. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. J Am Med Inform Assoc. 2019;26(4):364–79.

Wang L, Luo L, Wang Y, et al. Natural language processing for populating lung cancer clinical research data. BMC Med Inform Decis Mak. 2019;19(Suppl 5):239.

Banerji A, Lai KH, Li Y, et al. Natural language processing combined with ICD-9-CM codes as a novel method to study the epidemiology of allergic drug reactions. J Allergy Clin Immunol Pract. 2020;8(3):1032–1038.e1.

Zhang D, Yin C, Zeng J, et al. Combining structured and unstructured data for predictive models: a deep learning approach. BMC Med Inform Decis Mak. 2020;20(1):280.

Farmer R, Mathur R, Bhaskaran K, Eastwood SV, Chaturvedi N, Smeeth L. Promises and pitfalls of electronic health record analysis. Diabetologia. 2018;61:1241–8.

Haneuse S, Arterburn D, Daniels MJ. Assessing missing data assumptions in EHR-based studies: a complex and underappreciated task. JAMA Netw Open. 2021;4(2):e210184.

Groenwold RHH. Informative missingness in electronic health record systems: the curse of knowing. Diagn Progn Res. 2020;4:8.

Berkowitz SA, Traore CY, Singer DE, et al. Evaluating area-based socioeconomic status indicators for monitoring disparities within health care systems: results from a primary care network. Health Serv Res. 2015;50(2):398–417.

Kind AJH, Buckingham WR. Making neighborhood-disadvantage metrics accessible - the neighborhood atlas. N Engl J Med. 2018;378(26):2456–8.

Cantor MN, Thorpe L. Integrating data on social determinants of health into electronic health records. Health Aff. 2018;37(4):585–90.

Adler NE, Stead WW. Patients in context--EHR capture of social and behavioral determinants of health. N Engl J Med. 2015;372(8):698–701.

Chen M, Tan X, Padman R. Social determinants of health in electronic health records and their impact on analysis and risk prediction: a systematic review. J Am Med Inform Assoc. 2020;27(11):1764–73.

Goldstein BA, Bhavsar NA, Phelan M, et al. Controlling for informed presence bias due to the number of health encounters in an electronic health record. Am J Epidemiol. 2016;184(11):847–55.

Petersen I, Welch CA, Nazareth I, et al. Health indicator recording in UK primary care electronic health records: key implications for handling missing data. Clin Epidemiol. 2019;11:157–67.

Li R, Chen Y, Moore JH. Integration of genetic and clinical information to improve imputation of data missing from electronic health records. J Am Med Inform Assoc. 2019;26(10):1056–63.

Koonin LM, Hoots B, Tsang CA, Leroy Z, Farris K, Jolly T, et al. Trends in the use of telehealth during the emergence of the COVID-19 pandemic - United States, January-March 2020. MMWR Morb Mortal Wkly Rep. 2020;69(43):1595–9.

Barnett ML, Ray KN, Souza J, Mehrotra A. Trends in telemedicine use in a large commercially insured population, 2005-2017. JAMA. 2018;320(20):2147–9.

Franklin JM, Gopalakrishnan C, Krumme AA, et al. The relative benefits of claims and electronic health record data for predicting medication adherence trajectory. Am Heart J. 2018;197:153–62.

Devoe JE, Gold R, McIntire P, et al. Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers. Ann Fam Med. 2011;9(4):351–8.

Schmajuk G, Li J, Evans M, Anastasiou C, Izadi Z, Kay JL, et al. RISE registry reveals potential gaps in medication safety for new users of biologics and targeted synthetic DMARDs. Semin Arthritis Rheum. 2020 Dec;50(6):1542–8.

Izadi Z, Schmajuk G, Gianfrancesco M, Subash M, Evans M, Trupin L, et al. Rheumatology Informatics System for Effectiveness (RISE) practices see significant gains in rheumatoid arthritis quality measures. Arthritis Care Res. 2020. https://doi.org/10.1002/acr.24444 .

Angier H, Gold R, Gallia C, Casciato A, Tillotson CJ, Marino M, et al. Variation in outcomes of quality measurement by data source. Pediatrics. 2014;133(6):e1676–82.

Lin KJ, Schneeweiss S. Considerations for the analysis of longitudinal electronic health records linked to claims data to study the effectiveness and safety of drugs. Clin Pharmacol Ther. 2016;100(2):147–59.

Goldstein ND, Sarwate AD. Privacy, security, and the public health researcher in the era of electronic health record research. Online J Public Health Inform. 2016;8(3):e207.

U.S. Department of Health and Human Services (HHS). 45 CFR 46. http://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html .

Download references

Acknowledgements

The authors thank Dr. Annemarie Hirsch, Department of Population Health Sciences, Geisinger, for assistance in conceptualizing an earlier version of this work.

Research reported in this publication was supported in part by the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Award Number K01AR075085 (to MAG) and the National Institute Of Allergy And Infectious Diseases of the National Institutes of Health under Award Number K01AI143356 (to NDG). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author information

Authors and affiliations.

Division of Rheumatology, University of California School of Medicine, San Francisco, CA, USA

Milena A. Gianfrancesco

Department of Epidemiology and Biostatistics, Drexel University Dornsife School of Public Health, 3215 Market St., Philadelphia, PA, 19104, USA

Neal D. Goldstein

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors conceptualized, wrote, and approved the final submitted version.

Corresponding author

Correspondence to Neal D. Goldstein .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors have no competing interests to declare

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gianfrancesco, M.A., Goldstein, N.D. A narrative review on the validity of electronic health record-based research in epidemiology. BMC Med Res Methodol 21 , 234 (2021). https://doi.org/10.1186/s12874-021-01416-5

Download citation

Received : 02 July 2021

Accepted : 28 September 2021

Published : 27 October 2021

DOI : https://doi.org/10.1186/s12874-021-01416-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Electronic health records
• Data quality
• Secondary analysis

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Electronic Health Record

• Reference work entry
• First Online: 01 January 2022
• pp 1609–1614
• Cite this reference work entry

• Ping Yu 3 , 4 , 5  

60 Accesses

Electronic medical record

According to International Organization for Standardization (ISO) (ISO/TR20514 2005 ), electronic health record (EHR) is repository of information regarding the health status of a subject of care, in computer-processable form. The information can be retrospective, concurrent, and prospective. It can be transmitted securely and accessible by multiple authorized users. The primary purpose of EHR is to support continuing, efficient, and quality integrated healthcare.

Healthcare Information and Management Systems Society (HIMSS), the largest international health information management association, defines EHR as a longitudinal electronic record of patient health information ( https://www.himss.org/library/ehr ). The information is generated by one or more encounters in any care delivery setting. The content of EHR includes patient demographics, progress notes, problems, medications, vital signs, past medical history, immunizations, laboratory...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Abramson EL, Patel V, Malhotra S et al (2012) Physician experiences transitioning between an older versus newer electronic health record for electronic prescribing. Int J Med Inform 81(8):539–548. https://doi.org/10.1016/j.ijmedinf.2012.02.010

Article   Google Scholar  

Assis-Hassid S, Grosz BJ, Zimlichman E et al (2019) Assessing EHR use during hospital morning rounds: a multi-faceted study. PLoS One 14(2):e0212816. https://doi.org/10.1371/journal.pone.0212816

Bercovitz A, Sengupta M, Jamison P (2010) Electronic medical record adoption and use in home health and hospice. NCHS Data Brief 45(45):1–8

Google Scholar  

Blijleven V, Koelemeijer K, Wetzels M et al (2017) Workarounds emerging from electronic health record system usage: consequences for patient safety, effectiveness of care, and efficiency of care. JMIR Hum Factors 4(4):e27. https://doi.org/10.2196/humanfactors.7978

Bowman S (2013) Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 10:1c

Braunstein ML (2019) Health care in the age of interoperability part 5: the personal health record. IEEE Pulse 10(3):19–23. https://doi.org/10.1109/mpuls.2019.2911804

Byrne CM, Mercincavage LM, Bouhaddou O et al (2014) The department of veterans affairs’ (VA) implementation of the virtual lifetime electronic record (VLER): findings and lessons learned from health information exchange at 12 sites. Int J Med Inform 83(8):537–547. https://doi.org/10.1016/j.ijmedinf.2014.04.005

Clemens NA (2012) Privacy, consent, and the electronic mental health record: the person vs. the system. J Psychiatr Pract 18(1):46–50. https://doi.org/10.1097/01.pra.0000410987.38723.47

Colicchio TK, Borbolla D, Colicchio VD et al (2019) Looking behind the curtain: identifying factors contributing to changes on care outcomes during a large commercial EHR implementation. EGEMS (Wash DC) 7(1):21. https://doi.org/10.5334/egems.269

Detmer DE (2000) Information technology for quality health care: a summary of United Kingdom and United States experiences. Qual Health Care 9(3):181–189. https://doi.org/10.1136/qhc.9.3.181

Druss BG, Ji X, Glick G et al (2014) Randomized trial of an electronic personal health record for patients with serious mental illnesses. Am J Psychiatry 171(3):360–368. https://doi.org/10.1176/appi.ajp.2013.13070913

Entzeridou E, Markopoulou E, Mollaki V (2018) Public and physician’s expectations and ethical concerns about electronic health record: benefits outweigh risks except for information security. Int J Med Inform 110:98–107. https://doi.org/10.1016/j.ijmedinf.2017.12.004

Evans RS (2016) Electronic health records: then, now, and in the future. Yearb Med Inform (Suppl 1):S48–S61. https://doi.org/10.15265/IYS-2016-s006

Fragidis LL, Chatzoglou PD (2018) Implementation of a nationwide electronic health record (EHR). Int J Health Care Qual Assur 31(2):116–130. https://doi.org/10.1108/ijhcqa-09-2016-0136

Gardner RM (2016) Clinical information systems – from yesterday to tomorrow. Yearb Med Inform 25(Suppl 1):S62–S75. https://doi.org/10.15265/IYS-2016-s010

Gundlapalli AV, Olson J, Smith SP et al (2007) Hospital electronic medical record-based public health surveillance system deployed during the 2002 Winter Olympic Games. Am J Infect Control 35(3):163–171. https://doi.org/10.1016/j.ajic.2006.08.003

Hammond WE (1991) Health level 7: an application standard for electronic medical data exchange. Top Health Rec Manage 11(4):59–66

Häyrinen K, Saranto K, Nykänen P (2008) Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 77(5):291–304. https://doi.org/10.1016/j.ijmedinf.2007.09.001

ISO/TR20514 (2005) Electronic health record. https://www.iso.org/obp/ui/#iso:std:iso:tr:20514:en . Accessed 9 June 2019

Jamoom EW, Heisey-Grove D, Yang N et al (2016) Physician opinions about EHR use by EHR experience and by whether the practice had optimized its ehr use. J Health Med Inform 7(4):1000240. https://doi.org/10.4172/2157-7420.1000240

Kainz C, Lassmann R, Schaffer H et al (1992) Survey of computerized obstetric information systems in Austria. Arch Gynecol Obstet 252(2):87–91

Meeks DW, Smith MW, Taylor L et al (2014) An analysis of electronic health record-related patient safety concerns. J Am Med Inform Assoc 21(6):1053–1059. https://doi.org/10.1136/amiajnl-2013-002578

Munyisia E, Yu P, Hailey D (2014) The effect of an electronic health record system on nursing staff time in a nursing home: a longitudinal cohort study. Australas Med J 7(7):285–293. https://doi.org/10.4066/AMJ.2014.2072

Ornstein SM, Garr DR, Jenkins RG (1993) A comprehensive microcomputer-based medical records system with sophisticated preventive services features for the family physician. J Am Board Fam Pract 6(1):55–60

Pathak J, Kiefer RC, Bielinski SJ et al (2012) Applying semantic web technologies for phenome-wide scan using an electronic health record linked biobank. J Biomed Semantics 3(1):10. https://doi.org/10.1186/2041-1480-3-10

Stead WW, Hammond WE, Straube MJ (1983) A chartless record – is it adequate? J Med Syst 7(2):103–109

Technology TOotNCfHI (2019) What are the advantages of electronic health records? https://www.healthit.gov/faq/what-are-advantages-electronic-health-records . Accessed 9 June 2019

Virginio LA Jr, Ricarte IL (2015) Identification of patient safety risks associated with electronic health records: a software quality perspective. Stud Health Technol Inform 216:55–59. https://doi.org/10.3233/978-1-61499-564-7-55

Walsh KE, Marsolo KA, Davis C et al (2018) Accuracy of the medication list in the electronic health record-implications for care, research, and improvement. J Am Med Inform Assoc 25(7):909–912. https://doi.org/10.1093/jamia/ocy027

Weed LL (1971) The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 3(3):131–134

Zhang Y, Yu P, Shen J (2011) The benefits of introducing electronic health records in residential aged care facilities: a multiple case study. Int J Med Inform 81(10):690–704. https://doi.org/10.1016/j.ijmedinf.2012.05.013

Download references

Author information

Authors and affiliations.

Faculty of Engineering and Information Sciences, School of Computing and Information Technology, Centre for Digital Transformation, University of Wollongong, Wollongong, NSW, Australia

Illawarra Health and Medical Research Institute, University of Wollongong, Wollongong, NSW, Australia

Smart Infrastructure Facility, Faculty of Engineering and Information Sciences, University of Wollongong, Wollongong, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ping Yu .

Editor information

Editors and affiliations.

Population Division, Department of Economics and Social Affairs, United Nations, New York, NY, USA

Department of Population Health Sciences, Department of Sociology, Duke University, Durham, NC, USA

Matthew E. Dupre

Section Editor information

School of Computing and Information Technology, Faculty of Engineering and Information Sciences, University of Wollongong, Wollongong, NSW, Australia

Illawarra Health and Medical Research Institute, Wollongong, NSW, Australia

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this entry

Cite this entry.

Yu, P. (2021). Electronic Health Record. In: Gu, D., Dupre, M.E. (eds) Encyclopedia of Gerontology and Population Aging. Springer, Cham. https://doi.org/10.1007/978-3-030-22009-9_442

Download citation

DOI : https://doi.org/10.1007/978-3-030-22009-9_442

Published : 24 May 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-22008-2

Online ISBN : 978-3-030-22009-9

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Electronic Medical Records implementation in hospital: An empirical investigation of individual and organizational determinants

Contributed equally to this work with: Anna De Benedictis, Emanuele Lettieri

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Current address: Department of Healthcare Professions, University Hospital Campus Bio-Medico, Rome, Italy

Affiliations Department of Healthcare Professions, University Hospital Campus Bio-Medico, Rome, Italy, Faculty of Medicine & Surgery, University Campus Bio-Medico, Rome, Italy

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

Current address: Department of Economics, Management and Industrial Engineering, Politecnico of Milan, Milan, Italy

Affiliation Department of Economics, Management and Industrial Engineering, Politecnico of Milan, Milan, Italy

Roles Conceptualization, Data curation, Methodology, Writing – review & editing

Roles Conceptualization, Methodology, Writing – review & editing

Roles Formal analysis, Investigation, Project administration

Affiliation Department of Healthcare Professions, University Hospital Campus Bio-Medico, Rome, Italy

Roles Conceptualization, Writing – review & editing

• Anna De Benedictis, 
• Emanuele Lettieri, 
• Luca Gastaldi, 
• Cristina Masella, 
• Alessia Urgu, 
• Daniela Tartaglini

• Published: June 4, 2020
• https://doi.org/10.1371/journal.pone.0234108
• Reader Comments

The implementation of hospital-wide Electronic Medical Records (EMRs) is still an unsolved quest for many hospital managers. EMRs have long been considered a key factor for improving healthcare quality and safety, reducing adverse events for patients, decreasing costs, optimizing processes, improving clinical research and obtaining best clinical performances. However, hospitals continue to experience resistance from professionals to accepting EMRs. This study combines institutional and individual factors to explain which determinants can trigger or inhibit the EMRs implementation in hospitals, and which variables managers can exploit to guide professionals’ behaviours. Data have been collected through a survey administered to physicians and nurses in an Italian University Hospital in Rome. A total of 114 high-quality responses had been received. Results show that both, physicians and nurses, expect many benefits from the use of EMRs. In particular, it is believed that the EMRs will have a positive impact on quality, efficiency and effectiveness of care; handover communication between healthcare workers; teaching, tutoring and research activities; greater control of your own business. Moreover, data show an interplay between individual and institutional determinants: normative factors directly affect perceived usefulness (C = 0.30 **), perceived ease of use (C = 0.26 **) and intention to use EMRs (C = 0.33 **), regulative factors affect the intention to use EMRs (C = -0.21 **), and perceived usefulness directly affect the intention to use EMRs (C = 0.33 **). The analysis carried out shows that the key determinants of the intention to use EMRs are the normative ones (peer influence) and the individual ones (perceived usefulness), and that perceived usefulness works also as a mediator between normative factors and intention to use EMRs. Therefore, Management can leverage on power users to motivate, generate and manage change.

Citation: De Benedictis A, Lettieri E, Gastaldi L, Masella C, Urgu A, Tartaglini D (2020) Electronic Medical Records implementation in hospital: An empirical investigation of individual and organizational determinants. PLoS ONE 15(6): e0234108. https://doi.org/10.1371/journal.pone.0234108

Editor: Stefano Triberti, University of Milan, ITALY

Received: September 16, 2019; Accepted: May 19, 2020; Published: June 4, 2020

Copyright: © 2020 De Benedictis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Healthcare is the most complex and fast-moving industry that exists. New digital technologies are constantly being developed, all with the potential to support the clinical practice by bringing many advantages into the healthcare sector [ 1 ]. Nevertheless, the healthcare industry has lagged behind other sectors in the adoption of Information Technology (IT) in the workplace [ 2 ]. Electronic Medical Records (EMRs) have long been considered a key factor for improving healthcare quality and safety, reducing adverse events for patients, decreasing costs, optimizing processes, improving clinical research and obtaining best clinical performance [e.g., 3 – 5 ]. However, the pace of adoption of EMRs–as other digital technologies–in healthcare continues to lag [ 2 , 6 ], and hospitals continue to experience resistance from professionals to accepting digital technology [ 7 ]. Though many research and development programs exist and venture capital investment has been growing, successful IT projects in healthcare continue to be rare, and a plan to accelerate innovation is needed beginning with a diagnosis of the problem [ 2 ]. Some studies analyzed both individual and organizational factors that affect the acceptance and implementation of technology [ 8 ], but they have generated mixed results [ 9 ]. Indeed, mechanisms that drive the adoption and implementation of IT in hospitals remain unclear. Organizational studies conceive organizations as strongly institutionalized settings in which individual behaviours are influenced by regulations, social norms and cultural systems [ 10 , 11 ]. In contrast, Information Science has mostly adopted user acceptance models, which emphasise individuals’ rational and volitional assessment of the costs and benefits they would attain from the new digital technology [ 11 ].

Hospitals are highly institutionalized and regulated contexts, in terms of regulatory oversight and professional roles, and are operationally and technically complex [ 12 ]. Physicians and nurses have a high level of professionalism and they often affiliate within their specialities via professional training and participation in speciality-focused organizations [ 13 ]. Successful adoption or perceived usefulness of EMRs by others within their specialities may influence hospital professionals’ decisions, particularly if they are uncertain about individual benefits. Nevertheless, the majority of academic research in IT adoption in healthcare focused on the individual level [ 14 ]. The most widely used model to explore issues related to the acceptance of technology is the Technology Acceptance Model (TAM) [ 15 ], which identifies two main antecedents the perceived usefulness and the perceived ease of use of technology. The TAM has been validated in multiple settings [e.g. 16 – 18 ]. In its basic framework, the end user’s attitudes and perceptions regarding the use of new technology determine the user’s behavioural intention to use it. Institutional theory, instead, is based on the assumption that individual behaviours are modelled by regulations, social norms and meaning systems and that institutions embodied in routines rely on automatic cognition and uncritical processing of existing schemata and privilege consistency with stereotypes and speed over accuracy [ 19 ]. Thus, in this theory, normative and cultural conditions are co-determinants of the adoption of new technologies [ 20 ]. The use of institutional theory in Information Science is rare compared to other fields such as organization science [ 21 ]. However, several studies have used an institutional approach for exploring the adoption of technology considering institutional forces as crucial to shaping organizational actions and the opinions of the decision-makers [ 22 , 23 , 24 ].

Both institutional theory and user acceptance models have independently tried to incorporate elements of the other theory to enrich their explanatory power [ 2 ]. User acceptance models have incorporated the direct effects of social influences and organizational conditions on individuals’ behavioural intention [ 25 , 26 ], and institutional studies have demonstrated that even when professionals are subject to institutional influences, their self-determination plays an important role even in highly-institutionalized and regulated settings such as hospitals [ 27 ]. Previous studies about technology acceptance and adoption compared individual and social levels including environmental factors [ 22 , 28 – 30 ], typically based on the diffusion of innovation theory (DOI) [ 31 ] or the TOE (technology, organization, and environment) framework [ 32 ]. Moreover, only a few studies have tested both explanations (institutional and individual) in an integrative framework [ 23 ] to explain the behaviour of organizations.

The main purpose of this study was to explore which are the main determinants of hospital professionals’ intention to use EMRs through a novel theoretical model that combines organizational theories and technology acceptance models. By combining these theories, this study investigated the interplay between organizational and individual factors, thus offering novel insights on the determinants of hospital professionals’ acceptance of digital technology by showing how and to what extent the interplay between individual and organizational determinants might trigger or inhibit the acceptance of digital technology. This study focused on perceived usefulness and perceived ease of use as explanatory factors at the individual level, and on inter-hospital normative and regulative forces as explanatory factors at the organizational level. Intention to use has been preferred to repetitive use as the dependent variable. This choice is because of the still relatively low adoption rate of EMRs in many Countries such as Italy, where this study is located. In the specific case of Italy, a recent report issued by the Politecnico di Milano within the research activities of the Permanent Observatory of Digital Transformation in Health Care [ 33 ] pointed out that only 53% of Italian hospitals have in place an EMRs for therapy management, only 30% of Italian hospitals have in place an EMRs that collects vital parameters and informed consensus, and only 19% of Italian hospitals have in place an EMRs that supports clinical decision-making. In this view, a large number of Italian hospitals–as well as hospitals from other Countries who are still lagging in the adoption of EMRs–is expected to commit in the next years to adopt EMRs and the understanding of which individual or organizational factors might shape hospital professionals’ intention to use EMRs might contribute to the successful adoption and implementation of such and other digital technologies. In this view, the results of this study might be valuable for hospital managers and professionals of different countries who are going to invest in the digital transformation of their hospitals.

Material and methods

Ethics statement.

The study has been approved by the Ethics Board of the University Hospital Campus Bio-Medico of Rome. (Approval number: 61/16 OSS ComEt CBM), and written consent has been obtained by professionals involved in the study.

Theoretical background

To evaluate the potential interplay between individual and institutional variables, a research framework has been created ( Fig 1 ). The framework integrates into a coherent view of two theories that belong to two different bodies of literature:

• The Technology Acceptance Model (TAM), from Information Science, that has been widely used in the last decades in healthcare to understand what leads professionals or patients to accept or reject Information Technology [ 15 ];
• The Institutional Theory, from Public Management, that has been largely adopted in the last decades to assess how institutional factors shape professionals’ behaviours [ 34 – 36 ].

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0234108.g001

Technology acceptance model.

Davis introduced the TAM in 1989 [ 15 ]. The main problem raised by the author was to understand what leads people to accept or reject Information Technology. In this regards, two main variables have been identified: the perceived usefulness and the ease of use. Perceived usefulness measures “the degree to which a person believes that using a particular system would enhance his or her job performance” [ 15 ], and therefore induces individuals to use technology as it allows to obtain better results. On the other hand, the ease of use measures “the degree to which a person believes that using a system would be free of effort” [ 15 ] and induces the potential users to use a certain technology since it requires low energy expenditure while it may bring advantages. The first one induces an individual to use technology as it allows to obtain better results in his work; the ease of use, on the other hand, stimulates potential users to use a certain technology since many advantages are supported with low energy expenditure.

Institutional theory.

The Institutional Theory refers to a line of organizational research that recognize the significant organizational effects that are associated with the increase of cultural and social forces. According to Scott [ 34 – 36 ], “Institutions are made up of cultural-cognitive, normative and regulative elements, which together with associated activities and resources offer stability and meaning to social life.” These three forces are present in totally developed institutional systems, with economists and political scientists emphasizing regulative, sociological and normative factors, and anthropologists and organizational theorists emphasizing cognitive-cultural factors. According to this perspective, individuals are embedded in institutional pillars that limit the scope of their rational assessment and direct the engagement of specific behaviours [ 34 – 36 ]. Scott [ 34 – 36 ] defines the three institutional pillars as follows:

• regulative pillars : which regard the existence of regulations, rules and processes whose breach is monitored and sanctioned;
• normative pillars : which introduce a social dimension of appropriate behaviours in the organization;
• cultural pillars : which emphasize the use of common schemas, frames, and other shared symbolic representations that create an attachment to the ‘appropriate’ behaviour.

Research framework

Consistently to our research questions, we combined the two theories described above to develop an original, comprehensive research framework where individual and institutional determinants have been interlinked to explore their potential interplay in explaining hospital professionals’ intention to use an EMR. Coherently to past researches about user acceptance of new technologies [ 36 , 37 ], we considered age and job seniority as key control variables. Additionally, to narrow the knowledge gap about how hospital professionals belonging to either different profession (e.g., physicians vs. nurses) or different speciality (e.g., cardiology vs. orthopaedics) might be interested to use an EMR, we included clinical speciality and profession as control variables. Fig 1 offers a synoptic view of our research framework, where the independent variable (i.e., the intention to use an EMR) is explained by individual factors from TAM (i.e., perceived usefulness and perceived ease of use) as well as by institutional factors from Institutional Theory (i.e., regulative factors that refer to the degree of adhesion to hospital managers’ goals, and normative factors that explain the peer influence among hospital colleagues. Control variables have been also displayed.

According to the research questions and the research framework, the following research hypotheses (H) were stated: H1: Individual factors (perceived usefulness, perceived ease of use) directly affect the intention to use EMRs; H2: Organizational factors (normative and regulative factors) directly affect individual factors and the intention to use EMRs; H3: Some control variables (age, seniority, clinical specialities and different professions) directly affect individual factors and the intention to use EMRs.

Setting and research methodology

Given the explorative nature of this study, a single case study research design has been adopted. The choice of a single case study offers the opportunity to eliminate potential confounding factors due to the heterogeneity–in terms of strategy, legacy, professionals’ behaviours and technology infrastructure–that different hospitals might show. We selected the Teaching Hospital Campus Bio-Medico (CBM) in Rome (Italy) as an adequate setting for investigating our research questions. This hospital is mid-size (around 300 beds), many-disciplines, teaching and private. Being a teaching hospital, there is more room for divergent goals between professionals and managers, thus creating the correct setting where to investigate the interplay between individual and organizational factors. Being many-discipline, there is room to study the potential conflict among professionals from different disciplines concerning the intention to use EMRs. Finally, being mid-size, CBM is a valid setting to observe the potential divergence between nurses and doctors in the intention to use EMRs. A quantitative study has been performed using a survey administered to hospital professionals (physicians and nurses). The questionnaire has been designed based on the scales identified in the literature and reviewed in detail by the authors. Moreover, a pilot test of the questionnaire has been carried out before the survey. The initial questionnaire comprised 20 items that were reviewed for face validity by a panel of four experts, consisting of one nurse and one physician—with more than 9 years of work experience -, and two engineers with expertise in Information Science. Panel members were asked to evaluate each statement for clarity, ease of use and appropriateness. Based on their comments and suggestions, five items were removed and changes were made in the wording of several items to increase clarity.

This 15-item questionnaire was tested for content validity by 10 experts not involved in the preceding phase to identify its ability to measure the determinants of the intention to use EMRs in hospitals and to identify, for each item, utility, consistency with the research objectives, easy of reply and other important aspects to take into account. Audio-recorded individual interviews using a semi-structured grid were carried out with 10 experts including two nurses, three head nurses, two managers and three physicians. The interviews lasted 60 minutes on average and were conducted in a designated room by three researchers: one acted as the interviewer, the other two helped with audio-recording and with filling out the grid for item evaluation. Based on the expert evaluation, three items were modified.

The questionnaire consists of two main sections: scales and constructs of the proposed model; control variables and characteristics of respondents. Eleven items evaluated individual variables, in particular, the scale for the measurement of perceived usefulness has been adapted from the studies of Venkatesh [ 38 , 39 ]. Organizational variables were explored through 4 items related to normative and regulative factors. The scale for the measurement of normative and regulative factors has been adapted from the study of Scott [ 20 ]. The survey items are available in Annex ( S1 Table ). Additional questions have been designed to gather demographic and sample information. All questionnaire items related to the constructs of the proposed model were explored using a 7 point Likert scale with 1 indicating “strongly disagree” and 7 “strongly agree”. The first re-call has been made one week after the expiration date for compilation. Three days after the first follow-up, the second recall has been sent. Finally, three days after, the third recall has been sent.

The statistical analysis was performed using the software Stata 14.1®. The internal consistency was evaluated through Cronbach’s Alpha coefficients, the path analysis was performed to test the proposed model considering a p-value of <0.05 as significant. The correlation between profession (doctors vs. nurses) and the answers provided for each item was analyzed through the Fisher’s test; a p-value of <0.05 was considered significant.

The study has been approved by the General Management and the Ethics Board of CBM. The link for the online questionnaire was sent by e-mail to 380 nurses and 250 physician representatives of different clinical areas. All questionnaires were filled out anonymously in a period between February and September 2018. The final sample included 114 hospital professionals (response rate 19%), composed by 78 (68%) nurses and 36 (32%) physicians. They were 84 (74%) females and 30 (36%) males, aged 37.4 years on average (range 23–66, SD 9.6), with a mean work experience of 13.24 (range 0.5–41, SD 8.73). The sample of respondents has been compared–in terms of age, gender and clinical experience–to the whole population of doctors and nurses enrolled at CBM confirming the absence of potential response biases related to the non-respondents.

Questionnaire’s internal consistency

The internal consistency of constructs was evaluated through Cronbach's Alpha coefficients, values greater than or equal to 0.7 were considered acceptable. (α ≥ 0.90 were considered excellent; 0.8 ≤ α < 0.9 good; 0.7 ≤ α < 0.8 acceptable; 0.6 ≤ α < 0.7 questionable; 0.5 ≤ α < 0.6 poor; α < 0.5 unacceptable) ( Table 1 ).

https://doi.org/10.1371/journal.pone.0234108.t001

Determinants of current behaviours

Data show that both physicians and nurses expect many benefits from the use of EMRs. In particular, they think EMRs will have a positive impact on relevant factors such as quality, efficiency and effectiveness of care; handover communication among healthcare workers; teaching, tutoring and research activities; greater control of their tasks. Data confirm that perceived usefulness (C = 0.33**) directly affects the intention to use EMRs. Concerning the organizational factors, data prove that there does exist an interplay between them and individual determinants. In fact, normative factors directly affect perceived usefulness (C = 0.30**), perceived ease of use (C = 0.26**) and intention to use EMRs (C = 0.33**). Regulative factors affect the intention to use EMRs, with a negative sign (C = -0.21**). Control variables (i.e., age, seniority, clinical area and profession) have no impact on other variables in our model. Fig 2 offers a graphical representation of our results.

https://doi.org/10.1371/journal.pone.0234108.g002

Moreover, the findings show a significant correlation between being a nurse or a physician and the perceived ease of use and intention to use EMRs. In particular, more nurse than physicians perceive EMRs as easy to use (p = 0.019 for the item “the EMR will be easy to use”) and state that they would like to use it (p = 0.01 for the item “if I had the opportunity I would use the EMR for most of my work’s processes”).

This study sought to better clarify the relationship between organizational and individual determinants of the intention to use EMRs in a hospital setting by nurses and physicians. Previous studies [ 40 – 46 ] have focused mainly on either the barriers or the facilitators that might impact on the implementation of EMRs, but, to the best of authors’ knowledge, it has never been deepened if and how organizational and individual factors do interact and affect jointly hospital professionals’ motivation to use EMRs. Findings confirmed the positive role played by the perceived usefulness as driving individual factor to the intention to use EMRs and shed light on the significant positive role played by the normative (peer influence) factors [ 2 ], both with direct and indirect effects. In this view, hospital managers can leverage on lead peer influence (i.e., innovation champions) to motivate, generate and manage change and generate a virtuous circle inside the hospital to motivate the use of EMRs. The EMRs implementation process should take into account that professionals need proper time to re-establish control over their tasks and processes. The introduction of EMRs in daily clinical practice changes the status quo and, if, on one hand, it allows many new opportunities, on the other hand, it involves changes that can have different effects on hospital professionals also based on their own characteristics, knowledge, skills and work type. In general, this is what happens in the case of effective implementation, while the consequences of poorly managed implementation can be very complex and involve a greater expenditure of time, energy and money to restart the processes at the previous speed and functionality. In this sense, to increase the motivation of users in all phases of the project represent an essential point for effective management of change. This study confirms the importance of involving front-line professionals, as soon as the hospital decides to start the implementation phase to increase their motivation to use EMRs. In fact, as a result of their involvement, professionals will better understand the rationale of this technological shift and their perception of usefulness will increase consequently. Moreover, it is important to consider that, as reported by Gastaldi et al. [ 2 ] in the absence of coercive mechanisms, institutional pressures toward EMR use are primarily normative and/or mimetic [ 2 ].

In the study, the construct “Regulative factor” has been derived from the Institutional theory and is aimed at exploring the pressure that a hospital professional might perceive from the goals set by hospital managers. This pressure is intended to be independent of the specific strategy/initiative and to be a general availability of a hospital professional to align his/her behaviour to the goals set by hospital managers. An example of a question is: “I very much agree with most of the objectives of the management”. The regulative factor should be analyzed together with the construct “Normative factor” that crystallizes the perceived pressure from peers. Hospitals are intended as professional bureaucracies where professionals feel more the pressures from peers rather than from apex managers. What is interesting is that the regulative factor affects negatively the intention to use, meaning that more the general agreement with managers’ goals less the intention to use an EMR. This finding might appear as counter-intuitive and contrary to what has been found in other studies [ 47 ]. This result cannot be explained by the potential misalignment between hospitals managers’ goals and those of physicians and nurses, being the former more focused on the efficiency and the latter on the effectiveness of care delivery. Managers at CBM have proved to be committed to the quality of care and not to efficiency strategies that might reduce the effectiveness of care. This context is quite typical in Italy, where the tensions between “medicine” and “management” are less evident than in other countries, such as in the US. We think that the negative impact of the regulative factors on the perception of usefulness is because hospital managers did not detail enough their goals about the digital transformation of care delivery, thus impacting negatively on hospital professionals’ perception about the usefulness of an EMR. Being these goals enough general–e.g., providing support to research activities and care delivery, promoting efficiency and process redesign–while the linkage between the regulative factors and the perception of usefulness failed to materialize, the linkage between the regulative factors and the intention to use EMRs became negative as hospital professionals lost the connection between EMR usage and managers’ goals. In this view, more contextualized goals about the usage of EMR are expected to positively affect the intention to use it among those professionals who are more willing to be adherent to managers’ goals. This finding should be tested and confirmed by further replication studies that might capture more in detail the relationships between regulative factors and either the perceived usefulness or the intention to use. For instance, it might be valuable to understand whether and how the co-development of hospitals goals between managers and professionals might impact these relationships as well as the specific content of hospital goals (financial vs. quality of care, operative vs. research).

This study offers original insights to further the ongoing debate about the digital transformation of hospitals, with a focus to EMRs. Our results show that there is an interplay between individual and organizational factors in shaping hospital professionals’ intention to use EMRs. The study showed that the main determinants of the intention to use EMRs are the normative ones (peer influence) and the individual ones (perceive usefulness).

From an academic viewpoint, the study offers an original perspective and a new theoretical framework, which combines organizational theories and technology acceptance models to explain hospital professionals’ acceptance of EMRs. In particular, the results confirm the importance of individual variables, not only as directly related to the acceptance of new technology, but also as important mediators between institutional variables and acceptance, thus highlight and confirming the importance of the connections between organizational studies and information science.

Despite the original contributions, this study suffers at least two limitations that should be addressed by future research. First, the research design is based on a single case study. Further research should consider a multi-centre design, thus allowing the generalization of our results. Moreover, a multi-centre study will allow exploring the role that hospital characteristics–in terms of strategy, legacy, etc.–might have on shaping both the organizational and individual factors investigated in this study. Second, this study investigated the intention to use EMRs as the dependent variable. Further research should consider hospitals where EMRs are already mature technologies, thus allowing the investigation of the actual use and which factors might facilitate/inhibit the translation of the intention to use into actual use.

Supporting information

S1 table. questionnaire..

https://doi.org/10.1371/journal.pone.0234108.s001

S2 Table. Perceived usefulness.

https://doi.org/10.1371/journal.pone.0234108.s002

S3 Table. Perceived ease of use.

https://doi.org/10.1371/journal.pone.0234108.s003

S4 Table. Intention to use.

https://doi.org/10.1371/journal.pone.0234108.s004

S5 Table. Normative factors (Peer influence).

https://doi.org/10.1371/journal.pone.0234108.s005

S6 Table. Regulative factors (Adhesion to the management objectives).

https://doi.org/10.1371/journal.pone.0234108.s006

Acknowledgments

We want to thank Dr Federica Segato for her valuable comments in all phases of this study.

• 1. Barlow J. Managing Innovation in Healthcare. World Scientific Publishing Europe. London; 2017.
• View Article
• PubMed/NCBI
• Google Scholar
• 10. Scott W.R., & Davis G. Organizations and organizing: rational, natural and open systems perspectives. New Jersey: Prentice Hall; 2008.
• 12. Scott W.R., Ruef P., Mendel P., Caronna C. Institutional change and healthcare organizations: from professional dominance to managed care. University of Chicago Press. Chicago; 2000.
• 19. Lawrence T.B., Suddaby R. and Leca B. Institutional Work: Actors and Agency in Institutional Studies of Organizations. Cambridge University Press, Cambridge, UK; 2009.
• 27. Leca, B., Battilana, J. and Boxenbaum, E. Agency and Institutions: A Review of Institutional Entrepreneurship, Working Paper 08–096, Harvard Business School; 2008.
• 31. Rogers E. Diffusion of Innovations. 4th ed., Free Press, New York; 1995.
• 32. Tornatzky L., Fleischer M. The Process of Technology Innovation, Lexington Books, Lexington, MA; 1990.
• 33. https://www.osservatori.net/it_it/pubblicazioni/rapporti/sanita-digitale-italia-stato-dell-arte-e-trend .

The Impact of the Electronic Health Record on Moving New Evidence-Based Nursing Practices Forward

Affiliations.

• 1 The Ohio State University Wexner Medical Center, The Ohio State University College of Nursing, Columbus, OH, USA.
• 2 Translational Implementation Science Core, Helene Fuld Health Trust National Institute for EBP in Nursing and Healthcare, Columbus, OH, USA.
• 3 The Ohio State University Wexner Medical Center, Columbus, OH, USA.
• 4 Helene Fuld Health Trust National Institute for EBP in Nursing and Healthcare, The Ohio State University, Columbus, OH, USA.
• 5 Department of Critical Care Nursing, The Ohio State University Wexner Medical Center, Columbus, OH, USA.
• PMID: 32233009
• DOI: 10.1111/wvn.12435

Background: Anecdotal reports from across the country highlight the fact that nurses are facing major challenges in moving new evidence-based practice (EBP) initiatives into the electronic health record (EHR).

Purpose: The purpose of this study was to: (a) learn current processes for embedding EBP into EHRs, (b) uncover facilitators and barriers associated with rapid movement of new evidence-based nursing practices into the EHR and (c) identify strategies and processes that have been successfully implemented in healthcare organizations across the nation.

Methods: A qualitative study design was utilized. Purposive sampling was used to recruit nurses from across the country (N = 29). Nine focus group sessions were conducted. Semistructured interview questions were developed. Focus groups were conducted by video and audio conferencing. Using an inductive approach, each transcript was read and initial codes were generated resulting in major themes and subthemes.

Results: Five major themes were identified: (a) barriers to advancing EBP secondary to the EHR, (b) organizational structure and governing processes of the EHR, (c) current processes for prioritization of EHR changes, (d) impact on ability of clinicians to implement EBP and (e) wait times and delays.

Linking evidence to action: Delays in moving new EBP practice changes into the EHR are significant. These delays are sources of frustration and job dissatisfaction. Our results underscore the importance of a priori planning for anticipated changes and building expected delays into the timeline for EBP projects. Moreover, nurse executives must advocate for greater representation of nursing within informatics technology governance structures and additional resources to hire nurse informaticians.

Keywords: electronic health record; evidenced-based practice; hospital governance; nursing informatics.

© 2020 Sigma Theta Tau International.

• Electronic Health Records / standards*
• Electronic Health Records / trends
• Evidence-Based Practice / methods*
• Evidence-Based Practice / standards
• Evidence-Based Practice / trends
• Focus Groups / methods
• Nursing Research / instrumentation*
• Nursing Research / methods
• Nursing Research / trends
• Qualitative Research

Grants and funding

• Helene Fuld Health Trust National Institute for EBP in Nursing and Healthcare

• Introduction
• Conclusions
• Article Information

EHR indicates electronic health record.

The brown color represents the overlap between the US and non-US health systems in this overlaid histogram.

eFigure. Note Text by Templates, Copy/Paste, and Manual Entry

eTable 1. Regression Results Excluding Western Europe

eTable 2. Regression Results Excluding Canada

eTable 3. EHR Time Per Encounter

eTable 4. EHR Time Per Appointment: Time per Day Divided by Average Number of Clinician Appointments per Day

• Omission in the Author Contributions Section JAMA Internal Medicine Correction February 1, 2021
• Advancing Practice Science With Electronic Health Record Use Data JAMA Internal Medicine Invited Commentary February 1, 2021 Christine Sinsky, MD

See More About

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Others Also Liked

• Download PDF
• X Facebook More LinkedIn

Holmgren AJ , Downing NL , Bates DW, et al. Assessment of Electronic Health Record Use Between US and Non-US Health Systems. JAMA Intern Med. 2021;181(2):251–259. doi:10.1001/jamainternmed.2020.7071

Manage citations:

© 2024

• Permissions

Assessment of Electronic Health Record Use Between US and Non-US Health Systems

• 1 Interfaculty Initiative in Health Policy, Harvard University, Cambridge, Massachusetts
• 2 Harvard Business School, Boston, Massachusetts
• 3 Department of Medicine, Stanford University, Stanford, California
• 4 Clinical Excellence Research Center, Stanford University, Stanford, California
• 5 Department of General Internal Medicine, Brigham & Women’s Hospital, Boston, Massachusetts
• 6 Harvard Medical School, Boston, Massachusetts
• 7 Division of Hematology, Department of Medicine, Stanford University, Palo Alto, California
• 8 Department of Economics, Harvard University, Cambridge, Massachusetts
• 9 Graduate School of Business, Stanford University, Stanford, California
• Invited Commentary Advancing Practice Science With Electronic Health Record Use Data Christine Sinsky, MD JAMA Internal Medicine
• Correction Omission in the Author Contributions Section JAMA Internal Medicine

Question   Does the use of the electronic health record (EHR) differ between clinicians in the US and those in other countries?

Findings   In this cross-sectional study of the EHR metadata of 371 health systems in the US and abroad, US clinicians vs non-US clinicians were found to spend more time per day actively using the EHR, receive more system-generated messages, write a higher proportion of automatically generated note text, and spend more time using the EHR after hours.

Meaning   Findings from this study suggest that US clinicians compared with non-US clinicians had a higher EHR burden, which could be alleviated by minimizing EHR uncertainties and consolidating documentation requirements.

Importance   Understanding how the electronic health record (EHR) system changes clinician work, productivity, and well-being is critical. Little is known regarding global variation in patterns of use.

Objective   To provide insights into which EHR activities clinicians spend their time doing, the EHR tools they use, the system messages they receive, and the amount of time they spend using the EHR after hours.

Design, Setting, and Participants   This cross-sectional study analyzed the deidentified metadata of ambulatory care health systems in the US, Canada, Northern Europe, Western Europe, the Middle East, and Oceania from January 1, 2019, to August 31, 2019. All of these organizations used the EHR software from Epic Systems and represented most of Epic Systems’s ambulatory customer base. The sample included all clinicians with scheduled patient appointments, such as physicians and advanced practice practitioners.

Exposures   Clinician EHR use was tracked by deidentified and aggregated metadata across a variety of clinical activities.

Main Outcomes and Measures   Descriptive statistics for clinician EHR use included time spent on clinical activities, note documentation (as measured by the percentage of characters in the note generated by automated or manual data entry source), messages received, and time spent after hours.

Results   A total of 371 health systems were included in the sample, of which 348 (93.8%) were located in the US and 23 (6.2%) were located in other countries. US clinicians spent more time per day actively using the EHR compared with non-US clinicians (mean time, 90.2 minutes vs 59.1 minutes; P  < .001). In addition, US clinicians vs non-US clinicians spent significantly more time performing 4 clinical activities: notes (40.7 minutes vs 30.7 minutes; P  < .001), orders (19.5 minutes vs 8.75 minutes; P  < .001), in-basket messages (12.5 minutes vs 4.80 minutes; P  < .001), and clinical review (17.6 minutes vs 14.8 minutes; P  = .01). Clinicians in the US composed more automated note text than their non-US counterparts (77.5% vs 60.8% of note text; P  < .001) and received statistically significantly more messages per day (33.8 vs 12.8; P  < .001). Furthermore, US clinicians used the EHR for a longer time after hours, logging in 26.5 minutes per day vs 19.5 minutes per day for non-US clinicians ( P  = .01). The median US clinician spent as much time actively using the EHR per day (90.1 minutes) as a non-US clinician in the 99th percentile of active EHR use time per day (90.7 minutes) in the sample. These results persisted after controlling for organizational characteristics, including structure, type, size, and daily patient volume.

Conclusions and Relevance   This study found that US clinicians compared with their non-US counterparts spent substantially more time actively using the EHR for a wide range of clinical activities or tasks. This finding suggests that US clinicians have a greater EHR burden that may be associated with nontechnical factors, which policy makers and health system leaders should consider when addressing clinician wellness.

After the passage of the Health Information Technology for Economic and Clinical Health Act of 2009, health systems in the United States rapidly adopted electronic health records (EHRs). 1 - 3 Electronic health records have shown promise in improving quality, 4 , 5 improving communication, 6 and decreasing redundant use. 7 - 9 However, studies suggest that unintended consequences have emerged, including clinician frustration with EHRs and the large amount of time spent working in these systems. 10 - 15 Use of EHRs has been associated with decreased job satisfaction and burnout among clinicians 16 - 20 as well as disruptions of clinician-patient relationships. 11 , 21 - 24 In addition, EHRs enable clinicians to work outside of the physical walls of their facilities, resulting in so-called desktop medicine that expands to after-hours work. 25

It is critical to understand how EHRs have changed clinician work, productivity, and well-being so as to be aware of their association with clinician burnout. Most studies of EHRs have relied on data from a small number of organizations, largely in the US. 11 , 15 An early comparison of EHR documentation found that US clinician notes were 4 times longer than the notes of their counterparts in other countries and that non-US clinicians were more likely to report satisfaction with their EHR systems. 26

To date, few large-scale studies of clinician EHR work have been conducted and none have addressed the global variation, with the largest study focusing on cross-specialty variation within the US. 27 , 28 Little is known regarding the variation in time spent on specific EHR tasks, such as messaging (also known as in-basket), ordering, or documentation. To address this, we conducted a large, multiorganizational, cross-national comparison of EHR use by clinicians in ambulatory settings. Using a unique data set of EHR metadata, this study aimed to provide insights into which EHR activities clinicians spend their time doing, 21 the EHR tools they use, 26 , 29 the system messages they receive, 22 , 30 , 31 and the amount of time they spend in the EHR after hours. 30 , 32

This cross-sectional study was deemed exempt by the institutional review board at Stanford University because it used deidentified data and was not human subject research. The study was conducted from November 1, 2019, to July 1, 2020.

The data sample consisted of deidentified metadata from 371 ambulatory care health systems in the US, Canada, Northern Europe, Western Europe, the Middle East, and Oceania for the study period January 1, 2019, to August 31, 2019, obtained from the EHR software vendor Epic Systems. Epic Systems has the largest ambulatory market share in the US, and the study sample represented most of Epic System’s global ambulatory customer base. 33 , 34 Organizations that used the Epic EHR software but requested that their metadata not be shared were excluded from the sample. The sample included all clinicians with scheduled patient appointments, such as physicians and advanced practice practitioners (eg, physician assistants and nurse practitioners), and excluded nonclinical users of the EHR and clinicians who did not have scheduled appointments (eg, nurses and medical assistants).

In the data set, each organization represented a single EHR installation, which may include multiple suborganizations and facilities. For example, a hospital with several freestanding ambulatory clinics that used the same EHR was counted as 1 organization. Data were aggregated at the organizational level, with the means calculated over the duration of the study.

The data included organizational characteristics provided by Epic Systems. These characteristics included country (with organizations outside of the US grouped into geographic regions), region within the US, organizational structure (ie, ambulatory only, hospital and clinic, or other, such as retail clinic), organizational type (ie, teaching or academic, community, pediatric only, safety net, other ambulatory clinic, or other), whether the health system had an integrated health plan, size (measured by both the number of physicians and the number of outpatient encounters during the study period), and patient volume (ie, mean number of daily scheduled appointments per clinician during the study period).

The software has the capacity to monitor EHR activity at an extremely granular level, collecting metadata primarily with the Signal data extraction tool. Those metadata document the time the system is used, as indicated by keystrokes, mouse movements, clicks, scrolling, and interactions with the EHR. In this analysis, time was defined as the time a user was performing active tasks in the EHR. If no activity is detected for 5 seconds, the system stops counting time. This time measure captures active EHR engagement and excludes other time a clinician spends performing nonkeystroke tasks while the EHR is open.

Measurement of clinician work in the context of an EHR is a challenge. In this study, we defined work as the active interaction between the clinician and the tool that excluded other work, such as talking with the patient or digesting information while reading data in the EHR. This work time measure is inherently conservative but is available from metadata in a standardized fashion across all study sites. However, this measure may result in an underestimate of true EHR work time if clinicians spend time reading notes or otherwise performing EHR tasks without directly interacting with the system. For this reason, the EHR active work time should be considered as associated with but distinct from measurement methods, such as audit log data or time and motion studies used in previous studies of clinician EHR time.

We measured active EHR use time categorized into 4 main activities. 11 , 28 The first activity was clinical review, defined as time spent reviewing test results and patient history. The second activity was notes, defined as time spent documenting the clinical encounters. The third activity was in-basket messages, defined as time spent reading and writing messages and managing the messaging feature of the EHR. In-basket messages included those between clinical team members; messages from patients to the clinician; and a wide array of automatically generated messages regarding available results, orders, and prescribing, among others. The fourth activity was orders, defined as time spent entering orders for patients and other tasks related to such orders, such as associating a diagnosis. All EHR time was measured as the mean time per scheduled day in the study period to account for the differences in workload across clinicians who may not be practicing full-time.

For the notes activity, we measured the source of note text. We identified text generated with a manual process as the note text created by typing, transcription by a scribe or another nonclinician EHR user, or voice input using a dictation or text-to-speech program. We classified text generated with an automated process as the note text created by copying and pasting or using the software templates (eg, NoteWriter, SmartTool, and dot phrases) or any other mechanism to bring text from other parts of the EHR. We then calculated the proportion of note text (by character) generated manually vs automatically at the organizational level for the study period. To describe in detail the level of use of automatically generated text, we chose to measure the proportion of automated compared with manual text in the notes rather than the presence or absence of automated note text.

We measured the mean number of in-basket messages received per clinician per day, both overall and by source. In-basket messages in the EHR can be generated from 7 possible sources: system, team, results, prescription, patient, custom, and other. These categories were defined to be consistent with previous studies. 22

We calculated after-hours time in the EHR per day. This metric included only time spent performing clinical work and does not include time spent for purposes of tasks such as research, data analysis, performance measurement, or customization. We defined after-hours EHR time as any time between 5:30 pm and 7:00 am local time on weekdays and any time on weekends, unless the clinician was scheduled during those times, according to the Epic Systems definition of after-hours time, which was broadly consistent with the literature on after-hours EHR work. 28 , 30 Weekend days were established by specific locale for non-US health systems. Because of data limitations, the after-hours time was calculated using data from April 1, 2019, to August 31, 2019.

We compared the descriptive statistics of the organizational characteristics for US and non-US organizations. We tested for statistically significant differences using Fisher exact tests for organizational structure and organizational type and unpaired, 2-tailed t tests with unequal variance for number of physicians, number of outpatient visits, and number of scheduled appointments per clinician per day. We then calculated the descriptive statistics for EHR time per day across the 4 activities (clinical review, notes, in-basket messages, and orders) as well as the total EHR time per day for both US and non-US clinicians using unpaired, 2-tailed t tests with unequal variance to evaluate the statistical significance. We calculated the descriptive statistics and tests of statistical significance in comparing US with non-US health systems for manual and automated note text, in-basket messages received per day (in total and across message categories), and after-hours time per day.

To assess these associations while controlling for observable organizational characteristics, we created 4 ordinary least-squares regression models. Each model used a different dependent variable: total EHR time per day in minutes, after-hours EHR time per day in minutes, percentage of note text generated from automated sources, and EHR system–generated messages received per clinician per day. Each model included an indicator for the US or non-US location of the health system as well as organizational characteristics such as structure, type, health plan integration, size, and daily patient volume. All models had robust standard errors clustered at the health system level. In addition, we conducted a qualitative interview with a non-US hospital chief medical information officer who was previously employed by a health care organization in the study sample.

We conducted several robustness and sensitivity analyses. We calculated the source of the note text with 3 categories (manual or automated process [copy and paste or Epic template]) by disaggregating the copy-and-paste and template text. We ran the regression models several times (first excluding the health systems located in Western Europe and then excluding health systems located in Canada) to ensure that the results were not driven solely by the non-US health systems in areas most represented in the sample. We ran the model of total EHR time with the dependent variable expressed as time per encounter rather than time per day and time per day normalized by number of clinician encounters per day.

A 2-sided P  = .05 was used to indicate statistical significance. All statistical calculations and plots were done with Stata, version 16.1 (StataCorp LLC). Data were analyzed from December 1, 2019, to July 1, 2020.

Of the 371 health systems included in the sample, 348 (93.8%) were located in the US and 23 (6.2%) were located in other countries. Western Europe was the region most represented outside of the US, with 11 health systems (47.8%), followed by Canada with 6 systems (26.1%). The Middle East (3 [13.0%]), Northern Europe (2 [8.7%]), and Oceania (1 [4.3%]) composed the remaining health care organizations with the Epic EHR. Full descriptive statistics are available in Table 1 .

US clinicians spent a mean time of 90.2 minutes actively using the EHR per day compared with the 59.1 minutes spent per day by non-US clinicians ( P  < .001). Differences in time spent performing each of the 4 clinical activities were observed between the US and non-US clinicians, such as notes (40.7 minutes vs 30.7 minutes; P  < .001), orders (19.5 minutes vs 8.75 minutes; P  < .001), in-basket messages (12.5 minutes vs 4.80 minutes; P  < .001), and clinical review (17.6 minutes vs 14.8 minutes; P  = .01).

Clinicians in the US created more notes that were generated from automated sources compared with non-US clinicians (77.5% vs 60.8% of note text; P  < .001), and similar results were found when we disaggregated automated text into copy-and-paste and templated text (eFigure in the Supplement ). In addition, US clinicians compared with non-US clinicians received more messages per day in total (33.8 vs 12.8; P  < .001) and from various sources: system (11.5 vs 6.0; P  < .001), team (11.4 vs 3.27; P  < .001), results (6.49 vs 3.01; P  < .001), prescription (2.70 vs 0.14; P  < .001), patient (1.06 vs 0.10; P  < .001), and custom (0.35 vs 0.03; P  < .001). US clinicians worked in the EHR for a longer time after hours per day than did non-US clinicians (26.5 minutes vs 19.5 minutes; P  = .01) ( Figure 1 ). The distribution of US and non-US clinician total EHR time per day is shown in Figure 2 . The median time spent working in the EHR was 90.1 minutes per day for US clinicians compared with 58.3 minutes per day for non-US clinicians. The 99th percentile of EHR work time for US clinicians was 143.4 minutes per day, whereas the 99th percentile for non-US clinicians was 90.7 minutes per day.

The multivariable models found similar differences between US and non-US health systems. Compared with non-US clinicians (reference group), US clinicians spent 23.67 more minutes per day actively using the EHR (β = 23.67; 95% CI, 17.70-29.64; P  < .001), spent 7.23 more minutes after hours per day interacting with the EHR (β = 7.23; 95% CI, 2.30-12.16; P  < .001), composed 17 percentage points more note text with automated tools (β = 0.17%; 95% CI, 0.11%-0.22%; P  < .001), and received 5.28 more system-generated in-basket messages per day (β = 5.28; 95% CI, 3.18-7.38; P  < .001) ( Table 2 ). Regression model results were robust to excluding non-US health systems in Western Europe (eTable 1 in the Supplement ) or Canada (eTable 2 in the Supplement ) and to measuring total EHR time at the encounter level (eTables 3 and 4 in the Supplement ). For example, US clinicians spent more time per day in the EHR (β = 32.02 minutes; P  < .001), spent more after-hours time per day (β = 10.98 minutes; P  < .001), generated a higher proportion of note text from automated sources (β = 0.19%; P  < .001), and received more system messages per day (β = 4.03; P  = .01) (eTable 1 in the Supplement ).

We found large differences in EHR use between US and non-US clinicians. This finding is notable given that all of the organizations in the study used the same EHR software, although institutions may customize the system’s functionality. The results suggest that a portion of clinician EHR work was associated with contextual factors unique to national health systems rather than the technical demands of the EHR itself or the clinical demands of delivering care. Although some EHR features provide considerable value to patients and clinicians, quality of care delivered in the US is unlikely to be substantially better than that in the other countries examined in this study. 35 , 36 Furthermore, although health outcomes are associated with many factors ranging from technical clinical skills to social determinants, such as poverty, racism, and lack of access, recent research has shown that US-based health care organizations are not better than their counterparts in other countries at achieving high performance in most process quality measures during care delivery that information technology could help improve, such as reducing rates of medication errors. 37 Therefore, the finding that US clinicians spent more time actively using the EHR is concerning. This observation is consistent with other studies that suggested EHR adoption in the US is not associated with decreased administrative burdens 38 ; we found that the average US clinician spent as much time actively using the EHR as a non-US clinician in the 99th percentile of EHR work time. This finding suggests that if US clinicians could decrease their EHR work time to the level of their non-US peers, they may be able to increase the volume of patients seen or improve quality along multiple dimensions, such as longer visits and more patient-centered care.

The differences in EHR use between US and non-US clinicians are associated with multiple factors. For example, the reason that US clinicians spend more time on orders may be the need for diagnostic association, a billing requirement in the US that is absent from most non-US billing practices; alternatively, US clinicians may need to enter individual electronic orders for low-risk tasks, such as ear irrigation or immunizations. Similarly, additional time spent by US clinicians on in-basket messages may reflect policy-driven differences in volume of messages received. The Meaningful Use incentive program in the US mandated both secure messaging with patients and electronic prescribing, resulting in nearly universal adoption of those capabilities by US health systems. Connecting more members of the care team, such as pharmacists, to the EHR led to a higher volume of team messages. Given the low level of patient and prescription messages received by non-US clinicians and the wide variation in the implementation of electronic prescribing worldwide, 39 the requirements of the Meaningful Use program were likely a factor in the differences in in-basket messages time. US clinicians also spent more time working in the Notes function despite using more automatically generated text. Previous studies have shown that the length of notes is considerably longer in the US, 26 which may illustrate the implication of greater use of automatically generated text for nonclinical purposes such as billing, quality reporting, documentation to minimize legal liability, and other administrative tasks.

Some EHR areas in which US clinicians spend more time likely deliver value to patients, such as secure messaging, which patients may prefer to alternatives. Other functions may improve safety and save time elsewhere in the care delivery process, such as e-prescribing, which was considered EHR work for US clinicians in this study but was less likely to be observed among non-US clinicians who may write prescriptions manually. 40 Additional time in the orders activity may reflect higher levels of clinical decision support, which may improve safety, 41 , 42 albeit potentially at the cost of burnout and alert fatigue. 43 , 44 Studies have suggested associations between clinician burnout and total EHR time, 21 after-hours EHR time, 30 percentage of note text generated from automated sources, 26 and system-generated messages received per day. 22 , 30

We believe that findings from this study have important implications for health policy and practice. Although substantial attention has been focused on the role of EHRs in burnout, the results showed that US clinicians used EHRs in different ways compared with their non-US peers, suggesting that at least some of the time burden was associated with nontechnical aspects of EHR implementation that were specific to US market characteristics such as differences in workflow or policy. Factors such as a multipayer billing environment, in which clinicians rely on documentation to ensure their claims are not denied, as well as Meaningful Use and incorporation of quality measurement and administrative functions may have contributed to the results.

Policy makers who are concerned about the association of EHRs with clinician burnout should consider the implications of the study results. Most ambulatory practices contract with multiple payers and must be prepared to submit clinical documentation to substantiate reimbursable services, and they may opt to overdocument to guard against rejected claims. Similarly, US clinicians may overdocument if they are concerned that they may need to defend themselves from malpractice claims. In our qualitative interview with Chris Hayes, MD, a chief medical information officer in a non-US health system (email and telephone communication, May 2020), he indicated that, despite working in a fee-for-service reimbursement setting, there is no need to document anything that is not relevant to clinical care because medical claims are rarely challenged. Although the Centers for Medicare & Medicaid Services has clarified documentation requirements, 45 aggressive steps toward reducing inefficiencies in reimbursement are needed. Similarly, EHRs have promised to capture the information necessary to measure and improve quality. 46 However, quality reporting has been fragmented, involving multiple stakeholders. 47 , 48 Minimizing uncertainties and consolidating requirements could alleviate the EHR burdens of US clinicians.

This study has some limitations. First, this analysis is descriptive and cannot address the causal association between country of practice and EHR use. Second, the data set we obtained only accounted for ambulatory practice; we were unable to evaluate EHR use in an inpatient, emergency, or long-term care setting. Third, because the study used a narrowly defined measure of EHR work that counted only the time that clinicians spent actively working within the system, the results were difficult to compare with those of other investigations that used audit log data or time-and-motion studies and were likely to underestimate work time; the times represented in this study should not be taken as indicative of the length of a clinician workday. Furthermore, the sample included nonphysician clinicians and ambulatory care settings with EHR use patterns that were substantially different from those reported in studies of walk-in clinics, for example; this difference limits this study’s comparability to previous studies. However, the distribution of EHR time across activities in this study was similar to that in other studies. 11 , 28

Fourth, the data set did not include detailed information on clinician schedules, and thus we were unable to standardize the measure of scheduled days; the differences in clinician scheduling may be associated with some of the differences in EHR time between US and non-US clinicians. However, the multivariable analyses we performed, which controlled for the number of appointments per clinician per day, annual outpatient volume, and number of physicians, found similar results as our bivariate comparisons; the sensitivity analysis that compared EHR time per encounter also found results that were consistent with the time-per-scheduled-day measure. Similarly, although the US health systems had substantially greater annual patient volume compared with the non-US systems and only a small number of additional physicians, we did not have data on the exact number of nonphysician clinicians for health system deidentification purposes. However, US health systems are more likely to have more nurse practitioners and physician assistants, 49 , 50 which may explain why the differences in daily patient volume by clinician were much less pronounced in this study. In addition, although the measure of after-hours work time used a vendor-derived metric that combined a standard approach to defining a clinical work day with scheduled visit data that were consistent with data in the literature, we were unable to classify all after-hours time, which may result in some measurement error. Fifth, we had data for a broad range of ambulatory care facilities, but the data set came from a single EHR vendor and from a small number of non-US organizations, and thus the use of another vendor would have likely produced different results, limiting the external generalizability of this study. In addition, although the sample included nearly all Epic Systems customers that used the ambulatory EHR, the health systems that were also Epic customers but asked to be excluded from the sample may have different EHR use patterns. Sixth, the note text data included voice dictation and transcription by medical scribes, but we analyzed only the EHR work time spent by clinicians and not by scribes on behalf of clinicians. This exclusion may bias the results, although the data set indicated that dictation and transcription by medical scribes were much more common in the US, suggesting that we may have underestimated the difference between US and non-US clinicians. Seventh, although we included controls for health system characteristics, we did not have detailed organizational data. Understanding how EHR use varies across organizations is an important area for future research.

This cross-sectional study found that US clinicians spent substantially more time actively using the EHR than their non-US counterparts that interacted with the same technology. US clinicians had a higher EHR burden per day across 4 activities (clinical review, notes, in-basket messages, and orders). Policy makers and health system leaders who seek to address clinician wellness should consider minimizing uncertainties and consolidating documentation requirements to alleviate the burden of clinician EHR work, which is associated with US-specific market and policy factors.

Accepted for Publication: October 5, 2020.

Published Online: December 14, 2020. doi:10.1001/jamainternmed.2020.7071

Correction: This article was corrected on February 1, 2021, to indicate that Mr Holmgren and Dr Downing contributed equally as co–first authors.

Corresponding Author: A. Jay Holmgren, MHI, Harvard Business School, Soldiers Field Road, 324A Cotting House, Boston, MA 02163 ( [email protected] ).

Author Contributions: Mr Holmgren had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Mr Holmgren and Dr Downing contributed equally as co–first authors.

Concept and design: Holmgren, Downing, Bates, Milstein, Sharp, Cutler, Huckman, Schulman.

Acquisition, analysis, or interpretation of data: Holmgren, Downing, Bates, Shanafelt, Sharp, Cutler.

Drafting of the manuscript: Holmgren, Downing, Milstein, Sharp, Cutler, Schulman.

Critical revision of the manuscript for important intellectual content: Holmgren, Downing, Bates, Shanafelt, Sharp, Cutler, Huckman, Schulman.

Statistical analysis: Holmgren, Cutler, Huckman.

Obtained funding: Milstein.

Administrative, technical, or material support: Downing, Cutler, Schulman.

Supervision: Bates, Cutler, Huckman.

Conflict of Interest Disclosures: Dr Bates reported receiving grants and personal fees from EarlySense; personal fees from Center for Digital Innovation (Negev) Ltd; equity from Valera Health, CLEW, and MDClone Ltd; personal fees and other from AESOP; and grants from IBM Watson outside the submitted work. Dr Shanafelt reported being a coinventor of the Well-Being Index instruments and the Participatory Management Leadership Index, for which he receives a portion of any royalties paid to the copyright owner, Mayo Clinic, and reported receiving honoraria for providing grand rounds, keynote lectures, and advice to health care organizations. Dr Milstein reported being a co-founding scientist and paid scientific adviser of Dawnlight Technology and Prealize Health. Dr Huckman reported receiving personal fees from Kaiser Permanente, Partners Healthcare, MD Anderson Cancer Center, OhioHealth, and Ochsner Health; serving as an advisory board member for RubiconMD, Arena, and Carrum Health; and being an uncompensated trustee of Brigham Health and the Brigham and Women's Physicians Organization. Dr Schulman reported being a board member and shareholder for Grid Therapeutics and Reserve Therapeutics; being a managing member and shareholder for Faculty Connection LLC; being a shareholder for Prealize; being an investor in Altitude Ventures Inc and Excelerate Health Ventures; being a consultant for Novartis, Cytokinetics, Business Roundtable, Motley Rice LLC, and Frazier Healthcare Partners; being a speaker for Health Quest LLC and ISMIE Inc; being president of Business School Alliance for Health Management; being senior associate editor of Health Services Research ; and being on the advisory board of Civica RX. No other disclosures were reported.

Additional Contributions: Sam Choi, BS, and Josh Holzbauer, BA, Epic Systems, provided assistance with the data. These individuals received no additional compensation, outside of their usual salary, for their contributions.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

This paper is in the following e-collection/theme issue:

Published on 25.4.2024 in Vol 12 (2024)

Semantic Interoperability of Electronic Health Records: Systematic Review of Alternative Approaches for Enhancing Patient Information Availability

Authors of this article:

• Sari Palojoki 1, * , PhD ; 
• Lasse Lehtonen 2, * , MD, PhD ; 
• Riikka Vuokko 1, * , PhD

1 Department of Steering of Healthcare and Social Welfare, Ministry of Social Affairs and Health, , Helsinki, , Finland

2 Diagnostic Center, Helsinki University Hospital District, , Helsinki, , Finland

*all authors contributed equally

Corresponding Author:

Sari Palojoki, PhD

Background: Semantic interoperability facilitates the exchange of and access to health data that are being documented in electronic health records (EHRs) with various semantic features. The main goals of semantic interoperability development entail patient data availability and use in diverse EHRs without a loss of meaning. Internationally, current initiatives aim to enhance semantic development of EHR data and, consequently, the availability of patient data. Interoperability between health information systems is among the core goals of the European Health Data Space regulation proposal and the World Health Organization’s Global Strategy on Digital Health 2020-2025 .

Objective: To achieve integrated health data ecosystems, stakeholders need to overcome challenges of implementing semantic interoperability elements. To research the available scientific evidence on semantic interoperability development, we defined the following research questions: What are the key elements of and approaches for building semantic interoperability integrated in EHRs? What kinds of goals are driving the development? and What kinds of clinical benefits are perceived following this development?

Methods: Our research questions focused on key aspects and approaches for semantic interoperability and on possible clinical and semantic benefits of these choices in the context of EHRs. Therefore, we performed a systematic literature review in PubMed by defining our study framework based on previous research.

Results: Our analysis consisted of 14 studies where data models, ontologies, terminologies, classifications, and standards were applied for building interoperability. All articles reported clinical benefits of the selected approach to enhancing semantic interoperability. We identified 3 main categories: increasing the availability of data for clinicians (n=6, 43%), increasing the quality of care (n=4, 29%), and enhancing clinical data use and reuse for varied purposes (n=4, 29%). Regarding semantic development goals, data harmonization and developing semantic interoperability between different EHRs was the largest category (n=8, 57%). Enhancing health data quality through standardization (n=5, 36%) and developing EHR-integrated tools based on interoperable data (n=1, 7%) were the other identified categories. The results were closely coupled with the need to build usable and computable data out of heterogeneous medical information that is accessible through various EHRs and databases (eg, registers).

Conclusions: When heading toward semantic harmonization of clinical data, more experiences and analyses are needed to assess how applicable the chosen solutions are for semantic interoperability of health care data. Instead of promoting a single approach, semantic interoperability should be assessed through several levels of semantic requirements A dual model or multimodel approach is possibly usable to address different semantic interoperability issues during development. The objectives of semantic interoperability are to be achieved in diffuse and disconnected clinical care environments. Therefore, approaches for enhancing clinical data availability should be well prepared, thought out, and justified to meet economically sustainable and long-term outcomes.

Introduction

Over the past 2 decades, there has been growing interest in digital technologies and eHealth integration into national health care systems to promote health [ 1 ]. The World Health Organization (WHO) has launched the Global Strategy on Digital Health 2020-2025 [ 2 ]. To implement digital health strategy objectives, a toolkit was set up to help countries to integrate eHealth into their health care systems [ 3 ]. The objectives of the WHO strategy include standards for interoperability. Another current large-scale international initiative is the European Health Data Space (EHDS) regulation proposal. EHDS is a health-specific ecosystem comprised of rules, common standards and practices, infrastructures, and a governance framework. It supports the use of health data for better health care delivery, research, innovation, and policy making. Moreover, it aims at empowering patients through increased digital access to and control of their personal health data [ 3 - 6 ].

Interoperability ensures health data availability and use. It is the ability of different organizations and professionals to interact and share information according to standards of data transfer and common protocols that support data exchange [ 4 - 8 ]. In clinical context, interoperable electronic health records (EHRs) help health care practitioners gather, store, and communicate essential health information reliably and securely across care settings. This aims to guarantee coordinated and patient-centered care while creating many efficiencies in the delivery of health care [ 9 ]. EHRs use health-related information pertinent to an individual patient, whereas registries are mainly focused on population management and are designed to obtain information on predefined health outcomes data and data for public health surveillance, for example. Although technological possibilities for using various types of data grow, new demands are placed on data quality and usability and, consequently, on interoperability [ 5 , 10 , 11 ].

Moreover, semantic interoperability enhances the unambiguous representation of clinical concepts, supported by the use of international standard reference systems and ontologies. Since there are different types of health information, such as data from EHRs, patient registries, genomics data, and data from health applications, the development of international data standardization, common guidelines, and recommendations are needed [ 4 - 8 ]. Without applying appropriate semantic standards, such as domain-relevant terminologies, interoperability will be limited. This may diminish the availability and potential value of data. The various parties involved have to address the importance of shared digital health standards and especially semantic interoperability features [ 12 - 15 ]. In the clinical context, interoperability is required to enhance the quality, efficiency, and effectiveness of the health care system by providing information in the appropriate format whenever and wherever it is needed by eliminating unnecessary replication [ 16 ].

Therefore, our study aims to provide readers with up-to-date information about the different types of approaches to resolve semantic interoperability in EHRs specifically and to summarize the benefits of these choices. We aimed to research the topic with an emphasis on patient data availability and use. Our research questions were as follows: What are the key elements of and approaches for building semantic interoperability integrated in EHRs? What kinds of goals are driving the development? and What kinds of clinical benefits are perceived following this development?

Methodological Framework

With our research questions as a starting point, we set out to perform a systematic literature review of semantic interoperability. Regarding different layers of interoperability, legal interoperability ensures overcoming potential barriers for data exchange. Interoperability agreements are made binding via international- or national-level legislation and via bilateral and multilateral agreements. Organizational interoperability defines, for example, business goals and processes. Semantic interoperability ensures that the precise meaning of exchanged information is understandable by any other application. It enables systems to combine received information with other information resources and process it in a meaningful manner. Technical interoperability covers various issues of linking computer systems and services, such as open interfaces, data integration, data presentation and exchange, accessibility, and security services [ 6 , 7 ].

For the study design, we first defined our core concepts to refine the literature search strategy. The scope of the review was semantic interoperability, that is, organizational, legal, and technical interoperability were excluded [ 7 ]. Semantic interoperability was apprehended based on the European Interoperability Framework (EIF) that provides a common set of principles and guidance for the design and development of interoperable digital services. In the EIF, semantic interoperability covers both semantic and syntactic aspects. The semantic aspect refers to the meaning of data elements and their relationships, whereas the syntactic aspect refers to the format of the information to be exchanged. With semantic interoperability, it is ensured that data can be shared in such a way that the meaning of data does not change [ 7 , 15 , 17 , 18 ]. There are also other models for analyzing interoperability layers [ 18 ]. For example, in comparison to the European approach [ 7 ], the Healthcare Information and Management Systems Society defines 4 levels of interoperability for health care technology: foundational, structural, semantic, and organizational [ 19 , 20 ]. Since the EIF is a well-established and largely applied framework [ 6 ], we chose the EIF definitions to primarily guide our review framework, as illustrated in Figure 1 . Our review deals with semantic interoperability, which is highlighted in gray in the figure. Thus, we did not analyze, for example, standards that are related to processes or information quality.

As shown in Figure 1 , processing, storing, and exchanging health care data in EHRs and between EHRs or other clinical applications is, for example, governed and regulated at the legal layer. To continue, processes and workflows regarding information exchange are arranged at the organizational interoperability layer and resolved in the technical layer, for example, according to the principles of data protection and information security. To illustrate the point, for example, the EHDS proposal suggests that compliance with essential requirements on interoperability and data security may be demonstrated by the manufacturers of EHR systems through the implementation of common specifications. To that end, implementation can be grounded on common specifications, such as data sets, coding systems, technical specifications, standards, and profiles for data exchange, as well as requirements and principles related to security, confidentiality, integrity, patient safety, and he protection of personal data and so on [ 6 ].

The semantic interoperability layer in Figure 1 covers various approaches to resolve interoperability issues, such as more established international or domain-specific health care classifications, clinical terminologies, and ontologies and applications of international standards for EHRs. In Figure 1 , we provided some examples to illustrate various semantic aspects, but this is not an exhaustive list. Similarly, for other interoperability levels, real-world examples were given. Based on the EIF, semantic interoperability also covers syntactic features, such as data format and, for example, structured data content. We identified these key features of semantic interoperability based on previous research [ 8 , 16 - 19 , 21 ]. In our framework, a data model is a generic concept that describes various applications of data models from a reference information model (RIM) to a clinical information model. Data models define structures and semantics for storing, exchanging, querying, and processing health care data. Clinical information models can be implemented in an EHR, for example, as archetypes and templates, whereas RIMs refer to standards-based approaches to enable health care documentation and messages, such as the Health Level 7 (HL7) RIM or the International Organization for Standards’ EN/ISO 13606 standard for EHR communication [ 19 , 22 ]. When designing EHRs, for semantic interoperability, a dual-level method can be applied to represent both information and knowledge levels of interoperability requirements, properties, and structures for data. This approach is used, for example, for representing the dual levels of knowledge by an archetype model and information structures by the chosen RIM [ 16 , 21 , 22 ].

Study Design

In the design of the review, we applied the Cochrane review protocol [ 23 ] to ensure the scientific reliability and validity of our review ( Checklist 1 ). The search strategy (see Textbox 1 ) was defined based on the framework for semantic interoperability presented in Figure 1 . We performed the search in the PubMed database in December 2022. To conduct a systematic literature review, PubMed is regarded as a comprehensive database [ 24 ]. Therefore, no further data searches were performed. We documented the search so that it can be reproduced (see Textbox 1 ). The search resulted in 131 unique articles. One article was removed because it did not include an abstract, and 1 was removed because it was not in English. In total, the authors screened 129 articles.

• Search terms: (((((EHR) OR (EMR)) OR (“Electronic Health Record”)) OR (“Electronic Medical Record”)) AND (((((“Semantic interoperability”) OR ((“data model”) AND (“Semantic interoperability”))) OR ((((“classification”) OR (ontology)) OR (terminology)) AND (“Semantic interoperability”))) OR (((“data content”) OR (“data format”)) AND (“Semantic interoperability”))) OR ((“Semantic interoperability”) AND (standard)))
• Filters used: abstract, full text, and English

The research team first screened all the remaining articles by title and abstract from January to March 2023. After the first test reading, the researchers discussed the inclusion and exclusion criteria and coherence of the understanding. Researchers were blinded and performed the analysis independently based on the inclusion and exclusion criteria and then compared the results. Selecting the same alternative created a match. Choosing a different alternative or failing to recognize the category at all was considered a nonmatch. In data-model cases, discussion was needed for alignment, but no complex situations developed. During the first screening, after discussion by the research team, 71 articles were excluded from the review for the following four reasons: (1) EHR was not a key factor but a contextual factor in the original research setting; (2) the original research did not focus on semantic interoperability but on another level of interoperability; (3) the original study did not entail practical implementation goals, but the focus was predominantly theoretical or methodological; and (4) the original research was not a research article but, for example, a poster. The remaining 58 articles were sought for retrieval. For 4 articles, the full text was not available. To evaluate eligibility, full texts of the 54 remaining articles were read by the research team. At this point, 17 articles were excluded because the original research was out of scope, that is, semantic interoperability was not developed with practical goals for advancing the availability and use of interoperable patient data. In addition, 15 articles were excluded as the semantic interoperability case did not involve EHR use or development, 3 articles were excluded due to the absence of semantic interoperability altogether, and 5 more were excluded because they were not research articles. After agreeing upon the final exclusion within our research team, 14 articles were analyzed for semantic interoperability in EHRs. Our final inclusion criteria were grounded on our research questions: the research article should explore an EHR use or development case with the focus on semantic interoperability of clinical data. Preferably, the case would document the stage of interoperability development or use, expected or realized clinical benefits, semantic development goals, and aspects of interoperability to be implemented, as well as the method of application.

The extraction and documentation of the information from the research articles was informed by our research questions, the review framework ( Figure 1 ), and by previous research literature. At this stage, previous reviews [ 16 - 19 , 21 ] were especially used in compiling our study framework (see Figure 1 ). Based on our framework, the documentation of the review analysis included elements of interoperability already identified in the search strategy. Consequently, it was necessary to investigate which documented elements are typically examined in research and with what methods they are applied in EHRs [ 8 , 16 - 18 ]. Moreover, we deemed it important to document how semantic interoperability is described in the clinical use context, consisting of various EHRs, clinical applications, registers, and other data resources. Lastly, the information documentation had to include not only the semantic implementation, use goals, or intended benefits but also practical goals or benefits in the clinical use context (see Figure 2 ). We defined and agreed upon the information documentation categories within our research team to conduct a well-grounded analysis for the review.

Contextual Results

We identified 14 articles describing semantic interoperability in EHRs, published between 2011 and 2022, as shown in Multimedia Appendix 1 [ 24 - 37 ]. The results revealed predominantly European advances in the study topic. Most (n=11, 79%) of the research cases were affiliated with different types of institutions in the European Union member states or in European Economic Area countries. One of the publications was coproduced by authors from Columbia and Germany, and the authors of another article represented organizations from the United States, South Korea, China, and Egypt. We decided not to limit the included studies to a certain geographical area but to analyze any potential use case for enabling the interoperability of EHRs.

Two of the reported research cases focused on patients with heart failure [ 24 , 30 ], 1 focused on patients with neurosurgical tumors [ 28 ], 2 focused on patients in cancer care [ 33 , 37 ], and 1 focused on patients with type 1 diabetes [ 31 ]. Other clinical use domains described were a prehospital unit at the site of an incident or during transfer to the emergency department and a hospital emergency care unit where prehospital patient documentation must be reassessed. A primary care–related case documented experimental laboratory test results of a population of 230,000 patients. Examples of older adult medication care and multiprofessional health care were part of our sample. Two articles described multipurpose clinical use of physician’s notes and tertiary care data. One article concerned the domain of clinical research using data from different EHR systems, and another described semantic aspects for retrieval of medication, laboratory test, and diagnosis-related data.

Although all studies concerned data from the EHRs, some studies included more detailed descriptions on data sources. Heart failure summaries containing clinical situation data and diagnoses (severity and certainty), as well as heart failure summaries covering clinical situations and symptoms data (a symptom’s presence, absence, and severity), were represented in the sample. One study regarded clinical history, observations, and findings during tumor control. One study focused on histories of patients with diabetes and diabetes care plans (eg, insulin regimen, diet, and exercise plans) and patients’ self-monitoring of vital signs, and 1 study used self-monitoring data on daily activities, side effects, and patient-reported outcomes. One article reported results around diagnosis and laboratory data; 1 article reported on medication, laboratory, and diagnosis data; and another article reported on neurosurgical imaging and laboratory data, although the starting point in the paper was diagnosis and medication data. The remaining 4 studies generally applied prehospital patient case data, emergency care–related EHR data, laboratory data, and diagnosis data.

Interoperability Results

In our sample, data were transferred and shared between different EHRs and clinical applications with no loss of data or changes in their meaning ( Multimedia Appendix 2 [ 24 - 37 ]). Half (7/14, 50%) of the studies were aimed at developing semantic interoperability between different EHRs or within different EHR modules, such as a medication module in 1 EHR system. One case concentrated specifically on an EHR and a clinical application. Two articles reported results about the interoperability between EHRs and personal health records. Interoperability with the laboratory system and the EHR was the focus of study in 2 cases. Two studies reported advances in interoperability development between EHR and clinical research resources or a clinical registry. Regarding the state of development, the largest number of studies were categorized as “in development” (n=5, 36%) and “in use” (n=6, 43%). Two articles reported results regarding the testing phase, and the remaining study was in an implementation stage.

All articles reported clinical benefits of the selected approach to enhancing semantic interoperability. We identified 3 main categories of clinical benefits within the articles: increasing the availability of data for clinicians (n=6, 43%), increasing the quality of care (n=4, 29%), and enhancing clinical data use and reuse for varied purposes (n=4, 29%). The first category describes use cases where patient care would benefit from better availability of data. This was to be achieved by enhancing interoperable data and its transfer from clinical applications (eg, a laboratory system) to a central EHR and between EHRs to increase accessible data for making informed clinical decisions. These advances were in implementation to enhance the quality and effectiveness of care. Moreover, developing better access to health data and providing homogeneous access to heterogeneous data sets may facilitate resource effectiveness; patient management; and overall, the optimization of data for different purposes. The second category included benefits for the quality of care. The category had largely been implemented in EHRs already. Benefits entail better resource effectiveness and optimization of patient care planning and monitoring and better patient management, as well as the continuity of care based on interoperable and accessible health data that facilitates informed decision-making by clinicians. One of these cases documented improved patient safety based on interoperable health data across EHRs. The third category, enhancing clinical data use and reuse, included 2 use cases where data were used across EHRs. One use case described data transfer between an EHR and a national oncology registry, where interoperability enhanced data integration and redesign of the systems in use. The other 2 cases documented the evidence of data use, where better availability of data provided a means for developing new EHR integrated tools, such as clinical alerts, dynamic patient lists, and clinical follow-up dashboards. In summary, semantic development goals emphasized better access to data regardless of underlying standards and data structures or EHRs in use. The underlying assumption is that with better access to data, it is possible to facilitate better communication between professionals and the continuity of care.

In our analysis, semantic development goals were divided in 3 categories. All of these were closely coupled with the need to build usable and available data based on heterogeneous medical information that is accessible through various EHRs and databases, such as registers. Data harmonization and developing semantic interoperability between different EHRs or between EHRs and clinical application was the largest category (n=8, 57%). Enhancing health data quality through standardization (n=5, 36%) and developing EHR-integrated tools based on interoperable data (n=1, 7%) were the other identified categories. Semantic development goals were described as harmonizing data or otherwise processing semantically equivalent data across different medical domains and among different clinical data sources including EHRs and applications, thus facilitating clinicians’ availability of health data. One case included the formalization of data with a semantic converter to increase the interoperability of data. In 2 research cases, the main semantic development goals concentrated on advancing the interoperability of EHR data and patient-generated data or sensor data to monitor the situation of patients who are chronically ill. Regarding data standardization, 1 research case reported increasing data quality as the semantic interoperability development goal. Standardized data content decreased information overload of clinicians. Through data standardization, it was possible to increase conceptualization and, thus, access to data within an EHR regardless of the underlying standards and data structures, by providing a semantic standardized layer to facilitate clinicians’ data use, or by otherwise ensuring complete and coherent information with no errors due to the loss of meaning or context. One of these research cases documented improvements for system-level efficiency for EHR functions and integrated tools based on advances of semantic interoperability.

Features of semantic interoperability were described in all 14 articles. Most (9/14, 64%) of the analyzed cases incorporated 1 or more semantic aspects. In more detail, the aspects of semantic interoperability were described as follows: ontologies were the chosen aspect in 3 research cases, terminologies in 6 cases, classifications in 4 cases, various clinical documentation standards in 8 cases, and different data models in 10 cases. In this categorization, data model refers to various semantic model layers, namely, the use of various types of data models that include, for example, data content specifications, RIMs, and clinical information models depending on the development context. A dual model was discussed in 2 of the cases for the application of data models.

Closely related to the aspects of interoperability, several interoperability standard solutions were named. Named ontology solutions included a top-domain ontology for the life sciences (BioTopLite) in 2 cases, a HL7 Fast Health Interoperability Resources (FHIR) and semantic sensor network–based type 1 diabetes ontology for type 1 diabetes data, and a system of several ontologies to be used for building EHR interoperability. Systematized Nomenclature of Medicine Clinical Terminology was the common terminology application in 7 cases, whereas classification systems were applied in more heterogeneous ways. The following international classifications were named: International Classification of Diseases, Tenth Revision ; International Classification of Diseases, Ninth Revision, Clinical Modification ; The Anatomical Therapeutic Chemical Classification System; and Logical Observation Identifiers Names and Codes. One article documented national classification use. Applied health care–specific standards included the open standard specification in health informatics (openEHR; n=6), Digital Imaging and Communications in Medicine (n=1), HL7 FHIR (n=5), and the HL7 Clinical Document Architecture (n=2). Regarding data models or reference information models, several types were applied for distinct use environments. These included the Observational Medical Outcomes Partnership common data model, an EHR-specific data component model, the i2b2 common data model for data warehouse development, the HL7 FHIR RIM, and the EN/ISO 13606 standard–based model. Moreover, 1 case reported using openEHR as a data model reference.

The method for applying an interoperability framework or approach is related to the overall design of the data use purposes and the needs driving the semantic development. The chosen methodology for semantic development was based on ontology development or the application of an ontology framework in 4 research cases, data model–based development in 5 cases, archetype development in 1 case, and clinical data warehouse development to enhance access and processing of data in 1 case. In data model–based approaches, use cases document a method’s capability in separating different semantic levels of development, that is, system level, application level, clinical user interface level, or patient information level. The reusability of data model–based semantic approaches and related methods were assessed for resource savings in time and cost in development projects and, thus, to justify the choice of the approach. For example, clinical knowledge model–based development may allow recycling archetypes that further promote semantic interoperability.

Principal Findings

Our results are related to the main goals of semantic interoperability development, such as enabling patient data use regardless of which EHR the data originated from and by which terminologies, classifications, or other semantic features they are supported [ 16 - 19 , 21 ]. Regarding key elements of semantic interoperability, of the documented terminologies, Systematized Nomenclature of Medicine Clinical Terminology seemed to prevail as the dominant choice for clinical terminology [ 24 - 30 ]. For international classifications that are typically integrated into EHRs, a selection of well-established classifications was documented [ 25 , 26 , 31 , 32 ]. Likewise, several health care specific standards [ 24 - 26 , 28 , 31 , 33 ], ontologies [ 21 , 24 , 32 , 33 ], and data models [ 25 , 27 , 28 , 30 - 36 ] were presented, albeit in a relatively small sample in this study. One possible factor affecting the selection of interoperability features such as international standards may be open availability and the level of cost of the standard-specific resources and their deployment. Consequently, shared implementation experiences and recommendations from previous projects or from collaboration in international communities may promote and facilitate decision-making concerning future implementations.

Our review illustrates several approaches for building sematic interoperability. For ontologies and data models, based on the review, several layers may be deployed to address semantic interoperability development needs. For ontologies, deploying a system of ontologies seeks to bridge, for example, domain-specific ontologies and application-specific ontologies. In our sample, a case with a data model–based development approach enhanced the communication of clinical information with the application. The application was used by the patients in self-monitoring, and the EHR served as a clinical data repository to avoid the loss of meaningful information. In general, when applying data model–based approaches, a dual model or multimodel approach may be needed to address different semantic interoperability issues during development—from the clinician as an EHR user to the system transaction level.

Our review highlights several clinical benefits of semantic interoperability. Primarily semantic interoperability fulfills the need to support the implementation of applications that enhance the continuity of care and ensure access to safe and high-quality health care. The reported clinical benefits of developing semantic interoperability reflect well common international goals [ 2 , 3 , 5 ]. The results in our sample show that an evident goal driving the development in these studies is the following assumption: through increased access to patient information, better quality and outcomes in care can be achieved [ 24 , 26 , 27 , 33 , 37 ]. Better communication based on easily accessible data across EHRs is facilitated not only between clinicians but also between professionals and patients [ 28 , 34 , 35 ]. Further advances are related to efficiency and subsequent economic factors, for example, reducing the clinicians’ workload for documenting and evaluating extensive patient data, to avoid information overload and support multiprofessional care [ 26 , 31 - 33 , 35 ]. In addition, interoperable patient data provide opportunities for a wide range of EHR-related clinical development, for example, regarding decision-making support, other EHR integrated tools, clinical research, or other types of secondary use [ 25 , 28 - 31 , 33 , 36 ]. Essentially, the interoperability cases in our review demonstrated a well-documented selection of development goals in EHRs, including considerations of patient-generated, self-monitoring data and related interoperability features.

Finally, when reflecting on the goal-related semantic interoperability results, there is evidently not one universal approach available to tackle all interoperability-related needs and challenges. One reason for this is that interoperability is to be achieved in diffuse and disconnected clinical care settings and in registry data use across borders. However, regulations and international recommendations can support the choosing of common tools and standards for building interoperability for patient data generated in various EHRs and clinical applications. This may be the strongest selling point for evolving international frameworks, such as the EHDS regulation proposal. If adopted, unified toolkits of the most crucial means can be achieved for building international eHealth interoperability. Through these mechanisms, common solutions and standards can be agreed upon to remedy existing inconsistencies and avoid possible future imparities that hinder the realization of the common goals. It is noteworthy that all member states have steps to take to meet the international requirements with a country-specific road map to achieve the common goal [ 3 , 5 ]. Moreover, it would require cooperation to align on which level of interoperability should be reached when the operating environment consists of a diverse set of clinical practices and related data needs, such as between public and private care or between primary and specialized care. Additionally, it may be worthwhile to consider whether instead of promoting a single approach, semantic interoperability requirements should be assessed through several levels of semantic requirements, such as standards, data models, classifications, and terminologies. Moreover, developing the necessary skills and increasing capabilities is an essential component of this development.

Specifically, regarding European development, one of the main goals is to support the use of health data for better health care delivery and better research. The comprehensive and timely availability of EHR data is known to improve the quality of care and patient safety [ 26 , 38 ]. Concurrently, the lack of not only technical or organizational but also semantic interoperability has been recognized as one of the barriers for the cross-border exchange of health data [ 2 - 8 ]. Therefore, commonly recognized interoperability approaches and standards for the harmonization of semantic interoperability are needed.

Limitations

Our goal was to ensure that we did not overlook any important studies and to minimize any potential biases by conducting a thorough and comprehensive search of the available literature. However, it is worth noting that our search was limited to a single database, PubMed. Nevertheless, recent literature suggests that PubMed can serve as a primary search tool. It possesses the necessary capabilities for systematic reviews, including the ability to formulate and interpret queries accurately, as well as ensuring search reproducibility. It is important to acknowledge that even a well-performing system such as PubMed might not always yield the desired results in different scenarios [ 23 ]. Our data set was limited by a small sample size of 14 articles. Therefore, findings can only be regarded as descriptive in nature. Relatively large heterogeneity in study environments and selected research approaches limit us from drawing strong conclusions. Despite these limitations, this review demonstrates potentially feasible approaches for promoting semantic interoperability toward harmonized approaches. Additional real-world studies accounting for semantic interoperability are needed to reinforce understanding of the most promising, scalable examples such as international reference models (eg, HL7 RIM). Moreover, it was challenging to determine the “development status” category for certain studies. This was due to varying levels of details in the study reports, where some of the studies provided a wealth of detail, whereas some were more restricted in their scope.

Suggestions for Future Research

Future research directions are 2-fold from the current development perspective. First, evidence-based recommendations on semantic interoperability features, for example, data models and terminologies, are needed. Initially, the applicability of international data models and standards such as HL7 V2 might be evaluated. Second, more experiences of interoperability development should be reported in the peer-reviewed research literature to contribute evidence around successful and not so successful experiences instead of leaning solely on individual expert opinions. Presumably, due to the evolving implementation status of semantic interoperability cases illustrated in the research literature, systematic research–based evaluation of benefits and outcomes is still scarce.

Conclusions

We conclude that based on our review, the research literature highlights valuable aspects in promoting semantic interoperability in terms of the efficiency and feasibility of solutions integrated in EHRs and possibly for enhancing care. However, when heading toward semantic harmonization, more data, pilot experiences, and analyses are needed to assess how applicable the chosen specific solutions are for the standardization and semantic interoperability of patient data. Instead of promoting a single approach, semantic interoperability could be assessed through several levels of semantic approaches. A dual model or multimodel approach is usable to address different semantic interoperability issues during development—from the clinician as an EHR user to the system transaction level. Since interoperability is being implemented in complex and disconnected clinical care environments, choices should be well prepared and justified to meet sustainable and long-term outcomes. From that point of view, it is possible to outline future directions in selecting semantic interoperability approaches for the realization of the international patient data–related goals.

Acknowledgments

The study was supported by Finnish governmental study grant TYH2021319.

Conflicts of Interest

None declared.

Summary of study and sample characteristics.

Summary of results on semantic interoperability in electronic health records.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

• Iyamu I, Gómez-Ramírez O, Xu AXT, et al. Defining the scope of digital public health and its implications for policy, practice, and research: protocol for a scoping review. JMIR Res Protoc. Jun 30, 2021;10(6):e27686. [ CrossRef ] [ Medline ]
• Global strategy on digital health 2020-2025. World Health Organization; 2021. URL: https://www.who.int/docs/default-source/documents/gs4dhdaa2a9f352b0445bafbc79ca799dce4d.pdf [Accessed 2023-09-25]
• Godinho MA, Ansari S, Guo GN, Liaw ST. Toolkits for implementing and evaluating digital health: a systematic review of rigor and reporting. J Am Med Inform Assoc. Jun 12, 2021;28(6):1298-1307. [ CrossRef ] [ Medline ]
• Abboud L, Cosgrove S, Kesisoglou I, et al. TEHDAS Deliverable 4.1 Country factsheets: Mapping health data management systems through country visits: development, needs and expectations of the EHDS. TEHDAS Consortium Partners; Apr 28, 2023. URL: https:/​/tehdas.​eu/​app/​uploads/​2023/​04/​tehdas-mapping-health-data-management-systems-through-country-visits.​pdf [Accessed 2024-03-26]
• Hussein R, Scherdel L, Nicolet F, Martin-Sanchez F. Towards the European Health Data Space (EHDS) ecosystem: a survey research on future health data scenarios. Int J Med Inform. Feb 2023;170:104949. [ CrossRef ] [ Medline ]
• Proposal for a Regulation of the European Parliament and of the Council on the European Health Data Space: COM/2022/197 final. European Union. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52022PC0197 [Accessed 2023-09-25]
• Kouroubali A, Katehakis DG. The new European interoperability framework as a facilitator of digital transformation for citizen empowerment. J Biomed Inform. Jun 2019;94:103166. [ CrossRef ] [ Medline ]
• Stellmach C, Muzoora MR, Thun S. Digitalization of health data: interoperability of the proposed European Health Data Space. Stud Health Technol Inform. Aug 31, 2022;298:132-136. [ CrossRef ] [ Medline ]
• Gottumukkala M. Development, and evaluation of an automated solution for electronic information exchange between acute and long-term postacute care facilities: design science research. JMIR Form Res. Feb 17, 2023;7:e43758. [ CrossRef ] [ Medline ]
• Carlson J, Laryea J. Electronic health record-based registries: clinical research using registries in colon and rectal surgery. Clin Colon Rectal Surg. Jan 2019;32(1):82-90. [ CrossRef ] [ Medline ]
• Hohman KH, Martinez AK, Klompas M, et al. Leveraging electronic health record data for timely chronic disease surveillance: the multi-state EHR-based network for disease surveillance. J Public Health Manag Pract. 2023;29(2):162-173. [ CrossRef ] [ Medline ]
• 2015 Edition health information technology (health IT) certification criteria, 2015 Edition base electronic health record (EHR) definition, and ONC health IT certification program modifications. Federal Register. Oct 16, 2015. URL: https:/​/www.​federalregister.gov/​documents/​2015/​10/​16/​2015-25597/​2015-edition-health-information-technology-health-it-certification-criteria-2015-edition-base [Accessed 2023-02-21]
• Kush RD, Warzel D, Kush MA, et al. FAIR data sharing: the roles of common data elements and harmonization. J Biomed Inform. Jul 2020;107:103421. [ CrossRef ] [ Medline ]
• Horgan D, Hajduch M, Vrana M, et al. European Health Data Space-an opportunity now to grasp the future of data-driven healthcare. Healthcare (Basel). Aug 26, 2022;10(9):1629. [ CrossRef ] [ Medline ]
• Palojoki S, Vakkuri A, Vuokko R. The European cross-border health data exchange: focus on clinically relevant data. Stud Health Technol Inform. May 27, 2021;281:442-446. [ CrossRef ] [ Medline ]
• Gamal A, Barakat S, Rezk A. Standardized electronic health record data modeling and persistence: a comparative review. J Biomed Inform. Feb 2021;114:103670. [ CrossRef ] [ Medline ]
• Lee AR, Kim IK, Lee E. Developing a transnational health record framework with level-specific interoperability guidelines based on a related literature review. Healthcare (Basel). Jan 13, 2021;9(1):67. [ CrossRef ] [ Medline ]
• de Mello BH, Rigo SJ, da Costa CA, et al. Semantic interoperability in health records standards: a systematic literature review. Health Technol (Berl). 2022;12(2):255-272. [ CrossRef ] [ Medline ]
• Hwang KH, Chung KI, Chung MA, Choi D. Review of semantically interoperable electronic health records for ubiquitous healthcare. Healthc Inform Res. Mar 2010;16(1):1-5. [ CrossRef ] [ Medline ]
• Interoperability in healthcare. Healthcare Information and Management Systems Society (HIMSS). URL: https://www.himss.org/resources/interoperability-healthcare [Accessed 2023-12-21]
• Moreno-Conde A, Moner D, Cruz WD, et al. Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis. J Am Med Inform Assoc. Jul 2015;22(4):925-934. [ CrossRef ] [ Medline ]
• Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane; 2023. URL: http://www.training.cochrane.org/handbook [Accessed 2023-01-27]
• Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of Google scholar, PubMed, and 26 other resources. Res Synth Methods. Mar 2020;11(2):181-217. [ CrossRef ] [ Medline ]
• Martínez-Costa C, Schulz S. Ontology content patterns as bridge for the semantic representation of clinical information. Appl Clin Inform. Jul 23, 2014;5(3):660-669. [ CrossRef ] [ Medline ]
• Sun H, Depraetere K, de Roo J, et al. Semantic processing of EHR data for clinical research. J Biomed Inform. Dec 2015;58:247-259. [ CrossRef ] [ Medline ]
• Andersen SNL, Brandsborg CM, Pape-Haugaard L. Use of semantic interoperability to improve the urgent continuity of care in Danish ERs. Stud Health Technol Inform. May 27, 2021;281:203-207. [ CrossRef ] [ Medline ]
• Martínez-Costa C, Cornet R, Karlsson D, Schulz S, Kalra D. Semantic enrichment of clinical models towards semantic interoperability. the heart failure summary use case. J Am Med Inform Assoc. May 2015;22(3):565-576. [ CrossRef ] [ Medline ]
• Frid S, Fuentes Expósito MA, Grau-Corral I, et al. Successful integration of EN/ISO 13606-standardized extracts from a patient mobile app into an electronic health record: description of a methodology. JMIR Med Inform. Oct 12, 2022;10(10):e40344. [ CrossRef ] [ Medline ]
• Højen AR, Brønnum D, Gøeg KR, Elberg PB. Applying the SNOMED CT concept model to represent value sets for head and neck cancer documentation. Stud Health Technol Inform. 2016;228:436-440. [ Medline ]
• Pedrera M, Garcia N, Blanco A, et al. Use of EHRs in a tertiary hospital during COVID-19 pandemic: a multi-purpose approach based on standards. Stud Health Technol Inform. May 27, 2021;281:28-32. [ CrossRef ] [ Medline ]
• Boussadi A, Zapletal E. A Fast Healthcare Interoperability Resources (FHIR) layer implemented over i2b2. BMC Med Inform Decis Mak. Aug 14, 2017;17(1):120. [ CrossRef ] [ Medline ]
• González C, Blobel B, López DM. Ontology-based framework for electronic health records interoperability. Stud Health Technol Inform. 2011;169:694-698. [ CrossRef ] [ Medline ]
• Kropf S, Chalopin C, Lindner D, Denecke K. Domain modeling and application development of an archetype- and XML-based EHRs. practical experiences and lessons learnt. Appl Clin Inform. Jul 28, 2017;8(2):660-679. [ CrossRef ] [ Medline ]
• Marco-Ruiz L, Moner D, Maldonado JA, Kolstrup N, Bellika JG. Archetype-based data warehouse environment to enable the reuse of electronic health record data. Int J Med Inform. Sep 2015;84(9):702-714. [ CrossRef ] [ Medline ]
• El-Sappagh S, Ali F, Hendawi A, Jang JH, Kwak KS. A mobile health monitoring-and-treatment system based on integration of the SSN sensor ontology and the Hl7 FHIR standard. BMC Med Inform Decis Mak. May 10, 2019;19(1):97. [ CrossRef ] [ Medline ]
• Yang L, Huang X, Li J. Discovering clinical information models online to promote interoperability of electronic health records: a feasibility study of OpenEHR. J Med Internet Res. May 28, 2019;21(5):e13504. [ CrossRef ] [ Medline ]
• Terner A, Lindstedt H, Sonnander K. Predefined headings in a multiprofessional electronic health record system. J Am Med Inform Assoc. 2012;19(6):1032-1038. [ CrossRef ] [ Medline ]
• Vuokko R, Vakkuri A, Palojoki S. Systematized Nomenclature of Medicine-Clinical Terminology (SNOMED CT) clinical use cases in the context of electronic health record systems: systematic literature review. JMIR Med Inform. Feb 6, 2023;11:e43750. [ CrossRef ] [ Medline ]

Abbreviations

Edited by Christian Lovis; submitted 10.10.23; peer-reviewed by Hannes Ulrich, Xiaoshuo Huang; final revised version received 21.02.24; accepted 24.02.24; published 25.04.24.

© Sari Palojoki, Lasse Lehtonen, Riikka Vuokko. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 25.4.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/ , as well as this copyright and license information must be included.

• Open access
• Published: 26 April 2024

Clinician and staff experiences with frustrated patients during an electronic health record transition: a qualitative case study

• Sherry L. Ball 1 ,
• Bo Kim 2 , 3 ,
• Sarah L. Cutrona 4 , 5 ,
• Brianne K. Molloy-Paolillo 4 ,
• Ellen Ahlness 6 ,
• Megan Moldestad 6 ,
• George Sayre 6 , 7 &
• Seppo T. Rinne 2 , 8  

BMC Health Services Research volume  24 , Article number:  535 ( 2024 ) Cite this article

156 Accesses

Metrics details

Electronic health record (EHR) transitions are known to be highly disruptive, can drastically impact clinician and staff experiences, and may influence patients’ experiences using the electronic patient portal. Clinicians and staff can gain insights into patient experiences and be influenced by what they see and hear from patients. Through the lens of an emergency preparedness framework, we examined clinician and staff reactions to and perceptions of their patients’ experiences with the portal during an EHR transition at the Department of Veterans Affairs (VA).

This qualitative case study was situated within a larger multi-methods evaluation of the EHR transition. We conducted a total of 122 interviews with 30 clinicians and staff across disciplines at the initial VA EHR transition site before, immediately after, and up to 12 months after go-live (September 2020-November 2021). Interview transcripts were coded using a priori and emergent codes. The coded text segments relevant to patient experience and clinician interactions with patients were extracted and analyzed to identify themes. For each theme, recommendations were defined based on each stage of an emergency preparedness framework (mitigate, prepare, respond, recover).

In post-go-live interviews participants expressed concerns about the reliability of communicating with their patients via secure messaging within the new EHR portal. Participants felt ill-equipped to field patients’ questions and frustrations navigating the new portal. Participants learned that patients experienced difficulties learning to use and accessing the portal; when unsuccessful, some had difficulties obtaining medication refills via the portal and used the call center as an alternative. However, long telephone wait times provoked patients to walk into the clinic for care, often frustrated and without an appointment. Patients needing increased in-person attention heightened participants’ daily workload and their concern for patients’ well-being. Recommendations for each theme fit within a stage of the emergency preparedness framework.

Conclusions

Application of an emergency preparedness framework to EHR transitions could help address the concerns raised by the participants, (1) mitigating disruptions by identifying at-risk patients before the transition, (2) preparing end-users by disseminating patient-centered informational resources, (3) responding by building capacity for disrupted services, and (4) recovering by monitoring integrity of the new portal function.

Peer Review reports

Electronic health record (EHR) transitions present significant challenges for healthcare clinicians and staff. These transitions require adjustments in care delivery and may threaten care quality and value. It is critical that healthcare organizations undergoing these changes learn from others who have undergone similar transitions [ 1 , 2 ]. However, the current literature lacks adequate guidance on navigating EHR transitions, especially as they relate to how clinicians and staff interact with patients [ 3 ].

Embedded within EHRs, patient portals facilitate complete, accurate, timely, and unambiguous exchange of information between patients and healthcare workers [ 4 , 5 ]. These portals have become indispensable for completing routine out-of-office-visit tasks, such as medication refills, communicating laboratory results, and addressing patient questions [ 6 ]. In 2003, the VA launched their version of a patient portal, myHealtheVet [ 7 ] and by 2017 69% of Veterans enrolled in healthcare at the VA had registered to access the patient portal [ 8 ]. Similar to other electronic portals, this system allows Veterans to review test results, see upcoming appointments, and communicate privately and securely with their healthcare providers.

EHR transitions can introduce disruptions to patient portal communication that may compromise portal reliability, impacting patient and clinician satisfaction, patients’ active involvement in self-management, and ultimately health outcomes [ 9 ]. During an EHR transition, patients can expect reductions in access to care even when clinician capacity and IT support are increased. Patients will likely need for more assistance navigating the patient portal including and using the portal to communicate with their providers [ 10 ]. Staff must be prepared and understand how the changes in the EHR will affect patients and safeguards must be in place to monitor systems for potential risks to patient safety. Building the capacity to respond to emerging system glitches and identified changes must be included in any transition plan. Although portal disruptions are likely to occur when a new EHR is implemented, we know little about how these disruptions impact healthcare workers’ interactions and care delivery to patients [ 11 , 12 ].

Due to an urgency to raise awareness and promote resolution of these patient portal issues,, we utilized existing data from the first EHR transition site for the Department of Veterans Affairs (VA)’s enterprise-wide transition. We focused on end users’ responses to the question “How Veterans were affected by the transition?”. We used qualitative methods to begin to understand how provider and patient interactions were affected during and by the EHR transition. We explored the impact of the EHR transition on patients through healthcare workers’ vicarious and direct experiences with patients. Due to the high level of disruption in care delivery we draw on insights from an emergency preparedness framework [ 13 ] to generate a set of recommendations to improve healthcare workers’ experiences during EHR transitions. The emergency preparedness framework includes 4 phases of an iterative cycle that include: (1) building capacity to mitigate issues, (2) preparing for the inevitable onset of issues, (3) responding to issues as they emerge, and (4) strategies to recover from any damage incurred.

In early 2020, the VA embarked on an EHR transition from a homegrown, legacy EHR system, developed by VA clinicians and used since the 1990s, to a new commercial system by the Oracle-Cerner Corporation. The primary objectives of this transition were to standardize care and improve interoperability between VA Medical Centers nationwide and the Department of Defense (DoD). Spanning over a decade, this transition plan is scheduled to roll out to all VA medical centers and outpatient clinics.

In this manuscript, we present data from the Mann-Grandstaff VA Medical Center in Spokane, WA, VA’s first EHR transition site. The study uses qualitative methods with clinician and staff interviews as part of a larger multi-method evaluation of the EHR transition. Our overarching goal is to identify and share recommendations to improve VA’s EHR transition efforts; rather than be guided by a theoretical framework our study design including the interview guides [ 14 , 15 ] were based primarily on what was being experienced. An experienced team of ten qualitative methodologists and analysts conducted the study.

This evaluation was designated as non-research/quality improvement work by the VA Bedford Healthcare System Institutional Review Board deeming it exempt from needing an informed consent. Study materials, including interview guides with verbal consent procedures, were reviewed and approved by labor unions and by the VA Bedford Healthcare System Institutional Review Board; all methods were carried out in accordance with local and national VA guidelines and regulations.

Interview guides and an outline of the data collection plans were reviewed and approved by relevant national unions before beginning recruitment.

Recruitment

Recruitment began in July 2020, before the first site implemented the new EHR. Prior to collecting data, we met with site leadership to get buy-in and support for the study, understand local context, determine how the site was approaching the transition, and to obtain the names of clinicians and staff for potential interviews. All potential participants were invited by email to participate in a one-hour voluntary interview conducted on Microsoft Teams® about their experiences with this transition; we used snowball sampling during interviews to expand the pool of interviewees. Verbal permission for audio recording of interviews was obtained immediately prior to the interview. Interview participants were informed that they could skip any questions, pause or stop the recording, and stop the interview at any time and were invited to ask questions before beginning the interview.

Most participants were interviewed at multiple timepoints; these included pre-implementation interviews, brief check-ins, and post-implementation interviews (Table  1 ). At the end of the pre-implementation interview, participants were invited to participate in 3–4 additional, shorter (15–20 min), check-in interviews where information about any changes in the transition process, context, or experience could be discussed. Most initial interviewees, in addition to three new participants, participated in post-implementation interviews (35–60 min; approximately 2–3 months and 10–12 months after the implementation) to reflect on the entire transition process.

Data collection

Experienced qualitative interviewers included PhD trained qualitative methodologist and masters level qualitative analysts (JB, SB, AC, EK, MM, GS) conducted individual interviews with clinicians and staff, aligning to a semi-structured interview guide with follow-up probes using the participant’s words to elicit rich responses grounded in the data [ 16 ]. The guide was designed to inform ongoing efforts to improve the rollout of the new EHR. Six main categories were covered in our interview guides, including (1) attitudes toward the new software, (2) information communicated about the transition, (3) training and education, (4) resources, (5) prior experience with EHRs, and (6) prior experiences with EHR transitions. After piloting the interview guide with a clinician, initial interviews were completed between September and October 2020 and averaged  ∼  45 min in duration. Two-month and one-year post-implementation interview guides included an additional question, “Has the Cerner transition affected Vets?”; data presented here largely draw from responses to this question. Check-ins (October 2020– December 2020) took  ∼  15 min; two-month post-implementation interviews (December 2020– January 2021) and one-year post-implementation interviews (October 2020 - November 2021) took  ∼  45 min. Audio recordings of all interviews were professionally transcribed. To ensure consistency and relationship building, participants were scheduled with the same interviewer for the initial and subsequent interviews whenever feasible (i.e., check-ins and post-implementation interviews). Immediately following each interview, interviewers completed a debrief form where highlights and general reflections were noted.

Throughout the data collection process, interviewers met weekly with the entire qualitative team and the project principal investigators to discuss the recruitment process, interview guide development, and reflections on data collection. To provide timely feedback to leadership within the VA, a matrix analysis [ 17 ] was conducted concurrently with data collection using the following domains: training, roles, barriers, and facilitators. Based on these domains, the team developed categories and subcategories, which formed the foundation of an extensive codebook.

Data analysis

All interviewers also coded the data. We used inductive and deductive content analysis [ 18 ]. Interview transcripts were coded in ATLAS.ti qualitative data analysis software (version 9). A priori codes and categories (based on the overall larger project aims and interview guide questions) and emergent codes and categories were developed to capture concepts that did not fit existing codes or categories [ 18 ]. Codes related to patient experience and clinician interactions with patients were extracted and analyzed using qualitative content analysis to identify themes [ 18 ]. Themes were organized according to their fit within the discrete stages of an emergency preparedness framework to generate recommendations for future rollout. In total, we examined data from 111 interviews with 24 VA clinicians and staff (excluding the initial 11 stakeholder meetings (from the 122 total interviews) that were primarily for stakeholder engagement). We focused on participants’ responses related to their experiences interacting with patients during the EHR transition.

Exemplar quotes primarily came from participants’ responses to the question, “Has the Cerner transition affected Vets?” and addressed issues stemming from use of the patient portal. This included both clinicians’ direct experiences with the portal and indirect experiences when they heard from patients about disruptions when using the portal. We identified four themes related to clinicians’ and staff members’ reported experiences: (1) stress associated with the unreliability of routine portal functions and inaccurate migrated information; (2) concern about patients’ ability to learn to use a new portal (especially older patients and special populations); (3) frustration with apparent inadequate dissemination of patient informational materials along with their own lack of time and resources to educate patients on use of the new portal; and (4) burden of additional tasks on top of their daily workload when patients needed increased in-person attention due to issues with the portal.

Stress associated with the unreliability of routine portal functions and inaccurate migrated information

One participant described the portal changes as, “It’s our biggest stress, it’s the patients’ biggest stress… the vets are definitely frustrated; the clinicians are; so I would hope that would mean that behind the scenes somebody is working on it” (P5, check-in).

Participants expressed significant frustration when they encountered veterans who were suddenly unable to communicate with them using routine secure messaging. These experiences left them wondering whether messages sent to patients were received.

Those that use our secure messaging, which has now changed to My VA Health, or whatever it’s called, [have] difficulty navigating that. Some are able to get in and send the message. When we reply to them, they may or may not get the reply. Which I’ve actually asked one of our patients, ‘Did you get the reply that we took care of this?’ And he was like, ‘No, I did not (P11, 2-months post)

Participants learned that some patients were unable to send secure messages to their care team because the portal contained inaccurate or outdated appointment and primary site information.

I’ve heard people say that the appointments aren’t accurate in there… veterans who have said, ‘yeah, it shows I’m registered,’ and when they go into the new messaging system, it says they are part of a VA that they haven’t gone to in years, and that’s the only area they can message to, they can’t message to the [site] VA, even though that’s where they’ve actively being seen for a while now. (P20, 2-months post)

After the EHR transition, participants noted that obtaining medications through the portal, which was once a routine task, became unreliable. They expressed concern around patients’ ability to obtain their medications through the portal, primarily due to challenges with portal usability and incomplete migration of medication lists from the former to the new EHR.

I think it’s been negative, unfortunately. I try to stay optimistic when I talk to [patients], but they all seem to be all having continued difficulty with their medications, trying to properly reorder and get medications seems to still be a real hassle for them. (P17, one-year post) …the medications, their med list just didn’t transfer over into that list [preventing their ability to refill their medications]. (P13, 2-months post)

Concern about patients’ ability to learn to use a new portal

Clinicians and staff expressed concerns around veterans’ ability to access, learn, and navigate a new portal system. Clinicians noted that even veterans who were adept at using the prior electronic portal or other technologies also faced difficulties using the new portal.

They can’t figure out [the new portal], 99% of them that used to use our [old] portal, the electronic secure messaging or emailing between the team, they just can’t use [the new one]. It’s not functioning. (P13, one-year post) Apparently, there’s a link they have to click on to make the new format work for them, and that’s been confusing for them. But I still am having a lot of them tell me, I had somebody recently, who’s very tech savvy, and he couldn’t figure it out, just how to message us. I know they’re still really struggling with that. (P5, 2-months post) And it does seem like the My Vet [my VA Health, new portal], that used to be MyHealtheVet [prior portal], logging on and getting onto that still remains really challenging for a large number of veterans. Like they’re still just unable to do it. So, I do think that, I mean I want to say that there’s positive things, but really, I struggle (P17, one-year post)

Participants recognized difficulties with the new system and expressed empathy for the veterans struggling to access the portal.

I think that a lot of us, individually, that work here, I think we have more compassion for our veterans, because they’re coming in and they can’t even get onto their portal website. (P24, one-year post)

Participants acknowledged that learning a new system may be especially difficult for older veterans or those with less technology experience.

But, you know, veterans, the general population of them are older, in general. So, their technologic skills are limited, and they got used to a system and now they have to change to a new one. (P13, 2-months post) So, for our more elderly veterans who barely turn on the computer, they’re not getting to this new portal. (P8, check in) And you know, I do keep in mind that this is a group of people who aren’t always technologically advanced, so small things, when it’s not normal to them, stymie them.(P13, one-year post)

Concerns were heightened for veterans who were more dependent on the portal as a key element in their care due to specific challenges. One participant pointed out that there may be populations of patients with special circumstances who depend more heavily on the prior portal, MyHealtheVet.

I have veterans from [specific region], that’s the way they communicate. Hearing impaired people can’t hear on the phone, the robocall thing, it doesn’t work, so they use MyHealtheVet. Well, if that goes away, how is that being communicated to the veteran? Ok? (P18, Check-in)

Frustration with inadequate dissemination of information to veterans about EHR transition and use of new portal

Participants were concerned about poor information dissemination to patients about how to access the new portal. During medical encounters, participants often heard from patients about their frustrations accessing the new portal. Participants noted that they could only give their patients a phone number to call for help using the new system but otherwise lacked the knowledge and the time to help them resolve new portal issues. Some clinicians specifically mentioned feeling ill-equipped to handle their patients’ needs for assistance with the new portal. These experiences exacerbated clinician stress during the transition.

Our veterans were using the MyHealtheVet messaging portal, and when our new system went up, it transitioned to My VA Health, but that wasn’t really communicated to the veterans very well. So, what happened was they would go into their MyHealtheVet like they had been doing for all of these years, to go in and request their medications, and when they pulled it up it’d show that they were assigned to a clinician in [a different state], that they have no active medications. Everything was just messed up. And they didn’t know why because there was no alert or notification that things would be changing. (P8, check in) I field all-day frustration from the veterans. And I love my job, I’m not leaving here even as frustrated as I am, because I’m here for them, not to, I’m here to serve the veterans and I have to advocate for them, and I know it will get better, it can’t stay like this. But I constantly field their frustrations.… So, I give them the 1-800 number to a Cerner help desk that helps with that, and I’ve had multiple [instances of] feedback that it didn’t help. (P13, one-year post) And [the patients are] frequently asking me things about their medication [within the portal], when, you know, I can’t help them with that. So, I have to send them back up to the front desk to try to figure out their medications. (P17, one-year post)

Veteran frustration and the burden of additional tasks due to issues with the portal

Clinicians reported that veterans expressed frustration with alternatives to the portal, including long call center wait times. Some veterans chose to walk into the clinic without an appointment rather than wait on the phone. Clinicians noted an increase in walk-ins by frustrated veterans, which placed added workload on clinics that were not staffed to handle the increase in walk-ins.

It’s been kind of clunky also with trying to get that [new portal] transitioned. And then that’s created more walk-ins here, because one, the vets get frustrated with the phone part of it, and then MyHealtheVet (prior portal) not [working], so they end up walking [into the clinic without an appointment]. (P19, check-in) In terms of messages, they can’t necessarily find the clinician they want to message. We had a veteran who came in recently who wanted to talk to their Rheumatologist, and it’s like, yeah, I typed in their name, and nothing came up. So, they have to try calling or coming in. (P20, 2-months post)

In summary, participants described the new patient portal as a source of stress for both themselves and their patients.

In addition to their own direct experience using a new EHR to communicate with their patients, clinicians and staff can be affected by perceptions of their patients’ experiences during an EHR transition [ 19 ]. At this first VA site to transition to the new EHR, clinicians and staff shared their concerns about their patients’ experiences using the portal. They were particularly troubled by unreliability of the secure messaging system and challenges patients faced learning to use the new system without proper instruction. Moreover, clinicians were alarmed to hear about patients having to make in-person visits– especially unplanned (i.e., walk in) ones– due to challenges with the new portal. Each of these issues needs to be addressed to ensure veteran satisfaction. However, the only solution participants could offer to frustrated patients was the telephone number to the help desk, leaving them with no clear knowledge of a solution strategy or a timeline for resolution of the issues.

We propose applying emergency preparedness actions to future EHR rollouts: mitigate, prepare, respond, and recover (Fig.  1 ) [ 13 ]. By applying these actions, patient portal disruptions may be alleviated and patients’ communication with their clinicians and access to care can be maintained. For example, issues stemming from a disruption in the portal may be mitigated by first identifying and understanding which patients typically use the portal and how they use it. Sites can use this information to prepare for the transition by disseminating instructional materials to staff and patients on how to access the new portal, targeting the most common and critical portal uses. Sites can respond to any expected and emerging portal disruptions by increasing access to alternative mechanisms for tasks disrupted by and typically completed within the portal. After the transition, recovery can begin by testing and demonstrating the accuracy and reliability of functions in the new portal. These actions directly address reported clinician concerns and can help maintain patient-clinician communication, and access to care.

The emergency preparedness framework was applied. This framework includes 4 actions: (1) mitigate, (2) prepare, (3) respond, and (4) recover. These actions can be repeated. Recommendations for how each action (1–4) can be applied to a portal transition are included in each blue quadrant of the circle

Sites could mitigate issues by first understanding which patients will be most affected by the transition, such as those who rely heavily on secure messaging. Reliable use of secure messaging within the VA facilitates positive patient-clinician relationships by providing a mechanism for efficient between-visit communication [ 20 , 21 , 22 , 23 ]. During the EHR transition, clinicians and staff became concerned about the well-being of patients from whom they weren’t receiving messages and those who depended on the portal to complete certain tasks. Since secure messaging is often initiated by patients to clinicians [ 23 ], clinicians will likely be unaware that messages are being missed. Understanding how and which patients currently use the portal and anticipating potential portal needs is a first step toward mitigating potential issues.

Despite efforts to inform Veterans of the EHR transition and patient portal [ 24 ] including information sent to a Veteran by email, direct mail, postings on VA websites, and a town hall, our findings agree with those of Fix and colleagues [ 10 ] and suggest that many Veterans were unprepared for the transition. Our findings suggest that end users heard that more is needed to improve the dissemination of knowledge about the transition and how to navigate the new patient portal to both VA employees and the patients they serve.

Preparations for the transition should prioritize providing VA clinicians and staff with updated information and resources on how to access and use the new portal [ 25 ]. VA clinicians deliver quality care to veterans and many VA employees are proud to serve the nation’s veterans and willing to go the extra mile to support their patients’ needs [ 26 ]. In this study, participants expressed feeling unprepared to assist or even respond to their patients’ questions and concerns about using the new portal. This unpreparedness contributed to increased clinician and staff stress, as they felt ill-equipped to help their patients with portal issues. Such experiences can negatively affect the patient-clinician relationship. Preparing clinicians and patients about an upcoming transition, including technical support for clinicians and patients, may help minimize these potential issues [ 10 , 27 ]. Specialized training about an impending transition, along with detailed instructions on how to gain access to the new system, and a dedicated portal helpline may be necessary to help patients better navigate the transition [ 23 , 28 ].

In addition to a dedicated helpline, our recommendations include responding to potential changes in needed veteran services during the transition. In our study, participants observed more veteran walk-ins due to challenges with the patient portal. Health systems need to anticipate and address this demand by expanding access to in-person services and fortifying other communication channels. For example, sites could use nurses to staff a walk-in clinic to handle increases in walk-in traffic and increase call center capacity to handle increases in telephone calls [ 29 ]. Increased use of walk-in clinics have received heightened attention as a promising strategy for meeting healthcare demands during the COVID-19 pandemic [ 30 ] and can potentially be adapted for meeting care-related needs during an EHR transition. These strategies can fill a gap in communication between clinicians and their patients while patients are learning to access and navigate a new electronic portal.

Finally, there is a need for a recovery mechanism to restore confidence in the reliability of the EHR and the well-being of clinicians and staff. Healthcare workers are experiencing unprecedented levels of stress [ 31 ]. A plan must be in place to improve and monitor the accuracy of data migrated, populated, and processed within the new system [ 2 ]. Knowing that portal function is monitored could help ease clinician and staff concerns and mitigate stress related to the transition.

Limitations

This study has several limitations. First, data collection relied on voluntary participation, which may introduce self-selection response bias. Second, this work was completed at one VA medical center that was the first site in the larger enterprise-wide transition, and experiences at other VAs or healthcare systems might differ substantially. Third, we did not interview veterans and relied entirely on secondhand accounts of patient experiences with the patient portal. Future research should include interviews with veterans during the transition and compare veteran and VA employee experiences.

Despite a current delay in the deployment of the new EHR at additional VA medical centers, findings from this study offer timely lessons that can ensure clinicians and staff are equipped to navigate challenges during the transition. The strategies presented in this paper could help maintain patient-clinician communication and improve veteran experience. Guided by the emergency preparedness framework, recommended strategies to address issues presented here include alerting those patients most affected by the EHR transition, being prepared to address patients’ concerns, increasing staffing for the help desk and walk-in care clinics, and monitoring the accuracy and reliability of the portal to provide assurance to healthcare workers that patients’ needs are being met. These strategies can inform change management at other VA medical centers that will soon undergo EHR transition and may have implications for other healthcare systems undergoing patient portal changes. Further work is needed to directly examine the perspectives of veterans using the portals, as well as the perspectives of both staff and patients in the growing number of healthcare systems beyond VA that are preparing for an EHR-to-EHR transition.

Data availability

Deidentified data analyzed for this study are available from the corresponding author on reasonable request.

Abbreviations

Electronic health record

Department of Veterans Affairs

VA Medical Centers

Department of Defense

Huang C, Koppel R, McGreevey JD 3rd, Craven CK, Schreiber R. Transitions from one Electronic Health record to another: challenges, pitfalls, and recommendations. Appl Clin Inf. 2020;11(5):742–54.

Article   Google Scholar  

Penrod LE. Electronic Health Record Transition considerations. PM R. 2017;9(5S):S13–8.

Article   PubMed   Google Scholar  

Cogan AM, Haltom TM, Shimada SL, Davila JA, McGinn BP, Fix GM. Understanding patients’ experiences during transitions from one electronic health record to another: a scoping review. PEC Innov. 2024;4:100258. https://doi.org/10.1016/j.pecinn.2024.100258 . PMID: 38327990; PMCID: PMC10847675.

Article   PubMed   PubMed Central   Google Scholar  

Powell KR. Patient-perceived facilitators of and barriers to Electronic Portal Use: a systematic review. Comput Inf Nurs. 2017;35(11):565–73.

Google Scholar  

Wilson-Stronks A, Lee KK, Cordero CL, et al. One size does not fit all: meeting the Health Care needs of diverse populations. Oakbrook Terrace, IL: The Joint Commission; 2008.

Carini E, Villani L, Pezzullo AM, Gentili A, Barbara A, Ricciardi W, Boccia S. The Impact of Digital Patient Portals on Health outcomes, System Efficiency, and patient attitudes: updated systematic literature review. J Med Internet Res. 2021;23(9):e26189.

Home -. My HealtheVet - My HealtheVet (va.gov).

Nazi KM, Turvey CL, Klein DM, Hogan TP. A decade of veteran voices: examining patient Portal Enhancements through the Lens of user-centered design. J Med Internet Res. 2018;20(7):e10413. https://doi.org/10.2196/10413 .

Tapuria A, Porat T, Kalra D, Dsouza G, Xiaohui S, Curcin V. Impact of patient access to their electronic health record: systematic review. Inf Health Soc Care. 2021;46:2.

Fix GM, Haltom TM, Cogan AM, et al. Understanding patients’ preferences and experiences during an Electronic Health Record Transition. J GEN INTERN MED. 2023. https://doi.org/10.1007/s11606-023-08338-6 .

Monturo C, Brockway C, Ginev A. Electronic Health Record Transition: the patient experience. CIN: Computers Inf Nurs. 2022;40:1.

Tian D, Hoehner CM, Woeltje KF, Luong L, Lane MA. Disrupted and restored patient experience with transition to New Electronic Health Record System. J Patient Exp. 2021;18:8.

Emergency management programs for healthcare facilities. the four phases of emergency management. US Department of Homeland Security website: https://www.hsdl.org/?view&did=765520 . Accessed 28 Aug 2023.

Ahlness EA, Orlander J, Brunner J, Cutrona SL, Kim B, Molloy-Paolillo BK, Rinne ST, Rucci J, Sayre G, Anderson E. Everything’s so Role-Specific: VA Employee Perspectives’ on Electronic Health Record (EHR) transition implications for roles and responsibilities. J Gen Intern Med. 2023;38(Suppl 4):991–8. Epub 2023 Oct 5. PMID: 37798577; PMCID: PMC10593626.

Rucci JM, Ball S, Brunner J, Moldestad M, Cutrona SL, Sayre G, Rinne S. Like one long battle: employee perspectives of the simultaneous impact of COVID-19 and an Electronic Health Record Transition. J Gen Intern Med. 2023;38(Suppl 4):1040–8. https://doi.org/10.1007/s11606-023-08284-3 . Epub 2023 Oct 5. PMID: 37798583; PMCID: PMC10593661.

Sayre G, Young J. Beyond open-ended questions: purposeful interview guide development to elicit rich, trustworthy data [videorecording]. Seattle (WA): VA Health Services Research & Development HSR&D Cyberseminars; 2018.

Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12:6855–66.

Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62:1107–15.

Haun JN, Lind JD, Shimada SL, Simon SR. Evaluating Secure Messaging from the veteran perspective: informing the adoption and sustained use of a patient-driven communication platform. Ann Anthropol Pract. 2013;372:57–74.

Kittler AF, Carlson GL, Harris C, Lippincott M, Pizziferri L, Volk LA, et al. Primary care physician attitudes toward using a secure web-based portal designed to facilitate electronic communication with patients. Inf Prim Care. 2004;123:129–38.

Shimada SL, Petrakis BA, Rothendler JA, Zirkle M, Zhao S, Feng H, Fix GM, Ozkaynak M, Martin T, Johnson SA, Tulu B, Gordon HS, Simon SR, Woods SS. An analysis of patient-provider secure messaging at two Veterans Health Administration medical centers: message content and resolution through secure messaging. J Am Med Inf Assoc. 2017;24:5.

Jha AK, Perlin JB, Kizer KW, Dudley RA. Effect of the transformation of the Veterans Affairs Health Care System on the quality of care. N Engl J Med. 2003;348:22.

McAlearney AS, Walker DM, Gaughan A, Moffatt-Bruce S, Huerta TR. Helping patients be better patients: a qualitative study of perceptions about Inpatient Portal Use. Telemed J E Health. 2020;26:9.

https://www.myhealth.va.gov/mhv-portal-web/transitioning-to-my-va-health-learn-more .

Beagley L. Educating patients: understanding barriers, learning styles, and teaching techniques. J Perianesth Nurs. 2011;26:5.

Moldestad M, Stryczek KC, Haverhals L, Kenney R, Lee M, Ball S, et al. Competing demands: Scheduling challenges in being veteran-centric in the setting of Health System initiatives to Improve Access. Mil Med. 2021;186:11–2.

Adusumalli J, Bhagra A, Vitek S, Clark SD, Chon TY. Stress management in staff supporting electronic health record transitions: a novel approach. Explore (NY). 2021;17:6.

Heponiemi T, Gluschkoff K, Vehko T, Kaihlanen AM, Saranto K, Nissinen S, et al. Electronic Health Record implementations and Insufficient Training Endanger nurses’ Well-being: cross-sectional survey study. J Med Internet Res. 2021;23:12e27096.

Laurant M, van der Biezen M, Wijers N, Watananirun K, Kontopantelis E, van Vught AJ. Nurses as substitutes for doctors in primary care. Cochrane Database Syst Rev. 201;7(7):CD001271.

Elnahal S, Kadakia KT, Gondi S, How, U.S. Health systems Can Build Capacity to Handle Demand Surges. Harvard Business Review. 2021. https://hbr.org/2021/10/how-u-s-health-systems-can-build-capacity-to-handle-demand-surges/ Accessed 25 Nov 2022.

George RE, Lowe WA. Well-being and uncertainty in health care practice. Clin Teach. 2019;16:4.

Download references

Acknowledgments

We acknowledge and thank members of the EMPIRIC Evaluation qualitative and supporting team for their contributions to this work including Ellen Ahlness, PhD, Julian Brunner, PhD, Adena Cohen-Bearak, MPH, M.Ed, Leah Cubanski, BA, Christine Firestone, Bo Kim, PhD, Megan Moldestad, MS, and Rachel Smith. We greatly appreciate the staff at the Mann-Grandstaff VA Medical Center and associated community-based outpatient clinics for generously sharing of their time and experiences participating in this study during this challenging time.

The “EHRM Partnership Integrating Rapid Cycle Evaluation to Improve Cerner Implementation (EMPIRIC)” (PEC 20–168) work was supported by funding from the US Department of Veterans Affairs, Veterans Health Administration, Health Services Research & Development Quality Enhancement Research Initiative (QUERI) (PEC 20–168). The findings and conclusions in this article are those of the authors and do not necessarily reflect the views of the Veterans Health Administration, Veterans Affairs, or any participating health agency or funder.

Author information

Authors and affiliations.

VA Northeast Ohio Healthcare System, 10701 East Blvd., Research Service 151, 44106, Cleveland, OH, USA

Sherry L. Ball

Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System, Boston, MA, USA

Bo Kim & Seppo T. Rinne

Department of Psychiatry, Harvard Medical School, Boston, MA, USA

Center for Healthcare Organization and Implementation Research, VA Bedford Healthcare System, Bedford, MA, USA

Sarah L. Cutrona & Brianne K. Molloy-Paolillo

Division of Health Informatics & Implementation Science, Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, MA, USA

Sarah L. Cutrona

Seattle-Denver Center of Innovation for Veteran-Centered and Value-Driven Care, VHA Puget Sound Health Care System, Seattle, WA, USA

Ellen Ahlness, Megan Moldestad & George Sayre

University of Washington School of Public Health, Seattle, WA, USA

George Sayre

Geisel School of Medicine at Dartmouth, Hannover, NH, USA

Seppo T. Rinne

You can also search for this author in PubMed   Google Scholar

Contributions

S.R. designed the larger study. G.S. was the qualitative methodologist who led the qualitative team. S.B., E.A., and M.M. created the interview guides and completed the interviews; Data analysis, data interpretation, and the initial manuscript draft were completed by S.B. and B.K. S.C. and B.M. worked with the qualitative team to finalize the analysis and edit and finalize the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sherry L. Ball .

Ethics declarations

Ethics approval and consent to participate.

This evaluation was designated as non-research/quality improvement by the VA Bedford Healthcare System Institutional Review Board. All methods were carried out in accordance with local and national VA guidelines and regulations for quality improvement activities. This study included virtual interviews with participants via MS Teams. Employees volunteered to participate in interviews and verbal consent was obtained to record interviews. Study materials, including interview guides with verbal consent procedures, were reviewed and approved by labor unions and determined as non-research by the VA Bedford Healthcare System Institutional Review Board.

Consent for publication

Not applicable.

The findings and conclusions in this paper are those of the authors and do not necessarily represent the official position of the Department of Veterans Affairs.

Prior presentations

Ball S, Kim B, Moldestad M, Molloy-Paolillo B, Cubanski L, Cutrona S, Sayre G, and Rinne S. (2022, June). Electronic Health Record Transition: Providers’ Experiences with Frustrated Patients. Poster presentation at the 2022 AcademyHealth Annual Research Meeting. June 2022.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ball, S.L., Kim, B., Cutrona, S.L. et al. Clinician and staff experiences with frustrated patients during an electronic health record transition: a qualitative case study. BMC Health Serv Res 24 , 535 (2024). https://doi.org/10.1186/s12913-024-10974-5

Download citation

Received : 29 August 2023

Accepted : 09 April 2024

Published : 26 April 2024

DOI : https://doi.org/10.1186/s12913-024-10974-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• EHR transition
• Patient experience
• Clinician experience
• Qualitative analysis

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

ORIGINAL RESEARCH article

This article is part of the research topic.

Digital Health Applications: Acceptance, Benefit Assessment, and Costs from the Perspective of Patients and Medical Professionals

Implementation of the electronic health record (EHR) in the German healthcare system: An assessment of the current status and future development perspectives considering the potentials of health data utilisation by representatives of different stakeholder groups. Provisionally Accepted

• 1 West Saxon University of Applied Sciences of Zwickau, Germany

The final, formatted version of the article will be published soon.

Introduction: The digitalisation of the German healthcare system enables a wide range of opportunities to utilize healthcare data. The implementation of the EHR in January 2021 was a significant step, but compared to other European countries, the implementation of the EHR in the German healthcare system is still at an early stage. The aim of this paper is to characterise the structural factors relating to the adoption of the EHR in more detail from the perspective of representatives of stakeholders working in the German healthcare system and to identify existing barriers to implementation and the need for change. Methods: Qualitative expert interviews were conducted with one representative from each of the stakeholder groups health insurance, pharmacies, healthcare research, EHR development and panel doctors. Results: The interviews with the various stakeholders revealed that the implementation process of the EHR is being delayed by a lack of a viable basis for decision-making, existing conflicts of interest and insufficient consideration of the needs of patients and service providers, among other things. Discussion: The current status of EHR implementation is due to deficiency in legal regulations as well as structural problems and the timing of the introduction. For instance, the access rights of various stakeholders to the EHR data and the procedure in the event of a technical failure of the telematics infrastructure are remain unclear. In addition, insufficient information and communication measures have not led to the desired acceptance of EHR use among patients and service providers.

Keywords: Digital Health, electronic health record - (EHR), Personal health records, health data use, Digitalisation

Received: 15 Jan 2024; Accepted: 26 Apr 2024.

Copyright: © 2024 Rau, Tischendorf and Mitzscherlich. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mx. Tim Tischendorf, West Saxon University of Applied Sciences of Zwickau, Zwickau, 08056, Lower Saxony, Germany

People also looked at

This paper is in the following e-collection/theme issue:

Published on 29.4.2024 in Vol 26 (2024)

Exploring the Impact of In Basket Metrics on the Adoption of a New Electronic Health Record System Among Specialists in a Tertiary Hospital in Alberta: Descriptive Study

Authors of this article:

There are no citations yet available for this article according to Crossref .