Background: In South Korea, most graduated medical students undertake a 1-year internship before beginning residency and specialization. Interns usually work in a tertiary hospital and rotate between different, randomly assigned departments to be exposed to different medical specialties. Their jobs are mostly simple and repetitive but are still essential for the patient care process. However, owing to the lack of experience and overwhelming workload, interns at tertiary hospitals in South Korea are usually inefficient, often delaying the entire clinical process. Health care providers have widely adopted mobile electronic medical records (mEMRs) as they have been shown to improve workflow efficiency.
Objective: This study investigates the association between the frequency of mEMR usage and the clinical task completion interval time among interns in a tertiary hospital.
Methods: This mixed methods study was conducted at the Samsung Medical Center, Seoul, South Korea. Interns who worked at the Samsung Medical Center from March 2018 to February 2019 were included. The hospital electronic medical record (EMR) system known as DARWIN (Data Analysis and Research Window for Integrated kNowledge) was launched with PC and mobile. Both versions are actively used in hospitals by personnel in various positions. We collected the log data from the mEMR server and the intern clinical task time-series data from the EMR server. Interns can manage the process of identifying patients, assigning the clinical task, finishing the requested clinical intern tasks, etc, through the use of the mEMR system. We compared the clinical task completion interval among 4 groups of interns divided by the mEMR frequency quantile. Then, System Usability Score (SUS) questionnaires and semistructured interviews were conducted.
Results: The regular mEMR users were defined as those who logged in more than once a day on average and used the mEMR until the level after login. Among a total of 87 interns, 84 used the mEMR to verify the requested clinical tasks. The most frequently used item was “Intern task list.” Analysis of the 4 intern groups revealed an inverse relationship between the median time of the task completion interval and the frequency of mEMR use. Correlation analysis showed that the intern task completion time interval had a significant inverse relationship with the individual frequency of mEMR usage (coefficient=-0.27; 95% CI -0.46 to -0.04; P=.02). In the additional survey, the mean SUS value was 81.67, which supported the results of the data analysis.
Conclusions: Our findings suggest that frequent mEMR use is associated with improved work efficiency in hospital interns with good usability of the mEMR. Such finding supports the idea that the use of mEMR improves the effectiveness and workflow efficiency of interns working in hospitals and, more generally, in the context of health care.
Background and Significance
Professionals of various occupations, such as doctors, nurse pharmacists, and other supporters, provide patient care in hospitals. In most tertiary hospitals in South Korea, prescribers, such as specialists and residents, determine the appropriate care plan and use computerized provider order entry (CPOE) to order prescriptions. Then a nurse executes the order or passes it on to interns or other supporters. This computerized linear workflow benefits workflow and patient safety [, ]; however, when overloaded, it inevitably results in inefficiency and delay [ ].
Internship is the transition period between being a medical student and becoming a specialty doctor . In South Korea, most graduated medical students undertake a 1-year internship with their physician's license before beginning their specialty resident course [ ]. Interns usually work in a tertiary hospital, where they rotate monthly among different, randomly assigned departments to be exposed to different medical specialties. Their jobs are mostly simple and repetitive, but they are essential for the patient care process. These jobs include simple procedures (eg, catheterization, biopsies, monitoring, and sampling), documentation (eg, getting consent forms for radiology or procedures), and prescriptions that do not affect patient care plans (eg, meal changes, simple dressing) [ ].
The lack of experience and the workload of interns at tertiary hospitals, where patient needs are substantial and often overwhelming, make interns inevitably susceptible to inefficiency and fatigue . Such inefficiency can halt the entire clinical process and expose patients to the risk of errors. This can ultimately have an adverse influence on patient care and safety [ ].
Mobile Electronic Medical Records for Health Care Providers
Before the use of smartphones, interns usually received notifications pertaining to their jobs through various mediums (depending on the policies of each hospital), such as pagers, phone calls, or beepers for SMS texting exclusively used in the hospital communication system . With the widespread use of smartphones, health care professionals have widely adopted mobile electronic medical records (mEMRs) [ - ]. The mEMR has been shown to improve the efficiency and effectiveness of hospital workflow in previous studies [ , ]. However, none of the previous studies evaluated the time efficiency of interns’ job achievement in a clinical setting using mEMRs.
This study aims to determine the association between interns' clinical task completion time interval and the frequency of mEMR usage.
This mixed methods study was conducted at the Samsung Medical Center, Seoul, South Korea. We targeted and analyzed interns who worked at the Samsung Medical Center from March 2018 to February 2019. In South Korea, from the month of March to the following February, interns rotate between various medical departments. To examine the association between mEMR usage and intern performance, we collected the log data from the mEMR server and the intern clinical task time-series data from Samsung Medical Center's Electronic Medical Record (EMR). The study protocol was reviewed and approved by the institutional review board (IRB) of Samsung Medical Center (IRB No. SMC 2019-09-122-001).
Mobile Electronic Medical Records
In July 2016, the hospital EMR system known as DARWIN (Data Analysis and Research Window for Integrated kNowledge) was launched. DARWIN has both PC and mobile versions. DARWIN is actively used in hospitals, and its mobile version is used by hospital personnel in various positions. Mobile DARWIN (mDARWIN) includes a main menu, list-level features, and patient-level features. After login into the mDARWIN, users can select a list-level feature on the first screen from 8-9 main menus.
Interns' Clinical Task Implementation Process
There are 3 types of prescriptions that prescribers such as specialists or residents issue: (1) basic prescription (eg, vital-sign check term, input and output check term, meal, or simple daily care service for postoperation patients, (2) medication prescription, and (3) examination prescription. These prescriptions have associated tasks that are performed by health care providers. With the exception of the tagging of prescriptions (which is performed by nurses), most clinical tasks are performed by interns. When interns receive an alarm about a new task on their mobile device, they verify the clinical task and self-assign the prescriptions to themselves. Then, they conduct the clinical task according to the instruction (). Interns can manage this process (ie, identify, assign, and mark the task as complete after finishing the requested clinical task) through the mDARWIN mEMR ( ).
Outcome Measures and Sensitivity Analysis
The primary outcome was the comparison of the time interval to complete the intern tasks after dividing the interns into 4 groups based on the quantile of the frequency of mEMR usage. The definition of the task completion interval time was set from the time the task was requested to the task completion check time. For sensitivity analysis, we verified the correlation between the frequency of mEMR usage and the median time of interval to complete the intern's tasks individually. Subsequently, System Usability Score (SUS) questionnaires were administered and analyzed.
To investigate the feasibility of the mEMR in a clinical environment, SUS questionnaires were administered to interns . Interns were recruited through a notice posted on the bulletin board in the hospital for 2 months. In addition to the survey, interns were also interviewed at the same time. The survey included 10 questions, scored using a 5-point Likert scale (1=strongly disagree, 5=strongly agree). The SUS calculation formula is as follows:
During the interview, interns were asked questions such as when they mostly used the mEMR, which list they searched the most, where they mostly used the mEMR, and whether mDARWIN helps with their tasks. The interviews were semistructured.
We investigated the log data of interns' mEMRs during the study period. We compared the task completion time interval among the 4 groups using statistical analysis. We compared the task completion interval's median time and 95% confidence interval between the 4 groups.
For sensitivity analysis, we evaluated the correlation between the frequency of log data of interns' mEMR and the individual task completion time interval of interns using the Pearson product-moment correlation coefficient test. P values of <.05 were considered statistically significant. All data analyses were performed using R software (version 3.4.2; R Project for Statistical Computing).
Characteristics of the Subjects
In total, 87 interns performed intern tasks during the study period. A total of 1,081,413 tasks were performed by these interns. Of the 87 interns, 84 regularly used the mEMR and were included in the analysis. However, 3 interns were excluded because 2 had not used mEMR at all and 1 had a total of only 4 log records during the study period; thus, they were considered nonusers. In this context, regular mEMR users were defined as those who logged in more than once a day on average and used the mEMR until the next level after login.shows the intern information included in the study and the clinical tasks they received.
|Participant characteristics||Values, n (%)|
|Medical interns (n=87)|
|mEMRa users||84 (97)|
|Non-mEMRa users||3 (3)|
|Total intern clinical tasks performed (n=1,081,413)|
|Medical part||462,018 (42.70)|
|Surgical part||478,242 (44.20)|
|Other hospital-based part||141,153 (13.10)|
|By procedure category|
|Request order transcription (from nurse)b||348,805 (32.30)|
|Request order transcription (from doctor) b||163,886 (15.20)|
|Diagnostic test consent form||170,542 (15.80)|
|Wound dressing||134,503 (12.40)|
|Diagnostic test||94,596 (8.70)|
|Diagnostic test and treatment||30,521 (2.80)|
|Catheter tube insertion||16,058 (1.50)|
|Administrative paperwork||14,533 (1.30)|
|Influenza exam||5938 (0.50)|
|Writing slip||2733 (0.30)|
|Inject medicine||959 (0.10)|
amEMR: mobile electronic medical records.
bSouth Korea\'s medical system adopts a fee-for-service model for medical service. As the prescription order can only be authorized by a doctor, this category is in relation to the prescription after the act of the nurse or doctor.
Log Data Analysis
During the study period, 489,444 mEMR logs were created by interns. Interns used a total of 43 items within the mEMR, as shown in. From the 489,444 logs, 67,147 logs were made in a list-level feature. Among these records, “Intern task list” topped the list with 39,506 tasks. This was followed by “My patient list” and “Surgery history list,” as shown in .
|No.||List||Frequency, n (%)|
|1||Intern task list||39,506 (58.80)|
|2||My patient list||13,685 (20.40)|
|3||Surgery history list||8,545 (12.70)|
|4||Inpatient list||3,963 (5.90)|
|5||Emergency patient list||663 (1.00)|
|6||Integrated-view EMRa list||241 (0.40)|
|7||Outpatient list||137 (0.20)|
|8||Consultation list||110 (0.20)|
|9||Expected inpatient list||78 (0.10)|
|10||Scheduled surgery list||46 (0.10)|
|11||Patient search (through patient ID)||15 (0.00)|
aEMR: electronic medical record.
The comparison of clinical task completion interval consistently showed an inverse relationship between the median time of completion interval and the frequency of mEMR usage (). The frequent mEMR user group took a shorter time to complete the requested tasks. Sensitivity analysis with the Pearson product-moment correlation coefficient showed that the intern task completion interval time had a significant inverse relationship with individual frequency of mEMR usage (coefficient -0.27; 95% CI -0.46 to -0.04; P=.02). Using the mEMR once reduced the task completion time by approximately 16 seconds (P=.02).
SUS Survey Outcome
A total of 15 interns completed the SUS survey from December 2019 to January 2020. The mean SUS value for the intern clinical task item in the mEMR was 81.67 (). The interview and survey were conducted at the same time. shows the key points in the interview that may be useful for future research.
|No.||Question||Response, mean (SD)a|
|1||I think I use (intern task) frequently through mDARWINb.||4.7 (0.6)|
|2||I found that using (intern task) through mDARWINb is unnecessarily complex.||1.4 (0.5)|
|3||I thought that using (intern task) in mDARWINb was easy.||4.5 (0.5)|
|4||I think technical support is needed to use (intern task) in mDARWINb.||2.7 (1.4)|
|5||I found that (intern task) in mDARWINb was well integrated.||3.9 (1.0)|
|6||I thought there was too much inconsistency with (intern task) in mDARWINb.||1.7 (0.7)|
|7||I would imagine that most people would learn to use (intern task) through mDARWINb very quickly and easily.||4.7 (0.5)|
|8||I found that using (intern task) in mDARWINb is very cumbersome.||1.7 (0.9)|
|9||I felt very confident using (intern task) through mDARWINb.||4.0 (0.9)|
|10||I needed to learn many things before I could get going with (intern task) through mDARWINb.||1.5 (0.6)|
aResponses were scored using a 5-point Likert scale (1=strongly disagree, 5=strongly agree).
bmDARWIN: mobile Data Analysis and Research Window for Integrated kNowledge.
This study investigated the correlation between the frequency of mEMR usage and the intern task time interval based on mEMR log data and EMR timestamp data. Most interns use mEMR and the task completion time was shorter for interns who used mEMR more frequently. This suggests that mEMR use could effectively enhance hospital workflow time, leading to a faster response in real practice. This result supports the findings of previous studies that indicate that the mEMR is linked to improved workflow efficiency in hospitals by enabling faster responses [, , ].
In addition to log data analysis, we also interviewed interns to assess the use of the mEMR for job execution. All the interns who participated in the survey and interview were actively using the mEMR. The mean SUS value was >80, implying that the system is well utilized by the user . Doctors tend to underestimate the various positive workflow effects of mEMR usage [ ]; as such, our results are interesting and valuable enough to analyze motivation. We assume that the obvious and dominant benefit of mEMRs in terms of convenience and time efficiency would make all interns maximize the use of the mEMR compared to other systems such as computers and telephones. Further in-depth surveying and analysis can help increase mEMR usage among hospital health care providers. Our study shows that mEMR use offers both quantitative and qualitative strengths for intern job performance.
Comparison with Prior Work
Studies aiming to investigate the effects of mobile device use among health care providers in hospitals have proven their efficiency via surveys [, , ] and in simulations [ , ], and they have shown to be effective in limited spaces such as surgical rooms [ ] and emergency departments [ ]. However, there are limited quantitative studies assessing the efficiency of mEMR use in clinical practice. Our study results provide further evidence of the efficiency of mEMRs and suggest extending their use to other professionals with relatively similar daily tasks, such as physician assistants (PAs) who are responsible for clinical prescriptions in tertiary hospitals or nurses who are similarly overloaded with work. Further, the use of mEMRs by PAs or nurses would improve workflow efficiency, and ultimately, patient care and patient safety [ ].
In recent times, quick response code technology reduces time and errors in patient identification during patient care and procedures [- ]. Further, the closed-loop medication system, which integrates the barcode medication system and CPOE technology with automated dispensing technology (robots/units), prevents the adverse effects of medication due to administration errors [ ]. Future efforts should be directed at combining mEMR use with these technologies to simultaneously achieve efficient workflows and patient safety.
This study has some limitations. First, as this study was performed at a single center, the results have limited generalizability. Further, given that there are different job allocations for each occupation depending on the hospital, its feasibility and usability need to be validated in other institutions and environments.
Second, we could not identify the causality of log data as we analyzed the entire log dataset. There is no consideration for context or order between log data and interns’ jobs. Although the entire log was sufficient to achieve the study aim, a further observational study using the small cut log of mEMR is needed to analyze the association between behavior and mEMR usage.
Third, we did not consider the priority of specific jobs when assessing performance. Jobs related to emergency situations need to be prioritized over others that can be completed after the emergency situation. However, this study aimed to investigate the general trend of frequent mEMR users and not to compare nonfrequent and frequent users to assess the efficiency of the mEMR. Furthermore, the log data was large enough to distinguish between situations.
By retrospectively analyzing the mEMR log data of hospital interns, this study revealed that more frequent use of the mEMR led to quicker completion of intern jobs. This finding implies the effectiveness of mEMR use for the workflow of interns in hospitals. We used a SUS survey to examine the usability of mEMR, and the survey concluded that the mEMR has a good usability.
The authors would like to acknowledge the support of Samsung Medical Center.
Conflicts of Interest
Items on the mEMR menu.PDF File (Adobe PDF File), 242 KB
- Househ M, Ahmad A, Alshaikh A, Alsuweed F. Patient safety perspectives: the impact of CPOE on nursing workflow. Stud Health Technol Inform 2013;183:367-371. [Medline]
- Page N, Baysari MT, Westbrook JI. A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety. Int J Med Inform 2017 Dec;105:22-30. [CrossRef] [Medline]
- Zhivan NA, Diana ML. U.S. hospital efficiency and adoption of health information technology. Health Care Manag Sci 2011 Sep 16;15(1):37-47. [CrossRef]
- Brennan N, Corrigan O, Allard J, Archer J, Barnes R, Bleakley A, et al. The transition from medical student to junior doctor: today's experiences of Tomorrow's Doctors. Med Educ 2010 May;44(5):449-458. [CrossRef] [Medline]
- Lim H, Ahn D, Ahn S. A Comparative Study on Medical Education Systems in Korea, China and Japan. Korean J Med Educ 2007 Dec 31;19(4):271-278. [CrossRef]
- Yeo I, Kim J, Lee M, Shin S, Ryoo H, Park J. Work-related fatigue, stress and depression of the emergency department interns. Journal of The Korean Society of Emergency Medicine 2013;24(4):428-438.
- Kim K, Lee S, Choi YH. Relationship between occupational stress and depressive mood among interns and residents in a tertiary hospital, Seoul, Korea. Clin Exp Emerg Med 2015 Jun 30;2(2):117-122. [CrossRef]
- Garcia, Abreu, Ramos, Castro, Smiderle, Santos, Bezerra. Influence of Burnout on Patient Safety: Systematic Review and Meta-Analysis. Medicina 2019 Aug 30;55(9):553. [CrossRef]
- Tipping MD, Forth VE, O'Leary KJ, Malkenson DM, Magill DB, Englert K, et al. Where did the day go?-A time-motion study of hospitalists. J. Hosp. Med 2010 Aug 20;5(6):323-328. [CrossRef]
- Boulos MNK, Wheeler S, Tavares C, Jones R. How smartphones are changing the face of mobile and participatory healthcare: an overview, with example from eCAALYX. Biomed Eng Online 2011;10:24 [FREE Full text] [CrossRef] [Medline]
- Dimond R, Bullock A, Lovatt J, Stacey M. Mobile learning devices in the workplace: ‘as much a part of the junior doctors’ kit as a stethoscope’? BMC Med Educ 2016 Aug 17;16(1). [CrossRef]
- Schimpf B, Deanda K, Severenuk DA, Montgomery TM, Cooley GD, Kowalski RG, et al. Integration of Real-Time Electronic Health Records and Wireless Technology in a Mobile Stroke Unit. Journal of Stroke and Cerebrovascular Diseases 2019 Sep;28(9):2530-2536. [CrossRef]
- Jung KY, Kim S, Kim K, Lee EJ, Kim K, Lee J, et al. Frequent Mobile Electronic Medical Records Users Respond More Quickly to Emergency Department Consultation Requests: Retrospective Quantitative Study. JMIR Mhealth Uhealth 2020 Feb 14;8(2):e14487. [CrossRef]
- Duhm J, Fleischmann R, Schmidt S, Hupperts H, Brandt SA. Mobile Electronic Medical Records Promote Workflow: Physicians' Perspective From a Survey. JMIR Mhealth Uhealth 2016 Jun 06;4(2):e70 [FREE Full text] [CrossRef] [Medline]
- Brooke J. SUS-A quick and dirty usability scale. Usability evaluation in industry 1996;189(194):4-7. [CrossRef]
- Power JM, Spina SP, Forbes DA, Harder CK, Lalli SL, Loewen PS, et al. Integration of Smartphones into clinical pharmacy practice: An evaluation of the impact on pharmacists׳ efficiency. Health Policy and Technology 2014 Dec;3(4):296-305. [CrossRef]
- Wu R, Rossos P, Quan S, Reeves S, Lo V, Wong B, et al. An Evaluation of the Use of Smartphones to Communicate Between Clinicians: A Mixed-Methods Study. J Med Internet Res 2011 Aug 29;13(3):e59. [CrossRef]
- Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 2009;4(3):A.
- Panesar P, Jones A, Aldous A, Kranzer K, Halpin E, Fifer H, et al. Attitudes and Behaviours to Antimicrobial Prescribing following Introduction of a Smartphone App. PLoS ONE 2016 Apr 25;11(4):e0154202. [CrossRef]
- Kim S, Lee K, Hwang H, Yoo S. Analysis of the factors influencing healthcare professionals’ adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital. BMC Med Inform Decis Mak 2016 Jan 30;16(1). [CrossRef]
- Rao AS, Adam TJ, Gensinger R, Westra BL. Study of the factors that promoted the implementation of electronic medical record on iPads at two emergency departments. In: AMIA Annu Symp Proc. 2012 Presented at: AMIA Annu Symp Proc ; Medline; 2012; Chicago, US p. 744-752 URL: http://europepmc.org/abstract/MED/23304348
- Jung KY, Kim T, Jung J, Lee J, Choi JS, Mira K, et al. The Effectiveness of Near-Field Communication Integrated with a Mobile Electronic Medical Record System: Emergency Department Simulation Study. JMIR Mhealth Uhealth 2018 Sep 21;6(9):e11187. [CrossRef]
- Khanna V, Sambandam SN, Gul A, Mounasamy V. "WhatsApp"ening in orthopedic care: a concise report from a 300-bedded tertiary care teaching center. Eur J Orthop Surg Traumatol 2015 Jul;25(5):821-826. [CrossRef] [Medline]
- Gulacti U, Lok U, Hatipoglu S, Polat H. An Analysis of WhatsApp Usage for Communication Between Consulting and Emergency Physicians. J Med Syst 2016 Jun;40(6):130. [CrossRef] [Medline]
- Moore S, Jayewardene D. The use of smartphones in clinical practice. Nursing Management 2014 Jun 26;21(4):18-22. [CrossRef]
- Mathivanan P, Edward Jero S, Ramu P, Balaji Ganesh A. QR code based patient data protection in ECG steganography. Australas Phys Eng Sci Med 2018 Nov 5;41(4):1057-1068. [CrossRef]
- Upton J, Olsson-Brown A, Marshall E, Sacco J. Using QR codes to enable quick access to information in acute cancer care. British Journal of Nursing 2017 May 25;26(10):S4-S12. [CrossRef]
- Dixon JL, Smythe WR, Momsen LS, Jupiter D, Papaconstantinou HT. Quick Response codes for surgical safety: a prospective pilot study. Journal of Surgical Research 2013 Sep;184(1):157-163. [CrossRef]
- Burkoski V, Yoon J, Solomon S, Hall T, Karas A, Jarrett S, et al. Closed-Loop Medication System: Leveraging Technology to Elevate Safety. cjnl 2019 May 07;32(SP):16-28. [CrossRef]
|CPOE: computerized provider order entry|
|DARWIN: Data Analysis and Research Window for Integrated kNowledge|
|EMR: electronic medical record|
|mDARWIN: mobile Data Analysis and Research Window for Integrated kNowledge|
|mEMR: mobile electronic medical record|
|SUS: System Usability Score|
Edited by L Buis; submitted 18.08.20; peer-reviewed by R Krukowski, J Li; comments to author 22.08.20; revised version received 16.10.20; accepted 30.11.20; published 15.12.20Copyright
©SuJin Kim, Seulji Ku, Taerim Kim, Won Chul Cha, Kwang Yul Jung. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 15.12.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.