Background: The use of mobile apps in health care is growing. Current and future practitioners must be equipped with the skills to navigate and utilize apps in patient care, yet few strategies exist for training health care professional students on the usage of apps.
Objective: To characterize first-year pharmacy student use of medical apps, evaluate first-year pharmacy student's perception of skills in finding, evaluating, and using medical apps before and after a focused learning experience, and assess student satisfaction and areas for improvement regarding the learning experience.
Methods: Students listened to a recorded, Web-based lecture on finding, evaluating, and using mobile apps in patient care. A 2-hour, interactive workshop was conducted during which students were led by an instructor through a discussion on strategies for finding and using apps in health care. The students practiced evaluating 6 different health care–related apps. Surveys were conducted before and after the focused learning experience to assess students' perceptions of medical apps and current use and perspectives on satisfaction with the learning experience and role of technology in health care.
Results: This educational intervention is the first described formal, interactive method to educate student pharmacists on medical apps. With a 99% response rate, surveys conducted before and after the learning experience displayed perceived improvement in student skills related to finding (52/119, 44% before vs 114/120, 95% after), evaluating (18/119, 15% before vs 112/120, 93% after), and using medical apps in patient care (31/119, 26% before vs 108/120, 90% after) and the health sciences classroom (38/119, 32% before vs 104/120, 87% after). Students described satisfaction with the educational experience and agreed that it should be repeated in subsequent years (89/120, 74% agreed or strongly agreed). Most students surveyed possessed portable electronic devices (107/119, 90% mobile phone) and agreed with the concept of medical apps being an important part of the health care profession in the future (112/119, 94% before and 115/120, 96% after).
Conclusions: Student pharmacists recognize the key role technology plays in the future of health care. A medical apps workshop was successful in improving student pharmacists' perceptions of ability to find, evaluate, and use medical apps.
With the worldwide popularity of mobile devices (eg, mobile phones, tablet computers), mobile apps are increasingly being used by health care professionals in a variety of settings. A mobile device is defined as “a portable, wireless computing device that is small enough to be used while held in the hand” . In the health care field, it is estimated that there will be 500 million smartphone users worldwide with mobile health apps by 2015, and the global market for these apps may reach US $26 billion by 2017 [ ]. There are more than 43,000 health care, fitness, and medical apps available in English on iTunes [ ]. As mobile technology continues to gain popularity, health care practitioners, students, and residents are gravitating toward the utilization of mobile devices to assist in their clinical duties and education [ - ].
Clinicians have access to mobile medical apps that serve as drug references, clinical calculators, disease references, and clinical decision-processing aides [, ]. Pharmacists have also integrated mobile devices as a means to help process medical orders in the hospital, access clinical references, and increase communication with providers in their practices and professional duties [ - ]. Although medical apps are helping advance the field of health care in this digital era, there are certain pitfalls being noted by the health care and academic communities related to the quality of the apps and evidence of health information provided in them [ - ]. One of the biggest issues is the reliability and accuracy of information found within the plethora of medical apps available. Several studies have identified these concerns, with issues ranging from lack of medical evidence, nondisclosure of authorship, design flaws, and inaccurate information [ - ].
The Food and Drug Administration (FDA) has recently released guidelines on which subset of medical apps it will review and regulate . Whereas the safety of such apps is being ensured by FDA standards, apps that record life events, extract medical content, serve as clinical or drug references, or facilitate the communication between clinicians or health centers with patients will not be regulated by the FDA [ ]. Given the limited scope of FDA in monitoring medical apps, relevance and reliability of these apps will need to be determined by the users. As health care professionals, pharmacists should have the adequate background and understanding of these apps to critique the information provided, to be able to recommend to patients or colleagues, and to use them effectively when providing care.
Future health care practitioners will likely be the generation to formally adopt mobile technology into their workflow and clinical practice. Studies have identified that mobile devices and medical apps are increasingly being incorporated into formal medical education [- ]. However, strategies on educating current and future pharmacists to use apps effectively have not been reported. The objectives of this study were to characterize first-year pharmacy student use of medical apps, evaluate first-year pharmacy student's perception of skills in finding, evaluating, and using medical apps before and after a focused learning experience, and assess student satisfaction and areas for improvement regarding the learning experience.
Description of Learning Experience
Faculty from The Ohio State University College of Pharmacy (OSU COP) and Massachusetts College of Pharmacy and Health Sciences (MCPHS) University School of Pharmacy collaborated to create a novel learning experience for pharmacy students that focused on finding, evaluating, and using medical apps. The project was approved as exempt research by The Ohio State University Institutional Review Board. At OSU COP, foundational concepts related to drug information, including literature assessment, are taught in the spring semester of the PharmD curriculum as a distinct module that is part of an introductory pharmacy practice course series. During this module, students are engaged with didactic, workshop, and project-based experiences focused on finding, evaluating, and using drug information in patient care. This medical app educational experience was integrated into this course. This two-part learning experience involved a prerecorded, Web-based lecture that students were required to view before arriving for a 2-hour, small group workshop of approximately 30 students to apply concepts learned in the lecture. Available for 1 week before the workshop, the 38-minute lecture was created by the faculty member from MCPHS, video recorded via QuickTime, and uploaded onto a private view YouTube profile made available to OSU COP students. The lecture content was based on the role of mobile medical apps in clinical practice, issues and opportunities with utilization of medical apps, and how to review and assess medical apps. The lecture demonstrated to students how to evaluate mobile medical apps with several examples provided. In addition to the lecture, a tool for evaluating medical apps was shared with students; they were required to bring a copy of this tool with them to the workshop. This tool was developed by one of the coinvestigators and previously published incorporating previous strategies published [, ].
Prior to the workshop, students were sent a list of medical apps that could be used free of charge and were available for download to portable electronic devices with iOS or Android operating systems, such as mobile phones or tablets. Students were asked to download these apps to a device, if available, before the workshop. Medical apps were selected by faculty and then reviewed before the start of the workshop. Identification of apps was conducted via previously identified means , with an emphasis on apps related to pharmacy practice, drug information, medical calculators, and general clinical knowledge. Apps were reviewed for both positive and negative qualities to spur discussion among students regarding evaluation techniques during the course of the workshop. The medical apps selected for inclusion in the workshop are detailed in .
|Mobile app||Operating system|
|Medscape||iOS and Android|
|BodyXQ Heart||iOS and Android|
|Psych Drugs||iOS and Android|
|Cardiology Tool by Epocrates||iOS|
|Managing Dabigatran||iOS and Android|
When students arrived to class, a discussion was led by the instructor to review key concepts covered in the previously posted Web-based lecture, including how to use the medical app evaluation tool. The discussion also involved a facilitated conversation about where students find apps and students' experience with apps that had been useful or not useful. Students were split into groups of 4 to 5 students each (6 groups in total per workshop) to evaluate up to 6 medical apps. Each group presented their experience using the evaluation tool to the class and discussed what the individual groups had found with a focus on challenges in evaluating medical apps as well as how to determine what apps are useful in practice.
Evaluation of Learning Experience
Metrics were collected from YouTube Analytics on the number of views and visits received before the start of the workshop. To evaluate the effect of these medical apps on learning experience, prospective surveys were conducted before and after the involvement of first-year pharmacy students in the learning experience (seeand ). The surveys were created by the faculty collaborators on the project from OSU COP and MCPHS to assess changes in students' perceptions regarding how to find, evaluate, and use medical apps in pharmacy. Descriptive questions were asked to characterize the use of portable electronic devices and medical and nonmedical apps by this student population. The first survey included 27 questions involving 5-point Likert scale format (strongly agree to strongly disagree) for perceptions on student confidence with finding, evaluating, and using medical apps. Multiple choice, check all that apply, and open-ended questions were also included to gather information regarding app and device use. The second survey included 18 questions involving the same 5-point Likert scale format questions as well as many of the same multiple choice and check-all-that-apply questions for comparison with the first survey's responses. Demographic data collection about device ownership and population characteristics was not repeated. The second survey additionally asked about satisfaction with the learning experience. Deidentified data were entered into an Excel spreadsheet, then analyzed descriptively in aggregate with summary statistics, including 95% confidence intervals for proportions where appropriate, which were generated in SAS v9.2 (SAS Institute, Cary, NC).
Both surveys were conducted during the workshop as paper surveys. The first survey was conducted on the first day of workshop for the semester. The second survey was administered during the workshop 2 weeks after the medical apps workshop experience.
Pharmacy Student Use of Mobile Applications
The first-year class includes a total of 120 students. There were 96 visits to the YouTube video before the start of the workshop. From the first-year pharmacy class of 120 students, 119 students completed the first survey whereas 120 students completed the second survey. Differences in response total were due to 1 student being absent from class the day the first survey was conducted. Demographics of students gathered are shown inand are typical of a first-year PharmD class across the United States. Of 119 students, 107 (90%) own a mobile phone and 86 (72%) own a portable electronic device.
|Sex, female||75 (63)|
|African American or black||4 (3)|
|Device ownership||114 (96)|
On the basis of the results of the first survey, most students learned about apps via word of mouth, including obtaining information from classmates (96/119, 81%), social media such as Facebook, Twitter, and blogs (58/119, 49%), pharmacy staff at work (44/119, 37%), and family (44/119, 37%;).
|Source||Before (n=119)||After (n=120)|
|n||%||95% CI||n||%||95% CI|
|Friends or classmates in pharmacy school||96||81||74-88||100||83||77-90|
|Friends or classmates from other health professions schools||27||23||15-30||33||27||20-35|
|Medical or pharmacy Staff where I work||44||37||28-46||57||47||39-56|
|None of the above or N/Aa||7||6||2-10||6||5||1-9|
aN/A: not applicable.
In addition, most students reported regularly using 0 to 2 medical or pharmacy-related apps and approximately 55% (65/119) of students indicated currently having a drug information app installed on a portable electronic device ().
Impact of Learning Experience on Student Perceptions of Mobile Applications
Before the workshop, most students indicated that medical or pharmacy-related apps are beneficial to pharmacy practice (98/119, 82%) and that mobile technology will influence pharmacy practice in the future (112/119, 94%). However, less than 44% of students agreed or strongly agreed that they knew how to find (52/119, 44%), evaluate (18/119, 15%), or use medical or pharmacy-related apps (31/119, 26%;).
The students cited lack of knowledge of apps and inability to recognize when it was appropriate to use a mobile device in practice as the two main barriers to using mobile devices in pharmacy practice ().
|Barriers to using apps||Before (n=119)||After (n=120)|
|n||%||95% CI||n||%||95% CI|
|Lack of knowledge of apps||87||73||65-81||69||58||49-66|
|Purchasing a device||35||29||21-38||49||41||32-50|
|Recognizing when it\'s appropriate to use one||66||55||47-64||66||55||46-64|
The second survey's responses indicated changes in students' perceptions in a variety of areas after participating in the medical apps learning experience (). After the workshop, more than 90% of students agreed or strongly agreed that they knew how to find (114/120, 95%), evaluate (112/120, 93%), and use medical apps (108/120, 90%), compared with 43% before the workshop ( ). Most students also reported increased usage of apps than on the first survey with up to 5 medical or pharmacy-related apps regularly used ( ). Eighty percent of students (96/120) indicated that a drug information app was currently installed on one of their mobile devices. The source of information about new apps also changed after the workshop, with students indicating that they relied more on colleagues than social media and family to learn about new apps ( ). Perceptions on benefits and barriers to using mobile devices in pharmacy changed minimally, with the percentage of students agreeing or strongly agreeing that apps benefit pharmacy practice and that mobile technology will influence pharmacy practice increasing from 95% (113/119) to 96% (115/120). Lack of knowledge was indicated as a barrier less often in the second survey compared with the first survey ( ). Also, students' perceptions related to the cost of a medical or pharmacy-related app did not differ between first and second survey data. Most students were willing to pay less than US $2.99 for a medical or pharmacy-related app. Overall, most students found the medical apps presentation and the activities to be useful (at least 70% agreed or strongly agreed, 84/120) and that they would suggest offering the activity in the class next year (74% agreed or strongly agreed, 89/120).
This educational intervention is the first described formal, interactive method to educate pharmacy students on medical apps. Comparative surveys conducted before and after the experience display a perceived improvement in student skills related to finding, evaluating, and using medical apps. Students described satisfaction with the educational experience and agreed that this experience should be repeated in subsequent years. Most students surveyed possessed portable electronic devices, used apps regularly, and agreed with the concept of medical apps being an important part of the health care profession in the future.
Current literature reveals that medical trainees in a variety of practice environments share similar trends and perceptions of medical apps. A study of US urology trainees found that 77% reported downloading apps with 30.6% also paying for them; the mean number of apps downloaded was 4 (range 1-12). Approximately 44% of trainees indicated apps for mobile phones as being very useful in clinical practice . Our pharmacy students reported similar app-downloading habits, with a much higher percentage believing apps are integral to the practice. Methods to evaluate apps have been published, including the tool used in this project [ , , ]. No models for educating health care professionals on finding, evaluating, or using apps have been described. Greater emphasis on educating future health care practitioners in the classroom on the appropriate use of mobile technology has been suggested [ ].
Although our study demonstrated that students felt more comfortable with evaluating mobile medical apps, our results indicated that they still felt they would benefit from greater knowledge on when it is appropriate to use the technology in practice. Although this was not a focus of our study, our results suggest possible benefit from addressing the topic of e-professionalism and the integration of mobile devices into practice in pharmacy and possibly other health sciences curricula.
Limitations of this project relate to the narrow scope of the educational intervention. This intervention occurred in one course at one pharmacy school with first-year pharmacy students who have not yet experienced patient care at the level of more advanced students and may not be ready to apply the experience to real-life medication management in practice. Although the faculty attempted to choose a variety of medical apps to evaluate in one workshop session, we were able to accommodate up to 6 total apps. A greater variety may have allowed for a more in-depth discussion through application to core concepts. Approximately 80% of the class viewed the preparatory Web-based training module before the workshop, though it cannot be determined if each video view was conducted by each student alone or in groups. Despite this, data were analyzed in aggregate and showed overall improved perceptions by the majority of the class. Another limitation to considering the effect of this experience is that faculty evaluated student perception of skills with no formal assessment of student abilities. Data on students' general use of digital technology, including social media, were evaluated as demographic information to describe the population and not assessed in the second survey; thus, investigators are unable to identify any effects this learning experience may have had on this digital technology usage. Additionally, unique identifiers were not collected to match each student's responses for the first and second data collection. Consequently, no formal hypothesis testing for the changes before and after the learning experience could be performed and analyses of the data were limited to descriptive summaries.
Medical apps will be an inevitable component of health care in the 21st century. Pharmacy and other health care professionals must be equipped with the skills to navigate this new open-access world to provide safe and effective recommendations and care of patients. An important element of the portable technology world is medical apps. This paper describes a model for engaging pharmacy students in an active learning experience in finding, evaluating, and using medical apps. This model is transferable to other colleges of pharmacy as well as other health care professional training programs aimed at both current and future practitioners.
Conflicts of Interest
TDA helped author the tool used to evaluate mobile medical apps that was previously published, and he was also a previous editor for iMedicalApps.com where he reviewed and assessed mobile medical apps.
Multimedia Appendix 1
Pre-Survey.PDF File (Adobe PDF File), 78KB
Multimedia Appendix 2
Post-Survey.PDF File (Adobe PDF File), 74KB
- Definition of Medical Apps. 2016. URL: http://dictionary.reference.com/browse/mobile-device [accessed 2016-02-08] [WebCite Cache]
- Kamerow D. Regulating medical apps: which ones and how much? BMJ 2013 Oct;347:f6009. [Medline]
- IMS Institute for Healthcare Informatics. IMS Institute for Healthcare Informatics website. Patient Apps for Improved Healthcare: From Novelty to Mainstream URL: http://www.imshealth.com/en [accessed 2015-12-29] [WebCite Cache]
- Martínez-pérez B, De la torre-díez I, López-coronado M, Sainz-de-abajo B, Robles M, García-gómez J. Mobile clinical decision support systems and applications: a literature and commercial review. J Med Syst 2014;38(1):4.
- Boruff JT, Storie D. Mobile devices in medicine: a survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J Med Libr Assoc 2014 Jan;102(1):22-30 [FREE Full text] [CrossRef] [Medline]
- Robinson RL, Burk MS. Tablet computer use by medical students in the United States. J Med Syst 2013 Aug;37(4):9959. [CrossRef] [Medline]
- Franko OI, Tirrell TF. Smartphone app use among medical providers in ACGME training programs. J Med Syst 2012 Oct;36(5):3135-3139. [CrossRef] [Medline]
- Payne Karl Frederick Braekkan, Wharrad H, Watts K. Smartphone and medical related App use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak 2012;12:121 [FREE Full text] [CrossRef] [Medline]
- O'Connor P, Byrne D, Butt M, Offiah G, Lydon S, Mc IK, et al. Internstheir smartphones: use for clinical practice. Postgrad Med J 2014;90(1060):75-79.
- Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012;14(5):e128 [FREE Full text] [CrossRef] [Medline]
- Mosa Abu Saleh Mohammad, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak 2012;12:67 [FREE Full text] [CrossRef] [Medline]
- Ray SM, Clark S, Jeter JW, Treadway SA. Assessing the impact of mobile technology on order verification during pharmacist participation in patient rounds. Am J Health Syst Pharm 2013 Apr 1;70(7):633-636. [CrossRef] [Medline]
- Wilson C, Wu R, Lo V, Wong G. Effects of smartphones on pharmacist-physician clinical communication. J Pharm Technol 2012;28:234-242.
- Bucklen KW, Abbott BM. Promise and challenges with the use of mobile applications to support and improve patient care: an industry perspective. Clin Pharmacol Ther 2014 May;95(5):469-471. [CrossRef] [Medline]
- Fox BI, Felkey BG. An optimistic but cautionary outlook on mobile health. Hosp Pharm 2013 May;48(5):438-439 [FREE Full text] [CrossRef] [Medline]
- Gagnon L. Time to rein in the “Wild West” of medical apps. CMAJ 2014 May 13;186(8):E247 [FREE Full text] [CrossRef] [Medline]
- Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA 2014 May 14;311(18):1851-1852. [CrossRef] [Medline]
- Pandey A, Hasan S, Dubey D, Sarangi S. Smartphone apps as a source of cancer information: changing trends in health information-seeking behavior. J Cancer Educ 2013 Mar;28(1):138-142. [CrossRef] [Medline]
- Haffey F, Brady Richard R W, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf 2013 Feb;36(2):111-117. [CrossRef] [Medline]
- Cantudo-Cuenca MR, Robustillo-Cortés MA, Cantudo-Cuenca MD, Morillo-Verdugo R. A better regulation is required in viral hepatitis smartphone applications. Farm Hosp 2014;38(2):112-117 [FREE Full text] [Medline]
- Wolf J, Moreau J, Akilov O, Patton T, English JC, Ho J, et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol 2013 Apr;149(4):422-426 [FREE Full text] [CrossRef] [Medline]
- US Department of Health and Human Services, Food and Drug Administration. Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff. 2015. FDA URL: http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf [accessed 2015-12-29] [WebCite Cache]
- Airth-Kindree N, Vandenbark RT. Mobile applications in nursing education and practice. Nurse Educ 2014;39(4):166-169. [CrossRef] [Medline]
- Gaglani S, Topol E. iMedEd: The Role of Mobile Health Technologies in Medical Education. Acad Med 2014 Sep;89(9):1207-1209. [CrossRef]
- Short SS, Lin AC, Merianos DJ, Burke RV, Upperman JS. Smartphones, trainees, and mobile education: implications for graduate medical education. J Grad Med Educ 2014 Jun;6(2):199-202 [FREE Full text] [CrossRef] [Medline]
- Hanrahan C, Aungst T, Cole S. Evaluating Mobile Medical Applications. Bethesda, MD: American Society of Health-System Pharmacists, Inc; 2014.
- Aungst TD, Clauson KA, Misra S, Lewis TL, Husain I. How to identify, assess and utilise mobile medical applications in clinical practice. Int J Clin Pract 2014 Feb;68(2):155-162. [CrossRef] [Medline]
- Nason G, Burke M, Aslam A, Kelly M, Akram C, Giri S, et al. The use of smartphone applications by urology trainees. Surgeon 2015 Oct;13(5):263-266. [CrossRef] [Medline]
- Murfin M. Know your apps: an evidence-based approach to evaluation of mobile clinical applications. J Physician Assist Educ 2013;24(3):38-40. [Medline]
- Gaglani SM, Topol EJ. iMedEd: the role of mobile health technologies in medical education. Acad Med 2014 Sep;89(9):1207-1209 [FREE Full text] [CrossRef] [Medline]
|FDA: Food and Drug Administration|
|MCPHS: Massachusetts College of Pharmacy and Health Sciences|
|OSU COP: The Ohio State University College of Pharmacy|
Edited by P Bamidis; submitted 24.06.15; peer-reviewed by M Murfin, R Robinson; comments to author 28.07.15; revised version received 12.01.16; accepted 19.02.16; published 12.05.16Copyright
©Jennifer Rodis, Aungst TD, Nicole V Brown, Yan Cui, Leonard Tam. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 12.05.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.