Background: Drug-referencing apps are among the most frequently used by emergency health professionals. To date, no study has analyzed the quantity and quality of apps that provide information on emergency drugs.
Objective: This study aimed to identify apps designed to assist emergency professionals in managing drugs and to describe and analyze their characteristics.
Methods: We performed an observational, cross-sectional, descriptive study of apps that provide information on drugs for adult emergency care. The iOS and Android platforms were searched in February 2021. The apps were independently evaluated by 2 hospital clinical pharmacists. We analyzed developer affiliation, cost, updates, user ratings, and number of downloads. We also evaluated the main topic (emergency drugs or emergency medicine), the number of drugs described, the inclusion of bibliographic references, and the presence of the following drug information: commercial presentations, usual dosage, dose adjustment for renal failure, mechanism of action, therapeutic indications, contraindications, interactions with other medicinal products, use in pregnancy and breastfeeding, adverse reactions, method of preparation and administration, stability data, incompatibilities, identification of high-alert medications, positioning in treatment algorithms, information about medication reconciliation, and cost.
Results: Overall, 49 apps were identified. Of these 49 apps, 32 (65%) were found on both digital platforms; 11 (22%) were available only for Android, and 6 (12%) were available only for iOS. In total, 41% (20/49) of the apps required payment (ranging from €0.59 [US $0.64] to €179.99 [US $196.10]) and 22% (11/49) of the apps were developed by non–health care professionals. The mean weighted user rating was 4.023 of 5 (SD 0.71). Overall, 45% (22/49) of the apps focused on emergency drugs, and 55% (27/49) focused on emergency medicine. More than half (29/47, 62%) did not include bibliographic references or had not been updated for more than a year (29/49, 59%). The median number of drugs was 66 (range 4 to >5000). Contraindications (26/47, 55%) and adverse reactions (24/47, 51%) were found in only half of the apps. Less than half of the apps addressed dose adjustment for renal failure (15/47, 32%), interactions (10/47, 21%), and use during pregnancy and breastfeeding (15/47, 32%). Only 6% (3/47) identified high-alert medications, and 2% (1/47) included information about medication reconciliation. Health-related developer, main topic, and greater amount of drug information were not statistically associated with higher user ratings (P=.99, P=.09, and P=.31, respectively).
Conclusions: We provide a comprehensive review of apps with information on emergency drugs for adults. Information on authorship, drug characteristics, and bibliographic references is frequently scarce; therefore, we propose recommendations to consider when developing an app of these characteristics. Future efforts should be made to increase the regulation of drug-referencing apps and to conduct a more frequent and documented review of their clinical content.
Digital technologies are an increasingly relevant resource for health services because they can improve the quality, efficiency, and safety of health care, a particularly relevant issue in the event of emergencies, disasters, and other unplanned care situations . In recent years, there has been a significant increase in the quantity and quality of mobile health apps owing to the efforts made by health professionals and app developers. At the beginning of 2021, almost 50,000 medical apps were available on the main download platforms (Apple App Store and Google Play Store) [ ]. Mobile apps are changing the health care landscape because they facilitate the exchange of information among professionals, researchers, and patients and enable easy access to quality services during clinical practice [ , ].
The need for a quick response is one of the most prominent characteristics of emergency medicine. Examples of the high care burden experienced in emergency departments can be seen in the nearly 130 million visits in 2018 in the United States or the 30 million visits registered each year in Spain [, ]. A variety of apps have been developed in recent years to improve patient care in these departments [ , ]. Medical emergency apps are now a key element of clinical practice as they can be used as clinical decision tools, case management tools, and sources of clinical information. A desirable feature of these apps is that they can be used quickly because of the need to provide a rapid response to the broad spectrum of clinical scenarios occurring in emergency departments. Recent studies on mobile devices and medical apps in emergency rooms [ , ] have shown that the apps most frequently used by emergency health professionals are medical formulary and drug-referencing apps (84.4%), followed by disease diagnosis and management apps (69.5%) [ ].
Health care pressure, stressful situations, and the need for multiple high-alert medications make emergency departments the perfect setting for drug-related problems . Insufficient information on drugs is the most common cause of medication errors, which can lead to adverse drug events involving temporary or permanent harm to patients and higher health care costs [ , ]. The information needed in an emergency department includes multiple drug characteristics such as indications, dosing, administration, pharmaceutical compatibilities, adverse reactions, interactions, and contraindications [ , ]. The usefulness of medical apps as a source of information on drug-related characteristics should be highlighted, although the literature still contains relevant gaps concerning these tools. To date, no study has addressed the quantity and quality of smartphone apps that provide information on emergency drugs.
Therefore, the main objective of this study was to identify apps designed to assist health care professionals in managing drugs for adult emergency care and describe their main characteristics and functionalities. As secondary objectives, we designed a score to estimate the amount of drug information contained in each app and analyzed the relationship between this score and the relevant app characteristics. We also analyzed whether some of the variables selected could affect user satisfaction (app user ratings).
Search Strategy and App Selection
We performed an observational, cross-sectional, descriptive study of smartphone apps available on the iOS and Android platforms that provide information on drugs used for adult emergency care.
The methodology used for app selection was based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) system . To identify emergency drugs–related apps, a search was conducted between February 15 and February 19, 2021, on the digital distribution platforms Google Play Store (Android) and Apple App Store (iOS), which are the app stores with the most apps available at present [ ]. The search terms were “emergency drugs” OR “fármacos de urgencias” and “emergency medicine” OR “medicina de urgencias y emergencias.” We extracted text from app store descriptions and selected apps available in English or Spanish whose content was fully dedicated to drugs commonly used in the emergency room (hereafter referred to as emergency drugs apps) and apps related to the field of emergency medicine that contained a section on medications (emergency medicine apps). Apps aimed at pediatric emergencies were excluded because of relevant differences in the use of drugs in children (eg, dosage, treatment algorithms, and selection). Both free and paid apps were included. Apps from the Google Play Store were downloaded onto a Xiaomi Mi 9 SE (version 9 PKQ1.181121.001; Android), and apps from the Apple App Store were downloaded onto an iPhone 11 (version 14.4; iOS).
No patients were involved in the study and therefore ethical board approval was not sought, as it is considered unnecessary under RD 1090/2015 regulating clinical trials with medicinal products and the Ethics Committees for Research with medicinal products, and Law 14/2007 on Biomedical Research.
We collected the following information from the download platforms: app name, operating system (Android, iOS, or both), developer affiliation, country of origin, language, category, cost, publication date, date of last update, size, version, number of downloads, and user ratings. These indicators are commonly used in studies on health-related apps [- ]. The overall mean weighted user rating was calculated by considering the number of ratings from both app stores. For the rest of the analysis, when the same app was available on both platforms, we only considered the version available on the Google Play Store as Android is the leading operating system worldwide and the Apple App Store provides less information (no data on the number of downloads). Subsequently, all apps were downloaded and their contents were evaluated. We counted the number of drugs included in each app and determined whether they belonged to ≥1 drug classes. We then evaluated whether the apps contained information on the following fifteen drug-related characteristics: (1) commercial presentations, (2) usual dosage, (3) dose adjustment for renal failure, (4) mechanism of action, (5) therapeutic indications, (6) contraindications, (7) interaction with other medicinal products, (8) use in pregnancy and breastfeeding, (9) adverse reactions, (10) method of preparation and administration, (11) stability data and incompatibilities, (12) identification of high-alert medications, (13) positioning in treatment algorithms, (14) information about medication reconciliation, and (15) cost. The selection of these indicators was discussed by the research team based on the most frequent requests received from emergency medicine pharmacy services and drug information centers [ , ]. High-alert medications are defined as drugs that bear a heightened risk of causing significant patient harm when used erroneously [ ]. Medication reconciliation is defined as the formal process in which health care professionals partner with patients to ensure accurate and complete medication information transfer at transitions of care [ ].
We assigned a score of 0 to 15 according to the amount of drug information provided in the app. A score of 0 indicates that the app did not include any information about the 15 drug characteristics analyzed, and a score of 15 indicates that all characteristics were shown in the app. Finally, we also evaluated whether the apps included bibliographic references on drug-related concerns.
All apps were independently evaluated by 2 hospital clinical pharmacists (SGS and BSF). The variables were coded and entered in a Microsoft Excel spreadsheet. The Cohen κ coefficient was calculated using Reliability Calculator for 2 coders  to analyze the level of agreement between the data collected by each investigator. Following this analysis, disagreements on the reported results were resolved through iterative discussion and consensus.
A statistical analysis was performed using Stata (version IC-16; StataCorp). On the basis of previously published studies on mobile health apps, we measured the association between a series of app characteristics (developer, main topic, cost, and number of downloads) and user ratings (which indicate user satisfaction) or the score assigned to the app (which indicates the variety of content on drug information). We also analyzed whether the inclusion of bibliographic references could be influenced by the app developer (health-related or non–health-related). The Shapiro-Wilk test was used to evaluate whether continuous variables were normally distributed. For normally distributed data, differences were assessed using the 2-tailed Student t test for 2 categories and ANOVA for ≥2 categories; for nonnormally distributed data, the Mann-Whitney U test was used. The correlation between quantitative variables was evaluated using the Spearman correlation test. Categorical variables were compared using an uncorrected chi-square test or Fisher exact test, as appropriate. Statistical significance was set at P<.05.
Mobile App Search
Combined keyword searches of the Google Play Store and Apple App Store yielded 645 apps potentially related to emergency drugs. A flow diagram illustrating the selection and exclusion of apps at various stages of the study is shown in. We removed 88 apps as duplicates, with the same app name and developer appearing on both download platforms. The remaining 557 apps were further screened. We extracted information from the store app description and removed 293 apps that were not related to emergency medicine and 20 apps aimed at pediatric emergencies. We then exhaustively analyzed the descriptions of the remaining apps and downloaded them to determine whether the information was inaccurate. From the resulting apps, we removed 4 duplicates with different app names within the same store. We eventually excluded 191 apps that did not contain a specific section on drugs. Following this systematic search, we identified 49 apps that met the inclusion criteria. In total, 65% (32/49) of the apps were found on both digital distribution platforms, whereas 22% (11/49) were obtained only from the Google Play Store, and 12% (6/49) were only available from the Apple App Store.
Two independent researchers (SGS and BSF) further analyzed the characteristics, functionalities, and contents of the 49 apps selected. The mean Cohen κ coefficient for interrater reliability was 0.94 (SD 0.05).
Analysis of General Characteristics of Apps
shows the names of the 49 apps classified by their main topic (emergency drugs or emergency medicine).
By origin, 41% (20/49) of the apps were developed in North America, 35% (17/49) in Europe, 12% (6/49) in South America, 4% (2/49) in Asia, and 2% (1/49) in Africa. The origin of 6% (3/49) of the apps could not be determined. Of the 49 apps analyzed, 27 (55%) were published only in English, 21 (43%) were published only in Spanish, and 1 (2%) was available in both languages. Most apps (44/49, 90%) were classified in the category of medicine. The other categories were health and well-being (3/49, 6%) and education (2/49, 4%).
Slightly more than half of the apps were free to download (29/49, 59%), whereas the other 41% (20/49) required payment, with a cost ranging from €0.59 (US $0.64) to €179.99 (US $196.10) (median €8.99 [US $9.79]) and a mean cost of €20.82 (US $22.68) (SD €40.81 [US $44.46]). Two apps were for the exclusive use of workers at the center where they were developed, and 1 app could only be used with a code acquired after purchasing a book; therefore, they could not be fully analyzed. In addition, the content of 1 app was unavailable because of a download error that affected the latest versions of Android. In these cases, we collected as much information as possible from the description of the app and images available on the digital distribution platforms.
The average size of the apps was 23.89 (SD 23.28) MB. The content of 27% (13/49) of the apps was updated 6 months before the search. A further 14% (7/49) of the apps were updated in the previous year. A total of 59% (29/49) of the apps had not been updated for more than a year; of these, 12 (24% of the overall apps) had not been updated for more than 3 years. A total of 16% (8/49) apps had not been updated since the date of the first publication. The average time between the date of analysis and the date of the most recent update was 23.3 (SD 23.6) months. iOS apps were excluded from this last analysis because of the lack of information on the day of the most recent update.
About half of the apps were developed by private and for-profit organizations (22/49, 45%) as follows: health-related technology companies (n=12, 24%); non–health-related technology companies (n=9, 18%); and medical publishers (n=1, 2%). A total of 22% (11/49) apps were developed by non–health-related professionals. Among the 78% (38/49) apps developed by health care professionals, 29% (14/49) were developed by individual professionals, whereas the rest were developed by technology companies or medical publishers (13/49, 27%), or with the involvement of a health care organization (eg, hospital, public health agency, or professional society; 11/49, 22%). A complete list of developers is provided in.
The number of downloads can only be determined in the apps found in the Google Play Store, as this information is not available in the Apple App Store. The median number of downloads was >5000 (range >1 to >100,000). Detailed information regarding the number of downloads is presented in.
We evaluated the association between the cost of apps and the number of downloads. Owing to the small sample sizes, the number of downloads was broken down for this analysis into 3 categories: 1 to 1000, 1001 to 10,000, and >10,000 downloads. No statistically significant differences were found between the groups (F42=0.24; P=.70).
The analyses of user ratings included 40 apps, as no data were available for 9 apps. The mean overall weighted user rating of apps according to the number of valuations was 4.023 out of 5 (SD 0.71). The average user ratings were almost identical (t38=−0.01; P=.99) for apps developed by health professionals (n=30, mean 4.240, SD 0.707) and non–health professionals (n=10, mean 4.243, SD 0.470). Free apps were rated higher (n=27, mean 4.277, SD 0.680) than paid apps (n=13, mean 4.197, SD 0.621; t38=−2.27; P=.03).
List of emergency drugs apps and emergency medicine apps.
Emergency drugs apps
- 50 Drugs in emergency
- Common 50 drugs for emergency
- Drogas en emergencia y UCI
- ED drugs
- Emergency drugs
- Emergency drugs (Antonio Frontera)
- Emergency drugs (Ferrazza)
- Emergency medication reference
- EMS calculator or EMS drugs fast
- EMS drug cards
- Farmacos de urgencias SES or urgencias SES
- Goteo para vasoactivos
- Guía farmacológica
- Guía URG
- Medicina de urgencias
- Paramedic drug list
- Perfusiones urgencias
- Pocket drug guide EMS or EMS pocket guide
Emergency medicine apps
- AHS EMS MedicalProtocols
- Arritmias urgencias
- Basic emergency care
- Chuletario urgencias extrahospitalarias
- EMAT app
- Emergency central
- Emergency medicine on call
- EMR guide
- EMRA antibiotic guide
- EMRA PressorDex
- EMS ACLS guide
- EMS notes: EMT and paramedic
- EMS pro
- ICU ER facts made Incred quick
- iTox Urgencias intoxicación
- Manual de procedimientos SAMUR
- Médico de urgencias
- My Emergency Department
- Odonto emergencias
- Urgencia HBLT or Guia urgencia HBLT
- Urgencias Extrahospitalarias
- WikEM—Medicina de emergencia
- Zubirán. Manual Terapéutica 7e
|Developer||Value, n (%)|
|Individual health professional||14 (29)|
|Health-related technology company||12 (24)|
|Non–health-related technology company||9 (18)|
|Public health agency||3 (6)|
|Individual non–health professional||2 (4)|
|Medical or pharmaceutical society||2 (4)|
|Other health professional organization||2 (4)|
|Medical publisher||1 (2)|
|Number of downloads||Value, n (%)|
Analysis of Contents of Apps
Approximately half of the apps focused on emergency drugs (22/49, 45%), whereas the rest (27/49, 55%) focused on emergency medicine in a broader sense. We did not find statistically significant differences (z=−1.7; P=.09) between the average user rating of emergency drugs apps (4.163/5; 16 apps) and emergency medicine apps (4.296/5; 24 apps).
The median number of drugs included in the apps was 66 (range 4 to >5000). The apps classified according to the number of drugs analyzed are shown in.
Of 49 apps, 6 (12%) analyzed only a specific class of drugs: antidotes (n=2, 33%), vasopressors (n=2, 33%), antibiotics (n=1, 17%), and antiarrhythmics (n=1, 17%).
|Number of drugs||Value, n (%)|
shows the 15 drug characteristics of the apps analyzed. Most apps included information about therapeutic indications (38/48, 79%) and the most common doses (43/49, 88%). Other drug-related concerns found in more than half of the apps were commercial presentations (27/47, 57%), mechanism of action (26/47, 55%), contraindications (26/47, 55%), method of preparation and administration (25/48, 52%), and adverse reactions (24/47, 51%). Only 17% (8/47) of apps provided data on stability and incompatibilities. Identification of high-alert medications was found in 6% (3/47) of the apps. Information on drug costs was present in only 2% (1/47) of the apps. Similarly, information about medication reconciliation in the emergency room was found in only 2% (1/47) of the apps.
Most apps (29/47, 62%) did not include bibliographic references regarding drug-related concerns. The percentage of apps that included this kind of information was 44% (16/36) in the group of apps developed by health professionals and 18% (2/11) in the group of apps developed by non–health professionals (χ21=2.5; P=.12).
|Drug characteristic||Value, N||Value, n (%)|
|Commercial presentations||47||27 (57)|
|Usual dosage||49||43 (88)|
|Dose adjustment for renal failure||47||15 (32)|
|Mechanism of action||47||26 (55)|
|Therapeutic indications||48||38 (79)|
|Interaction with other medicinal products||47||10 (21)|
|Use in pregnancy and breastfeeding||47||15 (32)|
|Adverse reactions||47||24 (51)|
|Method of preparation and administration||48||25 (52)|
|Stability data and incompatibilities||47||8 (17)|
|Identification of high-alert medications||47||3 (6)|
|Positioning in treatment algorithms||47||19 (40)|
|Information about reconciliation||47||1 (2)|
Analysis of Drug Information Score
We assigned a score of 0 to 15 according to the number of drug characteristics provided in the app. The mean score was 5.89 (SD 2.91). Of the 47 apps, 22 (47%) apps received a score ranging from 0 to 5, a total of 21 (45%) apps received a score from 6 to 10, and 4 (8%) apps received a score from 11 to 13. There was no correlation between this score and the app user ratings (ρ=−0.17; P=.31).
The average score for apps developed by health professionals (n=36, mean 6.00, SD 3.04) was slightly higher than that for apps developed by non–health professionals (n=11, mean 5.55, SD 2.54), although the difference was not significant (t45=−0.45; P=.66). Similarly, no statistically significant differences (t45=−0.78; P=.44) were found between the average score of emergency drugs apps (n=21, mean 5.52, SD 2.75) and emergency medicine apps (n=26, mean 6.19, SD 3.06) or between the average score of free (n=27, mean 5.90, SD 2.91) and paid (n=20, 5.47, SD 3.07) apps (t45=−0.83; P=.40).
Finally, we compared the difference between the number of downloads and drug information score. The average score was 4.57 (SD 1.81) for apps with 1 to 1000 downloads (7/41, 17%), 6.69 (SD 3.28) for apps with 1001 to 10,000 downloads (16/41, 39%), and 6.61 (SD 2.55) for apps with >10,000 downloads (18/41, 44%). No statistically significant differences were found between the groups (F40=1.63; P=.21).
Studies on the content of mobile health apps are increasingly frequent, and apps related to relevant diseases such as cancer or COVID-19 infection have recently been analyzed [, , , ]. Nevertheless, research on apps designed for use in emergency rooms remains insufficient. In this study, we provide a comprehensive and unique review of smartphone apps that provide information on drugs for adult emergency care.
The use of mobile devices by emergency health professionals is common, and apps related to this field of medicine are proliferating [, ]. Emergency rooms are areas where a high volume of patients must be seen within a short period, and work interruptions are very frequent [ ]. In this complex environment, incorrect use of mobile devices can increase the risk of distraction and may affect patient safety [ ]. Nevertheless, when these devices are used properly, they have enormous potential to improve medical practice, for instance, by allowing quick access to relevant and evidence-based information, which facilitates decision-making and can help reduce error rates. In a recent survey of professionals in an emergency department, most respondents found mobile devices useful for better coordinating care among providers and beneficial for patient care [ ].
Principal Findings on General Characteristics and Comparison With Prior Studies
Our study provides a general perspective on apps designed to help health care professionals with drug management for adult emergency care. Given that medication errors are commonly caused by insufficient information on drugs , we analyzed these apps in detail. This is one of the most comprehensive studies of apps aimed at providing information about drugs for health care professionals. Recently, a study identified more than 600 drug-related apps, and approximately two-third of them were categorized within the medication information class [ ]. The authors distinguished among apps for patients, apps for health professionals, and apps that can be used by both groups. Recent studies on patient-focused drug apps have analyzed those that help patients understand and take their medications or those with a medication list function [ , ]. In addition, apps for treatment adherence have been the subject of intensive research [ - ]. Some papers have also been published on apps about drug-drug interactions [ , ]. This is an issue traditionally addressed by health care professionals, although nowadays many apps for checking interactions are intended to be used by patients rather than health care professionals.
Knowledge of the characteristics of drug apps designed to be used exclusively by health care professionals is still limited. Few studies have aimed to analyze the functionalities and content of these apps. A study conducted in 2013 identified 306 apps providing drug reference information and prescribing material, and analyzed cost, updates, user ratings, intended area of use, and medical involvement in app development . More recently, a study published in 2017 compared 8 apps for dosage recommendations, adverse reactions, and drug interactions [ ]. The quality of the apps targeting medication-related problems has been assessed. Of the 59 apps analyzed, 23 (39%) contained medication information features [ ]. Very recently, a study identified 23 drug reference apps with local drug information in Taiwan (including those aimed at both patients and professionals) and analyzed their quality and factors influencing user perceptions [ ]. In the field of emergency care, a recent study analyzed apps for the management of drug poisoning [ ]. Of the 17 apps identified, 14 (82%) presented diagnosis and treatment guides, and 3 (18%) were specifically on antidotes and their dosage.
In our study, we first collected the information available in the app marketplace descriptions (eg, number of downloads and user ratings) before downloading the apps and analyzing their content in detail. This strategy differs from those of other recent studies, in which a greater number of apps were identified but where the analysis was limited to the marketplace description [, , , ]. Among our main findings, we can highlight that 22% (11/49) of the apps were not developed by health care professionals. This is a lower percentage than that reported in other studies on mobile health apps [ , , ]. In 2013, there was no evidence of involvement of health care professionals in the development of 32.7% (100/306) of the apps available to support prescribing practice [ ]. It should also be noted that apps for patient medication management are developed mainly by the software industry, without the involvement of health care professionals [ , ]. Nevertheless, our findings should be considered relevant, given that the apps we analyzed are intended to be used in complex and emergency situations. In addition, information on authorship is scarce in many of the apps evaluated.
More importantly, we found that more than half of the apps (29/47, 62%) did not include bibliographic references or had not been updated for more than a year (29/49, 59%). Our results are in accordance with a previous study analyzing 23 apps with medication information, most of which did not provide supporting references . Of particular concern is the lack of updates in the apps analyzed in our study, as this indicator has worsened compared with the study conducted by Haffey et al [ ], in which 44.4% (136/306) of the apps had either been released or updated within the last 6 months, and a further 24.2% (74/306) within 1 year [ ]. These concerns raise doubts about the quality and reliability of the information provided by these apps aimed at emergency health care professionals, as incorrect drug information may remain for long periods.
Doubts arise when a health app is developed by non–health professionals [, ]. We found that bibliographic references were included in 44% (16/36) of the apps developed by health professionals and in only 18% (2/11) of the apps developed by non–health professionals. This result was not statistically significant (P=.12), probably because of the small sample size, although it highlights the uncertainty surrounding the sources of information provided in apps developed by non–health professionals. The reliability and authority of information should be analyzed by health care professionals who are more capable of evaluating, reviewing, and verifying the content of health-related apps. In the field of medication, pharmacists should play a vital role in reviewing apps.
About half of the apps (20/49, 41%) required payment to access all the content, with a cost ranging from €0.59 (US $0.64) to €179.99 (US $196.10). This is a similar percentage than that observed in a recent study on drug poisoning management apps . Nevertheless, it is considerably higher than that observed in other recent reviews of apps for medical emergencies [ ], medication management and adherence for patients [ , ], or checking for drug-drug interactions [ ]. We hypothesize that these differences could arise because the apps analyzed in our study are aimed exclusively at health care professionals and are designed for use in health care facilities. A study conducted in 2013 on apps to support drug prescribing or provide pharmacology education showed that 68% (208/306) of the apps required payment, with a mean price of £14.25 (US $18.57) per app and a range of £0.62 (US $0.81) to £101.90 (US $132.76) [ ]. The cost of apps also seems to be influenced by the origin of the developer [ ]. In any case, cost is an important determinant in the decision to adopt a mobile health app, regardless of age group and socioeconomic status [ ]. In addition, payment for the apps analyzed in our study could be a relevant limitation for health care professionals who only occasionally work in emergency rooms, as is common in many hospitals.
To date, few studies have analyzed the factors that influence user satisfaction with apps [, , ]. The number of downloads and user ratings are usually correlated and have been proposed as indicators of acceptability and satisfaction with mobile health apps [ , ]. A secondary objective of our study was to learn more about user behavior with emergency medicine apps, for which we analyzed whether factors such as cost, the main theme of the app (emergency medicine or emergency drugs), or the app developer (health-related or non–health related) could influence user ratings. The free apps analyzed in our study had higher user ratings than paid apps, although no association was found between the cost and number of downloads. We found no further statistically significant differences, probably because of the small sample size. The number of downloads and user ratings probably depend on multiple factors. Navigation, performance, visual appeal, credibility, and quantity of information have recently been identified as the most influential factors on higher user ratings in a study analyzing 23 drug reference apps [ ]. Previous studies have reported highly variable results for the influence of expert involvement in app development on user ratings and the number of downloads [ , , ]. In any case, user ratings and downloads should not be considered good predictors of the quality and reliability of medical apps because they could be influenced by other factors, such as low price, in-app purchase options, in-app advertisements, and recent updates [ , , ]. In our study, the number of downloads, cost, and user ratings were not associated with a score created to quantify the variety of relevant information on drug characteristics in the apps (P=.21, P=.40, and P=.31, respectively). Further research should analyze the reliability of the clinical content of drug information apps and corroborate its association with a greater intention to use or better user satisfaction.
Drug Information Gaps
At present, there are no standardized guidelines for assessing the clinical content and quality of mobile health apps . A highly specific quality assessment tool was developed to assess the quality of apps targeting medication-related problems, including those with medication information features [ ]. Nevertheless, the most commonly used methodology to assess the quality of medical apps is the Mobile Application Rating Scale [ - ], as well as in studies on drug apps [ , , , ]. The total number of features has been associated with the total Mobile Application Rating Scale score in a study on apps for potential drug-drug interaction decision support [ ].
In our study, we paid special attention to the information provided by the apps regarding relevant drug characteristics. We found that most apps included information about the usual dosage (43/49, 88%) and therapeutic indications (38/48, 79%). Nevertheless, other relevant characteristics were found in less than half of the apps, such as dose adjustment for renal failure (15/47, 32%) and use in pregnancy and breastfeeding (15/47, 32%). Interaction with other medicinal products was found in only 21% (10/47) of the apps, despite being a major problem in patient safety. Drug-drug interaction checks are one of the most frequent functional categories within the current medication-related app landscape [, ], but relevant quality and accuracy problems have been detected in apps, including this feature [ , ]. In addition, other relevant information on drug safety, such as contraindications (26/47, 55%) and adverse reactions (24/47, 51%), was found in approximately half of the apps analyzed in our study. Furthermore, it is worrying that only 6% (3/47) of the apps clearly identified high-alert medications, despite efforts made to avoid errors with these drugs [ ].
In clinical practice, many medication-related inquiries are about the method of administration; however, our study showed that this information is included in slightly more than half of the apps (25/48, 52%). In addition, stability data and incompatibilities were present in only 17% (8/47) of the apps. Nurses have also been reported to be frequent app users in daily practice, albeit at a slightly lower percentage than that observed by physicians . Therefore, apps for the use of drugs in the emergency department should be designed to provide more information on drug administration characteristics.
Finally, incorrect medication reconciliation in the emergency department can lead to relevant medication errors . We found only 1 app that appropriately addressed this issue, including information on the maximum time to carry out reconciliation or the possible presence of withdrawal syndrome. Given that medication reconciliation has been considered the most relevant activity carried out by pharmacists in emergency departments [ ], it would be desirable for apps related to emergency drugs to provide more information on this matter.
Recommendations for Development of an Emergency Drugs App
There are a growing number of health apps on the market with highly variable designs and content, and it is difficult to determine which are the most useful for health care professionals. Given the relative absence of legislation on medical apps  and the risks associated with drugs used in the emergency room, it would be interesting to propose a series of improvements in the content of apps for emergency drugs. The results of our study and clinical experience enable us to make several recommendations.
Design, ease of use, and the ability to quickly respond to questions that arise during daily clinical practice are especially relevant characteristics, considering that these apps are to be used in a stressful environment. The success of an app for emergency professionals depends on quickly obtaining a reliable response.
Our findings could help developers design apps that provide drug-related information most frequently demanded by health care professionals. Drug information centers have historically received the most inquiries regarding therapeutic indications, adverse reactions, and identification of medical products . In addition, information on contraindications, appropriate dosage, and major drug-drug interactions should be included to prevent major adverse events [ ]. We provided a score to measure the amount of drug information included in each app, and our results showed that a greater amount of information is not necessarily associated with better user ratings. Therefore, it could be beneficial to design apps with content aimed exclusively at doctors and apps for nurses, although with maximum information of interest for each of these professionals. For example, apps with information on drug administration and incompatibilities would have the potential to help nursing staff by reducing their workload and, ultimately, the risk of drug-related errors. Strategies to identify high-alert medications should be included in all emergency drug apps, regardless of the group of health care professionals they focus on [ ].
We recommend caution with respect to the sources of information used to elaborate the content of the app to ensure that it is reliable. Apps should only be considered reliable based on an extensive literature review, expert panel review, or peer review. The author’s affiliation and bibliographic references to scientific and clinical evidence should always be clearly shown , and health professionals participating in reviewing and app updates should be clearly identified.
As previously suggested , we believe that future legislation should require a more comprehensive description of the mobile app marketplace, with detailed information on authorship and the process used to review app functionalities and the clinical information provided. All information must be supported by appropriate bibliographic references, and developers should preferably be clinicians with experience writing or synthesizing medical evidence. Thus, the information provided, which should be checked by independent reviewers or endorsed by health organizations of recognized prestige, will be more reliable. In addition, we suggest that app developers clearly identify the target user group and provide the maximum amount of drug information relevant to each professional category. It may also be relevant for a partner with a technology company to make apps more attractive and user-friendly.
Limitations and Future Research Directions
First, our study was limited by the inclusion criteria. There are hundreds (perhaps thousands) of apps providing drug-related information, and some may be useful for emergency room professionals; however, they were not analyzed in this study because our aim was to review apps specifically related to emergency drugs or medicine in adults. Drug information indicators were selected and analyzed by the authors and were therefore not validated. A more comprehensive analysis of drug information apps may be the subject of future research, for which our methodology could prove useful. Our approach could be adapted to analyze apps related to child health care or to include indicators not described in our study, such as information on pharmacokinetic properties, therapeutic drug monitoring, and pharmacogenomics. Other limitations are associated with the study design. We only analyzed the Android version when the same app was available on Android and iOS platforms. It should be noted that some characteristics, such as the date of the last update, may vary among platforms. In addition, we analyzed apps in English and Spanish. Although Spanish is the language with the second highest number of native speakers, many health professionals are not sufficiently competent in the language to use these apps comfortably. Our study was also limited by the fact that it only analyzed whether a series of drug characteristics of interest were included in the app. Further research is needed to evaluate the clinical accuracy of the drug information provided by the apps. One possible approach would be a peer-review process to evaluate app contents in terms of the reliability and quality of information according to the best available clinical evidence.
We conducted a comprehensive and unique systematic review of apps that provide information on drugs for adult emergency care. We identified 49 apps according to the PRISMA methodology and conducted a content analysis on most of them. Health-related app developers, the main topic of the app (emergency drugs or emergency medicine), and a greater amount of drug information were not associated with higher app user ratings. Slightly less than half of the apps (20/49, 41%) required payment, with a cost ranging from €0.59 (US $0.64) to €179.99 (US $196.10). We noted that 22% (11/49) of the apps were not developed by health care professionals. Most apps include information about the usual dosage and therapeutic indications, although information on safety and drug administration is much less frequent. Very few apps provide relevant information, such as high-alert medication notices and instructions for drug reconciliation. In addition, more than half of the apps (29/47, 62%) did not include bibliographic references. These findings cast doubts on the quality of many apps. Therefore, we propose a series of issues that should be considered when developing an app of these characteristics and advocate for greater regulation and more frequent and documented review of app content.
The authors would like to thank Thomas O’Boyle for editing and proofreading the manuscript.
This study was the result of a collaboration between all the authors. SGS and BSF contributed equally to this work. SGS designed the study, supervised and performed the data collection, and drafted the manuscript. BSF conducted the data analysis and substantially contributed to data collection and drafting of the manuscript. AdLP and CON conceived the original research idea and made a considerable contribution to the design of the study. All authors critically revised the manuscript and approved its final version for publication.
Conflicts of Interest
- mHealth: use of appropriate digital technologies for public health: report by the Director-General. World Health Organization. 2017. URL: https://apps.who.int/iris/handle/10665/274134 [accessed 2022-04-01]
- Number of mHealth apps available at Google Play and App Store from 1st quarter 2015 to 3rd quarter 2020. Statista. URL: https://www.statista.com/statistics/779919/health-apps-available-google-play-worldwide/ [accessed 2022-04-01]
- Alonso SG, Marques G, Barrachina I, Garcia-Zapirain B, Arambarri J, Salvador JC, et al. Telemedicine and e-Health research solutions in literature for combatting COVID-19: a systematic review. Health Technol (Berl) 2021;11(2):257-266 [FREE Full text] [CrossRef] [Medline]
- de la Torre Díez I, Alonso SG, Hamrioui S, López-Coronado M, Cruz EM. Systematic review about QoS and QoE in telemedicine and eHealth services and applications. J Med Syst 2018 Aug 29;42(10):182. [CrossRef] [Medline]
- Emergency department visits. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/nchs/fastats/emergency-department.htm [accessed 2022-04-01]
- Informe Anual del Sistema Nacional de Salud 2018 Resumen ejecutivo. Ministerio de Sanidad. URL: https://www.mscbs.gob.es/estadEstudios/estadisticas/sisInfSanSNS/tablasEstadisticas/InfAnualSNS2018/ResumenEjecutivo2018.pdf [accessed 2022-04-01]
- Farmer B. Patient safety in the emergency department. Emerg Med 2016 Sep 01;48(9):396-404. [CrossRef]
- Plaza Roncero A, Marques G, Sainz-De-Abajo B, Martín-Rodríguez F, Del Pozo Vegas C, Garcia-Zapirain B, et al. Mobile health apps for medical emergencies: systematic review. JMIR Mhealth Uhealth 2020 Dec 11;8(12):e18513 [FREE Full text] [CrossRef] [Medline]
- Alameddine M, Soueidan H, Makki M, Tamim H, Hitti E. The use of smart devices by care providers in emergency departments: cross-sectional survey design. JMIR Mhealth Uhealth 2019 May 05;7(6):e13614 [FREE Full text] [CrossRef] [Medline]
- Hitti E, Hadid D, Melki J, Kaddoura R, Alameddine M. Mobile device use among emergency department healthcare professionals: prevalence, utilization and attitudes. Sci Rep 2021 Jan 21;11(1):1917 [FREE Full text] [CrossRef] [Medline]
- Hammond DA, Gurnani PK, Flannery AH, Smetana KS, Westrick JC, Lat I, et al. Scoping review of interventions associated with cost avoidance able to be performed in the intensive care unit and emergency department. Pharmacotherapy 2019 Mar;39(3):215-231. [CrossRef] [Medline]
- Ortmann MJ, Johnson EG, Jarrell DH, Bilhimer M, Hayes BD, Mishler A, et al. ASHP guidelines on emergency medicine pharmacist services. Am J Health Syst Pharm 2021 Jan 22;78(3):261-275. [CrossRef] [Medline]
- Rosenberg JM, Koumis T, Nathan JP, Cicero LA, McGuire H. Current status of pharmacist-operated drug information centers in the United States. Am J Health Syst Pharm 2004 Oct 01;61(19):2023-2032. [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009 Aug 18;151(4):264-9, W64 [FREE Full text] [CrossRef] [Medline]
- Mobile operating system market share worldwide. Statcounter Global Stats. URL: https://gs.statcounter.com/os-market-share/mobile/worldwide [accessed 2022-04-01]
- Charbonneau DH, Hightower S, Katz A, Zhang K, Abrams J, Senft N, et al. Smartphone apps for cancer: a content analysis of the digital health marketplace. Digit Health 2020;6:2055207620905413 [FREE Full text] [CrossRef] [Medline]
- Collado-Borrell R, Escudero-Vilaplana V, Villanueva-Bueno C, Herranz-Alonso A, Sanjurjo-Saez M. Features and functionalities of smartphone apps related to COVID-19: systematic search in app stores and content analysis. J Med Internet Res 2020 Aug 25;22(8):e20334 [FREE Full text] [CrossRef] [Medline]
- Biviji R, Vest JR, Dixon BE, Cullen T, Harle CA. Factors related to user ratings and user downloads of mobile apps for maternal and infant health: cross-sectional study. JMIR Mhealth Uhealth 2020 Jan 24;8(1):e15663 [FREE Full text] [CrossRef] [Medline]
- Aydin G, Silahtaroglu G. Insights into mobile health application market via a content analysis of marketplace data with machine learning. PLoS One 2021;16(1):e0244302 [FREE Full text] [CrossRef] [Medline]
- High-alert medications in acute care settings. Institute for Safe Medication Practices. URL: https://www.ismp.org/recommendations/high-alert-medications-acute-list [accessed 2022-04-01]
- Assuring medication accuracy at transitions in care: medication reconciliation. The High 5s Project. URL: https://www.who.int/patientsafety/implementation/solutions/high5s/h5s-guide.pdf?ua=1 [accessed 2022-04-01]
- Freelon D. ReCal: intercoder reliability calculation as a web service. Int J Internet Sci 2010;5(1):20 [FREE Full text]
- Collado-Borrell R, Escudero-Vilaplana V, Ribed-Sánchez A, Ibáñez-García S, Herranz-Alonso A, Sanjurjo-Sáez M. Smartphone applications for cancer patients; what we know about them? Farm Hosp 2016 Jan 01;40(1):25-35 [FREE Full text] [CrossRef] [Medline]
- Kondylakis H, Katehakis DG, Kouroubali A, Logothetidis F, Triantafyllidis A, Kalamaras I, et al. COVID-19 mobile apps: a systematic review of the literature. J Med Internet Res 2020 Dec 09;22(12):e23170 [FREE Full text] [CrossRef] [Medline]
- Gaziel-Yablowitz M, Schwartz DG. A review and assessment framework for mobile-based emergency intervention apps. ACM Comput Surv 2019 Jan 31;51(1):1-32. [CrossRef]
- Chisholm CD, Dornfeld AM, Nelson DR, Cordell WH. Work interrupted: a comparison of workplace interruptions in emergency departments and primary care offices. Ann Emerg Med 2001 Aug;38(2):146-151. [CrossRef] [Medline]
- Degelsegger-Márquez A, Trunner K, Piso B. Medication apps - a systematic search and classification. Stud Health Technol Inform 2020 Jun 23;271:207-214. [CrossRef] [Medline]
- Tabi K, Randhawa AS, Choi F, Mithani Z, Albers F, Schnieder M, et al. Mobile apps for medication management: review and analysis. JMIR Mhealth Uhealth 2019 Sep 11;7(9):e13608 [FREE Full text] [CrossRef] [Medline]
- Diaz-Skeete YM, McQuaid D, Akinosun AS, Ekerete I, Carragher N, Carragher L. Analysis of apps with a medication list functionality for older adults with heart failure using the mobile app rating scale and the IMS institute for healthcare informatics functionality score: evaluation study. JMIR Mhealth Uhealth 2021 Nov 02;9(11):e30674 [FREE Full text] [CrossRef] [Medline]
- Santo K, Richtering SS, Chalmers J, Thiagalingam A, Chow CK, Redfern J. Mobile phone apps to improve medication adherence: a systematic stepwise process to identify high-quality apps. JMIR Mhealth Uhealth 2016 Dec 02;4(4):e132 [FREE Full text] [CrossRef] [Medline]
- Ahmed I, Ahmad NS, Ali S, Ali S, George A, Saleem Danish H, et al. Medication adherence apps: review and content analysis. JMIR Mhealth Uhealth 2018 Mar 16;6(3):e62 [FREE Full text] [CrossRef] [Medline]
- Park JY, Li J, Howren A, Tsao NW, De Vera M. Mobile phone apps targeting medication adherence: quality assessment and content analysis of user reviews. JMIR Mhealth Uhealth 2019 Jan 31;7(1):e11919 [FREE Full text] [CrossRef] [Medline]
- Pérez-Jover V, Sala-González M, Guilabert M, Mira JJ. Mobile apps for increasing treatment adherence: systematic review. J Med Internet Res 2019 Jun 18;21(6):e12505 [FREE Full text] [CrossRef] [Medline]
- Armitage LC, Kassavou A, Sutton S. Do mobile device apps designed to support medication adherence demonstrate efficacy? A systematic review of randomised controlled trials, with meta-analysis. BMJ Open 2020 Jan 30;10(1):e032045 [FREE Full text] [CrossRef] [Medline]
- Al-Arkee S, Mason J, Lane DA, Fabritz L, Chua W, Haque MS, et al. Mobile apps to improve medication adherence in cardiovascular disease: systematic review and meta-analysis. J Med Internet Res 2021 May 25;23(5):e24190 [FREE Full text] [CrossRef] [Medline]
- Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer mobile apps for potential drug-drug interaction check: systematic review and content analysis using the mobile app rating scale (MARS). JMIR Mhealth Uhealth 2018 Mar 28;6(3):e74 [FREE Full text] [CrossRef] [Medline]
- Shen C, Jiang B, Yang Q, Wang C, Lu KZ, Gu M, et al. Mobile apps for drug-drug interaction checks in chinese app stores: systematic review and content analysis. JMIR Mhealth Uhealth 2021 Jun 15;9(6):e26262 [FREE Full text] [CrossRef] [Medline]
- Haffey F, Brady RR, Maxwell S. Smartphone apps to support hospital prescribing and pharmacology education: a review of current provision. Br J Clin Pharmacol 2014 Jan;77(1):31-38 [FREE Full text] [CrossRef] [Medline]
- Apidi NA, Murugiah MK, Muthuveloo R, Soh YC, Caruso V, Patel R, et al. Mobile medical applications for dosage recommendation, drug adverse reaction, and drug interaction: review and comparison. Ther Innov Regul Sci 2017 Jul;51(4):480-485. [CrossRef] [Medline]
- Loy JS, Ali EE, Yap KY. Quality assessment of medical apps that target medication-related problems. J Manag Care Spec Pharm 2016 Oct;22(10):1124-1140. [CrossRef] [Medline]
- Chen Y, Liao W, Su M, Lin Y. Personalized and self-management: systematic search and evaluation quality factors and user preference of drug reference apps in Taiwan. J Pers Med 2021 Aug 12;11(8):790 [FREE Full text] [CrossRef] [Medline]
- Hajesmaeel Gohari S, Bahaadinbeigy K, Tajoddini S, R Niakan Kalhori S. Drug poisoning management using smartphones: an apps review study based on use case classification model. BMJ Innov 2020 Jul 06;6(4):246-251. [CrossRef]
- Subhi Y, Bube SH, Rolskov Bojsen S, Skou Thomsen AS, Konge L. Expert involvement and adherence to medical evidence in medical mobile phone apps: a systematic review. JMIR Mhealth Uhealth 2015 Jul 27;3(3):e79 [FREE Full text] [CrossRef] [Medline]
- van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore the sanity. BMC Med Inform Decis Mak 2013 Feb 11;13:23 [FREE Full text] [CrossRef] [Medline]
- Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
- Peng W, Kanthawala S, Yuan S, Hussain SA. A qualitative study of user perceptions of mobile health apps. BMC Public Health 2016 Nov 14;16(1):1158 [FREE Full text] [CrossRef] [Medline]
- Mendiola MF, Kalnicki M, Lindenauer S. Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings. JMIR Mhealth Uhealth 2015 May 13;3(2):e40 [FREE Full text] [CrossRef] [Medline]
- Carlo AD, Hosseini Ghomi R, Renn BN, Areán PA. By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med 2019;2:54 [FREE Full text] [CrossRef] [Medline]
- Pereira-Azevedo N, Osório L, Cavadas V, Fraga A, Carrasquinho E, Cardoso de Oliveira E, et al. Expert involvement predicts mHealth app downloads: multivariate regression analysis of urology apps. JMIR Mhealth Uhealth 2016 Jul 15;4(3):e86 [FREE Full text] [CrossRef] [Medline]
- Huang H, Bashir M. Users' adoption of mental health apps: examining the impact of information cues. JMIR Mhealth Uhealth 2017 Jun 28;5(6):e83 [FREE Full text] [CrossRef] [Medline]
- Ghose A, Han SP. Estimating demand for mobile applications in the new economy. Manag Sci 2014 Jun;60(6):1470-1488. [CrossRef]
- Plante TB, O'Kelly AC, Macfarlane ZT, Urrea B, Appel LJ, Miller Iii ER, et al. Trends in user ratings and reviews of a popular yet inaccurate blood pressure-measuring smartphone app. J Am Med Inform Assoc 2018 Aug 01;25(8):1074-1079 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Amor-García MÁ, Collado-Borrell R, Escudero-Vilaplana V, Melgarejo-Ortuño A, Herranz-Alonso A, Arija A, et al. Assessing apps for patients with genitourinary tumors using the mobile application rating scale (MARS): systematic search in app stores and content analysis. JMIR Mhealth Uhealth 2020 Jul 23;8(7):e17609 [FREE Full text] [CrossRef] [Medline]
- Woulfe F, Fadahunsi KP, Smith S, Chirambo GB, Larsson E, Henn P, et al. Identification and evaluation of methodologies to assess the quality of mobile health apps in high-, low-, and middle-income countries: rapid review. JMIR Mhealth Uhealth 2021 Oct 12;9(10):e28384 [FREE Full text] [CrossRef] [Medline]
- Hammar T, Hamqvist S, Zetterholm M, Jokela P, Ferati M. Current knowledge about providing drug-drug interaction services for patients-a scoping review. Pharmacy (Basel) 2021 Mar 24;9(2):69 [FREE Full text] [CrossRef] [Medline]
- Santell JP. Reconciliation failures lead to medication errors. Jt Comm J Qual Patient Saf 2006 Apr;32(4):225-229. [CrossRef] [Medline]
- García-Martín Á, Maroun-Eid C, Campino-Villegas A, Oliva-Manuel B, Herrero-Ambrosio A, Quintana-Díaz M. Perception survey on the value of the hospital pharmacist at the emergency department. Farm Hosp 2017 May 01;41(3):357-370. [CrossRef] [Medline]
- Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med 2020;3:14 [FREE Full text] [CrossRef] [Medline]
- Health on the net foundation. HONcode principles. URL: https://www.hon.ch/HONcode/Webmasters/Conduct.html [accessed 2022-04-01]
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
Edited by L Buis; submitted 27.04.21; peer-reviewed by E Hitti, L Collado-Yurrita, V Cavadas, V Escudero-Vilaplana, H Soueidan; comments to author 22.11.21; revised version received 15.02.22; accepted 18.02.22; published 20.04.22Copyright
©Sebastián García-Sánchez, Beatriz Somoza-Fernández, Ana de Lorenzo-Pinto, Cristina Ortega-Navarro, Ana Herranz-Alonso, María Sanjurjo. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 20.04.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.