RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 06.11.18 in Vol 6, No 11 (2018): November

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/9651, first published Dec 13, 2017.

This paper is in the following e-collection/theme issue:

    Original Paper

    Medical Correctness and User Friendliness of Available Apps for Cardiopulmonary Resuscitation: Systematic Search Combined With Guideline Adherence and Usability Evaluation

    Department of Anaesthesiology, University Medicine Greifswald, Greifswald, Germany

    Corresponding Author:

    Bibiana Metelmann, MD

    Department of Anaesthesiology

    University Medicine Greifswald

    Ferdinand-Sauerbruch-Straße

    Greifswald, 17489

    Germany

    Phone: 49 38348760 ext 2848

    Fax:49 3834876092848

    Email:


    ABSTRACT

    Background: In case of a cardiac arrest, start of cardiopulmonary resuscitation by a bystander before the arrival of the emergency personnel increases the probability of survival. However, the steps of high-quality resuscitation are not known by every bystander or might be forgotten in this complex and time-critical situation. Mobile phone apps offering real-time step-by-step instructions might be a valuable source of information.

    Objective: The aim of this study was to examine mobile phone apps offering real-time instructions in German or English in case of a cardiac arrest, to evaluate their adherence to current resuscitation guidelines, and to test their usability.

    Methods: Our 3-step approach combines a systematic review of currently available apps guiding a medical layperson through a resuscitation situation, an adherence testing to medical guidelines, and a usability evaluation of the determined apps. The systematic review followed an adapted preferred reporting items for systematic reviews and meta-analyses flow diagram, the guideline adherence was tested by applying a conformity checklist, and the usability was evaluated by a group of mobile phone frequent users and emergency physicians with the system usability scale (SUS) tool.

    Results: The structured search in Google Play Store and Apple App Store resulted in 3890 hits. After removing redundant ones, 2640 hits were checked for fulfilling the inclusion criteria. As a result, 34 apps meeting all inclusion criteria were identified. These included apps were analyzed to determine medical accuracy as defined by the European Resuscitation Council’s guidelines. Only 5 out of 34 apps (15%, 5/34) fulfilled all criteria chosen to determine guideline adherence. All other apps provided no or wrong information on at least one relevant topic. The usability of 3 apps was evaluated by 10 mobile phone frequent users and 9 emergency physicians. Of these 3 apps, solely the app “HELP Notfall” (median=87.5) was ranked with an SUS score above the published average of 68. This app was rated significantly superior to “HAMBURG SCHOCKT” (median=55; asymptotic Wilcoxon test: z=−3.63, P<.01, n=19) and “Mein DRK” (median=32.5; asymptotic Wilcoxon test: z=−3.83, P<.01, n=19).

    Conclusions: Implementing a systematic quality control for health-related apps should be enforced to ensure that all products provide medically accurate content and sufficient usability in complex situations. This is of exceptional importance for apps dealing with the treatment of life-threatening events such as cardiac arrest.

    JMIR Mhealth Uhealth 2018;6(11):e190

    doi:10.2196/mhealth.9651

    KEYWORDS



    Introduction

    Background

    In case of a cardiac arrest, cardiopulmonary resuscitation (CPR) has to start as soon as possible [1]. However, even in the most advanced emergency systems, the emergency personnel needs a medium time of 5-8 min to arrive at the emergency site [2]. Therefore, it is crucial that a bystander—that is, a person noticing the cardiac arrest—starts CPR [3,4]. In the majority of cases, the bystander will be a medical layperson [5]. It has been shown in multiple studies that bystander CPR increases the rate of survival [1,6-9]. Nevertheless, the rate of bystander CPR is still relatively low [10]. Various reasons for this gap are discussed [11,12]. One of them might be that the bystander has probably never experienced a similar situation before and is therefore unsure what to do and fears to make mistakes [13]. The situation being highly time-critical further increases the cognitive workload. Decisions have to be made fast, leaving no time for elaborate reflections. It has been shown that cognitive aids can help reduce stress in these types of situations [14]. However, because cardiac arrests can happen anytime and at any place, it is unlikely that the bystander carries a traditional cognitive aid such as a textbook or leaflet with the required information.

    Objectives

    A possible solution to allow immediate assistance might be a mobile phone app offering real-time step-by-step instructions. Mobile electronic devices such as mobile phones are ubiquitously available and a bystander is likely to have them available at site [15]. Mobile phone apps have become a part of everyday culture and have changed daily life in nearly all aspects [16]. Especially digital natives are used to being able to receive information immediately; internet research and apps are their first choice in cases of questions and often the main source of information [16,17]. Likewise, the market of mHealth apps has grown exponentially over the last years [18-20]. Cognitive aids, which are based on medical guidelines, are increasingly accepted in health care [21]. Ahn et al described that the total number of downloads for CPR training apps is about several hundred thousand [22]. However, this field is getting increasingly complex and unmanageable [23]. Kumar et al raised concerns toward untested apps already at the mHealth Evidence Workshop at the US National Institutes of Health in 2011 [24]. They demanded rigorous research to examine the potentially negative consequences of ineffective mHealth apps or apps based on incorrect facts. This could lead to patient harm and higher medical costs [24]. It cannot be expected from medical laypersons to analyze all possible apps, evaluate the content, and decide whether it conforms to current medical guidelines.

    The aim of this study was to systematically detect apps giving German or English step-by-step instructions to perform CPR by an adult bystander and determine both their adherence to current medical guidelines and their usability.


    Methods

    Study Setup

    Our 3-step approach combined (1) a systematic review of currently available apps guiding a medical layperson through a resuscitation situation, (2) an adherence testing to medical guidelines, and (3) a usability evaluation of the determined apps. The study was approved by the institutional review board of Universitätsmedizin Greifswald with the case number BB 055/17.

    Systematic Review of Available Apps

    To date, there is no standardized search method for identifying mobile health apps. We used an approach similar to other studies [25-27]. The search was structured to an adapted preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram [28].

    We focused on the 2 largest and most popular stores for mobile apps—Apple App Store (for Apple iOS apps) and the Google Play Store (for Android apps). Smaller stores such as Amazon App store, Windows Store, Samsung Apps, or Blackberry World were not included in this study [29-31]. The apps offered in these 2 app stores are automatically preselected depending on the region from where the search is conducted. Google Play Store identifies the location based on the users’ IP address, whereas Apple App Store uses Apple ID. This default country setting can be changed [32,33]. An extensive search with country settings of all English-speaking countries would have led to an unmanageable amount of apps; whereas a restriction to just a few selected English-speaking countries would have been arbitrary. Therefore, we decided to restrict the search to the country setting of Germany. We defined 16 keywords and hand-searched each term separately. These keywords were the English words “CPR,” “resuscitation,” “chest compression,” “basic life support,” “BLS,” “first aid,” “cardiac arrest,” “112,” and similar German expressions (“Reanimation,” “Wiederbelebung,” “Thoraxkompression,” “Herzdruckmassage,” “Erste Hilfe,” “Herzstillstand,” “Kreislaufstillstand,” and “Notfall”). The systematic search was carried out on a MacBook Pro between May 26, 2017, and June 23, 2017. For Apple App Store, the iTunes search configuration was set to “all” (Mac, iPad, iPhone, and Apple Watch). Google Play Store was searched on the same MacBook Pro [34] accessed via Safari internet browser. Consistent with other studies, apps were identified if the keyword was either part of the title or the description of the app [27].

    All identified apps were screened. Apps that were found under different keywords but had the same name and were developed by the same company were considered redundant.

    For assessment of eligibility, the remaining apps were examined regarding the study’s inclusion criteria. Apps not coherent with 1 or more of these criteria were excluded from further evaluation. The following inclusion criteria were used in a descending rank order: availability on both Google Play Store and Apple App Store, language of the app either German or English, free of charge, covered the topic resuscitation, resuscitation of human beings, provides real-time step-by-step instructions, no duplicate under different names, and no technical problems. The inclusion criterion “availability in both stores” was chosen to find an app that can be recommended in, for example, basic life support trainings and is usable by the majority of mobile phone users without being restricted to a subgroup. Apps were classified as duplicates if they were developed by the same company and had the same interface but had different names. All remaining apps were included for evaluation of guideline adherence.

    Conformity to Guideline

    We analyzed the quality of content based on the adherence to the European Resuscitation Council Guidelines for Resuscitation 2015 [2], the American Heart Association Guidelines Update for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care 2015 [35], and the 2015 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations guidelines of the International Liaison Committee on Resuscitation [36,37]. A conformity checklist was developed in 2 successive brainstorming sessions of 8 emergency physicians and paramedics containing the following 9 items: the app should request the user to “check responsiveness,” “open the airway,” “assess whether the person is breathing normally (see, hear, and feel),” “consider no breathing or abnormal, ie, agonal breathing,” “call 112 (or 911) or ask somebody to call 112 (or 911),” “start chest compression,” “pay attention to correct positioning of hands,” “compress the chest at a rate of 100-120 bpm,” and “compress to a depth of at least 5 cm but not more than 6 cm.” If the app explained ventilation, it was expected to include “opening of the airway” and “verify rising of the chest.” To be rated as guideline-conform, all criteria needed to be covered in substance; verbatim coverage was not required.

    Usability Evaluation

    Remaining apps were evaluated using the system usability scale (SUS) developed by John Brooke [38]. This tool is based on the 3 categories of International Organization for Standardization norm 9241-11 for usability: “effectiveness,” “efficiency,” and “satisfaction” [38,39]. The SUS was rated as a highly robust and versatile tool to evaluate usability [40]. It is the most widely used scale to evaluate usability and has been cited in more than 600 research publications [41]. The questionnaire of the SUS consists of 10 statements on a 5-point Likert scale with 5 positive statements (item 1, 3, 5, 7, and 9) and 5 negative statements (item 2, 4, 6, 8, and 10). The user is asked to rate their level of agreement with these statements concerning the software under review. To get a total score between 0 and 100, the individual scores are calculated as follows: each item’s score ranges from 1 to 5 depending on the position. In case of the uneven items, the scale position minus 1 contributes to the total score. In case of the even items, the contribution to the total score is 5 minus the scale position. In the next step, the sum of these results is multiplied by 2.5. This results in a value on a scale between 0 and 100 [38]. This value does not represent a percentage of usability [42]. The German version of the SUS used is attached in Multimedia Appendix 1. When translating the SUS questionnaire from its original version, we changed the general term “system” into the more specific word “app.”

    Sauro and Lewis detected a rate of 11% of coding errors when working with a conventional SUS score with positive and negative statements [43]. Therefore, in our evaluation, 2 researchers calculated the SUS score independently to diminish the rate of mistakes in calculation. If their results did not match, a third researcher calculated the SUS score.

    Usability Evaluators

    Barnum recommends asking multiple groups to evaluate with SUS to emphasize different aspects of a given system [44]. Regarding this topic, 3 groups, whose SUS evaluation could show different perspectives, were identified: (1) people with a higher chance of having to use the app, (2) experienced app users, and (3) people with high experience regarding the medical content of the app. The highest risk of being confronted with cardiac arrest can be attributed to elderly people [45] as well as professionals of the medical field. However, the percentage of individuals owning a mobile phone is by far smaller among people aged 65 years and older than found in the average population [46]. People working in medical environments are educated and trained in basic life support and unlikely to need the help of an app to perform the basic steps of resuscitation. Thus, we decided not to interview these groups and focus on the remaining 2: those who frequently use related products (apps) and those whose work is relevant to the content of the product. Consequently, 1 group evaluating the app consisted of mobile phone frequent users, whereas the other group consisted of emergency physicians. Mobile phone frequent users were defined as individuals who had owned a mobile phone for more than 3 years, currently have more than 15 apps installed on their device, and use these for more than 1 hour per day. The emergency physicians are all currently employed in the German emergency system. The emergency physicians were asked to keep in mind that the apps were designed to teach the steps of basic life support to medical laypersons.

    In addition, the emergency physicians were asked to rank the apps according to the quality of teaching different aspects of high-quality CPR: 7 aspects were developed in 2 successive brainstorming sessions of 8 emergency physicians and paramedics based on the German translation of the European Resuscitation Council guidelines. Aspects, all researchers involved associated with high quality CPR were collected and evaluated. The criteria were as follows: “The app comprehensively explains the opening of the airway,” “The app points out the problem of agonal breathing,” “The app emphasizes the importance of complete recoil of the chest after each compression,” “The app indicates that pauses in chest compressions should be minimized,” “The app helps the medical layperson to find the correct frequency for chest compression,” “The design and user-interface supports an optimal execution of cardiopulmonary resuscitation,” and “The app requests the medical layperson to continue with cardiopulmonary resuscitation until the arrival of emergency service.”

    Statistical processing of the data was carried out using IBM SPSS Statistics, version 26.0 (IBM Corporation, Armonk, New York, USA), and Microsoft Excel 2010 (Microsoft Corporation, Redmond, Washington, USA). We assessed normal distribution by the Shapiro-Wilk test; median and interquartile range were calculated. In case of normal distribution, t test was used to assess significance levels. To assess significance level in the absence of normal distribution, Mann-Whitney U test was used between the 2 groups testing the same app and Wilcoxon test between the same people testing different apps.


    Results

    Systematic Review of Apps

    The results of the systematic review are depicted in Figure 1 as an adapted PRISMA flow diagram. The search of the 16 German and English keywords identified 3146 search results in Google Play Store and 744 in Apple App Store. After the exclusion process, 34 apps remained for the evaluation of guideline adherence (see Multimedia Appendix 2).

    Adherence to Guideline

    The results of the analysis of guideline adherence are depicted in Figure 2. A total of 7 apps taught hands-only CPR, whereas 27 also explained ventilation. Out of the 34 apps analyzed, 18 (53%, 15/34) did not indicate to consider “no breathing and abnormal, ie, agonal breathing,” 17 (50%, 17/34) did not request to assess whether the person is breathing normally, 18 (53%, 18/34) did not explain to compress to the recommended depth of at least 5 cm but not more than 6 cm, and 17 (50%, 17/34) did not recommend to open the airway. In our evaluation, only 5 out of the 34 (15%, 5/34) apps met all criteria of guideline adherence tested; these apps are listed in Table 1.

    Usability Evaluation

    The usability evaluation with the SUS tool was conducted in October 2017. Of the 5 apps that met the criteria, 2 had to be excluded before starting the usability evaluation: 1 app was no longer available in both app stores (“St John Wales First Aid”) and the other app (“Notfall-Hilfe”) was excluded because it showed fundamental differences between the version of the Google Play Store and Apple App Store. The version of the Google Play Store contained pictures and movies explaining all steps, and the text was read aloud, if desired. None of these features were available in the Apple App Store version. This gap would profoundly influence the results of the SUS, leading to the decision to exclude this app in the usability evaluation.

    The group of mobile phone frequent users consisted of 10 participants (7 females and 3 males) with a median age of 23 years (minimum=20 years and maximum=25 years) and the group of emergency physicians of 9 participants (4 females and 5 males) with a median age of 37 years (minimum=32 years and maximum=56 years). The SUS participants assessed the apps on their own mobile phone. iPhone 6Plus, provided by the researchers, was used by 2 emergency physicians.

    Figure 1. Results of the systematic review of apps providing step-by-step instructions for cardiopulmonary resuscitation (CPR) in case of a cardiac arrest.
    View this figure
    Figure 2. Apps in accordance with the criteria used to evaluate guideline adherence.
    View this figure
    Table 1. Apps that met all 9 of our criteria for guideline adherence.
    View this table

    The median SUS of “HELP Notfall” was significantly higher than “HAMBURG SCHOCKT” (87.5 vs 55; asymptotic Wilcoxon test: z=−3.63, P<.01, n=19) and also significantly higher than “Mein DRK” (87.5 vs 32.5; asymptotic Wilcoxon test: z=−3.83, P<.01, n=19). The median SUS of “HAMBURG SCHOCKT” was significantly higher than “Mein DRK” (55 vs 32.5; asymptotic Wilcoxon test: z=−2.81, P<.01, n=19). The median SUS scores did not differ significantly between the group of mobile phone frequent users and emergency physicians. The SUS results are shown in Figure 3.

    Table 2 depicts how the emergency physicians ranked the apps according to the quality of teaching different aspects of high-quality CPR. Of the 9 emergency physicians, 1 did not complete this part of the questionnaire. The participating emergency physicians rated the app “HELP Notfall” as the one, teaching the majority of relevant aspects (6 out of 7) best. There was no clear result for the aspect “The app emphasizes the importance of complete recoil of the chest after each compression.”

    Figure 3. Usability evaluation of the apps with system usability scale (SUS) score. IQR: interquartile range.
    View this figure
    Table 2. Number of emergency physicians rating the app listed as the one teaching a specific aspect of high-quality cardiopulmonary resuscitation best. A total of 8 emergency physicians evaluated the apps.
    View this table

    Discussion

    Principal Findings

    The structured search in 2 app stores resulted in 3890 hits. After removing redundant ones, 2640 hits were checked for fulfilling the inclusion criteria. Hereby, 34 apps were identified, meeting all inclusion criteria, of which only 5 (15%, 5/34) fulfilled all defined criteria of adherence to the guidelines of the European Resuscitation Council and the American Heart Association. All other apps gave no or incorrect information on at least one relevant topic. Regarding the usability, only 1 out of the 3 apps was evaluated with an SUS score above the published average of 68 [41].

    Systematic Review of Apps

    The systematic review of apps available on Google Play Store and Apple App Store took place between May 26, 2017, and June 23, 2017. After excluding redundant hits; duplicates apps; and apps that were not ubiquitously available, free of charge, or did not provide step-by-step instructions in English or German for resuscitation of human beings, 34 apps remained.

    Similar to other studies, the search in Google Play Store yielded far more results than the Apple App Store [47]. One reason for this striking difference might be the different submission systems. Although there are no admission requirements in the Google Play Store, Apple tests each submitted app for technical compatibility and conducts a content verification review [27,48]. However, the main goal of this content verification seems to be to ensure that the name and description of the app match with the content.

    Various aspects increase the difficulties for a medical layperson to find a suitable app: the sheer volume of apps to choose from can overwhelm the user [27]. Which app will be downloaded by the user depends on a number of factors such as user ratings, appealing of screenshots, keywords, and number of downloads [49]. Therefore, in the last years, the term “app store optimization” was coined describing strategies to increase the likelihood of an app being downloaded [49,50]. Moreover, the availability of apps differs not only between operators but also between countries, which influences and limits the user’s choice.

    Adherence to Guidelines

    Of the 34 examined apps, only 5 (15%, 5/34) apps fit all of our criteria of guideline-adherent resuscitation. This alarming result is concordant with that of other studies examining the content of mHealth apps [51-54]. The “European Resuscitation Council Guidelines for Resuscitation 2015” [55] and the “American Heart Association Guidelines Update for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care 2015” [35] are well known and highly respected international medical guidelines, which are evidence based on current literature. These guidelines offer clear and easy-to-understand advice on which steps should be taken to resuscitate a person. All flowcharts, pictures, and movies are freely available and translated into many different languages [56,57]. Nevertheless, only few apps implemented these recommendations. This might possibly lead to reduced probability of survival of the victim resuscitated.

    It has been shown in multiple studies [58-60] that a cardiac arrest victim showing abnormal, ie, agonal breathing has an increased chance of survival compared with cardiac arrest patients suffering from apnea. However, agonal breathing is often misjudged by medical laypersons not realizing the need for performing CPR in these cases. Therefore, it is crucial that CPR apps point out that patients presenting with agonal breathing are also in need of CPR. More than half of the examined apps (53%, 18/34) did not consider this important fact. Furthermore, 50% (17/34) of the apps did not even guide the user to open the airway and 50% (17/34) did not recommend to assess whether the person is breathing (see, hear, and feel). Of the 34 apps studied, 18 (53%, 18/34) recommended no or wrong compression depth, despite numerous studies suggesting to compress to a depth of at least 5 cm but not more than 6 cm to increase the chance of a positive medical outcome [61-65]. Of the 34 apps, 8 (24%, 8/34) apps did not show the correct compression rate of 100-120 bpm that has been proven to improve survival rates [66]. Of the 34 apps, 27 additionally explained ventilation, although the European Resuscitation Council as well as the American Heart Association recommend to teach medical laypersons hands-only CPR [35,55]. The 2 additional criteria for apps explaining ventilation as well were not met by all of these apps. Although this further diminishes the number of apps meeting all criteria, it is not the sole reason for the low adherence rate. A long conformity checklist certainly increases the risk of an app not meeting every single criterion. However, all chosen criteria for guideline adherence are important evidence-based aspects, which are taught in resuscitation courses worldwide.

    Determining a global quality management for apps is complicated by different legislations in different countries and multiple concerned governmental agencies (eg, health and privacy legislation) [67,68]. Although institutions and authorities from different states suggested approaches to a quality control of apps, there is no universally accepted procedure [69-73]. However, in our opinion, all efforts to increase the medical accuracy of mHealth products should be made. The aspiration of all persons teaching CPR should be that the general public is provided with correct information on resuscitation, independent on how the information is spread (by textbooks, movie clips, leaflets, or apps).

    Usability Evaluation

    Of the 3 apps examined with regard to usability, the app “HELP Notfall” had the highest median SUS score (87.5), followed by “HAMBURG SCHOCKT” (55) and “Mein DRK” (32.5). This difference was seen in both evaluating groups (mobile phone frequent users and emergency physicians). There was no significant difference between the evaluating groups.

    On the basis of a work by Sauro et al, an SUS score above 68 is rated as a value above average [41]. Such a score was solely achieved by “HELP Notfall.” The other 2 apps tested performed below average. The usability of an app is crucial for its implementation and usage. Sauro reported that products with an SUS score higher than 82 have a considerable chance of being recommended to a friend or colleague [41], which was reached by “HELP Notfall.” In his famous technology acceptance model, Davis (1986) stated that systems will only be used if they are perceived as useful [74]. If the user can see a clear advantage in comparison with their previous approach, users will utilize an app [75]. In the case of an app designed for the use in a time-critical and extremely challenging situation, it is even more important that the operation is highly intuitive. If a patient is in cardiac arrest and resuscitation becomes necessary, there is no time to first become acquainted with the software. Otherwise, there seems to be a relevant risk of apps not helping the user but leading him astray from doing best for the patient. Thus, a high usability of the app is crucial.

    Limitations

    The systematic review of apps was conducted in the 2 main app stores Apple App Store and Google Play Store, whereas smaller stores such as Amazon App Store, Windows Store, Samsung Apps, or Blackberry World were not included. The review was completed by only 1 researcher (LS), leaving a possibility of misjudging inclusion or exclusion criteria. However, the criteria were phrased distinctly and clearly to diminish this risk, and 2 other researchers (BM and CM) made spot checks. If an app was available either in Google Play Store or in Apple App Store but not in both stores, it was excluded because a recommendable CPR app should be usable by the majority of the population. This criterion led to the exclusion of 2218 search results, which was the majority of all search hits. Furthermore, we decided to choose “free of charge” as an inclusion criterion to enlarge the group of possible future users. A study by Lim et al conducted in different countries worldwide showed that the most important factor influencing people in the process of downloading an app was the price of the app and that 57% of users will not download apps they have to pay for [76]. We do not know how many apps explain the topic of resuscitation according to the medical guidelines in a user-friendly way but are not free of charge and have to be purchased or are available in just 1 app store. These inclusion criteria certainly influenced the amount and choice of apps analyzed. Furthermore, the world of mobile phone apps is fast moving with new apps entering the market and other ones vanishing. Hence, a review of apps always reflects availability at a certain time. We conducted the search with keywords in English and German language. We cannot say whether a search in other languages might lead to different results. The search was carried out with the app stores’ default country setting of Germany. This certainly further reduced the number of possible apps.

    The SUS was evaluated by mobile phone frequent users and emergency physicians. As described in the Methods section, groups at high risk of witnessing a cardiac arrest were not included. This might bias the results.

    To date, it is not known whether the use of an app providing step-by-step instructions in a CPR situation increases the rate or quality of bystander resuscitation and leads to higher survival rates among the victims. To broadly recommend the use of such apps, further studies are needed to evaluate positive and negative effects.

    Comparison With Prior Work

    Only few studies systematically evaluating CPR apps exist [22,47,51-54,77-79]. In contrast to our work, some studies did not have a structured search but evaluated only a representative sample of apps [47,77] or searched only in 1 operating system [51,78]. Only a few previous studies evaluated the adherence of an app to an existing medical guideline [53,54,79]. These 3 studies covered weight loss and pain management. The study of Kalz et al included both resuscitation-teaching apps as well as apps providing guidance in a resuscitation situation in real time [52]. However, teaching a topic in a classroom and giving step-by-step instructions in a real situation are different purposes and call for different designs of the app. We decided to focus only on apps offering real-time support. This is in contrast to the study of Ahn et al, concentrating solely on CPR-training apps [22].

    Contrary to all other studies, we did not select a reduced number of apps for the SUS evaluation but did a comprehensive evaluation of all apps that fit the inclusion criteria.

    Conclusions

    This work combined a systematic review of currently available resuscitation apps with an assessment of guideline adherence and an evaluation of usability. The search resulted in 3890 hits. Of 34 apps that met the inclusion criteria, only 5 (15%, 5/34) fulfilled all of the criteria applied to determine guideline adherence. All other apps gave no or incorrect information on at least one relevant topic. Furthermore, our evaluation of usability revealed that only 1 of the 3 apps tested had an above average usability rate according to SUS. Implementing a systematic quality control for health-related apps should be enforced to ensure medical accuracy and sufficient usability. This is of superior importance for apps focusing on the treatment of life-threatening events such as cardiac arrest.

    Acknowledgments

    The authors would like to thank Berthold Henkel, Anne Schiller, Stefan Mockler, and Jan Bartels for their ongoing support.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    The German version of the system usability scale used.

    PDF File (Adobe PDF File), 61KB

    Multimedia Appendix 2

    Apps analyzed for guideline adherence.

    PDF File (Adobe PDF File), 26KB

    References

    1. Sasson C, Rogers MA, Dahl J, Kellermann AL. Predictors of survival from out-of-hospital cardiac arrest: a systematic review and meta-analysis. Circ Cardiovasc Qual Outcomes 2010 Jan;3(1):63-81 [FREE Full text] [CrossRef] [Medline]
    2. Perkins GD, Handley AJ, Koster RW, Castrén M, Smyth MA, Olasveengen T, Adult basic life supportautomated external defibrillation section Collaborators. European Resuscitation Council Guidelines for Resuscitation 2015: Section 2. Adult basic life support and automated external defibrillation. Resuscitation 2015 Oct;95:81-99. [CrossRef] [Medline]
    3. Park GJ, Song KJ, Shin SD, Lee KW, Ahn KO, Lee EJ, et al. Timely bystander CPR improves outcomes despite longer EMS times. Am J Emerg Med 2017 Aug;35(8):1049-1055. [CrossRef] [Medline]
    4. Krammel M, Schnaubelt S, Weidenauer D, Winnisch M, Steininger M, Eichelter J, et al. Gender and age-specific aspects of awareness and knowledge in basic life support. PLoS One 2018;13(6):e0198918 [FREE Full text] [CrossRef] [Medline]
    5. Nord A, Svensson L, Karlsson T, Claesson A, Herlitz J, Nilsson L. Increased survival from out-of-hospital cardiac arrest when off duty medically educated personnel perform CPR compared with laymen. Resuscitation 2017 Nov;120:88-94. [CrossRef] [Medline]
    6. Fordyce CB. Reduced critical care utilization: another victory for effective bystander interventions in cardiac arrest. Resuscitation 2017 Oct;119:A4-A5. [CrossRef] [Medline]
    7. Riddersholm S, Kragholm K, Mortensen RN, Pape M, Hansen CM, Lippert FK, et al. Association of bystander interventions and hospital length of stay and admission to intensive care unit in out-of-hospital cardiac arrest survivors. Resuscitation 2017 Oct;119:99-106. [CrossRef] [Medline]
    8. Hasselqvist-Ax I, Riva G, Herlitz J, Rosenqvist M, Hollenberg J, Nordberg P, et al. Early cardiopulmonary resuscitation in out-of-hospital cardiac arrest. N Engl J Med 2015 Jun 11;372(24):2307-2315. [CrossRef] [Medline]
    9. Malta HC, Kragholm K, Pearson DA, Tyson C, Monk L, Myers B, et al. Association of bystander and first-responder intervention with survival after out-of-hospital cardiac arrest in North Carolina, 2010-2013. J Am Med Assoc 2015 Jul 21;314(3):255-264. [CrossRef] [Medline]
    10. Brinkrolf P, Bohn A, Lukas RP, Heyse M, Dierschke T, Van Aken HK, et al. Senior citizens as rescuers: is reduced knowledge the reason for omitted lay-resuscitation-attempts? Results from a representative survey with 2004 interviews. PLoS One 2017;12(6):e0178938 [FREE Full text] [CrossRef] [Medline]
    11. Starks MA, Schmicker RH, Peterson ED, May S, Buick JE, Kudenchuk PJ, Resuscitation Outcomes Consortium (ROC). Association of neighborhood demographics with out-of-hospital cardiac arrest treatment and outcomes: where you live may matter. J Am Med Assoc Cardiol 2017 Oct 1;2(10):1110-1118. [CrossRef] [Medline]
    12. Axelsson C, Herlitz J, Ekström L, Holmberg S. Bystander-initiated cardiopulmonary resuscitation out-of-hospital. A first description of the bystanders and their experiences. Resuscitation 1996 Nov;33(1):3-11. [Medline]
    13. Özbilgin Ş, Akan M, Hancı V, Aygün C, Kuvaki B. Evaluation of public awareness, knowledge and attitudes about cardiopulmonary resuscitation: report of İzmir. Turk J Anaesthesiol Reanim 2015 Dec;43(6):396-405. [Medline]
    14. Marshall S. The use of cognitive aids during emergencies in anesthesia: a review of the literature. Anesth Analg 2013 Nov;117(5):1162-1171. [CrossRef] [Medline]
    15. Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med 2015 Apr 15;7(283):283rv3. [CrossRef] [Medline]
    16. Catharine RB. Educational use of smart phone technology: a survey of mobile phone application use by undergraduate university students. Program 2013 Sep 23;47(4):424-436. [CrossRef]
    17. Prensky M. Digital natives, digital immigrants part 1. On the Horizon 2001 Sep;9(5):1-6. [CrossRef]
    18. Gee PM, Greenwood DA, Paterniti DA, Ward D, Miller LM. The eHealth enhanced chronic care model: a theory derivation approach. J Med Internet Res 2015;17(4):e86 [FREE Full text] [CrossRef] [Medline]
    19. Wiederhold BK. mHealth apps empower individuals. Cyberpsychol Behav Soc Netw 2015 Aug;18(8):429-430. [CrossRef] [Medline]
    20. Sama PR, Eapen ZJ, Weinfurt KP, Shah BR, Schulman KA. An evaluation of mobile health application tools. JMIR Mhealth Uhealth 2014;2(2):e19 [FREE Full text] [CrossRef] [Medline]
    21. Gálvez JA, Lockman JL, Schleelein LE, Simpao AF, Ahumada LM, Wolf BA, et al. Interactive pediatric emergency checklists to the palm of your hand-How the Pedi Crisis app traveled around the world. Paediatr Anaesth 2017 Aug;27(8):835-840. [CrossRef] [Medline]
    22. Ahn C, Cho Y, Oh J, Song Y, Lim TH, Kang H, et al. Evaluation of smartphone applications for cardiopulmonary resuscitation training in South Korea. Biomed Res Int 2016;2016:6418710 [FREE Full text] [CrossRef] [Medline]
    23. van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore the sanity. BMC Med Inform Decis Mak 2013 Feb 11;13:23 [FREE Full text] [CrossRef] [Medline]
    24. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
    25. Haskins BL, Lesperance D, Gibbons P, Boudreaux ED. A systematic review of smartphone applications for smoking cessation. Transl Behav Med 2017 Jun;7(2):292-299 [FREE Full text] [CrossRef] [Medline]
    26. Martínez-Pérez B, de la Torre-Díez I, López-Coronado M. Mobile health applications for the most prevalent conditions by the World Health Organization: review and analysis. J Med Internet Res 2013;15(6):e120 [FREE Full text] [CrossRef] [Medline]
    27. Arnhold M, Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res 2014;16(4):e104 [FREE Full text] [CrossRef] [Medline]
    28. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
    29. Mosa AS, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak 2012;12:67 [FREE Full text] [CrossRef] [Medline]
    30. Müller RM, Kijl B, Martens JK. A comparison of inter-organizational business models of mobile app stores: there is more than open vs. closed. J Theor Appl Electron Commer Res 2011;6(2):13-14. [CrossRef]
    31. Jansen S, Bloemendal E. Defining app stores: the role of curated marketplaces in software ecosystems. Springer 2013:195-206. [CrossRef]
    32. Igeeksblog. How to Change App Store Country Region in iPhone or iPad!   URL: https://www.igeeksblog.com/how-to-change-app-store-country-region-in-iphone-or-ipad/ [accessed 2018-04-18] [WebCite Cache]
    33. Thevpn.guru. How to Change Google Play Store Region/Country   URL: https://thevpn.guru/change-google-play-store-region-country-vpn-help [accessed 2018-04-18] [WebCite Cache]
    34. Apple. MacBook Pro   URL: https://www.apple.com/macbook-pro/ [accessed 2017-12-08] [WebCite Cache]
    35. Kleinman ME, Brennan EE, Goldberger ZD, Swor RA, Terry M, Bobrow BJ, et al. Part 5: adult basic life support and cardiopulmonary resuscitation quality: 2015 American Heart Association Guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation 2015 Nov 3;132(18 Suppl 2):S414-S435 [FREE Full text] [CrossRef] [Medline]
    36. Perkins GD, Travers AH, Berg RA, Castren M, Considine J, Escalante R, Basic Life Support Chapter Collaborators. Part 3: adult basic life support and automated external defibrillation: 2015 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science with Treatment Recommendations. Resuscitation 2015 Oct;95:e43-e69. [CrossRef] [Medline]
    37. Travers AH, Perkins GD, Berg RA, Castren M, Considine J, Escalante R, Basic Life Support Chapter Collaborators. Part 3: adult basic life support and automated external defibrillation: 2015 international consensus on cardiopulmonary resuscitation and emergency cardiovascular care science with treatment recommendations. Circulation 2015 Oct 20;132(16 Suppl 1):S51-S83 [FREE Full text] [CrossRef] [Medline]
    38. Jordan PW, Thomas B, McClelland IL, Weerdmeester B. Usability Evaluation in Industry. London: Taylor & Francis, Inc; 1996.
    39. Bevan N. Quality and usability: a new framework. Usability Services 1997:25-34 [FREE Full text]
    40. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008 Jul 30;24(6):574-594. [CrossRef]
    41. Sauro J. A Practical Guide to the System Usability Scale. Denver, CO: CreateSpace Independent Publishing Platform; 2011.
    42. Brooke J. SUS: a retrospective. J Usability Stud 2013;8:29-40 [FREE Full text]
    43. Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? 2011 Presented at: SIGCHI Conference on Human Factors in Computing Systems; May 7, 2011; Vancouver p. 2215-2224. [CrossRef]
    44. Barnum CM. Usability Testing Essentials. Burlington: Elsevier; 2010.
    45. Szczerbinski S, Ratajczak J, Lach P, Rzeszuto J, Paciorek P, Karlowska-Pik J, et al. Epidemiology and chronobiology of out-of-hospital cardiac arrest in a subpopulation of southern Poland: a two-year observation. Cardiol J 2018 Apr 3:1174 [FREE Full text] [CrossRef] [Medline]
    46. Pewinternet. Nearly half of American adults are smartphone owners   URL: http://www.pewinternet.org/2012/03/01/nearly-half-of-american-adults-are-smartphone-owners/ [accessed 2018-10-10] [WebCite Cache]
    47. Eng DS, Lee JM. The promise and peril of mobile health applications for diabetes and endocrinology. Pediatr Diabetes 2013 Jun;14(4):231-238 [FREE Full text] [CrossRef] [Medline]
    48. BinDhim N, Trevena L. There's an app for that: a guide for healthcare practitioners and researchers on smartphone technology. Online J Public Health Inform 2015;7(2):e218 [FREE Full text] [CrossRef] [Medline]
    49. Meatti. App Store Optimization Tips & Checklist   URL: https://www.meatti.com/blog/app-store-optimization-tips/ [accessed 2018-04-16] [WebCite Cache]
    50. En.wikipedia. App store optimization   URL: https://en.wikipedia.org/wiki/App_store_optimization [accessed 2018-04-16] [WebCite Cache]
    51. Abroms LC, Padmanabhan N, Thaweethai L, Phillips T. iPhone apps for smoking cessation: a content analysis. Am J Prev Med 2011 Mar;40(3):279-285 [FREE Full text] [CrossRef] [Medline]
    52. Kalz M, Lenssen N, Felzen M, Rossaint R, Tabuenca B, Specht M, et al. Smartphone apps for cardiopulmonary resuscitation training and real incident support: a mixed-methods evaluation study. J Med Internet Res 2014;16(3):e89 [FREE Full text] [CrossRef] [Medline]
    53. Alnasser AA, Amalraj RE, Sathiaseelan A, Al-Khalifa AS, Marais D. Do Arabic weight-loss apps adhere to evidence-informed practices? Transl Behav Med 2016 Dec;6(3):396-402 [FREE Full text] [CrossRef] [Medline]
    54. Breton ER, Fuemmeler BF, Abroms LC. Weight loss-there is an app for that! But does it adhere to evidence-informed practices? Transl Behav Med 2011 Dec;1(4):523-529 [FREE Full text] [CrossRef] [Medline]
    55. Monsieurs KG, Nolan JP, Bossaert LL, Greif R, Maconochie IK, Nikolaou NI, ERC Guidelines 2015 Writing Group. European Resuscitation Council Guidelines for Resuscitation 2015: Section 1. Executive summary. Resuscitation 2015 Oct;95:1-80. [CrossRef] [Medline]
    56. Cprguidelines. ERC Guidelines 2015 have arrived!   URL: https://cprguidelines.eu/ [accessed 2017-12-07] [WebCite Cache]
    57. Eccguidelines.heart. CPR & ECC Guidelines   URL: https://eccguidelines.heart.org/index.php/circulation/cpr-ecc-guidelines-2/ [accessed 2018-04-11] [WebCite Cache]
    58. Perkins GD, Walker G, Christensen K, Hulme J, Monsieurs KG. Teaching recognition of agonal breathing improves accuracy of diagnosing cardiac arrest. Resuscitation 2006 Sep;70(3):432-437. [CrossRef] [Medline]
    59. Perkins GD, Stephenson B, Hulme J, Monsieurs KG. Birmingham assessment of breathing study (BABS). Resuscitation 2005 Jan;64(1):109-113. [CrossRef] [Medline]
    60. Bobrow BJ, Zuercher M, Ewy GA, Clark L, Chikani V, Donahue D, et al. Gasping during cardiac arrest in humans is frequent and associated with improved survival. Circulation 2008 Dec 9;118(24):2550-2554 [FREE Full text] [CrossRef] [Medline]
    61. Vadeboncoeur T, Stolz U, Panchal A, Silver A, Venuti M, Tobin J, et al. Chest compression depth and survival in out-of-hospital cardiac arrest. Resuscitation 2014 Feb;85(2):182-188. [CrossRef] [Medline]
    62. Hellevuo H, Sainio M, Nevalainen R, Huhtala H, Olkkola KT, Tenhunen J, et al. Deeper chest compression- more complications for cardiac arrest patients? Resuscitation 2013 Jun;84(6):760-765. [CrossRef] [Medline]
    63. Stiell IG, Brown SP, Nichol G, Cheskes S, Vaillancourt C, Callaway CW, Resuscitation Outcomes Consortium Investigators. What is the optimal chest compression depth during out-of-hospital cardiac arrest resuscitation of adult patients? Circulation 2014 Nov 25;130(22):1962-1970 [FREE Full text] [CrossRef] [Medline]
    64. Hostler D, Everson-Stewart S, Rea TD, Stiell IG, Callaway CW, Kudenchuk PJ, Resuscitation Outcomes Consortium Investigators. Effect of real-time feedback during cardiopulmonary resuscitation outside hospital: prospective, cluster-randomised trial. Br Med J 2011 Feb 4;342:d512 [FREE Full text] [CrossRef] [Medline]
    65. Stiell IG, Brown SP, Christenson J, Cheskes S, Nichol G, Powell J, Resuscitation Outcomes Consortium (ROC) Investigators. What is the role of chest compression depth during out-of-hospital cardiac arrest resuscitation? Crit Care Med 2012 Apr;40(4):1192-1198 [FREE Full text] [CrossRef] [Medline]
    66. Idris AH, Guffey D, Pepe PE, Brown SP, Brooks SC, Callaway CW, Resuscitation Outcomes Consortium Investigators. Chest compression rates and survival following out-of-hospital cardiac arrest. Crit Care Med 2015 Apr;43(4):840-848. [CrossRef] [Medline]
    67. Yang YT, Silverman RD. Mobile health applications: the patchwork of legal and liability issues suggests strategies to improve oversight. Health Aff (Millwood) 2014 Feb;33(2):222-227. [CrossRef] [Medline]
    68. Parker L, Karliychuk T, Gillies D, Mintzes B, Raven M, Grundy Q. A health app developer's guide to law and policy: a multi-sector policy analysis. BMC Med Inform Decis Mak 2017 Oct 2;17(1):141 [FREE Full text] [CrossRef] [Medline]
    69. Charani E, Castro-Sánchez E, Moore LS, Holmes A. Do smartphone applications in healthcare require a governance and legal framework? It depends on the application!. BMC Med 2014 Feb 14;12:29 [FREE Full text] [CrossRef] [Medline]
    70. Barton AJ. The regulation of mobile health applications. BMC Med 2012 May 8;10:46 [FREE Full text] [CrossRef] [Medline]
    71. Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet Res 2014 Sep 15;16(9):e210 [FREE Full text] [CrossRef] [Medline]
    72. Cortez NG, Cohen IG, Kesselheim AS. FDA regulation of mobile health technologies. N Engl J Med 2014 Jul 24;371(4):372-379. [CrossRef] [Medline]
    73. Boulos MN, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform 2014;5(3):229 [FREE Full text] [CrossRef] [Medline]
    74. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manag Sci 1989 Aug;35(8):982-1003. [CrossRef]
    75. Grindrod KA, Li M, Gates A. Evaluating user perceptions of mobile medication management applications with older adults: a usability study. JMIR Mhealth Uhealth 2014 Mar 14;2(1):e11 [FREE Full text] [CrossRef] [Medline]
    76. Lim SL, Bentley PJ, Kanakam N, Ishikawa F, Honiden S. Investigating country differences in mobile app user behavior and challenges for software engineering. IIEEE Trans Software Eng 2015 Jan 1;41(1):40-64. [CrossRef]
    77. Rossi MG, Bigi S. mHealth for diabetes support: a systematic review of apps available on the Italian market. Mhealth 2017;3:16 [FREE Full text] [CrossRef] [Medline]
    78. Demidowich AP, Lu K, Tamler R, Bloomgarden Z. An evaluation of diabetes self-management applications for Android smartphones. J Telemed Telecare 2012 Jun;18(4):235-238. [CrossRef] [Medline]
    79. Reynoldson C, Stones C, Allsop M, Gardner P, Bennett MI, Closs SJ, et al. Assessing the quality and usability of smartphone apps for pain self-management. Pain Med 2014 Jun;15(6):898-909. [CrossRef] [Medline]


    Abbreviations

    CPR: cardiopulmonary resuscitation
    PRISMA: preferred reporting items for systematic reviews and meta-analyses
    SUS: system usability scale


    Edited by G Eysenbach; submitted 13.12.17; peer-reviewed by J Creutzfeldt, M Leary, S Nimbalkar; comments to author 20.03.18; revised version received 15.05.18; accepted 09.08.18; published 06.11.18

    ©Bibiana Metelmann, Camilla Metelmann, Louisa Schuffert, Klaus Hahnenkamp, Peter Brinkrolf. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 06.11.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.