Mobile health is growing [1]. Mobile apps are frequently used in daily clinical practice and enable immediate on-the-go access to key clinical information that supports clinical decision making [2-5]. Patients use apps for disease information, screening, self-treatment, and management [6-9]. One may rightly ask, “Who provides us our app content?” Currently, anyone can publish medical apps. Although some app stores check for fulfillment of a number of technical criteria (eg, whether the app crashes upon launch), no one validates the medical content and no expert approval or peer-review systems exist. Consequently, there are apps with variable quality: opioid-conversion apps suggest medication doses that may threaten patient safety [10], asthma self-treatment apps contain potentially life-threatening information [11], and very few apps on cardiopulmonary resuscitation are actually designed according to existing basic life-support guidelines [12].

From the aforementioned discussion, it is obvious that we need an overview of the literature to understand the extent of this problem. In this paper, we review studies that evaluate quality of medical apps by evaluating expert involvement or adherence of app content to medical evidence. We relate our findings to current initiatives that seek to encounter this problem.

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews [13]. We included studies evaluating expert involvement or adherence of app content to medical evidence in medical mobile phone apps. The following studies were considered eligible: (1) investigating medical mobile phone apps within a predefined topic using a search strategy, and (2) assessing expert involvement or adherence to relevant medical evidence. Given that the definition of an expert and acceptable credentials may vary widely, we did not restrict the inclusion of studies to our own definitions of these concepts. Similarly, the degree of adherence to relevant medical evidence was not defined in advance; instead, we noted the included studies’ own definitions and judgments. Language was restricted to only English. Case studies and reviews of a single app were excluded, because they did not include a search strategy to systematically review available apps.

We searched existing literature through the bibliographic databases PubMed, The Cochrane Library, and EMBASE using the following search terms: (“smartphone” OR “iPhone” OR “Android”) AND (“app” OR “application”). This broad search string was used to identify as many relevant studies as possible. The last search was performed March 17, 2015. One researcher (YS) removed all duplicates and screened all abstracts. All potentially eligible studies were read in full by 2 independent researchers (YS and SRB). Disagreements were resolved by discussion. References of all included studies were read to find additional eligible studies. We only included studies with original data.

The research group piloted a data-extraction form. We extracted information on topic, app stores searched, methods used for assessment of expert involvement/adherence to medical evidence, and study results. Two researchers (YS and SHB) extracted data independently. Disagreements were solved through discussion and consensus. Microsoft Excel (Redmond, WA, USA) was used for data collection and management. The heterogeneity of the studies did not permit pooling of study results to conduct a meta-analysis. All studies were included in a qualitative analysis.

The broad search strategy yielded 1936 records, of which a great number were duplicates or irrelevant (eg, mobile-phone-assisted data collection in biomedical research). Fifty-two studies were identified as relevant, and included in this review. These studies assessed 6520 apps. Details on search and study selection are presented in Figure 1.

Included studies are presented in Tables 1 and 2. Topics tended to be broader for studies of expert involvement (eg, dermatology [14], ophthalmology [15], or pain management [16-18]), whereas studies on adherence to medical evidence tended to be more specific (eg, asthma self-management [11], prostate cancer [19], pediatric obesity [20,21]). Studies included a median of 71 apps (interquartile range 41-148), and studies of expert involvement tended to have slightly higher number of included apps with a median of 85 apps (interquartile range 39-192 apps), compared with studies of adherence to medical evidence having a median of 63 apps (interquartile range 40-104 apps). Studies reviewed mostly included apps from the Apple App Store (n=49, 98%) and Google Android Market (n=36, 71%). Fewer studies included apps from less popular app stores such as BlackBerry Market (n=19, 38%), Windows Market (n=16, 32%), Nokia Ovi (n=11, 22%), and Samsung Market (n=9, 18%). Studies that included a search in these less popular app stores were often unable to find any relevant apps for inclusion [10,16,22-26].

Twenty-eight studies assessed 3852 apps for expert involvement (Table 1). These studies dealt with topics within a variety of medical specialties and topics. The following 2 topics were assessed more than once: pain management (n=3) [16-18] and bariatric surgery (n=2) [22,23]. Studies mostly used the app stores’ app description (n=28, 100%) and the developers’ website (n=15, 54%) to determine whether an app had expert involvement. Nine studies (32%) also downloaded the apps. All studies found that at least some of the assessed apps had expert involvement and none found expert involvement in all assessed apps. Overall, expert involvement was found in 9-67% of assessed apps.

Thirty studies assessed 3051 apps for adherence to medical evidence (Table 2). Six topics were investigated in more than 1 study: weight loss (n=4) [44-47], smoking cessation (n=3) [48-50], disease self-management (n=3) [11,51,52], pediatric obesity (n=2) [20,21], physical activity (n=2) [53,54], and sports injury (n=2) [55,56]. Remaining studies investigated apps on a diverse range of topics. Assessment was mostly based on downloaded app content (n=24, 86%). In 2 studies, it was unclear whether the assessment was based on downloaded app content [52,57]. Three studies only used the app stores’ app description for the assessment [38,44,58]. Studies compared the apps with a variety of forms of medical evidence. For example, smoking cessation apps were compared with US Public Health Service’s clinical practice guidelines for treating tobacco use and dependence [48,49]. Several studies correlated the app contents with available Cochrane reviews, other systematic reviews, or other published evidence [10,11,28,38,41,55,58-60]. In 6 studies, the assessment relied on criteria for ideal app contents as defined by the authors [33,61] or whether the app contents adhered to the general knowledge of the authors [19,43,46,62]. In 17 studies, none of the assessed apps (n=2237) adhered fully to the compared evidence [11,20,21,33,38,44-49,51-54,58,61]. In the remaining 13 studies, 10-87% of the assessed apps showed complete adherence to medical evidence [10,12,19,28,41,43,50,55-57,59,60,62]. Of these, only 5 studies found that more than half of the assessed apps showed complete adherence to medical evidence [19,41,56,60,62]; of note, 2 of these were based on the authors’ own self-stated expertise [19,62]. In most studies, a number of apps adhered partly to the assessed evidence. No topic was clearly associated with a higher or lower prevalence of adherence to available evidence—lack of adherence was highly prevalent in all studied topics.

Medical apps may save lives; with no regulation of the content, however, we fear that they may also do harm. Studies in this review focused on a wide range of medical topics, app platforms, and assessment methods and all reached one general conclusion: medical mobile phone apps generally lack expert involvement and do not adhere to relevant medical evidence. Expert involvement was found in 9-67% of assessed apps. Adherence to medical evidence was found in 10-87% of the assessed apps in 13 studies, and in none of the assessed apps in 17 studies. Medical professionals and patients should be aware of this, as mobile phones increasingly play a role in medical education [5], clinical decision making [2], and patient empowerment [6-9].

For the common user, it may be practically impossible to assess whether or not an app adheres to current evidence and guidelines. In some cases, the app descriptions include references to publications from which the content is based. Levels of evidence as defined by the Oxford Centre for Evidence-Based Medicine state that systematic reviews and individual studies rank higher than opinions of an expert, but an expert opinion ranks better than nothing [63]. Hence, although expert involvement does not guarantee adherence to relevant medical evidence, it may be safer to have an expert involved than none.

Cheap and technically simple methods enable experts and clinicians to develop medical apps on their own [64-67]. These methods are based on Web apps developed using tools with a simple interface, hosted online, and distributed by the experts and clinicians [64-67]. Published examples include 1 Web app with clinical instructional videos for joint examination and 1 Web app with videos on psychiatric assessments and psychopathology lessons [65,67]. These works demonstrate that it is possible for experts to develop Web apps on their own with useful results [64,66]. However, 1 study in our review assessed both expert involvement and adherence of content to published evidence among opioid-conversion apps, and found that expert involvement per se does not necessarily lead to medical correctness of the content [10].

Apps can be considered an interactive way of communicating knowledge. We already use peer-review systems for such purposes—at least in scholarly journals—and one way of ensuring medically correct apps could be through peer reviews, which due to the unregulated nature of app stores would arrive after app publication. There are examples of short publications in medical journals of a review of 1 or more apps [68,69], and app developers are able to get an independent app review by submitting a request to Journal of Medical Internet Research mHealth and uHealth [70]. In addition, dedicated Web pages for app reviews exist [71,72]. One example of this is the Health Apps Library, which is developed and supported by the National Health Service in the United Kingdom [72]. The Health Apps Library enables developers to submit their app for review by clinicians that assesses whether the app is relevant to people in the United Kingdom, provide information from trusted sources, and comply with relevant data protection regulations [72]. The clinician then decides whether the app can be approved and published on the Health Apps Library [72]. However, even if a review exists, the user may not be aware of this. If the review is undesirable, the app developer may omit from referring to the review, which creates a bias. Previous studies on health information on the Internet reported similar results—some sources provide medically correct information, and some do not [73]—therefore, the problem highlighted in our systematic review is not new. However, some differences do exist when dealing with apps, which may allow to address this problem in the future. Apps are already reviewed by app stores before publication and app stores provide a streamlined access to content. Therefore, one possible way of addressing this problem could involve the collaboration between app stores and a regulatory third party such as the Health Apps Library when publishing apps with medical content.

Limitations of our approach should be noted. Apps can have expert involvement without stating it to the user, and app content may be accurate without referring to medical publications. In addition, apps with expert involvement can also contain inaccurate information, and referring to medical publications does not prevent out-of-date or inaccurate content. None of the studies included assessment of the actual use of the apps, which would provide an interesting dimension to our research question, as owning an app does not necessarily mean that the apps is used. These dimensions may be enlightened by future studies. Our review found that different methods were used for the assessment of expert involvement and medical adherence. Some studies assessed expert involvement or adherence to medical evidence only by reviewing the app descriptions in the app stores and by visiting the app developers’ website (Tables 1 and 2). For example, 1 study reviewed apps dealing with alcohol abuse and categorized each app’s approach using the app description [58]. We acknowledge that in some cases, this approach may provide sufficient results. However, one should note that app descriptions do not necessarily reflect the actual app content. Therefore, future studies are encouraged to download and review actual app content. A clear consensus on a methodological golden standard does not exist, but we are currently seeing inspiring studies that explore different methods that evaluate authorship and content [47,74]. One recent example is the Mobile App Rating Scale (MARS), a 23-item assessment tool that provides quality scores for an app within 5 dimensions (engagement, functionality, aesthetics, information quality, and subjective quality), which demonstrated a high level of internal consistency and inter-rater reliability [74]. Reliable tools such as the MARS are important for the future direction of how and what to review, and may help future research in providing more comparable results.

In conclusion, most medical mobile phone apps lack expert involvement and do not adhere to relevant medical evidence. Because mobile phones are highly prevalent among medical professionals and patients, this poses a significant problem. Review services do exist, but additional effort is needed, and attention to the problem may help the community to figure out the solutions of the future.