There is mixed evidence to support current ambitions for mobile health (mHealth) apps to improve chronic health and well-being [1]. While some apps have demonstrated efficacy in definitive trials [2-5], others have performed poorly [6-9]. One proposed explanation for this variable effect is that users do not engage with apps as intended [10]. The construct of engagement has been quantitatively conceptualized as the amount, duration, breadth, and depth of intervention usage [11,12]. For many mHealth app evaluations, users can be segmented along a continuum of engagement; some will never use the app, some will use it but quickly abandon it, and some will use it in unexpected ways. Complex patterns of engagement with mHealth apps are emerging and challenge current conceptual paradigms for interpreting their impact on chronic health outcomes. These digitally mediated mechanisms of action require more granular evaluations capable of analyzing multilevel, temporally dense engagement data [13]. Evaluating engagement is therefore a priority and calls for the integration of nonintrusive measures of this construct in mHealth evaluation methodology [14].

Recently, scholars sought to further the conceptualization of engagement by proposing that it may be more valuable to identify the mechanisms that underlie effective engagement, defined as sufficient engagement with an intervention to achieve intended outcomes [14,15]. The construct of effective engagement differs conceptually from both engagement and adherence, which have historically been used interchangeably [16]. Sieverink et al reason that the following 3 elements are necessary to determine adherence to a digital health intervention: (1) the ability to measure usage behaviors, (2) an operationalization of intended use, and (3) an empirical, theoretical, or rational justification of intended use [17]. We propose that effective engagement is more intentional than engagement but less justified than adherence. It sits between both constructs and bridges the transition from identifying patterns of engagement toward evidencing their capacity to achieve intended outcomes.

There has been recognition that the definition of engagement has evolved to include offline interactions with the behavior change mediated by a digital health intervention. Yardley et al have been instrumental in furthering this conceptualization of engagement by suggesting that there are 2 levels of engagement: (1) the micro level of immediate engagement with the digital health intervention and (2) the macro level of engagement with the wider intervention-mediated behavior change [14]. They posit that engagement is a dynamic process marked by shifts in both micro and macroengagement, which will vary depending on the intervention, the user, and their context. Users may be macroengaging and experiencing positive behavior change, but this may not necessarily be reflected in their microengagement analytics data. In acknowledgment of this distinction between engagement with the technological and behavioral aspects of an intervention, Yardley et al critically posit that microengagement alone cannot be taken as a valid indicator of effective engagement. We do not dispute Yardley et al’s arguments and recognize the limitations of relying solely on microengagement data to infer effective engagement. However, we posit that measuring and reporting on microengagement is fundamental to understanding how people actually use an app to improve their health and well-being. In turn, these analytic insights can be coupled with measures of macroengagement to identify the mediating mechanisms that motivate effective engagement.

The application of analytics, defined as the use of data to generate new insights [18], is an emerging approach to study and interpret engagement with mHealth interventions [19]. Van Gemert-Pijnen et al have advanced the application of log data analysis to inform how an intervention works in practice and which components should be improved to yield greater benefit [20-22]. Arden-Close et al have developed and implemented a novel R-based tool to visually explore patterns of engagement [23]. Heckler et al have called for the adoption of a continuous optimization model of evaluation that leverages simulated computational models to predict how users might engage with an intervention before data collection [24]. Scherer et al have demonstrated the value of joint models in the analysis of longitudinal engagement data. In fact, Scherer et al recently participated in a workshop sponsored by the National Institutes of Health on emerging technology and data analytics for behavioral health, and espoused the need for new analytic methods that can scale to thousands of individuals and billions of data points [19]. Short et al recently published a viewpoint on engagement measurement options that can be employed in electronic health (eHealth) and mHealth behavior change intervention evaluations [25]. They found that system engagement data are the most commonly collected and reported measures of engagement in eHealth and mHealth interventions. From this, they recommend having shared ways of conceptualizing these data as the field progresses to consolidate categorization.

Motivated by the proven value of analytics to study engagement with mHealth apps, we sought to compile and catalog a library of analytic indicators of engagement with consumer mHealth apps for self-managing chronic conditions. We defined analytic indicators as proxy measures of engagement with an mHealth app based on objective usage that generates log data [14,22]. When positioned alongside other measures suitable for evaluating the subjective experience of mHealth app engagement, they may provide complementary data-driven insights into the objective extent of engagement. We propose that analytic indicators of engagement do exactly this: they indicate that users may be engaging effectively with a digital health intervention but do not definitively confirm a relationship between engagement and intended outcomes. Establishing this relationship requires adopting a mixed-methods multidimensional approach to measure effective engagement using multiple assessment strategies [14,25].

While many researchers have included analytic indicators as a study measure when evaluating apps, they are not consistent or systematic in their selection [26]. We propose that there is benefit to understanding how engagement with mHealth apps for chronic conditions has been defined, measured, and analyzed across evaluations. The aim of this scoping review was therefore to consolidate how analytic indicators of engagement have previously been applied across clinical and technological contexts to inform how they might be optimally applied in future evaluations.

This scoping review was guided by the methodological framework developed by Arksey and O’Malley [27] and advanced by Levac et al [28]. They endorse an iterative review process with 5 distinct steps: (1) identifying the research question, (2) searching for relevant studies, (3) selecting studies, (4) charting the data, and (5) collating, summarizing, and reporting results. This framework is particularly relevant to disciplines with emerging evidence, such as mHealth, in which the paucity of definitive research makes it difficult for researchers to undertake systematic reviews [28]. In this context, conducting a scoping review allowed us to incorporate a range of study designs beyond those accepted for inclusion in systematic reviews, to generate broad findings on how researchers are measuring engagement with consumer mHealth apps for chronic conditions. We made efforts to adhere to recommendations for each step, starting with the selection of a research question that was sufficiently broad to map the extent, range, and nature of mHealth engagement research activity. We conducted this review to explore the following research question: what analytic indicators of engagement are being used in evaluations of consumer mHealth apps for chronic conditions?

A literature search was conducted in the MEDLINE, PsycINFO, CINAHL, and EMBASE databases. In addition, the Journal of Medical Internet Research and its sister journals were independently searched given their frequent and high-impact publication of mHealth research. A combination of different keywords for the constructs “engagement” and “mHealth” was used. No search terms for chronic conditions were defined a priori to broaden search results. We adopted the World Health Organization’s definition of a chronic condition as a “non-communicable disease of long duration and slow progression [29].” Multimedia Appendix 1 presents our search strategy for MEDLINE on the Ovid platform.

Titles and abstracts retrieved from the search strategy were screened for inclusion against the following criteria: (1) the article described an evaluation or a protocol for an evaluation of a consumer mHealth app for self-managing a chronic condition; (2) the study included operationalization of an engagement-related construct—Multimedia Appendix 1 provides the full list of screened constructs; (3) the study included objective, quantifiable measurements using log data analytics; (4) the app was intended to be used more than once; (5) the article was published between November 1, 2015, and November 1, 2017; and (6) the article was published in English.

Studies were excluded if (1) the mHealth app was solely an appointment reminder service; (2) the primary app technology was short message service or interactive voice response; (3) the app was for an acute condition or preventive health purposes; (4) the app was a support tool for a patient’s circle of care; (5) the app did not require user input through active or passive (sensor) data entry; (6) the app only delivered educational content; and (7) the article primarily described the design, development, or usability testing of the app.

The first author conducted the electronic searches with support from a faculty-affiliated librarian and reviewed the reference lists of relevant articles. All identified titles and abstracts were downloaded and merged using Mendeley (Elsevier) [30] and duplicated records were removed. The first author independently screened all titles and abstracts against eligibility criteria. Any articles that caused the author uncertainty were retained until data extraction when more information was available to make an informed decision for inclusion in the review. Following title and abstract review, full papers of included abstracts were assessed for final selection by all study authors.

A data extraction form was developed by the first author to extract relevant study information. We referenced work by Sieverink [17] and Kelders [31] on analytic indicators of adherence to eHealth technologies to establish preliminary codes. The form was piloted on a sample of included articles to validate proposed codes and add emergent codes. The codes extracted from each study are presented in Textbox 1. All study data were entered into SPSS version 24 (IBM) [32]. Each study along with its corresponding data was treated as a separate case. We categorized studies according to app structure and application of engagement data and calculated descriptive data for each category. Chi-square and Fisher exact tests of independence were applied to calculate differences between coded variables. A Monte Carlo correction was applied when observed counts were below expected counts.

A total of 1873 articles were identified through the database search. Of the 60 full texts screened, 19 were excluded, 8 of which did not include objective, quantifiable measurements using log data analytics. In total, 41 articles comprising 33 studies and 8 protocols met the eligibility criteria and were included for review. Figure 1 presents the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of the study selection progress [33].

The first authors of reviewed studies were affiliated with institutions in the United States (46%, 19/41), Canada (20%, 8/41), the United Kingdom (10%, 4/41), Australia (5%, 2/41), Germany (5%, 2/41), the Netherlands (5%, 2/41), France (2%, 1/41), India (2%, 1/41), Singapore (2%, 1/41), Spain (2%, 1/41), Sweden (2%, 1/41), and Switzerland (2%, 1/41).

Researchers reported log data analytics across 14 different engagement-related constructs: engagement (27%, 11/41), adherence (17%, 7/41), usage (15%, 6/41), use (15%, 6/41), feasibility (10%, 4/41), acceptability (7%, 3/41), utilization (5%, 2/41), attrition (5%, 2/41), participation (5%, 2/41), activity (2%, 1/41), adoption (2%, 1/41), compliance (2%, 1/41), fidelity (2%, 1/41), and retention (2%, 1/41). There was significant variation in how constructs were defined across studies, which limited our ability to (1) extract reliable definitions for each construct, (2) map analytic indicators to specific constructs, and (3) conduct cross-construct comparisons of analytic indicators.

The majority of reviewed studies were experimental (51%, 21/41), with the two-group pretest-posttest randomized controlled trial (RCT) as the most prevalent experimental study design (48%, 10/21), followed by the one-group pretest-posttest design (43%, 9/21). Quasi-experimental design selection (17%, 7/41) was more diverse and included cohort (29%, 2/7), interrupted time-series (14%, 1/7), and single case (14%, 1/7) studies. The remaining 13 studies included for review were observational in design (32%, 13/41). Studies were on average 5 months long (median 152 days, interquartile range, IQR 106), with a sample size of over 100 participants (median 103, IQR 252). The longest reviewed observational study conducted by Serrano et al was 7 years long with over 1 million participants [34]. A total of 19 studies applied engagement-based segmentation and reported results for separate user cohorts (58%, 19/33). In total, 14 of the reviewed studies were published in the Journal of Medical Internet Research or its sister journals (34%, 14/41).

A wide range of chronic conditions were targeted through the apps under study, with mental health (29%, 12/41), chronic pain (12%, 5/41), asthma (10%, 4/41), cardiovascular disease (7%, 3/41), and diabetes (type 1 and 2; 15%, 6/41) leading the clinical charge. Researchers also evaluated apps for cancer (5%, 2/41), hypertension (5%, 2/41), obesity (5%, 2/41), chronic kidney disease (2%, 1/41), chronic obstructive pulmonary disease (2%, 1/41), cystic fibrosis and inflammatory bowel disease (2%, 1/41), Parkinson disease (2%, 1/41), and sleep apnea (2%, 1/41). Over half of the apps had a hybrid structure (59%, 24/41), 10 apps were unstructured (24%), and 7 apps were structured (17%). Nearly half of all structured apps were aimed at improving mental health (40%, 4/10). Health care provider services were accessible to users to support managing their condition in nearly half of all reviewed apps (44%, 18/41). Characteristics of the included studies are presented in Multimedia Appendix 2 alongside the full dataset of coded analytic indicators for each study, which are summarized below.

Across the reviewed studies, engagement was measured using the following 7 analytic indicators in order of prevalence: the number of measures recorded (76%, 31/41), the frequency of interactions logged (73%, 30/41), the number of features accessed (49%, 20/41), the number of log-ins or sessions logged (46%, 19/41), the number of modules or lessons started or completed (29%, 12/41), time spent engaging with the app (27%, 11/41), and the number or content of pages accessed (17%, 7/41). Table 1 presents a tally of the analytic indicators measured in each included study. On average, researchers applied 3 different analytic indicators to measure their engagement data (mean 3.20, SD 1.42; median 3, IQR 2). The Fisher exact test of independence indicated that engagement with unstructured apps was mostly measured by the number of features accessed (8/10, P=.04), and engagement with hybrid apps was mostly measured by the number of measures recorded (21/24, P=.03). Table 2 provides a descriptive overview of structured, hybrid, and unstructured apps across study characteristics and analytic indicators.

Of the 41 studies included for review, 24 presented, described, or summarized the data generated from applying analytic indicators to measure engagement. The remaining 17 studies used or planned to use these data to infer a relationship between engagement patterns and intended outcomes.

The majority of reviewed studies did not report on attrition (70%, 23/33). Of the 10 studies that did, 5 reported on dropout attrition (50%), 4 reported on nonusage attrition (40%), and 1 reported on both phenomena (10%). Researchers were more likely to descriptively summarize raw attrition proportions than statistically analyze them (70%, 7/10). Those that conducted comparisons across attrition curves used Kaplan-Meier survival curves (10%, 1/10), Cox regression models (10%, 2/10), and latent class growth models (10%, 1/10).

In conducting this scoping review, we sought to catalog the range of analytic indicators being used in evaluations of consumer mHealth apps for chronic conditions. We applied Arksey and O’Malley’s methods of reporting and provided a descriptive analysis of the extent, nature, and distribution of analytic indicators across 41 studies, as well as a narrative and thematic summary of collected data [27]. The average mHealth evaluation included for review was a two-group pretest-posttest RCT of a hybrid-structured app for mental health self- management, had 103 participants, lasted 5 months, did not provide access to health care provider services, measured 3 analytic indicators of engagement, segmented users based on engagement data, applied engagement data for descriptive analyses, and did not report on attrition.

Some methodological limitations of our scoping review warrant discussion, the most significant being that we only reviewed articles published over a 2-year period. This sampling frame may not have captured a representative sample of mHealth literature. As such, we may have missed relevant studies published before November 2015 and after November 2017 that would have met our eligibility criteria. While we acknowledge that our sampling frame is limited in scoping the entire field of mHealth, we believe it captures the application of analytics within the field of mHealth. From our review of the literature before conducting our search, we identified a paucity of papers that focused on mHealth log data analyses. The systematic review on usage-based adherence to eHealth interventions conducted by Sieverink et al reviewed 62 papers, of which 7 were on smartphone-based interventions [17]. Of those 7 papers, 5 were published after 2016, and the other 2 were both published in 2013. Perski et al conducted a systematic review on engagement with digital behavior change interventions that comprised all studies up to November 2015 [11]. They reviewed 113 studies, of which 13 were on mobile phone–based interventions. Only 4 of those studies applied log data analyses to study engagement with the intervention. These insights confirm that our scoping review did not include all studies that applied log data analyses to study engagement with mHealth apps. However, they also suggest that the number of studies we omitted is small. Our sampling frame of November 2015 to November 2017 directly follows Perski et al’s review and includes 41 studies to address our specific research questions. For these reasons, we posit that our sample is sufficiently robust to provide a representative understanding of how analytics are being applied to study engagement with mHealth apps. Due to limited resources, only 1 reviewer conducted the electronic searches and screened all titles and abstracts against eligibility criteria, thereby potentially introducing bias. We did not assess the quality of included articles; however, this is in line with our review framework, which does not mandate this methodological practice. Finally, we did not map analytic indicators to the 14 identified engagement-related constructs for analysis. We acknowledge that conceptual differences exist between some of these constructs (eg, usage, feasibility, and adherence), and it is possible to use multiple constructs in the same study. However, we reviewed each construct and its analytic operationalizations separately during our data extraction process and could not discern significant differences. As such, we feel that we have included a homogenous body of research in this review and provided accurate insights into how researchers have used analytic indicators to measure engagement.

To date, the potential for mHealth apps to positively impact chronic health outcomes has not yet been realized [89]. This is, in part, due to the difficulties of generating a solid evidence base to guide clinical, policy, and regulatory decision making [90]. Indeed, the mHealth field has been reproached for arguing that apps warrant digital exceptionalism given the iterative nature of their design and the prohibitive cost of trials compared with their perceived level of risk [91]. We propose that our review supports researchers to harness these natural attributes for conducting data-driven evaluations of digitally mediated behavior change. Without objective knowledge of how users engage with an app to care for themselves, the mechanisms of action that underlie complex models of digitally mediated behavior change cannot be identified.

Our proposed library of analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions may be of value to researchers as a resource to support their evaluative practice. Researchers can systematically incorporate these analytic indicators into their study measures by adding analytic tags to their app’s source code, allowing them to measure engagement without creating user burden or reactivity. Once generated, these data can be used in inferential analyses to delineate relationships with observed health outcomes. Researchers can further interrogate these data by conducting rapid cycles of research and development to validate hypothesized models of effective engagement. On the basis of these insights, researchers can (1) build a cumulative body of evidence for how users should engage with their app to achieve intended outcomes, (2) incrementally improve their app to optimize effective engagement, and (3) determine the optimal digital dose of effective engagement with their app for validation in a definitive trial to meet required levels of evidence for procurement and distribution [92]. Successful implementation of these practices may elevate the discourse of these apps beyond the coarse evaluations and monolithic policy recommendations against their value in health care.

Raising the standard of mHealth app efficacy through measuring analytic indicators of engagement may enable greater confidence in the causal impact of apps on improved chronic health and well-being. It is this opportunity afforded by data-driven research to close the gap between promised and realized health benefits that is most meaningful.