Participants were recruited from 3 hospital-based diabetes education programs (DEPs) in Ontario, Canada. Most health services in Ontario, Canada, are financed through the publicly funded Ontario Health Insurance Program (OHIP), which covers medically necessary services delivered by physicians, including primary, specialty, and emergency care. Patients with T2DM typically get most of their diabetes care in short visits from family physicians who may or may not have additional multidisciplinary support. In addition, OHIP covers services provided by DEPs, which are multidisciplinary, nonphysician -led programs designed to deliver self-management education of diabetes and self-management support [16]. The 3 recruitment sites included (1) a DEP located in an urban area in a large city center (>2 million people), (2) 1 located in a midsize city in a remote area of the province (<150,000 people), and (3) 1 located in a semiurban area surrounding a large city center (<600,000 people). These sites serve a diverse range of patients including a large immigrant community, rural patients, and a large Aboriginal population. The services of these programs are complementary to primary care delivered through the patients’ primary care provider (PCP) and usually do not include medication titration.

The study consisted of a multicenter, pragmatic randomized controlled trial with blinded outcome assessment designed to evaluate the effectiveness of the BlueStar app. A full description of the protocol has been previously published [17]. Participants with an HbA_1c level higher than 8.0% were recruited from the 3 DEPs, where they received support for diabetes management and randomized in a ratio of 1:1 to 2 groups: (1) immediate treatment group (ITG) or (2) wait-list control (WLC) group. The ITG received the intervention immediately for a total duration of 6 months. The WLC group received usual care for the first 3 months, at which point they received the intervention and used the app for a total of 3 months. Outcomes were measured at baseline as well as 3 and 6 months.

Participants were eligible for inclusion in the study if they met the following criteria: (1) adults aged older than 18 years; (2) obtaining care for T2DM at a participating DEP; (3) HbA_1c ≥8.0% (and at least 1% above the participant’s target level) on most recent laboratory report within the last 3 months; (4) currently using an active email address or able and willing to obtain one; and (5) able to read the English language (self-reported). Patients were excluded if they have type 1 diabetes, were on continuous glucose monitoring, had an insulin pump, were on dialysis, pregnant, or are unable to use a computer or mobile phone because of severe mental or physical impairment.

Potential participants were identified by a clinician at each site during their regular scheduled appointments at a participating DEP. Those wanting more information met with the site coordinator and were given a brochure on the intervention and a copy of study consent form to review. If interested, the site coordinator would facilitate a phone call between the participant and study research assistant to obtain verbal consent. Participants were then randomized to 1 of 2 arms. Baseline questionnaires were completed over the phone by the research assistant at that time or within 2 weeks of randomization. Patients randomized to the ITG would meet with the site coordinator to receive the phone loaded with the BlueStar app along with a training session designed by the Ontario Telemedicine Network. Participants in the WLC group would arrange an appointment with the site coordinator in 3 months to receive their intervention and training.

Randomization was done in a centralized fashion by the Applied Health Research Centre (AHRC) at the Li Ka Shing Knowledge Institute of St. Michael’s Hospital in Toronto, Canada. Subject randomization was computer generated and stratified by site, using block sizes of 2 or 4, through REDCap [18], a Web-based electronic data entry system at the AHRC. Once the participant completed a baseline questionnaire, the centralized research assistant accessed the randomization sequence and informed the patient of their allocation to receive 1 of 2 treatments with a 1:1 randomization scheme (ITG or WLC).

The intervention was the BlueStar mobile app, designed to act as a virtual coach for patients with T2DM. The app was preloaded onto a cellular network–connected Samsung smartphone (with all other features disabled). The phone was connected to a cellular data plan for internet connectivity and was able to connect to local Wi-Fi networks. If participants used the app without an internet connection, the information was saved and uploaded to the secure server when the phone regained an internet connection. Patients could enter information related to T2DM management into the app, including baseline health, daily blood glucose readings, exercise activity, and food intake (see Multimedia Appendix 1). The app used this information to deliver customized, evidence-based messages in real time that aim to impact motivation, behavior, and education. The messages, based on the Transtheoretical Model of Behavior Change, included educational and affirmational content to encourage sustained behavior changes. Educational messages were aligned with the American Association of Diabetes Educators 7 Standard of Care [19]. The app also facilitated the transfer of data to the user’s clinician through Smart Visit reports that provide a clinical overview of current diabetes management including recent blood sugar readings.

Patients in the WLC group received usual diabetes care by the DEP and their primary care physician for the first 3 months of the study. To align with the principles of pragmatic trials, the usual care received was not standardized among participants [20].

The primary outcome for the trial was glucose control measured by HbA_1c levels at 3 months. Secondary outcomes assessed intervention impact on patient self-management, experience of care, and self-reported health utilization using patient-reported outcomes measures and patient-reported experience measures. This included patient self-efficacy measured using 2 validated scales for diabetes, the Problem Areas in Diabetes [21] and the Summary of Diabetes Self-Care Activities [22], as well as quality of life measures using the EuroQol-5D (EQ-5D) [23].

Data were collected centrally by research assistants and inputted into the REDCap database. All outcomes were assessed at 3 and 6 months. Intervention usability, an additional secondary outcome, was evaluated by an adapted version of the Mobile App Rating Scale. App utilization data were routinely collected through the app. Utilization measures include the mean number of engagements per week and the frequency of use of each feature per week.

Patient characteristics and baseline HbA_1c levels were summarized using descriptive statistics, including means and SD for continuous variables and proportions for categorical variables. Data were analyzed according to the intention-to-treat principle. Primary analyses used analysis of covariance (ANCOVA) with all complete cases. A secondary analysis adjusting for study site, length of diabetes diagnosis, ethnicity, and length in DEP was also conducted. A sensitivity analysis to explore the impact of missing data was conducted by identifying all characteristics that significantly differed between those included and not included and then adding these to the primary model with the assumption that the data are at least missing at random. Self-reported health utilization data including hypoglycemic episodes, visits to a primary care physician, visits to a specialist, visits to the emergency department, and hospital admission were converted to binary outcomes (event vs no event) and analyzed using a logistic regression model.

After 6 months, HbA_1c levels among those participants in the ITG were compared using a paired t test to look for sustained impact of the intervention. App utilization data were analyzed descriptively, including frequency of use (mean uses per week) by site and feature. An exploratory analysis to assess the impact of app usage on 3-month HbA_1c and Problem Areas in Diabetes (PAID) scale values was conducted using general linear models that controlled for baseline values.

Power was determined assuming an ANCOVA analysis with an estimate correlation between baseline and follow-up HbA_1c measurements of 0.80. The power to detect a difference of 0.7% in HbA_1c levels using an SD of 2% between treatment groups at 3 months is 99.7% at a significance level of 5%, based on a sample size of 255 (which assumes a dropout rate of 15% from the target sample size of 300 participants).

Potential participants were identified based on the study criteria and enrolled in the study between June and December 2016. We invited 463 patients; of those, 145 were not interested, 74 were unreachable for follow-up, and 5 did not complete baseline questionnaires (Figure 1). Randomization was completed on 240 participants, but 17 were excluded (8 in the WLC group and 9 in ITG) because of an eligibility HbA_1c <8.0%. Thus, 223 participants were included in the study. On follow-up, 77.1% (172/223) of participants completed a baseline HbA_1c value, whereas 65.5% (146/223) completed the primary outcome (HbA_1c levels at 3 months). A comparison of baseline characteristics shows no significant differences among those who completed the primary outcome versus those who did not, except that nonwhite were less likely to have a 3-month HbA_1c value (Multimedia Appendix 2). In total, 120 participants (63 in the WLC group and 57 in the ITG) had both baseline and 3-month HbA_1c values completed.

Table 1 summarizes the demographic characteristics of the study population. There were no significant differences in patient characteristics including age, gender, ethnicity, education, and household income. About 18.0% (40/223) of participants were diagnosed with T2DM within the last 6 months, whereas 39.5% (88/223) had a diagnosis of T2DM for over 5 years. The average HbA_1c level for the study population was 8.96% (SD 1.68) and was similar between the 2 study arms, and the use of insulin was similar between the 2 groups. Additional clinical features, including baseline medication usage and comorbidities, were similar across study arms (Multimedia Appendix 3).

Overall, there was low app utilization among ITG participants with a mean number of log-in days of 42.4 (SD 52.1) over 26 weeks, of which 46.4% (51/110) of participants used the app for 10 days or less. There was a small percentage of high users, with 18.2% (20/110) of participants using the app 100 days or more over a 182-day period. Multimedia Appendix 6 shows average number of log-in days among ITG participants over 26 weeks, showing significant decreasing mean usage over time. Blood glucose tracking was the most utilized feature with an average of 76.6 entries over 14 weeks (SD 96.59), whereas exercise tracking was the least utilized (mean 26.7 [SD 53.4]; see Figure 3). Of note, this graph also shows high variability in usage by site. Over the first 14 weeks, site 2 showed the highest number of log-in days by participants (median 36; interquartile range [IQR] 10.5-124), whereas participants from site 3 used the app significantly less (median 9; IQR 6-51). Site 1 has intermediate usage (median 17; IQR 7-72). Users with a diagnosis of diabetes in the last 6 months were the most engaged as assessed by days of log-in (median 24.5; IQR 8.7-73.5), whereas those with a diagnosis for over 5 years also had high engagement (median 18; IQR 8-86).

User ratings were completed by 105 participants to assess satisfaction with the app. Almost half of those who responded (45.7, 48/105) stated they would recommend the app to all people like them. Moreover, 41.0% (43/105) stated they would use the app 50 times or more if they continued to have access to it. About half (53.3%, 56/105) gave the app a rating of 4 to 5 stars of 5, whereas 39.0% (41/105) gave the app a rating of 3 stars. When asked if they would be willing to pay for the app, the majority of participants (55.2%, 58/105) stated they would not.

The aim of this study was to evaluate the clinical impact of the BlueStar app for diabetes self-management in a real-world multisite implementation. The results showed no difference between intervention and control arms for the primary clinical outcome of glycemic control as measured by HbA_1c. Furthermore, we found no intervention effect on secondary outcomes measuring diabetes self-efficacy, quality of life, and health care utilization behaviors. Of note, there was relative low use of the app overall, with almost half of intervention group users having minimal engagement with the app. Many app features were poorly utilized, including diet and exercise tracking, which have previously shown to play an important role in T2DM self-management [10]. There was a small number of highly engaged users, and exploratory analysis suggests a correlation with app usage, and improvement in HbA_1c levels at 3 months analysis suggests that 25 days of usage associated with an improvement in HbA_1c level by 0.4%, a clinically significant change [24].

To our knowledge, this is the largest pragmatic multisite trial of evaluation of a mobile app for self-management of T2DM, and the results are in contrast to prior published studies of mobile app for diabetes self-management. These studies of mobile apps for T2DM largely consist of small single-site studies with a homogeneous population [8,25]. A meta-analysis of 10 studies of T2DM apps reported a medium reduction in HbA_1c level of 0.55% among those using an app, with all studies reporting some positive benefit. However, these tended to have small study populations, and 8 of the 10 studies included additional ongoing feedback from the PCP as part of the intervention [8]. Similarly, a previous study of the same mobile app, which showed significant decrease in HbA_1c level among intervention participants, was conducted with only 30 participants. Moreover, in that study, the intervention arm received the mobile app plus multiple follow-up interactions from the research team to the physician and patient [13]. This large multisite study likely represents a more realistic assessment of impact for a diabetes health app across a health system than smaller, higher touch, single-site studies.

Our findings suggest that when evaluating a mobile app for chronic disease management, it is important to ask not only if the app works but also in what context, for which patients, and how to promote ongoing engagement of use. Overall, there was low usage of the app among participants. However, results indicate that contextual factors, particularly site, had a significant impact on overall usage of the app. App usage overall and across features was almost twice as high among site 2 compared with site 3. Despite comprehensive implementation protocols, there were substantial differences in time spent training clinicians, time training patients, and ongoing engagement with patients between clinical test sites, with the highest use site spending the greatest time and resources on implementation. In addition, it is increasingly evident that digital health apps designed to improve chronic disease self-management require ongoing patient engagement as a key determinant of clinical impact [26-30]. Therefore, a successful implementation and evaluation of these apps require careful consideration of factors that impact patient app utilization [30]. In this study, patients with a new diagnosis of T2DM had significantly higher usage than those who were diagnosed more than 6 months prior. Previous studies have shown patient factors including age, internal motivation, and personal values impact utilization of mobile health technologies [31]. This aligns with the results of a qualitative evaluation conducted with a subset of patients from this study. It found that perceived self-efficacy, competing priorities, and beliefs about the usefulness of virtual solutions had a significant impact on app utilization [17].

A recent systematic review of factors that impact engagement with digital health interventions highlighted the importance of both patient factors and engagement and recruitment methods [32]. Several recent studies of apps for T2DM have emphasized the importance of an implementation that includes a strong clinical endorsement and ongoing clinical support to increase overall usage [33,34]. A qualitative study of patients who dropped out of a study evaluating a self-management app for T2DM cited lack of clinician support as the primary reason for leaving. Our complementary qualitative study found participants with high app utilization identified the health care provider and/or site coordinator as a significant source of support in app adoption. These align with our quantitative findings that variation in app usage across sites was at least in part driven by variation in implementation. Future implementations of digital health apps would benefit from a clear effort to include factors that improve engagement, including a strong clinical endorsement, ongoing physician involvement, and patient reminders [35].

Several limitations to this study warrant discussion. Importantly, the study was underpowered to detect small but potentially still important differences in HbA_1c levels. The studies’ high dropout rate of 34.5% (77/223), while in line with prior electronic health (eHealth) studies, may have led to an underestimation of the clinical impact among participants [36,37]. There were several study design factors that likely contributed to the low app usage and lack of a detected intervention effect. Instead of downloading the app, participants were given the intervention on a second phone they used for the duration of the study in an attempt to standardize implementation by the funder. However, the use of a second phone to deliver eHealth interventions has been a noted barrier to usage in previous studies, and future mobile app evaluations would likely benefit from allowing participants to use their own smartphones when possible [28]. Given previous evidence on the benefits of strong primary care participation in diabetes self-management apps, the use of DEPs as the primary site of recruitment likely had a negative impact on enrollment, usage, and clinical impact [38,39]. Clinicians at the selected DEPs did not have regular communication with PCPs, and therefore, there was no robust pathway to report use of the app or possible treatment enhancements to the PCP. Future implementations of this, or similar apps, would likely benefit from strong primary care involvement throughout the study who can support self-management through direct treatment changes including medication titration. Finally, as discussed previously, significant variations in implementation across sites likely also had significant impact on site usage and overall ability to detect a clinical effect.

In this large real-world evaluation of a mobile app for diabetes self-management, we found no significant difference in HbA_1c levels between the intervention and control groups. Future research into the patient and site-specific factors that would increase app utilization would be warranted.