This 12-month, randomized, open-label, parallel-group trial was performed at the Smoking Cessation Unit of Virgen del Rocío University Hospital in Seville (Spain) between October 24, 2016, and October 24, 2018, and it complied with the Declaration of Helsinki and Good Clinical Practice Guidelines. The recruitment period closed on October 23, 2017. The local ethics committee approved the study protocol, and written informed consent was obtained from each participant prior to inclusion. The clinical study design (NCT03553173) and technical study design (NCT03206619) have been published previously [23,24].

A technician generated a random-group table (n=240) using computer methods and following a 1:1 ratio between groups. Clinicians enrolled the participants and assigned them to the group mentioned in the table according to their enrolment sequence. Participants were blinded to this allocation, as those in the IG were told that the provided mobile app was part of usual care. Participants in the CG were not informed about the existence of the app and did not have access to it.

Smokers were recruited during routine visits to our outpatient clinic. The inclusion criteria were as follows: (1) age over 18 years and desire to stop smoking; (2) owning an Android smartphone (as the mobile app was only available for Android devices owing to time and resource constraints in the development phase of this study and Android phones were more likely to be owned by the target population owing to their lower entry price as compared with iPhones); and (3) ability to interact with the smartphone. Smartphone literacy was assessed by asking the participants if they commonly use other text exchange smartphone apps, such as Mail, SMS, and WhatsApp. The only exclusion criterion was any previous adverse effect related to the present pharmacological treatment.

A sample size of 236 was calculated during the study design phase, according to the following parameters: CI, 95%; statistical power, 80%; CG success rate, 35%; IG success rate, 55%; and expected dropout rate, 20%. A total of 240 participants were recruited and randomized for the stratified analysis.

Usual care in the Smoking Cessation Unit of Virgen del Rocío University Hospital consists of pharmacological therapy with bupropion (ie, Zyntabac 150 mg; GlaxoSmithKline, Brentford, UK) or varenicline (ie, Champix 0.5 mg or 1 mg; Pfizer, New York, New York, USA) plus behavioral therapy. To facilitate recruitment and avoid bias associated with treatment cost, the SmokeFreeBrain project financed the drugs for usual care. Thus, all participants received their assigned treatments free of charge. Behavioral therapy, which was provided during face-to-face follow-up consultations, included various psychological techniques, including motivational interviews and cognitive-behavioral therapy. This psychopharmacological therapy was provided to both CG and IG participants. CG participants (n=120) received psychopharmacological therapy alone, whereas IG participants (n=120) received psychopharmacological therapy and used the digital therapeutic solution. Further information regarding enrollment, allocation, and the study analysis phases is provided in Figure 1. The app utilized behavioral techniques by sending personalized motivational messages generated using AI with the intent to achieve better smoking cessation rates by improving program adherence and abstinence rates [20]. After the participant read each message, the app asked the participant to rate how relevant the message was for him or her, collecting feedback for the AI. Complementing the core messaging feature, the mobile app had other minor sections. These included a user profile linked to the performed physical activity level collected by Google Fit (Google, Mountain View, California, USA), four smoking cessation benefit indicators (savings, smoke-free days, regained life hours by not smoking, and number of nonsmoked cigarettes since quitting), text-based information about smoking cessation, and a section containing relaxing and distracting elements (breathing exercises and minigames). Figure 2 presents an overview of the app features.

Information from all participants included demographic (age and sex) and socioeconomic data (profession and employment status), consumption history (daily cigarettes smoked, living with smokers, partner smoking status, smoking cessation attempts, etc), clinical information (weight, height, blood pressure, comorbidities, etc), nicotine dependence measured using the Fagerström test for nicotine dependence [25], and motivation to stop smoking according to the Richmond test [26]. Safety was measured as the number of adverse events related to pharmacological therapy.

The main clinical outcome was the 1-year smoking abstinence rate measured by exhaled carbon monoxide (CO), which was assessed using a CO tester (Micro+ Smokerlyzer; CoVita, Santa Barbara, California, USA) and urine cotinine tests. Participants with an exhaled CO level greater than 6 ppm were considered smokers [27]. Urine cotinine (SmokeScreen test; Concept Smoke Screen Ltd, Lincolnshire, UK) is a colorimetric test that measures the main nicotine metabolites, including cotinine. Participants with a cotinine concentration greater than 200 ng/ml were considered smokers [28]. Participants were considered smokers when at least one of the aforementioned conditions was met.

The HRQoL was assessed using the 36-item Short-Form Health Survey (SF-36) that was validated in Spanish [29] and the EuroQol 5-dimension 5-level (EuroQoL-5D-5L) questionnaire [30]. Physical activity was measured using the International Physical Activity Questionnaire (IPAQ) [31], and a healthy lifestyle was interpreted via body mass index (BMI) variations during follow-up consultations. A case report form, built upon the OpenClinica [32] tool, was developed to facilitate information management in the study. Information subsets were registered according to the following schedule: basal and 15, 30, 60, 90, 120, 180, and 365 (±5) days after the basal consultation [23].

The HRS impact on smoking cessation was assessed according to the precision of the recommendations sent by the system. Precision was calculated for both the message content recommendations and the time to open the messages, as the HRS aimed to optimize both variables. Further, we considered the generated engagement of the user with the system and the user appreciation of the messages as part of the technical evaluation of HRS impact. The starting point to measure the evolution of these metrics was the first day that a participant rated a message (December 5, 2016), and the assessment continued for the following 18 months to evaluate the progress. Detailed descriptions of these metric calculations are presented below.

Descriptive analyses of participant characteristics according to absolute and relative frequencies for qualitative variables and mean (SD) for quantitative variables were conducted. For the primary endpoint and secondary outcomes (BMI, physical activity, and HRQoL), analyses were conducted on a per-protocol basis. These analyses included all participants who adhered to the schedule of eight consultations (basal through 365 days after the basal consultation).

Regarding incomplete data analyses, homoscedasticity tests were conducted among groups of cases with identical missing data patterns to evaluate whether data were missing completely at random [35]. A multinomial regression analysis was performed to assess the variables influencing the probability of a participant dropping out as compared with the probability of the treatment being effective. A “no efficacy” category was set as the reference category. As multinomial model effects are relative to the reference category, to assess whether the explicative variable effects were different between the “dropout” and “efficacy” categories, the same model was adjusted using the “dropout” category as a reference. Consequently, whether the variable effects were different between the “dropout” and “efficacy” categories was assessed. The relative risk ratio (RRR) with 95% CI was used for each model.

Lost data analysis and multinomial logistic models showed different patterns in participants who dropped out as compared with the patterns in those who completed the study, regardless of treatment efficacy. Therefore, smoking abstinence at 1 year was analyzed using logistic regression models on a per-protocol basis and intention-to-treat basis, and the effect measures were the OR and 95% CI.

For the selection of logistic regression and multinomial logistic models in per-protocol (n=94) analysis and intention-to-treat (n=240) analysis, a two-stage strategy was adopted. In the first step, the variable importance was quantified using Random Forest [36] with the mean decrease in accuracy as the score. Variables with a score above 0.5 were included in an Akaike information criterion–based stepwise selection strategy. The following two restrictions were applied to the final models selected: (1) absence of a pattern in model residuals and (2) variables with a generalized variance inflation factor [37] above 5 were not allowed in order to avoid collinearity.

To determine whether the HRS metrics had an impact on the clinical outcomes, we divided IG participants in cessation and no cessation subgroups at 12 months of the intervention. Thereafter, we conducted t tests for the 12-month results of precision, time to rate messages, and engagement, analyzing each of these two subgroups.

Both groups had similar baseline characteristics, except for the maximum abstinence time and smoking cessation attempts (Table 1).

IG and CG participants who completed the study (per-protocol analysis) achieved efficacy rates of 64.7% (CI 51.6%-77.8%) and 40.0% (CI 25.7%-54.3%), respectively (P=.02; OR 2.75, CI 1.20-6.29) and a number needed to treat of 4 (CI 2.0-19.0). In the intention-to-treat analysis, IG and CG participants achieved efficacy rates of 27.5% (CI 19.5%-35.5%) and 15.0% (CI 8.6%-21.4%), respectively (P=.02; OR 2.15, CI 1.13-4.08) and a number needed to treat of 8 (CI 4.0-43.0). Figure 3 shows the efficacy evolution during the 1-year follow-up in each study group according to both analytic approaches.

BMI variances were similar for both groups. In the IG and CG, the mean BMI variances were 1.01 (CI 045-1.57) and 1.10 (CI 0.72-1.48), respectively, at 6 months and were 1.47 (CI 0.90-2.03) and 1.22 (CI 0.67-1.75), respectively, at 12 months. The between-group mean BMI differences at 6 and 12 months were −0.09 (CI −0.77 to 0.60; P=.80) and 0.25 (CI −0.53 to 1.03; P=.52), respectively.

According to the observed IPAQ scores, physical activity evolution patterns were similar in both study groups. Moreover, in the IG, 12.8% (6/47) and 25.5% (13/51) of participants increased their physical activity at 6 and 12 months, respectively, whereas 14.9% (7/47) and 13.7% (7/51) of participants reduced it at the corresponding points. On the other hand, in the CG, 14.6% (6/41) and 24.4% (11/45) of participants increased their physical activity at 6 and 12 months, respectively, whereas 24.4% (10/41) and 8.9% (4/45) of participants reduced it at the corresponding points. Table 2 summarizes the BMI and IPAQ results.

Based on the EuroQoL-5D-5L questionnaire results, both groups had improved HRQoL scores at 6 months. Although the IG showed a greater improvement at 12 months, the difference was not statistically significant (Table 2). The mean between-group visual analog scale differences at 6 and 12 months were 0.17 (95% CI −6.36 to 6.70) and 3.01 (95% CI −3.14 to 9.16), respectively. Regarding the SF-36 dimensions, both groups showed improvement at 12 months, although there were no significant between-group differences. Table 2 summarizes the HRQoL results.

Nineteen adverse events (11/19 [58%] associated with bupropion and 8/19 [42%] with varenicline) were identified in 14 IG participants (11.7%), whereas 23 adverse events (10/23 [43%] associated with bupropion and 13/23 [57%] with varenicline) were identified in 13 CG participants (10.8%) (P=.84).

The most frequent events associated with bupropion were headache (6/21, 28.6%), insomnia (4/21, 19.0%), vertigo (4/21, 19.0%), acute abdominal pain (2/21, 9.5%), and others (5/21, 23.8%), and those associated with varenicline were nausea (6/21, 28.6%), acute abdominal pain (4/21, 19.0%), insomnia (3/21, 14.3%), vomiting (2/21, 9.5%), and others (6/21, 28.6%).

The pattern of missing efficacy variable values was not totally random (P<.001). Multimedia Appendix 1 shows the variables whose distributions differed depending on the group; the values were not equally distributed between the participants who dropped out and those who completed the study. Table 3 shows the parameters estimated by the multinomial logistic regression model for each of the variable effectiveness levels (no efficacy, efficacy, and dropout) with respect to the reference category.

Importantly, the probability of being in the “efficacy” category was higher for IG participants as compared with the “no efficacy” category (RRR=3.25; P=.01). Furthermore, increasing the baseline Fagerström score (higher nicotine dependence) in a unit rendered a smoker more likely to belong to the “dropout” category as compared with the “no efficacy” category (RRR=1.31; P=.03). Increasing the maximum smoking abstinence time reduced the probability of a smoker being in the “dropout” category as compared with the “no efficacy” category (RRR=0.81; P=.03). Furthermore, the probability of being in the “dropout” category increased for participants using bupropion as compared with the “no efficacy” category (RRR=0.36; P=.01).

Table 4 shows the logistic regression analysis results for participants who completed the study (per-protocol basis). Notably, for the IG participants, the efficacy was 3.45 times higher with adjustment for age, motivation to stop smoking (Richmond scale), and comorbidity level. The efficacy probability decreased by 28.9% when a participant was not in the IG, with adjustment for the rest of the variables.

The results of the lost data pattern analysis and those obtained from the multinomial logistic models showed different patterns in participants who dropped out as compared with those who completed the study, regardless of treatment efficacy. However, as the variables associated with the dropout probability indicated a lack of treatment efficacy (belonging to the CG, short withdrawal period duration, fewer attempts, greater number of daily cigarettes, and baseline Fagerström score), a value of 0 (no efficacy) was assigned to participants who dropped out of the study. Therefore, the worst possible scenario models efficacy by intention-to-treat using a logistic regression model (Table 5).

These findings demonstrate that intervention efficacy was 3.13 higher with adjustment for the rest of the variables. Hence, the probability of long-term smoking cessation using the digital therapeutic solution was over three times higher than the probability of cessation with usual care alone.

A total of 17,111 messages and 2617 ratings were provided in the study, and there were 2311 ratings during the 12 months of follow up, 261 in the presmoking cessation phase, and 45 at 13 months or later, which were not part of this analysis.

In previous similar studies, the dropout rate was usually very high, reflecting the difficulty of smoking cessation and high relapse rate [3]. In this study, the dropout rates were 57.5% and 62.5% in the IG and CG, respectively. A high dropout rate may bias the results; however, the dropout rate was similar between the groups, minimizing the possibility of bias. The efficacy of the intervention was verified by per-protocol and intention-to-treat analyses, providing a more realistic picture while reducing the importance of dropout or full treatment compliance. Moreover, an analysis of the characteristics of the dropout and no dropout categories within each group was carry out, and the same pattern was obtained between both groups (Multimedia Appendix 3). These findings suggest the potential benefit of the intervention, as participants did not experience abnormal adverse effects and were more likely to be abstinent after 1 year. Nevertheless, further exploration of its efficacy is needed for validation in other cultures, and studies should include a larger sample size and real-world data.

We found some random inconsistent values in the entries of the registered time, including duplicate entries and negative time values. The presented results correspond to filtered data (removing such values). Removal of these values reduced data quality for the metric, and the findings may not accurately represent the entire participant group. These inconsistent values may have been registered owing to glitches in communication between the smartphone app and the tracking system on the server.

Metrics for engagement at the aggregated level could not be provided as anticipated in the technical protocol because the software service used to track user data is not able to retrieve information from data older than 2 years. This information was not listed in the features of the service when it was selected for the purposes of the study. We encourage future researchers to use in-house user data tracking services to avoid relying on third-party software.

Previous studies have explored the use of mobile apps supporting smoking cessation. The SmartQuit study tested the feasibility, acceptability, preliminary efficacy, and mechanism of behavioral change of an innovative smartphone-delivered acceptance and commitment therapy app for smoking cessation in 196 adult participants and reported a successful cessation rate of 13% after 2 months of follow-up [47].

In another study, the authors assessed the efficacy of an interactive smoking cessation decision–aid app in 684 adults and reported a successful cessation rate of 10.2% [48]. Additionally, an app-based mindfulness training program for smoking cessation was assessed in 143 participants, and a successful cessation rate of 11.1% was achieved [49]. A parallel, double-blind, randomized, controlled, two-arm trial compared the efficacy of an evidence-informed smartphone app for smoking cessation (Crush the Crave) to that of an evidence-informed self-help guide (On the Road to Quitting) in 1599 subjects and reported successful cessation rates of 7.8% and 9.2%, respectively [50]. In all these studies, the abstinence rate was reported at 6 months from baseline. However, they did not include pharmacological treatment as part of the intervention, which is a clear difference from the approach in our study. It is remarkable that the dropout rates for both the IG and CG identified in this study are consistent with those previously reported in the literature [50].

Research has been performed on intervention efficacy for the use of an app specifically designed to support the smoking cessation process when added to pharmacological therapy. The preliminary efficacy of an app designed to prompt smokers to engage in physical activity was assessed at 6 months of follow-up in 44 regular smokers who received the app in addition to behavioral smoking cessation counselling, physical activity promotion, and pharmacological support, and the overall abstinence rates were 36% in an intention-to-treat analysis and 53% in a complete-case analysis [51]. In this previous study, the allocation of pharmacological treatment (varenicline and nicotine replacement therapy) was not controlled by the study protocol, which hinders the comparability of the results with those of our study. Additionally, in a randomized pilot study, a 12-week course of varenicline was prescribed to both arms to assess the feasibility and acceptability of the My Mobile Advice Program (MyMAP) smoking cessation app and estimate its effects on smoking cessation and medication adherence [52]; however, its efficacy could not be evidenced owing to the small sample size (n=33).

As stated in the conclusion section of a recent systematic review [53] that included 26 studies (n=33,849) involving text messaging and app-based smoking cessation interventions, there is moderate certainty evidence that automated text message–based smoking cessation interventions result in greater smoking cessation rates as compared with minimal smoking cessation support and there is moderate certainty evidence of the benefit of text messaging interventions in addition to other types of smoking cessation support as compared with smoking cessation support alone. The evidence of the comparison of smartphone apps with less intensive support had very low certainty, and further randomized controlled trials are needed to test these interventions. In this sense, the results of this study aim to contribute to an increase in the certainty level of the available evidence in this domain.

When focusing on the assessment of recommender systems, such as the one used in this study, the scientific community has extensively concentrated on improving the performance of recommender system algorithms with different metrics [54-58], mainly in nonhealth contexts, such as leisure [59] and e-commerce [60,61]. Among these approaches, common assessments for prediction accuracy are as follows: (1) mean absolute error and root mean squared error, as well as their normalized and averaged variants to predict ratings that users would give to items when the actual recommendation ratings of the items are known for the whole test set; (2) precision, recall, and false positive rate [58,62] for prediction of the usage of the recommendations; and (3) normalized discounted cumulative gain when the system presents a large list of elements (similar to a search in Google), where we expect that the most relevant elements are shown at the top of the list; and (4) coverage of the recommendation set [63]. Other authors have proposed the use of surveys to assess recommender systems, such as ResQue [33].

Regarding the evaluation of mobile Health (mHealth) apps for behavioral change, such as the one included in this study, some authors have proposed evaluation procedures, such as the National Institute for Health and Care Excellence adaptation [64], the Mobile App Rating Scale framework [65], and the Application Usage Factor, which is defined as the logarithm of the product of the number of active users of a mobile app and the median number of daily uses of the app [66]. Despite these attempts to set a methodological framework to assess mHealth apps, they still lack standardization and comprehensiveness [67]. In a recent study, McKay et al identified that there was no available related best practice [68]. Instead, they proposed the following generic guidelines for what these types of evaluations should include: (1) assessment of the quality of health-related content; (2) review of the usability and functionality of the app; and (3) critique of the app potential with regard to behavioral change promotion.

Despite the lack of consensus on specific evaluation methods, it is well accepted that engagement is a key element for mHealth solutions to be successful [69]. Engagement has been considered a positive early indicator for behavioral change [70], as the process of behavioral change requires time. Previous studies showed that technologies supporting reaching a specific behavior health goal help people stick to the desired goal [71,72]. Further, Scherer et al [73] found a significant positive correlation between engagement and fewer dropouts in an mHealth intervention. A recent approach to provide an engagement score for a mHealth app is the Engagement Index proposed by Taki et al [74]. To calculate the score, the authors combined the number of pages in the app that a participant visited each day, number of app accesses during the program, number of push notifications opened, elapsed time between app accesses, and subjective answers to a questionnaire. However, achieving user engagement is difficult [75], and its level could be improved [76-78]. Therefore, it seems reasonable to consider engagement as a key metric for digital health solutions. However, the definition of engagement varies among studies. For instance, Iacoviello et al [79] considered the number of times users opened an app, number of interactions they had, and number of weeks in which users had at least one interaction with the program as indicators of engagement. Owen et al [80] determined engagement by calculating several variables, such as the number of downloads and number of sessions (a session being basically opening of the app). Yet, it is common to just use opening the app as an indicator of engagement, as adopted in previous studies [40,81]. All these engagement interpretations fall under the categorization of “system usage data” proposed by Short et al [82], which is most frequently used. However, other engagement measurement approaches have been proposed, such as ecological momentary assessments, psychophysiological measurements, and qualitive methods.

In our So-Lo-Mo study, the HRS was intuitively related to user engagement, as some studies have shown that good and timely recommendations to stop smoking motivate users to read more future recommendations [74,83]. Hence, we proposed a combination of qualitative and quantitative metrics, which measure the intended purpose of the system (quality of the recommendations, engagement of the participants with the system, and participants’ perception of the system). To our knowledge, this is the first assessment involving a HRS. However, in the case of smoking cessation, low engagement may not necessarily result in low impact on behavioral change, as some studies have previously shown the “gateway effect” [84] and “happy abandonment” [85]. There are several reasons for this, including the burden of the required interactions as compared with perceived outcomes or the internalization of health habits denoting that support is perceived as no longer necessary [86]. Consequently, we needed to carefully analyze the engagement of users across time points keeping in mind all these factors to extract any conclusions. Thus, our system’s intended usage [82] is expected to be higher during the first weeks of the smoking cessation process and to progressively decrease until there is no engagement with the system, which will be reflected in engagement measurements.

The So-Lo-Mo intervention offers a promising strategy for smoking cessation. The use of this digital therapeutic solution alongside pharmacological treatment was much more efficacious in achieving tobacco abstinence as compared with pharmacological treatment alone at the 1-year follow-up. However, this intervention did not improve participant HRQoL and physical activity levels.

Analysis of the impact of the HRS showed that participants benefited from the recommendations, and those who engaged with the system were more likely to succeed in smoking cessation. Therefore, the proposed HRS had a positive impact on the participants and offered them personalized and relevant messages, although it could not determine when supportive messages should be sent to minimize the time until they are read.

Health care providers should consider incorporating this digital therapeutic solution in their usual care, as it can facilitate positive outcomes for participants willing to stop smoking.