Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 27.04.20 in Vol 8, No 4 (2020): April

Preprints (earlier versions) of this paper are available at, first published Dec 18, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    A Mobile Health Solution Complementing Psychopharmacology-Supported Smoking Cessation: Randomized Controlled Trial

    1Smoking Cessation Unit, Medical-Surgical Unit of Respiratory Diseases, Virgen del Rocío University Hospital, Seville, Spain

    2Centro de Investigación Biomédica en Red de Enfermedades Respiratorias, Carlos III Institute of Health, Madrid, Spain

    3Research and Innovation Group in Biomedical Informatics, Biomedical Engineering and Health Economy, Institute of Biomedicine of Seville, Virgen del Rocío University Hospital, Spanish National Research Council, University of Seville, Seville, Spain

    4Department of Architecture and Computer Technology, School of Computer Engineering, Universidad de Sevilla, Seville, Spain

    5Salumedia Labs, Seville, Spain

    6Medical Physics Laboratory, School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Corresponding Author:

    Francisco Jódar-Sánchez, PhD

    Research and Innovation Group in Biomedical Informatics

    Biomedical Engineering and Health Economy, Institute of Biomedicine of Seville

    Virgen del Rocío University Hospital, Spanish National Research Council, University of Seville

    Avenida Manuel Siurot S/N

    Seville, 41013


    Phone: 34 670943651



    Background: Smoking cessation is a persistent leading public health challenge. Mobile health (mHealth) solutions are emerging to improve smoking cessation treatments. Previous approaches have proposed supporting cessation with tailored motivational messages. Some managed to provide short-term improvements in smoking cessation. Yet, these approaches were either static in terms of personalization or human-based nonscalable solutions. Additionally, long-term effects were neither presented nor assessed in combination with existing psychopharmacological therapies.

    Objective: This study aimed to analyze the long-term efficacy of a mobile app supporting psychopharmacological therapy for smoking cessation and complementarily assess the involved innovative technology.

    Methods: A 12-month, randomized, open-label, parallel-group trial comparing smoking cessation rates was performed at Virgen del Rocío University Hospital in Seville (Spain). Smokers were randomly allocated to a control group (CG) receiving usual care (psychopharmacological treatment, n=120) or an intervention group (IG) receiving psychopharmacological treatment and using a mobile app providing artificial intelligence–generated and tailored smoking cessation support messages (n=120). The secondary objectives were to analyze health-related quality of life and monitor healthy lifestyle and physical exercise habits. Safety was assessed according to the presence of adverse events related to the pharmacological therapy. Per-protocol and intention-to-treat analyses were performed. Incomplete data and multinomial regression analyses were performed to assess the variables influencing participant cessation probability. The technical solution was assessed according to the precision of the tailored motivational smoking cessation messages and user engagement. Cessation and no cessation subgroups were compared using t tests. A voluntary satisfaction questionnaire was administered at the end of the intervention to all participants who completed the trial.

    Results: In the IG, abstinence was 2.75 times higher (adjusted OR 3.45, P=.01) in the per-protocol analysis and 2.15 times higher (adjusted OR 3.13, P=.002) in the intention-to-treat analysis. Lost data analysis and multinomial logistic models showed different patterns in participants who dropped out. Regarding safety, 14 of 120 (11.7%) IG participants and 13 of 120 (10.8%) CG participants had 19 and 23 adverse events, respectively (P=.84). None of the clinical secondary objective measures showed relevant differences between the groups. The system was able to learn and tailor messages for improved effectiveness in supporting smoking cessation but was unable to reduce the time between a message being sent and opened. In either case, there was no relevant difference between the cessation and no cessation subgroups. However, a significant difference was found in system engagement at 6 months (P=.04) but not in all subsequent months. High system appreciation was reported at the end of the study.

    Conclusions: The proposed mHealth solution complementing psychopharmacological therapy showed greater efficacy for achieving 1-year tobacco abstinence as compared with psychopharmacological therapy alone. It provides a basis for artificial intelligence–based future approaches.

    Trial Registration: NCT03553173;

    International Registered Report Identifier (IRRID): RR2-10.2196/12464

    JMIR Mhealth Uhealth 2020;8(4):e17530




    Tobacco use presents a major preventable public health problem; it is the leading cause of health deterioration and premature death. The World Health Organization recognizes smoking as a chronic systemic disease among the addictions capable of producing high physical damage related to multiple diseases [1]. Further, smoking kills over 7 million people annually, with global costs estimated at US $1.4 trillion [2].

    Validated approaches to facilitate smoking cessation include nicotine replacement therapy, pharmacological treatment (ie, bupropion and varenicline), and behavioral and psychological support. Combining behavioral and psychological support with pharmacological treatment is currently the most effective intervention for achieving tobacco abstinence [3]. Moreover, bupropion was shown to approximately double the likelihood of long-term tobacco abstinence as compared with placebo [4]. Furthermore, varenicline (2 mg, total daily dose) was shown to triple the likelihood of maintaining long-term tobacco abstinence as compared with placebo [5]. Despite the proven efficacy of these treatments, a meaningful number of smokers fail to stop smoking, and many factors influence their success, including the motivation to stop and continuous support.

    Mobile and web-based interventions have been used to facilitate effective behavioral changes in different health domains [6-9], including smoking cessation. Scientific evidence has indicated that mobile phone–based text messages supporting smoking cessation were nearly 1.7 times more successful than a control approach at 6 months [10]. It has been proven that the behavioral change impact is higher when users receive multiple tailored health recommendations [11,12]. However, in previous studies, a traditional computer-tailoring approach was used to generate such personalized health recommendations. Computer tailoring involves the generation of patient-specific recommendations, typically in the form of messages, by computers, and it is performed after personal assessment to match the characteristics, needs, and interests of the patients [13-16]. Cupertino et al [17] recently piloted a 12-week smoking cessation program involving text messaging with pharmacotherapy support, which showed promising abstinence rates at 3 months and high participant satisfaction. However, there is no evidence for the long-term abstinence efficacy of mobile-based tailored interventions combined with psychopharmacological therapies.

    As part of the SmokeFreeBrain H2020 European Commission project [18], the Social-Local-Mobile (So-Lo-Mo) study investigated mobile and artificial intelligence (AI) technologies (health recommender system [HRS]) as a complementary aid to pharmacological treatments. The experimental intervention focused on providing ubiquitous tailored support to patients willing to stop smoking through a digital therapeutic mobile app solution.

    Participants receiving the So-Lo-Mo intervention were provided with access to a code to be introduced in the app. The code activated the app, connecting it with the hospital patient database. The app automatically downloaded the necessary data to initialize the app profile, reducing the burden of having to create a user profile. The mobile app was connected with an AI system designed to learn from patient interests through their interactions with the app to dynamically (1) determine, personalize, and send motivational messages to support smoking cessation; (2) schedule the message delivery frequency according to the transtheoretical model of behavioral change [19]; and (3) calculate the most convenient time to send a motivational message to support smoking cessation for each patient. Specifically, the AI system used to tailor motivational messages is a hybrid HRS, whose design has been presented by Hors-Fraile et al [20].

    The motivational messages were designed after conducting semistructured interviews involving two smoking cessation experts (a pulmonologist and a psychologist). Based on their comments, we identified the following five different topics, which were the most relevant for participants to succeed in their smoking cessation process: (1) general motivation; (2) healthy diet recommendations; (3) recommendations for an active lifestyle; (4) positive reinforcement messages to meet activity goals; and (5) benefits of being a nonsmoker. A total of 150 different messages were written for each topic by a health communication and health promotion PhD candidate, and they were validated by the two smoking cessation experts. This number of messages was chosen to ensure that, even in the worst case scenario where the AI system determines that a user needs to receive messages of a single topic, no user would get the same message twice during the intervention, as the maximum number of messages that could be received is 150 during the 1-year study. This approach minimizes any potential robotic and static feelings.

    Messages included health communication and health promotion strategies, such as creating empathy, adding new knowledge, and changing existing misconceptions. The messages were short (less than 100 words), written in simple Spanish, written using a close and friendly tone, and written with easy-to-understand terms to facilitate comprehension by smokers of all educational levels. Other studies have used steps similar to the ones used by us for message creation [21]. However, in our design, the message content was not associated with any specific behavioral change model. Each message was checked to be suitable for all genders. When a message was clearly gender specific (ie, relation between smoking and erectile dysfunction in men), an alternate suitable message for the other gender was also designed (ie, risks of smoking during pregnancy).

    HRSs involve self-learning algorithms that can adapt to the constantly evolving needs and interests of users and offer a high level of personalization of recommendations, taking advantage of the so-called “collective intelligence.” This is a new recommendation generation paradigm that contrasts the traditional tailoring approach, which provides static recommendations according to user responses to usually lengthy questionnaires. Thus, as this novel approach of tailoring is still in its infancy [22], it is of high interest to determine the relationship between the clinical outcomes of the So-Lo-Mo study and this type of technology.

    The main objective of this So-Lo-Mo study was to compare usual psychopharmacological therapy (control group, CG) alone and alongside the aforementioned digital solution (intervention group, IG) for smoking cessation. The secondary objectives were to analyze health-related quality of life (HRQoL) and monitor healthy lifestyle and physical exercise habits. Complementarily, we assessed the impact of the AI-generated motivational messages on smoking cessation outcomes in the IG.


    Study Design

    This 12-month, randomized, open-label, parallel-group trial was performed at the Smoking Cessation Unit of Virgen del Rocío University Hospital in Seville (Spain) between October 24, 2016, and October 24, 2018, and it complied with the Declaration of Helsinki and Good Clinical Practice Guidelines. The recruitment period closed on October 23, 2017. The local ethics committee approved the study protocol, and written informed consent was obtained from each participant prior to inclusion. The clinical study design (NCT03553173) and technical study design (NCT03206619) have been published previously [23,24].


    A technician generated a random-group table (n=240) using computer methods and following a 1:1 ratio between groups. Clinicians enrolled the participants and assigned them to the group mentioned in the table according to their enrolment sequence. Participants were blinded to this allocation, as those in the IG were told that the provided mobile app was part of usual care. Participants in the CG were not informed about the existence of the app and did not have access to it.

    Study Population

    Smokers were recruited during routine visits to our outpatient clinic. The inclusion criteria were as follows: (1) age over 18 years and desire to stop smoking; (2) owning an Android smartphone (as the mobile app was only available for Android devices owing to time and resource constraints in the development phase of this study and Android phones were more likely to be owned by the target population owing to their lower entry price as compared with iPhones); and (3) ability to interact with the smartphone. Smartphone literacy was assessed by asking the participants if they commonly use other text exchange smartphone apps, such as Mail, SMS, and WhatsApp. The only exclusion criterion was any previous adverse effect related to the present pharmacological treatment.

    Power Calculation and Recruitment

    A sample size of 236 was calculated during the study design phase, according to the following parameters: CI, 95%; statistical power, 80%; CG success rate, 35%; IG success rate, 55%; and expected dropout rate, 20%. A total of 240 participants were recruited and randomized for the stratified analysis.

    Intervention and Control Groups

    Usual care in the Smoking Cessation Unit of Virgen del Rocío University Hospital consists of pharmacological therapy with bupropion (ie, Zyntabac 150 mg; GlaxoSmithKline, Brentford, UK) or varenicline (ie, Champix 0.5 mg or 1 mg; Pfizer, New York, New York, USA) plus behavioral therapy. To facilitate recruitment and avoid bias associated with treatment cost, the SmokeFreeBrain project financed the drugs for usual care. Thus, all participants received their assigned treatments free of charge. Behavioral therapy, which was provided during face-to-face follow-up consultations, included various psychological techniques, including motivational interviews and cognitive-behavioral therapy. This psychopharmacological therapy was provided to both CG and IG participants. CG participants (n=120) received psychopharmacological therapy alone, whereas IG participants (n=120) received psychopharmacological therapy and used the digital therapeutic solution. Further information regarding enrollment, allocation, and the study analysis phases is provided in Figure 1. The app utilized behavioral techniques by sending personalized motivational messages generated using AI with the intent to achieve better smoking cessation rates by improving program adherence and abstinence rates [20]. After the participant read each message, the app asked the participant to rate how relevant the message was for him or her, collecting feedback for the AI. Complementing the core messaging feature, the mobile app had other minor sections. These included a user profile linked to the performed physical activity level collected by Google Fit (Google, Mountain View, California, USA), four smoking cessation benefit indicators (savings, smoke-free days, regained life hours by not smoking, and number of nonsmoked cigarettes since quitting), text-based information about smoking cessation, and a section containing relaxing and distracting elements (breathing exercises and minigames). Figure 2 presents an overview of the app features.

    Figure 1. CONSORT diagram of the study.
    View this figure
    Figure 2. Structure of app features.
    View this figure


    Information from all participants included demographic (age and sex) and socioeconomic data (profession and employment status), consumption history (daily cigarettes smoked, living with smokers, partner smoking status, smoking cessation attempts, etc), clinical information (weight, height, blood pressure, comorbidities, etc), nicotine dependence measured using the Fagerström test for nicotine dependence [25], and motivation to stop smoking according to the Richmond test [26]. Safety was measured as the number of adverse events related to pharmacological therapy.

    The main clinical outcome was the 1-year smoking abstinence rate measured by exhaled carbon monoxide (CO), which was assessed using a CO tester (Micro+ Smokerlyzer; CoVita, Santa Barbara, California, USA) and urine cotinine tests. Participants with an exhaled CO level greater than 6 ppm were considered smokers [27]. Urine cotinine (SmokeScreen test; Concept Smoke Screen Ltd, Lincolnshire, UK) is a colorimetric test that measures the main nicotine metabolites, including cotinine. Participants with a cotinine concentration greater than 200 ng/ml were considered smokers [28]. Participants were considered smokers when at least one of the aforementioned conditions was met.

    The HRQoL was assessed using the 36-item Short-Form Health Survey (SF-36) that was validated in Spanish [29] and the EuroQol 5-dimension 5-level (EuroQoL-5D-5L) questionnaire [30]. Physical activity was measured using the International Physical Activity Questionnaire (IPAQ) [31], and a healthy lifestyle was interpreted via body mass index (BMI) variations during follow-up consultations. A case report form, built upon the OpenClinica [32] tool, was developed to facilitate information management in the study. Information subsets were registered according to the following schedule: basal and 15, 30, 60, 90, 120, 180, and 365 (±5) days after the basal consultation [23].

    The HRS impact on smoking cessation was assessed according to the precision of the recommendations sent by the system. Precision was calculated for both the message content recommendations and the time to open the messages, as the HRS aimed to optimize both variables. Further, we considered the generated engagement of the user with the system and the user appreciation of the messages as part of the technical evaluation of HRS impact. The starting point to measure the evolution of these metrics was the first day that a participant rated a message (December 5, 2016), and the assessment continued for the following 18 months to evaluate the progress. Detailed descriptions of these metric calculations are presented below.

    Precision of the System

    This measurement focused on the system’s effectiveness in recommending relevant messages. The mobile app required the user to rate all read messages at least once; otherwise, the user could no longer interact with the app. There were three message rating options (like, dislike, and neutral). The ratings were then coded as 1 for “like,” −1 for “dislike,” and 0 for “neutral.” The precision metric was calculated by dividing the number of hits (messages rated as “like”) by the total number of sent messages (precision_p). A modification of this measurement was performed by considering “likes” and “neutrals” as hits (precision_p_n,). We used two variants of precision to obtain different perspectives on how the system performed in selecting relevant messages. The second approach focused on minimizing the negatively rated messages. Regardless of the variant, when we code the hits as 1 and misses as 0, this metric is in fact an arithmetic mean (we sum the number of hits and divide them by the total number of sent messages). The value of this metric ranges from 0 to 1. A higher value is associated with more system precision. This is because on sending messages at random, we expect 33% positive feedback, 33% neutral feedback, and 33% negative feedback. However, when the system learns and becomes more “intelligent,” the percentage of negative feedback will decrease over time, increasing the value resulting from this metric.

    Time to Open Motivational Messages

    This pertained to the arithmetic mean of the elapsed time between the message being sent and opened in a 30-day period. We assumed that this metric would decrease over time because the system was expected to learn and become more “intelligent” in predicting the best time for the user to open and read the sent message.

    Engagement With the System

    To measure system engagement, we included a time stamp in each sent message and compared it with the time stamp sent by the app to the server when the user opened the given message. Thus, the engagement metric was determined by the ratio of rated messages calculated as the total of all rated messages from one user divided by the total number of messages sent to that user. As we coded each message rated with a “1” value and divided the finding by the total number of sent messages, this ratio coincides with the arithmetic mean. Thus, we assume that a higher value is associated with more user interest in the message and, consequently, higher user engagement. As this metric measured the engagement of the user with the system and not the engagement of the system with the user, we considered the set date to stop smoking for each user as the point to start counting each month. After this calculation, we offset the start date for each user to the first day the user rated the first message. This allowed us to compare the average engagement of each user across time. Therefore, the results are presented according the relative months to the user set date to stop smoking (eg, M3 results are the results of the third month after a participant stopped smoking). However, for one participant, M3 could be April if the participant stopped smoking in January, but for another participant, it could be December if the participant stopped smoking in September. The measurement of this follow-up was performed for the 12-month period that each participant in the IG was enrolled in the study, as participants in the CG did not use the mobile app.

    Subjective Quality of the System

    System quality was determined by the answers in an anonymized five-level Likert-type appreciation questionnaire [24], making it impossible to link the answers to the individual participants. The questionnaire was completed by each participant at the end of the 1-year follow-up period. The questions concerned the HRS according to the Recommender Systems Questionnaire of User Experience (ResQue) model [33] and the content of the messages according to the I-Change model [34]. Of the 15 constructs of the ResQue model, 12 were represented in the questions, as the remaining three did not apply to our HRS (ie, purchase intention). All the answers reflected the level of agreement of the participant with the proposed topics (1 indicating “totally disagree” and 5 indicating “totally agree”). We assessed how participants perceived the quality of the system after 12 months of usage by observing their questionnaire response distributions, means, and SDs. Group comparisons were not feasible owing to the complete anonymity of the questionnaire. The aim was to measure the degree of possible improvement of the system in the following areas included in the questionnaire: quality of recommended items, interaction adequacy, interface adequacy, perceived ease of use, perceived usefulness, control/transparency, attitudes, behavioral intentions, and message content to influence smoking risk perception, confidence, social support, and coping action support.

    Statistical Analysis

    Descriptive analyses of participant characteristics according to absolute and relative frequencies for qualitative variables and mean (SD) for quantitative variables were conducted. For the primary endpoint and secondary outcomes (BMI, physical activity, and HRQoL), analyses were conducted on a per-protocol basis. These analyses included all participants who adhered to the schedule of eight consultations (basal through 365 days after the basal consultation).

    Regarding incomplete data analyses, homoscedasticity tests were conducted among groups of cases with identical missing data patterns to evaluate whether data were missing completely at random [35]. A multinomial regression analysis was performed to assess the variables influencing the probability of a participant dropping out as compared with the probability of the treatment being effective. A “no efficacy” category was set as the reference category. As multinomial model effects are relative to the reference category, to assess whether the explicative variable effects were different between the “dropout” and “efficacy” categories, the same model was adjusted using the “dropout” category as a reference. Consequently, whether the variable effects were different between the “dropout” and “efficacy” categories was assessed. The relative risk ratio (RRR) with 95% CI was used for each model.

    Lost data analysis and multinomial logistic models showed different patterns in participants who dropped out as compared with the patterns in those who completed the study, regardless of treatment efficacy. Therefore, smoking abstinence at 1 year was analyzed using logistic regression models on a per-protocol basis and intention-to-treat basis, and the effect measures were the OR and 95% CI.

    For the selection of logistic regression and multinomial logistic models in per-protocol (n=94) analysis and intention-to-treat (n=240) analysis, a two-stage strategy was adopted. In the first step, the variable importance was quantified using Random Forest [36] with the mean decrease in accuracy as the score. Variables with a score above 0.5 were included in an Akaike information criterion–based stepwise selection strategy. The following two restrictions were applied to the final models selected: (1) absence of a pattern in model residuals and (2) variables with a generalized variance inflation factor [37] above 5 were not allowed in order to avoid collinearity.

    To determine whether the HRS metrics had an impact on the clinical outcomes, we divided IG participants in cessation and no cessation subgroups at 12 months of the intervention. Thereafter, we conducted t tests for the 12-month results of precision, time to rate messages, and engagement, analyzing each of these two subgroups.



    Both groups had similar baseline characteristics, except for the maximum abstinence time and smoking cessation attempts (Table 1).

    Table 1. Participant baseline characteristics.
    View this table

    Efficacy (Unadjusted): 1-Year Smoking Abstinence Rate

    IG and CG participants who completed the study (per-protocol analysis) achieved efficacy rates of 64.7% (CI 51.6%-77.8%) and 40.0% (CI 25.7%-54.3%), respectively (P=.02; OR 2.75, CI 1.20-6.29) and a number needed to treat of 4 (CI 2.0-19.0). In the intention-to-treat analysis, IG and CG participants achieved efficacy rates of 27.5% (CI 19.5%-35.5%) and 15.0% (CI 8.6%-21.4%), respectively (P=.02; OR 2.15, CI 1.13-4.08) and a number needed to treat of 8 (CI 4.0-43.0). Figure 3 shows the efficacy evolution during the 1-year follow-up in each study group according to both analytic approaches.

    Figure 3. Efficacy evolution during the 1-year follow-up.
    View this figure

    Body Mass Index and Physical Activity

    BMI variances were similar for both groups. In the IG and CG, the mean BMI variances were 1.01 (CI 045-1.57) and 1.10 (CI 0.72-1.48), respectively, at 6 months and were 1.47 (CI 0.90-2.03) and 1.22 (CI 0.67-1.75), respectively, at 12 months. The between-group mean BMI differences at 6 and 12 months were −0.09 (CI −0.77 to 0.60; P=.80) and 0.25 (CI −0.53 to 1.03; P=.52), respectively.

    According to the observed IPAQ scores, physical activity evolution patterns were similar in both study groups. Moreover, in the IG, 12.8% (6/47) and 25.5% (13/51) of participants increased their physical activity at 6 and 12 months, respectively, whereas 14.9% (7/47) and 13.7% (7/51) of participants reduced it at the corresponding points. On the other hand, in the CG, 14.6% (6/41) and 24.4% (11/45) of participants increased their physical activity at 6 and 12 months, respectively, whereas 24.4% (10/41) and 8.9% (4/45) of participants reduced it at the corresponding points. Table 2 summarizes the BMI and IPAQ results.

    Table 2. Body mass index, physical activity and health-related quality of life variances in each group.
    View this table

    Health-Related Quality of Life

    Based on the EuroQoL-5D-5L questionnaire results, both groups had improved HRQoL scores at 6 months. Although the IG showed a greater improvement at 12 months, the difference was not statistically significant (Table 2). The mean between-group visual analog scale differences at 6 and 12 months were 0.17 (95% CI −6.36 to 6.70) and 3.01 (95% CI −3.14 to 9.16), respectively. Regarding the SF-36 dimensions, both groups showed improvement at 12 months, although there were no significant between-group differences. Table 2 summarizes the HRQoL results.

    Safety: Adverse Events

    Nineteen adverse events (11/19 [58%] associated with bupropion and 8/19 [42%] with varenicline) were identified in 14 IG participants (11.7%), whereas 23 adverse events (10/23 [43%] associated with bupropion and 13/23 [57%] with varenicline) were identified in 13 CG participants (10.8%) (P=.84).

    The most frequent events associated with bupropion were headache (6/21, 28.6%), insomnia (4/21, 19.0%), vertigo (4/21, 19.0%), acute abdominal pain (2/21, 9.5%), and others (5/21, 23.8%), and those associated with varenicline were nausea (6/21, 28.6%), acute abdominal pain (4/21, 19.0%), insomnia (3/21, 14.3%), vomiting (2/21, 9.5%), and others (6/21, 28.6%).

    Lost Data Pattern Analysis

    The pattern of missing efficacy variable values was not totally random (P<.001). Multimedia Appendix 1 shows the variables whose distributions differed depending on the group; the values were not equally distributed between the participants who dropped out and those who completed the study. Table 3 shows the parameters estimated by the multinomial logistic regression model for each of the variable effectiveness levels (no efficacy, efficacy, and dropout) with respect to the reference category.

    Table 3. Multinomial logistic regression.
    View this table

    Importantly, the probability of being in the “efficacy” category was higher for IG participants as compared with the “no efficacy” category (RRR=3.25; P=.01). Furthermore, increasing the baseline Fagerström score (higher nicotine dependence) in a unit rendered a smoker more likely to belong to the “dropout” category as compared with the “no efficacy” category (RRR=1.31; P=.03). Increasing the maximum smoking abstinence time reduced the probability of a smoker being in the “dropout” category as compared with the “no efficacy” category (RRR=0.81; P=.03). Furthermore, the probability of being in the “dropout” category increased for participants using bupropion as compared with the “no efficacy” category (RRR=0.36; P=.01).

    Efficacy (Adjusted): 1-Year Smoking Abstinence Rate

    Table 4 shows the logistic regression analysis results for participants who completed the study (per-protocol basis). Notably, for the IG participants, the efficacy was 3.45 times higher with adjustment for age, motivation to stop smoking (Richmond scale), and comorbidity level. The efficacy probability decreased by 28.9% when a participant was not in the IG, with adjustment for the rest of the variables.

    The results of the lost data pattern analysis and those obtained from the multinomial logistic models showed different patterns in participants who dropped out as compared with those who completed the study, regardless of treatment efficacy. However, as the variables associated with the dropout probability indicated a lack of treatment efficacy (belonging to the CG, short withdrawal period duration, fewer attempts, greater number of daily cigarettes, and baseline Fagerström score), a value of 0 (no efficacy) was assigned to participants who dropped out of the study. Therefore, the worst possible scenario models efficacy by intention-to-treat using a logistic regression model (Table 5).

    These findings demonstrate that intervention efficacy was 3.13 higher with adjustment for the rest of the variables. Hence, the probability of long-term smoking cessation using the digital therapeutic solution was over three times higher than the probability of cessation with usual care alone.

    Table 4. Logistic regression analysis of the 1-year smoking abstinence rate (per-protocol analysis).
    View this table
    Table 5. Logistic regression analysis of the 1-year smoking abstinence rate (intention-to-treat analysis).
    View this table

    Health Recommender System Impact on Smoking Cessation Outcomes

    A total of 17,111 messages and 2617 ratings were provided in the study, and there were 2311 ratings during the 12 months of follow up, 261 in the presmoking cessation phase, and 45 at 13 months or later, which were not part of this analysis.


    The t test results for each month indicated that there were no significant differences for system precision_p or precision_p_n in the cessation and no cessation subgroups. The detailed results are presented in Multimedia Appendix 2. A bimonthly graphical representation of the evolution of precision_p and precision_p_n over time is shown in Figure 4.

    Figure 4. Evolution of health recommender system precision and number of users providing ratings by subgroup. M: month; precision_p: precision calculated by dividing the number of hits (messages rated as “like”) by the total number of sent messages; precision_p_n: precision calculated by dividing the number of hits (messages rated as “like” and “neutral”) by the total number of sent messages.
    View this figure
    Time to Open Messages

    We performed t tests (results are shown in Multimedia Appendix 2) and found no significant difference between the cessation and no cessation subgroups at any time point. A bimonthly graphical representation of the evolution of the time-to-open metric is shown in Figure 5.

    Figure 5. Mean time between a message being sent and opened. M: month.
    View this figure

    The t tests shown in Multimedia Appendix 2 indicate that there were significant differences between the cessation and no cessation subgroups in only the first 7 (P=.04), 8 (P=.03), and 10 (P=.04) months, showing a trend toward higher engagement in the cessation subgroup. A bimonthly graphical representation of the evolution of the IG participants’ engagement with the system over time is shown in Figure 6.

    Figure 6. Evolution of user engagement. M: month.
    View this figure
    Perceived Quality

    Only 32 participants volunteered to respond to the questionnaire. However, not all participants who responded to the questionnaire completed it. The results are presented in Table 6.

    Table 6. Answers to the perceived quality questionnaire.
    View this table


    Principal Findings

    Clinical Outcomes

    The combination of our digital therapeutic solution with psychopharmacological treatment was more effective in achieving tobacco abstinence at 12 months as compared with psychopharmacological treatment alone. According to the per-protocol analysis findings, abstinence was 2.75 times higher among IG participants than among CG participants (adjusted OR of 3.45 for age, motivation, and comorbidity level). According to the intention-to-treat analysis, abstinence was 2.15 times higher among IG participants than among CG participants (adjusted OR of 3.13 for age, drug used, maximum abstinence time, nicotine dependence, motivation, and physical activity). Moreover, our findings suggest that the probability of dropout increased for participants who used bupropion and had high nicotine dependence, whereas a high maximum smoking abstinence time reduced this probability.

    Despite mobile apps offering several advantages over traditional smoking cessation methods, few studies have examined the content quality or the effectiveness of apps promoting smoking cessation [38-40]. To the best of our knowledge, this study provides the first scientific evidence of the efficacy of a behavioral change intervention via a mobile app powered by AI for smoking cessation at a 1-year follow-up. Furthermore, no other study has provided statistically significant results for a mobile intervention used in combination with smoking cessation drugs. Previous studies published their cessation results with 6 months of follow-up at the most, as described in the comparison with prior work subsection below. We provide evidence for the long-term impact of our digital therapeutic solution, with a follow-up period that is double of the follow-up period in previous studies. Further, this study is the first to analyze physical activity via the IPAQ after smoking cessation. Physical activity evolution patterns were similar in the IG and CG, with an increase in the percentage of participants showing improved physical activity and no significant difference between the IG and CG participants.

    Weight gain during the first year of abstinence concerns smokers [41], and it may be a potential barrier to pursue smoking cessation. In this study, weight gain, measured by an increase in BMI, occurred in both groups. The app did not affect weight gain as there was no difference between the IG and CG.

    A previous review showed that a higher number of cigarettes smoked was associated with a lower QoL. Additionally, a low QoL and depression were related with lower odds of successfully stopping smoking [42]. Participants who received treatments, including varenicline and bupropion, reported QoL improvements and increased abstinence duration as compared with those who did not receive pharmacotherapy [42,43]. The present results are comparable to these findings. There was a positive trend in the HRQoL scores reported by the IG, but with no relevant difference as compared with the scores in the CG.

    Technical Outcomes

    The complementary technical results showed that the HRS is able to learn the participants’ interests regarding the support message topics for smoking cessation. The system was more precise at the end of the intervention than at the beginning, as was expected for a recommender system. However, the minimum precision reached was very high (over 96%). We believe that there may be different reasons for this extremely high value. For instance, it could be due to the implemented hybrid HRS algorithm that mitigates the cold start problem and accurately recommends messages from the beginning according to a weighting formula described in a previous publication [20]. Additionally, it could be due to participants believing that their ratings would be viewed by health care professionals; hence, they modified their ratings toward a higher value than they otherwise would. This potential Hawthorne effect would be a limitation for digital health not identified in a recent study [44], as HRSs are only starting to be used in health care. Another reason may be that the system did not allow sufficient granularity for the users to vote; the participants were only afforded three rating options. A wider spread of rating options may have contributed to the differentiation of message relevance to a greater extent (for instance, separating fair, good, and entirely accurate messages). The statistical analysis did not find a significant difference in the precision achieved between participants who were successful in smoking cessation and those who were not successful. Hence, the achieved precision over time cannot be considered a predictor of smoking cessation, as the HRS recommended motivational messages equally well for all participants.

    The time-to-open metric results showed that the HRS was not able to predict the best time to send a message for a decrease in the time between the message being sent and the user opening the message. This may be due to several factors influencing participant behavior other than the time they receive a message, which were not considered in the HRS. To improve the time, variables, such as the position of the phone, location, last activity, and cessation day, should be considered for inclusion as parameters in the HRS.

    The generated engagement by the system showed statistical significance (P=.04) between the cessation and no cessation subgroups after the first 6 months of the intervention, favoring a higher engagement for those who managed to stop smoking. This is in line with the intuitive assumption that participants who engaged with the system to a greater extent received greater benefit and consequently were less likely to relapse. However, these differences were not significant at 9, 11, and 12 months owing to a decrease in the number of participants to such an extent at these points that the statistical power was limited.

    As not all participants in the IG completed the perceived quality questionnaire and the questionnaire was anonymous, it is impossible to ensure that any conclusion derived from the collected results is not biased. However, the results showed a clearly positive perception of the quality of the system. Reasonably, only those who managed to stop smoking and presumably benefited from the system provided answers to the questionnaire, as they would have been more motivated to respond. On the other hand, participants who dropped out, probably due to relapse, would have not been interested in completing the questionnaire. However, it could also be argued that those who did not manage to stop smoking would have been willing to provide negative feedback on the system in the anonymized questionnaire. Nevertheless, the positive trend of the answers favors the first scenario. For similar research practices in the future, we suggest following a design that forces participants to respond to the questionnaire but still preserves their anonymity. This could be achieved by including the questionnaire in a previous stage of the trial to avoid an effect by the high dropout rate associated with digital interventions for smoking cessation.

    The features of the presented AI-based digital therapeutic solution enabled the system to effectively support smoking cessation by providing support and advice for facilitating abstinence, enhance motivation, and clearly show a benefit [45]. The HRS was well perceived by the participants. Over time, it identified the most relevant motivational messages to send to each participant, and those who were engaged with the system to a great extent managed to successfully stop smoking by the end of the intervention. Therefore, this digital therapeutic solution may alleviate the known drawbacks of intensive complementary behavioral interventions for smoking cessation, which, despite their benefits, require extensive availability of well-trained professionals, leading to limited scalability, poor accessibility, and smoker resignation owing to long waiting times [46].


    In previous similar studies, the dropout rate was usually very high, reflecting the difficulty of smoking cessation and high relapse rate [3]. In this study, the dropout rates were 57.5% and 62.5% in the IG and CG, respectively. A high dropout rate may bias the results; however, the dropout rate was similar between the groups, minimizing the possibility of bias. The efficacy of the intervention was verified by per-protocol and intention-to-treat analyses, providing a more realistic picture while reducing the importance of dropout or full treatment compliance. Moreover, an analysis of the characteristics of the dropout and no dropout categories within each group was carry out, and the same pattern was obtained between both groups (Multimedia Appendix 3). These findings suggest the potential benefit of the intervention, as participants did not experience abnormal adverse effects and were more likely to be abstinent after 1 year. Nevertheless, further exploration of its efficacy is needed for validation in other cultures, and studies should include a larger sample size and real-world data.

    We found some random inconsistent values in the entries of the registered time, including duplicate entries and negative time values. The presented results correspond to filtered data (removing such values). Removal of these values reduced data quality for the metric, and the findings may not accurately represent the entire participant group. These inconsistent values may have been registered owing to glitches in communication between the smartphone app and the tracking system on the server.

    Metrics for engagement at the aggregated level could not be provided as anticipated in the technical protocol because the software service used to track user data is not able to retrieve information from data older than 2 years. This information was not listed in the features of the service when it was selected for the purposes of the study. We encourage future researchers to use in-house user data tracking services to avoid relying on third-party software.

    Comparison With Prior Work

    Previous studies have explored the use of mobile apps supporting smoking cessation. The SmartQuit study tested the feasibility, acceptability, preliminary efficacy, and mechanism of behavioral change of an innovative smartphone-delivered acceptance and commitment therapy app for smoking cessation in 196 adult participants and reported a successful cessation rate of 13% after 2 months of follow-up [47].

    In another study, the authors assessed the efficacy of an interactive smoking cessation decision–aid app in 684 adults and reported a successful cessation rate of 10.2% [48]. Additionally, an app-based mindfulness training program for smoking cessation was assessed in 143 participants, and a successful cessation rate of 11.1% was achieved [49]. A parallel, double-blind, randomized, controlled, two-arm trial compared the efficacy of an evidence-informed smartphone app for smoking cessation (Crush the Crave) to that of an evidence-informed self-help guide (On the Road to Quitting) in 1599 subjects and reported successful cessation rates of 7.8% and 9.2%, respectively [50]. In all these studies, the abstinence rate was reported at 6 months from baseline. However, they did not include pharmacological treatment as part of the intervention, which is a clear difference from the approach in our study. It is remarkable that the dropout rates for both the IG and CG identified in this study are consistent with those previously reported in the literature [50].

    Research has been performed on intervention efficacy for the use of an app specifically designed to support the smoking cessation process when added to pharmacological therapy. The preliminary efficacy of an app designed to prompt smokers to engage in physical activity was assessed at 6 months of follow-up in 44 regular smokers who received the app in addition to behavioral smoking cessation counselling, physical activity promotion, and pharmacological support, and the overall abstinence rates were 36% in an intention-to-treat analysis and 53% in a complete-case analysis [51]. In this previous study, the allocation of pharmacological treatment (varenicline and nicotine replacement therapy) was not controlled by the study protocol, which hinders the comparability of the results with those of our study. Additionally, in a randomized pilot study, a 12-week course of varenicline was prescribed to both arms to assess the feasibility and acceptability of the My Mobile Advice Program (MyMAP) smoking cessation app and estimate its effects on smoking cessation and medication adherence [52]; however, its efficacy could not be evidenced owing to the small sample size (n=33).

    As stated in the conclusion section of a recent systematic review [53] that included 26 studies (n=33,849) involving text messaging and app-based smoking cessation interventions, there is moderate certainty evidence that automated text message–based smoking cessation interventions result in greater smoking cessation rates as compared with minimal smoking cessation support and there is moderate certainty evidence of the benefit of text messaging interventions in addition to other types of smoking cessation support as compared with smoking cessation support alone. The evidence of the comparison of smartphone apps with less intensive support had very low certainty, and further randomized controlled trials are needed to test these interventions. In this sense, the results of this study aim to contribute to an increase in the certainty level of the available evidence in this domain.

    When focusing on the assessment of recommender systems, such as the one used in this study, the scientific community has extensively concentrated on improving the performance of recommender system algorithms with different metrics [54-58], mainly in nonhealth contexts, such as leisure [59] and e-commerce [60,61]. Among these approaches, common assessments for prediction accuracy are as follows: (1) mean absolute error and root mean squared error, as well as their normalized and averaged variants to predict ratings that users would give to items when the actual recommendation ratings of the items are known for the whole test set; (2) precision, recall, and false positive rate [58,62] for prediction of the usage of the recommendations; and (3) normalized discounted cumulative gain when the system presents a large list of elements (similar to a search in Google), where we expect that the most relevant elements are shown at the top of the list; and (4) coverage of the recommendation set [63]. Other authors have proposed the use of surveys to assess recommender systems, such as ResQue [33].

    Regarding the evaluation of mobile Health (mHealth) apps for behavioral change, such as the one included in this study, some authors have proposed evaluation procedures, such as the National Institute for Health and Care Excellence adaptation [64], the Mobile App Rating Scale framework [65], and the Application Usage Factor, which is defined as the logarithm of the product of the number of active users of a mobile app and the median number of daily uses of the app [66]. Despite these attempts to set a methodological framework to assess mHealth apps, they still lack standardization and comprehensiveness [67]. In a recent study, McKay et al identified that there was no available related best practice [68]. Instead, they proposed the following generic guidelines for what these types of evaluations should include: (1) assessment of the quality of health-related content; (2) review of the usability and functionality of the app; and (3) critique of the app potential with regard to behavioral change promotion.

    Despite the lack of consensus on specific evaluation methods, it is well accepted that engagement is a key element for mHealth solutions to be successful [69]. Engagement has been considered a positive early indicator for behavioral change [70], as the process of behavioral change requires time. Previous studies showed that technologies supporting reaching a specific behavior health goal help people stick to the desired goal [71,72]. Further, Scherer et al [73] found a significant positive correlation between engagement and fewer dropouts in an mHealth intervention. A recent approach to provide an engagement score for a mHealth app is the Engagement Index proposed by Taki et al [74]. To calculate the score, the authors combined the number of pages in the app that a participant visited each day, number of app accesses during the program, number of push notifications opened, elapsed time between app accesses, and subjective answers to a questionnaire. However, achieving user engagement is difficult [75], and its level could be improved [76-78]. Therefore, it seems reasonable to consider engagement as a key metric for digital health solutions. However, the definition of engagement varies among studies. For instance, Iacoviello et al [79] considered the number of times users opened an app, number of interactions they had, and number of weeks in which users had at least one interaction with the program as indicators of engagement. Owen et al [80] determined engagement by calculating several variables, such as the number of downloads and number of sessions (a session being basically opening of the app). Yet, it is common to just use opening the app as an indicator of engagement, as adopted in previous studies [40,81]. All these engagement interpretations fall under the categorization of “system usage data” proposed by Short et al [82], which is most frequently used. However, other engagement measurement approaches have been proposed, such as ecological momentary assessments, psychophysiological measurements, and qualitive methods.

    In our So-Lo-Mo study, the HRS was intuitively related to user engagement, as some studies have shown that good and timely recommendations to stop smoking motivate users to read more future recommendations [74,83]. Hence, we proposed a combination of qualitative and quantitative metrics, which measure the intended purpose of the system (quality of the recommendations, engagement of the participants with the system, and participants’ perception of the system). To our knowledge, this is the first assessment involving a HRS. However, in the case of smoking cessation, low engagement may not necessarily result in low impact on behavioral change, as some studies have previously shown the “gateway effect” [84] and “happy abandonment” [85]. There are several reasons for this, including the burden of the required interactions as compared with perceived outcomes or the internalization of health habits denoting that support is perceived as no longer necessary [86]. Consequently, we needed to carefully analyze the engagement of users across time points keeping in mind all these factors to extract any conclusions. Thus, our system’s intended usage [82] is expected to be higher during the first weeks of the smoking cessation process and to progressively decrease until there is no engagement with the system, which will be reflected in engagement measurements.


    The So-Lo-Mo intervention offers a promising strategy for smoking cessation. The use of this digital therapeutic solution alongside pharmacological treatment was much more efficacious in achieving tobacco abstinence as compared with pharmacological treatment alone at the 1-year follow-up. However, this intervention did not improve participant HRQoL and physical activity levels.

    Analysis of the impact of the HRS showed that participants benefited from the recommendations, and those who engaged with the system were more likely to succeed in smoking cessation. Therefore, the proposed HRS had a positive impact on the participants and offered them personalized and relevant messages, although it could not determine when supportive messages should be sent to minimize the time until they are read.

    Health care providers should consider incorporating this digital therapeutic solution in their usual care, as it can facilitate positive outcomes for participants willing to stop smoking.


    The digital therapeutic solution used in the So-Lo-Mo study was jointly developed by Salumedia Tecnologías SLU, the Aristotle University of Thessaloniki, Servicio Andaluz de Salud, and Universidad de Sevilla based on the SmokeFreeBrain project. We acknowledge Prof Hein de Vries and Dr Francine Schneider who provided guidance in the data analysis of the technical outcomes. This research was funded by the H2020 European Commission research and innovation program (grant agreement 681120) as part of the SmokeFreeBrain project (

    Conflicts of Interest

    SHF is a product manager at Salumedia Tecnologías SLU, the company with exploitation rights to the digital therapeutic solution used in the So-Lo-Mo study (DigiQuit). He contributed to technical data extraction for the precision, time-to-open the messages, and perceived quality metrics and their analysis. He was not involved in the clinical trial design, execution, or analysis of the resulting clinical data. Some institutions of all the other authors (University of Seville, the Aristotle University of Thessaloniki, and the Servicio Andaluz de Salud as a legal representative institution for the Virgen del Rocío University Hospital) signed an exploitation agreement with Salumedia Tecnologías SLU to benefit from DigiQuit commercialization. This agreement was elaborated and signed before publishing the present results but after the trial was finished and its results were generated.

    Multimedia Appendix 1

    Incomplete data analysis.

    DOC File , 81 KB

    Multimedia Appendix 2

    Health recommender system performance analysis, Precision_p.

    DOCX File , 42 KB

    Multimedia Appendix 3

    Characteristics of the dropout and non-dropout within each group.

    DOC File , 126 KB

    Multimedia Appendix 4

    CONSORT-EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 3306 KB


    1. WHO. WHO Global Report On Trends In Prevalence Of Tobacco Smoking, 2015. Geneva: World Health Organization; 2015.
    2. Goodchild M, Nargis N, Tursan d'Espaignet E. Global economic cost of smoking-attributable diseases. Tob Control 2017 Jan 30;27(1):58-64. [CrossRef]
    3. Cahill K, Stevens S, Perera R, Lancaster T. Pharmacological interventions for smoking cessation: an overview and network meta-analysis. Cochrane Database Syst Rev 2013 May 31(5):CD009329. [CrossRef] [Medline]
    4. Reid RD, Pritchard G, Walker K, Aitken D, Mullen K, Pipe AL. Managing smoking cessation. CMAJ 2016 Dec 06;188(17-18):E484-E492 [FREE Full text] [CrossRef] [Medline]
    5. Cahill K, Lindson-Hawley N, Thomas K, Fanshawe T, Lancaster T. Nicotine receptor partial agonists for smoking cessation. Cochrane Database Syst Rev 2016 May 09(5):CD006103 [FREE Full text] [CrossRef] [Medline]
    6. Condon LA, Coulson NS. Chapter 2 - Designing and Delivering Interventions for Health Behavior Change in Adolescents Using Multitechnology Systems: From Identification of Target Behaviors to Implementation. In: Behavior Change Research and Theory. Cambirdge, Massachusetts: Elsevier; 2017:27-45.
    7. Elbert SP, Dijkstra A, Oenema A. A Mobile Phone App Intervention Targeting Fruit and Vegetable Consumption: The Efficacy of Textual and Auditory Tailored Health Information Tested in a Randomized Controlled Trial. J Med Internet Res 2016 Jun 10;18(6):e147 [FREE Full text] [CrossRef] [Medline]
    8. Zhao J, Freeman B, Li M. Can Mobile Phone Apps Influence People's Health Behavior Change? An Evidence Review. J Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [CrossRef] [Medline]
    9. Teasdale N, Elhussein A, Butcher F, Piernas C, Cowburn G, Hartmann-Boyce J, et al. Systematic review and meta-analysis of remotely delivered interventions using self-monitoring or tailored feedback to change dietary behavior. Am J Clin Nutr 2018 Feb 01;107(2):247-256 [FREE Full text] [CrossRef] [Medline]
    10. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database Syst Rev 2016 Apr 10;4:CD006611 [FREE Full text] [CrossRef] [Medline]
    11. Brug J, Glanz K, Van Assema P, Kok G, van Breukelen GJ. The impact of computer-tailored feedback and iterative feedback on fat, fruit, and vegetable intake. Health Educ Behav 1998 Aug 30;25(4):517-531. [CrossRef] [Medline]
    12. Dijkstra A, De Vries H, Roijackers J, van Breukelen G. Tailoring information to enhance quitting in smokers with low motivation to quit: Three basic efficacy questions. Health Psychology 1998;17(6):513-519. [CrossRef]
    13. Dijkstra A. The Psychology of Tailoring-Ingredients in Computer-Tailored Persuasion. Social Pers Psych Compass 2008 Mar;2(2):765-784. [CrossRef]
    14. Enwald HP, Huotari MA. Preventing the obesity epidemic by second generation tailored health communication: an interdisciplinary review. J Med Internet Res 2010 Jun 28;12(2):e24 [FREE Full text] [CrossRef] [Medline]
    15. Kamel Ghalibaf A, Nazari E, Gholian-Aval M, Tara M. Comprehensive overview of computer-based health information tailoring: a systematic scoping review. BMJ Open 2019 Jan 09;9(1):e021022 [FREE Full text] [CrossRef] [Medline]
    16. Kreuter M, Farrell D, Olevitch L, Brennan L. Customizing communication with computer technology. In: Routledge , editor. Tailored health messages. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc; 2000.
    17. Cupertino AP, Cartujano-Barrera F, Ramírez M, Rodríguez-Bolaños R, Thrasher JF, Pérez-Rubio G, et al. A Mobile Smoking Cessation Intervention for Mexico (Vive sin Tabaco... ¡Decídete!): Single-Arm Pilot Study. JMIR Mhealth Uhealth 2019 Apr 25;7(4):e12482 [FREE Full text] [CrossRef] [Medline]
    18. Bamidis P, Paraskevopoulos E, Konstantinidis E, Spachos D, Billis A. Multimodal e-health services for smoking cessation and public health: the smokefreebrain project approach. In: Proceedings of the 16th World Congress on Medical and Health Informatics. 2017 Presented at: MEDINFO 2017: Precision Healthcare through Informatics; August 2017; Hangzhou p. 5-9. [CrossRef]
    19. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot 1997 Aug 26;12(1):38-48. [CrossRef] [Medline]
    20. Hors-Fraile S, Nuñes Benjumea FJ, Carrasco Hernández L, Ruiz FO, Fernandez-Luque L. Design of two combined health recommender systems for tailoring messages in a smoking cessation app. In: arXiv preprint. 2016 Presented at: International Workshop on Engendering Health with RecSys co-located with ACM RecSys; 2016; Boston, Massachusetts p. 160807192   URL:
    21. Hors-Fraile S, de Vries H, Malwade S, Luna-Perejon F, Amaya C, Civit A, et al. Opening the Black Box: Explaining the Process of Basing a Health Recommender System on the I-Change Behavioral Change Model. IEEE Access 2019;7:176525-176540. [CrossRef]
    22. Hors-Fraile S, Rivera-Romero O, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit-Balcells A, et al. Analyzing recommender systems for health promotion using a multidisciplinary taxonomy: A scoping review. Int J Med Inform 2018 Jun;114:143-155. [CrossRef] [Medline]
    23. Jódar-Sánchez F, Carrasco Hernández L, Núñez-Benjumea FJ, Mesa González MA, Moreno Conde J, Parra Calderón CL, et al. Using the Social-Local-Mobile App for Smoking Cessation in the SmokeFreeBrain Project: Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2018 Dec 06;7(12):e12464 [FREE Full text] [CrossRef] [Medline]
    24. Hors-Fraile S, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit A, Spachos D, et al. Tailoring motivational health messages for smoking cessation using an mHealth recommender system integrated with an electronic health record: a study protocol. BMC Public Health 2018 Jun 05;18(1):698 [FREE Full text] [CrossRef] [Medline]
    25. Heatherton TF, Kozlowski LT, Frecker RC, Fagerström KO. The Fagerström Test for Nicotine Dependence: a revision of the Fagerström Tolerance Questionnaire. Br J Addict 1991 Sep;86(9):1119-1127. [CrossRef] [Medline]
    26. Richmond RL, Kehoe LA, Webster IW. Multivariate models for predicting abstention following intervention to stop smoking by general practitioners. Addiction 1993 Aug;88(8):1127-1135. [CrossRef] [Medline]
    27. Pearce MS, Hayes L, Newcastle Heart Project, Newcastle Thousand Families Study. Self-reported smoking status and exhaled carbon monoxide: results from two population-based epidemiologic studies in the North of England. Chest 2005 Sep;128(3):1233-1238. [CrossRef] [Medline]
    28. Pérez-Trullén A, Bartolomé CB, Banegas JR. [New perspectives in the diagnosis and evolution of the consumption of tobacco: markers of susceptibility and damage]. Med Clin (Barc) 2006 Apr 29;126(16):628-631. [CrossRef] [Medline]
    29. Vilagut G, Ferrer M, Rajmil L, Rebollo P, Permanyer-Miralda G, Quintana JM, et al. [The Spanish version of the Short Form 36 Health Survey: a decade of experience and new developments]. Gac Sanit 2005 Apr;19(2):135-150 [FREE Full text] [CrossRef] [Medline]
    30. Ramos-Goñi JM, Craig BM, Oppe M, Ramallo-Fariña Y, Pinto-Prades JL, Luo N, et al. Handling Data Quality Issues to Estimate the Spanish EQ-5D-5L Value Set Using a Hybrid Interval Regression Approach. Value Health 2018 May;21(5):596-604 [FREE Full text] [CrossRef] [Medline]
    31. Booth M. Assessment of physical activity: an international perspective. Res Q Exerc Sport 2000 Jun 13;71 Suppl 2(sup2):114-120. [CrossRef] [Medline]
    32. Cavelaars M, Rousseau J, Parlayan C, de Ridder S, Verburg A, Ross R, et al. OpenClinica. J Clin Bioinformatics 2015;5(Suppl 1):S2. [CrossRef]
    33. Pu P, Chen L, Hu R, editors. A user-centric evaluation framework for recommender systems. : ACM Digital Library; 2011 Oct Presented at: RecSys '11: Fifth ACM Conference on Recommender Systems; October 2011; Chicago, Illinois p. 157-164   URL: [CrossRef]
    34. de Vries H. An Integrated Approach for Understanding Health Behavior; The I-Change Model as an Example. PBSIJ 2017 Mar 9;2(2). [CrossRef]
    35. Jamshidian M, Jalal S. Tests of homoscedasticity, normality, and missing completely at random for incomplete multivariate data. Psychometrika 2010 Dec 3;75(4):649-674 [FREE Full text] [CrossRef] [Medline]
    36. Breiman L. Random forests. Machine learning 2001;45(1):5-32 [FREE Full text] [CrossRef]
    37. Fox J, Monette G. Generalized Collinearity Diagnostics. Journal of the American Statistical Association 1992 Mar;87(417):178-183. [CrossRef]
    38. Garrison KA, Pal P, Rojiani R, Dallery J, O'Malley SS, Brewer JA. A randomized controlled trial of smartphone-based mindfulness training for smoking cessation: a study protocol. BMC Psychiatry 2015 Apr 14;15(1):83 [FREE Full text] [CrossRef] [Medline]
    39. Heffner JL, Vilardaga R, Mercer LD, Kientz JA, Bricker JB. Feature-level analysis of a novel smartphone application for smoking cessation. Am J Drug Alcohol Abuse 2015 Jan 14;41(1):68-73 [FREE Full text] [CrossRef] [Medline]
    40. Ubhi HK, Michie S, Kotz D, Wong WC, West R. A mobile app to aid smoking cessation: preliminary evaluation of SmokeFree28. J Med Internet Res 2015 Jan 16;17(1):e17 [FREE Full text] [CrossRef] [Medline]
    41. Germeroth LJ, Levine MD. Postcessation weight gain concern as a barrier to smoking cessation: Assessment considerations and future directions. Addict Behav 2018 Jan;76:250-257 [FREE Full text] [CrossRef] [Medline]
    42. Goldenberg M, Danovitch I, IsHak WW. Quality of life and smoking. Am J Addict 2014 Sep 25;23(6):540-562. [CrossRef] [Medline]
    43. Fu S, Dao M, Wong CK, Yu W. SF-6D utility scores of smokers and ex-smokers with or without respiratory symptoms attending primary care clinics. Health Qual Life Outcomes 2019 Mar 15;17(1):48 [FREE Full text] [CrossRef] [Medline]
    44. Almario CV. The Effect of Digital Health Technology on Patient Care and Research. Gastroenterol Hepatol (N Y) 2017 Jul;13(7):437-439 [FREE Full text] [Medline]
    45. Kathleen FH, Young-Il K, Meifang C, Rekha R, Maria P, Rajani SS, et al. Web-Based Intervention for Transitioning Smokers From Inpatient to Outpatient Care: An RCT. Am J Prev Med 2016 Oct;51(4):620-629. [CrossRef] [Medline]
    46. Taylor G, Dalili M, Semwal M, Civljak M, Sheikh A, Car J. Internet-based interventions for smoking cessation. Cochrane Database Syst Rev 2017 Sep 04;9:CD007078 [FREE Full text] [CrossRef] [Medline]
    47. Baltrunas L, Ludwig B, Peer S, Ricci F. Context relevance assessment and exploitation in mobile recommender systems. Pers Ubiquit Comput 2011 Jun 21;16(5):507-526. [CrossRef]
    48. BinDhim NF, McGeechan K, Trevena L. Smartphone Smoking Cessation Application (SSC App) trial: a multicountry double-blind automated randomised controlled trial of a smoking cessation decision-aid 'app'. BMJ Open 2018 Jan 21;8(1):e017105 [FREE Full text] [CrossRef] [Medline]
    49. Garrison K, Pal P, O'Malley SS, Pittman B, Gueorguieva R, Rojiani R, et al. Craving to Quit: A Randomized Controlled Trial of Smartphone App-Based Mindfulness Training for Smoking Cessation. Nicotine Tob Res 2020 Mar 16;22(3):324-331. [CrossRef] [Medline]
    50. Baskerville NB, Struik LL, Guindon GE, Norman CD, Whittaker R, Burns C, et al. Effect of a Mobile Phone Intervention on Quitting Smoking in a Young Adult Population of Smokers: Randomized Controlled Trial. JMIR Mhealth Uhealth 2018 Oct 23;6(10):e10893 [FREE Full text] [CrossRef] [Medline]
    51. Hassandra M, Lintunen T, Hagger MS, Heikkinen R, Vanhala M, Kettunen T. An mHealth App for Supporting Quitters to Manage Cigarette Cravings With Short Bouts of Physical Activity: A Randomized Pilot Feasibility and Acceptability Study. JMIR Mhealth Uhealth 2017 May 26;5(5):e74 [FREE Full text] [CrossRef] [Medline]
    52. McClure JB, Anderson ML, Bradley K, An LC, Catz SL. Evaluating an Adaptive and Interactive mHealth Smoking Cessation and Medication Adherence Program: A Randomized Pilot Feasibility Study. JMIR Mhealth Uhealth 2016 Aug 03;4(3):e94 [FREE Full text] [CrossRef] [Medline]
    53. Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y, Dobson R. Mobile phone text messaging and app-based interventions for smoking cessation. Cochrane Database Syst Rev 2019 Oct 22;10:CD006611. [CrossRef] [Medline]
    54. Cremonesi P, Koren Y, Turrin R, editors. Performance of recommender algorithms on top-n recommendation tasks. : RecSys '10; 2010 Sep Presented at: Fourth ACM Conference on Recommender Systems; 2010; Barcelona, Spain p. 39-46   URL: [CrossRef]
    55. Hernández del Olmo F, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Systems with Applications 2008 Oct;35(3):790-804. [CrossRef]
    56. Ricci F, Rokach L, Shapira B, Kantor P. Introduction to recommender systems handbook. In: Recommender Systems Handbook. Boston, Massachusetts: Springer; 2011:1-35.
    57. Schröder G, Thiele M, Lehner W. Setting goals and choosing metrics for recommender system evaluations. In: The RecSys 2011 Workshops - Decisions@RecSys 2011 and UCERSTI-2. 2011 Oct 23 Presented at: UCERSTI2 Workshop at the 5th ACM conference on recommender systems; 2011; Chicago p. 78-85   URL:
    58. Shani G, Gunawardana A. Evaluating recommendation systems. In: Recommender systems handbook. Boston, Massachusetts: Springer; 2011:257-297.
    59. Bennett J, Lanning S. The netflix prize. In: Proceedings of KDD Cup and Workshop 2007. 2007 Aug 12 Presented at: KDD Cup and Workshop; 2007; San Jose, California   URL:
    60. Li S, Karahanna E. Online Recommendation Systems in a B2C E-Commerce Context: A Review and Future Directions. JAIS 2015 Feb;16(2):72-107. [CrossRef]
    61. Schafer J, Konstan J, Riedl J. E-commerce recommendation applications. Data mining and knowledge discovery 2001 Jan;5(1-2):115-153 [FREE Full text] [CrossRef]
    62. Herlocker JL, Konstan JA, Terveen LG, Riedl JT. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst 2004 Jan;22(1):5-53. [CrossRef]
    63. Ge M, Delgado-Battenfeld C, Jannach D. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: RecSys '10: Proceedings of the fourth ACM conference on Recommender systems. 2010 Sep Presented at: The fourth ACM conference on Recommender systems; 2010; Barcelona, Spain p. 257-260   URL: [CrossRef]
    64. McMillan B, Hickey E, Patel MG, Mitchell C. Quality assessment of a sample of mobile app-based health behavior change interventions using a tool based on the National Institute of Health and Care Excellence behavior change guidance. Patient Educ Couns 2016 Mar;99(3):429-435. [CrossRef] [Medline]
    65. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    66. Lewis TL, Wyatt JC. App Usage Factor: A Simple Metric to Compare the Population Impact of Mobile Medical Apps. J Med Internet Res 2015 Aug 19;17(8):e200 [FREE Full text] [CrossRef] [Medline]
    67. Grundy QH, Wang Z, Bero LA. Challenges in Assessing Mobile Health App Quality: A Systematic Review of Prevalent and Innovative Methods. Am J Prev Med 2016 Dec;51(6):1051-1059. [CrossRef] [Medline]
    68. McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M. Evaluating mobile phone applications for health behaviour change: A systematic review. J Telemed Telecare 2016 Oct 18;24(1):22-30. [CrossRef]
    69. Singh K, Drouin K, Newmark L, Rozenblum R, Lee J, Landman A. Developing a framework for evaluating the patient engagement, quality, and safety of mobile health applications. Issue Brief, Commonw Fund Pub 2016 Feb 18;5:11 [FREE Full text] [CrossRef]
    70. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions. Am J Prev Med 2016 Nov;51(5):833-842. [CrossRef] [Medline]
    71. Klasnja P, Consolvo S, Pratt W. How to evaluate technologies for health behavior change in HCI research. In: CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011 May Presented at: The SIGCHI conference on human factors in computing systems; 2011; Vancouver, Canada p. 3063-3072   URL: [CrossRef]
    72. Smith W, Ploderer B, Wadley G, Webber S, Borland R. Trajectories of engagementdisengagement with a story-based smoking cessation app. In: CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017 May Presented at: CHI '17: CHI Conference on Human Factors in Computing Systems; May, 2017; Denver, Colorado, USA p. 3045-3056   URL: [CrossRef]
    73. Scherer EA, Ben-Zeev D, Li Z, Kane JM. Analyzing mHealth Engagement: Joint Models for Intensively Collected User Engagement Data. JMIR Mhealth Uhealth 2017 Jan 12;5(1):e1 [FREE Full text] [CrossRef] [Medline]
    74. Taki S, Lymer S, Russell CG, Campbell K, Laws R, Ong K, et al. Assessing User Engagement of an mHealth Intervention: Development and Implementation of the Growing Healthy App Engagement Index. JMIR Mhealth Uhealth 2017 Jun 29;5(6):e89 [FREE Full text] [CrossRef] [Medline]
    75. O'Connor S, Hanlon P, O'Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak 2016 Sep 15;16(1):120 [FREE Full text] [CrossRef] [Medline]
    76. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    77. Kohl LF, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res 2013 Jul 16;15(7):e146 [FREE Full text] [CrossRef] [Medline]
    78. Anscombe J, Bacardit A. AT Kearney Insights. Chicago, Illinois: AT Kearney; 2011. Is Mobile the Cure for the Rising Costs of Healthcare?   URL: https:/​/www.​​documents/​10192/​422267/​EAXIV_1_Is_Mobile_a_Cure_for_the_Rising_Costs_of_Healthcare.pdf/​7337673b-bd46-4ae5-aa85-1906f83f6a05 [accessed 2016-02-20] [WebCite Cache]
    79. Iacoviello BM, Steinerman JR, Klein DB, Silver TL, Berger AG, Luo SX, et al. Clickotine, A Personalized Smartphone App for Smoking Cessation: Initial Evaluation. JMIR Mhealth Uhealth 2017 Apr 25;5(4):e56 [FREE Full text] [CrossRef] [Medline]
    80. Owen JE, Jaworski BK, Kuhn E, Makin-Byrd KN, Ramsey KM, Hoffman JE. mHealth in the Wild: Using Novel Data to Examine the Reach, Use, and Impact of PTSD Coach. JMIR Ment Health 2015 Mar 25;2(1):e7 [FREE Full text] [CrossRef] [Medline]
    81. Bricker JB, Copeland W, Mull KE, Zeng EY, Watson NL, Akioka KJ, et al. Single-arm trial of the second version of an acceptance & commitment therapy smartphone application for smoking cessation. Drug Alcohol Depend 2017 Jan 01;170:37-42 [FREE Full text] [CrossRef] [Medline]
    82. Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring Engagement in eHealth and mHealth Behavior Change Interventions: Viewpoint of Methodologies. J Med Internet Res 2018 Nov 16;20(11):e292 [FREE Full text] [CrossRef] [Medline]
    83. Okoshi T, Tsubouchi K, Taji M, Ichikawa T, Tokuda H. Attention and engagement-awareness in the wild: A large-scale study with adaptive notifications. 2017 Mar 17 Presented at: 2017 IEEE International Conference on Pervasive Computing and Communications, PerCom 2017; March, 2017; Big Island, United States p. 100-110   URL: [CrossRef]
    84. Schwanda V, Ibara S, Reynolds L, Cosley D. Side effects and gateway tools: advocating a broader look at evaluating persuasive systems. In: CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Side effectsgateway tools: advocating a broader look at evaluating persuasive systems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2011 May Presented at: CHI '11: CHI Conference on Human Factors in Computing Systems; May, 2011; Vancouver, Canada p. 345-348   URL: [CrossRef]
    85. Clawson J, Pater J, Miller A, Mynatt E, Mamykina L. No longer wearing: investigating the abandonment of personal health-tracking technologies on craigslist. In: UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2015 Sep Presented at: UbiComp '15: The 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing; September, 2015; Osaka, Japan p. 647-658   URL: [CrossRef]
    86. Epstein D, Caraway M, Johnston C, Ping A, Fogarty J, Munson S. Beyond Abandonment to Next Steps: Understanding and Designing for Life after Personal Informatics Tool Use. In: CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016 May Presented at: CHI'16: CHI Conference on Human Factors in Computing Systems; May 2016; San Jose, California, USA p. 1109-1113   URL: [CrossRef]


    AI: artificial intelligence
    BMI: body mass index
    CG: control group
    EuroQoL-5D-5L: EuroQol 5-dimension 5-level
    HRS: health recommender system
    IG: intervention group
    IPAQ: International Physical Activity Questionnaire
    ResQue: Recommender Systems Questionnaire of User Experience
    RRR: relative risk ratio
    SF-36: 36-item Short Form

    Edited by G Eysenbach; submitted 18.12.19; peer-reviewed by C Hao, A Vallée; comments to author 13.01.20; revised version received 03.03.20; accepted 21.03.20; published 27.04.20

    ©Laura Carrasco-Hernandez, Francisco Jódar-Sánchez, Francisco Núñez-Benjumea, Jesús Moreno Conde, Marco Mesa González, Antón Civit-Balcells, Santiago Hors-Fraile, Carlos Luis Parra-Calderón, Panagiotis D Bamidis, Francisco Ortega-Ruiz. Originally published in JMIR mHealth and uHealth (, 27.04.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.