This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
The Patient Experience Recommender System for Persuasive Communication Tailoring (PERSPeCT) is a machine learning recommender system with a database of messages to motivate smoking cessation. PERSPeCT uses the collective intelligence of users (ie, preferences and feedback) and demographic and smoking profiles to select motivating messages. PERSPeCT may be more beneficial for tailoring content to minority groups influenced by complex, personally relevant factors.
The objective of this study was to describe and evaluate the use of PERSPeCT in African American people who smoke compared with white people who smoke.
Using a quasi-experimental design, we compared African American people who smoke with a historical cohort of white people who smoke, who both received up to 30 emailed tailored messages over 65 days. People who smoke rated the daily message in terms of perceived influence on quitting smoking for 30 days. Our primary analysis compared daily message ratings between the two groups using a
The study included 119 people who smoke (African Americans, 55/119; whites, 64/119). At baseline, African American people who smoke were significantly more likely to report allowing smoking in the home (
Our study highlighted the potential of using a recommender system to personalize for African American people who smoke.
ClinicalTrials.gov NCT02200432; https://clinicaltrials.gov/ct2/show/NCT02200432
RR2-10.2196/jmir.6465
Computer-tailored health communication (CTHC) increases the personal relevance of health messaging by matching the messages to an individual or group’s characteristics [
Recommender systems have several advantages over rule-based approaches in health communication interventions, including the ability to continuously learn from user feedback (eg, liked product, products purchased) and enhance personal relevance [
This study is a pilot evaluation of our original evaluation of our recommender system in a new population, African American people who smoke. In our original experiment [NCT02200432], we recruited a general population of people who smoke, and most participants (92%) who enrolled were white. In this study, we recruited only African American people who smoke (n=55), and then compared the results with white people who smoke. Understanding the differential response to PERSPeCT between white and African American people who smoke may lead to improving PERSPeCT for African American people who smoke [
We recruited African American people who smoke (n=55) to the PERSPeCT intervention. Then, we conducted a quasi-experimental comparison of the effectiveness of PERSPeCT in the African American intervention with a nonconcurrent comparison group of white people who smoke (the historical cohort of 64 white people who smoke who had received PERSPeCT messages in a prior trial) [
The description of the recommender system is detailed elsewhere [
The system sent one message daily for 30 days. Messages were delivered to the person who smokes’s email address. At enrollment, we explained to people who smoke how the system worked. Each day, the participant was asked to rate the message by replying with a rating from 1 (strongly disagree) to 5 (strongly agree). Note that ratings are not required for the system to send more messages, however, the more ratings the system receives, the more personalized the daily messages can become. The daily messaging system was supported by a website with additional information. The website included functions designed to support cessation induction and maintenance such as information on smoking risks, tips on communicating with family members, and a library of cessation materials.
To train the machine learning artificial intelligence of PERSPeCT, we used the demographic and smoking behavior characteristics of previous participants (current or former people who smoke) and their message ratings. These participants generated 16,920 ratings of 261 messages. We comparatively tested the classical algorithms k-nearest neighbors, probabilistic matrix factorization, Bayesian probabilistic matrix factorization [BPMF], collective matrix factorization, and Bayesian collective matrix factorization to identify one that provided maximal prediction accuracy (ie, we evaluated the ability of the algorithms to generalize ratings to nontraining users). We used a strong generalization protocol that involved completely separating test users from train users, learning a model using all the train users’ ratings, freezing all non–user-specific parameters, and finally training the user-specific parameters on a subset of each test user’s observed ratings [
Recruitment of the African American PERSPeCT intervention and historical control participants was different. As such, we have conducted a detailed comparison of demographic and smoking behavior covariates. Our historical cohort of white people who smoke was recruited from the university hospital (2014) and affiliated outpatient clinics [
For both the African American people who smoke and comparison historical group, our inclusion criteria were the same (current people who smoke who were aged 18 years or older, English speaking, and had internet access). To confirm participation, all people who smoke had to complete an intake telephone call with study staff and complete an online registration. We provided incentives of up to $100 in Amazon gift cards for participation in the data collection.
As noted, all participants were required to complete an intake telephone call and log into the supportive website to complete an online consent form and a baseline questionnaire. Once registered, each participant was emailed messages selected by the PERSPeCT recommender system and asked to rate the influence of up to 30 messages within 65 days. At the end of this period, follow-up data collection was conducted with these people who smoke.
During registration, people who smoke provided information on their demographic characteristics (age, sex, race, and ethnicity), smoking behaviors, prior quit attempts, and readiness to quit (I am not thinking of quitting, I am thinking of quitting, I have set a quit date, I quit today, and I have already quit) [
During the intervention, we measured ratings of messages and engagement with the supportive website. For each message, people who smoke were asked to rate message’s influence on their motivation to quit smoking. Ratings were on a 5-point Likert scale (1=strongly disagree to 5=strongly agree). Also, participants’ visits to the supportive website was tracked using online scripts.
At 30 days of follow-up, people who smoke reported the perceived impact of the 30-day PERSPeCT experience (intervention impact) and self-reported their smoking status. Intervention impact was assessed using 7 questions. These questions included actions that are known to help a person who smokes prepare to quit (talk to a doctor about quitting smoking, get support from those around you to help quit smoking, make a list of reasons to quit smoking, and use behavioral strategies like distraction or substitution) and those that could help a person who smokes actively quitting (use NRT like the patch or gum, set a quit date, and quit smoking) [
The analytic plan followed the flow of data collection. We first compared data collected during the active intervention and then analyzed 30-day follow-up in the African American PERSPeCT group and the comparison white PERSPeCT group.
For each day, we created a daily rating defined as the mean of the ratings provided by all people who smoke in that group that day. We graphed this data by day and reported the percentage of days where the African Americans rated the message higher than whites. We then compared mean of daily ratings between African American and white people who smoke using a
For both the during intervention and 30-day follow-up analyses, we used Stata statistical software (StataCorp LLC). For the mediation analysis, we used the Stata medeff command and sought to quantify the effect of PERSPeCT that operates through the path of differential experience with messages (as measured by daily ratings). For the mediation analysis, we first fit a linear regression model evaluating the association of ethnicity and the mediator (daily ratings) and then a second logistic regression model with the main outcome as 30-day smoking cessation. The independent variable for the main model was ethnicity, and the mediator was daily ratings. We report the percentage of the total effect mediated by daily ratings.
African American people who smoke were significantly more likely to allow smoking in the home compared with whites (
Demographic characteristics and smoking behavior at baseline.
Characteristic | White people who smoke (n=64), n (%) | African American people who smoke (n=55), n (%) | ||
|
|
|
|
|
|
Male | 24 (37) | 22 (40) |
|
|
Female | 40 (63) | 33 (60) |
|
|
|
|
|
|
|
19-34 | 21 (33) | 17 (31) |
|
|
35-44 | 17(27) | 14 (25) |
|
|
45+ | 26(40) | 24 (44) |
|
|
|
|
|
|
|
Otherb | 51 (80) | 43 (78) |
|
|
Advanced college degree | 13 (20) | 47 (22) |
|
|
|
|
|
|
|
Not Hispanic | 59 (97) | 52 (95) |
|
|
Hispanic or Latino | 2 (3) | 3 (5) |
|
|
|
|
|
|
|
No | 40 (63) | 19 (35) |
|
|
Yes | 24 (37) | 36 (65) |
|
|
|
|
|
|
|
No | 52(81) | 48 (87) |
|
|
Yes | 12 (19) | 7 (13) |
|
|
|
|||
|
No | 36 (56) | 30 (55) |
|
|
Yes | 28 (44) | 25 (45) |
|
|
|
|
|
|
|
Not actively quitting | 52 (81) | 44 (80) |
|
|
Actively quitting | 12 (19) | 11 (20) |
|
aTested for trend using the Mantel-Haenszel method (mHodds command in Stata).
bSome high school, high school diploma, some college or technical school.
More African American people who smoke rated all 30 messages than white people who smoke (
African American people who smoke had significantly more visits to the Web-assisted tobacco intervention compared with white people who smoke (African American mean 5.5 [SD 1.3]; range 3.0-8.0 vs white mean 1.5 [SD 0.1]; range 1.2-1.7;
Proportion of people who smoke completing ratings assessments by study time periods.
African American and white people who smoke mean message ratings over time.
Perceived influence of use of NRT was significantly lower among the African American people who smoke compared with the white poeple who smoke (66% [35/53] vs 33% [17/51],
African American people who smoke were significantly more likely to quit than white people who smoke (African American 59% [30/51], white 38% [19/50],
Perceived influence on quitting strategies between African-American and white people who smoke.
Consistently across the 30 days, message ratings of African American people who smoke were higher than those of white people who smoke. Further, mean African American people who smoke ratings started higher than white people who smoke, and ratings within this group increased over time. On the contrary, mean white people who smoke ratings started lower than African American people who smoke, and ratings within this group decreased over time. Self-reported perceived influence of the intervention on use of NRT was lower among African American people who smoke than white people who smoke. Intervention engagement was higher among African Americans people who smoke compared with white people who smoke. African American people who smoke were significantly more likely to report 30-day cessation as compared with white people who smoke, and this different was mediated by experience with PERSPeCT.
There may be several reasons for the higher ratings of African American people who smoke than white people who smoek in the study. The ability to influence the message a perosn who smokes receives (by rating the message) may have provided an enhanced feeling of control over the intervention. African American people who smoke may have been more attracted to this increased autonomy, the extent to which a behavior or course of action is personally endorsed and engaged [
The lower perceived intervention impact for use of NRT among African American compared with white people who smoke may have highlighted a potential unintended consequence of using recommender systems. Evidence shows African American people who smoke are less likely to successfully quit than white people who smoke [
The 30-day cessation data highlights PERSPeCT’s potential as a cessation intervention for African American people who smoke. Culturally tailored materials have been shown to enhance the effectiveness of previous trials [
This study has some limitations. The difference in recruitment approaches may have influenced the results of our study. As noted, we were able to recruit white people who smoke within the local area (Central Massachusetts and surrounding areas). To recruit African Americans, we had to recruit in multiple states using ResearchMatch. However, as
Few systems have been able to select messages of higher influence, increasing engagement with the system and self-reported 30-day cessation rates. Additional innovations such as using a hybrid rule and recommender approach may be needed to effectively engage African American people who smoke while also motivating the use of NRT and other effective treatments. Also, since recommender systems can learn from user feedback and adapt over time, the system might be even more effective over a longer duration (6 months or a year). Future research is needed to test the long-term effectiveness of using a recommender system CTHC approach for smoking cessation in African American people who smoke to assess the true impact of PERSPeCT.
Bayesian probabilistic matrix factorization
computer-tailored health communication
nicotine replacement therapy
odds ratio
Patient Experience Recommender System for Persuasive Communication Tailoring
root mean square error
This work was supported by the National Cancer Institute under grants K07CA172677 and T32CA172009, in addition to the Patient-Centered Outcomes Research Institute award under grant PI12-001.
None declared.