This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.
SMS text messages as a form of mobile health are increasingly being used to support individuals with chronic diseases in novel ways that leverage the mobility and capabilities of mobile phones. However, there are knowledge gaps in mobile health, including how to maximize engagement.
This study aims to categorize program SMS text messages and participant replies using machine learning (ML) and to examine whether message characteristics are associated with premature program stopping and engagement.
We assessed communication logs from SMS text message–based chronic disease prevention studies that encouraged 1-way (SupportMe/ITM) and 2-way (TEXTMEDS [Text Messages to Improve Medication Adherence and Secondary Prevention]) communication. Outgoing messages were manually categorized into 5 message intents (informative, instructional, motivational, supportive, and notification) and replies into 7 groups (stop, thanks, questions, reporting healthy, reporting struggle, general comment, and other). Grid search with 10-fold cross-validation was implemented to identify the best-performing ML models and evaluated using nested cross-validation. Regression models with interaction terms were used to compare the association of message intent with premature program stopping and engagement (replied at least 3 times and did not prematurely stop) in SupportMe/ITM and TEXTMEDS.
We analyzed 1550 messages and 4071 participant replies. Approximately 5.49% (145/2642) of participants responded with
ML models enable monitoring and detailed characterization of program messages and participant replies. Outgoing message intent may influence premature program stopping and engagement, although the strength and direction of association appear to vary by program type. Future studies will need to examine whether modifying message characteristics can optimize engagement and whether this leads to behavior change.
Mobile health (mHealth) is increasingly being used to support individuals with chronic diseases in novel ways that leverage the mobility and capabilities of mobile phones [
Understanding how SMS text message–based program content affects participant replies may aid in optimizing future SMS text message–based programs. For example, SMS text message content that have been associated with premature program withdrawal can be avoided, and content that participants engage most with can be used more frequently. Engagement with mHealth programs has been considered an important factor in their effectiveness [
Our team has previously developed and supported patients with SMS text message–based mHealth programs who have chronic diseases, including CVD [
We analyzed our combined communication logs from 3 Australian SMS text message–based digital health programs (SupportMe, ACTRN12616001689460 [
SMS text message–based prevention programs for metabolic disease.
Project | Duration | 2-way communication encourageda | Population | Recruitment number | Number of replies | ||||
|
|
|
|
Total | Lost to follow-up | Withdrawn consent | Deaths |
|
|
TEXTMEDSb [ |
12 months | Yes | CVDc (recruited from hospital post-ACSd) | 1424 (716 in intervention arm; 1:1 allocation) |
39 (9 in intervention) |
6 (1 in intervention) |
15 (10 in intervention) | 2356 | |
ITM (support for patients with respiratory disease and CVD via integrated SMS text messaging) [ |
6 months | No | CVD and respiratory disease (recruited from community with one or more chronic conditions) | 316 (236 in intervention arm, 80 in control arm; 3:1 allocation) |
26 (22 in intervention) |
19 (12 in intervention) |
4 (3 in intervention) | 417 | |
SupportMe (SMS text messaging support for patients with chronic disease) [ |
6 monthse | No | CVD and diabetes (recruited from community and hospital with one or more chronic conditions) | 902 (454 in intervention arm; 1:1 allocation) |
15 (9 in intervention) |
7 (4 in intervention) |
9 (5 in intervention) | 1298 |
aTwo-way communication was possible with all the included SMS text message–based programs but only encouraged for TEXTMEDS (Text Messages to Improve Medication Adherence and Secondary Prevention).
bTEXTMEDS: Text Messages to Improve Medication Adherence and Secondary Prevention.
cCVD: cardiovascular disease.
dACS: acute coronary syndrome.
eA total of 7 patients in SupportMe at the conclusion of the 6-month intervention continued into a 6-month maintenance phase, which consisted of receiving texts at half the original frequency.
A total of 2 health professionals (HK and Anu Indrawansa, Westmead Hospital, Sydney, New South Wales, Australia) manually categorized all outgoing program SMS text messages in our SMS text message bank, all replies from SupportMe/ITM, and 829 TEXTMEDS replies.
Outgoing SMS text message intent (
“[person_name] by switching from full fat to low fat milk in tea & coffee you could remove 1 kg of saturated fat from your diet a year!” [TEXTMEDS; Text Messages to Improve Medication Adherence and Secondary Prevention]
“There are many ways to increase your activity levels [person_name]. Try Tai Chi, pilates, gardening, yoga or dancing” [ITM]
“Did you know a blood test called HbA1c measures your average blood sugar over the last 3 months? Ask your doctor for a check every 3-6 months” [SupportMe]
“Cardiac drugs are safe but if you have any side effects discuss with your Dr - there are many medication options [person_name].” [TEXTMEDS]
“If you are feeling more breathless than usual, try to relax, rest and practice your breathing techniques” [ITM]
“[person_name], use up vegies by mixing them with herbs, spices & water to cook up a hearty soup” [SupportMe]
“Are you having a good week [person_name]? Just reminding you that you can text us if we can be of help” [TEXTMEDS]
“Staying calm when you are breathless really helps. Is there someone in your household who can help you stay calm when you feel uptight?” [ITM]
“Hi [person_name], you may need extra carbohydrates before, during, or after exercise to prevent low blood sugars - discuss with your healthcare team” [SupportMe]
“Dont worry [person_name] if you have a bad day. Remember that there is another chance tomorrow to choose the healthy option.” [TEXTMEDS]
“Hi [person_name], when you are quitting smoking - if you have a bad day, don’t worry & keep trying” [ITM]
“Hi [person_name], did you exercise today?” [SupportMe]
“Hi [person_name], you are now halfway through TEXTMEDS. Soon we will ring to check how you are, but don’t tell us you have been receiving messages” [TEXTMEDS]
“Hi [person_name], welcome to the ITM study. We hope you enjoy the messages. Respond STOP to opt out” [ITM]
“Hi [person_name], welcome to the SupportMe study. You are in the group that will not receive regular messages. We will contact you at 6 months” [SupportMe]
Participant replies were categorized as follows:
If an SMS text message could belong to 2 different categories or
Before developing the ML models, the outgoing messages were grouped into
Each ML model was created using a DistilBERT model pretrained on the Toronto Book Corpus and full English Wikipedia to encode word meaning and sentence structure (
We assessed associations using univariate logistic regression between outgoing message characteristics (outgoing message intents: informative, instructional, motivational, supportive, and notification) to the outcomes (1) reply type
ML model accuracy was assessed using balanced accuracy, the area under the receiver operating characteristic (ROC) curve, and multiclass classification evaluators. This was done using the Scientific Python stack (Scikit-learn 0.22, Pandas 1.1, and Matplotlib 3.3) on Python 3.7 (Python Software Foundation). Associations of message characteristics to program termination and engagement were examined using SPSS Statistics (version 26.0; SPSS Inc). Chi-square tests of independence were performed to determine associations between the ML-derived program message intent and the outcomes (1) reply type
We analyzed a total of 1550 program messages and 4071 participant replies. The total number of patients in each group for each study and the received responses are shown in
Approximately 11.43% (302/2642) of participants met our definition of being engaged. For SupportMe/ITM, 8.05% (98/1218) of participants were engaged and contributed to 78.6% (1348/1715) of the total replies in SupportMe/ITM. For TEXTMEDS, 14.33% (204/1424) of participants engaged with the program and contributed to 89.13% (2100/2356) of the total replies in TEXTMEDS. Most replies during TEXTMEDS were received during the middle of the program, with the least number shouldering this period in response to motivational and instructional message intents (
Distribution of participant reply categories by program message intent during the 12-month TEXTMEDS (Text Messages to Improve Medication Adherence and Secondary Prevention) program. The 9 peak periods (4-day duration each) were defined as those which received >40 replies within each peak period. INFO: Informative; INST: Instructional; MOTI: Motivational; NOTI: Notification; TEXTMEDS: Text Messages to Improve Medication Adherence and Secondary Prevention; SUPP: Supportive.
Machine learning performance for program message intent.
Message intent | Sensitivity (SD) | Specificity (SD) | PPVa (SD) | NPVb (SD) | FPRc (SD) | FNRd (SD) | F1-score (SD) |
INFOe | 0.797 (0.144) | 0.868 (0.072) | 0.850 (0.070) | 0.840 (0.099) | 0.132 (0.072) | 0.203 (0.144) | 0.815 (0.089) |
INSTf | 0.761 (0.169) | 0.885 (0.093) | 0.795 (0.124) | 0.887 (0.064) | 0.115 (0.093) | 0.239 (0.169) | 0.759 (0.118) |
MOTIg | 0.778 (0.242) | 0.968 (0.033) | 0.671 (0.248) | 0.986 (0.016) | 0.032 (0.033) | 0.222 (0.242) | 0.702 (0.221) |
NOTIh | 0.800 (0.400) | 0.999 (0.002) | 0.900 (0.300) | 0.994 (0.015) | 0.001 (0.002) | 0.200 (0.400) | 1.000 (0.000) |
SUPPi | 0.697 (0.296) | 0.940 (0.046) | 0.635 (0.251) | 0.962 (0.036) | 0.060 (0.046) | 0.303 (0.296) | 0.741 (0.138) |
Averagej | 0.766 (0.175) | 0.932 (0.027) | 0.763 (0.148) | 0.934 (0.027) | 0.068 (0.027) | 0.234 (0.175) | 0.782 (0.100) |
aPPV: positive predictive value.
bNPV: negative predictive value.
cFPR: false positive rate.
dFNR: false negative rate.
eINFO: Informative.
fINST: Instructional.
gMOTI: Motivational.
hNOTI: Notification.
iSUPP: Supportive.
jMacroaveraged.
Receiver operating characteristic curves for predicting program message intent. Generated under one-vs-rest assumption (ie, each curve is generated assuming a binary scenario with the selected class against all other classes). AUC: area under the curve; INFO: Informative; INST: Instructional; MOTI: Motivational; NOTI: Notification; SUPP: Supportive.
Machine learning performance for participant reply categories.
Participant replies | Sensitivity (SD) | Specificity (SD) | PPVa (SD) | NPVb (SD) | FPRc (SD) | FNRd (SD) | F1-score (SD) |
General comment | 0.684 (0.121) | 0.893 (0.055) | 0.817 (0.074) | 0.815 (0.073) | 0.107 (0.055) | 0.316 (0.121) | 0.737 (0.079) |
Thanks | 0.911 (0.050) | 0.959 (0.026) | 0.863 (0.090) | 0.972 (0.027) | 0.041 (0.026) | 0.089 (0.050) | 0.771 (0.099) |
Question | 0.815 (0.213) | 0.976 (0.014) | 0.474 (0.157) | 0.995 (0.007) | 0.024 (0.014) | 0.185 (0.213) | 0.592 (0.174) |
Reporting healthy | 0.707 (0.097) | 0.940 (0.037) | 0.601 (0.198) | 0.960 (0.040) | 0.060 (0.037) | 0.293 (0.097) | 0.623 (0.111) |
Reporting struggle | 0.649 (0.167) | 0.979 (0.012) | 0.696 (0.136) | 0.976 (0.013) | 0.021 (0.012) | 0.351 (0.167) | 0.658 (0.106) |
Stop | 0.860 (0.147) | 0.993 (0.008) | 0.888 (0.131) | 0.992 (0.009) | 0.007 (0.008) | 0.140 (0.147) | 0.866 (0.116) |
Other | 0.818 (0.082) | 0.956 (0.029) | 0.740 (0.132) | 0.972 (0.016) | 0.044 (0.029) | 0.182 (0.082) | 0.885 (0.065) |
Averagee | 0.778 (0.048) | 0.957 (0.012) | 0.726 (0.071) | 0.955 (0.013) | 0.043 (0.012) | 0.222 (0.048) | 0.733 (0.054) |
aPPV: positive predictive value.
bNPV: negative predictive value.
cFPR: false positive rate.
dFNR: false negative rate.
eMacroaveraged.
Receiver operating characteristic curves for predicting participant reply type. Generated under one-vs-rest assumption (ie, each curve is generated assuming a binary scenario with the selected category against all other categories). AUC: area under the curve.
Overall,
Univariate logistic regression with program message intent and outcome variables (premature program stopping and engagement).
Message intent, outcome variables | Total | SupportMe/ITM | TEXTMEDSa | |||||||||||
|
ORc (95% CI) | β coefficient | OR (95% CI) | β coefficient | OR (96% CI) |
β coefficient |
|
|||||||
|
||||||||||||||
|
INFOd | 0.69 (0.49-0.98) | –0.37 | .04 | 0.35 (0.20-0.60) | –1.05 | <.001 | 1.25 (0.78-2.02) | 0.23 | .35 | <.001 | |||
|
INSTe | 0.98 (0.69-1.40) | –0.02 | .93 | 0.86 (0.54-1.38) | –0.15 | .54 | 1.03 (0.60-1.76) | 0.03 | .92 | .63 | |||
|
SUPPf | 0.53 (0.35-0.81) | –0.64 | .003 | 0.60 (0.29-1.26) | –0.51 | .18 | 0.58 (0.34-0.99) | –0.54 | .05 | .94 | |||
|
MOTIg | 1.00 (0.52-1.91) | –0.00 | .99 | 1.08 (0.43-2.73) | 0.08 | .87 | 0.98 (0.39-2.47) | –0.02 | .97 | .89 | |||
|
NOTIh | 4.01 (2.80-5.75) | 1.39 | <.001 | 5.76 (3.66-9.06) | 1.75 | <.001 | 1.89 (0.95-3.74) | 0.64 | .07 | .01 | |||
|
||||||||||||||
|
INFO | 1.76 (1.46-2.12) | 0.56 | <.001 | 2.16 (1.67-2.78) | 0.77 | <.001 | 1.62 (1.21-2.16) | 0.48 | <.001 | .14 | |||
|
INST | 1.47 (1.21-1.80) | 0.39 | <.001 | 1.68 (1.29-2.18) | 0.52 | <.001 | 1.51 (1.10-2.08) | 0.42 | .01 | .63 | |||
|
SUPP | 1.22 (1.01-1.49) | 0.20 | .04 | 1.77 (1.21-2.58) | 0.57 | .003 | 0.77 (0.60-0.98) | –0.27 | .04 | <.001 | |||
|
MOTI | 1.18 (0.82-1.70) | 0.17 | .37 | 0.82 (0.50-1.34) | –0.20 | .43 | 1.64 (0.92-2.93) | 0.50 | .10 | .07 | |||
|
NOTI | 0.14 (0.11-0.17) | –1.99 | <.001 | 0.07 (0.05-0.10) | –2.61 | <.001 | 0.28 (0.20-0.39) | –1.28 | <.001 | <.001 |
aTEXTMEDS: Text Messages to Improve Medication Adherence and Secondary Prevention.
b
cOR: odds ratio.
dINFO: Informative.
eINST: Instructional.
fSUPP: Supportive.
gMOTI: Motivational.
hNOTI: Notification.
Overall,
In this study, ML models were created to categorize program message intent and participant replies from SMS text message–based programs with good accuracy. Thus, they can enable the monitoring and detailed characterization of program messages and participant replies, which can be used to further customize SMS text message–based programs. Furthermore, this study found that program message type can influence premature program discontinuation and encourage participant engagement. However, some of these associations varied or were attenuated by program type. This suggests that participant engagement may be maximized by adjusting program message characteristics, that program type and patient population type are important to consider, and that larger studies to examine these interactions will enable further program refinement.
We have previously assessed the accuracy of ML to
A recent review of systematic reviews identified 3 reviews and 10 studies (clinical trials and feasibility studies) that measured engagement [
Features enabling remote contact with a health care professional can positively influence engagement with DHIs [
Unexpectedly,
The results of this study contribute to the field of SMS text message–based interventions by (1) demonstrating that using ML can automatically and accurately categorize SMS text messages sent to and from participants in an SMS text message–based program to support their health, and (2) providing new knowledge on how participants engage with SMS text messages and factors associated with engagement and premature program termination. Overall, our ML models for characterizing program message intent and user replies enable the ability to monitor and describe the way participants interact with different SMS text message–based prevention programs. This has implications for the optimized development of future SMS text message–based programs, as the results suggest that participant engagement may be maximized (and premature program termination avoided) by adjusting message characteristics, that is, the clinical implications of message content affecting participant withdrawal and engagement are the potential of using this knowledge to alter future messages automatically in real time to sustain engagement throughout the intervention duration. This could minimize participant withdrawal and maximize the likelihood of behavior change. This has not been assessed in previous studies, and a lack of knowledge of participant-program interactions has limited the utility of existing SMS text message–based programs.
As there may be differences in the degree of interactivity encouraged (ie, 1-way vs 2-way communication), when assessing engagement and premature program stopping, we performed an interaction analysis (in addition to analyzing the programs separately and combined) to assess if the associations were affected by program type (
Validation of these models with different SMS text message–based programs delivered to different population groups (different clinical and geographical settings across high-, middle-, and low-income countries) would assist in increasing generalizability and utility. Future research should explore the association between program engagement and intervention success or behavioral change. In addition, there is a need to determine whether modifying message characteristics can maximize participant engagement and whether this can lead to sustained behavior change.
Although manual categorization was done by health professionals, it is entirely possible that manual categorization may have differed if performed by a different group and, thus, affected the final ML models. In addition, some of the SMS text messages could be categorized into more than one category, which can also affect the final model; however, to minimize bias, we selected the majority category within each group of similar messages. Although there were differences in the populations between studies (
As discussed in the
This study compared a measure of engagement between 2 SMS text message–based programs that differ in the encouraged level of interaction (ie, 2-way vs 1-way communication). Thus, it is possible that participants replied depending on whether they were encouraged or not, and it is also possible that participants who did not engage with messages using our definition engaged with behavior change. However, almost one-third of the participants in SupportMe/ITM replied compared with one-fifth in TEXTMEDS, and, thus, using the frequency of replies (and excluding withdrawals) as a surrogate marker of engagement is a reasonable method of quantifying engagement, which is consistent with previous studies [
The association between participant sociodemographics and engagement with DHIs and behavior change was outside the scope of this study and should be explored in future research. The timing of messages sent and message length may affect engagement but was outside the scope of this study and will be assessed in future research.
In our study, using ML, we categorized outgoing and incoming messages from different SMS text message–based programs to support people with chronic diseases with good accuracy, enabling monitoring and detailed characterization of program messages and participant replies. Message intent can influence adherence (ie, not stopping the intervention) and participant engagement, although we suspect this association is affected by the type of interaction encouraged (ie, 1-way vs 2-way communication) and possibly the setting in which participants are recruited. The clinical implications include optimization of future SMS text message–based programs by using program message characteristics that maximize participant engagement and potentially behavior change.
Distribution of participant replies for each program message intent for TEXTMEDS (Text Messages to Improve Medication Adherence and Secondary Prevention) and SupportMe/ITM.
Distribution of response categories for TEXTMEDS (Text Messages to Improve Medication Adherence and Secondary Prevention) and SupportMe/ITM.
Participant reply category by program message intent for SupportMe/ITM.
Participant reply category by program message intent for TEXTMEDS (Text Messages to Improve Medication Adherence and Secondary Prevention).
cardiovascular disease
digital health intervention
mobile health
machine learning
National Health and Medical Research Council
odds ratio
randomized controlled trial
receiver operating characteristic
Text Messages to Improve Medication Adherence in Secondary Prevention
The authors wish to acknowledge the Sydney Informatics Hub at the University of Sydney for running the machine learning models, Tara Czinner for assisting with developing the project scope with the Sydney Informatics Hub, and Anu Indrawansa for assisting with managing the SMS text message bank database. HK is supported by the Royal Australian College of Physicians Fellows Research Entry Scholarship. JR is funded by the National Health and Medical Research Council (NHMRC) Career Development Fellowship (APP1143538). CKC is supported by the National Health and Medical Research Council Career Development Award (APP1105447), cofunded by a Future Leader Fellowship from the National Heart Foundation. The TEXTMEDS study was funded by an NHMRC project grant (ID APP1042290). The SupportMe study was funded by the Translational Research Grants Scheme of NSW Health. The ITM study was funded by the National Heart Foundation, NSW Cardiovascular Network Research Development project grant (101134).
HK and CKC contributed to the conception and study design. HK, CKC, AT, JR, and NWC contributed to the acquisition of raw data. HK, JN, DL, CS, and CKC contributed to the analysis and interpretation of data. HK and CKC drafted the manuscript. All authors critically reviewed the manuscript and gave final approval.
None declared.