Published on in Vol 8, No 6 (2020): June

Preprints (earlier versions) of this paper are available at, first published .
Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text–Based Secondary Prevention Program: Feasibility Study

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text–Based Secondary Prevention Program: Feasibility Study

Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text–Based Secondary Prevention Program: Feasibility Study

Original Paper

1Heart Research Institute, Sydney, Australia

2Faculty of Medicine and Health, University of Sydney, Sydney, Australia

3Westmead Applied Research Centre, University of Sydney, Sydney, Australia

4Department of Cardiology, Westmead Hospital, Sydney, Australia

Corresponding Author:

Nicole Lowres, BPhty, PhD

Heart Research Institute

Building D17, Charles Perkins Centre



Phone: 61 2 8627 5940


Background: SMS text messaging programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. SMS text messaging programs have the potential to increase the reach of an intervention, at a reduced cost, to larger numbers of people who may not access traditional programs. However, patients regularly reply to the SMS text messages, leading to additional staffing requirements to monitor and moderate the patients’ SMS text messaging replies. This additional staff requirement directly impacts the cost-effectiveness and scalability of SMS text messaging interventions.

Objective: This study aimed to test the feasibility and accuracy of developing a machine learning (ML) program to triage SMS text messaging replies (ie, identify which SMS text messaging replies require a health professional review).

Methods: SMS text messaging replies received from 2 clinical trials were manually coded (1) into “Is staff review required?” (binary response of yes/no); and then (2) into 12 general categories. Five ML models (Naïve Bayes, OneVsRest, Random Forest Decision Trees, Gradient Boosted Trees, and Multilayer Perceptron) and an ensemble model were tested. For each model run, data were randomly allocated into training set (2183/3118, 70.01%) and test set (935/3118, 29.98%). Accuracy for the yes/no classification was calculated using area under the receiver operating characteristics curve (AUC), false positives, and false negatives. Accuracy for classification into 12 categories was compared using multiclass classification evaluators.

Results: A manual review of 3118 SMS text messaging replies showed that 22.00% (686/3118) required staff review. For determining need for staff review, the Multilayer Perceptron model had highest accuracy (AUC 0.86; 4.85% false negatives; and 4.63% false positives); with addition of heuristics (specified keywords) fewer false negatives were identified (3.19%), with small increase in false positives (7.66%) and AUC 0.79. Application of this model would result in 26.7% of SMS text messaging replies requiring review (true + false positives). The ensemble model produced the lowest false negatives (1.43%) at the expense of higher false positives (16.19%). OneVsRest was the most accurate (72.3%) for the 12-category classification.

Conclusions: The ML program has high sensitivity for identifying the SMS text messaging replies requiring staff input; however, future research is required to validate the models against larger data sets. Incorporation of an ML program to review SMS text messaging replies could significantly reduce staff workload, as staff would not have to review all incoming SMS text messages. This could lead to substantial improvements in cost-effectiveness, scalability, and capacity of SMS text messaging–based interventions.

JMIR Mhealth Uhealth 2020;8(6):e19200



Cardiovascular disease (CVD) is the leading cause of death globally [1]. Secondary prevention interventions, such as cardiac rehabilitation, are very effective at reducing CVD and preventing cardiovascular events [2,3]. However, access to traditional cardiac rehabilitation and secondary prevention is limited due to many well-described barriers [4,5]. As a result, alternative methods of secondary prevention, including electronic health (eHealth) and mobile health (mHealth), are emerging with demonstrated effectiveness [6]. Because 88% of Australians (aged 18-75 years) now own a smartphone [7], the potential reach of mHealth programs to deliver CVD secondary prevention is significant. Smartphone mHealth interventions are shown to be an acceptable and feasible method of health care delivery [8,9]. Specifically, SMS text messaging programs have been shown to be effective for cardiac risk factor reduction [10-12], medication adherence [13,14], weight loss [15], and physical activity [16,17]. However, one study showed limited maintenance of behavior changes once the SMS text messaging intervention ceased [17]. SMS text messaging programs have also been utilized effectively in many other medical settings including diabetes [18], maternal health [19], hepatitis B [20], and smoking cessation [21].

SMS text messaging programs can be set up as either a one- or a two-way messaging communication system. The TEXTME program was an automated semipersonalized SMS text messaging program to support lifestyle change for people with coronary heart disease, which was originally delivered as a one-way communication system [12]. TEXTME sent participants 4 SMS text messages per week for 6 months, and demonstrated significant improvements in cholesterol, blood pressure, BMI, physical activity, and smoking reduction, compared with usual care [12]. In addition, participants were highly satisfied and engaged with the program, reporting numerous additional psychosocial benefits [12,22]. However, even in this one-way SMS text messaging program, where participants were specifically instructed not to respond, 32.9% (116/352) replied to the SMS text messages, with many replying multiple times [22]. The cost of delivering automated SMS text messages is relatively inexpensive (approximately AUD 13/person [US $8.60/person] for the entire program) [12]. However, when participants respond to SMS text messages there are various staffing and logistical requirements and a trained health professional is required to moderate replies. In TEXTME many replies were statements of thanks, but sometimes SMS text messages require a professional response, particularly in the case of medical distress [22], or if participants wish to withdraw from the program.

Machine learning (ML) programs have commonly been used in health care to predict outcomes [23,24], but in other applications they have been used to classify text [25,26]. It is therefore possible that a computerized ML program could be developed and trained to accurately identify which incoming SMS text messages require review by a health professional. If successful, an ML program could significantly reduce the health professional’s workload. Thus, the cost of running an SMS text messaging–based intervention would be significantly reduced, and the scalability and capacity of the program would improve substantially. Therefore, this study aimed to test the feasibility of developing an ML program to triage and identify SMS text messages requiring a health professional’s review. Specifically, the primary aim was to determine the accuracy of the ML program to sort which SMS text messaging replies require a staff member review (ie, binary classification of yes/no); and the secondary aim was to assess the accuracy of the ML program to classify each SMS text messaging reply into overall categories or themes (ie, multiclass classification).

Study Population

The SMS text messaging replies used in this analysis originated from the TEXTME and TEXTMEDS programs [12,27]. TEXTME was a 6-month program of SMS text messages for patients with coronary heart disease, where patients were explicitly instructed not to respond to SMS text messages (ACTRN12611000161921) [12]. TEXTMEDS was a 12-month program for patients with acute coronary syndrome, with a more interactive two-way SMS text messaging protocol (ACTRN12613000793718) [27]. Both studies recruited patients during hospitalization. The demographics of the TEXTME patients have previously been published [12], which included the following: mean age 57.9 (SD 9.1) years; 81.5% (287/352) men; and education level of 11 years (interquartile range 9-13).

Manual Coding of SMS Text Messaging Replies

In this study, de-identified SMS text messaging replies from the TEXTME and TEXTMEDS studies were extracted into a database. SMS text messages were entered into the database exactly as they were received, with no correction to spelling, grammar, or punctuation. The TEXTME and TEXTMEDS programs had created 22 categories of SMS text messaging replies for administrative and management purposes (eg, according to content, administrative request, or general commentary; Multimedia Appendix 1).

Two experienced cardiac health professionals (NL and Anu Indrawansa, Westmead Hospital, Sydney, Australia) independently reviewed and coded each SMS text message into the 22 categories according to the theme of the message. For the SMS text messages that expressed more than 1 theme, categorization was done according to the main theme or the theme that required a staff member review/action. For SMS text messages where a consensus on coding could not be reached, a third person (AD) was consulted. Using an iterative process, these 22 categories were condensed by NL and AD into 12 categories (Table 1), with consideration given to the type of response required for the SMS text message. Finally, each SMS text message was designated into one of two groups, depending on whether it required a staff member review or not (ie, yes or no).

Machine Learning Model Development

Five different ML models and an ensemble model were tested using Apache Spark (version 2.3.4) and the models, and their associated parameters or weights or both, were saved for further analysis. First, we assessed 4 models which are considered good models for classification purposes: Naïve Bayes, OneVsRest, Random Forest decision trees, and Gradient Boosted Trees. These models use a combination of different statistical data analytics and regression tools to assess the data and determine the likelihood of the correct response. Second, we assessed a convolutional neural network classification approach: Multilayer Perceptron. Neural networks use layers of iterative feedback cycles to calculate the outcome, and are commonly used for natural language translation in systems such as Google Translate, and for speech and image recognition. Lastly, we implemented a technique that combined the results from different models together as an Ensemble model. We evaluated the effect of using heuristics (ie, selected keywords) in each model, by programming and testing each model with and without the inclusion of heuristics. We also evaluated the addition of natural language programming which interprets and determines the intended context of written statements.

Each time a model was run, the data set was randomly divided into a training set (2183/3118, 70.01%) and a test set (935/3118, 29.98%), thus creating a unique allocation of SMS text messages to the training and test data each time. Each model was assessed and compared for determining if health professional review was required (ie, binary variable of yes/no; Figure 1). In addition, we assessed and compared each model that supported multiclass classification (Gradient Boosted Trees only supports binary classification) for its ability to correctly classify the SMS text messages into the 12 categorical variables as listed in Table 1. Each model was run 5 times on different random splits of the training and test data sets to validate the accuracy of the results.

Table 1. Manual classification of SMS text messaging replies (N=3118)
Classification categoryHealth professional review required
Yesa, n (%)Nob, n (%)
Health question/concern383 (12.28)
Administrative request155 (4.97)
Request to STOP/pause107 (3.43)
Ceased smoking24 (0.76)
SMS text message not delivered14 (0.44)
Timely response required4 (0.12)
General statement1132 (36.30)
Statement of thanks789 (25.30)
Reporting good health216 (6.92)
Blank SMS text message191 (6.12)
Positive emoticon74 (2.37)
Unrelated/accidental29 (0.93)

aTotal: 687/3118 (22.03%).

bTotal: 2431/3118 (77.96%).

Figure 1. Machine learning model development and testing.
View this figure

Statistical Analysis and Comparison of Models

For the primary outcome of determining which SMS text messaging replies require a staff member review (binary outcome of yes/no), a binary classification evaluator was used to calculate accuracy using the area under the receiver operating characteristics curve (AUC), the rate of true and false positives, and the rate of true and false negatives. For the secondary outcome of the accuracy of the ML program to classify the SMS text messaging replies into the 12 categories, a multiclass classification evaluator was used to compare each model for accuracy, precision, recall, and F1 score.

Manual Classification of SMS Text Messaging Replies

A manual review of 3118 SMS text messaging replies was performed. The manual classification identified 12 broad categories of replies (Table 1). Only 22.03% (687/3118) of all SMS text messages required someone to review and reply to the SMS text messages, and only 0.12% (4/3118) SMS text messages were deemed to warrant a timely response within 24-48 h; however no SMS text messages were considered urgent (Table 1).

Machine Learning Model Development

Natural language programming was evaluated as an option to improve the ML model’s ability to understand the sentiment behind each SMS text message. Ultimately, natural language programming was not included in any of the models as challenges arose in relation to the SMS text messaging content, due to the frequent use of nonstandard English grammar, nonstandard abbreviations, and spelling errors within the SMS text messages.

Accuracy for Determining If Staff Review Is Required

The results for each model, with and without the inclusion of heuristics, are outlined in Figure 2 and Multimedia Appendix 2. The inclusion of heuristics reduced the number of false negatives in each of the models, at the expense of an increase in the number of false positives. From the individual models, Naïve Bayes with heuristics produced the lowest false negatives (1.98%) and the highest false positives (14.54%). The Random Forest decision trees model resulted in the lowest number of false positives (1.87%), but had the highest rate of false negatives (13.44%).

The Multilayer Perceptron model correctly identified if staff review was required (yes/no) in 90.52% of cases (4.85% false negatives and 4.63% false positives from the test data sample). With addition of heuristics to the Multilayer Perceptron model, fewer false negatives were identified (3.19%) with a small increase in false positives (7.71%). Utilizing the Multilayer Perceptron model with heuristics, staff would have to review only 26.65% of all SMS text messaging replies (ie, 18.94% true positives + 7.71% false positives; Figure 3).

Figure 2. Machine learning model performance. +: model with heuristics added.
View this figure
Figure 3. Staff workload versus missed messages according to machine learning model. Staff review workload: true positives plus false positives; +: model with heuristics added.
View this figure

However, with the aim of reducing the false negatives to the lowest possible number, the ensemble model was found to be superior (1.43% false negatives and 16.2% false positives), giving a sensitivity of 93.5% for identifying the SMS text messages requiring review, and a specificity of 81.3%. Using the ensemble model with heuristics, the results indicated that health professionals would have to review 36.9% (ie, 20.7% true positives + 16.2% false positives) of all incoming SMS text messaging replies (Figure 3).

Examples of SMS text messages that were incorrectly coded by the Multilayer Perceptron model as “Yes – review required”:

I am now careful with my choices when i purches meats etc bacon is not eaten very often like it used to be
Pill are not a problem for me. To this point limited to no side effects. Just got to keep up with exercise regime.

Examples of SMS text messages that were incorrectly coded by the Multilayer Perceptron model as “No review required”:

Hi everything is excellent. Eating well exercising everyday taking medicine. You could probably take me of list as everything you suggest has already been implemented from day one of recovery. Thanks for your input. Regards
Just a query , I walk for 1/2 each day , is this sufficient?
Accuracy to Identify the Priority SMS Text Messages

Importantly, the model correctly identified the 4 priority SMS text messages that required a timely review. The model missed only 1 of 107 SMS text messages where the participant was requesting to opt out of the program or pause the SMS text messages. The only SMS text message that was missed read “no more texts now thanks.”

Accuracy for Classification Into 12 Categories

For classification into the 12 categories, OneVsRest and the Multilayer Perceptron models produced the best results. OneVsRest had slightly higher accuracy of 0.723, with precision 0.723, recall 0.723, and F1 score 0.719. For the Multilayer Perceptron model, accuracy was 0.717, precision 0.735, recall 0.717, and F1 score 0.723.

Principal Findings

Our study demonstrates the feasibility and accuracy of using an ML program as a triage system to sort which incoming SMS text messaging replies require health professional review, in a cardiovascular secondary prevention SMS text messaging intervention setting. Each of the ML models tested varied in their sensitivity and specificity for classifying the SMS text messages, and there appeared to be a trade-off between false negatives and false positives. From the individual models, Naïve Bayes produced the lowest false-negative rate, and Random Forest decision trees produced the lowest false-positive rate. In our study, the ensemble model (with all models combined) was the most accurate at identifying the SMS text messages needing review, with only 1.43% of the SMS text messages being false negatives. If this ensemble model were utilized as a triage for incoming SMS text messages, health professionals would have to review about 37% of all the incoming SMS text messaging replies. For sorting the SMS text messages into 12 categories (according to the theme of the SMS text message), the accuracy was only moderate (0.723), indicating further work is needed to create suitable categories and to train the model with larger data sets and examples.

To our knowledge this is the first study in a health setting to use an ML program to classify (ie, triage) incoming SMS text messages. ML has been widely validated and used for the purpose of filtering SPAM SMS text messages, classifying SMS text messages into a binary outcome (SPAM or not SPAM) which is similar to our study [28,29]. These studies on SPAM SMS text messages have employed different approaches for reviewing content, filtering, and feature extraction; however, they have produced similar accuracy results to our study [28,29]. The use of ML SPAM filters within the operating systems of smartphones is now commonplace, with early adoption dating back to Apple’s iOS 11 in 2017.

Within the health industry, ML algorithms are common in technology such as electrocardiogram analysis; however, the use of ML to interpret medical text or within patient–provider interactions has not been adopted in everyday practice. There may be larger ethical barriers to overcome in the adoption of ML programs if they are used to assist health care decisions and advice, due to the duty of care to our patients and concern over the consequences if the ML program misses a critical medical alert. Research using ML in the health setting has predominantly focused on extracting information from fields in health and medical records to predict the risk of an outcome such as mortality [30,31], emergency re-admissions [32], and myopia development [33]. It is difficult to compare accuracy across these studies as they measured different outcomes and reported accuracy using different metrics, however they generally performed well, with accuracy examples of AUC > 0.82 for predicting mortality after echocardiography [30]; and AUC range of 0.87-0.98 for predicting myopia development [33]. There is one study which has used ML as a triage for patients with congenital heart disease to estimate the need for a multidisciplinary review [34]. With all these studies, the ML models were trained to review fields in the medical records to determine the likelihood of the outcome, and therefore did not have to interpret the meaning or sentiment of the text in the medical notes [34].

More recently, ML has successfully been applied to more unstructured written health notes and dialogues [35,36]. One study was able to phenotype depression using unstructured notes in the medical records with a sensitivity of 93.5% and specificity of 68% [35]. Natural language processing has also been used to review secure online discussions between patients and health professionals, in which linguistic features were assessed to predict health literacy levels with an accuracy of 60.55% (C-statistic 0.63) [36]. Common to these studies, there was one ML model which performed better than the other models, and the model varied across the studies dependent on the primary data and outcome they were measuring, and this is also reflected in our results.

Clinical Implications

Although ML research in health care is still evolving, the possible breadth of using ML is promising. The most valuable feature of the ML program in our study was the ability to identify the SMS text messages that were needed to be reviewed by a staff member. Therefore, we were interested in identifying the model with the lowest false-negative rate. The AUC was not the best statistic for assessing the model performance, as for each small decrease in false negatives, there was a much larger increase in false positives, and thus a reduction in AUC.

If the ensemble model, with the lowest false negatives, was implemented in a secondary prevention SMS text messaging program, the minimal number of SMS text messages will be missed, but this would be at the expense of staff needing to review more overall SMS text messages (due to the higher false positives). However, even with reviewing additional false positives, staff would only need to review approximately 37% of all incoming SMS text messages, which will significantly reduce the workload of the health professionals running the program, permitting more efficient use of their time. Although it is thought that SMS text messaging interventions may be cost-effective [37], the cost-effectiveness of this solution would need to be evaluated prior to implementation.

Furthermore, prior to implementation it is also important to ensure that the model is capturing all of the most important SMS text messages requiring action from the health professional, and that there is no systemic misclassification. This would need to be tested with larger databases, with careful monitoring of the high-risk categories, to ensure the model is doing what we want it to do at scale. The use of heuristics in the model provides an additional safeguard to ensure that SMS text messages with certain keywords are reviewed. If participants are provided with specific words to opt out of SMS text messaging programs, such as STOP, then heuristics are even more effective at identifying all participants trying to opt out of the program.

Once implemented, the ML program would run alongside the SMS text messaging program, and would continue to be trained as the database grows, thereby improving the accuracy of the ML algorithms. To further improve the certainty of SMS text message classification, it would be possible to program BOT-like features into the model to automatically reply to the sender if clarification of the need to respond is required (eg, “Hi...Would you like me to contact you to discuss this?”). Furthermore, if the accuracy of the ML model could be improved for the classification of SMS text message into more specific categories, it may be possible to consider developing an automated system that sends an appropriate motivational response for the SMS text messages that do not require a staff member response.

Strengths and Limitations

This is the first study in a health setting to explore the use of ML for classification of health-related SMS text messages. It assesses and compares the accuracy of 7 different ML models, and employs a robust method for verification of results using 5 different splits of the training and test data sets. In all triage systems that require a human to decide on the priority and need for medical care, there is a certain element of subjective bias in the decision-making process. In our study where health professionals classified the incoming SMS text messages into whether or not a review was required, an element of subjectivity in that decision remained. This decision making is also complicated by many of the SMS text messages expressing one or more sentiments, and a subjective decision being made as to the most important sentiment to classify the SMS text messages. As there is no gold standard to define whether an SMS text message needs to be reviewed or not, it is possible that health professionals with different areas or levels of experience may manually classify the incoming SMS text messages differently. Our study demonstrates a proof-of-concept, as we tested a limited number of SMS text messaging replies (n=3118); however, to generalize the results beyond our study population, future work is required to validate the model against larger data sets, and on data sets that are coded by different health professionals.


In our feasibility study, the ML program displayed high sensitivity in identifying the SMS text messaging replies that require health professional input. There is a low false-negative rate, indicating few messages which need a response would be missed. Introduction of the ML program to SMS text messaging–based programs could therefore significantly reduce the need for health professionals to review every incoming SMS text message. This could lead to substantial improvements in scalability and capacity of SMS text messaging–based programs. The future implications for this technology are vast, including potential utilization in other interactive mHealth interfaces and cardiovascular health apps.


We gratefully acknowledge Ms Anu Indrawansa for her invaluable assistance in the running of the trials, database management, and coding of the SMS text messaging databases. NL is funded by an NSW Health Early Career Fellowship (H16/52168). JR is funded by an NHMRC Career Development Fellowship (APP1143538). CKC is funded by an NSW Health Clinician scientist grant. The TEXTME study was supported by peer-reviewed grants from the National Heart Foundation of Australia Grant-in-Aid (G10S5110) and a BUPA Foundation Grant. The TEXTMEDS study was funded by a National Health and Medical Research Council (NHMRC) Project grant (ID APP1042290). This work was presented in part as a poster presentation at the European Society of Cardiology Conference, Paris, in August 2019.

Authors' Contributions

All authors contributed to the conception and study design of the work. CKC, JR, and AT contributed to acquisition of the raw data. NL and AD contributed to the analysis and interpretation of data for the work. NL drafted the manuscript. All authors critically revised the manuscript and gave final approval and agree to be accountable for all aspects of work ensuring integrity and accuracy.

Conflicts of Interest

AD is an employee of Sandstone Technology: work on this study is not related to Sandstone Technology and was performed external to his employment. All other authors declare no conflicts of interest.

Multimedia Appendix 1

Original Categorisation used in TEXTME and TEXTMEDS study.

DOCX File , 33 KB

Multimedia Appendix 2

Results for identification of texts requiring review.

DOCX File , 41 KB

  1. World Health Organisation. Cardiovascular Disease Fact Sheet 2017. 2017.   URL: [accessed 2017-04-20]
  2. Clark AM, Hartling L, Vandermeer B, Lissel SL, McAlister FA. Secondary prevention programmes for coronary heart disease: a meta-regression showing the merits of shorter, generalist, primary care-based interventions. Eur J Cardiovasc Prev Rehabil 2007 Aug;14(4):538-546. [CrossRef] [Medline]
  3. Clark AM, Hartling L, Vandermeer B, McAlister FA. Meta-analysis: secondary prevention programs for patients with coronary artery disease. Ann Intern Med 2005 Nov 1;143(9):659-672. [Medline]
  4. Beswick AD, Rees K, Griebsch I, Taylor FC, Burke M, West RR, et al. Provision, uptake and cost of cardiac rehabilitation programmes: improving services to under-represented groups. Health Technol Assess 2004 Oct;8(41):iii-iv, ix [FREE Full text] [CrossRef] [Medline]
  5. Suaya JA, Shepard DS, Normand ST, Ades PA, Prottas J, Stason WB. Use of cardiac rehabilitation by Medicare beneficiaries after myocardial infarction or coronary bypass surgery. Circulation 2007 Oct 9;116(15):1653-1662 [FREE Full text] [CrossRef] [Medline]
  6. Park LG, Beatty A, Stafford Z, Whooley MA. Mobile phone interventions for the secondary prevention of cardiovascular disease. Prog Cardiovasc Dis 2016;58(6):639-650. [CrossRef] [Medline]
  7. Deloitte. 2017 Mobile Consumer Survery - Australia. 2017.   URL: [accessed 2018-04-20]
  8. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015;3(1):e20 [FREE Full text] [CrossRef] [Medline]
  9. Badawy SM, Kuhns LM. Economic evaluation of text-messaging and smartphone-based interventions to improve medication adherence in adolescents with chronic health conditions: a systematic review. JMIR Mhealth Uhealth 2016 Oct 25;4(4):e121 [FREE Full text] [CrossRef] [Medline]
  10. Adler AJ, Martin N, Mariani J, Tajer CD, Owolabi OO, Free C, et al. Mobile phone text messaging to improve medication adherence in secondary prevention of cardiovascular disease. Cochrane Database Syst Rev 2017 Apr 29;4:CD011851. [CrossRef] [Medline]
  11. Thakkar J, Redfern J, Thiagalingam A, Chow CK. Patterns, predictors and effects of texting intervention on physical activity in CHD - insights from the TEXT ME randomized clinical trial. Eur J Prev Cardiol 2016 Nov;23(17):1894-1902. [CrossRef] [Medline]
  12. Chow CK, Redfern J, Hillis GS, Thakkar J, Santo K, Hackett ML, et al. Effect of lifestyle-focused text messaging on risk factor modification in patients with coronary heart disease: a randomized clinical trial. JAMA 2015;314(12):1255-1263. [CrossRef] [Medline]
  13. Thakkar J, Kurup R, Laba T, Santo K, Thiagalingam A, Rodgers A, et al. Mobile telephone text messaging for medication adherence in chronic disease: a meta-analysis. JAMA Intern Med 2016 Mar;176(3):340-349. [CrossRef] [Medline]
  14. Huang H, Li YJ, Chou Y, Hsieh Y, Kuo F, Tsai W, et al. Effects of and satisfaction with short message service reminders for patient medication adherence: a randomized controlled study. BMC Med Inform Decis Mak 2013 Nov 16;13:127 [FREE Full text] [CrossRef] [Medline]
  15. Patrick K, Raab F, Adams MA, Dillon L, Zabinski M, Rock CL, et al. A text message-based intervention for weight loss: randomized controlled trial. J Med Internet Res 2009;11(1):e1 [FREE Full text] [CrossRef] [Medline]
  16. Martin SS, Feldman DI, Blumenthal RS, Jones SR, Post WS, McKibben RA, et al. mActive: a randomized clinical trial of an automated mHealth intervention for physical activity promotion. J Am Heart Assoc 2015 Nov 09;4(11):e002239 [FREE Full text] [CrossRef] [Medline]
  17. Müller AM, Khoo S, Morris T. Text messaging for exercise promotion in older adults from an upper-middle-income country: randomized controlled trial. J Med Internet Res 2016 Jan 07;18(1):e5 [FREE Full text] [CrossRef] [Medline]
  18. Dobson R, Carter K, Cutfield R, Hulme A, Hulme R, McNamara C, et al. Diabetes text-message self-management support program (SMS4BG): a pilot study. JMIR Mhealth Uhealth 2015;3(1):e32 [FREE Full text] [CrossRef] [Medline]
  19. Dobson R, Whittaker R, Bartley H, Connor A, Chen R, Ross M, et al. Development of a culturally tailored text message maternal health program: TextMATCH. JMIR Mhealth Uhealth 2017 Apr 20;5(4):e49 [FREE Full text] [CrossRef] [Medline]
  20. Hyun C, McMenamin J, Ko O, Kim S. Efficacy of a mobile texting app (HepTalk) in encouraging patient participation in viral hepatitis B care: development and cohort study. JMIR Mhealth Uhealth 2020 Apr 01;8(4):e15098 [FREE Full text] [CrossRef] [Medline]
  21. Gram IT, Larbi D, Wangberg SC. Comparing the efficacy of an identical, tailored smoking cessation intervention delivered by mobile text messaging versus email: randomized controlled trial. JMIR Mhealth Uhealth 2019 Sep 27;7(9):e12137 [FREE Full text] [CrossRef] [Medline]
  22. Redfern J, Santo K, Coorey G, Thakkar J, Hackett M, Thiagalingam A, et al. Factors influencing engagement, perceived usefulness and behavioral mechanisms associated with a text message support program. PLoS One 2016;11(10):e0163929 [FREE Full text] [CrossRef] [Medline]
  23. Zack CJ, Senecal C, Kinar Y, Metzger Y, Bar-Sinai Y, Widmer RJ, et al. Leveraging machine learning techniques to forecast patient prognosis after percutaneous coronary intervention. JACC Cardiovasc Interv 2019 Jul 22;12(14):1304-1311. [CrossRef] [Medline]
  24. Al'Aref SJ, Anchouche K, Singh G, Slomka PJ, Kolli KK, Kumar A, et al. Clinical applications of machine learning in cardiovascular disease and its relevance to cardiac imaging. Eur Heart J 2019 Jun 21;40(24):1975-1986. [CrossRef] [Medline]
  25. Lu Y. Automatic topic identification of health-related messages in online health community using text classification. Springerplus 2013;2:309 [FREE Full text] [CrossRef] [Medline]
  26. Chatterjee S, Deng S, Liu J, Shan R, Jiao W. Classifying facts and opinions in Twitter messages: a deep learning-based approach. Journal of Business Analytics 2018 Aug 23;1(1):29-39. [CrossRef]
  27. Chow CK, Thiagalingam A, Santo K, Kok C, Thakkar J, Stepien S, et al. TEXT messages to improve MEDication adherence and Secondary prevention (TEXTMEDS) after acute coronary syndrome: a randomised clinical trial protocol. BMJ Open 2018 Jan 27;8(1):e019463 [FREE Full text] [CrossRef] [Medline]
  28. Waheeb W, Ghazali R. Content-based SMS classification: statistical analysis for the relationship between number of features and classification performance. Comput Syst 2018 Jan 01;21(4):771-785. [CrossRef]
  29. Choudhary N, Jain A. Towards filtering of SMS spam messages using machine learning based technique. In: Singh D, Raman B, Luhach AK, Lingras P, editors. Advanced Informatics for Computing Research. Springer Singapore: Singapore; 2017:18-30.
  30. Samad MD, Ulloa A, Wehner GJ, Jing L, Hartzel D, Good CW, et al. survival from large echocardiography and electronic health record datasets: optimization with machine learning. JACC Cardiovasc Imaging 2019 Apr;12(4):681-689 [FREE Full text] [CrossRef] [Medline]
  31. Steele AJ, Denaxas SC, Shah AD, Hemingway H, Luscombe NM. Machine learning models in electronic health records can outperform conventional survival models for predicting patient mortality in coronary artery disease. PLoS One 2018;13(8):e0202344 [FREE Full text] [CrossRef] [Medline]
  32. Rahimian F, Salimi-Khorshidi G, Payberah AH, Tran J, Ayala Solares R, Raimondi F, et al. Predicting the risk of emergency admission with machine learning: development and validation using linked electronic health records. PLoS Med 2018 Nov;15(11):e1002695 [FREE Full text] [CrossRef] [Medline]
  33. Lin H, Long E, Ding X, Diao H, Chen Z, Liu R, et al. Prediction of myopia development among Chinese school-aged children using refraction data from electronic medical records: a retrospective, multicentre machine learning study. PLoS Med 2018 Nov;15(11):e1002674 [FREE Full text] [CrossRef] [Medline]
  34. Diller G, Kempny A, Babu-Narayan SV, Henrichs M, Brida M, Uebing A, et al. Machine learning algorithms estimating prognosis and guiding therapy in adult congenital heart disease: data from a single tertiary centre including 10 019 patients. Eur Heart J 2019 Apr 01;40(13):1069-1077 [FREE Full text] [CrossRef] [Medline]
  35. Geraci J, Wilansky P, de LV, Roy A, Kennedy JL, Strauss J. Applying deep neural networks to unstructured text notes in electronic medical records for phenotyping youth depression. Evid Based Ment Health 2017 Jul 24;20:83-87 [FREE Full text] [CrossRef] [Medline]
  36. Balyan R, Crossley SA, Brown W, Karter AJ, McNamara DS, Liu JY, et al. Using natural language processing and machine learning to classify health literacy from secure messages: The ECLIPPSE study. PLoS One 2019;14(2):e0212488 [FREE Full text] [CrossRef] [Medline]
  37. Badawy SM, Kuhns LM. Economic evaluation of text-messaging and smartphone-based interventions to improve medication adherence in adolescents with chronic health conditions: a systematic review. JMIR Mhealth Uhealth 2016 Oct 25;4(4):e121 [FREE Full text] [CrossRef] [Medline]

AUC: area under the receiver operating characteristics curve
CVD: cardiovascular disease
ML: machine learning

Edited by G Eysenbach; submitted 07.04.20; peer-reviewed by S Cartledge, A Maiorana, S Badawy; comments to author 05.05.20; revised version received 11.05.20; accepted 13.05.20; published 16.06.20


©Nicole Lowres, Andrew Duckworth, Julie Redfern, Aravinda Thiagalingam, Clara K Chow. Originally published in JMIR mHealth and uHealth (, 16.06.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.