To Text or Not to Text: Electronic Message Intervention to Improve Treatment Adherence Versus Matched Historical Controls

Background Ensuring treatment adherence is important for the internal validity of clinical trials. In intervention studies where touch points decrease over time, there is even more of an adherence challenge. Trials with multiple cohorts offer an opportunity to innovate on ways to increase treatment adherence without compromising the integrity of the study design, and previous cohorts can serve as historical controls. Electronically delivered nudges offer low-cost opportunities to increase treatment adherence. Objective This study aimed to evaluate the effectiveness of electronic messages (e-messages) on treatment adherence to the last cohort of a parent weight loss intervention during the second half of a year-long trial, when intervention checkpoint frequency decreases. Treatment adherence is measured by intervention class attendance and adherence to the intervention diet. Methods All participants in the last cohort (cohort 5, n=128) of a large randomized weight loss study were offered an e-message intervention to improve participant adherence during the last 6 months of a 1-year weight loss program. Overall, 3 to 4 electronic weekly messages asked participants about intervention diet adherence. A propensity score model was estimated using 97 participants who opted to receive e-messages and 31 who declined in cohort 5 and used to pair match cohort 5 e-message participants to a historical control group from cohorts 1 to 4. Moreover, 88 participants had complete data, yielding 176 participants in the final analyses. After matching, intervention and matched control groups were compared on (1) proportion of class attendance between the 6 and 12 month study endpoints, (2) diet adherence, as measured by total carbohydrate grams for low-carbohydrate (LC) and total fat grams for low-fat (LF) diets at 12 months, and (3) weight change from 6 to 12 months. The dose-response relationship between the proportion of text messages responded to and the 3 outcomes was also investigated. Results Compared with matched controls, receiving e-messages had no effect on (1) treatment adherence; class attendance after 6 months +4.6% (95% CI −4.43 to 13.68, P=.31), (2) adherence; LC −2.5 g carbohydrate, 95% CI −29.9 to 24.8, P=.85; LF +6.2 g fat, 95% CI −4.1 to 17.0, P=.26); or on (3) the secondary outcome of weight change in the last 6 months; +0.3 kg (95% CI −1.0 to 1.5, P=.68). There was a positive significant response correlation between the percentage of messages to which participants responded and class attendance (r=.45, P<.001). Conclusions Although this e-message intervention did not improve treatment adherence, future studies can learn from this pilot and may incorporate more variety in the prompts and more interaction to promote more effective user engagement. Uniquely, this study demonstrated the potential for innovating within a multicohort trial using propensity score–matched historical control subjects. Trial Registration ClinicalTrials.gov NCT01826591; https://clinicaltrials.gov/ct2/show/NCT01826591 International Registered Report Identifier (IRRID) RR2-10.1016/j.cct.2016.12.021

INTRODUCTION 1b-iv) RESULTS section in abstract must contain use data Does your paper address subitem 1b-iv?
1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials Does your paper address subitem 1b-v?
You're editing your response. Sharing this URL allows others to also edit your response. "Applying treatment adherence strategies to both treatment arms would extend participant contact and enhance strength of study conclusions. " "This paper highlights the deployment of a treatment adherence enhancement for the 5th cohort of a large 5-cohort, weight loss trial testing the differential effects of adhering to randomly assigned, high-quality, low carb or low fat diets. " 2a-ii) Scientific background, rationale: What is known about the (type of) system You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
2b) In INTRODUCTION: Specific objectives or hypotheses METHODS 3a) Description of trial design (such as parallel, factorial) including allocation ratio Does your paper address subitem 2a-ii? * "Treatment adherence strategies increase participants' enactment of the intervention delivered, thereby increasing the internal validity of the results and accuracy of conclusions drawn from the study [1,2]. Weight-loss diet interventions often show high recidivism, with adherence to the treatment an inherent challenge, and often built into the intervention itself [3][4][5][6]. With adherence being highly variable in diet studies, it becomes difficult to evaluate effectiveness of diet types. Applying treatment adherence strategies to both treatment arms would extend participant contact and enhance strength of study conclusions. " Does your paper address CONSORT subitem 2b? * "Our SMS intervention was designed to increase all participants' adherence to the treatment assigned, in this case a diet plan. Treatment adherence was measured by class attendance and adherence to the diet assigned. Weight loss was a secondary outcome. We used propensity score matching to historical controls in the previous four cohorts." You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 3a? * "This was a longitudinal observational study of those who opted into the Emessage intervention in Cohort 5, with a comparison to their matched historical controls from Cohorts 1-4." Does your paper address CONSORT subitem 3b? * there were no changes to the methods after trial commencement. 3b-i) Bug fixes, Downtimes, Content Changes Does your paper address subitem 3b-i?
You're editing your response. Sharing this URL allows others to also edit your response.

4a) Eligibility criteria for participants
Does your paper address CONSORT subitem 4a? * "The DIETFITS trial is a large randomized weight loss study in which 609 participants were randomly assigned to either a Low-Carb (LC) or Low-Fat (LF) diet, with 16 classes spread throughout months 1-6, and 6 monthly booster classes during month 7-12 period [16]. The trial was split into 5 cohorts over a 3year period, with the first cohort (Cohort 1) beginning in April 2013 and the final measurements in the last cohort (Cohort 5) taken in March 2016. Current Study: Cohort 5 had 128 participants offered the SMS intervention." 4a-i) Computer / Internet literacy Does your paper address subitem 4a-i?
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Does your paper address subitem 4a-ii? * "Attendance was recorded for all 22 assigned classes. Weight was collected at baseline, 3 months, 6 months, and 12 months. Demographics such as age, gender, race, were collected at baseline. E-message responses were collected after the introduction of E-messaging at the 6-month mark.
Dietary intake was assessed by using three unannounced 24-hour dietary recalls at each of the four major data collection time points throughout the yearlong study...Two questions were given at the end of the intervention for participants to evaluate the perceived usefulness of the E-messages and their preferred frequency of E-message receipt." 4a-iii) Information giving during recruitment You're editing your response. Sharing this URL allows others to also edit your response. 4b-i) Report if outcomes were (self-)assessed through online questionnaires You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
5) The interventions for each group with sufficient details to allow replication, including how and when they were actually administered Does your paper address subitem 4b-i? * "All Cohort 5 study participants who opted to receive the E-message intervention received one message comprised of four questions sent three to four times per week on randomly selected days each week. Message frequency was based on Spark et al.'s average contacts, with the goal of having regular, but not overwhelming, contact [15]. If the participant did not respond after 6-8 hours, one reminder was sent. REDCap was used to send E-messages to consenting participants in Cohort 5 [21]." 4b-ii) Report how institutional affiliations are displayed Does your paper address subitem 4b-ii?
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used Does your paper address subitem 5-v? 5-vi) Digital preservation webcitation.org Does your paper address subitem 5-vi?
You're editing your response. Sharing this URL allows others to also edit your response.

5-vii) Access
Does your paper address subitem 5-vii? * "All Cohort 5 study participants who opted to receive the E-message intervention received one message comprised of four questions sent three to four times per week on randomly selected days each week. Message frequency was based on Spark et al.'s average contacts, with the goal of having regular, but not overwhelming, contact [15]. If the participant did not respond after 6-8 hours, one reminder was sent. REDCap was used to send messages to consenting participants in Cohort 5 [21]."

5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
Does your paper address subitem 5-viii? * "The E-message intervention was grounded in the Nudge framework, whereby small touches can elicit significant behavior changes [17]. At a high level, receiving the E-message provides a cue to the treatment, a reminder that the participant is still in active treatment even though study interaction had decreased in frequency. The first question, "how adherent have you been to your eating plan since your last survey?", was based in the cybernetic model of selfcontrol, which suggests that monitoring for discrepancies between goal and current behavior can trigger behavioral corrections to mitigate the gap [18]. The subsequent three questions were based on feedback from the health coaches who taught the diet classes to participants in Cohorts 1-4."

5-ix) Describe use parameters
Does your paper address subitem 5-ix?
You're editing your response. Sharing this URL allows others to also edit your response.

5-x) Clarify the level of human involvement
Does your paper address subitem 5-x? 5-xi) Report any prompts/reminders used Does your paper address subitem 5-xi? * "If the participant did not respond after 6-8 hours, one reminder was sent.
REDCap was used to send E-messages to consenting participants in Cohort 5 [21]." You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed 5-xii) Describe any co-interventions (incl. training/support) Does your paper address subitem 5-xii? * "For more details, consult the separate methods paper for this study [16]. " Does your paper address CONSORT subitem 6a? * "After generating the control group, a two-sample t-test was used to assess the differences between the E-message group and the control group in the two primary study outcomes (attendance and diet adherence) and one secondary outcome (weight loss maintenance). " See above for how and when they were assessed.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed Does your paper address subitem 6a-i? 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored Does your paper address subitem 6a-ii?
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
6b) Any changes to trial outcomes after the trial commenced, with reasons 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Does your paper address subitem 6a-iii?
Does your paper address CONSORT subitem 6b? * There were not changes to the trial outcomes after the trial commenced.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
7b) When applicable, explanation of any interim analyses and stopping guidelines 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Does your paper address subitem 7a-i? Does your paper address CONSORT subitem 7b? * There were no interim analyses. Also no stopping guidelines since the intervention was seen by the study group and the Stanford IRB as exempt and non-invasive.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
8b) Type of randomisation; details of any restriction (such as blocking and block size) 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 8a? * Consented participants were included in the study and compared to propensityscored matched controls from another cohort of the same study.
Does your paper address CONSORT subitem 8b? * Consented participants were included in the study and compared to propensityscored matched controls from another cohort of the same study.
Does your paper address CONSORT subitem 9? * Consented participants were included in the study and compared to propensityscored matched controls from another cohort of the same study.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment Does your paper address CONSORT subitem 10? * Statisticians Joseph Rigdon and Ariadna Garcia were tasked with generating matched controls. The study was headed by Christopher Gardner and the participants were enrolled and randomly assigned to their dietary interventions by Dr. Gardner's study team. The consented participants in one cohort were compared with propensity-score matched controls from another cohort of the same larger study.
11a-i) Specify who was blinded, and who wasn't Does your paper address subitem 11a-i? * Neither participants nor study personnel were blinded to the conditions of this current study (E-message intervention vs. no E-message intervention).
You're editing your response. Sharing this URL allows others to also edit your response.

11b) If relevant, description of the similarity of interventions
(this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Does your paper address subitem 11a-ii?
Does your paper address CONSORT subitem 11b? * this is not relevant for this study.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
Does your paper address CONSORT subitem 12a? * "A propensity score model (PSM) was used to generate a matched control group for the E-message participants in Cohort 5 [22] to provide a relevant comparison for the effectiveness of the E-message intervention on the outcomes specified in the previous section...The PSM was estimated using data from Cohort 5 only as Cohorts 1-4 did not have the opportunity to refuse consent to the message intervention. Next, participants in the Cohort 5 who selected to participate in the E-message intervention were matched 1:1 to individuals in Cohorts 1-4 using the following variables: propensity score (P-score: propensity to participate or not, calculation described above), age, sex, race, proportion of classes attended before 6 months, and weight change before 6 months [23]. " 12a-i) Imputation techniques to deal with attrition / missing values Does your paper address subitem 12a-i? * no imputation techniques were used to deal with missing values.
You're editing your response. Sharing this URL allows others to also edit your response.

12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses X26) REB/IRB Approval and Ethical Considerations [recommended as subheading under "Methods"] (not a CONSORT item)
Does your paper address CONSORT subitem 12b? * "After generating the control group, a two-sample t-test was used to assess the differences between the E-message group and the control group in the two primary study outcomes (attendance and diet adherence) and one secondary outcome (weight loss maintenance)...Scatterplots and Spearmen correlations were used to assess the relationship between proportion of E-messages responded to and the three outcomes described in a previous section: (1) proportion of class attendance after 6 months, (2) diet adherence measured as average total carb grams for LC and average total fat grams for LF at 12 months, (3) 12-month minus 6-month weight change. All statistical analyses were performed using R version 3.3.3 [24]. " X26-i) Comment on ethics committee approval Does your paper address subitem X26-i?
You're editing your response. Sharing this URL allows others to also edit your response.

x26-ii) Outline informed consent procedures
Does your paper address subitem X26-ii?

X26-iii) Safety and security procedures
Does your paper address subitem X26-iii?
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center 13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13a? * "Control group before the matching refers to all of cohorts 1-4 with sufficient data for the match (n=376); control group post-matching was 1:1 nearest neighbor matched to cohort 5 E-message participants with no missing data weight data at 6 months, n=88. " "Because these results indicated that postmatching control and treatment groups were better balanced, the study hypotheses were only tested using post-matching groups. " These postmatching groups were n=88 for the control and n=88 for the treatment.
"Perceived usefulness of the E-message intervention was answered by 58 of the 88 participants. " Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT flow diagram) * Data from all participants in the active group were included. Control group was formed in the following manner. "Control group before the matching refers to all of cohorts 1-4 with sufficient data for the match (n=376); control group postmatching was 1:1 nearest neighbor matched to cohort 5 E-message participants with no missing data weight data at 6 months, n=88. " You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
14a) Dates defining the periods of recruitment and follow-up 13b-i) Attrition diagram Does your paper address subitem 13b-i? Does your paper address CONSORT subitem 14a? * "Our intervention demonstrates dynamic trial-design potential."Attendance was recorded for all 22 assigned classes. Weight was collected at baseline, 3 months, 6 months, and 12 months. Demographics such as age, gender, race, were collected at baseline. E-message responses were collected after the introduction of messaging at the 6-month mark...Dietary intake was assessed by using three unannounced 24-hour dietary recalls at each of the four major data collection time points throughout the yearlong study. " You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
14b) Why the trial ended or was stopped (early) 15) A table showing baseline demographic and clinical characteristics for each group NPT: When applicable, a description of care providers (case volume, qualification, expertise, etc.) and centers (volume) in each group 14a-i) Indicate if critical "secular events" fell into the study period Does your paper address subitem 14a-i? Does your paper address CONSORT subitem 14b? * the trial ended as the entire study and other measurements were also ending at the end of 12 months.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups Does your paper address CONSORT subitem 15? * The sample was intended as a non-clinical, non-diabetic, overweight or obese sample. The description of the participants is referred to in another publication. " "Stanton MV, Robinson JL, Kirkpatrick SM, et al. DIETFITS study (diet intervention examining the factors interacting with treatment success) -Study design and methods. Contemp Clin Trials. 2017;53:151-161. PMID: 28027950;doi:10.1016/j.cct.2016 15-i) Report demographics associated with digital divide issues Does your paper address subitem 15-i? * "Mean age for the treatment group was 39.5, while mean age for the control group was 41.2 and 39.6 pre-and post-matching respectively" " Figure 1 shows balance between treatment and control pre-and post-matching. P-score, age and Hispanic ethnicity were found to be significant pre-matching, but all variables were non-significant post-matching. " You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) 16-i) Report multiple "denominators" and provide definitions Does your paper address subitem 16-i? * Denominators were included in each analysis and can be found in text and in the figures and tables provided.

16-ii) Primary analysis should be intent-to-treat
Does your paper address subitem 16-ii?
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended Does your paper address CONSORT subitem 17a? * The paper includes results for each group and the estimated effect sizes and precisions as in this example -"No significant differences were seen in weight change between participants who received the intervention (M=2.0, SD=4.3) and the control group (M=2.0, SD=3.4); mean difference=0.3, 95% CI (-1.0, 1.5), P=.68 (see Table 2). " 17a-i) Presentation of process outcomes such as metrics of use and intensity of use Does your paper address subitem 17a-i?
You're editing your response. Sharing this URL allows others to also edit your response. 18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory Does your paper address CONSORT subitem 17b? * No binary outcomes were tested.
Does your paper address CONSORT subitem 18? * the paper explains that "We explored weight loss only as a secondary outcome." otherwise, all analyses were primary analyses.

18-i) Subgroup analysis of comparing only users
Does your paper address subitem 18-i?
You're editing your response. Sharing this URL allows others to also edit your response.

19) All important harms or unintended effects in each group
(for specific guidance see CONSORT for harms) Does your paper address CONSORT subitem 19? * In the revised document, we are including a sentence about the study being approved by the IRB.

19-i) Include privacy breaches, technical problems
Does your paper address subitem 19-i?
You're editing your response. Sharing this URL allows others to also edit your response.

DISCUSSION
22) Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group 19-ii) Include qualitative feedback from participants or observations from staff/researchers Does your paper address subitem 19-ii? 22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use) You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
20) Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses Does your paper address subitem 22-i? * "In response to a request from previous participants desiring more accountability and contact with the diet treatment study, a low-cost E-message intervention was offered to the final cohort. Receiving frequent E-message prompts during the second 6-month period of the diet intervention to monitor goal adherence, emotional responses, and behavioral responses to goal discrepancies did not significantly improve class attendance, diet adherence, or weight loss retention compared to matched controls. Within those who did receive the E-messages, there was a positive relationship between overall messages response rate and classes attended during the second 6-month period, however no relationship to diet adherence or weight change was found.
One explanation for this association is that response to the messages is a proxy of program engagement, reflecting action of participants already engaged..." 22-ii) Highlight unanswered new questions, suggest future research Does your paper address subitem 22-ii?
You're editing your response. Sharing this URL allows others to also edit your response.

21) Generalisability (external validity, applicability) of the trial findings
NPT: External validity of the trial findings according to the intervention, comparators, patients, and care providers or centers involved in the trial

20-i) Typical limitations in ehealth trials
Does your paper address subitem 20-i? * "The study had several limitations. While our E-messages elicited a participant response designed to increase the participant's awareness of their diet adherence and internal reaction to it, there was no feedback, tracking of progress, or even automatic reply sent when the participant did respond...It is possible that some of these features from past studies are critical to motivate behavior change...Another limitation is that although propensity score adjustment for treatment selection bias strengthens the causal interpretations of our findings, this adjustment cannot balance across all possible confounders, only the ones we included. A final limitation is low power due to our small sample size. The historically matched controls allowed for us to fully use cohort 5 for the intervention population; however, a larger sample size would have had more power to detect effects, and would have allowed for more stratified subgroup analyses." You're editing your response. Sharing this URL allows others to also edit your response.

21-i) Generalizability to other populations
Does your paper address subitem 21-i? 21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting Does your paper address subitem 21-ii?
You're editing your response. Sharing this URL allows others to also edit your response.