This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.
The dose–response relationship between patient engagement and long-term intervention effects in mobile health (mHealth) interventions are understudied. Studies exploring long-term and potentially changing relationships between patient engagement and health outcomes in mHealth interventions are needed.
This study aims to examine dose–response relationships between patient engagement and 3 psychosocial outcomes in an mHealth intervention, Run4Love, using repeated measurements of outcomes at baseline and 3, 6, and 9 months.
This study is a secondary analysis using longitudinal data from the Run4Love trial, a randomized controlled trial with 300 people living with HIV and elevated depressive symptoms to examine the effects of a 3-month mHealth intervention on reducing depressive symptoms and improving quality of life (QOL). We examined the relationships between patient engagement and depressive symptoms, QOL, and perceived stress in the intervention group (N=150) using 4–time-point outcome measurements. Patient engagement was assessed using the completion rate of course assignments and frequency of items completed. Cluster analysis was used to categorize patients into high- and low-engagement groups. Generalized linear mixed effects models were conducted to investigate the dose–response relationships between patient engagement and outcomes.
The cluster analysis identified 2 clusters that were distinctively different from each other. The first cluster comprised 72 participants with good compliance to the intervention, completing an average of 74% (53/72) of intervention items (IQR 0.22). The second cluster comprised 78 participants with low compliance to the intervention, completing an average of 15% (11/72) of intervention items (IQR 0.23). Results of the generalized linear mixed effects models showed that, compared with the low-engagement group, the high-engagement group had a significant reduction in more depressive symptoms (
This study revealed a positive long-term dose–response relationship between patient engagement and 3 psychosocial outcomes among people living with HIV and elevated depressive symptoms in an mHealth intervention over 9 months using 4 time-point repeat measurement data. The high- and low-engagement groups showed significant and widening differences in depressive symptoms, QOL, and perceived stress at the 3-, 6-, and 9-month follow-ups. Future mHealth interventions should improve patient engagement to achieve long-term and sustained intervention effects.
Chinese Clinical Trial Registry ChiCTR-IPR-17012606; https://www.chictr.org.cn/showproj.aspx?proj=21019
Mobile health (mHealth) interventions have gained increasing momentum in delivering easily accessible, patient-centered, individually tailored, and potentially cost-effective programs for a range of psychosocial disorders [
Limited mHealth studies have examined the dose–response relationship in program evaluation. Of the few such studies, the most used is the pre–post design with short-term follow-up, typically within 3 months of the intervention [
Long-term dose–response relationships have been examined more thoroughly in face-to-face interventions than in mHealth interventions. In face-to-face CBT interventions, homework assignments are considered indispensable to the effect of psychotherapy. Homework is defined as structured, specific, and therapeutic activities that are routinely completed by the participants between sessions. Homework tasks might include self-monitoring of mood, thoughts and behaviors, behavioral activation, or specific cognitive and behavioral skills, such as breathing exercises [
Furthermore, patient engagement in mHealth interventions may be different from face-to-face interventions as the former captures more multi-faceted aspects of patient engagement, such as log-ins, completion rate, frequency of items completed, and time spent on the program [
This study aims to examine the potential time-varying dose–response relationships between patient engagement and intervention effects in a randomized controlled trial of an mHealth intervention, the Run4Love program, and to fill gaps in the literature. The Run4Love trial aimed to reduce stress and depressive symptoms and to improve quality of life (QOL) among people who lived with HIV and were concurrently experiencing elevated depressive symptoms. We estimated the impact of patient engagement on depressive symptoms (the primary outcome of the intervention), QOL, and perceived stress at the 3-, 6-, and 9-month follow-ups. We hypothesize that better patient engagement in an mHealth intervention could lead to better and sustained health outcomes in the long term.
This study is a secondary analysis using data from the Run4Love trial, a parallel randomized controlled trial, to examine the effects of a WeChat (Tencent Holdings Limited)-based intervention on reducing depressive symptoms in people living with HIV and elevated depressive symptoms. The study design and primary results of the Run4Love trial have been published elsewhere [
A total of 300 people living with HIV and elevated depressive symptoms were recruited from the outpatient department of a large hospital designated for HIV treatment in Guangzhou, the third largest city in China, in 2017. The participants were recruited if they (1) were aged ≥18 years, (2) were HIV-seropositive, (3) had elevated depressive symptoms (measured using a Center for Epidemiologic Studies-Depression [CES-D] score of ≥16), (4) were active users of WeChat, and (5) were willing to provide hair samples (to measure cortisol as a biomarker of chronic stress). Participants were excluded if they were (1) currently on psychiatric or psychological treatment, (2) unable to finish the questionnaire, and (3) unable to engage in the intervention (read or listen to the materials on WeChat or engage in physical exercise because of medical or other reasons). The participants who met the eligibility criteria and were willing to participate completed a baseline survey and were randomized into the intervention or waitlist control group. A total of 150 participants in the intervention group received a 3-month Run4Love intervention and a 3-month booster session; the participants in the control group received a brochure on HIV-related nutrition in addition to usual care for HIV treatment. We used the data from the 150 participants in the intervention group in the analyses in this study.
The Run4Love intervention consisted of two components: adapted cognitive behavioral stress management (CBSM) courses and physical activity promotions [
This study collected data on individuals’ sociodemographic characteristics, patient engagement, and psychosocial outcomes, including depressive symptoms, QOL, and perceived stress. Psychosocial outcomes were assessed at baseline and 3-, 6-, and 9-month follow-ups, collected by research staff using electronic questionnaires on a tablet. Data on patient engagement were collected automatically using the enhanced WeChat platform. Sociodemographic characteristics included age, gender, marital status, sexual orientation, and educational level.
Patient engagement was assessed through the patient’s completion rate of course assignments and frequency of items completed as these 2 measurements were recommended as reliable measures of patient engagement in mHealth interventions targeting psychosocial outcomes [
Depressive symptoms were measured using the CES-D scale, one of the most widely used self-reported questionnaires on depressive symptoms in China [
QOL was measured using the World Health Organization Quality of Life HIV short version (WHOQOL-HIV BREF), with 31 items assessing 6 domains: physical, psychological, level of independence, social relationships, environment, and beliefs [
Perceived stress was assessed using the 10-item Chinese version of the Perceived Stress Scale (PSS-10) [
First, descriptive statistics of demographic characteristics, baseline depressive symptoms, QOL, and perceived stress were presented. Continuous variables with normal distribution were described using mean and SD, and those with skewed distribution were described using median and IQR. Categorical variables were described using numbers and percentages.
Second, hierarchical clustering was used to classify participants into different groups based on the 2 metrics of patient engagement. Cluster analysis is an exploratory classification technique to group participants into different categories based on their similarities in specific metrics. This statistical method helps identify different engagement groups taking different dimensions into account. Hierarchical clustering iteratively merges smaller clusters into larger clusters. The hierarchical clustering procedure was as follows:
A similarity distance matrix was constructed by calculating the pairwise distance between different observations. Each observation was assigned to an individual cluster; therefore, each observation represented 1 cluster.
The 2 clusters
Clusters
Steps 2 and 3 were repeated until the total number of clusters was 2 [
Through clustering, we categorized participants into high- and low-engagement groups based on both measurements of completion rate and frequency of items completed, which could be used to evaluate the effects of different levels of patient engagement on health outcomes. To verify the results of the cluster analysis, we used Wilcoxon rank-sum tests to compare patient engagement between the high- and low-engagement groups along the 2 metrics. In addition, Wilcoxon rank-sum tests were used to examine whether patient outcomes were balanced between the 2 engagement groups at baseline.
Finally, generalized linear mixed effects models (GLMMs) with fixed effects of the engagement groups and time and including time as a random effect were conducted to estimate the trajectories of patient outcomes of depressive symptoms, QOL, and perceived stress, by the 2 engagement groups over time. The random effects of time represented interindividual varying time trends. This allowed for the estimation of variance in the outcomes within and among these time groups. The GLMM allows for the simultaneous analysis of repeated measures in a longitudinal design, thus providing a more accurate estimation of changes in outcomes over time. It also allows for the inclusion of cases with missing data [
In total, 3 GLMMs were conducted to evaluate the relationships between levels of patient engagement and 3 health outcomes over time, adjusting for time and baseline characteristics. The dependent variables were depressive symptoms, QOL, and perceived stress measured repeatedly at baseline and 3, 6, and 9 months, whereas the independent variables were the patient engagement groups (high- and low-engagement group, the latter as reference) and the 4 time points (baseline and 3, 6, and 9 months, with baseline as reference). Baseline characteristics were included in the GLMMs as control variables, including age, gender, marital status, sexual orientation, and education. Only statistically significant characteristics were retained in the final models. The model estimate of the coefficient for engagement groups represented how the trajectories of health outcomes differed between the 2 groups. For example, in the GLMM with CES-D as the dependent variable, a significant negative coefficient suggested that the high-engagement group had a reduction in more depressive symptoms over 9 months than the low-engagement group. Statistical significance was defined as
The Run4Love trial recruited 300 participants, and the data from the 150 participants in the intervention group were used in this study (
Baseline characteristics and outcomes of the people living with HIV and elevated depressive symptoms in the intervention group (N=150).
Variables | Values | |
Age (years), mean (SD) | 28.0 (5.8) | |
|
||
|
Male | 142 (94.7) |
|
Female | 8 (5.3) |
|
||
|
Heterosexual | 20 (13.3) |
|
Homosexual, bisexual, or uncertain | 130 (86.7) |
|
||
|
High school or lower | 52 (34.7) |
|
More than high school | 98 (65.3) |
|
||
|
Single, divorced, or widowed | 132 (88) |
|
Married | 18 (12) |
CES-Da, mean (SD) | 23.9 (6.4) | |
QOLb, mean (SD) | 77.4 (9.0) | |
PSS-10c, mean (SD) | 20.0 (4.4) |
aCES-D: Center for Epidemiologic Studies-Depression.
bQOL: quality of life.
cPSS-10: Perceived Stress Scale.
The cluster analysis identified 2 clusters that were distinctively different from each other (
Differences in the engagement measurements between the high- and low-engagement groups (N=150).
Engagement metrics | High-engagement group (n=72), median (IQR) | Low-engagement group (n=78), median (IQR) | Wilcoxon rank-sum test | ||
|
|||||
|
3-month intervention | 0.77 (0.23) | 0.17 (0.25) | 3091.0 | <.001 |
|
Booster session | 0.43 (0.57) | 0.00 (0.14) | 3877.5 | <.001 |
|
Total | 0.74 (0.22) | 0.15 (0.23) | 3091.5 | <.001 |
|
|||||
|
3-month intervention | 80 (30.50) | 13 (21.25) | 3084.0 | <.001 |
|
Booster session | 3 (4.75) | 0 (1.00) | 3955.5 | <.001 |
|
Total | 82 (35.50) | 15 (23.25) | 3084.5 | <.001 |
The trajectories of the 3 outcomes at baseline and 3-, 6-, and 9-month follow-ups are shown in
Trajectories of depressive symptoms, quality of life (QOL), and perceived stress over time in high- and low-engagement groups. CES-D: Center for Epidemiologic Studies-Depression; PSS: Perceived Stress Scale.
Effects of patient engagement on intervention outcomes at the 3-, 6-, and 9-month follow-ups: results from generalized linear mixed effects modelsa.
Variables | ||||||
|
||||||
|
Intercept | 19.53 (1.86; 15.87 to 23.18) | <.001 | |||
|
|
|||||
|
|
High vs low | −1.93 (0.72; −3.34 to −0.51) | .008 | ||
|
|
|||||
|
|
3-month follow-up vs baseline | −6.02 (0.94; −7.87 to −4.16) | <.001 | ||
|
|
6-month follow-up vs baseline | −6.11 (0.99; −8.05 to −4.17) | <.001 | ||
|
|
9-month follow-up vs baseline | −5.78 (1.08; −7.91 to −3.65) | <.001 | ||
|
Age (years) | .16 (0.06; 0.03 to 0.28) | .01 | |||
|
|
|||||
|
|
High school or lower vs more than high school | 2.83 (0.75; 1.35 to 4.31) | <.001 | ||
|
||||||
|
Intercept | 78.55 (0.90; 76.77 to 80.33) | <.001 | |||
|
|
|||||
|
|
High vs low | 2.41 (0.93; 0.59 to 4.23) | .01 | ||
|
|
|||||
|
|
3-month follow-up vs baseline | 5.05 (1.22; 2.65 to 7.44) | <.001 | ||
|
|
6-month follow-up vs baseline | 6.01 (1.27; 3.51 to 8.50) | <.001 | ||
|
|
9-month follow-up vs baseline | 5.74 (1.34; 3.10 to 8.38) | <.001 | ||
|
|
|||||
|
|
High school or lower vs more than high school | −6.62 (0.97; −8.53 to −4.71) | <.001 | ||
|
||||||
|
Intercept | 20.28 (0.45; 19.39 to 21.17) | <.001 | |||
|
|
|||||
|
|
High vs low | −1.72 (0.45; −2.61 to −0.82) | <.001 | ||
|
|
|||||
|
|
3-month follow-up vs baseline | −4.25 (0.60; −5.44 to −3.07) | <.001 | ||
|
|
6-month follow-up vs baseline | −3.42 (0.61; −4.63 to −2.22) | <.001 | ||
|
|
9-month follow-up vs baseline | −3.78 (0.65; −5.05 to −2.50) | <.001 | ||
|
|
|||||
|
|
High school or lower vs more than high school | 1.51 (0.48; 0.57 to 2.44) | .002 |
aGeneralized linear mixed effects models included all participants in the intervention group at all time points. Analyses were adjusted for individual characteristics, including age, gender, marital status, sexual orientation, and education at baseline, and those that did not show significant effects were removed from the final models. Educational level and age remained in the final models.
bCES-D: Center for Epidemiologic Studies-Depression.
cQOL: quality of life.
dPSS-10: Perceived Stress Scale.
The average group difference in CES-D scores between the high- and low-engagement groups was 1.93 (
Similar trends were also observed in the other 2 outcome measures. The average group differences in the QOL and perceived stress scores between the high- and low-engagement group were 2.41 (
Covariates including age, gender, marital status, sexual orientation, and educational level at baseline were adjusted in the 3 GLMMs, and only statistically significant variables were retained in the final models. Educational level was significantly associated with the 3 outcomes, and age was only significantly associated with depressive symptoms. Specifically, compared with those with higher education, participants with lower educational levels (high school or lower) had poorer health outcomes (CES-D:
This study was among the first efforts to explore the potential time-varying dose–response relationships between patient engagement and various health outcomes over a span of 9 months using 4–time-point measurement data from the Run4Love mHealth intervention. The main finding was that patient engagement had a positive impact on the health outcomes, including depressive symptoms, QOL, and perceived stress, and such impacts were sustained over 9 months after the baseline. In addition, the dose–response relationship was not only sustained in the long term but also increased over time, as there were widening differences in health outcomes between the high- and low-engagement groups.
Both the high- and low-engagement groups benefited from the intervention, but the high-engagement group benefited more from the intervention consistently as the differences in health outcomes between the 2 groups became more pronounced over time. Such sustained and potentially increasing dose–response relationship in the long term has not been reported in previous mHealth studies. The increased effect sizes in depressive symptoms and QOL at 3, 6, and 9 months and increased effect sizes in perceived stress at 3 and 6 months were small to medium. The effect size of perceived stress at 9 months was medium [
Existing literature suggests that intervention effects tend to decrease or diminish over time after the intervention, with some demonstrating rebound effects [
The reasons for the sustained and widening dose–response relationship in this study are many; evidence-based interventions with rigorous design and good implementation are more likely to have sustained dose–response effects. What is missing in the literature on mHealth interventions is whether the momentum continues over time, for example, in 6- or 9-month follow-ups, and whether engagement level plays a role in this momentum. This study adds new evidence to this gap in the literature. Additional research to understand what factors predict patient engagement is also needed.
Our findings also revealed that education and age were important individual characteristics associated with the effects of the intervention, with participants with lower levels of education and of older age having poorer health outcomes. These findings are consistent with the literature [
Given the sustained positive impacts of patient engagement on health outcomes found in this study, it is critical to improve patients’ intervention adherence and engagement in both the intervention and subsequent booster sessions in mHealth interventions. There are some effective ways to improve patient engagement as suggested in the literature and evidenced in our Run4Love trial. First, the intervention content needs to be culturally tailored and personalized, which requires formative research and a pilot study. The Run4Love intervention was developed based on extensive formative research and a pilot study [
During the intervention, another way to improve patient engagement is to provide timely and personalized feedback. The enhanced WeChat platform automatically sends weekly feedback on the completion status to each participant. In addition to automatic weekly feedback, the Run4Love program also consisted of 5 phone calls made by the research staff at 1 week and 1, 2, 5, and 8 months to address technical challenges and motivate their participation. In addition, the backend platforms of mHealth interventions and wearable devices allow for the collection of passive data on various dimensions of patient engagement, such as physical activities, sleep hours and quality, log-in times, and time and duration of reading, listening to, or watching the intervention items [
There are several limitations to this study. First, the participants in this study were mostly from urban areas, young, and well-educated, and most were nonheterosexual men. Therefore, generalization of the results should be applied with caution. Second, measurement biases may exist in patient engagement. This study only measured patient engagement in CBSM courses but not in physical activity promotions as patient engagement data were not recorded or available in physical activities. However, from the qualitative interviews, we found that most of the engaged participants had a higher engagement in both CBSM courses and physical activities. Therefore, a long-term dose–response relationship between patient engagement in physical activities and intervention outcomes may exist, which needs to be further explored in future research. Moreover, the intervention items were considered as completed when clicked; therefore, we were not able to verify the actual completion or quality of completion. Nevertheless, the patient engagement metrics in this study served as a reliable measurement to assess the dose–response relationship, and the effect of such potential overestimation of patient engagement might only have diluted the observed dose–response relationship [
In conclusion, this study revealed a positive long-term dose–response relationship between patient engagement and 3 psychosocial outcomes in an mHealth intervention using 4–time-point measurement data over 9 months. High- and low-engagement groups showed significant and widening differences in depressive symptoms, QOL, and perceived stress at the 3-, 6-, and 9-month follow-ups in the Run4Love trial. Future mHealth interventions should improve patient engagement to achieve long-term and sustained intervention effects.
CONSORT-eHEALTH checklist (V 1.6.1).
cognitive behavioral stress management
cognitive behavioral therapy
Center for Epidemiologic Studies-Depression
generalized linear mixed effects model
mobile health
Perceived Stress Scale
quality of life
World Health Organization Quality of Life HIV short version
This study was supported by a grant from the National Natural Science Foundation of China (grant 71573290) and the China Medical Board (grant 17-271). The funder had no role in the study design, data collection and analysis, or writing of the manuscript.
YL analyzed the data and drafted the manuscript. YG and YAH contributed to obtaining funding, the study design, and manuscript revision. YZ helped with the study concept. AMW helped with manuscript revision. CZ, MZ, HZ, JQ, and ZX contributed to the clinical trial and data acquisition. WC, LL, and CL provided administrative, technical, and material support for the clinical trial.
None declared.