Original Paper
Abstract
Background: There are over 2 million newly diagnosed patients with breast cancer worldwide with more than 10,000 cases in Taiwan each year. During 2017-2018, the National Yang-Ming University, the Taiwan University of Science and Technology, and the Taiwan Breast Cancer Prevention Foundation collaborated to develop a breast cancer self-management support (BCSMS) mHealth app for Taiwanese women with breast cancer.
Objective: The aim of this study was to investigate the quality of life (QoL) of women with breast cancer in Taiwan after using the BCSMS app.
Methods: After receiving a first diagnosis of breast cancer, women with stage 0 to III breast cancer, who were recruited from social networking sites or referred by their oncologists or oncology case managers, were randomized 1:1 into intervention and control groups. Intervention group subjects used the BCSMS app and the control group subjects received usual care. Two questionnaires—the European Organization for Research and Treatment of Cancer (EORTC) Quality-of-Life Questionnaire Core 30 (QLQ-C30) and the EORTC Breast Cancer-Specific Quality-of-Life Questionnaire (QLQ-BR23)—were distributed to subjects in both arms. Paper-based questionnaires were used at baseline; paper-based or Web-based questionnaires were used at 1.5-month and 3-month follow-up evaluations. All evaluations were self-assessed and anonymous, and participants were blinded to their allocation groups. Descriptive analysis, the Pearson chi-square test, analysis of variance, and the generalized estimating equation were used to analyze the data. Missing values, with and without multi-imputation techniques, were used for sensitivity analysis.
Results: A total of 112 women were enrolled and randomly allocated to either the experimental group (n=53) or control group (n=59). The follow-up completion rate was 89.3% (100/112). The demographic data showed homogeneity between the two groups in age (range 50-64 years), breast cancer stage (stage II), marital status (married), working status (employed), and treatment status (receiving treatments). The mean total QoL summary scores from the QLQ-C30 (83.45 vs 82.23, P=.03) and the QLQ-BR23 (65.53 vs 63.13, P=.04) were significantly higher among the experimental group versus the control group, respectively, at 3 months.
Conclusions: This research provides support for using a mobile health care app to promote the QoL among women in Taiwan after a first diagnosis of breast cancer. The BCSMS app could be used to support disease self-management, and further evaluation of whether QoL is sustained is warranted.
Trial Registration: ClinicalTrials.gov NCT004174248; https://clinicaltrials.gov/ct2/show/NCT04174248
doi:10.2196/17084
Keywords
Introduction
Background
Breast cancer has been the most common type of malignant cancer in women in Taiwan for more than 10 years and is the fourth-leading cause of death [
]. Each year, more than 10,000 women are diagnosed with breast cancer, and about 2000 women die from breast cancer: in 2014, there were 12,714 new diagnoses and 2083 deaths. The incidence of breast cancer in Taiwan is common in younger women, 40-65 years of age, with stages 0 to II being the most common at diagnosis [ ]. With early detection and treatment of breast cancer, the treatment effect is good. With appropriate treatment, the 5-year relative survival rate approaches 87.2% and breast cancer is often regarded as a chronic disease. Breast cancer treatment includes breast preservation or resection, chemotherapy, hormone therapy, and targeted therapy [ - ]. The most common side effects differ by treatment method and include surgical-wound pain, nausea, and vomiting (see ). Treatment side effects can cause anxiety and depression, which can affect a woman’s willingness to undergo treatment and can severely decrease the quality of life (QoL) for women with breast cancer [ - ].Today, the self-management model is generally valued and promoted in Taiwan and abroad [
- ]. Self-management is different from traditional disease management; this includes education, emphasizing patient-centered and disease-oriented self-management, and taking the initiative to participate in health care activities. With self-management, the patient learns problem solving, disease control, life adjustment, physical and mental symptom management, and lifestyle changes in order to coexist with a chronic disease in daily life. A systematic review on chronic disease intervention studies showed that self-management can significantly improve knowledge of chronic disease, enhance self-care behavior, and increase self-efficacy [ ]. In a study of chronic disease self-management applied to women with breast cancer, it was found that self-management measures can effectively improve the QoL for women living with breast cancer, increase self-efficacy, and help women to better manage their medical and emotional tasks [ ].The World Health Organization defines QoL as “an individual's perception of their position in life in the context of the culture and value systems in which they live in relation to their goals, expectations, standards and concerns. It is a multifaceted concept affected by the person's physical health, psychological state, personal beliefs, social relationships and their relationship to salient features of their environment” [
]. QoL can be objectively measured. While breast cancer is the most common cancer among Taiwanese women, the survival rate is high. Providing patients with adequate physical and psychological treatment and care could improve their QoL, which becomes an important indicator of the quality of medical care [ - ]. Therefore, the measurement of QoL can objectively reflect the response of women with breast cancer to disease and provide important evaluation indicators for the effectiveness of interventions.The use of mobile apps by patients with cancer is becoming more common. More than half of patients with cancer are willing to transmit information through an app to support their treatment, and 84% of medical professionals, mainly physicians, support the use of apps among this population [
, ]. One study suggests that apps can support outpatient visits, documentation of adverse events, treatment, and medication reminders [ ]. Mobile apps have also been developed for breast cancer self-management [ - ]. However, we are unaware of any mobile self-management apps developed specifically for Taiwanese women with breast cancer. Therefore, our team developed the breast cancer self-management support (BCSMS) mHealth app to address this gap. The development and usability testing of the BCSMS app has been reported previously (publication is forthcoming). The purpose of this study was to evaluate the impact of using the BCSMS app on the QoL of Taiwanese women after an initial diagnosis of breast cancer.Treatment | Common side effects |
Surgery | Surgical-wound pain and lymphedema |
Chemotherapy | Nausea, vomiting, loss of appetite, oral mucositis, fatigue, hair loss, myelosuppression, neuropathy cognitive disorders, weight gain, and poor memory |
Radiotherapy | Dermatitis, sleep disturbance, and fatigue |
Hormone therapy | Menopause, lack of libido, hot flashes, headache, night sweats, and insomnia |
Prior Work
In Taiwan, since June 2017, the National Yang-Ming University, the Taiwan University of Science and Technology, and the Taiwan Breast Cancer Prevention and Research Foundation collaborated to develop the BCSMS app. Eight main features of the app included the following: (1) the evidence or knowledge about breast cancer, (2) exercise and rehabilitation after surgery, (3) diet and nutrition for breast cancer patients, (4) emotional support to prevent anxiety and depression, (5) a personal health record for tracking treatment and side effects, (6) social resource information, (7) experience sharing, and (8) expert consulting. We pilot-tested the BCSMS app with 45 Taiwanese women with breast cancer in 2018 using the modified technology acceptance model of mobile services survey. From this pilot test, our team found that the BCSMS app had sound usability and was accepted by the participants. The BCSMS app has since been outsourced to a technology company for long-term maintenance and can be downloaded for widespread use. To access the app, the keyword “ibreast” should be used to search for the iOS version and the keywords “pink passport” should be used for the Android version.
Methods
Study Design and Setting
The study was a single-blinded, parallel-group, randomized controlled trial with a pretest evaluation (T0), as well as 1.5-month (T1) and 3-month (T2) follow-up evaluations. The study sites included two medical centers— National Taiwan University Hospital and MacKay Memorial Hospital—and one area hospital—Hsinchu Mackay Memorial Hospital—in northern Taiwan. Every patient at each of the study sites received similar care from their health care team (eg, oncology clinic follow-up every 3 weeks during chemotherapy; oncology nurse case managers available to work face-to-face or via telephone with patients when needed on their cancer journey; and transfers by their physicians for consultations with health care professionals in various disciplines, such as pharmacists, physical therapists, nutritionists, psychologists, and social workers, when they had related health problems). The health care professionals’ care models influencing QoL improvement were assumed to be similar in this study.
Ethical Considerations
This study followed the ethical principles of the Declaration of Helsinki [
] and was approved by the Institutional Review Board (IRB) of National Yang Ming University in Taiwan (IRB No. YM107109E). Participation in the study was voluntary. Eligible participants were provided with two different informed consent forms according to their assigned groups. The consent forms included the same information about the protocol regarding their QoL data collection, and every participant was provided US $3 for each pretest evaluation and follow-up evaluation after the study. The information items that differed on the experimental group’s consent form were (1) a brief description about the contents of the BCSMS app and (2) a statement communicating that the BCSMS app would be installed on participants’ mobile phones with remote technology support. The protocol was determined to be of minimal risk to the participants. Nevertheless, all participant data were anonymized and stored on an encrypted, password-protected server.Recruiting the Study Participants, Sample Size, and Randomization
Women with breast cancer who met the following inclusion criteria were recruited: (1) first diagnosis of stage 0 to III breast cancer within the past year, (2) aged 20-65 years, to avoid barriers with respect to aging influencing mobile health usability for older adults [
], (3) had an Android or iOS mobile phone, (4) able to read and write in Chinese, and (5) willing to participate in the study and provide informed consent.A priori power analysis was calculated using G*Power, version 3.1 (Heinrich-Heine-Universität Düsseldorf), by performing F tests and repeated-measures, between-subjects factor analysis of variance; a medium effect size of 0.25, a significance level of .05, and a power of 0.8 were used for sample size calculation according to Cohen [
]. The a priori sample size was 41 in each group. We oversampled our eligible patient population to account for potential dropouts and expected to include a total of 106 patients in this study.Participants were recruited in two different ways. First, a recruitment ad was posted on the Taiwan Cancer Foundation social networking sites (eg, Facebook and Line) and interested participants contacted us through the online registration. Second, patients were referred by their oncologists and oncology case managers from the study settings. To blind the study group participants to allocation [
] and prevent selection bias (eg, technology novelty bias) [ ], the ad stated that participant recruitment was for a QoL evaluation following cancer treatment only, with no mention of the BCSMS app. The recruitment poster was used by recruiters to introduce the patients to the purpose of the study. For homogeneity of patients at each stage of breast cancer—stage 0 to III—between the two groups, the expected numbers of patients with stage 0, stage I, stage II, and stage III breast cancer were about 20, 36, 36, and 14, respectively: the incidence rate of each stage [ ] was used and multiplied by 106. Next, participants at each stage were randomly assigned 1:1 into one of two study groups: control or experimental. The randomization scheme was generated using the website Randomization.com [ ] for participants at each stage.Data Collection Procedures
The study team collected data from January to July 2019. Patients were blinded to their allocation groups—experimental or control—and their pretest data were collected via a paper-based instrument. After the pretest data were collected, the BCSMS app was installed on the mobile phones of the participants in the experimental group and they were taught how to use the app. Each participant in the experimental group could use BCSMS app any time as needed; there were no prompts or reminders from the study team. The entire pretest study design was done via face-to-face contact at the study sites. Data collection for the two follow-up evaluations was completed via a Web-based instrument, together with a phone call, email, and/or communication software (eg, Line). Paper-based instruments were also provided for participants that chose not to use the Web-based instrument. All evaluations were anonymous. During data collection, the content of the BCSMS app was frozen, however, the intervention group was supported in overcoming any technical problems associated with the BCSMS app (eg, log-in problems and data entry problems). During the study, participants in both groups received the same care, concurrently, from health care professionals at the study sites.
Instruments
The QoL instrument consisted of two parts. The first part included demographic items (eg, age, disease stage, marital status, and working status) and treatment-related items. The second part was the Taiwan Chinese version of two Quality-of-Life Questionnaires (QLQs) originally developed by the European Organization for Research and Treatment of Cancer (EORTC) [
, ]: the EORTC QLQ Core 30 (QLQ-C30), version 3, and the EORTC Breast Cancer-Specific QLQ (QLQ-BR23). These instruments have good test-retest reliability, high internal consistency in most scales, and show expected differences between patients in active chemotherapy and those in follow-up groups [ ]. The QLQ-C30 incorporates five functional scales—physical functioning, role functioning, emotional functioning, cognitive functioning, and social functioning—nine symptom scales—fatigue, nausea and vomiting, pain, dyspnea, insomnia, appetite loss, constipation, diarrhea, and financial difficulties—as well as global health status and QoL scales [ ]. The QLQ-BR23 is the module for breast cancer. It incorporates four functional scales—body image, sexual functioning, sexual enjoyment, and future perspective— and four symptom scales—systemic therapy side effects, breast symptoms, arm symptoms, and upset by hair loss [ ]. According to the EORTC QLQ scoring manual, a high score for a functional scale represents a high or healthy level of functioning when using the QLQ-C30 and QLQ-BR23, a high score for global health status or QoL represents a high QoL when using the QLQ-C30, but a high score for a symptom scale or item represents a high level of symptomatology or problems when using the QLQ-C30 and QLQ-BR23 [ ]. In this study, the instrument used was a paper-based or Web-based form. Participants could choose one of the evaluation form types according to their preferences (eg, desire to go paperless or efficiency when self-reporting [ ]) at the two follow-up evaluations. The average time required to complete any of the evaluation forms was approximately 15 minutes, and most patients required no assistance. Following data collection, the internal consistency reliability of both instrument types used by the two groups was assessed: the Cronbach alpha levels at baseline, 1.5-month follow-up, and 3-month follow-up were .90-.96 using the QLQ-C30 and .71-.95 using the QLQ-BR23. The results indicated that both types of forms—paper based and Web based—had adequate reliability [ ].Data Analysis
All completed questionnaires were coded and analyzed using SPSS Statistics for Windows, version 20.0 (IBM Corp). Each item of the QLQ-C30 and QLQ-BR23 underwent linear transformation to obtain a score of 0-100 in accordance with the EORTC QLQ scoring manual [
]. For data values that were missing at random (MAR), the multi-imputation technique was used [ , ] and the average score of the observed data at the same follow-up interval in the same group was adopted to impute the MAR. The dummy variables—summary scores of the QLQ-C30 and QLQ-BR23—were adopted to represent the overall QoL outcome of each respondent and the formula was the sum of scores from each functional item scale and each revised symptom item (eg, the difference between 100 and each symptom item scale score). A higher summary score represents a higher QoL. Frequency and percentage as well as mean and standard deviation were used for descriptive statistics for clinical variables, the QLQ-C30, and the QLQ-BR23. We compared the baseline results of the control and experimental groups using chi-square tests for categorical variables and t tests for continuous variables. All analyses were intention-to-treat using a repeated-measures analysis and the generalized estimating equation (GEE) [ ]. This method was used to account for the lack of adherence values over time and to detect any time × group effects among the target indicators. Significance was defined as a P value less than .05.Results
Randomization and Attrition
Randomization and attrition data were organized according to the Consolidated Standards Of Reporting Trials (CONSORT) guidelines [
] (see ). A total of 112 eligible women, after initial diagnosis of nonmetastatic breast cancer, were enrolled in the study and randomly allocated to the experimental (n=53) or control (n=59) group. Reasons for patient dropout included feeling sick during treatment (n=6), lost to follow-up (n=3), and concern for their personal privacy (n=3). A total of 48 participants remained in the experimental group and 52 remained in the control group at the 3-month follow-up; the follow-up completion rate was 89.3% (100/112). The statistical power of 100 participants was 0.99, which is adequate according to Cohen [ ].At baseline, all participants were assessed using paper-based instruments. At the 1.5-month follow-up evaluation, half of the participants were assessed using Web-based instruments: 48 used paper-based forms versus 55 who used Web-based forms. At the 3-month follow-up evaluation, most participants were assessed using Web-based instruments: 22 used paper-based forms versus 78 who used Web-based forms. The internal consistency reliability of the two groups at each follow-up were adequate; the Cronbach alpha levels are shown in the Instruments section.
Participant Demographics
The demographics and baseline QoL from the two groups were similar. The largest age group was 50-64 years (51/112, 45.5%). Participants had bachelor’s degrees or equivalent levels of education (34/112, 30.4%). Over half (78/112, 69.6%) of the participants were married. One-third of participants had two children (43/112, 38.4%). Close to half (52/112, 46.4%) of the participants were employed. The majority of participants did not have any comorbid diagnoses (98/112, 87.5%). The largest number of participants had stage II breast cancer (44/112, 39.3%) and received breast conservation surgery (76/112, 67.9%), chemotherapy (73/112, 65.2%), radiotherapy (74/112, 66.1%), and hormone therapy (60/112, 53.6%). There were no statistical differences between the control and experimental group participants with regard to any of the baseline characteristics. Detailed statistical results are shown in
.At the 1.5-month and 3-month follow-up evaluations, there were also no statistical differences between control and experimental group participants with respect to receiving chemotherapy (at 1.5 months, P=.40; at 3 months, P=.76), radiotherapy (at 1.5 months, P=.92; at 3 months, P=.31), and hormone therapy (at 1.5 months, P=.13; at 3 months, P=.06), which also showed homogeneity in the participants’ medical treatment statuses at the two follow-up evaluations.
Characteristic and treatment | Experimental group (n=53), n (%) | Control group (n=59), n (%) | All participants (N=112), n (%) | P value | |||||
Age in years | .33 | ||||||||
20-34 | 8 (15) | 4 (7) | 12 (10.7) | ||||||
35-49 | 22 (42) | 27 (46) | 49 (43.8) | ||||||
50-64 | 23 (43) | 28 (47) | 51 (45.5) | ||||||
Education | .05 | ||||||||
Junior high school | 4 (8) | 12 (20) | 16 (14.3) | ||||||
Senior high school | 14 (26) | 14 (24) | 28 (25.0) | ||||||
College | 9 (17) | 13 (22) | 22 (19.6) | ||||||
Bachelor or equivalent | 18 (34) | 16 (27) | 34 (30.4) | ||||||
Master or equivalent | 8 (15) | 4 (7) | 12 (10.7) | ||||||
Marital status | .21 | ||||||||
Unmarried | 13 (25) | 10 (17) | 23 (20.5) | ||||||
Married | 37 (70) | 41 (69) | 78 (69.6) | ||||||
Divorced | 1 (2) | 6 (10) | 7 (6.3) | ||||||
Widowed | 2 (4) | 2 (3) | 4 (3.6) | ||||||
Number of children | .28 | ||||||||
0 | 16 (30) | 15 (25) | 31 (27.7) | ||||||
1 | 7 (13) | 7 (12) | 14 (12.5) | ||||||
2 | 22 (42) | 21 (36) | 43 (38.4) | ||||||
More than 2 | 8 (15) | 16 (27) | 24 (21.4) | ||||||
Work status | .79 | ||||||||
Unemployed | 6 (11) | 8 (14) | 14 (12.5) | ||||||
Housewife | 18 (34) | 20 (34) | 38 (33.9) | ||||||
Retired | 4 (8) | 4 (7) | 8 (7.1) | ||||||
Employed | 25 (47) | 27 (46) | 52 (46.4) | ||||||
Past disease history | .18 | ||||||||
No | 44 (83) | 54 (92) | 98 (87.5) | ||||||
Yes | 9 (17) | 5 (8) | 14 (12.5) | ||||||
Breast cancer stage | .55 | ||||||||
0 | 5 (9) | 4 (7) | 9 (8.0) | ||||||
I | 22 (42) | 20 (34) | 42 (37.5) | ||||||
II | 17 (32) | 27 (46) | 44 (39.3) | ||||||
III | 9 (17) | 8 (14) | 17 (15.2) | ||||||
Surgery | .55 | ||||||||
Conservation therapy | 38 (72) | 38 (64) | 76 (67.9) | ||||||
Mastectomy | 15 (28) | 21 (36) | 36 (32.1) | ||||||
Chemotherapy | .32 | ||||||||
No | 21 (40) | 18 (31) | 39 (34.8) | ||||||
Yes | 32 (60) | 41 (69) | 73 (65.2) | ||||||
Radiotherapy | .11 | ||||||||
No | 14 (26) | 24 (41) | 38 (33.9) | ||||||
Yes | 39 (74) | 35 (59) | 74 (66.1) | ||||||
Hormone therapy | .08 | ||||||||
No | 20 (38) | 32 (54) | 52 (46.4) | ||||||
Yes | 33 (62) | 27 (46) | 60 (53.6) |
Changes in Quality-of-Life Questionnaire Core 30 Indicators
Before instruction (T0), the mean total summary scores of the QLQ-C30 for the experimental and control groups were 74.47 (SD 14.96) and 78.30 (SD 12.59), respectively, with no difference between the groups (P=.14).
At the 1.5-month and 3-month follow-up evaluations, these scores were 79.13 (SD 15.31) and 83.45 (SD 10.85) in the experimental group, respectively, and 79.49 (SD 12.41) and 82.23 (SD 12.07) in the control group, respectively. The mean summary scores for the QLQ-C30 in both groups showed a significant improvement by 3 months: the difference between the T2 and T0 scores was 8.98 in the experimental group and 3.93 in the control group. GEE analysis showed a statistically significant difference (P=.03) in the interaction between groups and time, which meant the experimental group had a greater mean summary score for the QLQ-C30 (difference=5.05) than the control group after the intervention at the 3-month assessment (see
).Step | Functional scalesa, mean | P value | Symptom scalesb, mean | P value | Global health status or quality of lifea, mean | P value | Total summary scoresa, mean | P value | |||||||||||||||
Expc | Cond | Diffe | Exp | Con | Diff | Exp | Con | Diff | Exp | Con | Diff | ||||||||||||
T0f | 73.59 | 77.18 | -3.58 | .25 | 24.71 | 20.99 | 3.73 | .16 | 57.08 | 63.28 | -6.20 | .08 | 74.47 | 78.30 | -3.82 | .14 | |||||||
T1g | 79.64 | 78.63 | 1.01 | .71 | 21.14 | 19.50 | 1.64 | .52 | 68.33 | 70.75 | -2.42 | .51 | 79.13 | 79.49 | -0.37 | .89 | |||||||
T2h | 82.76 | 80.64 | 2.12 | .38 | 16.10 | 16.62 | -0.52 | .79 | 73.44 | 74.36 | -0.92 | .76 | 83.45 | 82.23 | 1.22 | .55 | |||||||
T1-T0 | 6.05 | 1.46 | 4.59 | .06 | -3.58 | -1.49 | -2.09 | .32 | 11.26 | 7.48 | 3.78 | .38 | 4.65 | 1.19 | 3.46 | .09 | |||||||
T2-T0 | 9.17 | 3.47 | 5.71 | .04 | -8.61 | -4.37 | -4.25 | .06 | 16.36 | 11.08 | 5.28 | .18 | 8.98 | 3.93 | 5.04 | .03 |
aHigher scores correspond with better quality of life.
bLower scores correspond with better quality of life.
cExp: experimental group.
dCon: control group.
eDiff: difference between the experimental and control groups.
fT0: baseline or pretest evaluation.
gT1: 1.5-month follow-up evaluation.
hT2: 3-month follow-up evaluation.
Changes in Breast Cancer-Specific Quality-of-Life Questionnaire Indicators
Before instruction (T0), the QLQ-BR23 mean summary scores of the experimental and control groups were 59.68 (SD 12.46) and 61.21 (SD 12.65), respectively, with no difference between the groups (P=.52).
At 1.5-month and 3-month follow-up evaluations, these scores were 62.56 (SD 13.10) and 65.53 (SD 10.29) in the experimental group, respectively, and 62.13 (SD 12.74) and 63.13 (SD 12.66) in the control group, respectively. The mean summary scores for QLQ-BR23 in both groups increased significantly by 3 months: the difference between the T2 and T0 scores was 5.85 in the experimental group and 1.92 in the control group. GEE analysis showed a statistically significant difference (P=.04) in the interaction between groups and time, which meant the experimental group had a greater total summary score for the QLQ-BR23 (difference=3.93) than the control group after the intervention at the 3-month assessment (see
).Step | Functional scalesa, mean | P value | Symptom scalesb, mean | P value | Total summary scoresa, mean | P value | ||||||
Expc | Cond | Diffe | Exp | Con | Diff | Exp | Con | Diff | ||||
T0f | 46.54 | 47.32 | -0.78 | .76 | 27.38 | 25.45 | 1.93 | .51 | 59.68 | 61.20 | -1.53 | .52 |
T1g | 49.39 | 47.06 | 2.32 | .35 | 25.11 | 24.08 | 1.03 | .72 | 62.56 | 62.13 | 0.43 | .86 |
T2h | 34.65 | 32.82 | 1.83 | .22 | 21.69 | 23.46 | -1.78 | .51 | 65.53 | 63.13 | 2.40 | .24 |
T1-T0 | 2.85 | -0.25 | 3.10 | .17 | -2.27 | -1.37 | -0.90 | .71 | 2.88 | 0.93 | 1.95 | .30 |
T2-T0 | -11.89 | -14.49 | 2.61 | .31 | -5.69 | -1.99 | -3.70 | .14 | 5.85 | 1.92 | 3.93 | .04 |
aHigher scores correspond with better quality of life.
bLower scores correspond with better quality of life.
cExp: experimental group.
dCon: control group.
eDiff: difference between the experimental and control groups.
fT0: baseline or pretest evaluation.
gT1: 1.5-month follow-up evaluation.
hT2: 3-month follow-up evaluation.
Sensitivity Analysis
We performed a sensitivity analysis for the attrition cases using GEE analysis. The results showed that the mean summary scores also increased in the QLQ-C30 (difference between experimental and control groups was 5.05, P=.07) and the QLQ-BR23 (difference between experimental and control groups was 3.93, P=.07), with no statistically significant difference between the two groups noted at the 3-month assessment (see
).Step | QLQ-C30 mean summary scores | P value | QLQ-BR23 mean summary scores | P value | |||||
Expa | Conb | Diffc | Exp | Con | Diff | ||||
T0d | 74.47 | 78.30 | -3.83 | .14 | 59.68 | 61.21 | -1.53 | .52 | |
T1e | 79.27 | 79.49 | -0.22 | .89 | 63.04 | 62.13 | 0.91 | .87 | |
T2f | 83.45 | 82.23 | 1.22 | .99 | 65.53 | 63.13 | 2.40 | .40 | |
T1-T0 | 4.80 | 1.19 | 3.61 | .11 | 3.36 | 0.92 | 2.44 | .34 | |
T2-T0 | 8.98 | 3.93 | 5.05 | .07 | 5.85 | 1.92 | 3.93 | .07 |
aExp: experimental group.
bCon: control group.
cDiff: difference between the experimental and control groups.
dT0: baseline or pretest evaluation.
eT1: 1.5-month follow-up evaluation.
fT2: 3-month follow-up evaluation.
Discussion
Principal Findings
Our team developed the BCSMS app with eight main features, in line with the results from our previous study, to support self-management for women after a first diagnosis of breast cancer. After 3 months of repeated QoL evaluation, the principal results showed that for the QLQ-C30, both groups had improvement in their functional scales (9.17 for the experimental group vs 3.47 for the control group), symptom scales (-8.61 for the experimental group vs -4.37 for the control group), global health status (16.36 for the experimental group vs 11.08 for the control group), and total summary scores (8.98 for the experimental group vs 3.93 for the control group). The BCSMS app user group participants had significant improvement in their functional scores (difference=5.71, P=.04) and total summary scores (difference=5.04, P=.03) compared to the control group.
Based on the results of the QLQ-BR23, neither group had improvement in functional scales (-11.98 for the experimental group vs -14.49 for the control group) by the third month. However, improvement was noted in symptom scales (-5.69 for the experimental group vs -1.99 for the control group) and total summary scores (5.85 for the experimental group vs 1.92 for the control group) by the third month. The BCSMS app user group had a significant improvement in total summary scores (difference=3.93, P=.04) when compared to the control group.
After sensitivity analysis (ie, no imputation for attrition cases), the total summary scores for the QLQ-C30 (difference=5.05, P=.07) and the QLQ BR23 (difference=3.93, P=.07) also showed improvement by the third month between the two groups but it was not significant. Based on these findings, further evaluation of QoL among women with a first diagnosis of breast cancer after using the BCSMS app is warranted.
Comparison With Prior Work
The overall QLQ-C30 and QLQ-BR23 scores (see
and ) in both groups in this study were close to those from a multicenter, cross-sectional study in Taiwan [ ]. They were also similar to the QoL scores for women after 1 year following a breast cancer diagnosis in a 10-year, long-term follow-up study in Germany [ ]. This may be due to standardized medical treatments for each stage of breast cancer that were adopted regionally and globally [ - ]. Mobile health apps can effectively support patients with chronic disease self-management [ - ]. Patients with a first diagnosis of breast cancer need knowledge about the disease and to learn about self-management in order to live well with the cancer (eg, surgery-wound pain relief, lymphedema prevention, and controlling body weight). Others have demonstrated that mobile health apps are associated with improved knowledge of the disease and self-management, including self-efficacy with performing shoulder exercises during and after treatment, symptom relief, and QoL during chemotherapy [ - ]. In our study, the BCSMS app also demonstrated improved QoL for Taiwanese women with breast cancer. Among the eight main features included in the BCSMS app, which was developed with the user-centered approach [ ], was a feature that provided evidence or knowledge about breast cancer (eg, cancer stages, treatments, side effects, lymphedema prevention, and relapse) and addressed basic information for the patients. According to the prior study, higher levels of anxiety were experienced by preoperative breast cancer patients who received information delivered via a mobile app than by patients who did not use the app [ ]. In our study, every subject was postoperative; we did not know the BCSMS app would increase patients’ anxiety levels when they had knowledge about breast cancer before operation. Our results showed that the group who used the BCSMS app had higher scores in functional scales—physical functioning, role functioning, emotional functioning, cognitive functioning, and social functioning—than did the group who did not use the app at the 3-month follow-up. This might be because the BCSMS app not only provided the evidence or knowledge about breast cancer but also provided emotional support features to prevent anxiety and depression, including information on mental support, music therapy, mindfulness activities, sleeping well, and acupuncture point massage; social resource information, including information on nonprofit supporting organizations, financial support, and housekeeping; experience sharing, including encouraging information from other senior survivors and health care professionals; and expert consulting, including over 100 frequently asked questions. With this positive and useful information from the BCSMS app, patients were supported in overcoming cancer-related emotional disturbances.In addition, the BCSMS app provided videos that demonstrated effective body movements to prevent lymphedema and low-intensity exercises in the exercise and rehabilitation feature. This feature may have reduced patients’ symptom scale scores, including for fatigue, pain, dyspnea, insomnia, appetite loss, and constipation. Lowering of these scores benefitted symptom control during the first year of cancer treatment (eg, cancer-related fatigue improvement) among experimental subjects and had positive outcomes, similar to those seen in prior studies [
, ].In the personal health record feature for tracking treatments and side effects, subjects in the experimental group were able to record details about their personal medical treatments (eg, date of surgery, period of radiotherapy, medication for chemotherapy, and hormone therapy), record physical-related self-measurements (eg, body temperature and arm circumference), view data graphs, and receive abnormal-data warnings (eg, reminders to patients to consult their psychologists when their emotions were self-assessed as poor through the Brief Symptom Rating Scale) (publication is forthcoming). This feature was also similar to the patient-reported outcomes for symptom monitoring during routine cancer treatment; the results showed that integration of patient-reported outcomes into the routine care of patients with metastatic cancer was associated with increased survival compared with usual care [
]. Having all of the features integrated into one mobile app, as well as the app’s accessibility and ease of use regarding breast cancer self-management, may have contributed to the improvement in the patients’ QoL.Limitations
There were two main limitations of this study. The first was maturation bias, as our study followed participants for only 3 months. Although the control group did not use the BCSMS app, they still had supportive care from health care professionals (eg, physicians and oncology case managers), and perhaps their information-searching competence helped them with self-management and to increase their QoL. The second was that we did not know the actual frequency of use of the BCSMS app among the experimental group. To prevent the Hawthorne effect (ie, causing nonroutine behavior as a result of being observed) [
], the research team did not remind the subjects in the experimental group to use BCSMS app. Such limitations might have influenced the results in this study.Conclusions
The purpose of the study was to investigate the QoL of women with breast cancer in Taiwan after using the BCSMS app. According to the results, women with a first diagnosis of breast cancer appear to have experienced increased QoL after receiving cancer treatment. The BCSMS app provided users with supportive evidence to promote their QoL, compared with those who did not use the app. This might be because the BCSMS app delivered comprehensive information (eg, eight main features developed during our prior work) to women with breast cancer when they were first diagnosed and the app and its features supported their needs during their treatment. Next steps would be to introduce the BCSMS app to more breast cancer patients and to further evaluate whether QoL is sustained among these patients.
Implications
Using objective data, such as users’ log-in frequencies, most-used features, and content analysis from the personal health record for treatment and side effects, could be investigated to better understand how the BCSMS app helps women with breast cancer. In addition, subjective feedback from BCSMS app users is important for future modifications and enhancements of the app. In the future, the BCSMS app could be introduced to breast cancer patients in different countries after culturally sensitive content is added (eg, diet and nutrition for breast cancer patients and social resource information) and when multiple languages can be incorporated into the interface (eg, English).
Acknowledgments
We appreciate the funding from the Ministry of Science and Technology (MOST107 - 2314 - B - 010 - 052) and we acknowledge all the team members who participated in this research as well as the Taiwan Breast Cancer Foundation. We would like to give special thanks to Jhou-Liang Lian, Kong-Hao Chen, and Zih-Bin Chen for their support.
Authors' Contributions
ICH and HYL contributed to the study design and implementation, data analysis, interpretation of the findings, and preparation of the manuscript. SHS contributed to the technology support. KJC, HCT, and AJT contributed to the study implementation. PCD contributed to the editing and revision of the manuscript in English.
Conflicts of Interest
None declared.
This randomized study was only retrospectively registered, explained by authors with "We did not prospectively register this trial due to a lack of awareness or understanding of this requirement on our first Randomized Control Trial study submission". The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials because the risk of bias appears low and the study was considered formative, guiding the development of the application [or other reasons for the exception, as argued by the authors]. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness, as retrospective registration does not prevent authors from changing their outcome measures retrospectively.
CONSORT-EHEALTH checklist (V 1.6.1).
PDF File (Adobe PDF File), 384 KBReferences
- Health Promotion Administration. Cancer Registry Annual Report, 2015. Taipei, Taiwan: Ministry of Health and Welfare; 2017. URL: https://www.hpa.gov.tw/Pages/ashx/File.ashx?FilePath=~/File/Attach/8084/File_7635.pdf [accessed 2019-10-08]
- Health Promotion Administration. Cancer Registry Annual Report, 2016. Taipei, Taiwan: Ministration of Health and Welfare; 2018 Dec. URL: https://www.hpa.gov.tw/Pages/ashx/File.ashx?FilePath=~/File/Attach/10227/File_11644.pdf [accessed 2019-12-30]
- Lin CH. Chemotherapy in breast cancer. Formos J Med 2013;17(4):404-411. [CrossRef]
- Matsen CB, Neumayer LA. Breast cancer: A review for the general surgeon. JAMA Surg 2013 Oct;148(10):971-979. [CrossRef] [Medline]
- Diaby V, Tawk R, Sanogo V, Xiao H, Montero AJ. A review of systematic reviews of the cost-effectiveness of hormone therapy, chemotherapy, and targeted therapy for breast cancer. Breast Cancer Res Treat 2015 May;151(1):27-40 [FREE Full text] [CrossRef] [Medline]
- Lo C, Liao FZ, Chen SL. Symptom experiences of breast cancer patients. J Nurs 2017;64(2):19-27. [CrossRef]
- Browall M, Brandberg Y, Nasic S, Rydberg P, Bergh J, Rydén A, et al. A prospective exploration of symptom burden clusters in women with breast cancer during chemotherapy treatment. Support Care Cancer 2017 May;25(5):1423-1429 [FREE Full text] [CrossRef] [Medline]
- Hermelink K, Bühner M, Sckopke P, Neufeld F, Kaste J, Voigt V, et al. Chemotherapy and post-traumatic stress in the causation of cognitive dysfunction in breast cancer patients. J Natl Cancer Inst 2017 Oct 01;109(10):1-15. [CrossRef] [Medline]
- Wiśniewska I, Jochymek B, Lenart-Lipińska M, Chabowski M. The pharmacological and hormonal therapy of hot flushes in breast cancer survivors. Breast Cancer 2016 Mar;23(2):178-182 [FREE Full text] [CrossRef] [Medline]
- Savard J, Ivers H, Savard M, Morin CM. Cancer treatments and their side effects are associated with aggravation of insomnia: Results of a longitudinal study. Cancer 2015 May 15;121(10):1703-1711 [FREE Full text] [CrossRef] [Medline]
- Jung MS, Cimprich B. Cognitive deficits in Korean women treated with chemotherapy for breast cancer. Cancer Nurs 2014;37(3):E31-E42. [CrossRef]
- Yee C, Wang K, Asthana R, Drost L, Lam H, Lee J, et al. Radiation-induced skin toxicity in breast cancer patients: A systematic review of randomized trials. Clin Breast Cancer 2018 Oct;18(5):e825-e840. [CrossRef] [Medline]
- Barlow J, Wright C, Sheasby J, Turner A, Hainsworth J. Self-management approaches for people with chronic conditions: A review. Patient Educ Couns 2002 Oct;48(2):177-187. [CrossRef]
- McCorkle R, Ercolano E, Lazenby M, Schulman-Green D, Schilling LS, Lorig K, et al. Self-management: Enabling and empowering patients living with cancer as a chronic illness. CA Cancer J Clin 2011;61(1):50-62 [FREE Full text] [CrossRef] [Medline]
- Li MF, Pan LC. Applications on mobile health. J Long Term Care 2012;16(3):237-250.
- Kuo CC, Wang HH. Applying self-management intervention studies to patients with chronic illness: A systematic review. J Nurs Healthc Res 2012;8(1):50-60. [CrossRef]
- Loh SY, Packer T, Chinna K, Quek KF. Effectiveness of a patient self-management programme for breast cancer as a chronic illness: A non-randomised controlled clinical trial. J Cancer Surviv 2013 Sep;7(3):331-342. [CrossRef] [Medline]
- World Health Organization. 2019 Dec. WHOQOL: Measuring quality of life URL: http://www.who.int/healthinfo/survey/whoqol-qualityoflife/en/ [accessed 2019-12-12]
- Tsai H, Kuo NC, Chung KP. Quality of life of breast cancer survivors following breast-conserving therapy versus mastectomy: A multicenter study in Taiwan. Jpn J Clin Oncol 2017 Oct 01;47(10):909-918. [CrossRef] [Medline]
- Fanakidou I, Zyga S, Alikari V, Tsironi M, Stathoulis J, Theofilou P. Mental health, loneliness, and illness perception outcomes in quality of life among young breast cancer patients after mastectomy: The role of breast reconstruction. Qual Life Res 2018 Feb;27(2):539-543. [CrossRef] [Medline]
- Hamer J, McDonald R, Zhang L, Verma S, Leahey A, Ecclestone C, et al. Quality of life (QOL) and symptom burden (SB) in patients with breast cancer. Support Care Cancer 2017 Feb;25(2):409-419. [CrossRef] [Medline]
- Kessel KA, Vogel MM, Kessel C, Bier H, Biedermann T, Friess H, et al. Mobile health in oncology: A patient survey about app-assisted cancer care. JMIR Mhealth Uhealth 2017 Jun 14;5(6):e81 [FREE Full text] [CrossRef] [Medline]
- Kessel KA, Vogel MM, Schmidt-Graf F, Combs SE. Mobile apps in oncology: A survey on health care professionals' attitude toward telemedicine, mHealth, and oncological apps. J Med Internet Res 2016 Nov 24;18(11):e312 [FREE Full text] [CrossRef] [Medline]
- Zhu J, Ebert L, Guo D, Yang S, Han Q, Chan SW. Mobile breast cancer e-support program for Chinese women with breast cancer undergoing chemotherapy (Part 1): Qualitative study of women's perceptions. JMIR Mhealth Uhealth 2018 Apr 11;6(4):e85 [FREE Full text] [CrossRef] [Medline]
- Rincon E, Monteiro-Guerra F, Rivera-Romero O, Dorronzoro-Zubiete E, Sanchez-Bocanegra CL, Gabarron E. Mobile phone apps for quality of life and well-being assessment in breast and prostate cancer patients: Systematic review. JMIR Mhealth Uhealth 2017 Dec 04;5(12):e187 [FREE Full text] [CrossRef] [Medline]
- Giunti G, Giunta D, Guisado-Fernandez E, Bender J, Fernandez-Luque L. A biopsy of breast cancer mobile applications: State of the practice review. Int J Med Inform 2018 Feb;110:1-9 [FREE Full text] [CrossRef] [Medline]
- Lee H, Uhm KE, Cheong IY, Yoo JS, Chung SH, Park YH, et al. Patient satisfaction with mobile health (mHealth) application for exercise intervention in breast cancer survivors. J Med Syst 2018 Nov 06;42(12):254. [CrossRef] [Medline]
- World Medical Association. World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA 2013 Nov 27;310(20):2191-2194. [CrossRef] [Medline]
- Wildenbos GA, Peute L, Jaspers M. Aging barriers influencing mobile health usability for older adults: A literature based framework (MOLD-US). Int J Med Inform 2018 Jun;114:66-75. [CrossRef] [Medline]
- Cohen J. The analysis of variance and covariance. In: Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988:273-406.
- Day SJ, Altman DG. Statistics notes: Blinding in clinical trials and other studies. BMJ 2000;321(7259):504 [FREE Full text] [CrossRef] [Medline]
- Persaud N, Heneghan C, Catalogue of Bias Collaboration. Catalogue of Bias. Novelty bias URL: https://catalogofbias.org/biases/novelty-bias/ [accessed 2019-12-21]
- Randomization.com. URL: http://www.randomization.com [accessed 2018-12-01]
- Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: A quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993 Mar 03;85(5):365-376. [CrossRef] [Medline]
- EORTC Quality of Life. Questionnaires: Breast URL: https://qol.eortc.org/questionnaire/qlq-br23/ [accessed 2019-08-02]
- Chie W, Chang K, Huang C, Kuo W. Quality of life of breast cancer patients in Taiwan: Validation of the Taiwan Chinese version of the EORTC QLQ-C30 and EORTC QLQ-BR23. Psychooncology 2003;12(7):729-735. [CrossRef] [Medline]
- Sprangers MA, Groenvold M, Arraras JI, Franklin J, te Velde A, Muller M, et al. The European Organization for Research and Treatment of Cancer breast cancer-specific quality-of-life questionnaire module: First results from a three-country field study. J Clin Oncol 1996 Oct;14(10):2756-2768. [CrossRef] [Medline]
- Fayers P, Aaronson NK, Bjordal K, Groenvold M, Curran D, Bottomley A. EORTC QLQ-C30 Scoring Manual. 3rd edition. Brussels, Belgium: European Organisation for Research and Treatment of Cancer; 2001.
- Noyes JM, Garland KJ. Computer- vs paper-based tasks: Are they equivalent? Ergonomics 2008 Sep;51(9):1352-1375. [CrossRef] [Medline]
- Henson RK. Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Meas Eval Couns Dev 2019 Aug 29;34(3):177-189. [CrossRef]
- Fitzmaurice GM, Laird NM, Ware JH. Missing data and dropout: Multiple imputation and weighting methods. In: Applied Longitudinal Analysis. 2nd edition. Hoboken, NJ: John Wiley & Sons; 2011:515-550.
- Twisk J, de Vente W. Attrition in longitudinal studies. J Clin Epidemiol 2002 Apr;55(4):329-337 [FREE Full text] [CrossRef]
- Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. BMC Med 2010 Mar 24;8:18 [FREE Full text] [CrossRef] [Medline]
- Cohen J. The concepts of power analysis. In: Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988:1-17.
- Koch L, Jansen L, Herrmann A, Stegmaier C, Holleczek B, Singer S, et al. Quality of life in long-term breast cancer survivors: A 10-year longitudinal population-based study. Acta Oncol 2013 Mar 20;52(6):1119-1128. [CrossRef]
- Farzandipour M, Nabovati E, Sharif R, Arani MH, Anvari S. Patient self-management of asthma using mobile health applications: A systematic review of the functionalities and effects. Appl Clin Inform 2017 Oct;8(4):1068-1081 [FREE Full text] [CrossRef] [Medline]
- Rathbone AL, Prescott J. The use of mobile apps and SMS messaging as physical and mental health interventions: Systematic review. J Med Internet Res 2017 Aug 24;19(8):e295 [FREE Full text] [CrossRef] [Medline]
- Duke DC, Barry S, Wagner DV, Speight J, Choudhary P, Harris MA. Distal technologies and type 1 diabetes management. Lancet Diabetes Endocrinol 2018 Feb;6(2):143-156. [CrossRef] [Medline]
- Zhu J, Ebert L, Liu X, Wei D, Chan SW. Mobile breast cancer e-support program for Chinese women with breast cancer undergoing chemotherapy (Part 2): Multicenter randomized controlled trial. JMIR Mhealth Uhealth 2018 Apr 30;6(4):e104 [FREE Full text] [CrossRef] [Medline]
- Harder H, Holroyd P, Burkinshaw L, Watten P, Zammit C, Harris PR, et al. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv 2017 Dec;11(6):732-742 [FREE Full text] [CrossRef] [Medline]
- Cruz FOAM, Vilela RA, Ferreira EB, Melo NS, Reis PEDD. Evidence on the use of mobile apps during the treatment of breast cancer: Systematic review. JMIR Mhealth Uhealth 2019 Aug 27;7(8):e13245 [FREE Full text] [CrossRef] [Medline]
- Hou IC, Lan MF, Shen SH, Tsai PY, Chang KJ, Tai HC, et al. Developing a breast cancer self-management support mHealth application in Taiwan: A design thinking approach [In press]. JMIR mHealth and uHealth 2020 (forthcoming). [CrossRef]
- Foley N, O'Connell E, Lehane E, Livingstone V, Maher B, Kaimkhani S, et al. PATI: Patient accessed tailored information: A pilot study to evaluate the effect on preoperative breast cancer patients of information delivered via a mobile application. Breast 2016 Dec;30:54-58. [CrossRef] [Medline]
- van Vulpen JK, Peeters PH, Velthuis MJ, van der Wall E, May AM. Effects of physical exercise during adjuvant breast cancer treatment on physical and psychosocial dimensions of cancer-related fatigue: A meta-analysis. Maturitas 2016 Mar;85:104-111. [CrossRef] [Medline]
- van Waart H, Stuiver MM, van Harten WH, Geleijn E, Kieffer JM, Buffart LM, et al. Effect of low-intensity physical activity and moderate- to high-intensity physical exercise during adjuvant chemotherapy on physical fitness, fatigue, and chemotherapy completion rates: Results of the PACES randomized clinical trial. J Clin Oncol 2015 Jun 10;33(17):1918-1927. [CrossRef] [Medline]
- Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017 Jul 11;318(2):197-198 [FREE Full text] [CrossRef] [Medline]
- McCarney R, Warner J, Iliffe S, van Haselen R, Griffin M, Fisher P. The Hawthorne Effect: A randomised, controlled trial. BMC Med Res Methodol 2007 Jul 03;7:30 [FREE Full text] [CrossRef] [Medline]
Abbreviations
BCSMS: breast cancer self-management support |
CONSORT: Consolidated Standards Of Reporting Trials |
EORTC: European Organization for Research and Treatment of Cancer |
GEE: generalized estimating equation |
IRB: Institutional Review Board |
MAR: missing at random |
QLQ: Quality-of-Life Questionnaire |
QLQ-BR23: Breast Cancer-Specific Quality-of-Life Questionnaire |
QLQ-C30: Quality-of-Life Questionnaire Core 30 |
QoL: quality of life |
T0: pretest evaluation |
T1: 1.5-month follow-up evaluation |
T2: 3-month follow-up evaluation |
Edited by G Eysenbach; submitted 16.11.19; peer-reviewed by E Rincon, K Crew, Z Zhou; comments to author 12.12.19; revised version received 25.12.19; accepted 26.01.20; published 04.03.20
Copyright©I-Ching Hou, Hsin-Yi Lin, Shan-Hsiang Shen, King-Jen Chang, Hao-Chih Tai, Ay-Jen Tsai, Patricia C Dykes. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 04.03.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.