This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.
Text messaging interventions can be an effective and efficient way to improve health behavioral changes. However, most texting interventions are neither tested nor designed with diverse end users, which could reduce their impact, and there is limited evidence regarding the optimal design methodology of health text messages tailored to low-income, low–health literacy populations and non-English speakers.
This study aims to combine participant feedback, crowdsourced data, and researcher expertise to develop motivational text messages in English and Spanish that will be used in a smartphone app–based texting intervention that seeks to encourage physical activity in low-income minority patients with diabetes diagnoses and depression symptoms.
The design process consisted of 5 phases and was iterative in nature, given that the findings from each step informed the subsequent steps. First, we designed messages to increase physical activity based on the behavior change theory and knowledge from the available evidence. Second, using user-centered design methods, we refined these messages after a card sorting task and semistructured interviews (N=10) and evaluated their likeability during a usability testing phase of the app prototype (N=8). Third, the messages were tested by English- and Spanish-speaking participants on the Amazon Mechanical Turk (MTurk) crowdsourcing platform (N=134). Participants on MTurk were asked to categorize the messages into overarching theoretical categories based on the capability, opportunity, motivation, and behavior framework. Finally, each coauthor rated the messages for their overall quality from 1 to 5. All messages were written at a sixth-grade or lower reading level and culturally adapted and translated into neutral Spanish by bilingual research staff.
A total of 200 messages were iteratively refined according to the feedback from target users gathered through user-centered design methods, crowdsourced results of a categorization test, and an expert review. User feedback was leveraged to discard unappealing messages and edit the thematic aspects of messages that did not resonate well with the target users. Overall, 54 messages were sorted into the correct theoretical categories at least 50% of the time in the MTurk categorization tasks and were rated 3.5 or higher by the research team members. These were included in the final text message bank, resulting in 18 messages per motivational category.
By using an iterative process of expert opinion, feedback from participants that were reflective of our target study population, crowdsourcing, and feedback from the research team, we were able to acquire valuable inputs for the design of motivational text messages developed in English and Spanish with a low literacy level to increase physical activity. We describe the design considerations and lessons learned for the text messaging development process and provide a novel, integrative framework for future developers of health text messaging interventions.
Depression and diabetes are both highly disabling and often comorbid diseases that disproportionately affect patients with a low-income and ethnic minority status. For instance, ethnic and racial minority patients generally show a higher prevalence of these diseases [
Mobile phone SMS, or text messaging, interventions have shown great promise in helping individuals engage in healthy behaviors, including physical activity [
However, there is limited evidence regarding the design process of text messages aimed at promoting behavioral changes [
Moreover, very little has been described about the design process of text messages tailored to low-income, low–health literacy populations and non-English speakers. However, the current consensus is that digital interventions should emphasize usability and engagement with content [
In the context of health text messaging development, a UCD approach consists of conducting preliminary research within the target population to inductively identify any barriers that they face and collect feedback about the content as it is being developed [
Considering the limited literature available and the lack of consensus on best practices for designing a text messaging intervention, particularly for low-income Spanish-speaking populations, we aimed to develop messaging that was both evidence based and responsive to user feedback. Here, we describe the iterative design process of text messages for use within an adaptive smartphone app for low-income ethnic minority patients. We present a novel framework for health text messaging design, the theory-informed UCD framework, which integrates a UCD approach with crowdsourcing and expertise within the study team. In seeking to advance the body of knowledge of mHealth development methods, we provide recommendations and
This text messaging design is part of a larger randomized clinical trial, the
The messages were created in English and Spanish and tested in 5 phases (
Theory-based, user-centered design framework: phases and methods of the message development process. COM-B: capability, opportunity, motivation, and behavior; MTurk: Amazon Mechanical Turk.
This phase occurred between August 2017 and June 2018. To ground our intervention in an understanding of the enablers and barriers to behavior change, we developed a library of concise messages (n=200) based on health behavior change constructs from social cognitive theory and evidence-informed guidelines for mHealth interventions found in the literature (eg,
This phase took place between July and December 2018. It consisted of a feasibility study phase with English-speaking (n=5) and Spanish-speaking (n=5) primary care adult patients recruited from a safety-net health care setting. All participants (10/10, 100%) were diagnosed with depression, and 90% (9/10) had diabetes [
The individual semistructured interviews entailed a set of open-ended questions that ultimately informed the thematic content of messages and messaging characteristics [
This phase took place between January and April 2019. It consisted of a 2-week technology acceptance pilot with English-speaking (n=4) and Spanish-speaking (n=4) patients with comorbid diabetes and depression recruited from the same primary care setting as phase 1. Participants were enrolled on a rolling basis. The pilot entailed testing a subset of messages (n=36), which were updated based on findings from phase 1 (ie, patient feedback from the card sorting activity and interviews). Importantly, these messages were tested within the smartphone app prototype developed by authors AA and CRL and Audacious Software for the DIAMANTE trial [
Interface of the smartphone app and intervention text messages in a test user’s texting interface.
This phase took place between February 2019 and March 2019. A subset of 90 messages (30 per category) composed of the 36 messages tested in phase 2 and 54 chosen at random was surveyed among MTurk participants for motivational construct categorization (
Demographic data of Amazon Mechanical Turk participants (N=134).
Demographic category | Value, n (%) | |
|
||
|
English | 117 (87.3) |
|
Spanish | 17 (12.7) |
|
||
|
United States | 119 (88.8) |
|
Mexico | 4 (3) |
|
South America | 9 (6.7) |
|
Spain | 2 (1.5) |
This phase occurred in April 2019. After discarding messages (n=10) based on poor performance in phase 3, 80 preselected messages were rated for quality by the study team members using a 5-point Likert-type scale ranging from 1 (poor quality) to 5 (high quality). These ratings were independently performed to reduce bias. In addition, each team member was able to see the entire original set of messages (n=200) and all data from phases 1, 2, and 3.
A final bank of messages (n=54) was selected based on team ratings, MTurk scores, and participant feedback. Analysis of the final message selection and final content editing took place from June to August 2019.
We operationalized the constructs of social cognitive theory (ie, self-efficacy and outcome expectancies) [
The messages at this stage (n=200) were in a draft form. We aimed to refine the content after gaining a better understanding of the barriers of and beliefs about physical activity of our target population in the subsequent UCD phases, as we anticipated some discrepancy between participants’ identified beliefs and barriers and those we found in the literature. We also sought to identify any silent diabetes- and depression-related barriers for exercise and how well the individualistic, family, and peer orientations resonated with participants.
Phase 1 consisted of implementing the UCD framework of end user (patient) involvement to obtain qualitative feedback that would allow us to make design decisions. Detailed results of this phase have been published previously [
Feedback on the existing content consisted of liking messages that allowed for self-reflection, provided concrete ideas for engaging in physical activity, and were highly motivating. In general, participants disliked messages that they perceived as repetitive. Participants also suggested that sticking to highly motivating messages and personalized advice would strengthen the content. Overall, out of the 34 messages evaluated in this phase, 5 (15%) were predominantly liked (by 3 or more participants), 2 (6%) were predominantly considered neutral, and only 1 (3%) message was predominantly disliked.
Key findings revealed 3 main health-related barriers: physical limitations, chronic pain, and depression. We found that the primary facilitators of physical activity included being able to clearly visualize or understand the health-related benefits of engaging in regular physical activity, self-motivation and encouragement to change from a sedentary to an active lifestyle, and offering advice on how to engage in regular physical activity.
On the basis of participant feedback from the card sorting activity and qualitative interviews, we identified the COM-B model [
Accordingly, we reframed our library of draft messages to fit into each of the 3 COM-B model constructs [
Sample of final selected messages by theoretical construct category.
COM-Ba construct | Motivational orientation | |
|
Individual | Social |
Capability (benefits of physical activity) |
Physical activity is a great way to feel better, boost your health, and lift your mood. Aim for at least 30 minutes a day of walking at a medium speed. Managing your body pain can be hard. Exercise can help! Going for a walk can improve your mood and clear your mind. It may be painful to start walking, but walking often can help with pain relief. |
When you are active, you have more energy to spend time with your loved ones. Your loved ones will be proud to see the changes in your health over time. You are important to your loved ones. Being more active can help you take care of them. Being active can improve your mood. This can help you spend quality time with your loved ones. |
Opportunity (cues for engaging in physical activity) |
Exercise does not happen only at the gym. Try to find time for walking around your home, neighborhood, or workplace. Try not to sit for more than 30 minutes at a time. You can walk around your home for a few minutes. Too many things going on? Take a quick walk to destress and take care of your health. You do not need fancy things to get active. Comfortable shoes and a water bottle should work! |
Ask your loved ones for support to be more active. Invite a loved one to go on a walk with you. Use this time to catch up! Change your routine. Go for a walk with loved ones after dinner instead of watching TV. Think about walking as a way for you to visit new places or parks with your loved ones. |
Motivation (self-efficacy for physical activity) |
Be proud of yourself for making small changes. It is not always easy. You are a strong person. Use that strength to keep trying to reach your walking goal. Believing in yourself is the first step toward reaching your goal. Do not feel bad if you cannot walk very far. Start slow. You will get better! |
You can inspire others by being active and staying healthy. Other people will start to see the changes you are making for your health. Think about your loved ones when you do not feel like going for a walk. They are cheering for you. Do not worry if you are not walking as much as other people. Everyone has their own speed. |
aCOM-B: capability, opportunity, motivation, and behavior.
In addition, based on the findings of common constraints for physical activity among our target population, we integrated content that touched on physical symptoms, such as pain, and reframed messages to target the improvement of specific depressive symptoms with the purpose of increasing behavioral activation.
In the initial stage of this pilot, where we evaluated a representative subset of 36 messages (12 for each of the 3 theoretical categories) streamed through the smartphone app prototype, we found that patients tended to not carry their phone or not pay attention to the notification of text messages. However, patients generally started paying more attention to their phones and taking their phones with them more often after a few days into the pilot. In addition, although patients did not always remember receiving text messages when called by the research assistant, all patients reported liking the messages. A total of 75% (6/8) of patients reported feeling indifferent about the timing of the messages.
Furthermore, 38% (3/8) of patients reported not liking the messages in which they were compared with others (eg, other people have the same medical conditions and walked more). Messages about exercising with family members or about the benefits of being healthy to support family members were disliked by 25% (2/8) of participants, who reported not having strong family connections. However, all patients reported that they felt like they started walking more and that they wanted to continue to receive messages past their 2-week trial.
Following the completion of the pilot and based on the negative appraisal among some patients toward messages alluding to family member support, we removed any references to “family,” “peers,” or “friends” and replaced them with a blanket “loved ones” phrase. In the same vein, we decided to merge the
Message editing was performed by the first author. The purpose of editing was the overall quality assurance of all existing messages. The first round of edits focused on identifying and correcting grammar, spelling, and punctuation errors as well as simplifying syntax to a lower reading level. All messages were adapted to a reading level of sixth grade or below (average=3.5). A web-based readability tool [
The English text messages were translated into Spanish by the bilingual research staff. The translated messages were also reviewed by a native Spanish speaker fluent in English. In general, all translations were performed following a culturally sensitive communication standard [
In this phase, we submitted a representative sample of messages (n=90, 30 for each theoretical category) to the MTurk participants to test and validate the theoretical categorization. MTurk testing enabled us to ensure that the messages had been tagged in the right motivational categories and, therefore, refine the content and select the best ones. Messages were placed in the right category at an average of 58% (SD 0.21%) of the time. The MTurk respondents also rated messages for their overall quality from 1 to 5. The highest-rated messages were those in the
Amazon Mechanical Turk classification results of messages by theoretical category (N=30).
Theoretical category | Correct classification, mean n (%) |
Benefit | 21 (70) |
Self-belief | 16 (53) |
Opportunity | 15 (50) |
A challenge with using MTurk was to recruit a larger sample of Spanish-speaking workers to test the theoretical categorization of the messages. Otherwise, the task requirements of our categorization tests were easily implemented at the interface of the platform. However, as we found that the qualitative feedback about the message content from the testers was either incomplete or not helpful (ie, not reflective of the needs of our target population), we decided to discontinue this task after the second round of testing.
To assess the content adequacy of the messages, all coauthors independently rated the high-performing messages of phase 3 (n=80) for subjective quality using a scale of 1 to 5. The coauthors gave the messages an average rating of 3.6 (SD 0.59) and reached a quality rating consensus for a given message only 3% (2/80) of the time. A total of 60% (48/80) of the messages received a rating of 3.5 and higher. Quantitative ratings and qualitative feedback were collected from each coauthor and analyzed in aggregate.
Feedback from all 4 phases was combined to select the best messages from the preselected subset from phase 4 (n=80). Messages that compared patients with others with the same medical conditions, which were predominantly disliked by participants in phases 1 and 2, were discarded (4/80, 5%). Of the remaining messages, those that were both sorted into the correct theoretical categories at least 50% of the time during phase 3 and rated at least 3.5 or higher in quality during phase 4 were considered as the clearest and highest impact content to include in the final text message bank. This combined feedback resulted in a final bank of 54 messages, with 18 messages per motivational category.
Using a theory-based UCD framework, which consists of an iterative development process blending UCD methods with crowdsourcing and expert input, we produced a library of 54 motivational messages for a physical activity SMS intervention for low-income minority patients with comorbid diabetes and depression over a 2-year period. The development process comprised 5 phases: feasibility testing, pilot testing, crowdsourcing, expert evaluation, and final selection.
Patient feedback gathered in the UCD phases (1 and 2) was overwhelmingly positive: most patients reported no preference for the timing of message delivery, were receptive to receiving motivational text messages, and reported wanting to continue receiving messages upon conclusion of the pilot testing. However, user feedback is prone to response bias, and it is possible that such positive responses were because of patients not wanting to disappoint the study investigators. Findings from the UCD phases were especially helpful for discarding unappealing messages and editing thematic aspects of messages that did not resonate well with patients. For example, after phase 2, we rephrased any references to peer or familial relations to a broader social denomination (
The strengths of this study include the multistaged and evidence-based development of cogent messaging that is grounded in the available scientific best practices, health behavior change and user engagement theories, and feedback from target users. This process allowed us to design content to match and adapt to the relevant needs of our patient population. As such, we wrote all messages at a sixth-grade or lower reading level, took cultural and linguistic appropriateness into account during editing and translation, and tested for acceptability and clarity among target patients. However, this study has some limitations. For instance, the sample sizes for the UCD phases were small and from one geographic location. However, the purpose of qualitative user research is not to produce findings that can be generalized to other populations but rather to generate data about attitudes or behaviors based on direct observation. Indeed, the UCD phases in our study were used to assess the acceptability and usability of a set of motivational text messages for use in a future physical activity intervention study tailored to a diverse clinical sample of low-income patients with diabetes and depression. A second limitation of our study is that although we did not collect demographic information beyond language and the country of residence, MTurk respondents are not likely to be representative of our target population. Thus, despite the aforementioned benefits of sampling from this crowdsourcing platform, findings from phase 4 were interpreted with caution, only used for theoretical category validation, and not used to inform subsequent content changes. A third limitation is that MTurk respondents did not receive the full set of draft messages (n=200) in the theoretical category validation task, which means that we were not able to collect categorization ratings for every message. However, we had raters assess only a subset of messages (n=90) as a measure to reduce the overall burden of the task and not overwhelm the respondents. We also noted messages with motivational construct ambiguity that were not tested in MTurk during the internal expert review phase and later discarded them from the final selection. A fourth limitation is that the MTurk categorization responses are subject to method bias as a function of low motivation; that is, participants may have not been willing or able to expend the required amount of cognitive effort and thus were less thorough in making accurate response selections. Online recruitment methods such as MTurk have been criticized as prone to yield samples full of low effort respondents [
This paper describes in detail the process for designing and validating the content of an mHealth intervention that utilizes an adaptive learning algorithm to deliver optimized and tailored motivational text messages that promote physical activity. Previous studies have utilized similar methods to inform health text messaging development, such as message drafting based on behavior change theory [
We present a novel, integrative framework for combining UCD, expert input, and crowdsourcing to determine, create, and improve the content of text messages of a smartphone app to increase physical activity in low-income, English- and Spanish-speaking patients with depression and diabetes. This provides a research-based design approach for future developers of health text messaging interventions. This framework can potentially be extended to mHealth programs targeting other vulnerable populations and health behaviors.
Translation key.
capability, opportunity, motivation, and behavior
Diabetes and Mental Health Adaptive Notification Tracking and Evaluation
mobile health
Amazon Mechanical Turk
user-centered design
None declared.