Published on in Vol 9, No 5 (2021): May

Preprints (earlier versions) of this paper are available at, first published .
Developing Messaging Content for a Physical Activity Smartphone App Tailored to Low-Income Patients: User-Centered Design and Crowdsourcing Approach

Developing Messaging Content for a Physical Activity Smartphone App Tailored to Low-Income Patients: User-Centered Design and Crowdsourcing Approach

Developing Messaging Content for a Physical Activity Smartphone App Tailored to Low-Income Patients: User-Centered Design and Crowdsourcing Approach

Original Paper

1School of Social Welfare, University of California, Berkeley, Berkeley, CA, United States

2Center for Vulnerable Populations, University of California, San Francisco, San Francisco, CA, United States

3Department of Psychiatry, University of California, San Francisco, San Francisco, CA, United States

4Department of Computer Science, University of Toronto, Toronto, ON, Canada

5Division of General Internal Medicine, Zuckerberg San Francisco General Hospital, San Francisco, CA, United States

Corresponding Author:

Adrian Aguilera, PhD

School of Social Welfare

University of California, Berkeley

120 Haviland Hall, MC 7400

Berkeley, CA, 94720

United States

Phone: 1 510 642 8564


Background: Text messaging interventions can be an effective and efficient way to improve health behavioral changes. However, most texting interventions are neither tested nor designed with diverse end users, which could reduce their impact, and there is limited evidence regarding the optimal design methodology of health text messages tailored to low-income, low–health literacy populations and non-English speakers.

Objective: This study aims to combine participant feedback, crowdsourced data, and researcher expertise to develop motivational text messages in English and Spanish that will be used in a smartphone app–based texting intervention that seeks to encourage physical activity in low-income minority patients with diabetes diagnoses and depression symptoms.

Methods: The design process consisted of 5 phases and was iterative in nature, given that the findings from each step informed the subsequent steps. First, we designed messages to increase physical activity based on the behavior change theory and knowledge from the available evidence. Second, using user-centered design methods, we refined these messages after a card sorting task and semistructured interviews (N=10) and evaluated their likeability during a usability testing phase of the app prototype (N=8). Third, the messages were tested by English- and Spanish-speaking participants on the Amazon Mechanical Turk (MTurk) crowdsourcing platform (N=134). Participants on MTurk were asked to categorize the messages into overarching theoretical categories based on the capability, opportunity, motivation, and behavior framework. Finally, each coauthor rated the messages for their overall quality from 1 to 5. All messages were written at a sixth-grade or lower reading level and culturally adapted and translated into neutral Spanish by bilingual research staff.

Results: A total of 200 messages were iteratively refined according to the feedback from target users gathered through user-centered design methods, crowdsourced results of a categorization test, and an expert review. User feedback was leveraged to discard unappealing messages and edit the thematic aspects of messages that did not resonate well with the target users. Overall, 54 messages were sorted into the correct theoretical categories at least 50% of the time in the MTurk categorization tasks and were rated 3.5 or higher by the research team members. These were included in the final text message bank, resulting in 18 messages per motivational category.

Conclusions: By using an iterative process of expert opinion, feedback from participants that were reflective of our target study population, crowdsourcing, and feedback from the research team, we were able to acquire valuable inputs for the design of motivational text messages developed in English and Spanish with a low literacy level to increase physical activity. We describe the design considerations and lessons learned for the text messaging development process and provide a novel, integrative framework for future developers of health text messaging interventions.

JMIR Mhealth Uhealth 2021;9(5):e21177




Depression and diabetes are both highly disabling and often comorbid diseases that disproportionately affect patients with a low-income and ethnic minority status. For instance, ethnic and racial minority patients generally show a higher prevalence of these diseases [1], lower treatment rates [2,3], and worse outcomes [4,5]. There is a need for the design of more effective self-management interventions that can target both these diseases and are affordable, acceptable, and tailored to vulnerable populations [6]. Promoting physical activity is a potentially effective strategy that has positive effects on both mental health, with a particularly strong effect on depression [7,8], and common chronic diseases such as diabetes [9].

Mobile phone SMS, or text messaging, interventions have shown great promise in helping individuals engage in healthy behaviors, including physical activity [10,11] and diabetes self-management [12-14]. Most individuals currently own a mobile phone. Smartphone usage across a wide range of demographic groups is high (around 81%) and is on the rise. For instance, the percentage of Black individuals (80%) and Latino individuals (79%) that own smartphones is now similar to that of White individuals (82%) [15]. Therefore, feedback and motivational text messages might be effective in helping individuals with depression and diabetes increase their levels of physical activity and improve their overall health.

However, there is limited evidence regarding the design process of text messages aimed at promoting behavioral changes [16]. Detailed accounts of complete content development processes, including formative research and pretesting methods, are underreported in the mobile health (mHealth) literature [17]. Among the published health text messaging studies that describe their content design processes, there is great variability in the methods utilized. Researchers have reported the development of messaging based on public health guidelines [18-21], health education curricula [14,22], theoretical models [16,18,20,21,23], findings from quantitative surveys or focus groups [16,18,19,21,23,24], evaluation by members of the research team or outside experts [16,18,20,22], and pilot testing within a subset of the target population [19,20,25]. Thus, this study aims to integrate various methods such as participant feedback, pilot testing, crowdsourcing, and expert knowledge.

Moreover, very little has been described about the design process of text messages tailored to low-income, low–health literacy populations and non-English speakers. However, the current consensus is that digital interventions should emphasize usability and engagement with content [26] and should be developed with users who are intended to benefit from the intervention [27]. This perspective is fundamental to user-centered design (UCD), a design approach derived from the multidisciplinary field of study of human-computer interaction. UCD entails the active participation of end users in product development to enhance the understanding of user and task requirements as well as the iteration of design and evaluation [28].

In the context of health text messaging development, a UCD approach consists of conducting preliminary research within the target population to inductively identify any barriers that they face and collect feedback about the content as it is being developed [21,29,30]. However, recruiting large groups of participants within patient populations to test the content is challenging. Online crowdsourcing platforms, such as Amazon’s Mechanical Turk (MTurk), are increasingly being used as a means of acquiring feedback from a relatively large pool of participants. However, relying solely on MTurk respondents might not be entirely feasible as these participants are not always reflective of typical patient populations. Therefore, specifically for research on the design of text messaging in underserved populations, an argument can be made to combine crowdsourcing with participant feedback.


Considering the limited literature available and the lack of consensus on best practices for designing a text messaging intervention, particularly for low-income Spanish-speaking populations, we aimed to develop messaging that was both evidence based and responsive to user feedback. Here, we describe the iterative design process of text messages for use within an adaptive smartphone app for low-income ethnic minority patients. We present a novel framework for health text messaging design, the theory-informed UCD framework, which integrates a UCD approach with crowdsourcing and expertise within the study team. In seeking to advance the body of knowledge of mHealth development methods, we provide recommendations and lessons learned from designing text messages in this population.


This text messaging design is part of a larger randomized clinical trial, the Diabetes and Mental Health Adaptive Notification Tracking and Evaluation (DIAMANTE) trial (NCT 03490253) [31]. This trial examines the effectiveness of a smartphone app that uses a reinforcement learning algorithm to predict the categories of text messages that are most effective in increasing a participant’s physical activity, given the participant’s contextual variables, such as previous physical activity, as well as demographic and clinical variables (eg, age, gender, and depression scores) [32]. Although we plan to report the effectiveness results of the trial in future studies, this paper explains the development process of the different categories of motivational text messages for increasing physical activity targeted to individuals with comorbid diabetes and depression. The text messages were developed in 5 phases that incorporated crowdsourced feedback, study team feedback, and UCD techniques (ie, card sorting, interviews, and surveys). The process was iterative in nature as the findings from each step informed the subsequent steps.

Message Development Phases

The messages were created in English and Spanish and tested in 5 phases (Figure 1).

Figure 1. Theory-based, user-centered design framework: phases and methods of the message development process. COM-B: capability, opportunity, motivation, and behavior; MTurk: Amazon Mechanical Turk.
View this figure
Phase 0: Message Drafting

This phase occurred between August 2017 and June 2018. To ground our intervention in an understanding of the enablers and barriers to behavior change, we developed a library of concise messages (n=200) based on health behavior change constructs from social cognitive theory and evidence-informed guidelines for mHealth interventions found in the literature (eg, gain and loss framing, proximal outcomes framing) [33-37].

Phase 1: Feasibility Testing

This phase took place between July and December 2018. It consisted of a feasibility study phase with English-speaking (n=5) and Spanish-speaking (n=5) primary care adult patients recruited from a safety-net health care setting. All participants (10/10, 100%) were diagnosed with depression, and 90% (9/10) had diabetes [30,38]. A subset of messages (n=34) was tested using a mixed methods approach that included a card sorting task and individual semistructured interviews. For the card sorting activity, participants were asked to sort text messages in either English or Spanish (depending on their preferred language) into 3 piles: liked messages, disliked messages, and messages that were neither disliked nor liked (neutral). Participants also provided the reason behind the placement of the card in a particular pile. Although card sorting has been previously used to validate expert-developed health text messages among a group of nonexperts (ie, university students and staff) [39], the use of this method in our study entailed modification from the traditional card sorting protocol to adapt it to our population of low-income patients [40]. For example, the subset of messages represented in the cards was written at a sixth-grade reading level. In addition, interviewers modified the administration of the activity to accommodate participants with limited literacy or communication barriers by providing audiovisual cues, including reading the cards aloud and successively probing for feedback after reading each card [40].

The individual semistructured interviews entailed a set of open-ended questions that ultimately informed the thematic content of messages and messaging characteristics [30]. Participants were also asked about their perceived barriers and facilitators of regular physical activity. Upon conclusion of this feasibility phase, all messages (n=200) were further refined to incorporate participant feedback and a cognitive framework for behavior change—the capability, opportunity, motivation, and behavior (COM-B) model—and edited and translated by members of the research team.

Phase 2: Pilot Testing

This phase took place between January and April 2019. It consisted of a 2-week technology acceptance pilot with English-speaking (n=4) and Spanish-speaking (n=4) patients with comorbid diabetes and depression recruited from the same primary care setting as phase 1. Participants were enrolled on a rolling basis. The pilot entailed testing a subset of messages (n=36), which were updated based on findings from phase 1 (ie, patient feedback from the card sorting activity and interviews). Importantly, these messages were tested within the smartphone app prototype developed by authors AA and CRL and Audacious Software for the DIAMANTE trial [32] (Figure 2). A total of 12 messages from each theoretical category of the full message bank were selected for testing to ensure that the subset (n=36) was representative of our theoretical construct categorization. Moreover, because we wanted to maintain some variability in messaging to evaluate the impact of the adaptive learning algorithm, any messages that received negative feedback in the previous phase were still tested in this phase. The study team members monitored the messages received by the participants. In addition, participants were called by a team member every weekday during business hours at a selected time chosen by the participant to ask them about their experiences with the app and to provide qualitative feedback on the messages. Individualized implementation methods have been recommended for enhancing participant buy-in in maintaining commitment to the intervention once enrolled and minimizing dropout rates in mHealth studies [41,42]. Therefore, in addition to eliciting regular and reliable feedback from participants on the messaging content, we used a daily phone call strategy to enhance engagement with the app and minimize dropouts during pilot testing, particularly among Spanish-speaking participants and those with limited health and digital literacy.

Figure 2. Interface of the smartphone app and intervention text messages in a test user’s texting interface.
View this figure
Phase 3: Crowdsourcing

This phase took place between February 2019 and March 2019. A subset of 90 messages (30 per category) composed of the 36 messages tested in phase 2 and 54 chosen at random was surveyed among MTurk participants for motivational construct categorization (Table 1). Although a handful of messages (n=4) tested in phases 1 and 2 received predominantly negative participant feedback, we still included these messages in the MTurk sample for variability to assess the impact of the algorithm. As educational attainment and income are correlated [43], we selected participants with high school education or less to inform an intervention targeting low-income users. MTurk participants were asked to read different messages and determine their category (among the 3 overarching theoretical categories: benefit, self-efficacy, and opportunity cues) to the best of their abilities, based on the examples and descriptions provided. Categorization questions were set up as multiple-choice questions with the 3 categories as possible options. The order of the options was not randomized, but the sequence of the questions was randomized. Correct rates (scores) of the English and Spanish categorization questions were calculated for each participant.

Table 1. Demographic data of Amazon Mechanical Turk participants (N=134).
Demographic categoryValue, n (%)

English117 (87.3)

Spanish17 (12.7)

United States119 (88.8)

Mexico4 (3)

South America9 (6.7)

Spain2 (1.5)
Phase 4: Expert Evaluation

This phase occurred in April 2019. After discarding messages (n=10) based on poor performance in phase 3, 80 preselected messages were rated for quality by the study team members using a 5-point Likert-type scale ranging from 1 (poor quality) to 5 (high quality). These ratings were independently performed to reduce bias. In addition, each team member was able to see the entire original set of messages (n=200) and all data from phases 1, 2, and 3.

Phase 5: Final Selection

A final bank of messages (n=54) was selected based on team ratings, MTurk scores, and participant feedback. Analysis of the final message selection and final content editing took place from June to August 2019.

Phase 0: Message Drafting

We operationalized the constructs of social cognitive theory (ie, self-efficacy and outcome expectancies) [33] into practical cues and tips for physical activity in a subset of messages. In addition, given past research that shows that exposure to proximal positive exercise outcomes increases intrinsic motivation among individuals who report lower levels of past physical activity [34], we framed most messages in terms of proximal outcomes and underscored the positive health-related feelings associated with exercising [35]. Furthermore, we wrote half of the messages with a gain framing, emphasizing the benefits of physical activity, and half with a loss framing, emphasizing what is avoided by doing physical activity [36]. Health communication research has shown that gain and loss framing has the capacity to both increase engagement with message content and change behavior [37,38]. Finally, we wrote messages that followed individualistic, family, and peer orientations to cue participants to increase their current physical activity levels as well as messages that addressed environmental constraints for physical activity [44].

The messages at this stage (n=200) were in a draft form. We aimed to refine the content after gaining a better understanding of the barriers of and beliefs about physical activity of our target population in the subsequent UCD phases, as we anticipated some discrepancy between participants’ identified beliefs and barriers and those we found in the literature. We also sought to identify any silent diabetes- and depression-related barriers for exercise and how well the individualistic, family, and peer orientations resonated with participants.

Phase 1: Feasibility Testing

Phase 1 consisted of implementing the UCD framework of end user (patient) involvement to obtain qualitative feedback that would allow us to make design decisions. Detailed results of this phase have been published previously [30,40].

Card Sorting

Feedback on the existing content consisted of liking messages that allowed for self-reflection, provided concrete ideas for engaging in physical activity, and were highly motivating. In general, participants disliked messages that they perceived as repetitive. Participants also suggested that sticking to highly motivating messages and personalized advice would strengthen the content. Overall, out of the 34 messages evaluated in this phase, 5 (15%) were predominantly liked (by 3 or more participants), 2 (6%) were predominantly considered neutral, and only 1 (3%) message was predominantly disliked.

Qualitative Interviews

Key findings revealed 3 main health-related barriers: physical limitations, chronic pain, and depression. We found that the primary facilitators of physical activity included being able to clearly visualize or understand the health-related benefits of engaging in regular physical activity, self-motivation and encouragement to change from a sedentary to an active lifestyle, and offering advice on how to engage in regular physical activity.

Theoretical Inclusion

On the basis of participant feedback from the card sorting activity and qualitative interviews, we identified the COM-B model [41] as the optimal theoretical framework for message categorization and conceptualization. According to the COM-B model, behavior change is part of an interacting system involving the key ingredients of capability, opportunity, and motivation [45]. This model lays out a mechanism for the development of effective behavioral health interventions: changing one or more of these components to rearrange the system and minimize the risk of default [45]. Although the COM-B model draws from social cognitive theory, which was the original chosen theoretical base for the messages, we incorporated the COM-B model because it allowed for more optimal linking with our intervention design, as it was a better fit for the opportunity cues to be organized within our message development process.

Accordingly, we reframed our library of draft messages to fit into each of the 3 COM-B model constructs [30]. In targeting the capability construct, which is influenced by internal processes that direct behavior, we reconstructed the messages that provided information on the determined benefits of physical activity to highlight physical and social outcome expectations as motivators for physical activity (72/200, 36%). The opportunity construct explains the social or environmental factors that allow change to occur. Therefore, we decided to redefine existing messages that addressed environmental constraints for physical activity to emphasize cues for exercise opportunities or suggestions on how to reduce or circumvent these constraints (eg, restructuring routines and planning for social support; 73/200, 36.5%). Finally, for the motivation construct, which is demonstrated by the skills and knowledge necessary for change, we used existing messages that targeted self-efficacy (55/200, 27.5%) through tips for building self-management skills and self-beliefs of motivation and confidence to exercise (Table 2).

Table 2. Sample of final selected messages by theoretical construct category.
COM-Ba constructMotivational orientation

Capability (benefits of physical activity)
  • Physical activity is a great way to feel better, boost your health, and lift your mood. Aim for at least 30 minutes a day of walking at a medium speed.
  • Managing your body pain can be hard. Exercise can help!
  • Going for a walk can improve your mood and clear your mind.
  • It may be painful to start walking, but walking often can help with pain relief.
  • When you are active, you have more energy to spend time with your loved ones.
  • Your loved ones will be proud to see the changes in your health over time.
  • You are important to your loved ones. Being more active can help you take care of them.
  • Being active can improve your mood. This can help you spend quality time with your loved ones.
Opportunity (cues for engaging in physical activity)
  • Exercise does not happen only at the gym. Try to find time for walking around your home, neighborhood, or workplace.
  • Try not to sit for more than 30 minutes at a time. You can walk around your home for a few minutes.
  • Too many things going on? Take a quick walk to destress and take care of your health.
  • You do not need fancy things to get active. Comfortable shoes and a water bottle should work!
  • Ask your loved ones for support to be more active.
  • Invite a loved one to go on a walk with you. Use this time to catch up!
  • Change your routine. Go for a walk with loved ones after dinner instead of watching TV.
  • Think about walking as a way for you to visit new places or parks with your loved ones.
Motivation (self-efficacy for physical activity)
  • Be proud of yourself for making small changes. It is not always easy.
  • You are a strong person. Use that strength to keep trying to reach your walking goal.
  • Believing in yourself is the first step toward reaching your goal.
  • Do not feel bad if you cannot walk very far. Start slow. You will get better!
  • You can inspire others by being active and staying healthy.
  • Other people will start to see the changes you are making for your health.
  • Think about your loved ones when you do not feel like going for a walk. They are cheering for you.
  • Do not worry if you are not walking as much as other people. Everyone has their own speed.

aCOM-B: capability, opportunity, motivation, and behavior.

In addition, based on the findings of common constraints for physical activity among our target population, we integrated content that touched on physical symptoms, such as pain, and reframed messages to target the improvement of specific depressive symptoms with the purpose of increasing behavioral activation.

Phase 2: Pilot Testing


In the initial stage of this pilot, where we evaluated a representative subset of 36 messages (12 for each of the 3 theoretical categories) streamed through the smartphone app prototype, we found that patients tended to not carry their phone or not pay attention to the notification of text messages. However, patients generally started paying more attention to their phones and taking their phones with them more often after a few days into the pilot. In addition, although patients did not always remember receiving text messages when called by the research assistant, all patients reported liking the messages. A total of 75% (6/8) of patients reported feeling indifferent about the timing of the messages.

Furthermore, 38% (3/8) of patients reported not liking the messages in which they were compared with others (eg, other people have the same medical conditions and walked more). Messages about exercising with family members or about the benefits of being healthy to support family members were disliked by 25% (2/8) of participants, who reported not having strong family connections. However, all patients reported that they felt like they started walking more and that they wanted to continue to receive messages past their 2-week trial.

Following the completion of the pilot and based on the negative appraisal among some patients toward messages alluding to family member support, we removed any references to “family,” “peers,” or “friends” and replaced them with a blanket “loved ones” phrase. In the same vein, we decided to merge the family and peer orientations as social and therefore reclassify the messages as either individualistic or social.


Message editing was performed by the first author. The purpose of editing was the overall quality assurance of all existing messages. The first round of edits focused on identifying and correcting grammar, spelling, and punctuation errors as well as simplifying syntax to a lower reading level. All messages were adapted to a reading level of sixth grade or below (average=3.5). A web-based readability tool [46] was used to measure the reading level of each message using the Flesch-Kincaid grade level formula. Messages with scores higher than 6 were edited until they scored lower than the sixth-grade cutoff. A second round of edits focused on rewording to make the messages more concise to improve the experience of patients reading them. As the number of characters that can fit in a standard SMS is 160, messages with more than 160 characters were edited and reduced to fit this parameter. However, as the research staff had identified patient preferences for shorter messages in the previous phases, most messages were already under 140 characters.

Translation From English Into Spanish

The English text messages were translated into Spanish by the bilingual research staff. The translated messages were also reviewed by a native Spanish speaker fluent in English. In general, all translations were performed following a culturally sensitive communication standard [47]. We adapted syntax, lexical content, and idiosyncratic phrases that have been identified as key cultural aspects of translation in previous text messaging intervention research [47]. This means that the messages were not translated literally; rather, they were adapted and reframed to consider the ethnic diversity and other important cultural characteristics of one of the target populations of the DIAMANTE study—Spanish-speaking immigrants from Latin American countries. As such, neutral Spanish [48] words were used to the extent possible, and formal Spanish was used throughout (ie, using “usted” and its derivations in second person pronouns). A translation key was developed to standardize commonly translated phrases or words (Multimedia Appendix 1). During editing, any changes to English messages were simultaneously applied to the translated Spanish messages. Similar measures to reduce the literacy level of English content were applied to the Spanish content.

Phase 3: Crowdsourcing

In this phase, we submitted a representative sample of messages (n=90, 30 for each theoretical category) to the MTurk participants to test and validate the theoretical categorization. MTurk testing enabled us to ensure that the messages had been tagged in the right motivational categories and, therefore, refine the content and select the best ones. Messages were placed in the right category at an average of 58% (SD 0.21%) of the time. The MTurk respondents also rated messages for their overall quality from 1 to 5. The highest-rated messages were those in the benefit category. A total of 70% (63/90) of the messages were put in the right category at least 50% of the time. Furthermore, 11% (10/90) of the messages yielded a mean percent correct categorization rate of 40% or less. These messages were discarded from subsequent evaluations because of their thematic ambiguity. Participants completed the survey in an average of 6.5 minutes (Table 3).

Table 3. Amazon Mechanical Turk classification results of messages by theoretical category (N=30).
Theoretical categoryCorrect classification, mean n (%)
Benefit21 (70)
Self-belief16 (53)
Opportunity15 (50)

A challenge with using MTurk was to recruit a larger sample of Spanish-speaking workers to test the theoretical categorization of the messages. Otherwise, the task requirements of our categorization tests were easily implemented at the interface of the platform. However, as we found that the qualitative feedback about the message content from the testers was either incomplete or not helpful (ie, not reflective of the needs of our target population), we decided to discontinue this task after the second round of testing.

Phase 4: Expert Evaluation

To assess the content adequacy of the messages, all coauthors independently rated the high-performing messages of phase 3 (n=80) for subjective quality using a scale of 1 to 5. The coauthors gave the messages an average rating of 3.6 (SD 0.59) and reached a quality rating consensus for a given message only 3% (2/80) of the time. A total of 60% (48/80) of the messages received a rating of 3.5 and higher. Quantitative ratings and qualitative feedback were collected from each coauthor and analyzed in aggregate.

Phase 5: Final Selection

Feedback from all 4 phases was combined to select the best messages from the preselected subset from phase 4 (n=80). Messages that compared patients with others with the same medical conditions, which were predominantly disliked by participants in phases 1 and 2, were discarded (4/80, 5%). Of the remaining messages, those that were both sorted into the correct theoretical categories at least 50% of the time during phase 3 and rated at least 3.5 or higher in quality during phase 4 were considered as the clearest and highest impact content to include in the final text message bank. This combined feedback resulted in a final bank of 54 messages, with 18 messages per motivational category.

Principal Findings

Using a theory-based UCD framework, which consists of an iterative development process blending UCD methods with crowdsourcing and expert input, we produced a library of 54 motivational messages for a physical activity SMS intervention for low-income minority patients with comorbid diabetes and depression over a 2-year period. The development process comprised 5 phases: feasibility testing, pilot testing, crowdsourcing, expert evaluation, and final selection.

Lessons Learned

Patient feedback gathered in the UCD phases (1 and 2) was overwhelmingly positive: most patients reported no preference for the timing of message delivery, were receptive to receiving motivational text messages, and reported wanting to continue receiving messages upon conclusion of the pilot testing. However, user feedback is prone to response bias, and it is possible that such positive responses were because of patients not wanting to disappoint the study investigators. Findings from the UCD phases were especially helpful for discarding unappealing messages and editing thematic aspects of messages that did not resonate well with patients. For example, after phase 2, we rephrased any references to peer or familial relations to a broader social denomination (loved ones) based on the finding that participants who reported not having a family disliked messages that alluded to family support. The number of content changes we incorporated over 2 rounds of UCD testing underscores the importance of taking a user-centered approach to the development of a text messaging intervention. Results from the MTurk testing phase 3 showed that this platform is a relatively inexpensive, accessible, and rapid source of data for the validation of message classification. Thus, we learned that crowdsourcing methods are valuable for certain aspects of the design process in which a larger group of participants is desired or when seeking specific feedback that is not about engagement with content, such as for determining the correct thematic categorization of messages. Therefore, it is valuable to leverage crowdsourcing platforms as a more accessible and inexpensive source of feedback. Finally, we found that expert input and review (phase 4) rendered cohesiveness and reliability to the design process, given that this stage consisted of making independent ratings of the final set of messages to achieve consensus on the best ones. This was particularly important when it came to translating theory and evidence into succinct and engaging yet actionable and motivational messaging.

Strengths and Limitations

The strengths of this study include the multistaged and evidence-based development of cogent messaging that is grounded in the available scientific best practices, health behavior change and user engagement theories, and feedback from target users. This process allowed us to design content to match and adapt to the relevant needs of our patient population. As such, we wrote all messages at a sixth-grade or lower reading level, took cultural and linguistic appropriateness into account during editing and translation, and tested for acceptability and clarity among target patients. However, this study has some limitations. For instance, the sample sizes for the UCD phases were small and from one geographic location. However, the purpose of qualitative user research is not to produce findings that can be generalized to other populations but rather to generate data about attitudes or behaviors based on direct observation. Indeed, the UCD phases in our study were used to assess the acceptability and usability of a set of motivational text messages for use in a future physical activity intervention study tailored to a diverse clinical sample of low-income patients with diabetes and depression. A second limitation of our study is that although we did not collect demographic information beyond language and the country of residence, MTurk respondents are not likely to be representative of our target population. Thus, despite the aforementioned benefits of sampling from this crowdsourcing platform, findings from phase 4 were interpreted with caution, only used for theoretical category validation, and not used to inform subsequent content changes. A third limitation is that MTurk respondents did not receive the full set of draft messages (n=200) in the theoretical category validation task, which means that we were not able to collect categorization ratings for every message. However, we had raters assess only a subset of messages (n=90) as a measure to reduce the overall burden of the task and not overwhelm the respondents. We also noted messages with motivational construct ambiguity that were not tested in MTurk during the internal expert review phase and later discarded them from the final selection. A fourth limitation is that the MTurk categorization responses are subject to method bias as a function of low motivation; that is, participants may have not been willing or able to expend the required amount of cognitive effort and thus were less thorough in making accurate response selections. Online recruitment methods such as MTurk have been criticized as prone to yield samples full of low effort respondents [49]. However, we operationalized measures to reduce such bias in this content validation phase, including providing clear and precise definitions of the 3 motivational categories and providing a trial run during the onboarding stage for MTurk participants.

Future Directions

This paper describes in detail the process for designing and validating the content of an mHealth intervention that utilizes an adaptive learning algorithm to deliver optimized and tailored motivational text messages that promote physical activity. Previous studies have utilized similar methods to inform health text messaging development, such as message drafting based on behavior change theory [16,23] and public health guidelines [18], survey responses of participants’ preferences [23], focus groups and expert evaluation [18], and end user ratings of content understanding and appeal [16]. However, the process described herein is unique in that it integrates various design methods previously reported in literature and incorporates crowdsourcing via MTurk as an alternative content pretesting method. This is important because beyond a focus on usability, engagement, and implementation, alternative methods for intervention development in general and text messaging content in particular are a largely untapped area of inquiry in the field of mHealth. This paper thus helps advance mHealth research addressing intervention design processes, including content pretesting methods, which are not commonly detailed in the literature [17]. However, there continues to be a need for research on text messaging intervention design methodology, especially that which explores how to best integrate behavioral health theory in messaging content. There is also a need to continue evaluating new empirical models of content testing, including those that leverage crowdsourcing and online recruitment methods such as Amazon’s MTurk and Facebook.


We present a novel, integrative framework for combining UCD, expert input, and crowdsourcing to determine, create, and improve the content of text messages of a smartphone app to increase physical activity in low-income, English- and Spanish-speaking patients with depression and diabetes. This provides a research-based design approach for future developers of health text messaging interventions. This framework can potentially be extended to mHealth programs targeting other vulnerable populations and health behaviors.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Translation key.

DOCX File , 7 KB

  1. Thomas J, Jones G, Scarinci I, Brantley P. A descriptive and comparative study of the prevalence of depressive and anxiety disorders in low-income adults with type 2 diabetes and other chronic illnesses. Diabetes Care 2003 Aug;26(8):2311-2317. [Medline]
  2. Alegría M, Chatterji P, Wells K, Cao Z, Chen C, Takeuchi D, et al. Disparity in depression treatment among racial and ethnic minority populations in the United States. Psychiatr Serv 2008 Nov;59(11):1264-1272 [FREE Full text] [CrossRef] [Medline]
  3. Chow EA, Foster H, Gonzalez V, McIver L. The disparate impact of diabetes on racial/ethnic minority populations. Clin Diabetes 2012 Jul 16;30(3):130-133. [CrossRef]
  4. Katon WJ, Rutter C, Simon G, Lin EH, Ludman E, Ciechanowski P, et al. The association of comorbid depression with mortality in patients with type 2 diabetes. Diabetes Care 2005 Nov;28(11):2668-2672. [Medline]
  5. Colon E, Giachello A, McIver L, Pacheco G, Vela L. Diabetes and depression in the Hispanic/Latino community. Clin Diabetes 2013 Jan 17;31(1):43-45. [CrossRef]
  6. Aguilera A, Lyles CR. The case for jointly targeting diabetes and depression among vulnerable patients using digital technology. JMIR Diabetes 2017 Jan 17;2(1):e1 [FREE Full text] [CrossRef] [Medline]
  7. Craft LL, Perna FM. The benefits of exercise for the clinically depressed. Prim Care Companion J Clin Psychiatry 2004;6(3):104-111 [FREE Full text] [Medline]
  8. Silveira H, Moraes H, Oliveira N, Coutinho ES, Laks J, Deslandes A. Physical exercise and clinically depressed patients: a systematic review and meta-analysis. Neuropsychobiology 2013;67(2):61-68 [FREE Full text] [CrossRef] [Medline]
  9. Hamasaki H. Daily physical activity and type 2 diabetes: a review. World J Diabetes 2016 Jun 25;7(12):243-251 [FREE Full text] [CrossRef] [Medline]
  10. Buchholz SW, Wilbur J, Ingram D, Fogg L. Physical activity text messaging interventions in adults: a systematic review. Worldviews Evid Based Nurs 2013 Aug;10(3):163-173. [CrossRef] [Medline]
  11. Müller AM, Khoo S, Morris T. Text messaging for exercise promotion in older adults from an upper-middle-income country: randomized controlled trial. J Med Internet Res 2016 Jan 07;18(1):e5 [FREE Full text] [CrossRef] [Medline]
  12. Ferrer-Roca O, Cárdenas A, Diaz-Cardama A, Pulido P. Mobile phone text messaging in the management of diabetes. J Telemed Telecare 2004;10(5):282-285. [CrossRef] [Medline]
  13. Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015 Mar 18;36:393-415 [FREE Full text] [CrossRef] [Medline]
  14. Fortmann AL, Gallo LC, Garcia MI, Taleb M, Euyoque JA, Clark T, et al. Dulce Digital: an mHealth SMS-based intervention improves glycemic control in Hispanics with type 2 diabetes. Diabetes Care 2017 Oct;40(10):1349-1355. [CrossRef] [Medline]
  15. Demographics of mobile device ownership and adoption in the United States. Pew Research Center. 2021.   URL: [accessed 2020-01-05]
  16. Diez-Canseco F, Zavala-Loayza JA, Beratarrechea A, Kanter R, Ramirez-Zea M, Rubinstein A, et al. Design and multi-country validation of text messages for an mHealth intervention for primary prevention of progression to hypertension in Latin America. JMIR Mhealth Uhealth 2015 Feb 18;3(1):e19 [FREE Full text] [CrossRef] [Medline]
  17. Fitts WJ, Furberg R. Underdeveloped or underreported? Coverage of pretesting practices and recommendations for design of text message-based health behavior change interventions. J Health Commun 2015 Apr;20(4):472-478. [CrossRef] [Medline]
  18. Parra S, Ortega A, Kanter R, Kain J. Process of developing text messages on healthy eating and physical activity for Chilean mothers with overweight or obese preschool children to be delivered via WhatsApp. Cogent Social Sciences 2018 Oct 08;4(1):1-11. [CrossRef]
  19. Redfern J, Thiagalingam A, Jan S, Whittaker R, Hackett ML, Mooney J, et al. Development of a set of mobile phone text messages designed for prevention of recurrent cardiovascular events. Eur J Prev Cardiol 2014 Apr;21(4):492-499. [CrossRef] [Medline]
  20. Shaw RJ, Bosworth HB, Hess JC, Silva SG, Lipkus IM, Davis LL, et al. Development of a theoretically driven mHealth text messaging application for sustaining recent weight loss. JMIR Mhealth Uhealth 2013;1(1):e5 [FREE Full text] [CrossRef] [Medline]
  21. Ybarra ML, Holtrop JS, Bağci BA, Emri S. Design considerations in developing a text messaging program aimed at smoking cessation. J Med Internet Res 2012;14(4):e103 [FREE Full text] [CrossRef] [Medline]
  22. Arora S, Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial. Ann Emerg Med 2014 Jun;63(6):745-754. [CrossRef] [Medline]
  23. Woolford SJ, Clark SJ, Strecher VJ, Resnicow K. Tailored mobile phone text messages as an adjunct to obesity treatment for adolescents. J Telemed Telecare 2010;16(8):458-461 [FREE Full text] [CrossRef] [Medline]
  24. Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: examples from New Zealand. J Health Commun 2012;17 Suppl 1:11-21. [CrossRef] [Medline]
  25. Nundy S, Dick JJ, Goddu AP, Hogan P, Lu CE, Solomon MC, et al. Using mobile health to support the chronic care model: developing an institutional initiative. Int J Telemed Appl 2012;2012:871925 [FREE Full text] [CrossRef] [Medline]
  26. Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for internet interventions. Ann Behav Med 2009 Aug;38(1):18-27 [FREE Full text] [CrossRef] [Medline]
  27. Owens C, Farrand P, Darvill R, Emmens T, Hewis E, Aitken P. Involving service users in intervention design: a participatory approach to developing a text-messaging intervention to reduce repetition of self-harm. Health Expect 2011 Sep;14(3):285-295. [CrossRef] [Medline]
  28. Mao J, Vredenburg K, Smith PW, Carey T. The state of user-centered design practice. Commun ACM 2005 Mar 01;48(3):105-109. [CrossRef]
  29. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012;Suppl:49-56. [CrossRef] [Medline]
  30. Avila-Garcia P, Hernandez-Ramos R, Nouri SS, Cemballi A, Sarkar U, Lyles CR, et al. Engaging users in the design of an mHealth, text message-based intervention to increase physical activity at a safety-net health care system. JAMIA Open 2019 Dec;2(4):489-497 [FREE Full text] [CrossRef] [Medline]
  31. Diabetes and Depression Text Messaging Intervention (DIAMANTE). 2018.   URL: [accessed 2020-01-20]
  32. Aguilera A, Figueroa CA, Hernandez-Ramos R, Sarkar U, Cemballi A, Gomez-Pathak L, et al. mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE Study. BMJ Open 2020 Aug 20;10(8):e034723 [FREE Full text] [CrossRef] [Medline]
  33. Bandura A. Health promotion by social cognitive means. Health Educ Behav 2004 Apr;31(2):143-164. [CrossRef] [Medline]
  34. Evans MB, Cooke LM, Murray RA, Wilson AE. The sooner, the better: exercise outcome proximity and intrinsic motivation. Appl Psychol Health Well Being 2014 Nov;6(3):347-361. [CrossRef] [Medline]
  35. White KM, Terry DJ, Troup C, Rempel LA. Behavioral, normative and control beliefs underlying low-fat dietary and regular physical activity behaviors for adults diagnosed with type 2 diabetes and/or cardiovascular disease. Psychol Health Med 2007 Aug;12(4):485-494. [CrossRef] [Medline]
  36. Rothman A, Bartels R, Wlaschin J, Salovey P. The strategic use of gain- and loss-framed messages to promote healthy behavior: how theory can inform practice. J Communication 2006;56(s1):202-220. [CrossRef]
  37. Mays D, Turner MM, Zhao X, Evans WD, Luta G, Tercyak KP. Framing pictorial cigarette warning labels to motivate young smokers to quit. Nicotine Tob Res 2015 Jul;17(7):769-775 [FREE Full text] [CrossRef] [Medline]
  38. Larsson LS, Champine D, Hoyt D, Lin L, Salois E, Silvas S, et al. Social marketing risk-framing approaches for dental sealants in rural American Indian children. Public Health Nurs 2015;32(6):662-670 [FREE Full text] [CrossRef] [Medline]
  39. Farris KB, Salgado TM, Batra P, Piette JD, Singh S, Guhad A, et al. Confirming the theoretical structure of expert-developed text messages to improve adherence to anti-hypertensive medications. Res Social Adm Pharm 2016 Jul;12(4):578-591 [FREE Full text] [CrossRef] [Medline]
  40. Nouri SS, Avila-Garcia P, Cemballi AG, Sarkar U, Aguilera A, Lyles CR. Assessing mobile phone digital literacy and engagement in user-centered design in a diverse, safety-net population: mixed methods study. JMIR Mhealth Uhealth 2019 Aug 29;7(8):e14250 [FREE Full text] [CrossRef] [Medline]
  41. DiMatteo MR, Haskard-Zolnierek KB, Martin LR. Improving patient adherence: a three-factor model to guide practice. Health Psychol Rev 2012 Mar;6(1):74-91. [CrossRef]
  42. Linardon J, Fuller-Tyszkiewicz M. Attrition and adherence in smartphone-delivered interventions for mental health problems: a systematic and meta-analytic review. J Consult Clin Psychol 2020 Jan;88(1):1-13. [CrossRef] [Medline]
  43. Galobardes B, Shaw M, Lawlor DA, Lynch JW, Davey Smith G. Indicators of socioeconomic position (part 1). J Epidemiol Community Health 2006 Jan;60(1):7-12 [FREE Full text] [CrossRef] [Medline]
  44. Korkiakangas EE, Alahuhta MA, Laitinen JH. Barriers to regular exercise among adults at high risk or diagnosed with type 2 diabetes: a systematic review. Health Promot Int 2009 Dec;24(4):416-427. [CrossRef] [Medline]
  45. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011;6:42 [FREE Full text] [CrossRef] [Medline]
  46. Readable. 2021.   URL: [accessed 2019-04-09]
  47. Buchholz SW, Sandi G, Ingram D, Welch MJ, Ocampo EV. Bilingual text messaging translation: translating text messages from English into Spanish for the Text4Walking program. JMIR Res Protoc 2015 May 06;4(2):e51 [FREE Full text] [CrossRef] [Medline]
  48. González AM. Español neutro — español latino: hacia una norma hispanoamericana en los medios de comunicación. Roczniki Humanistyczne 2019 Jul 24;67(5):7. [CrossRef]
  49. Zickar MJ. Measurement development and evaluation. Annu Rev Organ Psychol Organ Behav 2020 Jan 21;7(1):213-232. [CrossRef]

COM-B: capability, opportunity, motivation, and behavior
DIAMANTE: Diabetes and Mental Health Adaptive Notification Tracking and Evaluation
mHealth: mobile health
MTurk: Amazon Mechanical Turk
UCD: user-centered design

Edited by L Buis; submitted 09.06.20; peer-reviewed by C Stiles-Shields, S Guo; comments to author 27.07.20; revised version received 21.09.20; accepted 13.04.21; published 19.05.21


©Laura Elizabeth Pathak, Adrian Aguilera, Joseph Jay Williams, Courtney Rees Lyles, Rosa Hernandez-Ramos, Jose Miramontes, Anupama Gunshekar Cemballi, Caroline Astrid Figueroa. Originally published in JMIR mHealth and uHealth (, 19.05.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.