Insights From a Text Messaging–Based Sexual and Reproductive Health Information Program in Tanzania (m4RH): Retrospective Analysis

Introduction

Mobile phones are a well-established platform for health education and behavior change, and as global levels of phone and internet access continue to rise, so does the application of mobile health (mHealth) approaches across multiple public health sectors [1-3]. Many mHealth interventions have the potential to generate and store vast amounts of system-generated data that could provide insight into user engagement, programmatic strengths, and areas that need improvement to maximize efficacy. However, despite the popularity of mHealth interventions, there is little documentation on how to use these data to monitor and improve programming or to evaluate impact.

The utilization of mHealth program data for process evaluation creates an opportunity to complement conventional impact evaluation approaches. The analysis of mobile phone program use data allows researchers to examine information from all participants compared with traditional process evaluation approaches (such as in-depth interviews or focus group discussions and questionnaires) that collect information from a subset of program participants. It is more cost effective than other methods, and because analytics are collected in real-time, the approach improves efficiency and eliminates the potential for recall or interview bias [4,5]. Research demonstrates that the analysis of these data does provide valuable information for program evaluation. For example, the examination of user experience data, including the number of menus accessed, from an mHealth intervention delivered to young and middle-aged male alcohol users in Scotland allowed researchers to measure the fidelity of the intervention delivery, user engagement, and message comprehensibility—all critical indicators of programmatic success [4].

As summarized by Perski et al, several conceptual models have been presented in the literature to depict the behavior change theory underlying digital behavior change interventions [6]. Across these, the level and quality of user engagement with digital health interventions are considered key factors in the intervention’s effectiveness [6-8]. The literature presents a broad definition of user engagement that often encompasses the number of intervention interactions, the relevance and relatability of the content, and repeat use [9,10]. User engagement is influenced by multiple aspects of intervention design such as resonance of messages, message delivery pattern, and program interactivity [11].

To better understand how users of the Mobile for Reproductive Health (m4RH) mHealth intervention engaged with the program, we conducted a retrospective analysis of m4RH system use data from 1 country (Tanzania) from September 2013 to August 2016. m4RH is a short message service (SMS) text messaging-based sexual and reproductive health (SRH) information program. It was initially developed in 2009 to provide information about the full range of contraceptive methods available locally and was first implemented in Kenya and Tanzania. The program has since been adapted in multiple countries, including Uganda and Rwanda, and its content has been expanded to include additional SRH topics such as HIV, sexually transmitted infections (STIs), and puberty.

User engagement can be measured using both subjective (qualitative) and objective (quantitative) measures [6,10]. Previously published research describes the results of qualitative data collection efforts to understand the acceptability and quality of interactions with the m4RH system [12,13]. This publication focuses on the retrospective analysis of subjective measures of engagement: system use metrics. The concept of user engagement, for the purposes of this analysis, builds upon the work of previous authors and is characterized by the following attributes as defined in Table 1 [6,14].

m4RH is an on-demand system that requires users to navigate and request information before accessing content. Users can access content in 2 ways: by accessing the navigation menu, which takes an average of 3 steps to access content or directly entering a content code after the welcome menu (ie, 2 steps). In Tanzania, participants access the m4RH program by SMS text messaging “m4RH” to short code “15014” to receive a menu of choices for accessing information on a variety of SRH topics including contraceptive methods, family planning (FP) clinic locations, role model stories (story installments modeling positive health attitudes, norms, and behaviors), and so forth. To access any of the menu options, respondents reply with another SMS text message using a numerical code corresponding to each menu item. Respondents only receive SMS text messages as they request them; no follow-up or reminder SMS text messages are sent after the initial query. To encourage respondent participation, respondents incur no cost for either sending or receiving SMS text messages when interacting with the m4RH system.

Methods

Data Analysis

Secondary data analysis was conducted using longitudinal data collected by mobile SMS text messaging from respondents of the m4RH program from 127 districts in Tanzania from September 2013 to August 2016. Participant interactions were extracted from system logs that captured each query to the m4RH system, including respondent mobile phone number (used to identify unique system interactions), information requested (eg, contraceptive methods, HIV and STI prevention, sex and pregnancy, and puberty), and date and time of the query. The analysis utilized descriptive analysis (proportions and means) of participant interactions and adapted the conversion funnel framework and indicators (mainly used in e-commerce) to map out and assess the journey of a user in a Web-based system. Conversion funnels are grounded in consumer decision-making theory and lifecycle, which mirrors behavior change models such as the transtheoretical model [17].

This study was reviewed and approved by the federally registered institutional review board of FHI 360, the Protection of Human Subjects Committee, and the National Institute for Medical Research Tanzania Institutional Review Board. Respondents did not receive reimbursement for participating in any study activity.

Data Cleaning

From September 2013 to August 2016, there were 4,112,460 system queries. Among those queries, 35,978 were invalid menu selections or text, 402,780 were duplicated menu selections sent in succession, and 5663 were m4RH specified incorrectly (ie, mr4h, m4hr, mrh4, or mfrh). After removing invalid and duplicate queries, the total population for analysis included 3,673,702 queries among 409,768 unique visitors.

Data cleaning and analysis was conducted using Stata 13 (Stata Corp, College Station, TX, USA). For analysis, all data were combined into 1 dataset consisting of multiple data points per user. This repeated measures dataset was sorted by users’ mobile number (unique ID) and date of system query. In this analysis, the proportions and frequencies of respondent queries are aggregated yearly unless specified, for ease of interpretability in user engagement over time. The data were examined for completeness (queries with missing time or date) and “correctness” (invalid menu option, duplicated menu selections sent in succession, and incorrectly specifying the program name, ie, mr4h, m4hr, mrh4, or mfrh). In addition, data were examined to identify illegitimate or implausible entries, for example, users sending duplicate queries in succession. Implausible entries identified during this process were excluded from analysis. No missing data were imputed. Respondents with missing or incorrect values were dropped from the analyses. To date, m4RH has reached 409,768 unique users in Tanzania living in 127 of the country’s 129 districts.

Results

Environment

The environment is hypothesized to influence engagement as social norms about seeking health information and mobile phone use, access to digital technology, and time use patterns are often similar in like settings [6]. Setting or location is one dimension of context. Through this analysis, we were able to examine the location using clinic location as a proxy. Only 5.67% (23,236/409,768) of new users requested information on the location of nearby clinics, and among these users, 33.30% (7,737/23,236) requested information on clinics within a given district. The region with the most requests was Dar es Salaam, representing 50.91% (3939/7737) of all requests, followed by Dodoma, Mwanza, and Arusha, representing 9.65% (747/7737), 8.52% (659/7737), and 7.60% (588/7737), respectively (see Table 2). These are all urban districts.

Interaction

During the analysis period, 409,768 new users initiated m4RH in Tanzania. New users represented roughly 11.15% (409,768/3,673,702) of all queries. Table 3 shows total interactions for new and return users. Across all time points, on average 381 new users accessed the system per day and interacted with the system 5 times within 24 hours and 7 times per month. Among new, return, and repeat queries, the average duration between the first and last daily query was 64 minutes, and the duration between each query was 17 minutes.

Most daily queries occurred between 12 pm and 2 pm and between 6 pm and 8 pm (796,678/3,673,702, 21.69%, and 740,682/3,673,702, 20.16%, respectively). The fewest queries took place between 12 am and 5 am (see Figure 1). Access time shows similar trends across years (Figure 2).

‎View this figure

‎View this figure

Depth

After first accessing the m4RH main menu, 89.58% (169,218/188,904) of users requested FP, contraceptive methods, youth-specific information, role model stories, or clinic locations submenus. The majority of all users (216,422/409,768, 52.82%) requested the contraceptive methods submenu. Among those that requested the submenu, 24.15% (98,969/409,768) requested additional information on natural FP methods and only 9.02% (36,968/409,768) requested information on intrauterine contraceptive devices (IUCDs).Close to the same number of users accessed FP and role model stories.

The types of information requested on contraceptive methods over the 36-month period followed similar trends relative to the number of new users requesting contraceptive method information each year. Few users requested information on clinic location (see Table 4).

As shown in Table 5, across all years, users were most interested in FP methods. Within the pregnancy prevention menu, roughly one-quarter (98,969/409,768, 24.15%) of all queries were requests for information on natural FP, as shown in Table 6 below.

Queries for information on other pregnancy prevention methods were more equally distributed, ranging from roughly 10% to 17%. The pregnancy prevention method with the fewest queries was IUCD, representing only 9.02% (36,968/409,768) of all pregnancy prevention queries (Table 6). Among respondents seeking information on youth content (Table 7), the most commonly requested topic was About Sex (34,542/249,368, 13.85%), while requests for Puberty and Choices represented similar proportions, at 6.43% (16,043/249,368) and 5.31% (13,238/249,368), respectively.

Menu content for “General Family Planning” and “Youth Content” was not implemented in Tanzania until 2015 and can help explain the lower frequencies among the 2 menu options for 2014 and 2015.

On average, users queried the system 5.56 times per month on the m4RH content described above. The rate of users that accessed m4RH and become active users declined slightly from to 98.05% (11,246/11470) in 2013 to 87.54% (56,969/97,465) in 2016.

Loyalty

The active use rate was highest during the first 2 years of the program, with nearly all users (11,246/11,470, 98.05%, and 33,551/34,830, 96.33%, respectively) who accessed m4RH continuing on to query more specific content from the system. An explanation for the decrease in conversion rates may be a result of the timing and frequency of promotions, for example, radio and magazine advertisements (Table 8).

Table 9 shows respondent retention and loyalty. Overall, slightly more than one-third (150,915/409,768) of all new users accessing m4RH sent queries at least once per month for 2 or more months and 67.86% (278,088/409,768) of new and return users requested information multiple times per month.

On average, users accessed the system for 3 months over the 4-year period. This is also demonstrated by the average number of keywords queried per month presented in sections “Activate” and “Active Use” (Figure 3). Of the 409,768 new users acquired from September 2013 to October 2016, 41.30% (169,218/409,768) became active users seeking further content (Figure 3). At the end of the study period, there were a total of 169,218 active users (users who requested specific keyword content). Among these users, the majority (117,805/169,218, 69.62%) of them requested content on pregnancy prevention methods, and only 11.98% (20,277/169,218) sought information on FP clinics.

Promotional periods were present for 15/36 months during the study period. However, as shown in Table 8, nearly all promotional activities took place in 2013 and 2014, and no promotional activities were reported for 2016. Looking specifically at 2014, 84.88% (99,012/116,654) of new users and 84.27% (31,084/36,887) of return users were acquired during months when promotional activities were being conducted. The unequal distribution of promotional activities makes it difficult to establish the influence of promotional activities on user engagement.

‎View this figure

Discussion

Principal Findings

Our study shows that approximately 67.86% (278,088/409,768) of users accessed the system more than once within a month period, averaging 5.56 times a month. Since m4RH is an on-demand system requiring people to access the service voluntarily, our results may suggest that the users have a keen interest in information to enhance their contraceptive knowledge. Our results also suggest a high degree of demand and motivation among users who access the service. A conversion rate of 41.30% (169,218/409,768) was observed among all users who accessed m4RH in a 3-year period, and an active use rate of 89.58% (169,218/188,904) among all who chose to navigate through the service menu.

More users were lost in the first step than in the second step of the navigation menu, the churn rates being 53.90% (220,864/409,768) and 10.42% (19,686/188,904), respectively. The churn rate could be explained by several factors. First, the fact that most users drop during the first stage of the navigation may be reflective of curious users who are trying out the service but decide they are not interested in going further. Second, it could suggest problems with message delivery (technology challenges have been observed contributing to delayed, truncated, or lack of message delivery). Third, although m4RH was developed with considerable formative work that allowed end users to contribute to the design of the service [18], this dropoff could raise the question of whether our “pull” design across all stages, with its multiple navigation steps structure, discourages users from accessing content. Alternatively, the dropoff could signify that menu option descriptions are unclear to users. Despite these potential challenges, we observed a high level of repeat use across all years. Close attention to churn rate data could help the program implement measures to reduce user attrition to further increase active use. For example, a cluster-randomized trial using hybrid text messaging and face-to-face interventions to reduce teenage pregnancies in Denver, Colorado, USA, was able to reduce user drop-out rates through the use of follow-ups by the program facilitators [19]. Analysis of churn rate data can allow program managers to identify the need to implement innovative solutions such as those applied by the Denver team. Because few programs have reported their activation rate, we are unable to determine how our findings compare to what is observed in similar programs.

Our study examined user engagement with service promotion. While we were unable to determine a statistically significant relationship between promotion and increased use, we did observe the greatest number of new users during the months when promotion activities occurred. Our findings show that during 2014, 84.88% (99,012/116,654) of new users and 84.27% (31,084/36,887) of return users were acquired during months when promotional activities were being conducted. This finding supports the rationale for investing in mass media promotion to influence m4RH use. Information about patterns in the time of day when users access the system can provide important data for future promotional efforts. Peak times for users to access m4RH across all 4 years were 1 pm and 8 pm. This finding helps to inform the timing of radio promotions to get maximum responses, as studies have shown that people are likely to respond to cues to action soon after they receive them [20].

Behavioral experts recommend that health messages should be tailored to audiences so that they can be personally relevant and affect behavioral change [21]. Past surveys to assess the demographic profile of users repeatedly showed that m4RH is mostly utilized by young people. This finding influenced the addition of youth content in 2015 as a means to tailor messages to this audience segment. However, only 11.28% (46,237/409,768) of all users and 26.56% (44,943/169,218) of active users requested youth content. This could suggest a higher demand for specific contraceptive information compared with general adolescent and youth SRH information that is also offered through other communication channels, for example, through HIV or AIDS health promotion programs. However, because the m4RH system did not routinely collect demographic data from users, we are unable to disaggregate information-seeking behavior by age or sex. Our results show that m4RH users mostly requested content on contraceptive methods and role model stories. Relatively few users requested information about clinic locations, suggesting that this content may not be interesting to most users of the m4RH system or users may have been discouraged by the need to enter additional information (first 3 or 4 letters of their ward). It is notable that the 4 regions with the most requests were all urban areas. This may be a result of greater promotional activity or higher levels of mobile phone ownership in urban areas [22]. However, this finding may be skewed as we know certain populations (key populations and youth) will travel over long distances to visit clinics in urban settings to protect their confidentiality. The request for clinic information from these locations may reflect this trend.

Analysis of data about information-seeking behavior has important implications for future iterations of the m4RH program and other health promotion programs with similar target audiences and information-sharing goals.

Challenges and Limitations

The m4RH system design presented some challenges in understanding respondent progression through the menus. As described above, the system was designed so that a user first queried information from the main menu and then from submenus. However, it appears that over time, respondents would become familiar with numerical codes assigned to menu items and directly access content without first accessing the main menu or submenus. Initially, we intended to identify repeat users by summing the total number of unique phone numbers that sent “m4RH” more than once. However, we realized that this definition of repeat use did not account for users who re-entered the system without first SMS text messaging “m4RH.” This required us to create a more sophisticated Stata code that instead summed the total number of unique phone numbers that sent any code to the system on >1 day. Respondent queries (either initial or follow-up), including invalid numerical codes, text, or a combination of both, also presented challenges for analysis. For example, a respondent interested in pregnancy prevention methods (menu code 66) might send a message reading “m4RH 66,” and although this query would be received by the m4RH system, the respondent would not receive any information on pregnancy prevention methods.

Information on use patterns alone does not paint a full picture of engagement [10]. Previously conducted and published research on the m4RH program in Tanzania and elsewhere sheds light on the program’s feasibility and acceptability [18,23]. However, the m4RH program in Tanzania did not routinely collect qualitative and demographic data, information that would have helped us understand the nature of engagement. For example, questions about why users drop off the navigation menu before they access content or whether there are variations in information preferences across age and gender groups would inform further design improvements.

Conclusions

The vertical and horizontal scale-up and wide reach of m4RH in Tanzania has been a major success story [24,25]. The rich data captured over the 3-year timeframe of this analysis and the results of our analyses provide insights into a useful framework to measure the degree and nature of user engagement using routine system data. For example, the conversion rate suggests a need to identify approaches to improve user engagement beyond their first contact with the m4RH system, and our data on the clinic locater use suggests that it may not provide value in its current form. Additional data collection efforts could provide a deeper understanding of our findings. These analyses contribute to knowledge about how users engage with SMS text messaging-based health promotion interventions and demonstrate how data generated on user interactions could inform improvements to the design and delivery of a service, thereby enhancing its effectiveness.