Background: The Hookup is a collaborative project reaching young people in California with valuable sexual and reproductive health information and linkage to local resources. Due to limited access to subscriber contact information, it has been a challenge to evaluate the program.
Objective: The aims of this study were to determine the feasibility of using text messaging (short message service, SMS) as an evaluation tool for an educational text message-based program and to evaluate the program itself.
Methods: All subscribers of The Hookup were sent four survey questions via SMS about age, gender, location, referral source and behavior change. An incentive was offered for completing the survey and an opt-out option was provided in the initial message.
Results: All existing subscribers of The Hookup (N=2477) received a request to complete the survey using the SMS application on their mobile phones. A total of 832 (33.6%) subscribers responded to the initial question and 481 (20%) answered all four questions. Of the responses, 85% were received in the first two hours of the initial request. Respondents who answered the question about behavior change, 90% reported having made some positive change since subscribing to Hookup, including getting tested for STDs and HIV.
Conclusions: The survey methodology initiated a high response rate from The Hookup subscribers. The survey was able to provide data about subscribers in a short time period at minimal cost. The results show potential for using mobile SMS applications to evaluate SMS campaigns. The findings also support using SMS to provide young people with sexual health prevention messaging and linkage to health services.
Mobile phone use has exploded in the United States (US) and across the world in the last decade, providing a cheap, easy and fast way to communicate with others. Five billion text messages (short message service, SMS) are sent daily in North America . The volume of texting among teens has risen from 50 texts per day in 2009 to 60 texts in 2011 for the median teen texter. Of all teens, 63% say they exchange text messages with people everyday. This far surpasses the frequency with which they use other forms of daily communication [ , ].
Preliminary data has shown that SMS can be used successfully to increase knowledge around reproductive health issues [, ], as well as to promote short-term behavior change for sexual and reproductive health [ - ]. Behavior changes included intent to use condoms, increased access to reproductive and sexual health services, including sexually transmitted disease (STD) and HIV testing, and intent to have fewer sex partners.
A study in South Africa used a randomized controlled study design to test whether SMS messages could motivate people to test for HIV . Participant interactions were conducted via SMS and multimedia message service (MMS). The study found that 10 motivational-style SMS/MMS messages encouraged a statistically significant amount of people to test for HIV as compared to the control group. Another randomized control study in Australia, which used SMS as a health promotion tool, found a significant increase in STD knowledge and STD testing in the intervention group as compared to the control [ ].
There exists a small yet limited database of using SMS for sexual health research in the context of randomized control trials . Such information has typically been collected through surveys and interviews conducted face to face, by telephone, or online. These methods have the disadvantage of being costly and time consuming. According to a recent meta analysis, half of all online surveys received less than a 26% response rate, with the majority of responses (96.5%) taking two weeks to arrive [ , ]. Additionally, response rates to telephone surveys have continuously declined over the past decade [ , ].
As SMS is the most commonly used mobile app worldwide, extremely fast and highly reliable , known to be opened and read in short time after delivery [ ], this study aimed to discover if SMS could be a feasible tool to use to collect user data, particularly from a program concerning sensitive content around sexual and reproductive health for teens.
Designed and implemented as a partnership between California Family Health Council (CFHC) and youth+tech+health (YTH) in April 2009, The Hookup is a California statewide sexual health text messaging service for adolescents between the ages of 13 and 24. The goals of the program are to connect users with relevant and accurate sexual and reproductive health information, and serve as a convenient, confidential resource for accessing local clinics that offer free or low-cost sexual and reproductive health services.
By sending the word “Hookup” as a text message to 61,827, young people subscribe for weekly sexual health tips. Subscribers can also enter the word “Clinic” plus their zip code to obtain local clinic referrals for youth-friendly, free and low cost STD/HIV testing, and reproductive health services. Subscribers are also linked to Teensource, a youth-focused sexual health website, for further information. All content for the messages is developed through inputs from the youth community and reviewed by internal medical, sexual health, and adolescent experts. Promotion of the Hookup was conducted through outreach in school-based health centers, other community-based partnerships using print materials, social media, and through the Teensource website. To increase sign-ups, we distributed branded youth-selected collateral such as lip balm, palm cards, condom tins, and keychains. While the program staff has access to subscribers’ cell phone numbers, this access is for the sole purpose of sending The Hookup messages. This has posed a challenge in evaluating the program, and led to the design of an SMS-based survey tool.
The aims of this study were: (1) to assess the feasibility of using SMS as an evaluation tool for a text message-based sexual health program for California youth, and; (2) to evaluate the effectiveness of the program itself.
Feasibility of SMS Evaluation
At the time of the survey in January 2011, The Hookup had 2477 subscribers receiving weekly messages. The evaluation team, comprised of members from the community, CFHC and YTH, identified and developed the survey questions () adhering to the 160 character limitation of a standard text message in the United States. Additional elements included: specific format of responses “Text ___”; examples to avoid confusion “Like ___”; requirement for respondent to be concise.
The evaluation format included an introductory text message with an opt-out option, four survey question messages and a concluding message. The first three questions had limited response options, while the fourth question was open-ended. The text evaluation was conducted on the day and at the same time that the usual weekly messages were released. In order to increase response rates, the team offered an incentive in the form of a draw for US $50 Target gift cards for 10 subscribers who completed the survey.
All responses were exported from the SMS platform to excel to conduct the analysis and included phone number, date/time of response, and body of response. The evaluation team additionally tracked response time for completing the survey and compared it to traditional survey data collection methods.
Evaluation of The Hookup
The three main questions that the evaluation of the program looked to answer were: (1) are we reaching target population (California youth)?; (2) are our promotion efforts successful?; and (3) are our messages effective?
Throughout the program, YTH collected aggregate process data during the first year of operations to track overall program usage. This included the number of new subscribers, opt-outs, and clinic searches. In order to evaluate the program by answering the evaluation questions, in 2011 YTH and CFHC designed and conducted the following survey to understand the demographics of the population, how they heard about the program, and potential impact in terms of behavior change. For the program evaluation, The Hookup subscribers were asked to answer four questions about their age and gender, location, referral source, and behavior change efforts using an SMS survey via their mobile phones ().
Hookup: No tip this week. Your turn to help the Hookup! 4 questions coming your way. Answer all and you can win prizes. Text stop 2 end.
Hookup: 1st a little about u. Text back GUY or GIRL then ur age. Like ‘GIRL 16’ or 'GUY 20'. Msg&rates apply.
Hookup: We want to know if we’re reaching youth all over CA. Text back ZIP then ur zipcode to tell us where ur at! Like ‘ZIP 94117’. Msg&rates apply
Hookup: How’d u get hooked up with the Hookup? A friend tell u? Teacher? Saw a poster? Text back HOOKED and ur reason, like ‘HOOKED my bff’. Msg&rates apply
Hookup: How has the Hookup changed u? Text back CHANGE then ur change or something new, like ‘CHANGE STD test last week’. Msg&rates apply.
Hookup: Thx 4 helping us make Hookup better 4u! Stay tuned 2 learn if u won our raffle 2 win $25 iTunes card. Back 2 ur regular tips next week! Msg&rates apply.
Feasibility of SMS Evaluation
In total, 2477 Hookup subscribers were invited to participate in the survey, of those 832 (33.6%) responded and 58% (482) completed the full survey (responded to all four questions). Of all responses received, 85% were received within two hours,13% were received within the subsequent 22 hours, for a total of 98% of responses received within 24 hours.
Respondents were lost with each subsequent question, 22% were lost at second question, 34% at third and 50% at fourth question (cumulative).
Evaluation of The Hookup
Question 1: Are We Reaching Target Population (california Youth)?
Analysis of the responses showed that of the people who responded, over 90% were in the age range of 14-21 with 92% girls and 8% boys. Of the people who responded, 95% lived in zip codes that were California-based. The results appear to indicate that The Hookup is reaching the target population of California youth with a higher female than male representation. This finding has guided improved program promotion to reach males.
Question 2: Are Our Promotion Efforts Successful?
Due to the multi-faceted promotion plan, the question assessing how users heard about the Hookup was left open-ended. The responses reflected the program’s promotion efforts. The main sources of referral to the campaign were: teachers/school (40%), friend (25%), posters/stickers/tattoos (22%), and online/web (10%). This has resulted in enhancing school-based partnerships for The Hookup promotion, as well as increased web-based promotion efforts.
Question 3: Are Our Messages Effective?
Of the respondents who answered the question about behavior change (n=482), 90% indicated that they had made some behavior change since they started receiving the text messages, including using condoms (33%), increased knowledge and awareness (24%), initiation of birth control (15%), and getting tested for HIV and other STDs (15%).
Feasibility of SMS Evaluation: with traditional data collection methods facing modern day challenges, using text messaging for short surveys can be an effective and efficient method. This study demonstrates the feasibility of using text messaging for evaluating health campaigns in terms of response rates, turnaround time, and ability to reach the target audience. As compared to online surveys, the response rate was comparable and the turnaround time was much faster [, ].
Evaluation of The Hookup: the data collected correlates with key Hookup program objectives, raising awareness/knowledge, increasing safer sex and condom use, and getting tested for STD/HIV among youth in California. The findings on sources of referral to Hookup indicate that teachers and other school staff have been key in promoting the uptake of this program among youth.
The challenges to using SMS as a survey tool include fitting questions and potential responses within the 160 character limit of text messaging and accounting for the loss of respondents as the number of questions increase. Our findings indicate the need to keep the surveys short and the questions clear to minimize drop-out rates. Additionally, the methodology may have presented a selection bias in those who chose to respond, potentially posing a threat to validity. Although preliminary calculations indicate that this method could be cost-effective, more in-depth study is needed to conduct a cost-effective analysis of text messaging for program evaluation.
Using SMS messaging for prevention has high potential for reaching young people consistently and supporting their decision making process around their sexual and reproductive health. SMS messaging shows immense potential to increase access to free and low cost health care, and increase knowledge in the United States among hard to reach populations such as adolescents. With text messaging being seen as a low cost, viable method of reaching communities with valuable information and critical resources, this study sheds some light on actual field experience through a functioning statewide campaign, as opposed to a pilot demonstration project. Additionally, SMS is an innovative and effective tool for conducting evaluations of SMS-based programs.
We would like to acknowledge the support from Hipcricket, a mobile marketing and advertising company that allowed us to use SMS to reach underserved youth and conduct the evaluation.
Conflicts of Interest
- Lenhart A, Purcell K, Smith A, Zickuhr K. Social Media & Mobile Internet Use Among Teens and Young Adults. 2010 URL: http://www.pewinternet.org/2010/02/03/social-media-and-young-adults/ [accessed 2014-08-27] [WebCite Cache]
- Lenhart A. Teens, Smartphones and Texting. 2012 Mar 19. 2012 URL: http://www.pewinternet.org/2012/03/19/teens-smartphones-texting/ [accessed 2014-08-27] [WebCite Cache]
- Boyar R, Levine D, Zensius N. TECHsex USA: Youth Sexuality and Reproductive Health in the Digital Age. 2011. TECHsex USA URL: http://www.yth.org/wp-content/uploads/YTH-youth-health-digital-age.pdf [accessed 2014-08-27] [WebCite Cache]
- de Tolly KM, Constant D. Integrating mobile phones into medical abortion provision: intervention development, use, and lessons learned from a randomized controlled trial. JMIR Mhealth Uhealth 2014;2(1):e5 [FREE Full text] [CrossRef] [Medline]
- Risk A, Dzenowagis J. Review of internet health information quality initiatives. J Med Internet Res 2001;3(4):E28 [FREE Full text] [CrossRef] [Medline]
- Lim M, Hocking J, Hellard, ME, Aiken CK. SMS STI: a review of the uses of mobile phone text messaging in sexual health. Int J STD AIDS 2008:287-290. [CrossRef]
- Levine D, McCright J, Dobkin L, Woodruff A, Klausner JD. SEXINFO: A Sexual Health Text Messaging Service for San Francisco Youth. Am J Public Health 2008 Mar;98(3). [CrossRef]
- Perry RC, Kayekjian KC, Braun RA, Cantu M, Sheoran B, Chung PJ. Adolescents' perspectives on the use of a text messaging service for preventive sexual health promotion. J Adolesc Health 2012 Sep;51(3):220-225. [CrossRef] [Medline]
- de Tolly K, Skinner D, Nembaware V, Benjamin P. Investigation into the use of short message services to expand uptake of human immunodeficiency virus testing, and whether content and dosage have impact. Telemed J E Health 2012 Jan;18(1):18-23. [CrossRef] [Medline]
- Lim MS, Hocking JS, Aitken CK, Fairley CK, Jordan L, Lewis JA, et al. Impact of text and email messaging on the sexual health of young people: a randomised controlled trial. J Epidemiol Community Health 2012 Jan;66(1):69-74. [CrossRef] [Medline]
- Lim MS, Sacks-Davis R, Aitken CK, Hocking JS, Hellard ME. Randomised controlled trial of paper, online and SMS diaries for collecting sexual behaviour information from young people. J Epidemiol Community Health 2010 Oct;64(10):885-889. [CrossRef] [Medline]
- Baruch Y, Brooks C. Erratum for PMID 21180585. Therap Adv Gastroenterol 2012 Sep;5(5):371 [FREE Full text] [CrossRef] [Medline]
- Hamilton MB. Hamilton, MB. Online Survey Response Rates and Times, Background and Guidance for Industry URL: http://www.supersurvey.com/papers/supersurvey_white_paper_response_rates.htm [accessed 2014-08-27] [WebCite Cache]
- Sheehan K. E-mail Survey Response Rates: A Review. J Comput Mediat Commun 2001 Jan;6(2). [CrossRef]
- Pew Research: Center for the people & the press. The Problem of Declining Response Rates URL: http://www.people-press.org/methodology/collecting-survey-data/the-problem-of-declining-response-rates/ [accessed 2014-08-27] [WebCite Cache]
- Lenhart A, Purcell K, Smith A, Zickuhr K. Pew Research Internet Project. Social Media and Young Adults URL: http://www.insightexpress.com/pdfs/InsightExpress_1Q2012%20Digital%20Consumer%20Portrait_Jan2012.pdf [accessed 2014-08-27] [WebCite Cache]
- Mahmud N, Rodriguez J, Nesbit J. A text message-based intervention to bridge the healthcare communication gap in the rural developing world. Technol Health Care 2010 May;18(2):137-144. [CrossRef] [Medline]
|CFHC: California Family Health Council|
|SMS: short message service|
|STD: sexually transmitted disease|
Edited by G Eysenbach; submitted 05.06.14; peer-reviewed by M Lim, L Hightow-Weidman; comments to author 14.08.14; revised version received 27.08.14; accepted 27.08.14; published 03.11.14Copyright
©Bhupendra Sheoran, Rebecca A Braun, Jenna Patrice Gaarde, Deborah K Levine. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 03.11.2014.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.