Published on in Vol 5, No 3 (2017): March

Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study

Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study

Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study

Original Paper

1Rollins School of Public Health, Department of Epidemiology, Emory University, Atlanta, GA, United States

2Department of Medicine, University of Washington, Seattle, WA, United States

3Center for Sexuality and Health Disparities, School of Nursing, University of Michigan, Ann Arbor, MI, United States

4Department of Medicine, Department of Medicine, University of Washington, Seattle, WA, United States

Corresponding Author:

Patrick S Sullivan, DVM, PhD

Rollins School of Public Health

Department of Epidemiology

Emory University

1518 Clifton Road NE

Atlanta, GA, 30322

United States

Phone: 1 4047272038

Fax:1 4043603039


Background: Men who have sex with men (MSM) are the group most impacted by the human immunodeficiency virus (HIV) epidemic and the only subgroup in the United States among which new HIV diagnoses are not decreasing. To achieve the US National HIV/AIDS (acquired immunodeficiency syndrome) Strategy goals of reducing new diagnoses by 25%, high (eg, 30-50%) coverage of multiple HIV prevention interventions is needed in both urban and rural areas. Mobile phone “apps” are an important channel through which prevention services could be provided at scale and at low marginal cost.

Objective: The aim of this study was to evaluate the usability and acceptability of a theory-based Android mobile phone app for HIV prevention.

Methods: The app included self-assessment tools; prevention recommendations; commodity (condoms, HIV self-tests) ordering; reminders to MSM for basic HIV prevention services, HIV testing, condom use, screening for preexposure prophylaxis (PrEP) and nonoccupational postexposure prophylaxis (nPEP); and prevention and treatment provider locators. The study recruited HIV-negative, Android-using MSM in Atlanta and Seattle who were asked to use the app for 4 months and complete a post-use survey. We measured the use of the app and its features, ordering of commodities, self-report of establishing an HIV testing plan, being HIV tested in the community, and starting PrEP or using nPEP. Usability was assessed using the system usability scale (SUS).

Results: A total of 121 MSM were enrolled (59.5%, 72/121 from Atlanta; 40.5%, 49/121 from Seattle). Median age was 28. Nearly half (48.8%, 59/121) were nonwhite, and most (85.9%, 104/121) were gay-identified. Most had tested for HIV in the past (85.1%, 103/121), and 52 (43.0%, 52/121) had a plan to test for HIV regularly. Men used the app for an average of 17.7 minutes over the first 4 months. Over the 4-month period, over half ordered condoms (63.6%, 77/121) and HIV test kits (52.8%, 64/121) on the app. Eight of 86 (9%) PrEP-eligible MSM started PrEP during the 4-month period; of those, 6 of the 8 reported that the app influenced their decision to start PrEP. The mean SUS was 73 (above average).

Conclusions: A theory-based mobile phone app was acceptable to MSM and was rated as having above-average usability. Most men used the commodity-ordering features of the app during the 4-month evaluation period, and nearly 1 in 10 PrEP-eligible men started PrEP, with most attributing their decision to start PrEP in part to the app. A broader, randomized controlled study of the impact of the app on uptake of prevention behaviors for MSM is warranted.

JMIR Mhealth Uhealth 2017;5(3):e26



Human immunodeficiency virus (HIV) prevention has become an HIV sero-status-dependent practice, in which an HIV test is the first step toward either a prevention continuum for HIV-negative individuals, or a treatment and care continuum for those testing HIV-positive. In other words, HIV prevention must rest on a foundation of accurate knowledge of HIV sero-status among key populations, followed by sero-status-specific prevention approaches. For those who are HIV-negative, biomedical interventions such as preexposure prophylaxis (PrEP) hold promise to reduce susceptibility to HIV [1-3].

Men who have sex with men (MSM) are a key risk group in the United States and are disproportionately impacted in terms of HIV prevalence [4-6] and incidence [7-9]. MSM are the only US risk group for whom HIV incidence increased after 2000 [8]; increases are especially alarming among young (15-24 years old) MSM [7] and MSM of color [10]. This has resulted in profound health disparities for both MSM relative to other adult men and within the MSM community, with a burden of HIV infection that is a staggering 67 times greater than for other men in the US population [11]. Disparities are especially pronounced among MSM of color [4].

Multiple models of HIV incidence in MSM suggest that to decrease HIV incidence in MSM, we will need to achieve 30-50% coverage of multiple prevention services and interventions (eg, condom promotion, HIV testing, PrEP, treatment as prevention) in at-risk MSM [12-15]. However, the uptake of routine HIV testing and PrEP is low: less than half of MSM test for HIV yearly [16] and in 2013, <5% were utilizing PrEP [17]. A recent summary of electronic tools for HIV prevention in MSM noted that promotion of certain types of prevention services are most amenable to provision through new technologies. Services for which eligibility can be determined through an algorithm are good candidates to bring to scale with technologies [18]. For example, behavioral eligibility for PrEP has well-described criteria and eligibility algorithms [19,20]. Using technology to promote uptake of prevention services for MSM would also make services more accessible to rural MSM [18]. This is especially important because MSM in rural areas may have lower access to HIV prevention services delivered in community-based organizations [21,22].

Here, we present briefly the development of a comprehensive mobile HIV prevention app for MSM, and describe and report the initial evaluation of the app for usability and acceptability.

Previous Work and App Development

Needs assessment for an HIV prevention app for MSM was conducted prior to app development using a 3-phase, iterative process [23,24]. Phase 1 consisted of separate focus group discussions with MSM, HIV testing counselors, and key community informants to identify preferences and requirements to consider including in a mobile HIV prevention app. Preliminary data from phase 1 was used to build an alpha version of the app. The alpha version was then theater tested with additional focus group discussions. Data from all phases were then used to develop features, language, and security to build into the beta version of the app. All of the app development was completed by Keymind (McLean, VA), a technology firm that specializes in creating data systems and mobile apps, including apps for health care providers and systems. The authors (PS, RS, JS, and TG) were directly involved in creating the requirements for the app; Keymind staff produced the app.

Theoretical Basis

The app content was developed based on the social cognitive theory of behavior [25]. Briefly, the app features enumerated in the following section were developed to fit into a framework of several health outcome behaviors (eg, making a plan to test regularly for HIV, using condoms, self-screening for PrEP, seeking HIV care for those living with HIV). For each health or prevention behavior, there were specific app features that were designed to promote goal setting, self-efficacy, outcome expectations, and self-regulation. For example, for the behaviors of HIV testing, the “Make a plan” app feature promoted goal setting, the presentation of several testing options and information promoted self-efficacy, information about the benefits of testing promoted positive outcome expectations, and a customizable reminder system for testing promoted self-regulation.

App Features

A list of app features and descriptions are shown in Table 1; screenshots of the app are available in Multimedia Appendix 1. Highlighted features included monthly risk assessment quizzes that offered tailored HIV-prevention related recommendations, quizzes to self-assess PrEP and nonoccupational postexposure prophylaxis (nPEP) eligibility, resources to create and schedule a custom HIV testing plan with reminders, a global positioning system (GPS) enabled map of HIV testing locations with their operational details, and ordering of free condoms and at-home HIV test kits.

Table 1. Features of the HealthMindr app during the pilot study, United States, 2015.
Initial and Monthly Risk AssessmentsProvides tailored, HIV-related prevention suggestions for users to consider based on quiz responses. Monthly Assessments used responses from the previous month’s assessment to ask if there had been any changes to give up-to-date suggestions.
PrEPa ScreenerAssesses PrEPa eligibility using seven questions developed by the CDC. The screener asks about time since last HIVb test, number of partners in past 3 months, condom use frequency, partner’s HIV status, bacterial STIsc in the past 12 months, and if engaged in exchange sex.
nPEPd ScreenerAssesses nPEP eligibility using a three question series about contact with bodily fluids, recency of exposure, and confidence in partners\' HIV status [26].
Find My Frequency (HIV Testing)Suggests HIV testing frequency of every 3 or 6 months based on five questions, including number of partners, partners\' HIV status, bacterial STI infections in the last 12 months, and injection drug, meth, or poppers use [27,28].
Compare HIV Tests/ Help Me ChooseAllows users to prioritize the most important aspects of an HIV testing experience based on location type, sample collection method, cost, HIV counseling available, wait time for results, and window period of test. Users can filter tests based on their preferences or complete a quiz for recommendations based on their stated preferences.
My Test PlanUsers can plan an HIV test by date, time, and location. Automated reminders can be set based on a chosen testing frequency. After being tested, users can record their HIV/STI test results within the app to keep a record of testing history.
RemindersPreferences can be set for how users receive testing and assessment reminders as pop-up notification, email, or neither. Users can choose the text of the reminder from a list of preset phrases or write their own message.
OrderingFree at-home HIV test kits (OraQuick and Home Access), a variety condom styles, and silicone and water-based personal lubricants were offered.
Location Details & MapProvides a map and details about testing locations, including address, phone number, type of organization, web address, days/hours of operation, service eligibility requirements (if any), fee information, languages available, and clinical services offered (HIV testing, HIV treatment, PrEP, nPEP, vaccinations, and so on). GPSe was enabled to show user’s location relative to testing locations. Locations were able to be filtered by the above characteristics to display locations with select characteristics.
FAQsFrequently asked questions related to HIV were included for users to reference. Users were also able to submit questions via the app to study staff.

aPrEP: preexposure prophylaxis.

bHIV: human immunodeficiency virus.

cSTI: sexually transmitted infection.

dnPEP: nonoccupational postexposure prophylaxis.

eGPS: global positioning system.

Pilot Study Overview

The purpose of this study was to assess the usability and acceptability of the HealthMindr app among MSM living in the metro areas of Atlanta, Georgia, and Seattle, Washington. The 2 cities were chosen because the availability of high-quality, gay-friendly prevention services differs in the 2 cities; we hypothesized that men who live in a city like Seattle where services are readily available and culturally competent might have less interest in accessing services through a mobile app. MSM were recruited on the Web and asked to install HealthMindr on their mobile phone, keep it on their phones for 4 months, and complete an evaluation survey at the end of the study period. Demographic and HIV prevention behaviors were collected during study enrollment. Brief periodic assessments were delivered monthly; the assessment of 10 risk questions allowed for prevention recommendations to be updated based on recent behaviors. App-based usage data was collected for all in-app actions participants made, including in-app button clicks, page views, and assessment or quiz responses. At the end of the participant’s study period, a Web-based evaluation survey was sent to participants to assess their HIV-related prevention behaviors during the pilot and app features that they did and did not find useful. Selected participants who were recommended to receive PrEP, including all who started PrEP, were invited to participate in individual in-depth interviews about their decision to start PrEP or not, and how the app influenced their decision-making process.

This study was approved by the institutional review boards of Emory University and the University of Washington. Participants were compensated US $25 each for completion of the baseline and 4-month follow-up surveys and US $5 each for the 3 periodic assessments administered through the app. Men who participated in individual in-depth interviewed were compensated US $40.

Recruitment and Enrollment

Participants were recruited from May 2015 and August 2015 using advertisements on Facebook and a social or sexual networking mobile phone app for MSM. Advertisements targeted adult male Facebook users residing in Atlanta or Seattle who indicated being a man interested in men. Advertisements on the MSM networking app used geolocation to deliver advertisements to men who opened the app on an Android device while in the Atlanta or Seattle metro area.

Interested participants who clicked on an advertisement were taken to a Web-based screening and enrollment survey and presented with a brief description of the study. Men completed an electronic informed consent to be screened for study eligibility and then completed a brief screening survey; to be eligible for the study, participants must have been ≥ 18 years of age, English-speaking, living in the Atlanta or Seattle metropolitan areas, assigned male sex at birth, and identifying as male at the time of the screening; must have had sex with a man in the past year; must have never tested positive for HIV; and must have owned an Android mobile phone device with current service. Eligible men were asked to complete an electronic informed consent for study participation. Disqualified respondents were not given a reason for ineligibility and were provided the principal investigator’s contact information.

Consenting participants were next shown a 7-minute introductory video embedded within the enrollment survey (Multimedia Appendix 2). The video introduced study staff members, explained study procedures including compensation schedule, provided detailed instructions with app screenshots on how to download and register the app, and demonstrated different app features. Mobile phone ownership was verified by sending participants an SMS text message (short message service, SMS) with a confirmation number that the participant was required to enter into the enrollment survey before continuing. The last section of the enrollment survey asked for demographic information and HIV testing history during the previous 24 months, use of free condoms in the last year, and whether the participant had ever used PrEP, nPEP, or at-home HIV test kits.

Access to the app was limited to participants through the use of a unique registration code provided only to participants; password and personal identification number (PIN) protection were provided. After successfully registering the app, participants were asked to complete an initial in-app screening assessment, which completed their enrollment into the study. Enrollment completions that were suspected to be fraudulent based on duplicate or similar phone numbers, Internet protocol (IP) addresses, or email addresses were screened and verified by calling and speaking with the participant before accepting him into the study. In all cases, study staff called all study participants within a week of study enrollment to introduce themselves and answer any questions or concerns.


Enrollment and Baseline Survey

Participants were asked demographic and baseline characteristics during study enrollment; including age in years; city of residence; race or ethnicity; sexual identity; recent HIV testing history; HIV status; plans for future HIV testing; and past use of PrEP, nPEP, condoms, and at-home HIV testing kits.

Evaluation Survey

After 4 months of use, participants were asked about motivation to use the app, HIV testing during the study period, PrEP and nPEP use during the study period, and at-home test kit and condom use for those who placed in-app orders. Participants were also asked to assess the app’s features, usability, design, content, and functionality using both Likert scales and optional open text fields. The usability of the app was further assessed using the system usability scale (SUS), a validated, industry standard scale used to evaluate a variety of products and services, including websites, mobile phones, computer software, and more [29]. The scale uses a series of questions to generate a usability score ranging from 0-100. An SUS score below 50 is not considered acceptable while above 70 is above average and >90 is superior [30].


The usage log was used to calculate the number of days participants used the app, pages of the app accessed, and the total time spent in the app. Time spent engaged within the app was quantified by calculating time passed between each action a user took and totaling the time for the visit. The longest 1% of time between actions (ie, longer than 2 minutes 38 seconds) was considered to not be representative of active app engagement. Time engaged within the app per person and per person-month was calculated. Descriptive statistics were used to examine app engagement and are reported as mean with range for time and action measurements. Participants’ ordering histories were kept for all at-home test kits, condoms, and personal lubricant orders placed. App pages accessed and features used by participants are reported as participant counts with percent. Evaluation responses are reported as percent of users who completed the evaluation survey. SUS results are reported as an aggregate score, using the method by which the scale was validated [30].

All analyses were performed using SAS 9.4 (SAS Institute Inc).

Study Population

Of the 919 Web-based survey responses, 244 (26.5%, 244/919) left the survey after reading the study description, 108 (11.7%, 108/919) did not complete the screening survey, and 257 (28.0%, 257/919) did not meet eligibility requirements. Reasons for ineligibility included not owning an Android phone (42.8%, 110/257), being HIV positive (27.6%, 71/257), and living outside of the study area (18.3%, 47/257). Of the 309 eligible survey responses, 127 (41.1%, 127/309) did not complete the postscreening enrollment survey, 21 (6.7%, 21/309) completed the survey but did not download the app, and 40 (12.9%, 40/309) were determined to be fraudulent attempts to enroll multiple times and were disqualified. Final study enrollment was 121 MSM, including 72 in Atlanta and 49 in Seattle. App usage data were available for 90.0% (109/121) of participants. Ninety-eight (81.0%, 98/121) participants completed the 4-month evaluation survey. Participation in the evaluation survey did not differ by age (median test: P=.34); race (chi-square test: P=.90); or knowing of a local place to be tested for HIV (chi-square test: P=.99).

Table 2. Select baseline characteristics of men who have sex with men (MSM) participating in a 4-month pilot study of a human immunodeficiency virus (HIV) prevention app, United States, 2015.
Total (n=121)Atlanta (n=72)Seattle (n=49)

n (%)

n (%)

n (%)

121 (100)

72 (100)

49 (100)

Age in years, median IQRa

28 (24-34)

28 (24-35)

28 (23-33)

Race or ethnicity

White or Caucasian62 (51.2)

34 (47.2)

28 (57.1)

Black or African American25 (20.7)

24 (33.3)

1 (2.0)

Hispanic or Latino10 (8.3)

3 (4.2)

7 (14.3)

Asian or Pacific Islander12 (9.9)

5 (6.9)

7 (14.3)

Multiracial or other12 (9.9)

6 (8.3)


Sexual orientationb

Gay or homosexual104 (86.0)

64 (88.9)

40 (81.6)

Bisexual14 (11.6)

8 (11.1)

6 (12.2)

Times tested for HIVcin last 24 months

012 (9.9)

8 (11.1)

4 (8.2)

1-250 (41.3)

26 (36.1)

24 (49.0)

3-432 (26.4)

23 (31.9)

9 (18.4)

5+27 (22.3)

15 (20.8)

12 (24.5)

Most recent HIV test result

Negative103 (85.1)

58 (80.6)

45 (91.8)

Never tested or unsure18 (14.9)

14 (19.4)

4 (8.2)

Knows local places to get an HIV test

Yes75 (62.0)

43 (59.7)

32 (65.3)

No or don\'t know24 (19.8)

14 (19.4)

10 (20.4)

Did not answer22 (18.2)

15 (20.8)

7 (14.3)


Never used106 (87.6)

65 (90.3)

41 (83.7)

Previously used4 (3.3)

3 (4.2)

1 (2.0)

Currently use11 (9.1)

4 (5.6)

7 (14.3)


Ever used6 (5.0)

5 (6.9)

1 (2.0)

Free condoms in last 12 months

Received and used55 (45.5)

33 (45.8)

22 (44.9)

Received and did not use21 (17.4)

12 (16.7)

9 (18.4)

At-home HIV test

Ever used33 (27.3)

14 (19.4)

19 (38.8)

aIQR: interquartile range.

b1 missing, 1 pansexual, 1 queer for Seattle.

cHIV: human immunodeficiency virus.

dPrEP: preexposure prophylaxis.

enPEP: nonoccupational postexposure prophylaxis.

Select baseline characteristics of participants are shown in Table 2. The median age of participants was 28 years and did not differ between Atlanta and Seattle. Nearly half of study participants were nonwhite; Seattle had a higher proportion of white participants than Atlanta. Nearly two-thirds knew of a local location to get an HIV test. During the previous 24 months, almost half of participants had tested for HIV 3 or more times; 15.7% (19/121) had never been tested or were unsure of their HIV status. Over 1 in 10 participants had used PrEP before; reported PrEP usage (ever) was higher in Seattle (16%, 8/49) than in Atlanta (10%, 7/72%).

App Engagement

Participants’ app engagement is presented in Table 3. Over 4 months, participants used the app on average for 17 minutes 40 seconds and made 133 clicks. There were no differences by median test between participants from Atlanta and Seattle in terms of total time spent on the app (P=.91) or total clicks (P=.62). There were also no differences in total time spent on the app by age (P=.21), race or ethnicity (P=.65), or whether participants knew of a local place to be tested for HIV (P=.99).

Total engaged time ranged from 25 seconds up to 77 minutes. Typically, the first visit was the longest (average first visit time: 7 minutes). Although the number of participants using the app each month declined, returning participants continued to engage with the app consistently with engagement during months 2-4 averaging 6.5 minutes and 49 clicks per month among active users. Most participants returned to the app multiple times over the 4-month period: 35% used the app on between 2 and 4 days, and 42% used the app on 5 or more days. Participants averaged using the app on a mean of 4.9 days.

Table 3. Time engaged and user clicks made in a human immunodeficiency virus (HIV) prevention app by men who have sex with men (MSM) participants during 4-month pilot study, United States, 2015.

Time engagedClicksa
nAverage (minutes) per userRangeAverage per userRange

Total pilot usage10917.70.4-76.81337-572

First visit usage1097.00.4-22.2527-131
Participant no.





aClicks capture all single actions made by a user, including logins, button clicks, and app navigation.

The percent of participants that used app features are reported in Figure 1. All participants completed the initial assessment, a requirement for enrollment into the study. Fewer participants completed the monthly assessments. HIV testing features were frequently accessed; ordering test kits and condoms was the most frequently accessed feature. Forty percent viewed PrEP information, a quarter used the PrEP screener, and nearly 1 in 5 opened the provider map feature from their PrEP screener result to view locations that offered PrEP. nPEP information was accessed by 25% of participants and 7% screened themselves for nPEP eligibility at least once.

Figure 1. Percent of MSM participants that used features or viewed pages in the app during a 4-month pilot study, United States, 2015 (n=109). MSM: men who have sex with men.
View this figure


Orders placed for free condoms and at-home HIV test kits are summarized in Table 4. Nearly two-thirds of men ordered condoms and over half ordered an at-home HIV test kit at least once. Of 154 kits ordered, most (84%, 129/154) were OraQuick kits. Most who placed an order did so on their first visit. Many participants placed multiple orders for condoms (38% of all who ordered) and for at-home HIV test kits (41%). Most at-home HIV test kits received were used by the participants to test themselves; 10% gave their test kit to someone else. Test kit use varied between the cities; 12% of Atlanta participants did not use the test kit compared with 41% of Seattle participants.

Table 4. Condoms and at-home human immunodeficiency virus (HIV) test kit ordered from an HIV prevention app by men who have sex with men (MSM) participants during a 4-month pilot study, United States, 2015.
Total (n=121)
n (%)
Atlanta (n=72)
n (%)
Seattle (n=49)
n (%)
Condom ordersa

On 1st visit64 (52.9)

38 (52.8)

26 (53.1)

At least once during pilot77 (63.6)

47 (65.3)

30 (61.2)

Place repeat order29 (24.0)

16 (22.2)

13 (26.5)

At-home HIV test ordersa

On 1st visit37 (30.6)

28 (38.9)

9 (18.4)

At least once during pilot64 (52.9)

39 (54.2)

27 (55.1)

Placed a repeat order26 (21.5)

18 (25.0)

8 (16.0)

Used the ordered condomsb

Yes61 (87.1)

40 (88.9)

21 (84.0)

No9 (12.9)

5 (11.1)

4 (16.0)

Condoms replaced condoms would have bought or received elsewhereb

Yes51 (72.9)

31 (68.9)

20 (80.0)

No18 (25.7)

14 (31.1)

4 (16.0)

Had condoms at time of orderb

Yes40 (57.1)

20 (44.4)

20 (80.0)

No30 (42.9)

25 (55.6)

5 (20.0)

Use of human immunodeficiency virus (HIV) home testb

Self34 (68.0)

25 (75.8)

9 (52.9)

Significant other3 (6.0)

3 (9.1)

0 (0.0)

Friend1 (2.0)

1 (3.0)

0 (0.0)

Acquaintance1 (2.0)

0 (0.0)

1 (5.9)

Not yet used11 (22.0)

4 (12.1)

7 (41.2)

Prior intentions of at-home HIV test usersb

Not planning to be tested34 (68.0)

21 (63.6)

13 (76.5)

Replaced a planned test16 (32.0)

12 (36.4)

4 (23.5)

aOrder history analyses include all pilot participants (n=121).

bReported in final evaluation survey (n=98).

Of those who ordered condoms, 87.1% reported using them. When asked about their motivations to place a condom order, participants said it was because the condoms were free (76%), it was convenient to do so in the app (67%), and they wanted to try different condom types (66%). Over two-thirds of participants who ordered test kits said they did not plan on being tested for HIV but ordered an at-home HIV test kit because it was offered in the app.


HIV and sexually transmitted infection (STI) testing behaviors during the pilot study are also shown in Table 5. Almost 80% of participants tested for HIV at least once during the pilot, and 56% tested multiple times. Additionally, almost half were screened for STIs. Three Atlanta participants tested newly positive for HIV. Their ages ranged from 18-29 years; 1 had never been tested for HIV before, and the other 2 had been tested for HIV 3 and 10 months before their enrollment. One participant had ordered and received an HIV test kit from the study. Two of the 3 who tested positive said their main motivation for being tested for HIV was the recommendation of the app to be tested; one said he tested routinely and had planned the test regardless of the app recommendation. Among participants who did not have a set HIV testing schedule at the start of the pilot, 63% had a schedule at the end of the pilot.

Table 5. Human immunodeficiency virus (HIV) and sexually transmitted infection (STI) testing history of men who have sex with men (MSM) participants during a 4-month pilot study of an HIV prevention app, United States, 2015.
Health Behavior at Post-Use Survey
Total (n=98)Atlanta (n=61)Seattle (n=37)
n (%)

n (%)

n (%)

HIV/STI Testing

Tested for HIVa75 (77)

48 (79)

27 (73)

Tested HIV positive3 (4)

3 (6)

0 (0)

Tested for STIsb46 (47)

29 (48)

17 (47)

HIV Test Plan

Had a previous HIV testing plan52 (53)

34 (56)

18 (49)

Did not have a previous plan, but now does29 (30)

18 (30)

11 (30)

Does not have an HIV testing plan17 (17)

9 (15)

8 (19)

Among those never tested or tested > 1 year ago at baseline

Tested during pilot13 (68)

8 (67)

5 (71)

Tested HIV positive1 (5)

1 (8)

0 (0)

HIV Test Plan

Had a previous HIV testing plan6 (32)

4 (33)

2 (29)

Did not have a previous plan, but now does9 (47)

7 (58)

2 (29)

Does not have an HIV testing plan4 (21)

1 (8)

3 (43)

aHIV: human immunodeficiency virus.

bSTIs: sexually transmitted infections.

At the beginning of the study, 24% of participants reported not having heard of PrEP and 53% reported not knowing about nPEP. During the app pilot, 9% (8/86) of PrEP-eligible participants not already taking PrEP began taking PrEP, and 1 participant used nPEP. Among the 8 men who started PrEP, 6 reported that the app influenced their decision to start PrEP for one or more reasons (because the participant did not know what PrEP was before using the app [1/8]; because the app recommended PrEP based on behavioral assessments [1/8]; because the app provided information about PrEP [3/8]; because the app allowed them to find a PrEP provider [3/8]).

App Evaluation

The content of the app was thought of positively overall, with 88% finding the level of detail and 81% finding the assessment recommendations to be useful or very useful. Additionally, 66% felt the app content helped them to stick to an HIV prevention plan. Most participants felt the app was a good balance of personal and professional language (71%) and the information was easy to understand (90%). Most participants felt confident in app security (86%), including a password or PIN offering sufficient protection (85%) and the app name and icon not readily associated as an HIV prevention app (84%).

The usability of the app was well received by participants with findings shown in Table 6. Overall, participants found the app to be easy to use and felt confident that they would be able to learn how to use it quickly and without technical assistance. The average composite score for the app was 73.4 (above average) [30].

Table 6. System usability scale (SUS) scores of an human immunodeficiency virus (HIV) prevention app by men who have sex with men (MSM) participants during 4-month pilot study, United States, 2015 (n=98).
StatementMeana (SD)Absoluteb
I would like to use this app frequently.3.7 (SD 1.1)3.7
The app was unnecessarily complex.2.2 (SD 1.0)3.8
The app was easy to use.4.1 (SD 0.8)4.1
I would need support from a technical person to be able to use this app.1.8 (SD 1.0)4.3
Various functions in the app were well integrated.3.9 (SD 1.0)3.9
There was too much inconsistency in this app.2.1 (SD 0.9)3.9
Most people would learn to use this app very quickly.4.1 (SD 1.0)4.1
The app was very cumbersome to use.2.4 (SD 1.0)3.6
I felt very confident using the app.4.0 (SD 0.8)4.0
I had to learn many things before I could get going with this app.2.0 (SD 1.0)4.0
Calculated score73.4 (SD 16.7) 

aScoring based on a scale from 1=totally disagree to 5=totally agree.

bAdjusts scores of negative statements so larger numbers are associated with positive statements.

When participants were asked about future app use, most said they would probably or definitely download the app again (69%), recommend it to a friend (71%), and continue to use it as part of their HIV prevention plan (66%). Very few participants reported they would probably or definitely not download it again (5%), not recommend the app to a friend (3%), or not continue to use the app themselves (13%).

Principal Findings

There are many reasons to be excited about the use of mobile apps to increase the uptake of basic HIV prevention services among MSM. Achieving our national strategy goal of reducing new HIV infections by 25% by 2020 will require making substantial improvements in HIV prevention for MSM. Based on current use of HIV prevention services [31], achieving the 30%-50% coverage of basic prevention services that will likely be needed to achieve such reductions would require either broad expansions of funding for existing service provision mechanisms or development of new modes of delivery that can scale with low incremental costs [18]. And the very communities of MSM most impacted by ongoing transmissions—young MSM and MSM of color—are the same men most likely to have mobile phones [32]. In short, there is an intersection of public health need, good fit of technology to programmatic needs, and high coverage of required device ownership among those at greatest need of services. Men in rural areas of the United States are in equal need of HIV prevention services, but are less likely to report receiving HIV testing and condom distribution, and are less likely to be aware of PrEP or have accessed PrEP [17]. Our data indicate that men will use a mobile phone app for comprehensive HIV prevention services, including engagement in HIV testing, condom use, and PrEP assessment, and that such an app is acceptable to MSM at risk for HIV acquisition.

Various metrics have been proposed to characterize engagement with mobile phone apps [ [33], although there are very limited data specific to health-related apps. With respect to engagement in the app, our data indicate that there was substantial variation in the extent of engagement with the app. Overall usage time was about 1.5 minutes per week over the period, which suggests that usage of our app was above average (above the median level of engagement observed in a recent analysis of usage of 22,000 mobile phone apps by over 600,000 mobile phone users) [33]. Our user database from the trial participants, 121 users, places our app above the 80th percentile in terms of numbers of users of available apps [33]. Thus, even in this limited implementation, our app shows engagement characteristics that are more favorable than most available apps.

Another way to measure engagement is the ordering and use of commodities by app users. Most users ordered condoms, HIV test kits, or both. It is also significant that most users who ordered condoms or HIV test kits reported using them. In a survey of MSM who received free condoms in bar or club settings, less than three quarters reported using the condoms they received [34]; in our study, nearly 90% of those who ordered condoms used them. It is important to recognize that many health jurisdictions already provide distribution of free condoms, sexual lubricants, and HIV test kits. In most places, this is accomplished through paying staff to distribute commodities in high risk venues. Distributing these prevention commodities by mail to those who want them and order them might offer a more targeted and cost-efficient way to fulfill public health functions of condom distribution and HIV testing promotion.

PrEP is an emerging biomedical approach to reducing HIV acquisition risk in high-risk MSM, but uptake of PrEP among MSM has been slow. Levels of awareness of PrEP are also low, especially among younger MSM and MSM in rural areas [17]. When less than 5% of US MSM have ever used PrEP, it is striking that nearly 10% of PrEP-eligible men initiated PrEP within a 4-month period of app use. According to our theoretical model of the PrEP continuum [35] and our qualitative data, the app likely promoted PrEP uptake by increasing awareness of PrEP (PrEP information features), facilitating access to PrEP providers, and helping men identify their personal risk of HIV infection and indications for PrEP. We recognize that some additional information such as information on costs and clinical procedures are currently lacking from the app, and based on the feedback we received from the participants, we will develop and add new materials to the PrEP component of the app.

According to a broadly accepted standard assessment of usability, our app was assessed as being above average using traditional criteria [30]; using a validated translation to qualitative terms, our app would be assessed as being between good and excellent [36]. Our data also suggest the possibility of social diffusion of the app, when it is made more broadly available: over 7 in 10 participants reported being willing to recommend the app to a friend. MSM have reported that recommendation from a trusted source was a major factor influencing their willingness to use an app [37]. By these standards, we conclude that our app is acceptable to the MSM whom we would have use the app.


Our study had several limitations. First, our participants were subject to selection bias across several dimensions. We recruited men who were using Facebook or a Web-based dating app, and who may have been more comfortable using mobile apps than other men. We restricted our study to MSM in Seattle and Atlanta because of the need to provide enhanced resource directories; men in other cities might view and use the app differently. We limited this evaluation to men whose phones used the Android operating system; users of Apple or Windows operating system (OS) phones might have different use experiences or opinions of the app. However, we note that Android phone ownership is higher among Americans of color and among younger Americans, who are the groups with the highest rates of new HIV infections [32]. Our usage might have been overestimated because participants were enrolled in a research study with compensation for study assessments. Our pilot occurred only among MSM in urban areas, so generalizability to other MSM (including those in rural areas) was limited. Although nearly half of our sample was nonwhite, we had limited enrollment of Hispanic MSM. Enrolling racial or ethnic minorities in Web-based prevention research has been historically challenging, and future studies should over-recruit MSM of color, including Hispanic MSM [38].

There is a broad interest in the use of mobile apps for HIV prevention and a scientific evidence base to support the idea that mobile apps can influence health behaviors. We have developed a theory-based mobile phone app to provide a basic package of HIV prevention services to MSM, and found it to be acceptable to users in Seattle and Atlanta. Furthermore, our data on usage of specific components and order of commodities provide examples of how engaging with the app could improve health outcomes and provide baseline estimates of uptake, which can be used to power future randomized studies of the app. We recommend that, because of the high costs of app development, prevention scientists use a staged approach of qualitative formative work, theater testing, and usability or acceptability testing to ensure that mobile apps that are moved into larger, more expensive efficacy trials meet basic standards of acceptability and usability. The HealthMindr app has been demonstrated as being acceptable to MSM, as being usable, and as being associated with use of prevention services. HealthMindr should be considered for further evaluation in a randomized controlled trial with outcomes of the uptake of prevention behaviors.


This work was generously supported by the MAC AIDS Fund and the Center for AIDS Research at Emory University (P30AI050409). The funders had no role in the design or conduct of the research, or the analysis.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Screenshots of HealthMindr app.

PDF File (Adobe PDF File), 1MB

Multimedia Appendix 2

Welcome video provided to participants to introduce the study and procedures.

MP4 File (MP4 Video), 9MB

  1. McCormack S, Dunn DT, Desai M, Dolling DI, Gafos M, Gilson R, et al. Pre-exposure prophylaxis to prevent the acquisition of HIV-1 infection (PROUD): effectiveness results from the pilot phase of a pragmatic open-label randomised trial. Lancet 2016 Jan 02;387(10013):53-60 [FREE Full text] [CrossRef] [Medline]
  2. Molina J, Capitant C, Spire B, Pialoux G, Cotte L, Charreau I, ANRS IPERGAY Study Group. On-demand preexposure prophylaxis in men at high risk for HIV-1 infection. N Engl J Med 2015 Dec 03;373(23):2237-2246. [CrossRef] [Medline]
  3. Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med 2010 Dec 30;363(27):2587-2599 [FREE Full text] [CrossRef] [Medline]
  4. Sullivan P, Wolitski R. HIV infection among gaybisexual men. In: Wolitski RJ, Stall R, Valdiserri RO, editors. Unequal Oppportunity: Health Disparities Affecting Gay and Bisexual Men in the United States. New York: Oxford University Press; 2007:220-247.
  5. CDC. 2013. HIV/AIDS Surveillance Report 2011   URL: [accessed 2017-02-14] [WebCite Cache]
  6. Rosenberg ES, Grey JA, Sanchez TH, Sullivan PS. Rates of prevalent HIV infection, prevalent diagnoses, and new diagnoses among men who have sex with men in US States, metropolitan statistical areas, and Counties, 2012-2013. JMIR Public Health Surveill 2016 May 17;2(1):e22 [FREE Full text] [CrossRef] [Medline]
  7. Prejean J, Song R, Hernandez A, Ziebell R, Green T, Walker F, HIV Incidence Surveillance Group. Estimated HIV incidence in the United States, 2006-2009. PLoS One 2011;6(8):e17502 [FREE Full text] [CrossRef] [Medline]
  8. Hall HI, Song R, Rhodes P, Prejean J, An Q, Lee LM, HIV Incidence Surveillance Group. Estimation of HIV incidence in the United States. J Am Med Assoc 2008 Aug 6;300(5):520-529 [FREE Full text] [CrossRef] [Medline]
  9. Koblin B, Mayer KH, Eshleman SH, Wang L, Mannheimer S, Rio CD, et al. Correlates of HIV incidence among black men who have sex with men in 6 U.S. cities (HPTN 061).   URL: [accessed 2017-02-15] [WebCite Cache]
  10. CDC. 2016 Sep 01. HIV and AIDS among Gay and Bisexual Men   URL: [accessed 2017-02-14] [WebCite Cache]
  11. Purcell DW, Johnson CH, Lansky A, Prejean J, Stein R, Denning P, et al. Estimating the population size of men who have sex with men in the United States to obtain HIV and syphilis rates. Open AIDS J 2012;6:98-107 [FREE Full text] [CrossRef] [Medline]
  12. Brookmeyer R, Boren D, Baral SD, Bekker L, Phaswana-Mafuya N, Beyrer C, et al. Combination HIV prevention among MSM in South Africa: results from agent-based modeling. PLoS One 2014;9(11):e112668 [FREE Full text] [CrossRef] [Medline]
  13. Sullivan PS, Carballo-Diéguez A, Coates T, Goodreau SM, McGowan I, Sanders EJ, et al. Successes and challenges of HIV prevention in men who have sex with men. Lancet 2012 Jul 28;380(9839):388-399 [FREE Full text] [CrossRef] [Medline]
  14. Sullivan PS, Jones J, Kishore N, Stephenson R. The roles of technology in primary HIV prevention for men who have sex with men. Curr HIV/AIDS Rep 2015 Dec;12(4):481-488. [CrossRef] [Medline]
  15. Jenness SM, Goodreau SM, Rosenberg ES, Beylerian E, Hoover K, Smith DK, et al. Impact of the Centers for Disease Control's HIV pre-exposure prophylaxis guidelines for men who have sex with men in the United States. J Infect Dis 2016 Jul 14;214(12):1800-1807. [Medline]
  16. Sanchez TH, Sineath RC, Kahle EM, Tregear SJ, Sullivan PS. The annual American men's internet survey of behaviors of men who have sex with men in the United States: protocol and key indicators report 2013. JMIR Public Health Surveill 2015 Apr 17;1(1):e3. [CrossRef]
  17. Sullivan PS, Sineath C, Kahle E, Sanchez TH. Awareness, willingness and use of oral pre-exposure prophylaxis (PrEP) among a national sample of US men who have sex with men. 2015 Presented at: AIDS Impact; July 2015; Amsterdam.
  18. Sullivan PS, Grey JA, Simon Rosser BR. Emerging technologies for HIV prevention for MSM: what we have learned, and ways forward. J Acquir Immune Defic Syndr 2013 Jun 1;63(Suppl 1):S102-S107 [FREE Full text] [CrossRef] [Medline]
  19. CDC. Interim guidance: preexposure prophylaxis for the prevention of HIV infection in men who have sex with men. MMWR Morb Mortal Wkly Rep 2011;60(03):65-68. [Medline]
  20. Smith DK, Grohskopf LA, Black RJ, Auerbach JD, Veronese F, Struble KA, US Department of Health and Human Services. Antiretroviral postexposure prophylaxis after sexual, injection-drug use, or other nonoccupational exposure to HIV in the United States: recommendations from the US Department of Health and Human Services. MMWR Recomm Rep 2005 Jan 21;54(RR-2):1-20 [FREE Full text] [Medline]
  21. Williams ML, Bowen AM, Horvath KJ. The social/sexual environment of gay men residing in a rural frontier state: implications for the development of HIV prevention programs. J Rural Health 2005;21(1):48-55 [FREE Full text] [Medline]
  22. Bowen AM, Williams ML, Daniel CM, Clayton S. Internet based HIV prevention research targeting rural MSM: feasibility, acceptability, and preliminary efficacy. J Behav Med 2008 Dec;31(6):463-477 [FREE Full text] [CrossRef] [Medline]
  23. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Preferences for a mobile HIV prevention app for men who have sex with men. JMIR Mhealth Uhealth 2014;2(4):e47 [FREE Full text] [CrossRef] [Medline]
  24. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Building a mobile HIV prevention app for men who have sex with men: an iterative and community-driven process. JMIR Public Health Surveill 2015;1(2):e18 [FREE Full text] [CrossRef] [Medline]
  25. Bandura A. Social cognitive theory of self-regulation. Organ Behav Hum Decis Process 1991;50:248-287. [CrossRef]
  26. CDC. 2014. Preexposure prophylaxis for the prevention of HIV infection in the United States - 2014   URL: [accessed 2017-02-14] [WebCite Cache]
  27. Branson BM, Handsfield HH, Lampe MA, Janssen RS, Taylor AW, Lyss SB, Centers for Disease ControlPrevention (CDC). Revised recommendations for HIV testing of adults, adolescents, and pregnant women in health-care settings. MMWR Recomm Rep 2006 Sep 22;55(RR-14):1-17 [FREE Full text] [Medline]
  28. Khanna A, Goodreau SM, Wohlfeiler D, Daar E, Little S, Gorbach PM. Individualized diagnosis interventions can add significant effectiveness in reducing human immunodeficiency virus incidence among men who have sex with men: insights from Southern California. Ann Epidemiol 2015 Jan;25(1):1-6 [FREE Full text] [CrossRef] [Medline]
  29. Bangor A, Kortum P, Miller J. The system usability scale (SUS): An empirical evaluation. Int J Hum Comput Interact 2008;24(6):574-594. [CrossRef]
  30. Jordan PW. SUS-A quick and dirty usability scale. In: Usability evaluation in industry. London: Taylor & Francis; 1996.
  31. Centers for Disease Control and Prevention (CDC). HIV testing and risk behaviors among gay, bisexual, and other men who have sex with men - United States. MMWR Morb Mortal Wkly Rep 2013 Nov 29;62(47):958-962 [FREE Full text] [Medline]
  32. Smith A. Pew Research Center. 2015. US Smartphone Use in 2015   URL: [accessed 2017-02-15] [WebCite Cache]
  33. Xu Q, Erman J, Gerber A, Mao Z, Pang J, Venkataraman S. Identifying diverse usage behaviors of smartphone apps. In: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference. 2011 Presented at: ACM SIGCOMM Conference on Internet Measurement; November, 2011; Berlin, Germany   URL:
  34. Khosropour C, Sullivan PS. Receipt and use of free condoms among US men who have sex with men. Public Health Rep 2013;128(5):385-392 [FREE Full text] [Medline]
  35. Kelley CF, Kahle E, Siegler A, Sanchez T, Del RC, Sullivan PS, et al. Applying a PrEP continuum of care for men who have sex with men in Atlanta, Georgia. Clin Infect Dis 2015 Nov 15;61(10):1590-1597. [CrossRef] [Medline]
  36. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. J Usability Stud 2009;4(3):114-123.
  37. Horvath KJ, Alemu D, Danh T, Baker JV, Carrico AW. Creating effective mobile phone apps to optimize antiretroviral therapy adherence: perspectives from stimulant-using HIV-positive men who have sex with men. JMIR mHealth and uHealth 2016;4(2):-.
  38. Sullivan PS, Khosropour CM, Luisi N, Amsden M, Coggia T, Wingood GM, et al. Bias in online recruitment and retention of racial and ethnic minority men who have sex with men. J Med Internet Res 2011 May;13(2):e38 [FREE Full text] [CrossRef] [Medline]

AIDS: acquired immunodeficiency syndrome
GPS: global positioning system
HIV: human immunodeficiency virus
MSM: men who have sex with men
nPEP: nonoccupational postexposure prophylaxis
PrEP: preexposure prophylaxis
PIN: personal identification number
STI: sexually transmitted infection
SUS: system usability scale

Edited by G Eysenbach; submitted 19.12.16; peer-reviewed by I Holloway, D Thompson, S Baral, E Sanders; comments to author 11.01.17; revised version received 26.01.17; accepted 28.01.17; published 09.03.17


©Patrick S Sullivan, Robert Driggers, Joanne D Stekler, Aaron J Siegler, Tamar Goldenberg, Sarah J McDougal, Jason Caucutt, Jeb Jones, Rob Stephenson. Originally published in JMIR Mhealth and Uhealth (, 09.03.2017.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.