This paper is in the following e-collection/theme issue:

mHealth for Wellness, Behavior Change and Prevention (2728) Online Dating, Sexual Health Behavior (148) HIV/AIDS/STI Prevention and Care (816) Design and Formative Evaluation of Mobile Apps (661) Risky Sexual Behavior (137) Formative Evaluation of Digital Health Interventions (2193)

Published on in Vol 8, No 5 (2020): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/15888, first published .
A Sexual Health Promotion App for Transgender Women (Trans Women Connected): Development and Usability Study

A Sexual Health Promotion App for Transgender Women (Trans Women Connected): Development and Usability Study

A Sexual Health Promotion App for Transgender Women (Trans Women Connected): Development and Usability Study

Authors of this article:

Christina J Sun1 Author Orcid Image ;   Kirsten M Anderson1 Author Orcid Image ;   Tamara Kuhn2 Author Orcid Image ;   Liat Mayer3 Author Orcid Image ;   Charles H Klein3 Author Orcid Image
Article Authors Cited by (13) Tweetations (5) Metrics

    Original Paper

    1School of Public Health, Oregon Health & Science University-Portland State University, Portland, OR, United States

    2dfusion, Scotts Valley, CA, United States

    3Department of Anthropology, Portland State University, Portland, OR, United States

    Corresponding Author:

    Christina J Sun, PhD, MS

    School of Public Health

    Oregon Health & Science University-Portland State University

    506 SW Mill St., Suite 450

    SCH

    Portland, OR, 97201

    United States

    Phone: 1 503 725 8858

    Email: chrsun@pdx.edu


    Background: HIV severely impacts the transgender communities in the United States, and transgender women have the highest HIV incidence rates among any identified risk group. Guided by formative research with transgender women and by an expert advisory panel of transgender women, we designed a prototype mobile app to promote HIV prevention among transgender women.

    Objective: This study aimed to develop and test the usability and acceptability of the prototype Trans Women Connected mobile app.

    Methods: We engaged in a 3-phase prototype development process. After conducting formative research about the health needs of this population, we outlined a theory-based app framework and developed three prototype activities (ie, a vision board, a pre-exposure prophylaxis [PrEP] education activity, and an interactive map). We then tested the usability and acceptability of the mobile app and activities with 16 transgender women using pre- and posttests, think-aloud protocols, and open-ended questions.

    Results: Participants reported high acceptability for the mobile app; the mean rating across all usability and likability questions was 5.9 out of 7. Service utilization intention, goal setting, and social support increased at posttest compared with pretest. Increases in self-efficacy in finding lesbian, gay, bisexual, transgender, and queer–friendly services; intention to seek online social support; and PrEP knowledge were statistically significant. Participants described the app as attractive and useful and perceived all three activities positively.

    Conclusions: This study describes the development and usability and acceptability evaluation of a prototype mobile app designed for and with transgender women for HIV prevention. The usability testing findings provided important insights toward refining and the further development of the Trans Women Connected mobile app. The results suggest that a mobile health intervention can support positive changes. The remaining development and efficacy randomized trial of the Trans Women Connected mobile app is currently underway.

    JMIR Mhealth Uhealth 2020;8(5):e15888

    doi:10.2196/15888

    Keywords

    transgender persons; HIV; sexual health; minority health; health care disparities; health status disparities 


    According to a 2016 meta-analysis, nearly 1 million adults in the United States are estimated to be transgender [1]. Transgender (or trans, in short) is a term for individuals whose gender expression and/or gender identity do not align with cultural expectations and gender norms associated with their sex assignment at birth [2]. Transgender women in the United States have been severely impacted by HIV [3-8], and more than 25% of them are living with HIV [5]. Transgender women of color are particularly affected; 56% of black/African American transgender women are living with HIV [5]. HIV incidence rates among transgender people are much higher than the national average [9] and highest among any group specifically tracked by the Centers for Disease Control and Prevention (CDC) [10]. Currently, the only transgender-specific program in the CDC’s Effective Interventions and Compendium of Evidence-Based Interventions and Best Practices for HIV [11,12] is an in-person, couples-based intervention designed for transgender women coupled with a cisgender male partner [13].

    Internet/electronic health (eHealth) and mobile health (mHealth) modalities may be particularly appropriate for transgender women, many of whom are socially marginalized, who live in areas that do not offer transgender-specific programming and/or may be uncomfortable participating in face-to-face interventions because of confidentiality concerns and fear of stigmatization [5,14-16]. Although empirical data on internet and mobile app use among transgender persons is limited, existing studies show that transgender women consistently use their phones for information gathering, socializing, and making sexual connections [17-19]. These communication practices provide opportunities for engaging transgender women in HIV prevention and sexual health promotion issues. Mobile phone apps also have the capacity to deliver individually tailored content based on motivations for use and app use patterns [20], making them particularly appropriate for addressing the heterogeneous experiences, identities, and needs of transgender women across the life course. Furthermore, although there are very few studies evaluating transgender-specific eHealth or mHealth programs [21], a growing body of research about diverse communities shows that new media intervention programs are acceptable and can be effective in reducing HIV/sexually transmitted infection–related risk behaviors and linking individuals to prevention and care services [22-34]. Given these realities, program developers and practitioners are increasingly calling for transgender-specific internet-, social media-, and mobile-based programs to expand the reach of HIV prevention and health promotion activities [3,35-38].

    The purpose of this study was to develop and evaluate the usability and acceptability of a prototype of the Trans Women Connected mobile app that addresses the unique HIV prevention and sexual health needs of transgender women. The goal was to engage transgender women through a strengths-based approach to HIV prevention and sexual health promotion, leveraging the power of social networks to identify and encourage protective factors.


    Overview

    We developed the prototype in three phases (see Figure 1). This process occurred in collaboration with a research team, an expert advisory panel (EAP), and a technology team. The EAP consisted of 3 transgender women of color, a Latinx woman, a Filipina woman, and an African American woman, each of whom holds a high-level position in an organization providing services to transgender women. They provided consultation and feedback throughout the prototype design and development process and were compensated as project consultants. The technology team included a technology director, an app developer, a database engineer, and a user experience and graphic designer and was responsible for the design and programming of the prototype. The ETR Institutional Review Board approved the study protocols and provided oversight. In phase one, the research team conducted formative research with potential users of the mobile app to understand the health needs of this population and how an app might support their health and to gather specific suggestions for the development and content of the app. Phase two included prototype design and development of the overall app structure and three activities (ie, a vision board, a pre-exposure prophylaxis [PrEP] education activity, and an interactive map). In phase three, a group of potential users used, rated the usability and acceptability of, and described their experiences with the prototype.

    ‎
    Figure 1. Prototype development process. PrEP: pre-exposure prophylaxis; STD: sexually transmitted disease.
    View this figure

    Phase One: Formative Research

    The research team conducted four focus groups and 20 in-depth qualitative interviews with transgender women between the ages of 18 and 59 years living in urban and rural areas in every region of the United States [39]. Briefly, women were recruited through partnerships with community-based organizations, fliers in community spaces and on social media, and word of mouth. Focus group participants received US $100 in compensation; interview participants received US $50. Topics included overall well-being and connectedness, transgender health, sexual health, use of internet and social media, and recommendations for a trans-specific app. Focus groups and interviews were audio-recorded and transcribed. We used the grounded theory open coding methodology and ethnographic methodologies to classify key themes [40,41].

    Phase Two: Prototype Design and Development

    On the basis of the findings of the formative research, the research team developed a preliminary curriculum outline, which was discussed with and refined by the EAP and, at times, additional transgender women from their networks. The technology team then determined the optimal formats for conveying the curriculum in an engaging manner, in small self-paced segments, with sufficient levels of interactivity, and customization. The technology team next created wireframes and storyboards to mock-up the layout, workflow, and interaction of all app elements. Through an iterative process of review and feedback, the EAP guided the designer to create a final set of imagery and style guide (see Figure 2 for the overall look). Following final approval of the storyboards, the app developer created the prototype framework using Adobe Animate. Using this cross-platform development method, the app could be built for several mobile platforms, including Android and iOS, and would allow the utilization of integrated mobile device (eg, mobile phone and tablet) features such as GPS and camera.

    ‎
    Figure 2. Welcome screen for the Trans Women Connected mobile app.
    View this figure

    Phase Three: Usability Testing

    Participants

    Two community-based organizations, one in San Francisco, California and one in Portland, Oregon, recruited participants. Inclusion criteria were ages 18 to 55 years, self-identify as a transgender woman, and English speaking. All participants provided written informed consent. Participants received US $100 in compensation.

    Procedures and Measures

    Using a standard think-aloud protocol, the usability test administrator (TK) asked each participant to complete a series of tasks associated with typical use case scenarios, such as using the menu to locate activities and completing activities. We also objectively measured participants’ task completion success and time to completion and asked them to articulate their questions and thinking process [42,43].

    Participants rated the difficulty, appearance, perceived usefulness, and educational value following each key task and after completion of all activities and completed a pre- and posttest that assessed service utilization, goal setting, social support, and HIV PrEP knowledge. We also solicited open-ended comments regarding their perceptions of the app and suggestions to improve it.

    Analysis

    We conducted descriptive analyses of the ratings and assessed changes from pre- to posttest using nonparametric tests in IBM SPSS version 24 (IBM Corp, Armonk, NY). The research team reviewed open-ended comments and identified key suggestions.


    Phase One: Formative Research

    Themes About Transgender Health

    We identified four main themes from the focus groups and interviews with 57 transgender women (45/57, 79% who were transgender women of color) related to the health of transgender women:

    1. Employment and structural factors play a key role in the health of transgender women.
    2. Transition-related care and mental health care are highly prioritized by transgender women. However, knowledgeable and affirming providers are difficult to find.
    3. The sexual health needs of transgender women are much broader than just HIV and sexually transmitted infections.
    4. Finding community and social support is challenging, but an important part of health for transgender women.
    Conceptual Feedback About a Mobile App for Transgender Health

    Our formative research found that most participants regularly used social media channels such as Facebook, Instagram, Snapchat, and various social and sexual networking apps to connect with friends, family, romantic/sexual partners, and work opportunities. However, the women also believed that the Trans Women Connected mobile app could fill a gap by providing resources specific to transgender women and an opportunity to connect to other transgender women (see Textbox 1). One Atlanta focus group participant described how this app could be a tool for “lifting the curtain and letting go of baggage.” In particular, participants valued possible app features, such as forums and direct messaging, to enable users to engage with each other and for the possibility of formal mentoring. Participants had questions and concerns about how the app would be structured, including issues of privacy and safety, and the possibility of forums and messaging being used for dating or hooking up. In general, however, participants were positive about the mobile app, and many expressed that it would fulfill an important need in the community.

    Suggestions for Trans Women Connected mobile app features and activities.
    • Give transgender women a voice in the process at every step of development
    • Images and videos used in the app depict actual transgender women
    • Space for trans women to connect and share information
    • Optional and customizable notifications
    • Customizable privacy settings to allow users to control how their information is displayed
    • Information about sexual health, including sexual pleasure
    • Support for healthy relationships and relationship skills
    • Information to help women locate both trans-specific resources and trans friendly businesses and organizations
    • Ways to block and report other users
    Textbox 1. Suggestions for Trans Women Connected mobile app features and activities.

    Phase Two: Prototype Development

    Guided by the formative research and EAP, the technology team created three prototype interactive activities. The first activity, entitled Create Your Own Vision (Board), provides participants with the opportunity to imagine how they would like their lives to be in eight domains: health, living situation, school/work, sex/love, making a difference, people, free time, and spirituality (see Figure 3). This activity seeks to help transgender women situate their well-being and sexual health within the social determinants of health and structural factors that emerged as the dominant theme of the focus groups and interviews.

    This interactive vision board allows participants to use their existing pictures, take pictures, create doodles/artwork, select quotes, and add their own text or other images to create a vision for each of the eight life domains. These eight collages are merged into a final scrollable collage that represents the user’s vision for their future. The activity also has the user identify a primary goal for their life today, lay out the steps needed to reach that goal, and identify who can help them execute each step. This goal setting activity provides a roadmap to reaching their goal that users can return to anytime. In addition, the vision board can be printed or shared with other app users—an option that all usability testers said they would use.

    The second activity—Is PrEP Right for Me or My Partners?—presents a basic overview of PrEP and guides participants through a benefits and risks assessment to help them decide if PrEP might be appropriate for them or their partners (see Figure 4). The activity primarily presents information through a series of videos featuring a charismatic African American transgender woman who is getting ready for and enjoying a night out with friends, including 2 transgender women who tell personal stories about using PrEP (see Figure 5). Interactive elements were included through questions and a poll that allows the user to see what other users are thinking about PrEP and a map that identifies PrEP providers in the users’ area.

    ‎
    Figure 3. A screenshot of the Create Your Own Vision (Board) prototype activity.
    View this figure
    ‎
    Figure 4. A screenshot of the Is pre-exposure prophylaxis Right for Me or My Partners? prototype activity.
    View this figure

    Responding to the need for helping transgender women find culturally competent and welcoming services, the staff developed the third activity: Interactive Resource Map. The map displays providers/organizations in five service categories: medical, support, educational, employment, and housing/living. On the basis of GPS location, users see icons for each entry and can customize the map to view all or certain categories. When users choose an icon, they see details about the resource including name, contact information, and user ratings and reviews (see Figure 6). In addition, the map allows users to add locations to the map as they identify transgender-friendly providers in their area.

    ‎
    Figure 5. A screenshot of a video within the Is pre-exposure prophylaxis Right for Me or My Partners? prototype activity.
    View this figure
    ‎
    Figure 6. A screenshot of the Interactive Resource Map prototype activity.
    View this figure

    Phase Three: Usability Testing

    Participant Characteristics

    A total of 16 racially diverse transgender women participated in the usability testing (see Table 1). Participants were aged 19 to 52 years, with an average of 34.5 years, and all were experienced with using mobile devices.

    Table 1. Participant characteristics.
    CharacteristicsValues
    Location, n (%)

    		San Francisco11 (69)

    		Portland5 (31)
    Gender identity, n (%)

    		Transgender woman14 (88)

    		Nonbinary2 (13)
    Racea, n (%)

    		American Indian or Alaska Native3 (19)

    		Asian5 (31)

    		Black or African American3 (19)

    		Native Hawaiian or other Pacific Islander1 (6)

    		White/Caucasian6 (38)

    		Other3 (19)
    Ethnicity, n (%)

    		Hispanic or Latinx2 (13)

    		Non-Hispanic or Latinx14 (88)
    Age (years), mean (SD)34.5 (9.28)
    Experience with mobile devices, n (%)

    		Very experienced14 (88)

    		A lot of experience1 (6)

    		Some experience1 (6)

    		Not much experience0 (0)

    		No experience0 (0)
    Primary mode to connect with transgender women, n (%)

    		Online0 (0)

    		In person7 (44)

    		Both online and in person8 (50)

    		Other1 (6)
    Primary mode to meet transgender women, n (%)

    		Online3 (19)

    		In person9 (56)

    		Both online and in person4 (25)

    		Other0 (0)

    aParticipants could select more than one option.

    App Usability

    The usability test results demonstrated that participants liked the mobile app and found it highly usable (see Table 2). The mean rating across all usability/likability questions was 5.9 out of 7. The study benchmark was a score of 4 or better for each individual rating component.

    Table 2. Usability and likability ratings of the Trans Women Connected mobile app.
    VariableValues, mean (SD)a
    Main menu

    		Likeability of opening screen6.25 (1.24)

    		Likeability of menu system5.66 (1.81)
    Create Your Own Vision (Board)

    		Likeability of the appearance6.28 (1.49)

    		Likeability of the interface6.44 (0.73)

    		Likeability of the narrator5.80 (1.14)
    Is PrEPb Right for Me or My Partners?

    		Likeability overall5.57 (1.60)

    		Likeability of women in the videos6.14 (1.03)
    Interactive Resource Map

    		Usefulness6.73 (0.59)
    Overall ratings of Trans Women Connected

    		Appearance6.38 (1.15)

    		Contents6.39 (0.65)

    		Ease of use6.25 (1.29)

    		Interest6.08 (1.16)

    		Professionally designed5.92 (1.38)

    		Enjoyability6.25 (1.48)

    		Usefulness6.50 (1.24)

    		Educational value6.25 (0.87)

    		Likelihood of using6.58 (0.68)

    		Likelihood of recommending6.62 (1.39)

    a7 represents the most positive or highest rating; the study benchmark was 4.

    bPrEP: pre-exposure prophylaxis.

    Changes in Service Utilization, Goal Setting, Social Support, and Pre-Exposure Prophylaxis Knowledge

    There was a positive trend and increase across multiple measures (eg, perceptions of available social support, service utilization intention, intention to mentor other transgender women on the app, and self-efficacy in discussing PrEP; see Table 3) after using the app. Increases in self-efficacy in finding lesbian, gay, bisexual, transgender, and queer (LGBTQ)–friendly services; intention to seek online social support; and PrEP knowledge were statistically significant.

    Table 3. Self-efficacy, intention, knowledge, goal setting, and social support: pre– and post–usability testing scores.
    VariableUsability testing scoresP value

    		PrePost
    Self-efficacy in finding LGBTQa-friendly services, mean (SD)3.19 (0.75)3.69 (0.48).01
    Likelihood of seeking LGBTQ-friendly services, mean (SD)3.69 (0.48)3.56 (0.81).48
    Intention to make medical appointmentb, mean (SD)3.88 (0.34)3.88 (0.50)>.99
    Created health goals, n (%)13 (81)13 (81)>.99
    Considered steps to reach goalc, n (%)9 (69)9 (69)>.99
    Have social support to help reach goal, n (%)12 (75)14 (88).16
    Availability of social support, mean (SD)2.63 (1.02)3.00 (0.97).71
    Intention to seek online supportb, mean (SD)2.56 (1.09)3.06 (1.00).03
    Intention to seek local support servicesb, mean (SD)3.38 (0.81)3.50 (0.63).48
    Helpfulness of mentoring from another trans woman, mean (SD)4.06 (1.24)4.19 (1.28).91
    Intention to mentor other transgender women on app, mean (SD)3.38 (0.89)3.56 (0.63).48
    Can identify friends/family for support, n (%)15 (94)16 (100)>.99
    PrEPd knowledge, mean (SD)2.50 (1.37)3.56 (0.63).008
    Intention to seek more info about PrEPb, mean (SD)2.80 (1.21)2.81 (1.22).58
    Intention to discuss PrEP with providerb, mean (SD)2.07 (0.80)2.00 (0.97)>.99
    Intention to discuss PrEP with partnerb,e, n (%)6 (40)6 (40)>.99
    Self-efficacy in discussing PrEP, mean (SD)3.00 (1.13)3.25 (0.86).16
    Connection to other transgender women, mean (SD)6.44 (2.60)6.63 (2.33).78
    Satisfaction with connection to other transgender women, mean (SD)5.00 (1.67)5.19 (1.83).26
    Level of social support from other transgender women, mean (SD)5.19 (1.87)5.21 (1.81).72

    aLGBTQ: lesbian, gay, bisexual, transgender, and queer.

    bIn the next 30 days.

    cOut of 13 participants.

    dPrEP: pre-exposure prophylaxis.

    eOut of 15 participants.

    Qualitative Findings From Usability Testing

    Usability testing participants also provided a number of qualitative comments about their experience using the prototype and suggestions for how it might be improved (see Textbox 2). Overall, this qualitative feedback was positive. As one participant told us, “Helping trans people navigate their health is a win.” As in the quantitative ratings, women expressed that they felt the app was useful and attractive. The welcome screen was described as being friendly and approachable, and many participants listed the esthetics of the app as one of their favorite parts. The women liked all three activities, with the map being a particular favorite. The potential for interactivity and connection with other transgender women was described as another attractive feature. The suggestions provided ranged from broad to very specific; a list of some of the most common or useful suggestions can be seen in Textbox 2.

    Suggestions to improve the Trans Women Connected mobile app prototype.

    General

    • More information or instructions on welcome screen and main menu
    • Tidier appearance/colors
    • More photos with a variety of transgender women (more representative of the community)
    • Options for text instead of audio; subtitles for all audio
    • Change or remove audio when returning to a screen
    • More black trans women developers
    • Provide suggestions for donating or volunteering

    Create Your Own Vision (Board)

    • Improve instructions and make navigation clearer
    • Ability to draw directly on vision board
    • Ability to add songs, gifs, or other “moving” content
    • Option to turn off narration
    • Option to share on social media

    Is PrEP Right for Me or My Partners?

    • Option to pause videos
    • Shorter videos
    • Option to view more scientific information
    • Clarify some information

    Interactive Resource Map

    • Add a search bar and list of locations
    • Ability to filter user-added locations
    • Filter or information on Americans with Disabilities Act-accessible locations
    • Add other types of resources, such as gender-neutral restrooms
    • Add a filter or new category for free activities
    Textbox 2. Suggestions to improve the Trans Women Connected mobile app prototype.

    Principal Findings

    We conducted a 3-phase process to develop a prototype HIV prevention mobile app for and with transgender women. In the first phase, we completed formative research, which indicated that HIV prevention for transgender women must address the larger context of women’s lives. In the second phase, we developed and refined the overall app framework and three prototype activities based on feedback from the EAP. In the third phase, we conducted usability testing. Participants rated the app highly on usability and likeability. There was a significant increase in self-efficacy in finding LGBTQ-friendly services, intention to seek online social support, and PrEP knowledge. Overall, the participants were supportive of the app, with many saying they would use it regularly with no additional incentive beyond the information and opportunity to connect with other transgender women.

    Strengths of the Approach

    There are a number of strengths of the Trans Women Connected prototype mobile app and research process. The 3-phase development process allowed us to collect formative data about the health priorities of transgender women and incorporate this information into the app prototype. We developed these activities because they addressed issues or requests observed in the formative research. This approach, which is considerably more extensive than the typical app development approach, helps to ensure that the selected prototype activities were addressing real needs of the community. The extensive formative research and the rigor of the analysis in this project, when combined with an iterative usability testing approach, are unique among mobile app developers.

    Although researchers have called for digital health interventions to (1) be grounded in behavioral theory [44,45], (2) utilize an in-depth qualitative understanding of the intended population [46], and (3) be iteratively developed with multiple stages of user feedback [47,48], the process has been rarely implemented consistently, with variations in the development process between academic researchers and technology industry developers. Most mHealth interventions designed by those within the industry do not yet incorporate theory-based strategies known to drive changes in health behaviors or undergo systematic testing to demonstrate their effectiveness [45,49,50]; however, they typically employ a rapid iterative development process of the technology. In contrast, interventions developed within an academic environment are more often grounded in behavioral theory [51], but researcher-driven mobile interventions often do not benefit from rapid iterative prototyping and standardized usability testing with iterative feedback [52].

    The development of this app integrated the strategies used in both academia and the industry and, in some instances, used methodologies that were more rigorous. For example, rather than use a local convenience sample for focus groups, as is typical of most software and app developers, the project conducted focus groups with transgender women in four regions and interviews with transgender women throughout the United States, with key transgender community stakeholders actively assisting in recruitment activities. By using a community-driven, grounded theory approach, the research team was able to identify and delve deeply into themes that would not have been detected through simply examining focus group transcripts from convenience samples.

    Furthermore, input from transgender women was integrated throughout the prototype development in coordination with the EAP and women from their networks. These iterative feedback loops began earlier than typically in app development and included gathering feedback on colors and image representations of transgender women and several other functional and aesthetic elements of the app before moving on to feedback on the app interface and activity content, function, and appearance. The inclusion of members of the LGBTQ community as part of the development process extended beyond formative research into voice talent/narration, video development, and design, with the designer being an LGBTQ-identified woman of color, the narrators being transgender women of color, and all video participants being transgender women. These multiple forms of community engagement not only resulted in a highly tailored app but also further provide a foundation for dissemination and active utilization once the Trans Women Connected mobile app is finalized. As suggested by other researchers, community engagement increases the likelihood the knowledge gained and the intervention, in this case an app, will benefit the community [53-55]. We recognize and appreciate the importance of the community’s priorities and solutions, which enhances the relevance and use of data and quality and validity of research [56-58].

    Additional strengths of the usability testing were the multiple types of data and an evaluation of impact. We gathered both quantitative and qualitative data about the experience of using the prototype mobile app; the quantitative data allowed us to verify that we were meeting numeric targets for usability and acceptability (as is typical for usability testing), whereas the qualitative data gave participants the opportunity to provide more in-depth feedback and suggestions. However, measuring the impact of the prototype on knowledge, attitudes, and intentions during the development process rather than when the app is completed goes considerably beyond standard usability testing, which is typically limited only to identifying problems within the user experience [59]. This approach helps determine whether the learning and engagement activities are performing as intended before building out the entire app.

    Finally, a central strength of this mobile app is the emphasis on addressing HIV holistically using a strengths-based perspective. This approach recognizes the resilience of the transgender community and the social and structural barriers that transgender women face [60]. Developing an effective intervention requires a comprehensive approach that intentionally targets community strengths and challenges [60-63]. Our formative research confirmed that attention to the larger context of trans women’s lives was key to HIV prevention [14], [61,64,65], and this understanding guided our development process.

    Limitations

    A few limitations of our study need to be considered. The usability testing sample was limited to 16 participants in two different cities in the West Coast. Research has determined that 7 to 10 usability participants will uncover 80% to 90% of usability problems [66,67], and we were able to test the mobile app with a diverse group of 16 participants. This sample allowed the examination of some of the different experiences and identities of transgender women. We likely identified functional problems with the app and the app was well received as usable and acceptable; however, it is possible that more geographically diverse participants or participants from rural areas would have qualitatively experienced the app differently.

    The inclusion of pre- and posttest knowledge and intention measures during a usability test is unusual and presents some limitations. Although significant increases in knowledge and intent were found on some variables, the sample size, although adequate for usability testing, is small for such testing and again may not represent the effect that would have been found with a more diverse or larger sample. However, the results are promising given that there were positive changes in key measures after viewing only three prototype activities, and this would seem to bode well for the efficacy of the complete mobile program. For several measures, there was no significant increase between pre- and posttest scores; we may have observed a ceiling effect. Given the short duration of app use, we were unable to measure behavior change. However, we measured antecedents of behavior change according to behavior change theories, and in the full trial when participants have a longer period to use the mobile app, we expect to observe increases in these antecedents and more protective behaviors.

    A final limitation of the project was that all members of the research team identify as cisgender. Although the input of transgender women was included throughout the development process, no transgender individuals participated directly in the interpretation and analysis of data. As a result, there may be aspects of the data that were missed or misinterpreted because of the absence of this perspective.

    Next Steps

    Given the positive results from the prototype development and usability testing, our next steps are to complete the development of the mobile app, including the storyboards and curriculum/content for the remaining activities, and conduct a cluster randomized controlled trial to assess the effectiveness of the Trans Women Connected mobile app. The expansion of the app to include additional activities will be based on a theoretical framework of gender affirmation, resilience, and cognitive behavioral theory and will be implemented using an agile development process with iterative testing. Through the entire process, we will continue to work with an expanded expert and community advisory group of transgender women who will provide feedback on the app; they will review materials to ensure the content reflects their experiences and those of their community. The project team, in collaboration with the expert advisors, will identify the most essential topics, which will then be reviewed and revised through an iterative process of refinement. A newly formed community advisory board, consisting of 10 geographically and racially diverse transgender women, will provide feedback on the activity plans and storyboards. The project team will then revise the documents, with the process continuing until the content is approved by the community advisors. In addition, the community advisory board will test and review completed interactive activities throughout the project.

    Conclusions

    Strong formative research, followed by an iterative review from members of the intended audience, resulted in a usable and acceptable mobile app that has been well received by transgender women. Usability testing findings provided important insights toward refining and the further development of the Trans Women Connected mobile app. The results of this study suggest that an mHealth intervention can address critical structural factors that shape transgender women’s lives and support positive changes in the knowledge and attitudes about social connection and health access that are antecedents to increasing protective and health promotion behaviors.

    Acknowledgments

    The authors would like to extend their gratitude for the time and effort of the transgender women who provided input and feedback on their app in the formative research interviews and focus groups, as members of the EAP, and in usability testing. They would also like to thank the community-based organizations that supported this project. This project was supported by grant R43MD012279 from the National Institute on Minority Health and Health Disparities and K12HS022981 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Minority Health and Health Disparities and the Agency for Healthcare Research and Quality.

    Conflicts of Interest

    None declared.

    1. Institute of Medicine, Board on the Health of Select Populations, Transgender Health Issues and Research Gaps and Opportunities Committee on Lesbian, Gay, Bisexual. The Health Of Lesbian, Gay, Bisexual, And Transgender People: Building A Foundation For Better Understanding. Washington DC: The National Academies Press; 2011.
    2. Meerwijk EL, Sevelius JM. Transgender population size in the United States: a meta-regression of population-based probability samples. Am J Public Health 2017 Feb;107(2):e1-e8. [CrossRef] [Medline]
    3. Centers for Disease Control and Prevention. 2019. HIV And Transgender People   URL: http://www.cdc.gov/hiv/group/gender/transgender/ [accessed 2019-08-11]
    4. Clements-Nolle K, Marx R, Guzman R, Katz M. HIV prevalence, risk behaviors, health care use, and mental health status of transgender persons: implications for public health intervention. Am J Public Health 2001 Jun;91(6):915-921. [CrossRef] [Medline]
    5. Herbst JH, Jacobs ED, Finlayson TJ, McKleroy VS, Neumann MS, Crepaz N, HIV/AIDS Prevention Research Synthesis Team. Estimating HIV prevalence and risk behaviors of transgender persons in the United States: a systematic review. AIDS Behav 2008 Jan;12(1):1-17. [CrossRef] [Medline]
    6. Operario D, Soma T, Underhill K. Sex work and HIV status among transgender women: systematic review and meta-analysis. J Acquir Immune Defic Syndr 2008 May 1;48(1):97-103. [CrossRef] [Medline]
    7. Poteat T, Scheim A, Xavier J, Reisner S, Baral S. Global epidemiology of HIV infection and related syndemics affecting transgender people. J Acquir Immune Defic Syndr 2016 Aug 15;72(Suppl 3):S210-S219 [FREE Full text] [CrossRef] [Medline]
    8. Reback CJ, Fletcher JB. HIV prevalence, substance use, and sexual risk behaviors among transgender women recruited through outreach. AIDS Behav 2014 Jul;18(7):1359-1367 [FREE Full text] [CrossRef] [Medline]
    9. Grant JM, Mottet LA, Tanis J, Harrison J, Herman JL, Keisling M. National LGBTQ Task Force. Washington, DC: National Center for Transgender Equality and National Gay and Lesbian Task Force; 2011. Injustice at Every Turn: A Report of the National Transgender Discrimination Survey   URL: https://www.thetaskforce.org/wp-content/uploads/2019/07/ntds_full.pdf [accessed 2019-08-12]
    10. Centers for Disease Control and Prevention. 2014. CDC-Funded HIV Testing: United States, Puerto Rico, and US Virgin Islands, 2014   URL: https://www.cdc.gov/hiv/pdf/library/reports/cdc-hiv-funded-testing-us-puerto-rico-2014.pdf [accessed 2019-08-12]
    11. Centers for Disease Control and Prevention. 2019. Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention   URL: https://www.cdc.gov/hiv/research/interventionresearch/compendium/rr/index.html [accessed 2019-08-12]
    12. Centers for Disease Control and Prevention. 2019. Effective Interventions: HIV Prevention That Works   URL: https://www.cdc.gov/hiv/effective-interventions/index.html [accessed 2019-08-12]
    13. Reisner SL, Menino D, Leung K, Gamarel KE. 'Unspoken Agreements': Perceived Acceptability of Couples HIV Testing and Counseling (CHTC) among cisgender men with transgender women partners. AIDS Behav 2019 Feb;23(2):366-374. [CrossRef] [Medline]
    14. Hines DD, Ryan M. It's not just about condoms and sex: using syndemic theory to examine social risks of HIV among transgender women. In: Wright ER, Carnes N, editors. Understanding the HIV/AIDS Epidemic in the United States. Social Disparities in Health and Health Care. Cham, Switzerland: Springer; 2016:99-130.
    15. Andrasik MP, Yoon R, Mooney J, Broder G, Bolton M, Votto T, HVTN 505 study team, NIAID HIV Vaccine Trials Network. Exploring barriers and facilitators to participation of male-to-female transgender persons in preventive HIV vaccine clinical trials. Prev Sci 2014 Jun;15(3):268-276 [FREE Full text] [CrossRef] [Medline]
    16. Kosenko K, Rintamaki L, Raney S, Maness K. Transgender patient perceptions of stigma in health care contexts. Med Care 2013 Sep;51(9):819-822. [CrossRef] [Medline]
    17. Benotsch EG, Zimmerman RS, Cathers L, Heck T, McNulty S, Pierce J, et al. Use of the internet to meet sexual partners, sexual risk behavior, and mental health in transgender adults. Arch Sex Behav 2016 Apr;45(3):597-605. [CrossRef] [Medline]
    18. Krueger EA, Young SD. Twitter: a novel tool for studying the health and social needs of transgender communities. JMIR Ment Health 2015;2(2):pii: e16 [FREE Full text] [CrossRef] [Medline]
    19. Young SD, Jaganath D. Online social networking for HIV education and prevention: a mixed-methods analysis. Sex Transm Dis 2013 Feb;40(2):162-167 [FREE Full text] [CrossRef] [Medline]
    20. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health 2016 Mar 1;3(1):e7 [FREE Full text] [CrossRef] [Medline]
    21. Rhodes SD, McCoy TP, Tanner AE, Stowers J, Bachmann LH, Nguyen AL, et al. Using social media to increase HIV testing among gay and bisexual men, other men who have sex with men, and transgender persons: outcomes from a randomized community trial. Clin Infect Dis 2016 Jun 1;62(11):1450-1453 [FREE Full text] [CrossRef] [Medline]
    22. Lightfoot M, Comulada WS, Stover G. Computerized HIV preventive intervention for adolescents: indications of efficacy. Am J Public Health 2007 Jun;97(6):1027-1030. [CrossRef] [Medline]
    23. Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015 Mar;12(1):173-190 [FREE Full text] [CrossRef] [Medline]
    24. Card JJ, Kuhn T, Solomon J, Benner TA, Wingood GM, DiClemente RJ. Translating an effective group-based HIV prevention program to a program delivered primarily by a computer: methods and outcomes. AIDS Educ Prev 2011 Apr;23(2):159-174. [CrossRef] [Medline]
    25. Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV treatment & prevention: a systematic review of the literature. Open AIDS J 2013;7:17-41 [FREE Full text] [CrossRef] [Medline]
    26. Klein CH, Kuhn T, Huxley D, Kennel J, Withers E, Lomonaco CG. Preliminary findings of a technology-delivered sexual health promotion program for black men who have sex with men: quasi-experimental outcome study. JMIR Public Health Surveill 2017 Oct 24;3(4):e78 [FREE Full text] [CrossRef] [Medline]
    27. Noar SM. Computer technology-based interventions in HIV prevention: state of the evidence and future directions for research. AIDS Care 2011 May;23(5):525-533 [FREE Full text] [CrossRef] [Medline]
    28. Noar SM, Willoughby JF. eHealth interventions for HIV prevention. AIDS Care 2012;24(8):945-952 [FREE Full text] [CrossRef] [Medline]
    29. Schnall R, Travers J, Rojas M, Carballo-Diéguez A. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res 2014 May 26;16(5):e134 [FREE Full text] [CrossRef] [Medline]
    30. Swanton R, Allom V, Mullan B. A meta-analysis of the effect of new-media interventions on sexual-health behaviours. Sex Transm Infect 2015 Feb;91(1):14-20. [CrossRef] [Medline]
    31. Conserve DF, Jennings L, Aguiar C, Shin G, Handler L, Maman S. Systematic review of mobile health behavioural interventions to improve uptake of HIV testing for vulnerable and key populations. J Telemed Telecare 2017 Feb;23(2):347-359 [FREE Full text] [CrossRef] [Medline]
    32. Fuchs JD, Stojanovski K, Vittinghoff E, McMahan VM, Hosek SG, Amico KR, et al. A mobile health strategy to support adherence to antiretroviral preexposure prophylaxis. AIDS Patient Care STDS 2018 Mar;32(3):104-111 [FREE Full text] [CrossRef] [Medline]
    33. Dillingham R, Ingersoll K, Flickinger TE, Waldman AL, Grabowski M, Laurence C, et al. PositiveLinks: a mobile health intervention for retention in HIV care and clinical outcomes with 12-month follow-up. AIDS Patient Care STDS 2018 Jun;32(6):241-250 [FREE Full text] [CrossRef] [Medline]
    34. Sun CJ, Stowers J, Miller C, Bachmann LH, Rhodes SD. Acceptability and feasibility of using established geosocial and sexual networking mobile applications to promote HIV and STD testing among men who have sex with men. AIDS Behav 2015 Mar;19(3):543-552 [FREE Full text] [CrossRef] [Medline]
    35. Arayasirikul S, Wilson EC, Raymond HF. Examining the effects of transphobic discrimination and race on HIV risk among transwomen in San Francisco. AIDS Behav 2017 Sep;21(9):2628-2633 [FREE Full text] [CrossRef] [Medline]
    36. Garofalo R, Kuhns LM, Reisner SL, Mimiaga MJ. Behavioral interventions to prevent HIV transmission and acquisition for transgender women: a critical review. J Acquir Immune Defic Syndr 2016 Aug 15;72(Suppl 3):S220-S225 [FREE Full text] [CrossRef] [Medline]
    37. Horvath KJ, Iantaffi A, Swinburne-Romine R, Bockting W. A comparison of mental health, substance use, and sexual risk behaviors between rural and non-rural transgender persons. J Homosex 2014;61(8):1117-1130 [FREE Full text] [CrossRef] [Medline]
    38. LeGrand S, Muessig KE, Pike EC, Baltierra N, Hightow-Weidman LB. If you build it will they come? Addressing social isolation within a technology-based HIV intervention for young black men who have sex with men. AIDS Care 2014;26(9):1194-1200 [FREE Full text] [CrossRef] [Medline]
    39. Sun CJ, Anderson KM, Mayer L, Kuhn T, Klein CH. Findings from formative research to develop a strength-based HIV prevention and sexual health promotion mHealth intervention for transgender women. Transgend Health 2019;4(1):350-358 [FREE Full text] [CrossRef] [Medline]
    40. Schensul JJ, LeCompte MD. Essential Ethnographic Methods: A Mixed-Methods Approach. Second Edition. Lanham, MD: AltaMira Press; 2012.
    41. LeCompte MD, Schensul JJ. Analysis and Interpretation of Ethnographic Data: A Mixed-Methods Approach. Second Edition. Lanham, MD: AltaMira Press; 2012.
    42. Nielsen J. Usability Engineering. San Diego, CA: Academic Press; 1993.
    43. Dumas JS, Redish JC. A Practical Guide to Usability Testing. Second edition. Exeter, UK: Intellect; 1999.
    44. Azar KM, Lesser LI, Laing BY, Stephens J, Aurora MS, Burke LE, et al. Mobile applications for weight management: theory-based content analysis. Am J Prev Med 2013 Nov;45(5):583-589. [CrossRef] [Medline]
    45. Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med 2011 Mar;1(1):53-71 [FREE Full text] [CrossRef] [Medline]
    46. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015 Jan 30;17(1):e30 [FREE Full text] [CrossRef] [Medline]
    47. Tate EB, Spruijt-Metz D, O'Reilly G, Jordan-Marsh M, Gotsis M, Pentz MA, et al. mHealth approaches to child obesity prevention: successes, unique challenges, and next directions. Transl Behav Med 2013 Dec;3(4):406-415 [FREE Full text] [CrossRef] [Medline]
    48. Hekler EB, King AC, Banerjee B, Robinson T, Alonso M, Cirimele J, et al. A Case Study of BSUED: Behavioral Science-informed User Experience Design. In: Proceedings of the Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery; 2011 Presented at: CHI'11; May 7-12, 2011; Vancouver, BC, Canada   URL: http://v1.personalinformatics.org/docs/chi2011/hekler.pdf
    49. Pagoto S, Schneider K, Jojic M, DeBiasse M, Mann D. Evidence-based strategies in weight-loss mobile apps. Am J Prev Med 2013 Nov;45(5):576-582. [CrossRef] [Medline]
    50. Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res 2016 Dec 16;18(12):e317 [FREE Full text] [CrossRef] [Medline]
    51. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015 Feb 26;3(1):e20 [FREE Full text] [CrossRef] [Medline]
    52. Pagoto S, Bennett GG. How behavioral science can advance digital health. Transl Behav Med 2013 Sep;3(3):271-276 [FREE Full text] [CrossRef] [Medline]
    53. O'Mara-Eves A, Brunton G, Oliver S, Kavanagh J, Jamal F, Thomas J. The effectiveness of community engagement in public health interventions for disadvantaged groups: a meta-analysis. BMC Public Health 2015 Feb 12;15:129 [FREE Full text] [CrossRef] [Medline]
    54. Cyril S, Smith BJ, Possamai-Inesedy A, Renzaho AM. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Glob Health Action 2015;8:29842 [FREE Full text] [CrossRef] [Medline]
    55. Centers for Disease Control and Prevention (US). The Principles of Community Engagement. Second Edition. Bethesda, MD: National Institutes of Health; 2011.
    56. Cargo M, Mercer SL. The value and challenges of participatory research: strengthening its practice. Annu Rev Public Health 2008;29:325-350. [CrossRef] [Medline]
    57. Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health 1998;19:173-202. [CrossRef] [Medline]
    58. Dankwa-Mullan I, Rhee KB, Williams K, Sanchez I, Sy FS, Stinson N, et al. The science of eliminating health disparities: summary and analysis of the NIH summit recommendations. Am J Public Health 2010 Apr 1;100(Suppl 1):S12-S18. [CrossRef] [Medline]
    59. Nielsen J. Usability Inspection Methods. In: Proceedings of the Conference Companion on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery; 1994 Presented at: CHI'94; April 24 - 28, 1994; Massachusetts, Boston, USA p. 413-414. [CrossRef]
    60. Gahagan J, Colpitts E. Understanding and measuring LGBTQ pathways to health: a scoping review of strengths-based health promotion approaches in LGBTQ health research. J Homosex 2017;64(1):95-121. [CrossRef] [Medline]
    61. Operario D, Nemoto T. HIV in transgender communities: syndemic dynamics and a need for multicomponent interventions. J Acquir Immune Defic Syndr 2010 Dec;55(Suppl 2):S91-S93 [FREE Full text] [CrossRef] [Medline]
    62. Cahill S, Valadéz R, Ibarrola S. Community-based HIV prevention interventions that combat anti-gay stigma for men who have sex with men and for transgender women. J Public Health Policy 2013 Jan;34(1):69-81. [CrossRef] [Medline]
    63. Reback CJ, Ferlito D, Kisler KA, Fletcher JB. Recruiting, linking, and retaining high-risk transgender women into HIV prevention and care services: an overview of barriers, strategies, and lessons learned. Int J Transgend 2015;16(4):209-221 [FREE Full text] [CrossRef] [Medline]
    64. Brennan J, Kuhns LM, Johnson AK, Belzer M, Wilson EC, Garofalo R, Adolescent Medicine Trials Network for HIV/AIDS Interventions. Syndemic theory and HIV-related risk among young transgender women: the role of multiple, co-occurring health problems and social marginalization. Am J Public Health 2012 Sep;102(9):1751-1757 [FREE Full text] [CrossRef] [Medline]
    65. Mayer KH, Grinsztejn B, El-Sadr WM. Transgender people and HIV prevention: what we know and what we need to know, a call to action. J Acquir Immune Defic Syndr 2016 Aug 15;72(Suppl 3):S207-S209 [FREE Full text] [CrossRef] [Medline]
    66. Hwang W, Salvendy G. Number of people required for usability evaluation. Commun ACM 2010;53(5):130-133. [CrossRef]
    67. Macefield R. How to specify the participant group size for usability studies: a practitioner's guide. J Usability Stud 2009;5(1):34-45 [FREE Full text]

    ‎
    CDC: Centers for Disease Control and Prevention
    EAP: expert advisory panel
    eHealth: electronic health
    LGBTQ: lesbian, gay, bisexual, transgender, and queer
    mHealth: mobile health
    PrEP: pre-exposure prophylaxis

    Edited by G Eysenbach; submitted 14.08.19; peer-reviewed by A Tanner, W Goedel; comments to author 02.10.19; revised version received 22.11.19; accepted 15.12.19; published 12.05.20

    Copyright

    ©Christina J Sun, Kirsten M Anderson, Tamara Kuhn, Liat Mayer, Charles H Klein. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 12.05.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.