We conducted beta testing of JomPrEP to assess its usability and acceptability among MSM living in the Greater Kuala Lumpur region, Malaysia. Beta testing of an app helps to identify any final areas for improvement [31]. We hypothesized that beta testing for 30 days of observation (N=50) would allow us to evaluate the app's design, functionality, and usability. The sample size of 50 was determined based on the pragmatics of recruitment and the need to examine feasibility [32-34].

We partnered with the Centre of Excellence for Research in AIDS (CERiA) at the University of Malaya, Kuala Lumpur, Malaysia, to conduct this study. Working closely with several other local and international institutions, CERiA conducts innovative and interdisciplinary research that combines epidemiological, biomedical, and sociobehavioral approaches, focusing on the implementation of HIV prevention and treatment. As part of JomPrEP integration with existing clinics, we partnered with 2 local clinics—the Red Clinic (private clinic) and the Community Health Care Clinic (nongovernmental organization [NGO]-based clinic)—to provide clinical services (eg, HIV testing, sexually transmitted infection [STI] testing, PrEP services) virtually via the app.

Eligibility criteria included (1) being 18 years or older; (2) identifying as a cis-gender man; (3) self-reporting an HIV-negative or HIV status unknown at screening; (4) not having used PrEP previously (ie, PrEP naive); (5) self-reporting evidence of being at risk for HIV acquisition, as defined by the World Health Organization PrEP clinical guidelines [35]; (6) owning a smartphone; and (7) currently residing in the Greater Kuala Lumpur region.

In total, 50 participants were recruited between March and April 2022 using in-person and online recruitment strategies. For in-person recruitment, flyers were distributed to potential participants as well as posted at local partner organizations (eg, clinics, lesbian, gay, bisexual, transgender [LGBT]–friendly community-based organizations). Additionally, we used various general and MSM-specific social media platforms as venues for participant recruitment. These included placing advertisements in geosocial networking (GSN) apps popular among MSM in Malaysia (ie, Hornet) as well as posting study flyers on Malaysian MSM–focused Facebook pages. Interested individuals who clicked on an advertisement were directed to the study website [36], where they were presented with a brief description of the study and web-based screening.

After meeting enrollment criteria, eligible participants were asked to provide electronic informed consent for study participation, followed by undergoing a baseline assessment. Study staff then assisted enrolled participants with downloading JomPrEP and provided them with brief instructions on the purpose of the app and an overview of how to use it. To restrict access to JomPrEP to the study participants, participants were provided with a single-use registration code needed to gain access to the app. Upon downloading the app, participants were asked to complete an onboarding process, which included creating log-in credentials. They were then redirected to the JomPrEP landing screen (home screen), which contains several icons representing key app functions (Table 1). Screenshots of the app are available in Multimedia Appendix 1. Participants were requested to keep and use the app for 30 days and encouraged to use all app features. On day 30, participants completed a posttest survey and were asked to provide a synthesis of issues regarding the app (ie, exit interviews). For exit interviews, 20 (40%) participants were randomly sampled and interviews were conducted until data saturation was reached. The 1-on-1 sessions were conducted online via licensed videoconferencing software. Participants were given the choice of turning on or off their cameras and were asked to use a pseudonym/nickname.

Participants received point-based rewards (known as JomPrEP points [JPP]) for completing specific activities or meeting milestones via the app (eg, 100 JPP for baseline and follow-up assessment each, 50 for completing app onboarding, 20 for an HIVST in-app order, 50 for completing a lab test, 50 for completing an e-consultation for PrEP, 30 for an in-app PrEP order, 50 for optimal PrEP adherence; maximum points that could be earned: 740 or US $18.50). Participants were allowed to redeem points for cash at any point during the study period (10 JPP=RM 1, or US $0.24).

The Institutional Review Board at the University of Connecticut approved this study (H22-0049), with an institutional reliance agreement with the University of Malaya. Eligible participants provided electronic informed consent for study participation.

All quantitative data were managed and analyzed using IBM SPSS Statistics version 28. Means for continuous variables and frequencies for categorical variables were calculated to describe the participants at baseline. App usability and acceptability were based on descriptive statistics from the app analytics and acceptability measure. For example, evaluation responses are reported as the percentage of users who completed the posttest survey. SUS results are reported as an aggregate score, with a score of ≥50 indicating that the app is acceptable [39,40]; the percentage of participants with scores ≥50 is also reported. Descriptive statistics of app analytics were used to examine app engagement and are reported as the mean with the range for time and action measurements. For qualitative data, all the exit interviews were audio-recorded, transcribed, and analyzed. The comments and issues were grouped and categorized according to common themes relative to specific app functions by 3 coders (including 2 senior coders) and agreed upon by all authors. Dedoose version 9.0.54 was used throughout to assist in data management and analysis.

The mean age of the 50 participants was 27.9 years (range 21-45 years), with most being single (36/50, 72%), Malay (26/50, 52%), university graduates (34/50, 68%) and living in a house/apartment with other people (36/50, 72%). Almost all participants reported having been tested for HIV (49/50, 98%), and 39/50 (78%) participants had done so in the past 6 months. Of all 50 participants, 26 (52%) reported using HIVST and only 5 (10%) had used PrEP previously. Regarding sexual behaviors in the past 6 months, 47/50 (94%) participants reported anal sex with another man, while only 16/50 (32%) participants reported consistent condom use, 4/50 (8%) reported having engaged in sexualized drug use, and 9/50 (18%) reported having engaged in group sex (Table 2).

During the 30-day beta-testing phase of JomPrEP, 42/50 (84%) participants ordered an HIVST kit using the app. Almost all (46/50, 92%) participants used the app to get on PrEP. Specifically, 30/46 (65%) participants chose same-day PrEP versus traditional PrEP, and the majority of them picked up the PrEP medication at the pharmacy (28/46, 61%). Additionally, 44/50 (88%) participants used the online assessment tool to screen for depression, and 39/44 (89%) of them met the criteria for moderate-to-severe depressive symptoms [37] and were provided with a referral letter (Table 3).

During the beta-testing phase, 29/50 (58%) participants were Android users, while the remainder (21/50, 42%) were iOS users. Usability measures by participants included app use, with an average of 8 (SD 5.0, range 2-18) unique visits over 30 days, with an average duration of 28 (SD 38.9) minutes per session. The app had a mean of 34.9 (SD 14.7) daily users, with 939.3 (SD 597.9) daily page views, 63.4 (SD 28.5) daily sessions on average, and consistent returning visits (eg, >10: 29/50, 58%; 6-10: 22/50, 44%).

The mean acceptability score was 73.8 (SD 10.1) on the SUS, well above the minimum criteria (≥50) set for the acceptability of the app [39,40], with all participants reporting acceptability scores of >50. Almost all participants reported that they were satisfied with JomPrEP (46/50, 92%) and that the app was useful in addressing their HIV prevention needs (49/50, 98%); see Table 4.

When participants were asked about future app use, most said they were likely to continue using the app as part of their HIV prevention plan (42/50, 84%), would download the app if publicly available (43/50, 86%), and would recommend the app to their friends or colleagues (50/50, 100%). Most participants felt confident in in-app security (43/50, 86%), including autologout after 5 minutes of inactivity (44/50, 88%), an email and password log-in (42/50, 84%), a 4-digit personal identification number (36/50, 72%), and the app name and icon not associated with HIV (32/50, 64%); see Table 5.

Participants found JomPrEP to be easy to use and felt confident that they would be able to learn how to use it quickly and without technical assistance (Table 6).

In follow-up exit interviews (n=20, 40%), participants indicated a high level of acceptability for the content, interface, and features of JomPrEP. Participants found the app to be user friendly, easy to navigate, and with a good layout. Participants also appreciated the ability to earn reward points for using specific app features, facilitating user engagement and retention.

Participants noted that ordering HIVST kits via the app was straightforward and that the multimedia instructions helped them use HIVST kits and interpret test results.

Furthermore, participants endorsed that the app helped them initiate PrEP use and maintain optimal adherence by facilitating a safe and stigma-free virtual platform to access PrEP services. Participants commented on the relevant information presented in the app, and many noted that they “didn’t know anything about [PrEP] until [they] used the app.”

Participants also provided suggestions to improve the app and specific feedback on additional resources and features that they found interesting and helpful. For example, participants suggested that the app include an option to make an appointment with a mental health counselors and support groups and the ability to connect with other JomPrEP users through private messaging or discussion forums. Participants shared occasional issues with lagging app response time, difficulty setting up reminder notifications, and missing notifications. A few participants indicated that the test result feature of the app was a little challenging to use and required multiple clicks to view the results. A few participants also noted that some of the information in the app is repetitive or is not updated frequently. One participant recommended that the app allow the users to make the app more discreet (eg, the ability to change app icons).

Participants indicated that they would continue to use the JomPrEP app after the final version is released to the public.

Using innovative tools, such as mHealth, in public health programming and the health care system can help bridge gaps in the adoption of needed health and prevention services, particularly among underserved populations [41-44]. In this study, we sought to investigate the usability and acceptability of JomPrEP, a clinic-integrated smartphone app, as an additional platform to promote routine HIV testing and PrEP uptake among MSM in Malaysia. Our findings demonstrated that Malaysian MSM will use a smartphone app to virtually access HIV prevention services and that such an app is acceptable to this at-risk group, as indicated by the participants’ empiric use of the app.

Prior studies have demonstrated several apps for HIV prevention and treatment efforts to be promising and cost-effective strategies to reach and engage stigmatized and hidden populations, such as MSM [25,45-47]. In Malaysia, the use of mobile technology over the past decades has grown markedly, particularly among MSM, with a mobile phone penetration rate of 97.5% and an internet penetration rate of 71.1% [18,19,48]. Importantly, our beta testing of JomPrEP revealed that an overwhelming majority of men used the app to receive HIV prevention services: ordering an HIVST kit (84%) and getting on PrEP (94%). Participants reported that they were satisfied and comfortable using JomPrEP and would recommend it to friends or colleagues. These findings indicate the potential utility of JomPrEP for Malaysian MSM to promote HIV prevention services.

One of the key innovations on JomPrEP includes incorporating on-demand features, such as home-based HIVST, e-consultations, and discrete door-to-door delivery, to provide a scalable model for remote HIV prevention services delivery in the LMIC setting. Although the users would still be required to visit a laboratory for clinical testing, this would not require them to be face-to-face with their clinicians. Moreover, the platform allows users to self-assess their HIV risk, consult online with clinicians from the participant clinics, and have their medication delivered to their preferred location, thus minimizing the need for in-person interactions with the clinician. This represents a significant and much-needed innovation over traditional clinic-based and O2O models of HIV service delivery to keep at-risk individuals wedded into the virtual clinical ecosystem and boost the uptake of clinical services [28,29]. This is particularly important in LMIC settings, such as Malaysia, as the virtual platform allows users to bypass barriers to care for marginalized populations and feel safer and less vulnerable to potential legal or social harm (eg, by reducing face-to-face interactions with providers).

Prior research has documented low user retention and a lack of sustained use after adoption as key challenges to the effectiveness of existing app-based interventions [49]. Obtaining high engagement and retention is necessary to maintain the integrity and long-term sustainability of effective mHealth interventions [50]. Strategies to integrate other features that do not include individual input, such as passive data collection using inputs from their smartphones or unobtrusive wearable devices, may strengthen the features of the app.

Results from our beta testing, however, revealed that MSM were actively engaged in the app and that retention was excellent through the beta testing. Although it is possible that the perfect retention rate could be because of the shorter follow-up time, it is likely that the incorporation of additional components, such as the ability to customize profiles, personalized messages, and gaming elements (ie, the ability to “level up,” earn and redeem points), may have allowed for enhanced user engagement. In recent years, the utility of gamification features (eg, challenges, tasks, rewards, badges, leaderboards) in nontraditional gaming contexts has increased significantly, thus providing opportunities for greater user engagement in mHealth interventions [51,52]. As confirmed in the exit interviews, the overall high engagement and usage of key features of JomPrEP suggest that an app-based intervention, such as JomPrEP, has a high degree of feasibility to ensure equitable access to HIV testing and PrEP services for MSM in Malaysia.

Although there was consensus on the usability and acceptability of JomPrEP, with no significant differences between different subgroups of Malaysian MSM, the app would benefit from continued refinement to address some of the shortcomings identified by men. For example, most participants who completed screening for depressive symptoms (88%) received referral letters to seek care offline (ie, outside the app). Given the focus of JomPrEP to offer holistic HIV prevention services within the online ecosystem, it would be important for the app to incorporate online consultation with mental health counselors and linkage to support groups via the app. Additionally, as part of the continued effort to ensure the safety and security of users, it would be important that JomPrEP incorporate added security measures, including 2-factor authentication and a discreet app icon (DAI). The availability of a DAI allows users to replace the default JomPrEP app log on their phone with another symbol (of their choice). This will help protect users when there is a possibility that someone may accidentally look at users’ phones and recognize that they have an app that might link them to the HIV or lesbian, gay, bisexual, transgender, queer, and others (LGBTQ+) community. A study conducted with MSM of Malaysia also highlighted the importance of privacy and confidentiality features in the mobile apps targeted for HIV prevention and treatment to minimize harm and safeguard users' privacy and confidentiality [53]. Furthermore, it is important that the app be available for users outside of Kuala Lumpur, the capital city, and be linked to both private and government clinics/hospitals. This will ensure widespread implementation of JomPrEP to scale-up HIV prevention services for MSM across Malaysia.

The results of this study should be viewed in the context of the limitations. First, this pilot study included a small sample size and short-term follow-up (ie, 30-days) and used a single-arm design that is commonly used in beta testing; therefore, it was not powered or designed to evaluate efficacy. Second, our participants were subject to selection bias across several dimensions. For example, we recruited men using Facebook or a dating app (ie, Hornet) who may have been more comfortable using mobile apps than other men (ie, hidden MSM). Moreover, participants were already engaged, at least in part, due to their prior high levels of HIV testing. Participants were enrolled in the Greater Kuala Lumpur area only, potentially limiting the generalizability of the findings. Third, social desirability bias may have led participants to speak more positively about their app experience during the survey and exit interviews. This was in part observed by participants who responded favorably to app features, but our usability testing had not confirmed they used the specific feature. Finally, HIVST kits and PrEP services were free to the participants, which may have led to an overestimation of the actual uptake of HIV testing and PrEP services.

Further research is warranted to examine the implementation of JomPrEP in a more real-world setting. Regardless of these limitations, we believe that our findings carry important implications for efforts to improve the uptake of HIV testing and PrEP services among Malaysian MSM using an app-based intervention.

Overall, the JomPrEP app represents a feasible and acceptable tool for Malaysian MSM to access HIV prevention services. Importantly, it incorporates several on-demand features to support the remote delivery of HIV prevention services, thus representing a significant innovation on traditional clinic-based and O2O service delivery models [28,29]. The reported outcomes are promising and indicate the benefits of systematically implementing this platform to foster HIV prevention efforts in LMICs, such as Malaysia, where MSM are disenfranchised from existing prevention efforts [2-4]. A large-scale randomized controlled trial is warranted to establish the efficacy of JomPrEP among this at-risk group.