Web-based trial recruitment, enrollment, and data collection are increasingly common in research, particularly those focused on the use of mobile health (mHealth) apps. The benefits of using web-based methods include faster and cheaper recruitment, particularly in rural areas [1]. Traditional in-person recruitment strategies are more complicated with adolescent men who have sex with men as they are not able to frequent bars and may not attend gay pride events and other common locations for recruitment in studies of men who have sex with men. This, coupled with the common use of social media apps in this age group, makes web-based recruiting efficient and more generalizable. mHealth apps hold promise to increase the provision of prevention services [2,3] and to reach populations such as adolescent men who have sex with men and rural men who have sex with men who may face interpersonal and structural barriers to seeking in-person prevention services [4-6]. These apps and the use of web-based study methods can be particularly useful for adolescent men who have sex with men residing in rural communities who may not have shared their sexuality with their family and friends and where access to services is challenging. The distribution of these apps through trials is more widespread on the internet, and these apps increase the scale and reach of prevention services, including HIV testing and prevention counseling, pre-exposure prophylaxis, condom distribution, and education, all of which are required to decrease incidence rates [7]. However, web-based trials increase the need for careful scrutiny of forms of fraudulent activity both as an issue of data quality (ie, multiple entries and ineligible participants providing inaccurate age to participate) and as an issue of protection for the adolescent men who have sex with men enrolling in the study. It is critical to ensure that the data collected through this modality are valid and reliable [8].

Authentication for fully web-based studies requires a multimodal approach of electronic and manual verification that may require substantial effort compared with traditional in-person studies [9-11]. There are many examples in the literature on the frequency of fraud in web-based studies. In 2019, Ballard et al [12] categorized 28.7% of their web-based surveys as fraudulent and another 10.1% as potentially fraudulent. In addition, a fully web-based youth-specific HIV study in 2008 identified 675 persons suspected of fraudulent enrollment through multimodal processes [9], and an adolescent men who have sex with men–specific survey published in 2013 found 559 fraudulent cases [13]. Two analyses in 2020 examining issues with web-based recruitment in men who have sex with men concluded that fraud was common, that manual methods work but are resource intensive, and that additional research should be completed to find affordable methods to limit fraudulent enrollment in studies [14,15].

Trials of mHealth tools for HIV prevention pose unique challenges, including the authentication of potential participants and the prevention of fraudulent attempts to enroll in studies [16,17]. This study describes the authentication and fraud prevention protocols used in the iReach project, a randomized controlled trial (RCT) of a multilevel life skills intervention that uses mobile apps to reduce vulnerability among men adolescent men who have sex with men [18]. We present the multistep validation process for this web-based adolescent trial, which included electronic programmed comparisons; the use of a manual checklist; and fraud detection methods, including social media. We describe the application of these steps in the trial and provide examples and metrics for the more common types of fraudulent activities, including a brief case scenario for illustration. The strategies used could benefit others who are working on recruitment and enrollment in web-based studies.

Methods for conducting the ongoing iReach trial have been described elsewhere [18]. In brief, the trial aimed to explore the efficacy of a multilevel life skills intervention delivered through a web app to 499 adolescents (aged 13-18 years), same-sex attracted, cisgender males (people assigned male at birth who were identified as men) in four US regions, and an additional 101 adolescent men who have sex with men nationally. The participants were a racially and ethnically diverse sample with at least 50% (300/600) identity as people of color or from rural communities. After enrollment, eligible participants assigned to the experimental arm had access to the iReach web app over 12 months of the study. Within the web app, they had access to activity-based life skills modules across 14 key life areas, set goals, monitor progress toward these goals, work on these goals using the peer mentor video chat feature, and access to a locator feature to find community resources that are welcoming to lesbian, gay, bisexual, transgender, queer, questioning, and other sexual and gender minority individuals to support a healthy life and achieve their goals. Participants randomized to the control arm of the trial had access to the locator feature of the intervention due to adolescent men who have sex with men’s vulnerability to HIV and sexually transmitted infections. At the end of the 12-month period, participants in the control arm were given full access to the iReach web app for 3 additional months. The primary outcomes of the study were cognitive factors linked to the ability to use HIV prevention and behavioral intentions to use HIV prevention. Participants received a US $30 Amazon gift card for surveys at baseline and 12-month follow-up; control participants received an additional US $30 gift card for a 15-month follow-up. Participants received a US $25 Amazon gift card for the 3-, 6-, and 9-month follow-up surveys. The University of Pennsylvania Institutional Review Board (IRB) served as the IRB of the record for this study. In accordance with the National Institutes of Health Common Rule, the University of Michigan and Emory University IRBs entered IRB authorization agreements with the University of Pennsylvania IRB. A waiver of parental consent was obtained for minor participants under the age of 18 years.

Recruitment was primarily completed through demographically targeted banner advertisements on social media platforms (ie, Snapchat and Facebook). Additional community engagement using print and media advertisements supplemented the social media advertisements with a link to screening eligibility on the web. All recruitment advertisements asked young men to help test a new health app and included photos of racially and ethnically diverse young men and a link to follow for eligibility.

Interested participants who clicked on the banner ads or accessed the screening survey link were screened for eligibility on the web and completed an electronic informed consent process [19].

Of the 19,709 visitors to the iReach project screening survey, 13,931 (70.68%) completed the screening survey. Of these, 23.35% (3253/13,931) were eligible for the study. Of the 3253 visitors who were deemed eligible, 2544 (78.2%) consented to participate in the study. After excluding potential participants who did not provide contact information or who failed the 3-digit phone verification, 92.44% (1224/1324) started the baseline questionnaire. Figure 1 shows a flow diagram of enrollment based on the responses of potential participants.

Upon completion of the baseline questionnaire, responses were verified by automatic electronic authentication using the SAS program. This automatic authentication was completed for potential participants in bulk every Monday, Wednesday, and Friday. Table 1 demonstrates that although 1224 participants started the baseline questionnaire, 492 were excluded because of SAS-programmed automatic authentication failures, most of which (n=252) excluded at this stage were excluded because they did not complete 60% (17/27) of the random subset of questions or 70% (44/62) of primary outcome questions. As iReach is a regionally bound RCT for HIV seronegative youth, IP addresses outside the United States were excluded from moving forward (n=8). In addition, several duplications of participants (n=177) who already existed within the study or the SMART participant management system were excluded. The remaining participants were referred for manual verification.

Tables 2 and 3 show the verification and failure of potential participants referred for manual authentication, respectively. The manual review process was used to exclude 132 individuals (Multimedia Appendix 1). As several areas of data were assessed for irregularities before exclusion, the causes for exclusion in this stage were not mutually exclusive but are noted in Table 3.

A total of 132 potential participants failed manual authentication during manual checklist completion. The reasons for failure during manual authentication varied and could overlap if potential participants had multiple issues with the data they provided. The most common reasons for manual authentication failure were a time stamp failure (n=33) or suspicious survey response patterns (n=33). A time stamp failure occurred when potential participants had unusually short or unusually long completion times for their screener survey or baseline questionnaire. A suspicious survey response pattern occurred when the potential participant provided conflicting responses or responded in a pattern (eg, selecting the same response for every answer). Potential participants were also excluded if manual authentication revealed that they did not meet eligibility age requirements for the study, if their social media profile or profiles (if provided) revealed that the potential participant was not who they said they were (ie, did not meet eligibility requirements for age, gender, and location), or if it was discovered that they were duplicates who managed to pass through the automatic authentication process. One specific cluster of 23 potentially fraudulent enrollments was highlighted in the case study provided.

While manually verifying a potential participant for the iReach project, a study team member noted that the zip code provided by the potential participant did not match the SMART-derived zip code (note that the street name has been changed). This flagged an electronic review of the participant data that found the address provided was a partial match with an already-enrolled participant and differed only by house number. In addition, the contact phone number provided by the potential participant was the secondary phone number of the currently enrolled participants with the same street name.

A study team member attempted to contact the potential participant to ask for additional verification to ensure that the potential participant could be authenticated. When the potential participant did not respond after multiple attempts at contact, they were not enrolled in the study. The previously enrolled participants with a similar address and phone number were sent a message that attempts to duplicate enrollment would be grounds for dismissal from the study.

Twenty-three additional attempts to enroll in the study came from different house numbers on Wisteria Street. Owing to the electronic and manual verification systems in place in the study, multiple attempts for duplication of enrollment were identified and prevented. The enrolled participants were contacted and advised that they were being removed from the study. This example demonstrates the effectiveness of using electronic strategies that include both exact matches and partial matches for addresses to discover potential fraud cases.

This study examines methods for authenticating unique participants for the iReach project, a fully web-based RCT multilevel skills intervention for HIV prevention with adolescents, a population that is easier to enroll on the web. Using a multimodal approach to authentication, 624 potential participants were excluded from enrollment, including those who attempted to enroll more than once. Most participants were excluded by automated data reviews, with a smaller number requiring manual authentication by the study staff.

Over the past 15 years, researchers have acknowledged the threat that potentially fraudulent cases can have on the internal and external validity of their trials [8,9,17]. Several of the authentication checks used in the iReach project mirror those of previous studies, including the electronic verification strategies of IP addresses, the comparison of demographic information at multiple time points (ie, age), and survey time stamp review. Similar to other human verification strategies, manual reviews that look for patterns of illogical responses and requests for the submission of additional proof of eligibility were also included in iReach. The iReach project has also extended previously reported verification processes by improving the methods of possible fraud detection, particularly through the use of electronic authentication strategies. By using automated systems to identify exact and partial matches of demographic information (eg, name and email address) and the SMART participant management system to verify addresses based on GPS locations, many possibly fraudulent cases were eliminated before manual authentication approaches were engaged. Furthermore, the completion thresholds used for all questions and primary outcome questions—60% (17/27) and 70% (44/62), respectively—reduced the number of possibly fraudulent cases that underwent manual authentication. An additional highlight of the iReach project approach was the use of social media for verification. Given that estimates of social media use among youth are as high as 97% [21], using social media data can increase the sensitivity of detecting possible fraudulent enrollments, and some researchers have gone a step further by asking participants to provide a current selfie to match to social media profiles, as described by Bonar et al [22]. Although not a requirement for this study, most participants provided a social media profile. Finally, the use of a single report provided to the study team that detailed possible inaccuracies offered an efficient checklist to ensure a systematic approach to manual authentication.

It is important to consider how best to characterize the sensitivity and specificity of fraud detection systems. Enrolling fraudulent participants introduces bias into the data, and the detection consumes resources. When working with youth, authentication can limit the potential harm of youth inadvertently interacting with fraudulent accounts or nonminors. However, there is a balance needed in fraud detection, as we strive to include a diversity of participants in the studies. Automated electronic fraud detection methods have the potential to introduce selection bias, as it is plausible that residents of high-density housing developments are at higher risk of being classified as potentially fraudulent based on the similarity of addresses or shared IP addresses than residents of single-family dwellings. This could also be true for participants in college, given that this study included those aged 18 years. A fraud detection system that is only automated might make 1 determination, whereas a fraud detection system that uses both automated and manual verification will be more likely to uncover the reasons for the similarities. Furthermore, 22.8% (28/123) of potential participants who failed manual authentication were discovered to be duplicates, even though they had avoided detection during the initial automatic authentication process, which demonstrates the importance of a combination of both automated and manual authentication for data quality. Potential iReach participants whose data indicated a need for manual review were not automatically or always excluded due to the manual check if staff found explanations for the issue that triggered the manual review. This included participants with long completion times for the baseline questionnaire or those with an IP address that was different from their state address, often due to travel or moving. Although manual reviews certainly take longer and are more time consuming for staff, the targeted use of manual review triggered by the automated review of data resolved issues and inconsistencies by contacting participants. As suggested by Bauermeister et al [23] and Ballard et al [12], this focused manual review allows opportunities to resolve more nuanced situations and improve the specificity of the fraud detection algorithm and keep these participants in the study if the manual review is passed. Whenever possible, study teams strive to strike a balance between automatic and manual authentication strategies to produce the highest quality of research.

Research teams recruiting on the web should be vigilant to maintain scientific rigor in the methods of recruitment and retention. As technology continues to advance, researchers should periodically update methods to ensure the authenticity and uniqueness of participants. Reviews of the design and implementation of electronic and manual strategies for authentication should be performed periodically to ensure that the validity of the study sample is maintained. The careful construction of ways to avert fraud in the design stages can help prepare research teams for unanticipated challenges within this environment, save time and money with detection efforts, and preserve data quality.