The University of Virginia Institutional Review Board approved this study. The setting for this study was four Ryan White clinics that pursued PL implementation in 2018. Ryan White clinics provide HIV primary medical care, medications, and essential support services for people living with HIV who are low income, uninsured, and underserved [12]. The clinics implementing PL varied in their location and organizational structure. They included three health system-affiliated Ryan White clinics in Virginia and one in Texas. All four clinics were located in cities, with the two largest clinics serving primarily an urban population and the two remaining clinics reaching a catchment area that included both urban and rural regions. At the time of this evaluation, of the four sites, one ceased participation in PL before enrolling patients due to difficulties with leadership buy-in, one was unable to obtain information security approvals for PL use within the parent health system, and two progressed to the implementation stage and began enrolling patients.

All sites received PL implementation support from the University of Virginia PL implementation team that incorporated evidence-based interventions from an implementation research taxonomy [13]. Implementation support included a comprehensive training package consisting of a training manual, onsite PL training for HIV care team members, a PL learning management system accessible directly from PL’s clinic/provider tool suite, and ongoing and on-demand program support from an experienced PL coordinator.

We anticipated that successful implementation of PL would require processes and infrastructure on multiple levels. At the individual level, PL members (individuals living with HIV) must download and install the app and interact with the app for self-monitoring. At the interpersonal level, PL members communicate with the implementation site–specific PL coordinator and their care team. PL coordinators at the sites also interact with the University of Virginia PL implementation team for training and support. PL workflow must be integrated into the clinic site and processes developed for allocation of phone resources, member enrollment, and support and engagement for members and providers. At the organizational level, the PL program requires leadership approval and information technology (IT) infrastructure to support it.

Purposive sampling was used to recruit employees from the Ryan White clinics that expressed intent to adopt the PL program. Intent to adopt PL was ascertained from direct email inquiries, personal contact with the investigators, or requests for information on the PL website. The PL implementation coordinator at the University of Virginia tracked these clinics in their progression toward PL implementation. Figure 2 displays the stages of the implementation pathways from information exchange when potential sites contact PL to learn about the program to preimplementation after the decision is made to implement PL and then to either implementation or failed to progress. Sites were included in the evaluation when the clinic made the decision to adopt PL (preimplementation). The evaluation team was notified to initiate interview recruitment activities and received contact information for the individual confirming intent to implement PL. This individual was contacted by phone or email and asked to identify primary stakeholders involved with the PL implementation process at the site. The individuals were then approached to participate in the study via email.

We specifically targeted clinic managers, PL coordinators, and PL providers for interviews. PL coordinators were the point people for PL at each site; they were responsible for enrollment and support of PL members. We defined PL providers as physicians, nurses, psychologists, social workers, case managers, community health workers, or any other staff members who were end-users of the PL platform.

An implementation science determinants framework informed the interview guide development and analysis, and it allowed for consideration of the multilevel factors that may contribute to implementation success [14,15]. The CFIR [16] captures a broad range of constructs that fit the goals of this evaluation and provides interview guides and codebook templates that facilitate the application of this framework to evaluation projects [17]. At the time of our evaluation, the CFIR was being used widely in health-related implementation research [18-27] but had a smaller presence in the field of mHealth [28-31].

The CFIR consists of the following five domains: intervention characteristics, outer setting, inner setting, characteristics of individuals, and implementation process. Across the primary domains are 39 specific constructs corresponding to factors of successful implementation [16,17]. The CFIR is designed to be flexible in its use, with users able to determine which constructs are most relevant to their project’s implementation.

Two sources guided the selection of specific CFIR constructs for inclusion in the evaluation. The University of Virginia PL implementation team reviewed and identified all five domains and six priority constructs (innovation source, evidence strength and quality, patient needs and resources, self-efficacy, and engaging opinion leaders) as most relevant to the PL implementation based on their experiences with early adopting sites. Additionally, a 2016 systematic review summarized factors that influence the implementation of all types of eHealth interventions [31]. We used findings from the systematic review to select CFIR constructs most relevant to mHealth interventions. Unlike mHealth interventions described in the report by Ross et al [31], cost was not anticipated to be a barrier to initial PL implementation. All four Ryan White clinics had received grant funding to support their PL programs. In total, 22 CFIR constructs were included in the analysis. The chosen constructs with their PL-specific operational definitions are included in Table 1. We also included an additional code for technology because the mHealth aspect of the intervention was particularly salient to our implementation, and we wanted to be sure that there was a code to quickly capture all data related to this theme.

We developed three interview guides to permit tailoring of questions to clinic managers, PL coordinators, and PL providers. PL coordinators were asked questions about the PL program at two time points (before implementation and during implementation) using the same interview guide. PL providers were asked questions about the PL program during implementation only (ie, after the clinic began enrolling patients in the PL program). Interview guides incorporated questions associated with 22 CFIR constructs (Table 1). For each construct, we began by adapting questions suggested by the CFIR developers [17] and added questions as necessary for our particular project needs.

Recruitment began at all clinics on February 15, 2018. Interviews were completed between March 1, 2018, and July 10, 2019. All participants verbally provided their informed consent to participate and were allowed to discontinue participation at any time after giving consent. Interviews were conducted over the phone and were audio recorded and professionally transcribed. Interview lengths differed based on the guide used, ranging from 30 to 60 minutes. Interviewers were trained members of the evaluation team, and they were not involved with PL implementation at the clinics. This was done to promote candid and honest responses from the interview participants when asked to describe their experiences. Interviews were stored as audio files on a secure drive labeled by study ID number. Individual interviews were not discussed with the entire team to preserve confidentiality.

Eight interviews were completed, including three interviews that occurred during the preimplementation stage and five interviews that occurred after the clinics began enrolling patients in their PL programs. Six of the interviews were contributed by two clinics that successfully implemented PL during the study period. The remaining clinics were unable to implement PL as of June 30, 2019. Each of these clinics contributed a single interview. One of the clinics experienced difficulties in garnering leadership buy-in for PL use. The other clinic was unable to obtain information security approvals for PL use within the parent health system.

Because only two sites progressed to enrolling patients in PL, PL providers were only eligible to participate in interviews at two sites. Of 13 providers we attempted to recruit via email, two declined interviews, one was lost to follow-up, nine did not respond, and one successfully completed the interview.

An analytical template was used by trained personnel (WC, CC, JS, and TEF) using the CFIR codebook [17], which provided operational definitions for CFIR constructs along with example inclusion and exclusion text. We restricted the codebook to those constructs selected for inclusion as described above, with the addition of the technology code as previously described. Transcripts were independently coded by two investigators who then worked together to achieve consensus on the coded content. Coding and analyses were performed using Dedoose Version 8.2.14 [32].

After reaching consensus on the coded content, the two coders independently summarized the barriers and facilitators that emerged from each interview. These summaries also underwent consensus discussions. The summaries were used to generate a master list of PL implementation determinants. Each determinant was listed with its corresponding CFIR construct–specific examples from the interviews and potential action items to report to the implementation team. Facilitators were evaluated by the team to assess whether there were corresponding actions that would enhance or strengthen the facilitator and enable future clinics to benefit from explicit recommended actions.

The master list of PL implementation determinants, including facilitators and barriers, was updated and shared with the implementation team as each determinant summary was completed. This rapid evaluation process enabled iterative changes to the implementation model prior to study completion. The median time from interview completion to determinant summary completion was 131 days and ranged from 36 to 228 days. All information presented to the implementation team was delivered in a deidentified and aggregated manner.

Based on our analyses, nine CFIR constructs were associated with 10 facilitators. A summary of facilitators and recommended action items is shown in Table 2. PL compatibility was an important facilitator to its implementation. Components of fit included alignment between (1) PL and clinic needs, and (2) PL and clinic goals and values. The intersection between patient needs and resources and compatibility constructs was notable. For example, the respondents mentioned that accessing the clinic was a challenge for some of their patients. Having a phone and connectivity to the clinic through PL helped to address this barrier. The following two action items arose from findings related to PL compatibility: (1) remind sites that it is important to budget for cell phones for their most at risk patients and (2) help sites identify their own needs and articulate how PL addresses those needs during preimplementation.

Engaging key stakeholders was also identified as an important facilitator of PL implementation. First, stakeholder engagement with PL implementation activities contributed to leadership and end-user excitement for the program. Second, stakeholder input during implementation planning prepared teams to integrate PL into their existing workflows. Encouraging sites to include clinic staff in implementation planning was identified as an action item.

The planning and engaging innovation participants constructs identified useful strategies for boosting engagement by PL providers and users. Action items included proactive planning for these engagement strategies and timing marketing of PL features that are dependent on group participation. The remaining CFIR constructs (and their associated facilitators) highlighted aspects of the implementation process that were going well and did not require additional action. Examples include accessibility and quality of PL training materials, ease of PL use, and PL’s ability to be adapted to clinic needs. For these facilitators, the evaluation team recommended that the implementation team continue current practices.

Six CFIR constructs were associated with eight common barriers. A summary of those barriers and their recommended action items are shown in Table 3. Barriers associated with PL compatibility were common (n=3) and related to mHealth technology either directly or indirectly. Specific examples expressed by the respondents included not being able to get Wi-Fi in the clinic to download PL on their phones and the release of laboratory results on PL prior to review by the medical provider, which was in opposition to some clinic’s usual workflows.

The remaining barriers were associated with different CFIR constructs. Policy and incentives represented a single barrier related to privacy concerns, both from clients and administrators concerned about regulatory compliance. Issues in implementation planning arose in relation to obtaining institutional approvals for PL use, in part because of unclear internal review processes. Barriers associated with available resources focused on phone availability and staffing effort to manage the PL program. PL’s adaptability by clinics created potential barriers to uptake, with suggestions for allowing tailoring of the PL platform appearance to appeal to unique clinic populations. To address these barriers, the team developed the following action items: (1) create a document that helps sites anticipate potential IT challenges with tips on how to address in advance and (2) provide examples from other sites, including stories or case studies related to how sites addressed common problems and how long their implementation process took.

This work demonstrates the feasibility of applying the CFIR to the evaluation of mHealth implementations. Rapid evaluation methods using a determinants framework were deemed ideal for interim analysis, simultaneously providing for rigorous assessment of PL implementation processes, identification of specific barriers and facilitators of implementation, and timely refinement of our implementation support program [14].

One of the goals of this analysis was to determine if CFIR could adequately capture the important factors in mHealth implementation. We found that the constructs within CFIR are sufficient to pick up mHealth-specific considerations that may impact successful implementation. Although most content captured within our functional technology code was also coded with a CFIR construct, its inclusion in our final codebook enabled more rapid data reduction and analysis to identify instances of delayed or stalled PL uptake due to the clinic’s or health system’s technology-related barriers. Technology played a role in each of the CFIR domains. The addition of a functional code for technology allowed the evaluation team to extract segments of interviews specific to technology aspects of the implementation support program. This enabled us to target our rapid analysis; however, CFIR alone is suitable to cover the concepts arising in mHealth intervention-related interviews.

Preliminary results from the first four PL expansion sites identified compatibility, engaging key stakeholders, and innovation participants and planning as important CFIR constructs associated with early PL implementation. The primary barriers identified in our early results were related to technology and mHealth. mHealth interventions require support from a broad range of stakeholders, including clinic leadership and administration, clinic staff, security and privacy officers, and IT personnel. One recommendation that emerges from this finding is that sites intending to implement an mHealth intervention should engage their security and technology leaders and staff early in the process to streamline implementation and help to avoid or minimize technology-related barriers, such as multiple levels of review and permissions that can be required by large systems. Further, mHealth interventions are unique in their need for continuous updates following their initial implementation. Unlike more discrete interventions, mHealth interventions are not complete once they are implemented but instead require ongoing technological maintenance and support. We identified a useful application of the CFIR for identifying barriers and facilitators at PL sites while PL implementation processes were underway. This enabled us to provide specific action items for the PL implementation team, and it resulted in iterative refinement of our PL implementation strategy. The evaluation was successful in providing interim CFIR-informed feedback to implementation stakeholders rather than waiting for the end of implementation to assess its success. This model could be useful in other implementations that occur on a rolling or ongoing basis.

The results of our evaluation led to revisions in a detailed implementation manual that is provided to new expansion sites. This manual includes information about the PL program itself, including a detailed description of all app features and components of the web portal. Revisions to the implementation manual included (1) a description of important milestones in the implementation process including developing a budget and IT flows, (2) information about gathering necessary security and privacy approvals, and (3) phone logistics. This manual is essential to the successful implementation of PL at new sites and is continually updated based on feedback learned from the implementation interviews using the rapid feedback approach taken by the evaluation and implementation teams.

Rapid evaluation was critical to providing timely feedback to the implementation team and expansion sites. Because the implementation team is continuously implementing PL and working to meet the immediate needs of expansion sites, it is important to plan regular communication between the evaluation and implementation teams. The frequency of communication should be based on timing of interviews and the emergence of new sites. Both teams should establish an integrated flow that allows for two-way channels of communication from the implementation team to the evaluators regarding the status of pending sites, as well as from the evaluators to the implementers regarding recommended changes to the implementation process.

The results of this rapid interim evaluation suggest that health systems looking to adopt new mHealth apps to improve patient engagement and outcomes will need to consider adopting a streamlined approach to decision making and IT infrastructure, including security, to have successful mHealth implementation. Our findings are consistent with other recent evaluations of mHealth interventions [33,34] that used CFIR to identify factors influencing implementation, including characteristics of the innovations themselves, as well as the local and institutional contexts in which they are being adopted. Scale-up of technological innovation is a challenging process and can be inhibited by organizational factors external to the intervention.

Future work will include incorporating interviews with patients who represent another important stakeholder group in the implementation of mHealth. Additional next steps will include incorporating the CFIR valence attribution as the number of sites increase and we become able to identify the positive and negative attributions of each implementation.

One of the main limitations of this evaluation was the lack of provider interviews. Our provider recruitment experience echoes others in the implementation literature, requiring perseverance of our PL evaluation team and considerably longer recruitment times than predicted [35]. Providers represent a key stakeholder group in mHealth chronic disease interventions, and their participation in the evaluation of such interventions is critical. For interventions like PL that are designed to improve connections between users with chronic medical conditions and members of their health care teams, intervention effectiveness is dependent on provider uptake. Engaging providers in evaluations of PL implementation is essential for the recognition of and response to the unique barriers to PL use.

This study describes the use of the CFIR to guide iterative refinement of an implementation strategy to facilitate dissemination of our mHealth intervention. Our findings highlight the unique characteristics of mHealth interventions and the multilevel factors that must be considered when planning for their implementation in health care settings. The flexibility and comprehensiveness of CFIR appear to be sufficient to capture concepts within the interviews that we conducted and likely would be applicable to the evaluation of other mHealth interventions. Strategies for rapid evaluation may be particularly important in the realm of mHealth, where the field can move quickly. Rapid evaluation methods that are rigorous and responsive to the experiences of early mHealth adopters can better inform best practices for mHealth implementation. Increasing provider feedback will also enable more impactful and utility-focused evaluation.