This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Digital tools accessed via smartphones can promote chronic condition management, reduce disparities in health care and hospital readmissions, and improve quality of life. However, whether digital care strategies can be implemented successfully on a large scale with traditionally underserved populations remains uncertain.
As part of a randomized trial comparing care delivery strategies for Medicaid and Medicare-Medicaid beneficiaries with multiple chronic conditions, our stakeholders identified implementation challenges, and we developed stakeholder-driven adaptions to improve a digitally delivered care management strategy (high-tech care).
We used 4 mechanisms (study support log, Patient Partners Work Group log, case interview log, and implementation meeting minutes) to capture stakeholder feedback about technology-related challenges and solutions from 9 patient partners, 129 participants, and 32 care managers and used these data to develop and implement solutions. To assess the impact, we analyzed high-tech care exit surveys and intervention engagement outcomes (video visits and condition-specific text message check-ins sent at varying intervals) before and after each solution was implemented.
Challenges centered around 2 themes: difficulty using both smartphones and high-tech care components and difficulty using high-tech care components due to connectivity issues. To respond to the first theme’s challenges, we devised 3 solutions: tech visits (eg, in-person technology support visits), tech packet (eg, participant-facing technology user guide), and tailored condition-specific text message check-ins. During the first 20 months of implementation, 73 participants received at least one tech visit. We observed a 15% increase in video call completion for participants with data before and after the tech visit (n=25) and a 7% increase in check-in completion for participants with data before and after the tech visit (n=59). Of the 379 participants given a tech packet, 179 completed care during this timeframe and were eligible for an exit survey. Of the survey respondents, 76% (73/96) found the tech packet helpful and 64% (62/96) actively used it during care. To support condition-specific text message check-in completion, we allowed for adaption of day and/or time of the text message with 31 participants changing the time they received check-ins and change in standard biometric settings with 13 physicians requesting personalized settings for participants. To respond to the second theme’s challenges, tech visits or phone calls were made to demonstrate how to use a smartphone to connect or disconnect from the internet, to schedule video calls, or for condition-specific text message check-ins in a location with broadband/internet.
Having structured stakeholder feedback mechanisms is key to identify challenges and solutions to digital care engagement. Creating flexible and scalable solutions to technology-related challenges will increase equity in accessing digital care and support more effective engagement of chronically ill populations in the use of these digital care tools.
ClinicalTrials.gov NCT03451630; https://clinicaltrials.gov/ct2/show/NCT03451630.
Over the last 10 years, ownership and use of smartphones has more than doubled in the United States, from 35% to 81% of the population [
Digital care tools have a growing evidence base, including evidence supporting the effectiveness of such technologies for patients managing chronic conditions. For example, these tools can support individuals with diabetes in lowering hemoglobin A1c levels, improve quality of life and lower number of hospital readmissions for individuals with heart failure [
Despite increased smartphone use and evidence supporting the benefits of digital care strategies such as remote patient monitoring, biosensors, and wearable devices, barriers to care are ever-present when implementing digital tools on a large scale. Notably, there is a lack of research on the challenges that occur when implementing digital care with traditionally underserved populations and those with high-burden, high-cost medical conditions [
Overcoming these challenges to realize the full potential of digital care to support the management of chronic conditions and reduce health disparities requires an iterative development approach that includes ongoing consumer and community stakeholder input. We are conducting a large-scale, randomized controlled trial comparing the effectiveness of 3 care management strategies (ie, high-touch, high-tech, and usual care) delivered by a commercial insurance organization for adult Medicaid and Medicare-Medicaid beneficiaries living with multiple chronic conditions. To address the unique needs and challenges experienced by this population and to ensure that our digital interventions are patient-centered and pragmatic, we describe early implementation challenges and our stakeholder-driven process adaptions specific to the digitally delivered chronic disease care management strategy (high-tech care).
The 3 study comparators are approaches to care delivered by a chronic disease care management program and incorporate fundamental, evidence-based components of integrated care models including interdisciplinary care management [
All study participants work, one on one, with a care manager (ie, nurse, social worker, and licensed professional counselor) to create individualized care plans [
At the onset of implementation, the study team was acutely aware of the need for continued stakeholder engagement and feedback to promote effective high-tech care implementation. On the basis of the Patient-Centered Outcome Engagement principles [
Using information from these 3 feedback mechanisms, we developed solutions to the identified technology-related challenges. Solutions were discussed, refined, and implemented with input from the study team, care managers, and clinical leadership. Solutions were also vetted through the Patient Partners Work Group. A work group of patient partners, who have similar characteristics and lived experiences similar to those experienced by our study population, was established through a collaboration with the National Alliance on Mental Illness Southwestern Pennsylvania’s Consumer Action Response Team. Patient partner feedback was tracked in the Patient Partners Work Group log (
Data source and information collected and provided:
Study support log
Study team created a study-specific, toll-free, hotline staffed during office hours
Hotline supports care managers and participants with study-related questions or challenges
Implementation meeting minutes
Study team meets with clinical leadership weekly and meets monthly with all care managers
Meetings provide a time and space for care managers and their clinical leadership to voice implementation challenges and to strategize potential solutions
Case interview log
Semistructured interviews were conducted with care managers to identify technology-related challenges, participants experience, and workflow impacts
Patient Partners Work Group log
Study team meets regularly with the work group to discuss high-tech care implementation
The work group provides feedback on materials and solutions supporting high-tech care engagement/implementation
In order to assess a change in participants’ abilities to overcome technology-related challenges, we analyzed intervention engagement outcomes that may have been impacted by stakeholder-driven implementation solutions. We reviewed the following 3 sources of engagement data pertaining to care activity from April 23, 2018, to December 31, 2019: (1) the participants’ ability to complete a video visit as defined by answering the video call from their care manager, (2) the participants’ ability to answer condition-specific text message check-ins, as defined by receiving a check-in via text message and submitting all answers to condition-specific questions, and (3) participant responses to exit survey questions sent via the remote monitoring platform. All pre- and postdata presented are based on the first in-person technology support visit completed by the participant.
Stakeholders reported challenges centered around 2 major themes: (1) difficulties using basic functionalities of the smartphone and high-tech care components and (2) difficulties using high-tech care components due to cellular reception and internet connectivity issues. Approximately 500 study hotline phone calls, about technology-specific challenges, were made by 129 participants and 32 care managers to the study team, and the calls were tracked in the study support log. Feedback was also provided during clinical leadership meetings (n=82), monthly care management staff meetings (n=16, tracked via implementation meeting minutes), and semistructured interviews with care managers (n=4, tracked via case interview log). For each thematic challenge, we present: (1) specific stakeholder feedback that leads to solution development, (2) the description of the stakeholder-driven solutions as they are a direct result of stakeholder feedback, and (3) changes in participant engagement data after solution implementation.
With support from care managers and clinical leadership, the study team focused on common functionality challenges experienced by high-tech care participants and devised 3 main solutions: (1) tech visits (technology support visits), (2) a tech packet (participant-facing technology user guide), and (3) tailored condition-specific text message check-in.
Sources of information and solutions: utilizing functionalities of the smartphone and high-tech components.
Data source | Information provided | Informed solution |
Case interview log and study support log |
Care managers concerned about time spent teaching participants basic smartphone functionalities Specific technology challenges faced by participants include screen pressure difficulties, home screen navigation, phone charging, text message access, including opening condition-specific check-ins, and phone volume manipulation |
Tech visits and tech packet Tech packet |
Implementation meeting minutes |
Clinical leadership interpreted technology-education time concerns as a workflow issue in which care managers had to make up time to ensure participant clinical care needs were met |
Tech visits and tech packet |
Study support log |
Smartphone factory resets and reconfiguration were time consuming for care managers Smartphone volume manipulation and battery power were 2 participant challenges that resulted in missed high-tech care video visits and condition-specific check-ins Check-in assignments were sometimes automatically scheduled at inconvenient times for participants (ie, work, school or sleeping hours) Biometric check-in settings were automatically standardized for each participant |
Tech visits and tech packet Tech packet Tailored condition-specific check-ins Tailored condition-specific check-ins |
Tech visits are structured to allow a study team member to assist participants with time-consuming digital literacy challenges, either at the participant’s home or at a community location. Tech visits do not replace initial face-to-face training that care managers provide to high-tech care participants; rather, it is a form of supplemental training to ensure that participants are able to use their smartphone to receive care and to reduce the time care managers spend on high-tech care training during the initial appointment. Participants are selected for tech visits if (1) they have called the study hotline multiple times with issues that could not be completely resolved, (2) a care manager is unable to connect with the member due to technology challenges, and (3) clinical supervisors believe a participant’s level of digital literacy requires a substantial amount of care manager’s time.
During the first 20 months of implementation, 73 participants received at least one tech visit. Before the tech visit, 23% (17/73) of the participants completed a video call with their care manager. Within 30 days of a tech visit, the average rate for video call completion increased to 51% (n=33). In total, 21 members, who had never been able to connect with their care manager via video calls, before the tech visit completed a video call after receiving the tech support. Of the 73 participants, 25 (34%) had video call data for both before and after the tech visit. For these participants, the average rate of completed calls increased by 15% after the completion of a tech visit (
Video call data for participants with completed pre- and postdata. CM: care manager.
Specific to condition-specific check-ins sent via text message, most participants completed at least one check-in on their own before their first tech visit (60/73, 82%), with an average rate of 36% (n=59). Within 30 days of a tech visit, the average rate for engagement with check-ins increased to 43% (n=59). Two participants completed check-ins after receiving support, who had never completed a check-in prior. Conversely, 15 participants completed check-ins before receiving support but never completed a check-in after the visit. Of the 73 participants, 59 (81%) had check-in data both before and after the tech visit. For these participants, the average rate of completed check-ins increased by 7% after the completion of a tech visit (
Check-in data for participants with completed pre- and postdata.
The participant-facing technology user guide (tech packet) is an educational resource for participants learning how to use the smartphone and high-tech care digital components. The tech packet outlines key smartphone functions (ie, how to answer a phone call, how to navigate to the
Tech packet additions based on study support log.
Patient Partners Work Group technology user guide feedback.
Work group feedback | Examples of feedback | Feedback incorporated |
Use less abbreviations/ jargon as these are difficult to follow |
App has many meanings to different participants Remote monitoring may have a negative connotation for participants |
Addition of definitions page and spelling out abbreviations: app to application Replacement of technical jargon: Check-ins with your care manager instead of remote monitoring |
Provide an easy |
Each section begins on the smartphone home screen, which assumes participants can find the home screen |
Addition of instructions on how to navigate to home screen at the beginning and end of each section |
Dexterity and pressure difficulties may be a concern |
Press, click, touch, and open were used interchangeably |
Used touch for screen actions and press for the home screen button to distinguish amount of pressure to be applied |
Highlight important contact information associated with care management and study activities |
Participants may not be able to distinguish who is sending a text message, and some may be concerned about the legitimacy of messages |
Stated explicitly messages from the remote monitoring platform come from the same phone number each time and make the number visible on all sections Displayed the study hotline number frequently |
Participants can be difficult to reach |
Participants may not be reached during business hours |
Included a section on how to set up and check a voicemail box |
Tech packet updates pre- and postpatient Partners Work Group feedback.
We distributed the tech packets to 379 participants. Of the individuals who received a packet, 179 participants were eligible (eg, completed high-tech care before January 1, 2020) to receive the high-tech care exit survey. Of the 179 participants, 96 responded to the survey; 73 respondents strongly agreed or agreed that the tech packet was useful and 62 actively used the guide at least 1 to 2 times a week during care. See
High-tech care exit survey (96/179, 53.6%).
Not all challenges presented to the study team were solved through tech visits and the tech packet. To support participants in completing their condition-specific check-ins, the study team adapted their workflow to allow for a modification of when check-ins are assigned (ie, day of the week and time) based on participant preference. In total, 31 participants requested to change the day/time when check-in text messages were to be received. Second, as all biometric check-in settings (eg, normal boundary parameters for pulse, blood glucose level, weight, and blood pressure) were standardized across participants by default, the study team allowed individual check-in settings to be modified based on the agreement of a participant’s primary medical provider. For 13 participants, the primary medical provider requested biometric setting changes to reflect the participant’s
Check-in assignments modified based on participant preferences
Participant rescheduled their diabetes check-in for the morning, based on when their medical provider had instructed them to check their glucose
Participant rescheduled their check-in assignment to the day they are off work
Biometric settings modified based on provider preferences
Primary medical provider verified that the participant takes glucose readings before taking insulin and requested setting alerts to be set at 270 or above
Primary medical provider approved to change a participant’s blood pressure settings; allowing notification to only send to the care manager when the participant is out of their expected range >170/100 or <90/60
It was reported, by both care managers and participants themselves, that participants were having difficulties using high-tech care digital components (eg, video visits and condition-specific check-ins) due to limited cellular reception and internet connectivity. Depending on the circumstances, solutions include (1) participant education on how to connect a smartphone to the internet, (2) schedule video visits or condition-specific check-ins when the participant is in a location with cellular service or internet, and (3) participant education on how to disconnect from the internet.
Sources of information and solutions: understanding limited cellular reception and intervention connectivity.
Data source | Information provided | Informed solution |
Study support log and implementation meeting minutes |
Server error messages reported by participants preventing check-in messages from opening |
Education on connecting a smartphone to the internet |
Study support log and implementation meeting minutes |
No internet at the home preventing digital tool use No cellular reception at home prevented digital tool use |
Scheduling video visits/check-ins when participant has access to cellular service or internet |
Study support log and implementation meeting minutes |
Poor internet service at home chosen as default connection method prevented digital tool use Poor internet service in community chosen as default connection method prevented digital tool use |
Education on disconnecting from the internet |
Care managers communicate with each participant regarding the best way to provide care, given limited cellular reception and internet connectivity. All 3 connectivity solutions are addressed via a tech visit or phone call. Providing education on connecting a smartphone to the internet is often a solution for participants receiving a
This paper highlights the impact of stakeholder-driven solutions on early implementation challenges specific to a digital care strategy. To support participants in engaging with their smartphone and high-tech care digital components (theme 1 challenge), 3 main solutions were implemented (eg, in-person technology support visits, participant-facing technology user guides, and tailored condition-specific check-ins). For participants who received an in-person technology support visit, we saw an overall increase in engagement with video calls and condition-specific check-ins. For participants who received the participant-facing technology user guide and completed both high-tech care and the exit survey, we found that most used the tech packet while receiving care and or believed it was useful. Finally, condition-specific check-ins were tailored for participants to support engagement and meet their primary medical provider’s care goals. To support participants experiencing difficulties engaging in high-tech care due to limited cellular reception or internet connectivity (theme 2 challenges), 3 solutions were devised and implemented as needed to support engagement in the program (eg, education on both connecting and disconnecting a smartphone to the internet, scheduling high-tech care video visits or condition-specific check-ins at times when the participant is in a physical location that allows connectivity to occur).
Our findings suggest that concurrent stakeholder feedback has the potential to increase implementation success; therefore, it is pivotal to provide stakeholders with multiple and continuous avenues for communicating challenges to the study team. Furthermore, the results stress the importance of working collaboratively with stakeholders early in the implementation of digital interventions to design scalable solutions such as educational materials (tech packet) and activities (tech visits and telephonic support) and refine condition-specific check-ins that suit the specific needs of the patient and their primary medical provider. Moreover, although measuring the success of solutions created during implementation is not always preplanned, early results indicate positive changes in participant technology engagement after tech visits are implemented. Our positive trends in engagement highlight the need for earlier identification of patients who require tech visits to promote early engagement, reduce demoralization, and potentially achieve earlier clinical benefits. Understanding the nuanced challenges of delivering interventions and engaging patients—as well as how to create effective solutions—will advance the efficiency and reach of digital care.
Current digital care literature focuses on either how tool engagement impacts desired health outcome(s) [
Our work is an important addition to the discussion on digital care provision as we provide a systematic framework for how digital care providers can work with stakeholder groups to identify and address care delivery implementation challenges. Although most research on remote patient monitoring focuses on single-diagnosis care for older Americans, our intervention targets adults aged 21 years and older who are managing multiple chronic physical and or behavioral health conditions [
Finally, as participants in this research are exclusively eligible for Medicaid, our work promises to reduce health disparities by improving access to digitally delivered evidence-based care management for low-income patients. Supporting consumer adoption of digital health tools is one way to both support patients in their management of chronic conditions and the ethical imperative of reducing health disparities. However, realizing the full potential of digital tools to positively impact health disparities requires continued work on understanding the ways to best support traditionally underserved populations to use digital tools (and how to design those tools and their implementation protocols to meet diverse consumer needs).
Although this study contributes novel stakeholder-driven solutions to stakeholder-reported implementation challenges that affect participants’ engagement in a digital care intervention, there are several limitations. Data collected via the high-tech care exit survey may be limited due to a nonresponse bias, as participants who completed the survey had to be able to access the check-in and be willing to complete the survey after completing all care goals. However, our current response rate of 54% indicates that this bias may be less salient [
To better understand digital care barriers specific to a patient population or care program, it is critical to develop and employ methods for obtaining feedback from key stakeholders before and during implementation. Key stakeholders may include care providers and implementation teams, digital tool creators, and community organizations that represent the population of interest and patients. Our stakeholder-informed solutions include in-person tech visits, a tech packet detailing how to access and use key technology components, along with the tailoring of digital care components to meet both patient and provider needs. Using high-tech care exit survey responses and remote monitoring engagement data, we measured the impact of our solutions and continued to improve high-tech care delivery. As solutions to challenges develop, detailed tracking of their implementation may positively impact patient engagement with digital tools and ultimately show increased participation in care resulting in improved health outcomes and reductions in health outcome disparities.
Patient-Centered Outcomes Research Institute
The authors thank all care managers, clinical leadership, Patient Partners Work Group, Stakeholder Advisory Board, and our study participants for their collaboration and input into each stage of the study thus far and for their continued commitment to this research. The authors also thank the study team for their dedication and preserving adaptability. ClinicalTrials.gov identifier: NCT03451630. This study was partially funded by the Patient-Centered Outcomes Research Institute (PCORI). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee. The participation of JS in the research project occurred before JS’s appointment to the Board of Governors of the PCORI (R).
JK has received medication supplies for investigator-initiated trials from Pfizer and Indivior. He received compensation for an educational webinar from Otsuka (condition-specific, not product-focused) within the past 12 months. He receives ongoing compensation for editorial work from the American Association for Geriatric Psychiatry and Physicians Postgraduate Press.