This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Most evidence-based practices (EBPs) do not find their way into clinical use, including evidence-based mobile health (mHealth) technologies. The literature offers implementers little practical guidance for successfully integrating mHealth into health care systems.
The goal of this research was to describe a novel decision-framing model that gives implementers a method of eliciting the considerations of different stakeholder groups when they decide whether to implement an EBP.
The decision-framing model can be generally applied to EBPs, but was applied in this case to an mHealth system (Seva) for patients with addiction. The model builds from key insights in behavioral economics and game theory. The model systematically identifies, using an inductive process, the perceived gains and losses of different stakeholder groups when they consider adopting a new intervention. The model was constructed retrospectively in a parent implementation research trial that introduced Seva to 268 patients in 3 US primary care clinics. Individual and group interviews were conducted to elicit stakeholder considerations from 6 clinic managers, 17 clinicians, and 6 patients who were involved in implementing Seva. Considerations were used to construct decision frames that trade off the perceived value of adopting Seva versus maintaining the status quo from each stakeholder group’s perspective. The face validity of the decision-framing model was assessed by soliciting feedback from the stakeholders whose input was used to build it.
Primary considerations related to implementing Seva were identified for each stakeholder group. Clinic managers perceived the greatest potential gain to be better care for patients and the greatest potential loss to be cost (ie, staff time, sustainability, and opportunity cost to implement Seva). All clinical staff considered time their foremost consideration—primarily in negative terms (eg, cognitive burden associated with learning a new system) but potentially in positive terms (eg, if Seva could automate functions done manually). Patients considered safety (anonymity, privacy, and coming from a trusted source) to be paramount. Though payers were not interviewed directly, clinic managers judged cost to be most important to payers—whether Seva could reduce total care costs or had reimbursement mechanisms available. This model will be tested prospectively in a forthcoming mHealth implementation trial for its ability to predict mHealth adoption. Overall, the results suggest that implementers proactively address the cost and burden of implementation and seek to promote long-term sustainability.
This paper presents a model implementers may use to elicit stakeholders’ considerations when deciding to adopt a new technology, considerations that may then be used to adapt the intervention and tailor implementation, potentially increasing the likelihood of implementation success.
ClinicalTrials.gov NCT01963234; https://clinicaltrials.gov/ct2/show/NCT01963234 (Archived by WebCite at http://www.webcitation.org/78qXQJvVI)
The vast majority of practices shown to be effective by research remain unused in health care. It takes an estimated 17 years for an evidence-based practice (EBP) to be used in clinics, but only 14% of EBPs ever make it into use [
The focus of this paper is a novel model for implementation that can generally be applied to EBPs. The model was developed through an exploratory analysis conducted in the context of an mHealth implementation research trial funded by the US National Institutes of Health (NIH) [
Fundamentally, implementing an EBP is a social process [
People frame their decisions on the basis of their own unique perspectives. Tversky and Kahneman’s paper, “The Framing of Decisions and the Psychology of Choice,” [
Decision making involves not just a person’s perception of the acts, outcomes, and contingencies related to a specific choice, but also a subjective evaluation that determines the perceived value associated with a given choice. The perceived value of a particular choice depends critically on each decision maker’s unique perspective as a stakeholder in a health care system. In this context,
This simple equation (wherein value equals perceived gains minus perceived losses) becomes complicated in light of Tversky and Kahneman’s pioneering work showing that the everyday choices people make are typically not governed by rationality, as had been long assumed in classical economic theory [
Schematic representation of decision framing in terms of gains and losses (adapted from Tversky and Kahnamen [
These decision-making biases are hard to address even among people well versed in them [
Clearly, individual choices favor the status quo. Compounding the issue, implementation involves many different individuals making choices—all of them with different decision-making considerations. Game theory provides a framework for organizing implementation as a series of decisions made by members of different health care stakeholder groups [
In EBP adoption, the players can be defined as the 4 stakeholder groups named above (payers, clinic managers, staff, and patients); the players’ actions are either to adopt or resist adopting the EBP; and each player’s payoff corresponds to the perceived net benefits of adopting or not adopting
At each stage of implementation, different stakeholder groups make informal assessments of the value of adopting the EBP. Abstaining from decision making by failing to participate in the implementation process is common and tantamount to not adopting. If members of a stakeholder group do not perceive that they will benefit significantly by adopting, they may choose not to adopt and maintain the status quo instead. For example, if management promotes an EBP that staff members find onerous, staff will likely not adopt unless they are strongly compelled to adopt. The serial cooperation required for successful implementation will be broken at this level, and patients will not have access to the EBP because their access depends on the cooperation of clinical staff. Staff members are acting rationally in this example, because, by not adopting, they are selecting the option with greater value
The game theory conception of implementation helps further explain the dismal statistics on implementation success cited at the start of this paper [
This paper provides a systematic model that implementers and researchers may use to gather input from health care stakeholders whose cooperation is essential to the successful implementation of EBPs. The model was applied specifically to an mHealth implementation study, and it therefore offers insights specific to mHealth, in addition to a method for designing and operating an effective implementation strategy. Someone who wants to introduce and implement an EBP into a system would benefit from understanding the considerations—the gains and losses—that potential adopters perceive as they think about adopting new practices. These considerations can then be used to modify the intervention or the implementation strategy (or potentially both) to better align with the considerations expressed by potential adopters and improve the likelihood of implementation success.
The mHealth intervention analyzed is called Seva, an evidence-based mHealth intervention designed to help prevent relapse in people recovering from substance use disorders [
The study protocol was designated minimal risk and approved by the University of Wisconsin’s Health Sciences Institutional Review Board (protocol number: 2012-0937-CP019). The parent study is registered with ClinicalTrials.gov (NCT01963234).
The parent study [
The model used to frame decisions around EBP adoption is based on procedures for eliciting stakeholder considerations and defining a decision-analytic structure described by Edwards et al in their 2007 text,
Clinic managers at each implementation site were interviewed one-on-one by the decision analyst. Clinic managers continually make decisions about whether to undertake new projects, such as implementing Seva. Such decision making often occurs in the context of formal meetings intended to establish consensus around organizational goals (eg, monthly board meetings). An initial question posed during one-on-one interviews with managers from each site was, “What factors do you consider in deciding whether to introduce a new EBP like Seva to the staff and patients in your organization?” This initial inquiry was followed with specific questions about the factors the manager named, as well as questions that arose in the context of the discussion. Follow-up questions included the following: “At the organizational level, is there a process for deciding what new practices to implement? What barriers did you face in introducing Seva to your clinic? What would make it easier for you to implement Seva?”
Teams of staff members who participated in the implementation of Seva were interviewed in a group setting at each of the 3 implementation sites. When it comes to adopting a new EBP, clinic staff members can usually choose to adopt the new practice (such as Seva) or maintain the status quo.
During group interviews with staff members, the decision analyst asked participants to reflect on the following question: “What do you think about when you are asked to do something new—a new procedure, a new technology, or some new evidence-based practice?” The decision analyst then gave staff members time to generate ideas individually. These ideas were then shared within the group in a round-robin fashion. After eliciting key considerations with respect to adopting new practices, the decision analyst used open-ended questions to expand on concepts presented by participants. Follow-up questions included the following: “In your different roles, how are you judged to be successful? Are metrics used (eg, number of patients seen, patient surveys, and other things)?”
Baseline characteristics of participating clinics, clinic staff, and patients.
Characteristics | Site 1 (Madison, Wisconsin): Primary care and mental health | Site 2 (Missoula, Montana): Primary care, mental health, and addiction treatment | Site 3 (Bronx, New York): Primary care and mental health | ||
Participants | 5 | 9 | 9 | ||
Manager | 1 | 2 | 3 | ||
Physician | 1 | 2 | 1 | ||
PhD psychologist | 1 | 0 | 0 | ||
Therapist, counselor, or social worker | 0 | 2 | 3 | ||
Care manager | 0 | 2 | 1 | ||
Medical assistant | 2 | 0 | 0 | ||
Clinic data manager | 0 | 1 | 0 | ||
Other | 0 | 0 | 1 | ||
Participants, n | 0 | 3 | 3 | ||
Age (years), range | —a | 43-56 | 40-63 | ||
Gender (female), n | — | 2 | 1 | ||
Some high school | — | 0 | 3 | ||
Some college | — | 2 | 0 | ||
Associate’s degree | — | 1 | 0 | ||
Alcohol | — | 2 | 0 | ||
Cocaine | — | 0 | 1 | ||
Marijuana | — | 0 | 2 | ||
Multiple drugs | — | 0 | 1 | ||
Ethnicity, Hispanic/Latinx, n | — | 0 | 1 | ||
White | — | 3 | 1 | ||
African American/black | — | 0 | 2 |
aNot applicable.
Decision-framing model. EBP: evidence-based practice.
Patients were interviewed in a group setting at both the Missoula and Bronx sites. (Owing to staggered implementation timing and turnover of a key staff member, patients at the first implementation clinic could not be reached for follow-up interviews). During these group interviews, patients were asked to reflect upon the considerations they had about adopting Seva and upon how Seva complemented other addiction treatment options, such as outpatient addiction treatment services offered by the clinic and traditional Alcoholics Anonymous/Narcotics Anonymous meetings. An initial question posed to patients was, “Assume you have a friend struggling with drug or alcohol problems who wants to know if you would recommend Seva to him or her. What would you say and why?” Each patient participant reflected on this question and shared responses, prompting group discussion. Follow-up questions posed by the decision analyst included the following: “What problems (if any) did you have using Seva? To what extent was cost a barrier to your using Seva?”
Input derived from this series of stakeholder interviews was used to establish the primary values (ie, trade-offs between perceived gains and losses) that governed stakeholders’ decisions about the implementation of Seva. Considerations gathered through the interviews were systematically reviewed by the decision analyst and another researcher, and they were compiled into a decision-framing model that expressed the considerations as perceived gains and losses from the perspective of each stakeholder group.
The first-order approach to assessing the validity of any model focuses on face validity—that is, the degree to which the model concords with holistic judgments of validity by the stakeholders whose input was used to develop it [
The subsequent results and discussion should be understood in the context of the parent implementation study [
Stakeholder implementation considerations.
Stakeholder group | Decision alternatives | Considerations: perceived gains and losses | Notes on implementation |
Clinic managers | Support implementation of Seva versus allocate resources to competing projects | Gain: increased quality of patient care; Loss: additional clinical staff time required to implement and operate the intervention; Gain: advances organizational mission; Loss: uncertainty about sustainability potential of new intervention; Loss: opportunity cost of time for clinic champion to lead change efforts; Loss: lack of integration of new intervention into existing clinical workflows | Perceived gains were evident at outset. Clinics were compensated for staff time during grant period to offset costs. Management at all clinics supported introduction and use of Seva throughout the implementation period. Though management at 2 of 3 sites supported ongoing use of Seva, the challenges of transferring from grant funding to a long-term sustainable operational plan could not be successfully addressed, and system use ended at all 3 sites |
Clinic staff | Adopt Seva or maintain status quo clinical practice for addiction | Loss: time required to learn and use a new system; Loss: disruption of current workflows, including integration with the electronic health record (EHR); Gain: improved quality of patient care; Loss: uncertainty about long-term sustainability; Gain: potential to automate clinical functions currently done manually | Seva was heavily used and valued by clinic champions, but penetration beyond clinic champions was limited. Failure to integrate Seva data into EHR made accessing Seva data infeasible for most clinicians |
Patients | Use Seva (in addition to standard addiction treatment offered by the clinic) or continue with standard addiction treatment offered by the clinic or seek other treatment (eg, Alcoholics Anonymous) | Gain: access to a safe means of recovery support (anonymous and private, as well as coming from a trusted source); Gain: promotes access to resources and connections to similar others; Gain: promotes autonomy in recovery management (ie, voluntary use on patient’s own time); Loss: cost to operate (including smartphone and data plan, covered by grant during intervention but transferred to patients after 12 months) | Patient out-of-pocket costs for Seva were paid with National Institutes of Health grant funding. Patient use during the study was high; use fell to zero when costs shifted to patients after grant funding ended. Logistical challenges made it difficult to transfer payment arrangements for data plans from the research team to individuals |
In determining whether to adopt Seva, clinic managers considered the greatest potential gain (or
According to 1 clinic leader who championed the Seva project, “If a program like Seva costs us money, on balance, it will be very hard for us to sustain. If it saves the organization money, it has a chance.” As the director of behavioral health put it at another clinic, “We don’t want to spend the resources to create something new that won’t last.” Many Federally Qualified Health Centers are accountable care organizations (or part of such organizations) that are responsible for the total cost of care for each patient. If an intervention like Seva helps patients maintain healthy and stable lives—and helps avoid the costly emergency room visits, detoxification stays, and hospitalizations often associated with addiction—the cost of implementing it may be worthwhile. If an innovation cannot be paid for by a grant or insurance reimbursement, it is, by definition, unsustainable, and an unsustainable innovation is not worth implementing. Managers also weigh costs in terms of staff time and integration with existing workflows, and small aspects of a proposed innovation can affect those costs. One management group uses a visual representation of clinic workflows to see how an intervention might fit because altering existing operations and workflow is expensive in labor and opportunity cost. As this manager put it, “How many hoops will staff have to jump through to do this?”
At all 3 clinics, virtually every clinic staff member interviewed cited time requirements as the foremost consideration in deciding to adopt a new practice. An intervention that costs staff extra time (expressed in terms of learning and using a new method or methods) is perceived negatively; a proposed intervention that might save staff time is perceived positively. An addiction psychiatrist whose patients used Seva shed light on what clinicians are trying to accomplish in the limited time they have with patients:
I’ve got 20 minutes with each patient every 6 weeks, if they show up. If you’re my patient and you’re not getting better, I want to know what is going on in your life. Are you not taking the medications I’ve prescribed? Are you using alcohol or other drugs? Are you having trouble with your family? Are you not sleeping? What’s going on?
This clinician values learning about a patient’s problems as efficiently as possible—time is her most limited resource. An mHealth system like Seva has the potential to save clinicians time by continuously gathering and summarizing patient data so that they can quickly get an accurate picture of their patients’ lives. Clinicians also said (like managers) that sustainability was important in their decision making, but to clinic staff, sustainability meant, in part, integration with the electronic health record (EHR). This EHR consideration arose in the group interviews because Seva data could not be integrated into the EHR, which caused inefficiency and frustration among clinicians. Clearly, the EHR is central to clinicians because it structures and monitors clinical work. This sets a high bar for EBP implementation because making changes to the EHR can be so difficult. In the health system that 1 clinic is part of, incorporating data into the EHR from external systems is reportedly so onerous that integration takes place only when changes are legally mandated (eg, a change to the EHR was enacted only after the state legislature passed a law requiring physicians to check the state’s Prescription Drug Monitoring Program database before prescribing an opioid).
Because patients were asked about their decision making in the context of the Seva project, their responses reflected their thoughts about Seva and addiction treatment more specifically than the responses of managers and clinic staff, whose comments reflected their experience with the adoption of EBPs generally. Patients cited safety as their foremost consideration with regard to Seva. In the context of recovering from a substance use disorder, the concept of safety includes anonymity and privacy, as well as feeling confident that the innovation came from a trusted source (ie, on the basis of a recommendation from the patient’s clinician). Patients chose to use Seva in part because they weighed it against specific alternatives—Alcoholics Anonymous or Narcotics Anonymous meetings or not receiving treatment—that felt less safe to them. Patients’ comments also suggested that a successful innovation—all 6 patients interviewed regarded Seva as a valuable intervention—must promote connection to others and access to recovery resources. In both the rural setting of Missoula, Montana, and the urban setting of the Bronx, New York, patients reported feeling isolated. A patient in the Bronx said it was hard to get to Alcoholics Anonymous meetings, especially those held at night, because of concerns for her physical safety and the temptation to use drugs. In Missoula, getting to meetings was challenging because of difficulties with transportation. Seva addressed the isolation of these patients by enabling them to connect with peers and get help 24/7. A participant from the Bronx described how Seva helped him when he was on the brink of relapse. He decided to use Seva to reach out to one of the group’s monitors (a member of the research team). “If I didn’t have that phone, I don’t know what would’ve happened,” he reported. He also said the following:
Reaching out through Seva was the only thing that I could’ve done at that moment. I just needed someone to talk to; to listen to what was going on with me; to give me a push in the right direction.
Finally, patients wanted to feel in control of their choices to use Seva rather than be coerced. “I like the fact that this is not something I’m forced to do,” said one woman from the Bronx. She also said the following:
I can do it [use Seva] when I want to. This is my option. If I don’t feel like listening, I won’t listen!
The 6 clinic managers and staff members who were consulted to provide face validity offered no corrections to the results presented.
This research conceptualizes the problem of implementing EBPs in a new way, borrowing key ideas from behavioral economics and game theory and integrating them with stakeholder feedback. This new conceptualization applies to the implementation of EBPs generally, but in this case, it was applied to one of the largest mHealth implementation trials to date, thereby specifically producing insights about mHealth implementation. The presumption of implementation researchers is that it is valuable to adopt EBPs
Payers are key stakeholders in health care systems, but payers were not directly interviewed for this study. In interviews with clinic managers and through interactions with a payer at 1 site, a single payer consideration was perceived as dominant: cost. An EBP may be perceived positively if it reduces the total cost of care—as Seva showed the potential to do through its effects on hospitalizations and emergency room visits [
The results of this inductively constructed model will now be tested prospectively, in a process using deductive reasoning, for its ability to predict the adoption of mHealth in a forthcoming implementation study funded by the US NIH (1R01DA04415901A1). In this test, the considerations reported here will be ranked (eg, clinic managers in the forthcoming trial will rank the 6 considerations reported by clinic managers in this paper), and then the decision alternatives will be rated (eg, clinic managers will rate how well the mHealth intervention addresses the considerations). See
A nesting structure of values emerged through the interviews with stakeholders, both within each stakeholder group and between 1 stakeholder group and the next. For instance, management implied that for an intervention to be maximally appealing for adoption, it must first and foremost be valuable to patients, then palatable to staff, and then sustainable from a cost and reimbursement perspective. In a sense, management is implicitly incorporating the key values of patients, staff, and payers when deciding whether to approve implementation projects.
Illustration of prospective ranking and rating procedures.
Although many potentially useful instruments and frameworks are available from the implementation research literature to aid in the implementation process [
The need for systematic tailoring of implementation strategies has been identified as essential in the implementation research literature [
For logistical reasons, the decision-framing model was constructed retrospectively, at roughly the end of the implementation period at each site. Stakeholders’ responses may have been different if the process had been undertaken before Seva was adopted. Prospective application of decision framing will take place in a forthcoming randomized trial, an NIH-funded implementation trial that was funded in 2018 (1R01DA04415901A1).
The face validity of decision framing was established in the context of a single study involving 1 type of health care setting (Federally Qualified Health Centers) and 1 EBP (an mHealth intervention for substance use disorders). Further research will be needed to validate the model and examine its usefulness with other interventions in other settings. The data reported also represent small samples, especially with only 6 patients interviewed, warranting caution about the generalizability of the findings.
Decision framing is a simple model that seeks to capture essential decision-making processes related to implementation research. More quantitatively robust decision-analytic techniques certainly exist (eg, multiattribute utility theory), but trade-offs are inevitable between pragmatism and research sophistication in selecting a model. Decision framing was selected in part because it is simple and intuitive enough for wider uptake.
Finally, decision modeling of any type invariably simplifies the complexity of any actual implementation process. Implementation does not always unfold in an orderly fashion, and assigning accurate weights to considerations can be difficult. For example, unreimbursed cost sealed the fate of Seva, despite patients’ positive perceived value and the efforts of leadership in 1 clinic to find funding, and it may be that unreimbursed cost commonly plays such a role in implementation. In addition, the implementation of some practices may not require cooperation from all stakeholder groups—for example, patients may choose to use certain EBPs (such as mHealth apps) without any support or involvement from clinic management or staff. Demand for innovations can bubble up from patients and staff; indeed, such origins may bode more favorably for successful implementation than the top-down approach to implementation that is common in the health care system.
Though the decision-framing model is new to implementation research, the rationale for it is both simple and pragmatic: implementing an EBP is a fundamentally social process [
Decision-framing to incorporate stakeholder perspectives in implementation.
evidence-based practice
electronic health record
mobile health
National Institutes of Health
The author wishes to thank Chantelle Thomas, Mary Jane Nealon, Virna Little, Thomas McCarry, and Victoria Ward for their research collaboration. The author also wishes to thank David Gustafson, Randall Brown, John Mullahy, Barbara Bowers, Oguzhan Alagoz, and Ramon Aldag for their mentoring, Mark McGovern and Joann Kirchner for their valuable feedback on the manuscript, and Roberta Johnson and Nick Schumacher for their ongoing support.
AQ has a shareholder interest in CHESS Health, a public benefit corporation that disseminates Web-based health care intervention for patients and family members struggling with addiction. This relationship is extensively managed by the author and the University of Wisconsin–Madison’s Conflict of Interest Committee.