This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
To facilitate decision-making capacity between options of care under real-life service conditions, clinical trials must be pragmatic to evaluate mobile health (mHealth) interventions under the variable conditions of health care settings with a wide range of participants. The mHealth interventions require changes in the behavior of patients and providers, creating considerable complexity and ambiguity related to causal chains. Process evaluations of the implementation are necessary to shed light on the range of unanticipated effects an intervention may have, what the active ingredients in everyday practice are, how they exert their effect, and how these may vary among recipients or between sites.
Building on the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth) statement and participatory evaluation theory, we present a framework for the process evaluations for mHealth interventions in multiple cultural settings. We also describe the application of this evaluation framework to the implementation of DREAM-GLOBAL (Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC [low- and middle-income countries]), a pragmatic randomized controlled trial (RCT), and mHealth intervention designed to improve hypertension management in low-resource environments. We describe the evaluation questions and the data collection processes developed by us.
Our literature review revealed that there is a significant knowledge gap related to the development of a process evaluation framework for mHealth interventions. We used community-based participatory research (CBPR) methods and formative research data to develop a process evaluation framework nested within a pragmatic RCT.
Four human organizational levels of participants impacted by the mHealth intervention were identified that included patients, providers, community and organizations actors, and health systems and settings. These four levels represent evaluation domains and became the core focus of the evaluation. In addition, primary implementation themes to explore in each of the domains were identified as follows: (1) the major active components of the intervention, (2) technology of the intervention, (3) cultural congruence, (4) task shifting, and (5) unintended consequences. Using the four organizational domains and their interaction with primary implementation themes, we developed detailed evaluation research questions and identified the data or information sources to best answer our questions.
Using DREAM-GLOBAL to illustrate our approach, we succeeded in developing an uncomplicated process evaluation framework for mHealth interventions that provide key information to stakeholders, which can optimize implementation of a pragmatic trial as well as inform scale up. The human organizational level domains used to focus the primary implementation themes in the DREAM-GLOBAL process evaluation framework are sufficiently supported in our research, and the literature and can serve as a valuable tool for other mHealth process evaluations.
ClinicalTrials.gov NCT02111226; https://clinicaltrials.gov/ct2/show/NCT02111226 (Archived by WebCite at http://www.webcitation.org/6oxfHXege)
Explanatory randomized controlled trials (RCTs) are the gold standard for measuring efficacy, the cause and effect relationship in treatment intervention research in medicine, and are instrumental in providing evidence for Clinical Practice Guidelines (CPGs). A key requirement of explanatory RCTs is the stringent standardization of the tested intervention, which (while challenging) is realistic; for example, in pharmaceutical trials that occur in clinical settings where a single variable can be reasonably well controlled. Standardization in explanatory RCTs removes bias and thus results in substantial rigor and strong internal validity; however, in contrast, external validity (ie, the applicability in diverse clinical settings) is often compromised and consequently applying RCT results to populations outside the study’s scope can be problematic [
Health care interventions are typically difficult to standardize because they are complex with long, nonlinear implementation chains that may have unexpected outcomes in disparate settings. Similarly, mobile health (mHealth) interventions usually contain multiple active components that may require changes in the behavior of patients and providers. These behaviors in turn are driven by numerous social, cultural, and environmental factors, thus creating considerable complexity and even ambiguity related to causal chains [
To be truly useful for decision making, pragmatic trials require detailed and standardized reporting with sufficient transparency to communicate what works and what does not work for whom and under what circumstances. The prevailing international standard for reporting trials in general is the CONSORT statement [
Although the key research outcome of a complex intervention focuses on the effectiveness of the intervention in everyday practice, the process evaluation in contrast assesses how closely the intervention was actually implemented as intended in the study protocol. Process evaluation data explain why a study unexpectedly did not achieve anticipated outcomes or if achieved outcomes are truly a consequence of the intervention. Process evaluations therefore help to distinguish between the reasons for lack of outcome as follows: (1) a consequence of either implementation failure or (2) the failure of the intervention itself. In multisetting studies, the process evaluation also facilitates an examination of differences in outcome and documents program adaptations at various sites. Process evaluations shed light on the range of unanticipated effects an intervention may have, what the active ingredients in everyday practice are, how they exert their effect, and how these may vary among recipients or between sites [
Although standardized reporting of pragmatic trials is crucial to our understanding of how and why complex interventions work, a difficulty related to good reporting is that neither all the active components nor implementation barriers are known before the start of a trial; instead issues are often discovered during the implementation process and may vary between settings, thus requiring an emergent rather than predetermined study of implementation. Consequently, a key opportunity for the discovery and description of the unanticipated active components is the process evaluation. We argue that standardized reporting frameworks for pragmatic RCTs must allow for flexibility to permit investigators to explore emergent processes, once work on the trial has commenced, to ensure rigorous documentation of the evolving and capricious aspects of implementation. Therefore, a process evaluation plan that systematically seeks to document not only the standardized implementation items but also the emergent aspects of implementation must be a critical piece in the documentation of pragmatic trials, particularly when diverse cultural settings are involved.
Recommendations for the design of process evaluations of complex interventions have been published [
Although there can be no single definition of what elements comprise the process evaluation of the implementation of mHealth interventions, we deconstruct the functional components that are most significant to researchers, policy makers and implementation science in this study. Building on the CONSORT-EHEALTH statement and participatory evaluation theory, we present a framework for the process evaluations for mHealth interventions focusing on active components of the intervention in multiple cultural settings. We also describe the application of this evaluation framework to the implementation of DREAM-GLOBAL (Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC [low- and middle- income countries]), a pragmatic RCT and mHealth intervention designed to improve hypertension management in low-resource environments. In addition, we describe data collection tools and processes that were developed to collect the process evaluation data necessary to inform implementation of the pragmatic RCT and to inform future scale up of DREAM-GLOBAL in various geographic and cultural settings.
DREAM-GLOBAL is a pragmatic RCT that can be described as a complex mHealth intervention. It is a research project designed to increase the capacity for affordable, evidence-based, guidelines-driven hypertension management interventions at the patient, provider, and community level by leveraging existing mobile telecommunication technology and task shifting (findings by Yeates K et al, unpublished data, 2016). This study was funded by The Canadian Institutes of Health Research, Grand Challenges Canada, and by the Global Alliance for Chronic Diseases and involves diverse cultural and geographic settings, including Indigenous communities in Canada and the Kilimanjaro region of Tanzania.
The DREAM-GLOBAL mobile technology consisted of health care short messaging services (SMS) texting to support patient hypertension self-management and to facilitate decision support for health care providers. The program theory was informed by the Canadian Hypertension Education Program CPGs drawing on evidence for the prevention of high Blood Pressure (BP) through dietary sodium restriction, BP measurement, education interventions for health care providers and patients, inter-professional care, health systems, and interventions such as automated reminder systems [
Specifically, the DREAM-GLOBAL RCT was designed to test the effectiveness of mobile phone–based SMS feedback to patients and providers tailored to patients’ BP measurements using artificial intelligence. Briefly, in this intervention, nonmedical workers used Bluetooth-enabled BP monitors to record and transmit BP readings from patients with hypertension. Mobile phones were used for storing and transmitting the patient roster and BP readings to a server. The server then sent text messages to the participating patients with their individual results. Study participants were randomized into two groups: one group received a set of cardiovascular health promotion messages and the second group received the same set of messages and additional “active” messages that support self-management by indicating requirement for attendance at medical appointments, rationale for drug therapy, and adherence to medication regimes, and messages directly reflective of their blood pressure readings. In addition, DREAM-GLOBAL closed the loop for sharing of clinical health information among health care providers.
Our review of the RCT reporting and process evaluation literature revealed that there is a significant knowledge gap related to the development of a process evaluation framework that can accommodate the values of community-based participatory research (CBPR) and the importance of acknowledging diverse cultural perspectives and settings. We used a medical anthropology approach to develop a framework reflective of participatory evaluation theory and specifically focused on meeting the information needs of academic and community stakeholders for mHealth interventions in diverse settings. The process evaluation framework was informed by the following 3 main factors:
A review of current process evaluation theories applied to complex interventions and established RCT reporting requirements;
Formative DREAM-GLOBAL CBPR in diverse cultural settings, including ethnographic notes, research engagement and implementation research notes, and reflective discussion sessions with research and community teams collected during formative research. This data represents a substantial set of qualitative data collected over the period of 1.5 years [
mHealth-specific topics within the context of international, culturally, and geographically diverse low-resource settings [
Various approaches to the evaluation of complex interventions have been described in the literature. Bamberger and colleagues stressed that no single methodology can address all dimensions of complexity and that a careful mapping of complexity dimensions is necessary to design an evaluation [
In contrast, Damschroder and coworkers conducted a review of the literature and identified 5 major domains for the evaluation of implementation: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation. The respective domains are further broken down into a total of 37 subcategories such as cost, policies, implementation climate, stage of change, and engagement [
Realist evaluation approaches have also been applied to process evaluations of complex interventions [
The Medical Research Council’s (MRC) guidance report is a framework for process evaluation of complex interventions and builds on the 2008 MRC guidance document for complex interventions, which recognized the importance of process evaluation within trials [
While all these frameworks will result in the documentation of important aspects of a process evaluation, there are also serious limitations. For example, although these approaches advocate to examine the setting of the intervention, it is not clear which aspects of the setting evaluators should be focused on. Another limitation specifically associated with realist evaluation approaches is that realist philosophy presupposes that “reality can be experienced and shared by everyone in precisely the same way” [
We argue that given the complexity of mHealth interventions and the multiple cultural settings of trials such as DREAM-GLOBAL, it would be naive for an evaluator to identify a priori the most important active components of the intervention and related evaluation research questions. Local knowledge and expertise of community stakeholders is urgently needed not only for successful implementation of DREAM-GLOBAL but also to understand implementation barriers and required adaptations. We therefore intentionally moved away from notions of the detached objectivity of a positivistic oriented evaluation which would marginalize local knowledge and emergent understandings. Instead of applying a constructivist evaluation approach, the research team entered the development of the process evaluation as “learners, not claiming to know preordinately what is salient” [
CBPR has been defined as “a collaborative research approach that is designed to ensure and establish structures for participation by communities affected by the issue being studied, representatives of organizations, and researchers in all aspects of the research process to improve health and well-being through taking action, including social change” [
The finalization of the process evaluation domains was facilitated by monthly team discussions and several evaluation meetings over the period of 12 months where the literature, formative research evidence, ethnographic notes, and stakeholder discussion notes were analyzed, and eventually consensus was reached on the most significant process evaluation domains.
The pragmatic trial is designed to test the program theory of DREAM-GLOBAL—to facilitate the control of BP in low-resource environments through evidence-based text messaging, targeting patients, and enabling feedback to health care providers. The formative research was designed to generate technological, cultural, social, and health systems data to determine feasibility and, if necessary, plan for tailoring the implementation for each DREAM-GLOBAL site [
DREAM-GLOBAL process evaluation relationship to formative research and trial research (I-RREACH, Intervention and Research Readiness Engagement and Assessment of Community Health Care).
The study involved Indigenous people and is built foremost on strong commitment to respectful, CBPR with First Nations communities as partners in research as outlined in the Tri-Council Policy Statement (TCPS 2) [
The implementation of the RCT (ClinicalTrials.gov registration: NCT02111226. 2014) is currently ongoing and is expected to be completed by December 2017. The data analysis had not begun at the time of submission of this protocol.
Four human organizational levels of participants impacted by the mHealth intervention were identified in our analysis and included patients, providers, community and organizations actors, and health systems and settings (see
These four organizational levels critically impact on the primary outcome of the mHealth intervention, that is, improvements in BP. The process evaluation framework therefore examines implementation from each of these four organizational perspectives.
Patient participants comprise the primary target population of the intervention, in this case people with hypertension are enrolled in the study.
Provider participants are care and service providers whose work is to some degree altered by the intervention.
Community and organization members are people whose immediate social environment impacts on the intervention, for example, those who permit the implementation or approve changes in work flow. They may also be decision makers.
Health system and setting members are people or structures that impact on the implementation at a systems level, such as local and district level decision makers and National health policy makers.
Major factors that interact with the DREAM-GLOBAL mHealth intervention.
We thematically reanalyzed qualitative data collected during the formative research and found four primary implementation themes that interact with the four organizational domains described above. The primary implementation themes for each of the organizational domains were (1) identification of the major active component of the intervention (or program theory), (2) the technology of the intervention, (3) cultural congruence, and (4) task shifting. In addition, participant observation research and ongoing collaboration meetings with community stakeholders helped to identify emergent issues and the importance of these issues. We therefore learned the importance of documenting unintended consequences, which became our fifth primary implementation theme.
The interaction between the organizational levels and the primary implementation themes will be explored in this process evaluation. Using the four organizational domains and their interaction with the primary implementation themes, we developed detailed evaluation research questions and the identified data or information sources that we concluded would best answer our questions. The evaluation questions and data sources are presented in
For example, one issue identified as a primary implementation theme in our formative work was the notion of task shifting. Task shifting in health care is a process where specific tasks are delegated to health care workers who have fewer qualifications but have received competency training [
Patients have to be comfortable with the task shifting.
Nonmedical providers should be confident and willing to take on this new task, and medical providers should be confident on the nonmedical providers to accept that measurements are accurate, in order to act on them
The organizational decision makers have to be in a position where they can assign the new tasks.
Health system decision makers must support the shift instead of posing barriers (eg, funding or licensing issues).
The DREAM-GLOBAL RCT constitutes a complex mHealth intervention because it is designed to affect change in the behavior of patients, providers, and local systems in order to achieve the primary outcome: lowering BP through management according to CPG. The DREAM-GLOBAL RCT intervenes at multiple points in the care normally provided to patients with hypertension. It combines a chronic care management approach with medical task shifting to community health resource and community health workers in order to ensure hypertension care is delivered according to CPGs, even in low-resource environments such as Indigenous communities in Canada and Tanzania.
Given the complexity of the program, extensive formative research was necessary to understand the local health care system before initiating program implementation. This was necessary for the program to be adapted to patients and providers in diverse cultural contexts and health systems [
The research team anticipated the possibility of emerging elements that should be examined in a process that may not have been obvious during implementation planning. The research team preplanned a discovery process through formative and CBPR.
The DREAM-GLOBAL pragmatic trial involves several interacting components targeting changes in the patient, provider, health organizations, and health systems, which makes it a complex intervention. This is arguably true for almost all mHealth interventions, and our proposed framework based on human organizational level domains is therefore a useful and a clearly focused framework for process evaluations of complex mHealth interventions.
Our framework chart identified several primary implementation themes related to the intervention that we claim to be explored in process evaluations of mHealth interventions:
Major active components of the intervention: Craig and colleagues discussed for the importance of describing and evaluating how the intervention works by clearly describing how the active ingredients exert their effect. “Only by addressing this kind of question can we build a cumulative understanding of causal mechanisms, design more effective interventions and apply them appropriately across group and setting [
Technology: Technology is by definition the most integral component of any mHealth intervention and therefore requires close scrutiny in process evaluation at different organizational levels.
Cultural congruence: The importance of the significant additional complexity introduced with the inclusion of diverse cultural settings of implementation has been stressed in the literature [
Task shifting: Task shifting is one way to expand the health work force in low-resource countries, and it is a necessary component of mHealth when the interventions require new ways for providers to interact with patients.
Unintended consequences: Many teams have underscored the need for process evaluations to capture unintended consequences and the importance of using a variety of methods to describe these in detail [
Clearly, the primary implementation themes of DREAM-GLOBAL are strongly supported in the literature, which underscores their relevance to process evaluations of mHealth interventions, in general. When the framework is applied to interventions other than DREAM-GLOBAL, additional primary implementation themes could be added by researchers once identified, and added to the framework chart. Evaluation questions within the framework chart should be adapted for each intervention.
The DREAM-GLOBAL process evaluation protocol provides a framework to document how this intervention was implemented in various Indigenous communities in Canada and rural villages in Tanzania; and if the same intervention was implemented and received in similar ways in different communities or not. Our framework has several strengths related to work in diverse cultural settings.
First, this framework allows for exploration of emergent topics and verification with multiple stakeholder perspectives instead of imposing a rigid linear framework that would have put us at risk of not identifying and analyzing key unanticipated implementation factors. The four human organizational levels (patients, providers, community and organizations, and health systems and settings) can be applied to other mHealth interventions to guide the development of the most salient evaluation research questions. Although implementation themes may vary slightly in other mHealth interventions, following our approach will help to identify the relevant implementation themes.
Second, our collaborative social constructivist I-RREACH approach allowed us to tap into the tacit knowledge of stakeholders by collecting qualitative data through field notes and extensive dialogue and negotiation. Community stakeholders and providers have tacit local knowledge which can support, strengthen and optimize the intervention in the local context; academic team members on the other hand have technical expertise to ensure that essential elements of an intervention are preserved in various settings [
Third, CBPR—the research approach employed for DREAM-GLOBAL—is an excellent epistemological fit with a social constructivist evaluation approach. Clearly, CBPR shares the orientation of social constructivist evaluation in that it incorporates and legitimizes multiple value systems, perspectives, and stakeholders and emphasizes the role of dialogue and negotiation and their link to action and improvements. Therefore, as part of the study implementation, the community was involved in shaping how the research would be implemented to provide the maximal benefit for the community and these processes were critically important for the successes of the study.
Fourth, the process evaluation is intended to support the succinct reporting of the DREAM-GLOBAL RCT according to the CONSORT-EHEALTH.
Finally, we found that the formative qualitative research on community engagement and the development of the SMS messages we conducted before implementation of DREAM-GLOBAL [
We acknowledge that an important factor in our approach is that it relies on a collaborative team approach which includes local community experts and a well-functioning interdisciplinary academic team.
Our research supports the notion that process evaluations of complex mHealth interventions cannot be fully designed in advance and instead should employ a framework that allows for emergent research. When an mHealth intervention is to be deployed in multiple cultural settings, our research suggests that a participatory constructivist approach to the development of the process evaluation is necessary to identify relevant evaluation questions that incorporate local knowledge and expertise.
Eysenbach and colleagues stated that the CONSORT-EHEALTH statement is “only the first step and the guideline will be very much a living document in an iterative and ongoing development process” [
Framework chart with evaluation research questions.
blood pressure
community-based participatory research
Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth
clinical practice guidelines
Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC
Intervention and Research Readiness Engagement and Assessment of
low- and middle-income countries
mobile health
Medical Research Council
randomized controlled trial
short message service
The authors would like to thank health care staff and leadership of participating communities in both the countries for their collaboration. In Canada these include Aundeck Omni Kaning First Nation, Cree Nation of Chisasibi, Eel River Bar First Nation, M’Chigeeng First Nation, Noojmowin Teg Health Centre, Sheshegwaning First Nation, Wikwemikong Unceded Indian Reserve. In Tanzania they include Kisiki and Lambo villages in Hai district, Kilimanjaro. We also thank the translators who supported the project in Tanzania. The project was funded by The Canadian Institutes of Health Research, Grand Challenges Canada, and the Global Alliance for Chronic Diseases.
None declared.