Published on 20.04.17 in Vol 5, No 4 (2017): April
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/7044, first published Nov 24, 2016.
A Framework for the Study of Complex mHealth Interventions in Diverse Cultural Settings
Background: To facilitate decision-making capacity between options of care under real-life service conditions, clinical trials must be pragmatic to evaluate mobile health (mHealth) interventions under the variable conditions of health care settings with a wide range of participants. The mHealth interventions require changes in the behavior of patients and providers, creating considerable complexity and ambiguity related to causal chains. Process evaluations of the implementation are necessary to shed light on the range of unanticipated effects an intervention may have, what the active ingredients in everyday practice are, how they exert their effect, and how these may vary among recipients or between sites.
Objective: Building on the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth) statement and participatory evaluation theory, we present a framework for the process evaluations for mHealth interventions in multiple cultural settings. We also describe the application of this evaluation framework to the implementation of DREAM-GLOBAL (Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC [low- and middle-income countries]), a pragmatic randomized controlled trial (RCT), and mHealth intervention designed to improve hypertension management in low-resource environments. We describe the evaluation questions and the data collection processes developed by us.
Methods: Our literature review revealed that there is a significant knowledge gap related to the development of a process evaluation framework for mHealth interventions. We used community-based participatory research (CBPR) methods and formative research data to develop a process evaluation framework nested within a pragmatic RCT.
Results: Four human organizational levels of participants impacted by the mHealth intervention were identified that included patients, providers, community and organizations actors, and health systems and settings. These four levels represent evaluation domains and became the core focus of the evaluation. In addition, primary implementation themes to explore in each of the domains were identified as follows: (1) the major active components of the intervention, (2) technology of the intervention, (3) cultural congruence, (4) task shifting, and (5) unintended consequences. Using the four organizational domains and their interaction with primary implementation themes, we developed detailed evaluation research questions and identified the data or information sources to best answer our questions.
Conclusions: Using DREAM-GLOBAL to illustrate our approach, we succeeded in developing an uncomplicated process evaluation framework for mHealth interventions that provide key information to stakeholders, which can optimize implementation of a pragmatic trial as well as inform scale up. The human organizational level domains used to focus the primary implementation themes in the DREAM-GLOBAL process evaluation framework are sufficiently supported in our research, and the literature and can serve as a valuable tool for other mHealth process evaluations.
Trial Registration: ClinicalTrials.gov NCT02111226; https://clinicaltrials.gov/ct2/show/NCT02111226 (Archived by WebCite at http://www.webcitation.org/6oxfHXege)
JMIR Mhealth Uhealth 2017;5(4):e47
- mobile health;
- health care texting;
- process evaluation;
- process assessment (health care);
- health services, Indigenous;
- community-based participatory research;
Pragmatic RCTs and mHealth Interventions
Explanatory randomized controlled trials (RCTs) are the gold standard for measuring efficacy, the cause and effect relationship in treatment intervention research in medicine, and are instrumental in providing evidence for Clinical Practice Guidelines (CPGs). A key requirement of explanatory RCTs is the stringent standardization of the tested intervention, which (while challenging) is realistic; for example, in pharmaceutical trials that occur in clinical settings where a single variable can be reasonably well controlled. Standardization in explanatory RCTs removes bias and thus results in substantial rigor and strong internal validity; however, in contrast, external validity (ie, the applicability in diverse clinical settings) is often compromised and consequently applying RCT results to populations outside the study’s scope can be problematic [, ].
Health care interventions are typically difficult to standardize because they are complex with long, nonlinear implementation chains that may have unexpected outcomes in disparate settings. Similarly, mobile health (mHealth) interventions usually contain multiple active components that may require changes in the behavior of patients and providers. These behaviors in turn are driven by numerous social, cultural, and environmental factors, thus creating considerable complexity and even ambiguity related to causal chains [, ]. Therefore, a shortcoming of the robust explanatory RCT methodology is that this approach often falls short of providing answers needed in mHealth interventions and everyday clinical care practice [ ]. To facilitate the decision-making capacity between options of care under real-life service conditions, clinical trials must be pragmatic to evaluate an intervention under the variable conditions of health care settings with a wide range of participants [ ].
Evaluating and Reporting Pragmatic RCTs of mHealth Interventions
To be truly useful for decision making, pragmatic trials require detailed and standardized reporting with sufficient transparency to communicate what works and what does not work for whom and under what circumstances. The prevailing international standard for reporting trials in general is the CONSORT statement , which has been expanded by Zwarenstein and colleagues [ ] to reflect the complexity of issues that may impact on pragmatic trials. For Web-based and mHealth interventions, Eysenbach and colleagues have called for further expansion of reporting requirements. They developed the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth) statement to ensure reporting of sufficient detail for replication and theory building specifically to eHealth [ ]. The reporting standards are designed to provide important guidance for reporting trials in general, but they do not distinguish specific elements that should be researched during the process of implementation and juxtaposed with outcome research at the end of the trial.
Although the key research outcome of a complex intervention focuses on the effectiveness of the intervention in everyday practice, the process evaluation in contrast assesses how closely the intervention was actually implemented as intended in the study protocol. Process evaluation data explain why a study unexpectedly did not achieve anticipated outcomes or if achieved outcomes are truly a consequence of the intervention. Process evaluations therefore help to distinguish between the reasons for lack of outcome as follows: (1) a consequence of either implementation failure or (2) the failure of the intervention itself. In multisetting studies, the process evaluation also facilitates an examination of differences in outcome and documents program adaptations at various sites. Process evaluations shed light on the range of unanticipated effects an intervention may have, what the active ingredients in everyday practice are, how they exert their effect, and how these may vary among recipients or between sites .
Investigating Implementation Factors in Multisetting RCTs
Although standardized reporting of pragmatic trials is crucial to our understanding of how and why complex interventions work, a difficulty related to good reporting is that neither all the active components nor implementation barriers are known before the start of a trial; instead issues are often discovered during the implementation process and may vary between settings, thus requiring an emergent rather than predetermined study of implementation. Consequently, a key opportunity for the discovery and description of the unanticipated active components is the process evaluation. We argue that standardized reporting frameworks for pragmatic RCTs must allow for flexibility to permit investigators to explore emergent processes, once work on the trial has commenced, to ensure rigorous documentation of the evolving and capricious aspects of implementation. Therefore, a process evaluation plan that systematically seeks to document not only the standardized implementation items but also the emergent aspects of implementation must be a critical piece in the documentation of pragmatic trials, particularly when diverse cultural settings are involved.
Recommendations for the design of process evaluations of complex interventions have been published [- ]. We maintain, however, that little work has focused on developing theoretical approaches that would help researchers to reduce the complexity of the process evaluation by focusing on those functional components that are most relevant to community stakeholders as well as researchers in mHealth interventions.
Although there can be no single definition of what elements comprise the process evaluation of the implementation of mHealth interventions, we deconstruct the functional components that are most significant to researchers, policy makers and implementation science in this study. Building on the CONSORT-EHEALTH statement and participatory evaluation theory, we present a framework for the process evaluations for mHealth interventions focusing on active components of the intervention in multiple cultural settings. We also describe the application of this evaluation framework to the implementation of DREAM-GLOBAL (Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC [low- and middle- income countries]), a pragmatic RCT and mHealth intervention designed to improve hypertension management in low-resource environments. In addition, we describe data collection tools and processes that were developed to collect the process evaluation data necessary to inform implementation of the pragmatic RCT and to inform future scale up of DREAM-GLOBAL in various geographic and cultural settings.
The DREAM-GLOBAL RCT
DREAM-GLOBAL is a pragmatic RCT that can be described as a complex mHealth intervention. It is a research project designed to increase the capacity for affordable, evidence-based, guidelines-driven hypertension management interventions at the patient, provider, and community level by leveraging existing mobile telecommunication technology and task shifting (findings by Yeates K et al, unpublished data, 2016). This study was funded by The Canadian Institutes of Health Research, Grand Challenges Canada, and by the Global Alliance for Chronic Diseases and involves diverse cultural and geographic settings, including Indigenous communities in Canada and the Kilimanjaro region of Tanzania.
The DREAM-GLOBAL mobile technology consisted of health care short messaging services (SMS) texting to support patient hypertension self-management and to facilitate decision support for health care providers. The program theory was informed by the Canadian Hypertension Education Program CPGs drawing on evidence for the prevention of high Blood Pressure (BP) through dietary sodium restriction, BP measurement, education interventions for health care providers and patients, inter-professional care, health systems, and interventions such as automated reminder systems [, ].
Specifically, the DREAM-GLOBAL RCT was designed to test the effectiveness of mobile phone–based SMS feedback to patients and providers tailored to patients’ BP measurements using artificial intelligence. Briefly, in this intervention, nonmedical workers used Bluetooth-enabled BP monitors to record and transmit BP readings from patients with hypertension. Mobile phones were used for storing and transmitting the patient roster and BP readings to a server. The server then sent text messages to the participating patients with their individual results. Study participants were randomized into two groups: one group received a set of cardiovascular health promotion messages and the second group received the same set of messages and additional “active” messages that support self-management by indicating requirement for attendance at medical appointments, rationale for drug therapy, and adherence to medication regimes, and messages directly reflective of their blood pressure readings. In addition, DREAM-GLOBAL closed the loop for sharing of clinical health information among health care providers.
The DREAM-GLOBAL Constructivist Approach to the Process Evaluation
Our review of the RCT reporting and process evaluation literature revealed that there is a significant knowledge gap related to the development of a process evaluation framework that can accommodate the values of community-based participatory research (CBPR) and the importance of acknowledging diverse cultural perspectives and settings. We used a medical anthropology approach to develop a framework reflective of participatory evaluation theory and specifically focused on meeting the information needs of academic and community stakeholders for mHealth interventions in diverse settings. The process evaluation framework was informed by the following 3 main factors:
- A review of current process evaluation theories applied to complex interventions and established RCT reporting requirements;
- Formative DREAM-GLOBAL CBPR in diverse cultural settings, including ethnographic notes, research engagement and implementation research notes, and reflective discussion sessions with research and community teams collected during formative research. This data represents a substantial set of qualitative data collected over the period of 1.5 years [ , ];
- mHealth-specific topics within the context of international, culturally, and geographically diverse low-resource settings [ ].
Current Process Evaluation Theories
Various approaches to the evaluation of complex interventions have been described in the literature. Bamberger and colleagues stressed that no single methodology can address all dimensions of complexity and that a careful mapping of complexity dimensions is necessary to design an evaluation [, ]. They offered a conceptual framework of 5 sources of complexity in the evaluation of interventions as follows: (1) the nature of the program (what does the intervention look like), (2) the context within which the program is embedded, (3) the interactions among the different stakeholders and agencies involved in the program,(4) the nature of processes of change and causality (how does the program effect change in society and how can change be captured), and (5) the nature of the evaluation process (how to deal with divergent stakeholder interests and incentives, data availability, resources, etc) [ ].
In contrast, Damschroder and coworkers conducted a review of the literature and identified 5 major domains for the evaluation of implementation: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation. The respective domains are further broken down into a total of 37 subcategories such as cost, policies, implementation climate, stage of change, and engagement .
Realist evaluation approaches have also been applied to process evaluations of complex interventions . The realist evaluation questions include: “What works, for whom, in what respects, to what extent, in what contexts, and how?” To answer these questions, realist evaluators tried to identify the underlying mechanisms that explain how the outcomes were caused and the influence of context on the intervention [ ]. Although the main realist evaluation questions can be adapted to various projects, there is a lack of consensus and consistency in defining the major domains of “mechanism” and “context.” This limited its utility in the development of a concise framework for process evaluations aimed at mHealth interventions, particularly when sources of complexity include culturally and geographically diverse low-resource sites, such as the DREAM-GLOBAL trial [ ].
The Medical Research Council’s (MRC) guidance report is a framework for process evaluation of complex interventions and builds on the 2008 MRC guidance document for complex interventions, which recognized the importance of process evaluation within trials [, ]. It builds on three concepts for a process evaluation design namely the description of the intervention, the mechanisms, and the context. These concepts align with realist evaluation theory, although the authors do not recommend specific evaluation theories or approaches. The authors argue for clear descriptions of intervention theory and identification of key implementation process questions and tailoring of the evaluation to the intervention.
While all these frameworks will result in the documentation of important aspects of a process evaluation, there are also serious limitations. For example, although these approaches advocate to examine the setting of the intervention, it is not clear which aspects of the setting evaluators should be focused on. Another limitation specifically associated with realist evaluation approaches is that realist philosophy presupposes that “reality can be experienced and shared by everyone in precisely the same way” . However, in a multicultural context, there is often little knowledge and experience that is shared between the researchers and those who interact with the intervention. It is precisely the lack of a shared culture that makes implementation particularly difficult to predict, unless the differences in how the intervention is experienced in various cultural settings are acknowledged and actively researched. We argue that it is particularly important when dealing with evaluations in multiple cultural settings to use a constructivist approach which acknowledges the knowledge and experience constructed by the individuals and societies. Participatory evaluation theory provides an excellent framework, as this approach incorporates the perspective and lived experience of local stakeholder in all aspects of evaluation. The participatory evaluation approach includes a community-based discovery process of implementation issues that integrates local knowledge and is essential to the development of a good evaluation framework for health interventions designed to work in diverse settings.
We argue that given the complexity of mHealth interventions and the multiple cultural settings of trials such as DREAM-GLOBAL, it would be naive for an evaluator to identify a priori the most important active components of the intervention and related evaluation research questions. Local knowledge and expertise of community stakeholders is urgently needed not only for successful implementation of DREAM-GLOBAL but also to understand implementation barriers and required adaptations. We therefore intentionally moved away from notions of the detached objectivity of a positivistic oriented evaluation which would marginalize local knowledge and emergent understandings. Instead of applying a constructivist evaluation approach, the research team entered the development of the process evaluation as “learners, not claiming to know preordinately what is salient” . The collaborative process, characteristic of constructivist inquiry supported a discovery process informed by dialogue, negotiation, and verification with stakeholders to identify the most relevant evaluation domains related to implementation [ ].
Formative Community-Based Participatory Research
CBPR has been defined as “a collaborative research approach that is designed to ensure and establish structures for participation by communities affected by the issue being studied, representatives of organizations, and researchers in all aspects of the research process to improve health and well-being through taking action, including social change” . CBPR principles have been incorporated into DREAM-GLOBAL. During formative CBPR research for this project, we developed an information gathering and dialogue tool, consisting of the following 3 phases: (1) a community profile tool, (2) an interview guide to facilitate the discussion of strategic topics related to implementation with key stakeholders in the community, and (3) a focus group guide to lead a dialogue on community-specific issues related to the intervention. The methodology has been previously published [ ]. A thematic analysis of this qualitative data as well as participant observation data from community visits and team meetings provided initial process evaluation domains and evaluation research questions. We then further compared and contrasted our formative data with the evaluation literature, specifically the MRC guidelines [ ] and CONSORT-EHEALTH extension. We ensured that the basic process evaluation domains would address the most important research questions that are meaningful and of immediate importance to community and policy maker stakeholders.
The finalization of the process evaluation domains was facilitated by monthly team discussions and several evaluation meetings over the period of 12 months where the literature, formative research evidence, ethnographic notes, and stakeholder discussion notes were analyzed, and eventually consensus was reached on the most significant process evaluation domains.
Integration of the Process Evaluation Research Into the DREAM-GLOBAL Trial
The pragmatic trial is designed to test the program theory of DREAM-GLOBAL—to facilitate the control of BP in low-resource environments through evidence-based text messaging, targeting patients, and enabling feedback to health care providers. The formative research was designed to generate technological, cultural, social, and health systems data to determine feasibility and, if necessary, plan for tailoring the implementation for each DREAM-GLOBAL site [, ]. The process evaluation protocol was nested within the pragmatic trial and followed the formative research before implementation of the trial and preceded outcome research (see ). The purpose of the process evaluation was primarily to evaluate the process of implementation of the trial; however, it will also generate information that will be valuable for potential posttrial scale up of DREAM-GLOBAL. The objectives of the process evaluation were to examine (1) what was delivered, including the quality (fidelity) and quantity (dose); (2) how delivery of the intervention was achieved; as well as (3) how the mHealth intervention required adaptations or innovations within the context of international, culturally, and geographically diverse low-resource settings [ ] and 4) any unanticipated outcomes [ ].
The study involved Indigenous people and is built foremost on strong commitment to respectful, CBPR with First Nations communities as partners in research as outlined in the Tri-Council Policy Statement (TCPS 2) . We therefore sought academic ethics review as well as community-based First Nations ethics review for the clinical trial as well as the process evaluation research. Furthermore, the research team also sought formal approval and permission for DREAM-GLOBAL implementation from First Nations decision makers in each of the participating communities after local, in-person presentations. Academic ethics approvals include the following: (1) Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board, Kingston, Ontario (DMED-1603-13); (2) Sunnybrook Health Sciences Centre Research Ethics Board, Toronto, Ontario, (#182-2013) and (3) the National Institute for Medical Research Tanzania (NIMR/HQ/R.8a/Vol.IX/1698). Community-based ethics review in First Nations communities included The Cree Board of Health and Social Services of James Bay, Ontario and Manitoulin Anishinaabek Research Review Committee (MARRC), Ontario. The study was also formally approved by decision-making bodies in all participating communities through Band Council Resolutions in participating First Nations.
The implementation of the RCT (ClinicalTrials.gov registration: NCT02111226. 2014) is currently ongoing and is expected to be completed by December 2017. The data analysis had not begun at the time of submission of this protocol.
Organizational Levels of the Process Evaluation
Four human organizational levels of participants impacted by the mHealth intervention were identified in our analysis and included patients, providers, community and organizations actors, and health systems and settings (see). These four levels correspond with the organizational evaluation domains (see ).
These four organizational levels critically impact on the primary outcome of the mHealth intervention, that is, improvements in BP. The process evaluation framework therefore examines implementation from each of these four organizational perspectives.
Primary Implementation Themes in the Process Evaluation
We thematically reanalyzed qualitative data collected during the formative research and found four primary implementation themes that interact with the four organizational domains described above. The primary implementation themes for each of the organizational domains were (1) identification of the major active component of the intervention (or program theory), (2) the technology of the intervention, (3) cultural congruence, and (4) task shifting. In addition, participant observation research and ongoing collaboration meetings with community stakeholders helped to identify emergent issues and the importance of these issues. We therefore learned the importance of documenting unintended consequences, which became our fifth primary implementation theme.
Development of Evaluation Questions
The interaction between the organizational levels and the primary implementation themes will be explored in this process evaluation. Using the four organizational domains and their interaction with the primary implementation themes, we developed detailed evaluation research questions and the identified data or information sources that we concluded would best answer our questions. The evaluation questions and data sources are presented in.
For example, one issue identified as a primary implementation theme in our formative work was the notion of task shifting. Task shifting in health care is a process where specific tasks are delegated to health care workers who have fewer qualifications but have received competency training . In DREAM-GLOBAL, the task of measuring BP is shifted from a medical to a nonmedical worker. Applying this implementation theme to the four domains of our framework, we developed queries to explore the impact of task shifting on each of the four human organizational levels:
- Patients have to be comfortable with the task shifting.
- Nonmedical providers should be confident and willing to take on this new task, and medical providers should be confident on the nonmedical providers to accept that measurements are accurate, in order to act on them
- The organizational decision makers have to be in a position where they can assign the new tasks.
- Health system decision makers must support the shift instead of posing barriers (eg, funding or licensing issues).
The DREAM-GLOBAL RCT constitutes a complex mHealth intervention because it is designed to affect change in the behavior of patients, providers, and local systems in order to achieve the primary outcome: lowering BP through management according to CPG. The DREAM-GLOBAL RCT intervenes at multiple points in the care normally provided to patients with hypertension. It combines a chronic care management approach with medical task shifting to community health resource and community health workers in order to ensure hypertension care is delivered according to CPGs, even in low-resource environments such as Indigenous communities in Canada and Tanzania.
Given the complexity of the program, extensive formative research was necessary to understand the local health care system before initiating program implementation. This was necessary for the program to be adapted to patients and providers in diverse cultural contexts and health systems . However, we also realized that some adaptations to optimize integration with existing health services and to minimize undue increased workload and systemic barriers could not be anticipated before implementation and that required an openness to flexibility during the process of implementation. In line with Bumbarger and Perkins [ ], we wanted to document the unanticipated issues and distinguish between “innovation” (skilled implementers actively attempting to make an intervention better fit their population or setting) and “drift” (unintentional shortcomings arising from barriers to full implementation) [ ].
The research team anticipated the possibility of emerging elements that should be examined in a process that may not have been obvious during implementation planning. The research team preplanned a discovery process through formative and CBPR.
Applicability of the DREAM-GLOBAL Framework to Other mHealth Interventions
The DREAM-GLOBAL pragmatic trial involves several interacting components targeting changes in the patient, provider, health organizations, and health systems, which makes it a complex intervention. This is arguably true for almost all mHealth interventions, and our proposed framework based on human organizational level domains is therefore a useful and a clearly focused framework for process evaluations of complex mHealth interventions.
Our framework chart identified several primary implementation themes related to the intervention that we claim to be explored in process evaluations of mHealth interventions:
Major active components of the intervention: Craig and colleagues discussed for the importance of describing and evaluating how the intervention works by clearly describing how the active ingredients exert their effect. “Only by addressing this kind of question can we build a cumulative understanding of causal mechanisms, design more effective interventions and apply them appropriately across group and setting .” We believe that it is necessary to explore active components from the perspective of the four human organizational levels of patients, providers, organizations, and systems.
Technology: Technology is by definition the most integral component of any mHealth intervention and therefore requires close scrutiny in process evaluation at different organizational levels.
Cultural congruence: The importance of the significant additional complexity introduced with the inclusion of diverse cultural settings of implementation has been stressed in the literature [, ]. With increasing cultural diversity within many regions of the world due to human migration as well as the application of interventions in various settings, a focus on the important role of cultural factors and the arising need for adaptation for cultural congruence is an important emergent area of study of mHealth interventions.
Task shifting: Task shifting is one way to expand the health work force in low-resource countries, and it is a necessary component of mHealth when the interventions require new ways for providers to interact with patients.
Unintended consequences: Many teams have underscored the need for process evaluations to capture unintended consequences and the importance of using a variety of methods to describe these in detail [, , ].
Clearly, the primary implementation themes of DREAM-GLOBAL are strongly supported in the literature, which underscores their relevance to process evaluations of mHealth interventions, in general. When the framework is applied to interventions other than DREAM-GLOBAL, additional primary implementation themes could be added by researchers once identified, and added to the framework chart. Evaluation questions within the framework chart should be adapted for each intervention.
Implications for mHealth Interventions in Diverse Cultural Settings
The DREAM-GLOBAL process evaluation protocol provides a framework to document how this intervention was implemented in various Indigenous communities in Canada and rural villages in Tanzania; and if the same intervention was implemented and received in similar ways in different communities or not. Our framework has several strengths related to work in diverse cultural settings.
First, this framework allows for exploration of emergent topics and verification with multiple stakeholder perspectives instead of imposing a rigid linear framework that would have put us at risk of not identifying and analyzing key unanticipated implementation factors. The four human organizational levels (patients, providers, community and organizations, and health systems and settings) can be applied to other mHealth interventions to guide the development of the most salient evaluation research questions. Although implementation themes may vary slightly in other mHealth interventions, following our approach will help to identify the relevant implementation themes.
Second, our collaborative social constructivist I-RREACH approach allowed us to tap into the tacit knowledge of stakeholders by collecting qualitative data through field notes and extensive dialogue and negotiation. Community stakeholders and providers have tacit local knowledge which can support, strengthen and optimize the intervention in the local context; academic team members on the other hand have technical expertise to ensure that essential elements of an intervention are preserved in various settings .
Third, CBPR—the research approach employed for DREAM-GLOBAL—is an excellent epistemological fit with a social constructivist evaluation approach. Clearly, CBPR shares the orientation of social constructivist evaluation in that it incorporates and legitimizes multiple value systems, perspectives, and stakeholders and emphasizes the role of dialogue and negotiation and their link to action and improvements. Therefore, as part of the study implementation, the community was involved in shaping how the research would be implemented to provide the maximal benefit for the community and these processes were critically important for the successes of the study.
Fourth, the process evaluation is intended to support the succinct reporting of the DREAM-GLOBAL RCT according to the CONSORT-EHEALTH.
Finally, we found that the formative qualitative research on community engagement and the development of the SMS messages we conducted before implementation of DREAM-GLOBAL [, ] were also useful in informing the process evaluation. The I-RREACH process was implemented in five Indigenous communities and two rural villages in Tanzania. A total of 135 informants participated in 12 focus groups and 7 interviews. After the sessions were completed, 83 informants participated in an evaluation of the I-RREACH session. During the development of culturally safe SMS messages, a total of 45 informants from 3 Indigenous communities and 1 community in Tanzania participated in four focus groups. This rich formative qualitative research provided deeper understanding of community issues related to the implementation of a hypertension mHealth intervention in our thematic analysis and confirmation for the emerging evaluation domains for the process evaluation protocol.
We acknowledge that an important factor in our approach is that it relies on a collaborative team approach which includes local community experts and a well-functioning interdisciplinary academic team.
Our research supports the notion that process evaluations of complex mHealth interventions cannot be fully designed in advance and instead should employ a framework that allows for emergent research. When an mHealth intervention is to be deployed in multiple cultural settings, our research suggests that a participatory constructivist approach to the development of the process evaluation is necessary to identify relevant evaluation questions that incorporate local knowledge and expertise.
Eysenbach and colleagues stated that the CONSORT-EHEALTH statement is “only the first step and the guideline will be very much a living document in an iterative and ongoing development process” . We have developed from this checklist, to identify items that are well suited to be studied during implantation of an mHealth intervention during the implementation using process evaluation methodologies. We succeeded in creating an uncomplicated approach to the development of a process evaluation framework for mHealth interventions that provide key information to stakeholders, which can optimize implementation of the pragmatic trial and can be used to inform scale up. The human organizational level domains used to focus primary implementation themes in the DREAM-GLOBAL process evaluation framework are sufficiently supported in our research and literature that they can serve as a valuable tool for other mHealth process evaluations.
The authors would like to thank health care staff and leadership of participating communities in both the countries for their collaboration. In Canada these include Aundeck Omni Kaning First Nation, Cree Nation of Chisasibi, Eel River Bar First Nation, M’Chigeeng First Nation, Noojmowin Teg Health Centre, Sheshegwaning First Nation, Wikwemikong Unceded Indian Reserve. In Tanzania they include Kisiki and Lambo villages in Hai district, Kilimanjaro. We also thank the translators who supported the project in Tanzania. The project was funded by The Canadian Institutes of Health Research, Grand Challenges Canada, and the Global Alliance for Chronic Diseases.
Conflicts of Interest
Multimedia Appendix 1
Framework chart with evaluation research questions.PDF File (Adobe PDF File), 55KB
- Hotopf M. The pragmatic randomised controlled trial. Adv Psychiatr Treat 2002;8(5):326-333. [CrossRef]
- McAlister FA, van Diepen S, Padwal RS, Johnson JA, Majumdar SR. How evidence-based are the recommendations in evidence-based guidelines? PLoS Med 2007 Aug;4(8):e250 [FREE Full text] [CrossRef] [Medline]
- Audrey S, Holliday J, Parry-Langdon N, Campbell R. Meeting the challenges of implementing process evaluation within randomized controlled trials: the example of ASSIST (A Stop Smoking in Schools Trial). Health Educ Res 2006 Jun;21(3):366-377. [CrossRef] [Medline]
- Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. Br Med J 2000 Sep 16;321(7262):694-696 [FREE Full text] [Medline]
- Manchikanti L, Hirsch JA, Smith HS. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: Part 2: randomized controlled trials. Pain Physician 2008;11(6):717-773 [FREE Full text] [Medline]
- Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. Br Med J 2008 Nov 11;337:a2390 [FREE Full text] [Medline]
- CONSORT Transparent Reporting of Trials. Consort-statement. Welcome to the CONSORT Website URL: http://www.consort-statement.org/ [accessed 2016-11-22] [WebCite Cache]
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008;337:a1655 [FREE Full text] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. MRC. 2006. Developing and evaluating complex interventions: new guidance URL: https://www.mrc.ac.uk/documents/pdf/complex-interventions-guidance/ [accessed 2017-03-14] [WebCite Cache]
- Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. MRC. 2014. Process evaluation of complex interventions: UK Medical Research Council (MRC) guidance URL: https://www.mrc.ac.uk/documents/pdf/mrc-phsrn-process-evaluation-guidance-final/ [accessed 2017-03-14] [WebCite Cache]
- Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. Br Med J 2015;350:h1258 [FREE Full text] [Medline]
- Tobe SW, Touyz RM, Campbell NR, Canadian Hypertension Education Program. The Canadian hypertension education program - a unique Canadian knowledge translation program. Can J Cardiol 2007 May 15;23(7):551-555 [FREE Full text] [Medline]
- Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Interventions used to improve control of blood pressure in patients with hypertension. Cochrane Database Syst Rev 2010 Mar 17(3):CD005182. [CrossRef] [Medline]
- Maar M, Yeates K, Barron M, Hua D, Liu P, Moy LM, et al. I-RREACH: an engagement and assessment tool for improving implementation readiness of researchers, organizations and communities in complex interventions. Implement Sci 2015;10:64 [FREE Full text] [CrossRef] [Medline]
- Maar MA, Yeates K, Toth Z, Barron M, Boesch L, Hua-Stewart D, et al. Unpacking the black box: a formative research approach to the development of theory-driven, evidence-based, and culturally safe text messages in mobile health interventions. JMIR Mhealth Uhealth 2016 Jan 22;4(1):e10 [FREE Full text] [CrossRef] [Medline]
- W Tobe S, Global Alliance for Chronic Diseases Hypertension Research Teams With the World Hypertension League. The global alliance for chronic diseases supports 15 major studies in hypertension prevention and control in low- and middle-income countries. J Clin Hypertens (Greenwich) 2016 Jul;18(7):600-605 [FREE Full text] [CrossRef] [Medline]
- Bamberger M, Vaessen J, Raimondo E. Complexity in development evaluation: the framework of the book. In: Bamberger M, Vaessen J, Raimondo E, editors. Dealing with Complexity in Development Evaluation: A Practical Approach. Thousand Oaks, CA: SAGE Publications; 2015.
- Vaessen J, Raimondo E, Bamberger M. Impact evaluation approaches and complexity. In: Bamberger M, Vaessen J, Raimondo E, editors. Dealing with Complexity in Development Evaluation: A Practical Approach. Thousand Oaks, CA: SAGE Publications; 2015.
- Raimondo E, Bamberger M, Vaessen J. Introduction. In: Bamberger M, Vaessen L, Raimondo E, editors. Dealing with Complexity in Development Evaluation. Thousand Oaks, CA: SAGE Publications; 2015.
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
- Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation (Lond) 2016 Jul;22(3):286-303 [FREE Full text] [CrossRef] [Medline]
- Marchal B, Van Belle S, Westhop G. Better Evaluation. Realist evaluation URL: http://betterevaluation.org/approach/realist_evaluation [accessed 2016-11-23] [WebCite Cache]
- Marchal B, Westhorp G, Wong G, Van Belle S, Greenhalgh T, Kegels G, et al. Realist RCTs of complex interventions - an oxymoron. Soc Sci Med 2013 Oct;94:124-128. [CrossRef] [Medline]
- Campbell B. Realism versus constructivism: which is a more appropriate theory for addressing the nature of science in science education? EJSE 1998;3(1).
- Guba E, Lincoln Y. Fourth Generation Evaluation. Newbury Park, CA: SAGE Publications; 1989.
- Viswanathan M, Ammerman A, Eng E, Gartlehner G, Lohr K, Griffith D, et al. AHRQ. 2004. Community-based participatory research: assessing the evidence URL: http://archive.ahrq.gov/downloads/pub/evidence/pdf/cbpr/cbpr.pdf [WebCite Cache]
- Bumbarger B, Perkins D. After randomised trials: issues related to dissemination of evidence‐based interventions. J Child Serv 2008 Apr 12;3(2):55-64. [CrossRef]
- Government of Canada. Pre-ethics. 2016. TCPS 2 - Chapter 9: Research involving the first nations, inuit and metis peoples of Canada URL: http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/chapter9-chapitre9/ [accessed 2017-03-14] [WebCite Cache]
- World Health Organization. 2008. Task shifting: rational redistribution of tasks among health workforce teams: global recommendations and guidelines URL: http://www.who.int/healthsystems/TTR-TaskShifting.pdf [accessed 2016-11-23] [WebCite Cache]
- Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013;14:15 [FREE Full text] [CrossRef] [Medline]
|BP: blood pressure|
|CBPR: community-based participatory research|
|CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth|
|CPG: clinical practice guidelines|
|DREAM-GLOBAL: Diagnosing hypertension—Engaging Action and Management in Getting Lower BP in Indigenous and LMIC|
|I-RREACH: Intervention and Research Readiness Engagement and Assessment of|
|LMIC: low- and middle-income countries|
|mHealth: mobile health|
|MRC: Medical Research Council|
|RCT: randomized controlled trial|
|SMS: short message service|
Edited by A Keepanasseril; submitted 24.11.16; peer-reviewed by R Dobson; comments to author 26.01.17; accepted 27.02.17; published 20.04.17
©Marion A Maar, Karen Yeates, Nancy Perkins, Lisa Boesch, Diane Hua-Stewart, Peter Liu, Jessica Sleeth, Sheldon W Tobe. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 20.04.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.