This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Mobile health (mHealth) offers new opportunities to improve access to health services and health information. It also presents new challenges in evaluating its impact, particularly in linking the use of a technology intervention that aims to improve health behaviors with the health outcomes that are impacted by changed behaviors. The availability of data from a multitude of sources (paper-based and electronic) provides the conditions to facilitate making stronger connections between self-reported data and clinical outcomes. This commentary shares lessons and important considerations based on the experience of applying new research frameworks and incorporating maternal and child health records data into a pseudo-randomized controlled trial to evaluate the impact of mMitra, a stage-based voice messaging program to improve maternal, newborn, and child health outcomes in urban slums in India.
Over the past 10 years, there has been a rapid increase in the adoption and use of mobile technology by the health sector globally as a tool to increase access to health services and health information, strengthen health systems, improve the quality of care by health professionals, and increase efficiency in the delivery of health services. From the earliest days of mobile health (mHealth), the peer-reviewed literature has primarily focused on usability and feasibility of apps and has largely been published in the computer science literature [
Several systematic reviews highlight the lack of rigorous experimental and quasi-experimental trials; they also cite the lack of research measuring health outcomes as a key challenge to the advancement of the field of mHealth [
Therefore, in this paper, we will share our experience and lessons learned from the evaluation of a large-scale mHealth voice message service implementation in India, where
The Mobile Alliance for Maternal Action (MAMA) was a 4-year public-private partnership focused on harnessing the power of mobile technology to send stage-based health messages to pregnant women and new mothers. There were country implementations of MAMA in Bangladesh and South Africa followed by India and Nigeria. Thus, the implementation and evaluation in India built on the program design and lessons from the earlier MAMA implementations and research studies.
In 2015, the MAMA partnered with the nonprofit organization Advancing Reduction in Mortality and Morbidity of Mothers, Children and Neonates (ARMMAN) to implement a mobile phone–based voice messaging service in the urban slums of Mumbai, India, called mMitra [
A total of 145 voice messages were developed. The voice was female, and there were 2 languages available—Hindi and Marathi—the languages commonly spoken in the urban slums of Mumbai. The message development process was rigorous and involved BabyCenter, ARMMAN, and representatives from the Federation of Obstetric and Gynecologic Societies of India and the Indian Academy of Pediatrics. They were field tested with local health experts and community focus groups. The messages were delivered as a prerecorded phone call and covered the period from 6 weeks of pregnancy to an infant’s first birthday. A call-back service was integrated into the package for women to access within 2 days after the original call.
A MAMA research agenda was developed to provide a standardized approach to evaluating the impact of the intervention in different countries [
Mobile Alliance for Maternal Action theory of change and priority outcomes. ARV: antiretroviral; HIV: human immunodeficiency virus; MNC: maternal, newborn, and child; MNCH: maternal, newborn, and child health; PMTCT: prevention of mother to child transmission (of HIV); SMS: short message service.
The underlying hypothesized pathway to change for mMitra was that if women receive educational messages that are interesting, easy to understand, and aligned with their physiological state during pregnancy and post delivery, then they would be motivated to take the needed self-care and seek the needed health services. There was an assumption that women would also have access to other sources of similar information, such as community health workers who enroll them for pregnancy care; health care providers who treat them at health centers; and mass media messages on radio, television, and posters.
As the MAMA studies assessed behavior, it was important to collect data from real-world sources, triangulate the data as a method of data validation, and juxtapose the self-reported data with objective clinical data (see
Prioritized health and behavior outcomes for Mobile Alliance for Maternal Action programs in Bangladesh, South Africa, and India with data sources.
Prioritized Health Outcomes | Bangladesh | South Africa | India | |
|
||||
|
Hemoglobin levels/ anemia in mothers | —b | — | Clinic records |
Babies’ anthropometric measurements | — | — | Clinic records | |
CD4c count from mothers with HIV | — | Clinic records | — | |
WHOd stage of mothers with HIV | — | Clinic records | — | |
Tuberculosis status of mothers with HIV | — | Clinic records | — | |
Babies HIV test result | — | Clinic records | — | |
|
||||
|
Mother’s nutrition/diet/folic acid/iron tablets | Self-report | Self-report | Self-report |
Breast-feeding–exclusive and colostrum | Self-report | Self-report | Self-report | |
Antenatal care | Self-report | Self-report | Self-report | |
Gestational age at first ANCe | Self-report | Self-report | Self-report | |
Facility-based births | Self-report | Self-report | Self-report | |
Postnatal care | Self-report | Self-report | Self-report | |
HIV counseling and testing | Self-report | Self-report | Self-report | |
Antiretroviral therapy | Self-report | Self-report | Self-report | |
Immunizations | Self-report | Self-report | Self-report | |
Early detection and action for risk factors | Self-report | Self-report | Self-report | |
Empowerment/gender/self-efficacy | Self-report | Self-report | Self-report |
aMNHC: maternal, neonatal, and child health.
bNot applicable.
cCD4: Cluster of differentiation 4.
dWHO: World Health Organization.
eANC: antenatal care.
The lack of objective clinical data in the Bangladesh evaluation led to the collection of retrospective clinical data in South Africa [
Given the evolving technology landscape (ie, more affordable technologies and emergence of new technologies), evaluation studies are often not relevant by the time full-scale implementation occurs [
Whittaker et al’s detailed process on rigorous mHealth experimental evaluation approaches and types of measures that should be included [
Roess’ publication that outlines process evaluation and implementation science approaches that can be used to measure reach, fidelity of use, and the
The mHealth evidence reporting and assessment checklist, which is often used to guide reporting on the effectiveness of digital health [
We ultimately decided to use a pseudo-randomized controlled trial design to assess the impact of mMitra on the desired outcomes [
All women interested in participating in the study and implementation provided written informed consent. The study was approved by the Foundation for Research in Health System’s institutional review board (under protocol no. HHS00009235) and registered with ISRCTN (registration no. 88968111). Specially designed questionnaires were administered to women in the control and intervention groups at 3 time points: at enrollment in the study while the women were pregnant (baseline), after delivery, and when their baby reached 1 year of age. The surveys were conducted in-person with consent in Hindi and Marathi with the enumerator using an Android-based data collection platform. At the same in-person encounters, data were abstracted, with consent, from the woman’s Maternal and Child Health (MCH) card. The MCH card is issued to women by hospitals where they register for prenatal care. It serves as their health record, and they are instructed to bring the card with them on every visit for staff to update. Women are incentivized to keep their cards as it determines their eligibility to receive government compensation after delivery. Of the women interviewed post delivery 1040/1113 (93.44%) in intervention and 381/402 (94.8%) in control group showed their MCH card to the enumerators.
Pathway to change for hemoglobin levels in mMitra. ANC: antenatal care; Hb: hemoglobin; HH: household; KABP: knowledge, attitudes, behavior, and practices; MNC: maternal, newborn, and child.
Several key lessons were learned through this process. Major insights were incorporated into the study design based on the previous evidence and approach to research of other mobile messaging programs, the literature related to mHealth research, and the cumulative experience in research across the initial 2 MAMA country programs. This led to the adoption of a prospective pseudo-randomized controlled trial that proactively measured dose response based on the duration of exposure to messages by trimester and integrated objective clinical data with self-reported information.
As we had assumed that women would be exposed to external sources of information that would influence their behavior and practices, we prioritized real-world study designs over pure experimental designs. This pragmatic approach allowed us to have a comparison group and ensure that we would appropriately attribute any potential impact of the intervention on behavior and health outcomes, if any. We were also able to ensure that the research aligned with India’s health priorities by ensuring the study’s health outcomes matched the country’s priorities.
The triangulation of the data helped advance linkages between the intervention and clinical outcomes. Through this process, we learned that even though the health record (MCH card) seemed like a reliable data source because of government incentives, it did not preclude missing and poor-quality data. This was particularly true for maternal health data but less so for child health data. Only 24% of MCH cards in the control group and 34% of cards in the intervention group had entries for maternal hemoglobin—a key maternal health outcome of interest; whereas 65% of cards in both groups had anthropometric child health data that could be used to evaluate key child health outcomes. However, the data from the MCH cards were insufficient, and findings detected from the subsample of MCH cards with complete data were not statistically significant. This impacted our ability to use the data from the MCH cards to assess clinical health outcomes, but it provided an authentic, real-world perspective. It is acknowledged that the issue of missing data may vary by data element as well as by data source as well as use case and context.
From the outset of the study, we kept track of the duration of exposure women had to the messages. This was important to track to determine if a dose response was present. However, we also recognized that greater confidence could be placed in the message exposure findings by having both self-reported and electronic data sources. Subsequently, this brought attention to a future consideration—the inclusion of
With the increase in the digitization and use of decision-support and community-case management tools by frontline health workers, there are opportunities to further leverage existing data collection processes to validate and supplement self-reported and health record data. Furthermore, future assessments on the nature of the missing data may provide useful insights into how it can be addressed through statistical methods or through efforts made upstream to the data collection (eg, frontline health workers). As research moves toward real-world evidence and data generated through digital systems, it is important for the initial design to abide by data ownership and governance regulations, including mechanisms for obtaining consent for the use of routine health data to be used for research purposes as well as promoting data completeness and data quality. A rigorous focus on these aspects will allow evaluations to harness data more effectively and better lend themselves to high-quality research efforts that generate sound evidence.
The MAMA research experience, especially that of mMitra in India, provides important considerations for how to approach health outcomes research in evaluating the impact of an mHealth intervention. Having a clear understanding of the context and local health priorities is important to ensure that the research agenda is in alignment with local and national priorities and, thus, can have greater relevance to policy makers, program implementers, and beneficiaries. The inclusion of biomarkers and use of clinical records were an attempt to complement self-reported information on health outcomes. These opportunities are increasing as more health systems increase adoption of electronic medical records and shared health records. However, as our experience highlights, real-world data has its limitations—particularly in the areas of data completeness and quality. There is also an opportunity to use the data exhaust and meta-data generated from digital tools as an additional data source. Further study is recommended to assess the impact of missing data and the quality and viability of these types of records for future use in health outcomes–focused mHealth research. As more technology is adopted by health systems, the ability to move toward real-world evidence in the evaluation of digital health interventions and health interventions alike will increase—but great care and adoption of ethical practices will be required in the way that data are captured, structured, managed, analyzed, and shared.
Advancing Reduction in Mortality and Morbidity of Mothers, Children and Neonates
Mobile Alliance for Maternal Action
maternal and child health
mobile health
maternal, neonatal, and child health
The authors would like to thank Aparna Hegde of ARMMAN India for her support in providing access to the mMitra messages during the study and answering questions about the service throughout the research and writing process. They would also like to thank BabyCenter for information on the process of developing, reviewing, and testing the mMitra mobile messaging content. This study was funded by Johnson & Johnson. The funders contributed to critical review of the manuscript and decision to submit the paper for publication.
AG is employed by with Saati Health, Mumbai, India, although contributions to this work though were done in his personal capacity. SC is currently the Ethical Legal Social Issues lead in the
PNM provided overall oversight and contributed to manuscript writing. NM provided scientific and study oversight, helped design and analyze the study, and assisted in writing the initial draft of the manuscript. SC and NNK contributed to writing final versions of the manuscript. MPP performed all data analyses and contributed to data interpretation. AG and JP contributed to overall study design, data interpretation, and research concept. All authors critically reviewed the manuscript and approved the final version for publication.
JP is an employee of Johnson & Johnson, one of the funders of mMitra. No other competing interests are declared.