Implementation of a Technology-Based Mobile Obstetric Referral Emergency System (MORES): Qualitative Assessment of Health Workers in Rural Liberia

Introduction

Despite increased attention, funding, and innovation, maternal mortality continues to disproportionately affect low- and middle-income countries (LMICs) [1]. Liberia in West Africa has one of the highest maternal mortality ratios in the world with 652 deaths per 100,000 births [2]. This inequity is due to many factors including resource limitations, workforce shortages, and an overall unstable and underfunded health system following decades of conflict, the 2014-2016 Ebola epidemic, and COVID-19 pandemic [3]. A major contributing factor impacting Liberia’s high maternal mortality is a lack of standardized obstetric triage protocols and prehospital communication systems between health facilities [4]. Such systems allow for efficient patient transfer and have consistently been shown to improve maternal and neonatal health outcomes [5,6]. In Liberia, a lack of standardized triage, referral, and transfer systems leads to care delays and increases morbidity at the prehospital and intra-facility level, also known as the “third delay,” which occurs after patients arrive at health facilities [7]. This lack of infrastructure creates a critical blind spot for timely maternal care, especially for patients in rural areas who must travel hours on foot or by personal car to reach a health facility.

Over the last decade, the Liberian Ministry of Health has emphasized the importance of combating maternal and perinatal mortality at community and national levels [8]. Rural health facilities (RHFs) are community clinics established throughout rural areas of Liberia to serve as the first line of care for patients living far from district hospitals [9]. RHF staff, midwives, and nurses are the frontline obstetric health workers in Liberia, responsible for preliminary assessments, treatment and stabilization, and escalation of care by referring patients to district hospitals when necessary [10]. However, care delays persist due to system limitations. Verbal autopsy reports of 35 maternal deaths from Bong County, Liberia revealed 2 major contributors to maternal death: inadequate staff training and ineffective communication between RHFs and hospitals [4]. A promising solution to improve maternal transfer and prehospital communication systems leverages mobile health interventions (mHealth), including messaging platforms to transmit patient information from a referring facility to the hospital prior to arrival [11]. Such systems may be a promising intervention to reduce prehospital and intrafacility delays by allowing health workers to prepare resources for immediate intervention upon patient arrival [12]. Several mHealth approaches have been successful in reducing care delays in LMICs including Ghana, and a recent study by our team showed high desirability for such a program among 130 health workers from RHFs and hospitals in Bong County [13,14].

In response, our team developed and implemented the Mobile Obstetric Referral Emergency System (MORES) among 20 RHFs and 2 district referral hospitals in Bong County, Liberia. MORES has two main components as follows: (1) triage capacity building training among obstetric health workers, and (2) implementation of a mobile, WhatsApp-based communication platform, aimed at reducing prehospital and intrafacility care delays [15]. Specifically, the intended aims of MORES were to reduce care delays through improved prehospital transfer, increased health worker attentiveness, and improved patient care. MORES works in the following ways: when a pregnant patient presents to an RHF in critical condition, she will undergo standardized triage by a community nurse midwife to assess acuity. If the RHF midwife determines that this patient requires higher level or urgent care at the district hospital, the RHF midwife will send a templated WhatsApp message to the district hospital with information including the patient’s condition, past medical and obstetric history, any interventions administered, and method of transport to the hospital. Immediately upon receiving this message from the RHF midwife, hospital obstetric nurses begin preparing for that patient’s arrival including gathering needed medications, informing physicians, readying the emergency department, and preparing the operating room if urgent surgical intervention is anticipated. If delays emerge during patient transport, the RHF and facility providers can work collaboratively through MORES messaging to overcome these barriers. Upon arrival at the hospital, the referred obstetric patient will undergo an additional standardized triage by the obstetric nurses and receive a green, yellow, or red wristband corresponding to acuity level before admission or initiation of urgent intervention. Implemented in March 2022, MORES demonstrated significant improvements in triage knowledge and was used by more than 50 health workers to send 359 referral messages to transfer patients over the course of a year.

In this qualitative study, we evaluated the implementation outcomes of MORES including usability, fidelity, effectiveness of accomplishing intended aims, sustainability, and scalability. We also evaluated if MORES had additional impacts on the health system more broadly beyond its intended impacts.

Methods

Study Setting

This study took place in Bong County, Liberia, the third most populous and primarily rural county in Liberia [14]. Researchers at the University of Liberia and the University of Michigan partnered with the Bong County Health Team to design and implement MORES. Further detail of the MORES intervention, including its impact on health outcomes, is published elsewhere [15]. This study was conducted 1 year post implementation. Bong County was selected as the study site following efforts by the Liberia Ministry of Health to focus on improving maternal outcomes in this region given recent increases in maternal deaths [4]. Additionally, we have strong institutional partnerships between researchers and the County Health Team. Prior to MORES, Bong County had limited prehospital infrastructure consisting of 2 ambulances, no medical dispatch, and limited avenues for RHFs to communicate with hospitals. Health workers previously documented all medical information on a referral paper, which the patient was responsible for carrying and presenting to the referral hospital upon arrival. Often taking hours to arrive at said hospital, these papers would frequently be lost or damaged in transit.

Mores Implementation

In March 2022, specialists from Ghana and the University of Michigan with extensive experience in mHealth prehospital obstetric systems conducted capacity-building training among health workers from 20 RHFs and 2 district hospitals. Details regarding the templated communication messages and initial MORES training have been described [15]. Following this training, health workers returned to their respective health facilities to train colleagues in MORES. The 2 district hospitals were provided with smartphones to be used in the emergency department solely for MORES, while RHF workers used their personal cell phones. RHF and hospital workers were supplied cellular scratch cards to reload data at regular intervals throughout the study.

Study Design

To determine health worker perceptions of MORES implementation outcomes, we conducted a qualitative study using content analysis, following 1 year of MORES implementation in Bong County. First, US and Liberian experts in obstetric care and qualitative methods developed a structured interview script. The interview script was reviewed with Liberian health workers to adjust the language to the local context, purge repetitive items, and consolidate questions. The final interview script contained 20 items evaluating 6 major metrics including demographics, MORES purpose, usability, impact on referrals, facility linkages and feedback, and areas of strength and improvement (Multimedia Appendix 1).

Data Collection and Participants

In March 2023, 10 research assistants (RAs) from the University of Liberia traveled to Bong County to conduct interviews. The RAs consisted of both male and female Master of Public Health Students who had completed their course of study and were working on their Master theses and Master students who had completed a qualitative methods course. All were bachelor degree holders in various health-related fields and had extensive prior experience conducting qualitative interviews, which was a criterion for RA selection. During the data collection period, participants knew that RAs were students from the University of Liberia with an interest in obstetric health access and part of the research team who would be studying the implementation of MORES within their health system. A prior relationship with the study team had been established, given that this group was the same that implemented MORES 1 year prior.

In collaboration with the Bong County Health Team, RAs invited all health workers from the 20 RHFs and 2 district hospitals who had used MORES in the past year to participate using purposive sampling. Health workers were eligible if they were 18 years or older, currently employed by a Bong County health facility, and had used MORES at least once. Participants were excluded if they were unable to allocate sufficient time for an uninterrupted interview (10 minutes) or had never used MORES. RAs approached eligible participants face-to-face and invited them to complete the interview. Written consent was collected from eligible health workers who were willing to participate. It was made clear that the interview was anonymous, voluntary, and that their answers or decision of whether to participate would not influence their employment or ability to use MORES in the future. Individual interviews were conducted in a private location at the RHF or hospital according to participant preference. Each interview took approximately 10 minutes, and the interviews were audio recorded and stored in a secure lock box for transport to the University of Liberia. Researchers from Liberia and the University of Michigan met periodically throughout the data collection period to determine data saturation. Though saturation occurred after approximately 30 interviews, the opportunity to participate remained available so that all health workers who wished to share their perceptions were able to.

Data Analysis

Interviews were transcribed in Microsoft Word and immediately uploaded to a secure Dropbox folder only available to study researchers. All identifiable information was omitted during transcription. Two researchers began analysis through iterative immersion in the data, using content analysis with a deductive approach to categorize implementation outcomes of usability, fidelity, effectiveness, sustainability, and scalability to guide codebook development. Upon immersion in the data, it was determined that additional, unanticipated themes of MORES impact on the wider health system were frequently mentioned throughout the interviews. Therefore, a combined deductive-inductive content analysis approach was used to generate codes that categorized study aims while also accounting for the emergence of new themes. Once domains and major themes were identified, the codebook was further refined to identify subthemes under the major themes [16]. Transcripts were imported to NVivo (version 12; Lumivero) and coded using the codebook with a content analysis approach to answer the research question: “What are health workers’ perceptions on the implementation of MORES in light of its intended aims and impact on the wider health system?” [17]. Multiple measures were taken to ensure trustworthiness including (1) triangulation of interviews from multiple health specialists, (2) creation and validation of a consensus codebook, (3) inclusion of Liberian health workers as core members of the analytic team, and (4) member checking with Liberian health workers regarding key results.

Ethical Considerations

This study was approved by the University of Michigan institutional review board (HUM00195449) and the University of Liberia institutional review board (IRB00013730), whose protocols were followed throughout the entire study period. All participants underwent a process of written, informed consent prior to participation and all data were deidentified prior to analysis. This study adheres to COREQ (consolidated criteria for reporting qualitative research) guidelines for reporting in qualitative research [18].

Results

Participant Demographics

Sixty-two health workers from 19 RHFs and 2 district hospitals participated, representing 95% of total facilities taking part in MORES (Figure 1). Health workers from one facility included in the original training did not transmit any messages due to the lack of a smartphone with WhatsApp capability, therefore health workers from 19 facilities were included in our data collection. Participants included registered nurses, midwives, physician assistants, and medical doctors (Table 1). Three participants had additional specialization as neonatal specialists, and 12 occupied leadership roles, including nursing officer-in-charge (n=9), obstetrics supervisor (n=2), and chief medical officer (n=1). Most participants were between 35 and 45 years old, with years of professional experience ranging from 1 to 19 years (median 8 years, IQR 5-10). Across facilities, participants working in hospitals were slightly older and had more work experience than those in RHFs.

Four major domains were used to evaluate the implementation of MORES: (1) usability and fidelity, (2) effectiveness of intended impact, (3) sustainability and scalability, and (4) health system impact. Among these domains were 15 themes and 43 subthemes, outlined in Figure 2.

‎

‎

MORES Usability and Fidelity

MORES was evaluated for its usability and fidelity, meaning if participants used MORES as it was intended to be implemented. For usability, all 62 participants affirmed the MORES intervention was easy to use and beneficial to the patients they served (100%). They described MORES as easily integrated into existing workflows within their health system, as most had access to smartphones, knew how to use WhatsApp, and used MORES as part of a preexisting RHF-hospital referral system. For fidelity, participants described how they used MORES as it was intended to be implemented, including for making patient referrals, interfacility communication, and triage upon hospital arrival: “We get the patient’s name, age, reasoning of referral, mode of transfer. If referring through an ambulance or commercial car, we can place it in the WhatsApp group chat and send it” [P41].

Upon arrival at the hospital, patients were immediately triaged with a green, yellow, or red wristband according to acuity: “The band is there to help us to separate patients according to the conditions. The red is a critical condition; yellow is for someone that has a bad condition but not very critical; and the green is for normal patients” [P29]. Bands were worn until discharge, with health workers continually monitoring patients for changes in the category and exchanging wristbands appropriately.

All participants expressed that they used MORES for acutely sick patients at appropriate times. For hospitals, this was multiple times per day, given their high volume: “We use it all of the time, when the patient arrives in the facility. Any patient that comes in the labor room or the entry point, the ER” [P15]. At RHFs with smaller catchment areas, rates of emergency presentation were less frequent: “Sometimes in a month, we can get 4-6 [emergencies]. All depends on how the patients will come” [P31].

There was a clear understanding of the purpose of MORES from all providers and facilities, as every participant (62/62, 100%) accurately defined at least 1 goal of the program:

[MORES] is a quick emergency guide that will inform you where to refer, because you put the patient’s information on the WhatsApp platform and send it to the referral site. They will receive it and be preparing for the patient
[P56]

MORES Effectiveness

MORES was evaluated for effectiveness, meaning whether or not the program accomplished its intended purpose. These goals included improving prehospital transfer, increasing health worker attentiveness, and improving patient care and outcomes. Nearly all participants reported improved prehospital transfer (57/62, 92%). A major factor for improved transfer was that MORES established an avenue for rapid, prehospital communication between RHF and hospital health professionals, which facilitated closed-loop communication:

If [RHFs] communicate that they are sending a patient and we reply, ‘yes, we’re awaiting the patient’ then they are aware that we have received the message. Then we have to give them feedback that the patient came and arrived at this time. We send information like the patient came with this kind of condition and we’ve started management or the patient is improving or the patient is on treatment. Until that patient can be discharged, we still have to give them feedback.
[P1]

Participants believed that this communication resulted in more timely referrals from RHFs to hospitals (53/62, 85%). MORES also allowed health professionals to collaboratively overcome referral and transport facilitation barriers and informed clinicians on patient arrival times and conditions: “Before, people were not there to receive the patient on time. But now, if you send information, they will already be standing at the entrance waiting to receive the patient” [P48]. This led to a reduction in transfer time and intrafacility, or third delays, at hospitals, which was especially important for urgent cases requiring surgical intervention: “It has helped us limit delays from patients being referred. It informs the ER and OB Ward in times of emergency and gives us prior notice about a referred patient that has limited time for surgery” [P21].

Participants reported being able to prepare operating rooms and gather needed materials for urgent intervention before patient arrival, to more quickly evaluate and treat critically ill patients upon arrival: “As a health worker, it’s helpful in the sense that before my patient arrives, I’ve already received their information and setup. The moment they arrive, immediate intervention starts” [P6].

Second, health workers believed that MORES increased their attentiveness toward patients (39/62, 63%). One stated “It’s useful because it helps to keep us focused and makes us know that there’s an emergency that you need to act right away” [P9]. Another agreed that MORES trained her to recognize clinical deterioration faster: “Most often, if a patient comes, I will already know and be on guard to monitor because any time the patient’s condition can change” [P7].

This attentiveness was partially attributed to improved triage capacity following the triage training sessions and wristband implementation (35/62, 56%). One hospital health worker described the case of a woman with peripartum hemorrhage who received timely care because of the triage system. Another affirmed that the triage wristbands helped focus health workers’ urgency, with patients with red wristbands being seen first on clinician rounds (Table 2).

Finally, more than half of the participants (34/62, 55%) believed that MORES contributed to improving patient outcomes. They perceived MORES as helping to improve maternal and neonatal mortality, as well as patient satisfaction:

It has helped me to save lives that were to be lost, especially in pregnant women cases. If someone has a ruptured ectopic, before the person comes, we already know the person’s condition…and the person will receive the care in time.
[P14]

MORES Sustainability and Scalability

Interview responses reflected an overwhelmingly positive desire for MORES to continue and expand (62/62, 100%): “My recommendation is for the program to continue because it’s really helpful to us and reduces our maternal deaths” [P35]. Participants discussed the benefit of scaling MORES to other RHFs, who: “complained on why they are not part of the program,” and citing that, “there are some terrible cases that come from the other facilities that are not trained” [P6]. They believed benefits could be realized in standardization of medical care, reduction in error, and ultimately improvement in maternal and neonatal mortality if the program was scaled to other counties in Liberia. Participants directly cited MORES usability, feasibility, and effectiveness as justification for its scaling potential.

Despite successes, there were ongoing challenges to be addressed prior to scale-up, with 89% (n=55) of participants naming at least 1 issue. Challenges included financial support for cellular data (39/62, 63%), poor cellular networks preventing connectivity (21/62, 34%), and staff unavailability due to high turnover and limited motivation (19/62, 31%). RHF health workers were more likely to mention poor cellular networks than hospital staff, who worked in urban and suburban settings. Three participants also explained how conflict could arise between patient care and MORES, particularly when checking messages on a busy service: “Especially at the emergency unit, sometimes we are busy and forget to check the WhatsApp phone. If you have an emergency, you will be busy with that while other clinics are sending messages.” [P10].

From these challenges, participants offered suggestions for improvement. For workforce shortages, participants encouraged mentoring more staff to be MORES trainers to onboard new health workers: “If the knowledge of nurses can be increased and incorporating more nurses and increasing the training method, it will help support the program.” [P21]. Others suggested increasing staff motivation through cash incentives and providing facility phones to all RHFs with mechanisms for data recharging, rather than relying on personal phone use.

MORES Health System Impact

In addition to evaluating MORES implementation based on intended aims, there were multiple effects of MORES on the wider health system that were not part of the original design. In total, 61 (98%) participants described ways that MORES implementation strengthened aspects of the health system beyond the intended aims of implementation. Participants identified 6 ways in which MORES contributed to broader health systems strengthening (Figure 3). First, participants noted increased coordination and accountability among health workers and facilities when caring for patients (55/62, 89%). This was perceived through rapidly updated information and the ability to coordinate care between receiving hospitals which previously had limited communication:

I appreciate the program because care is on time these days and people are aware. When a patient is coming but C.B Dunbar is crowded and there’s no space, as long we get the message, we send it to Phebe for preparations. It helps a lot.
[P1]

Coordination and accountability were also experienced at a personal level, with health workers reporting increased responsibility to their patients, including their scope of care: “[MORES] got us to be on our guard and to know which case is above our limit and needs to be transferred to a bigger facility” [P54]. Workers felt more comfortable acknowledging personal limitations and used the MORES system to consult higher-level colleagues more frequently than prior.

Second, feedback mechanisms for hospitals and RHFs were a key benefit of the 2-way WhatsApp messaging (48/62, 77%). Hospital staff offered constructive feedback to RHF workers on their care and referral process. This feedback included the accuracy of diagnoses and opportunities for quality improvement:

Based on the feedback, we will know that we are doing good work or understanding the referral system. Using the paper-based system, getting feedback used to be difficult, but now it comes as soon as the patient arrives
[P37]

Feedback was rapid, with 89% of RHF health workers receiving updates within the first 24 hours (n=55). Such communication also facilitated improved interprofessional teamwork (21/62, 34%). Participants reported reduced work tensions between RHF and facility providers and improved interdisciplinary collaboration through network building: “It is helpful because my community health assistants and midwives are working together. If a patient is not willing to come, they will continue encouraging the patient” [P40]. RHF health workers appreciated that they could consult dozens of other professionals when managing complex patients, establishing consultation avenues for health workers who are often the sole clinicians working in a low-resource facility, and learning from the outcomes of one another: “I use it to refer, but also see if my colleagues are using it, because we all are trying to stop the maternal death rate in Liberia. I use it to see the outcome of my colleagues’ referral” [P47].

Fourth, MORES facilitated longitudinal follow-up care for RHFs (20/62, 32%), especially through patient tracking while still in the hospital. These updates helped family members unable to visit loved ones: “It makes us know more about our patients because the hospital will give [updates]. If the patient’s relatives don’t have a phone, we will be able to inform them on the status” [P35]. Hospitals additionally used MORES to communicate discharge instructions to RHFs for patients requiring follow-up. One RHF midwife elaborated on how MORES empowered her to help a patient continue her hospital-prescribed medication following discharge (Table 3).

‎

Fifth, MORES created a record of hidden care delays in Bong County by documenting reasons for delays as they arose (17/62, 27%). The most common included resource insufficiencies and worker capacity: “The purpose of [MORES] is to do quick referral and…to know where the delay comes from; whether from the community, the relatives or the midwife or from the county because they are responsible to send the ambulance” [P55]. In such situations, even when MORES worked properly, patients still experienced delays. More than half of respondents listed factors outside of MORES that worsened care delays for patients (35/62, 56%), including a lack of medications, intravenous fluids, and ambulance insufficiency.

Finally, MORES benefited the health system by creating a preliminary electronic health record (EHR) infrastructure (13/62, 21%). No EHR exists in Bong, and all documentation occurs on paper charts. Participants appreciated the ease, accessibility, and speed that MORES provided, believing it sped information dissemination:

[MORES] enables me to do my work with ease. The time I take to write a whole lot of papers…wastes the patient’s time, and it will also waste time at the referral hospital…With [MORES], the hospital will already get the information before the patient can arrive so they will start to attend immediately.
[P47]

This EHR also started a permanent record of patient conditions, transfers, and treatments administered, so that health teams could review patients requiring special accommodations or difficult cases. Finally, health workers appreciated the protections that having a permanent record of electronic communication offered in terms of liability for malpractice.

Discussion

Principal Findings

From this qualitative study of frontline health workers in rural Liberia, we found that implementation of the MORES intervention was perceived as having high usability, fidelity, and effectiveness for its intended aims. Furthermore, MORES had the unanticipated impact of strengthening multiple components of the health system and demonstrated exciting potential for scaling and sustainability. These findings confirmed the results of our preintervention study which showed high desirability and acceptability of the WhatsApp-based triage, transfer, and referral system among rural Liberian health workers to better care for obstetric patients [14]. Our analysis further revealed health worker perceptions regarding the benefits and health system improvement opportunities through MORES. These in-depth results contribute to the findings from other studies on the impact of MORES, including increased rates of cesarean section and lower odds of nonvigorous symptoms in neonates [15].

Users perceived MORES to be highly usable, as it could be seamlessly integrated into existing workflows and referral pathways. MORES use also demonstrated high fidelity, as it was implemented as intended and at appropriate times to refer critical patients. MORES participants reported the program to be effective across its intended aims: improving prehospital transfer, increasing health worker attentiveness, and improving patient care. The triage training and templated, bidirectional WhatsApp communication components of MORES worked collaboratively to address 2 key gaps in rural care delivery in Liberia: a lack of standardized obstetric triage protocols and a limited prehospital communication system. Increasing capacity and infrastructure to support triage protocols and prehospital emergency systems have been shown to significantly reduce patient morbidity and mortality, particularly in LMICs [19]. This opportunity for impact is especially prevalent for maternal mortality, where the third delay accounts for a massive burden of maternal and neonatal deaths, in some cases as high as 80% [20]. Participants were enthusiastic about the impact of MORES and desired its sustainability and scaling to other health facilities throughout Liberia. Continuing to implement and evaluate programs that increase capacity in obstetric triage, prehospital communication, and facility referral and transfer will be important for improving care and understanding the effects of mHealth technologies in resource-limited environments.

Though not an intended aim of implementing MORES, we found that the program facilitated health systems strengthening across 6 themes: coordination and accountability, feedback mechanisms for hospitals and RHFs, interprofessional teamwork, longitudinal follow-up care with RHFs, a record of hidden care delays, and establishment of a preliminary EHR infrastructure. While not the primary goals of MORES, these effects highlight important benefits across many internal and external domains of health system strengthening. For internal effects, participants reported MORES improved personal accountability, quality improvement opportunities, network building, and reduced workplace tensions and stress from liability. Externally, participants perceived faster information dissemination and facility preparation, consultation mechanisms, and care continuity. These findings highlight how MORES contributed to capacity building and the beneficial effects that can be realized when interventions are implemented to improve systems beyond individual disease states [21]. For example, while MORES was intended for obstetric and peripartum referrals, it exhibited the potential to be adapted to other specialties such as emergency medicine and surgery [22]. MORES provides a pragmatic example of how horizontally implementing interventions across multiple facilities, worker levels, and patient conditions can have multiplicative effects on health systems improvement.

This study adds to the growing literature on mHealth implementation in LMICs to reduce delays and improve care access. One comprehensive review of mHealth interventions for maternal health in LMICs showed that health worker buy-in and motivation, perceived intervention usability, and efficient messaging were key implementation-related mechanisms for success [23]. Our findings on MORES account for all 3, showing high usability and desires for sustainability and scaling, with efficient communication mechanisms that were used appropriately, bidirectionally, and rapidly. A study in South Africa demonstrated high usability for a mobile health platform to aid health workers in the management of pregnant women at risk for preeclampsia [24]. From this program, most user issues resulted from phone features including scroll wheels and touch screens. In MORES, we leveraged the free, WhatsApp communication platform already familiar to Liberian health workers, thereby mitigating user errors from unfamiliarity. Similar to our findings on the impact on interprofessional dynamics, other studies have demonstrated the collaborative effects of a WhatsApp messaging platform among obstetric providers, including as avenues for consultation, network building, and interprofessional communication [25]. Recently in Ghana, a WhatsApp messaging technology was implemented among 13 facilities and 81 health workers for more than 600 patient referrals from rural to district hospitals [26]. Findings from this study mirrored ours, including exposing hidden care delays and creating a preliminary EHR. Specifically, the program identified delay factors, referral characteristics associated with faster transport, and timely updates on patient arrival and outcomes more quickly and permanently compared with paper charting. A mHealth intervention among traditional birth attendants for indigenous women in rural Guatemala demonstrated increased referrals to facility-level care but similar complication rates across intervention and control groups [27]. Such inconclusive outcomes data suggest opportunities for future evaluation of maternal mHealth interventions. Results of a meta-analysis from 2016 revealed that out of 15 articles, only 2 studies showed a low risk of bias and 1 demonstrated a mortality benefit when text messaging women during pregnancy [28]. Future quantitative findings regarding the impact of MORES on maternal outcomes and referral times could clarify the potential clinical impact of well-implemented and highly acceptable maternal mHealth in LMICs.

Limitations

This study had several limitations. First, our qualitative methods and geographic homogeneity limit the generalizability of results. However, this approach was considered most effective given the study aims which allowed for in-depth explanations of MORES usability, fidelity, and effectiveness. Second, our sample had an unevenly distributed representation of health worker roles, with most participants being midwives and nurses. Increasing recruitment of physicians and other personnel who interact with MORES such as ambulance drivers, administrative staff, and patients being referred within the system could capture more in-depth data beyond what was offered by frontline health workers. Finally, we elected not to utilize a preexisting framework for the conceptualization of our results, given that commonly referenced usability, acceptability, and feasibility metrics for mHealth did not align with the evaluative approach and subsequent immersion in the data. This was apparent, especially for the impact of MORES on health system strengthening. To increase study rigor, we used a validated codebook and 4 separate metrics to ensure accuracy including member checking. However, developing flexible frameworks to better conceptualize usability, effectiveness, and broader benefits of mHealth technologies in LMICs could be highly valuable for future studies.

Conclusions

The MORES intervention was perceived to have high usability, fidelity, and effectiveness for reducing delays and optimizing referrals in rural Liberia. MORES also demonstrated 6 components of health systems strengthening which positively impacted accountability among health workers, feedback mechanisms, interprofessional dynamics, longitudinal follow-up care, and system-level efficiencies through recording delay factors and electronic health record infrastructure. Findings from this study suggest that MORES demonstrates high feasibility for scaling to other facilities and counties in Liberia to improve maternal and neonatal referral pathways. These conclusions can have direct policy and program implications for supporting MORES funding, implementation, and scaling at a national level. More widely, these findings can be highly useful to policymakers, clinicians, and researchers intending to implement and scale technology-based obstetric referral systems in low-resource contexts to accomplish intended aims, anticipate challenges prior to implementation, and enhance impacts on health system strengthening.