Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 13.02.20 in Vol 8, No 2 (2020): February

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/16081, first published Sep 18, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    Operability, Acceptability, and Usefulness of a Mobile App to Track Routine Immunization Performance in Rural Pakistan: Interview Study Among Vaccinators and Key Informants

    1Division of Woman and Child Health, Aga Khan University, Karachi, Pakistan

    2Department of Community Health Sciences, Karachi, Pakistan

    3Department of Health, Government of Sindh, Karachi, Pakistan

    4E-Health Resource Centre, Aga Khan Development Network, Karachi, Pakistan

    5Department of Medicine, Aga Khan University, Karachi, Pakistan

    6Department of Community Health Sciences, Aga Khan University, Karachi, Pakistan

    Corresponding Author:

    Shehla Zaidi, MBBS, MSc, PhD

    Department of Community Health Sciences

    Aga Khan University

    National Stadium Road

    Karachi

    Pakistan

    Phone: 92 2134864852

    Email: shehla.zaidi@aku.edu


    ABSTRACT

    Background: There has been a recent spate of mobile health (mHealth) app use for immunizations and other public health concerns in low- and middle-income countries. However, recent evidence has largely focused on app development or before-and-after effects on awareness or service coverage. There is little evidence on the factors that facilitate adoption of mHealth programs, which is critical to effectively embed digital technology into mainstream health systems.

    Objective: This study aimed to provide the qualitative experiences of frontline health staff and district managers while engaging with real-time digital technology to improve the coverage of routine childhood immunization in an underserved rural district in Pakistan.

    Methods: An Android-based app was iteratively developed and used for a 2-year period in 11 union councils of the Tando Muhammad Khan district, an underserved rural district with poor immunization coverage in Pakistan. We used iterative methods to examine the (1) acceptability and operability of the app, (2) validity of the collected data, and (3) use of the collected data. In addition, we collected the barriers and enablers for uptake of the mHealth app. Each of these topics was further explored related to changes in work as well as the enabling factors for and barriers to app use. In-depth interviews were conducted with the 26 vaccinators posted in the 11 union councils and 7 purposively selected key informants (government district managers) involved with the Expanded Program for Immunization. Findings were triangulated in line with the three broad research areas.

    Results: Digital immunization tracking was considered acceptable by vaccinators and district managers. Real-time immunization data were used to monitor vaccination volume, track children with incomplete vaccinations, develop outreach visit plans, correct existing microplans, and disburse a fuel allowance for outreach sessions. The validity of the app data was perceived to be superior to that of data from manual records. Ease of operability, satisfaction with data, personal recognition, links to field support, and a sense of empowerment served as powerful enablers. Taking twice the time to complete both manual and digital entries and outdated phones over time were considered constraints. An unintended knock-on effect was improved coordination and strengthening of Expanded Program for Immunization review platforms across district stakeholders through digitalized data.

    Conclusions: Embedding digital technology into mainstream health systems relies on use by both end users and district stakeholders. Ease of operability, satisfaction with data reliability, personal recognition, links to field support, and empowerment are powerful enablers, whereas improved coordination as a result of easy, transparent data access can be an important by-product of digitalization. Findings are relevant not only for wide-scale implementation of immunization tracking apps in Pakistan but also for informing the use of digital technology for results-based delivery by frontline health workers.

    JMIR Mhealth Uhealth 2020;8(2):e16081

    doi:10.2196/16081

    KEYWORDS



    Introduction

    There has been a recent spate of mobile phone–led programs, known as mobile health (mHealth) [1], to tackle health care issues such as immunization, tuberculosis (TB), and malaria in low- and middle-income countries (LMICs). Mobile phones are used by 97 per 1000 people in LMICs [2], reaching remote communities that previously had little interaction with public authorities or private companies [3]. In rural Tanzania, mobile technology has been used to provide the current inventory of anti-malarial medicine to improve access to drugs and other medical supplies [4]. In rural Uganda, an app called TB Detect enables health care providers to access TB-related educational material [5]. Another area where mHealth is being increasingly applied is routine immunization services in LMICs. SMS reminders about immunization and the use of electronic registries to trace cases who missed vaccination are being trialed in an increasing number of LMICs in Latin America and Sub-Saharan Africa [6-16]. There is some evidence that these programs improve vaccination completion rates [6,8-16]. However, many have not progressed beyond the pilot phase, and evidence for the effectiveness of digital solutions is limited [17].

    Despite the proliferation of mHealth apps, not all digital health programs perform as intended, and assessments of the technology, health systems, and behavioral factors need careful consideration [18]. In particular, mHealth programs that rely on the performance of health workers would benefit from qualitative research to understand the extent to which end users and decision makers are prepared to engage with mHealth technology [19,20]. Experience from high-income settings emphasizes a user-centric approach to the development of digital technology [21]. This is also an important consideration to assess the implementation of these technologies within LMICs. Technological factors such as internet access, data volume, and app usability can affect implementation [22]. In addition, other factors such as personal motivations for data use, work culture, and health system support should be reviewed when adopting mHealth programs.

    In Pakistan, 66% of children are completely immunized; this is much lower than the country target of 90% set by the Global Vaccine Action Plan [23]. Routine childhood vaccinations are provided by a dedicated cadre of government-employed vaccinators at static health centers and during outreach visits to villages >5 km from the health center. Poor vaccinator performance, which is exacerbated by poor district supervision, has been a chronic issue for Pakistan’s vaccine delivery system. A minimum of 1 vaccinator is allocated to each union council (UC), which is the smallest district administrative tier with a population of 10,000-20,000 people. Vaccinators are responsible for registering eligible children, providing vaccinations, updating Expanded Program on Immunization (EPI) cards, maintaining EPI records, and counseling parents for routine immunization awareness. The vaccinators are supplied with a motorbike, fuel support, and vaccines for routine immunization by the EPI program. UC–based EPI activities are administratively supervised by the district health office. However, paper records are poorly maintained and have questionable data. In addition, evidence suggests that vaccinators make too few outreach visits, provide little routine immunization education for parents, and maintain poor records [19]. Anecdotal evidence also suggests that vaccinators are resistant to supervision and often enjoy political patronage from local legislators protecting them from accountability on work performance [24]. In 2014, an app to track vaccinator movements through GPS was introduced across the populous Punjab province to ensure vaccinators made a sufficient number of outreach visits. Called e-Vaccs, it was designed, championed, and implemented by the government with provincially driven vertical accountability. Since then, policy commitment for digital immunization monitoring has also increased in other provinces, with the aim of effectively mobilizing the vaccinator workforce and moving away from relying on questionable paper-based records. However, a lack of the required systems support has stalled the roll-out process [20].

    In this paper, we report the end-user experiences with a mobile app (Teeko, Aga Khan University Pediatrics Department, Pakistan) to track the delivery of routine childhood immunizations as well as the enablers and barriers for implementation in local health systems. Teeko is a mobile app that tracks vaccinators’ routine immunization performance. It was co-designed with the sub-national government as part of a larger health system strengthening research program for routine childhood immunization and piloted in a rural district of Sindh, Pakistan in 2015-2017. The app generates quality real-time data, and its key features include GPS tracking of vaccinator outreach visits, digital records of the immunization volume at the static health center and during outreach visits, creation of the next scheduled immunization encounter, and identification of children who missed vaccination. We drew on the experiences of Teeko usage by vaccinators and district managers, including acceptability of the app, use of the digitalized data, perceptions of the data validity, and enabling factors and barriers for its adoption. We aim to inform how digital technology can be embedded within district health systems in LMICS.


    Methods

    Overview

    Our study explored the experiences with an Android-based immunization app to improve vaccination coverage in a rural, disadvantaged district of Pakistan. The overall objective was to evaluate the experiences of vaccinators and district managers during their use of the app to track routine immunization encounters. The specific objectives were to determine the acceptance and operability of the app as a tool for tracking vaccine encounters within the district health system, assess the data validity and data-related concerns of stakeholders, and understand how the technology is being used within the vaccine delivery system.

    Setting

    The mHealth initiative was part of a larger implementation research program to strengthen the health system and routine childhood immunization conducted by the Aga Khan University (AKU) in collaboration with the provincial EPI in the Tando Muhammad Khan (TMK) district in Sindh Province in 2015-2017 (Figure 1). This district had poor vaccination performance, with a Penta coverage rate of only 23% and PCV coverage of only 11% at the start of the study. A package of integrated interventions including a digital app was implemented based on formative research to identify interventions that was conducted at the start of the study. In addition to the mHealth intervention, other complementary interventions included facilitation of a district EPI review platform, microplanning training, fuel support routed through the district health office for outreach activities, and co-financing of EPI motorbikes for UCs lacking functional bikes for outreach sessions. Interventions were implemented in 2 district sub-divisions (Talukas) comprised of 11 UCs. The Teeko intervention was implemented over 24 months, after which vaccinator and district stakeholder interviews were conducted. The immunization program resulted in a significantly higher number of completed vaccinations at the intervention sites than at the control sites, as measured in terms of PENTA3 and PCV3 coverage (Table 1).

    Figure 1. Map of the Tando Muhammad Khan (TMK) district in Sindh province, Pakistan, including locations of union councils (UCs) and Expanded Program on Immunization (EPI) centers.
    View this figure
    Table 1. Improvement in the vaccine coverage rate from 2014 to 2017 during the Health Systems Strengthening (HSS) immunization pilot study [19].
    View this table

    mHealth Intervention

    Vaccinators and their sub-district and district EPI supervisors were provided with smartphones to track the immunization encounters. The app contained a list of UCs, villages, and health facilities assigned to each vaccinator. Vaccinators could access the app via unique access numbers to enter immunization data for the villages listed in their UCs (Figure 2). The app tracked vaccinator visits, vaccination volume at the static health center and outreach points, identified children that had missed a vaccination, tracked vaccine availability, and provided communication features for parents. The app had two components: data entry and a Web portal for data visualization.

    Figure 2. Vaccinator using the Teeko app.
    View this figure

    The immunization app was developed over a 6-month field testing period in 2014-2015 and was implemented in 2015-2017. The principal investigator (SZ) conducted a baseline assessment of routine immunization delivery in the district and identified the reporting of incomplete, questionable immunization EPI data and poor use in the TMK district [19]. During the assessment, management information system records were reviewed, and EPI officials, vaccinators, health workers, and health facility managers were interviewed. Further follow-up meetings were held with EPI provincial staff, the district health team, and vaccinators by the principal investigator (SZ) and a research specialist (SH) to identify intervention points within the EPI record flow that could be supported with an immunization tracking app, the support required for vaccinators and district health officers to use the app, and the health system interventions required to facilitate decision making. An in-house technology firm (Aga Khan Development e-Resource Centre) was contracted for digital programming. App design and development were overseen by a steering committee chaired by the Provincial Director General of Health and comprised of the study principal investigator (SZ), a research specialist (SH), UNICEF, WHO, the EPI focal person for the TMK district, the District Health Officer for TMK, and a representative from the technology firm (SS). Using a participatory process, the following app features were identified: compulsory photo identification; defaulter (children who missed a vaccination) status using traffic light colors; alignment with EPI’s management information system; client data to include the parents’ national ID card numbers, household number, and village details; and adjustment for community migration. Meetings were held during the design phase and continued into the app roll-out, with recorded minutes. Iterations included the addition of new vaccines, identification of areas of misreporting, adjustment for vaccinations received outside the study area, and adjustment for household migration. To improve the technological features, the study team and technology partner field-tested different versions of the app with vaccinators and the district EPI focal person. Due to poor internet connectivity, the ability to upload data offline was added after the field testing. SMS messaging was poorly received and replaced with a robotic call. The study team randomly checked the photo entries and uploaded data for errors.

    Finally, the app features included registration, immunization encounter record, offline data mode, vaccine stock management, awareness content, text messaging, a central database, and a Web portal.

    Registration

    The vaccinator registered the child at the first encounter, entering the child’s father’s name, child’s date of birth, household phone number, village name, and UC (Figure 2). This created an individual profile for the child. The child registration form was the same as the manually completed EPI child registration form. The data for the registered child was synced with the central database, in which all data were stored.

    Immunization Encounter Record

    The vaccinator then vaccinated the child and uploaded a mandatory photo of the vaccination or updated EPI card (Figure 3) as proof of vaccination. A GPS-based location was generated for the vaccination site. The app sent the immunization record data to the central server in real time.

    Figure 3. Examples of registration, an immunization record, and the image capture view on the Teeko app.
    View this figure
    Offline Data Mode

    The app’s offline feature allowed vaccination data to be recorded when internet connectivity was not available. The records were stored locally in the app and uploaded once an internet connection was restored during the outreach visit or at the health facility. Vaccinations could occur within the health facility or during an outreach visit.

    Vaccine Inventory Management

    To track inventory and ensure timely pick up of required supply, vaccinators could enter the amount of vaccine they had with them as well as the inventory at their respective health facilities.

    Awareness Content

    The app had certain communication features. A 60-second awareness video on routine immunization could be played by the vaccinator to educate the parents during a vaccination encounter.

    Text Messaging

    Following the vaccination, robotic calls and SMS messages about the routine immunization schedule and next visit were sent to the caregivers of each registered child.

    Central Database

    The central system (Figure 4) allowed the administrator to register all users (vaccinators, lady health workers, and viewers). In addition to during the vaccination encounters, the app occasionally sent the GPS coordinates of the vaccinators to the central database (Figure 5). If internet connectivity was not available, the coordinates were stored in the local database and later uploaded to the central database. The vaccinator directory listed the UC assigned to each vaccinator.

    Figure 4. Flow of data through the Teeko app and Web portal.
    View this figure
    Figure 5. Vaccinator tracking screen in the Teeko Web portal.
    View this figure
    Web Portal

    Through the Web portal, the management and supervisory staff remotely monitored vaccinators’ activities and movements in real time (Figure 5). The SMS Panel allowed the administrator to send an SMS to the vaccinators and lady health workers about meetings, schedules, and data corrections. The portal also generated district and UC reports on routine immunization coverage, defaulters, and individual vaccinator performance. These reports were generated based on formats already in use by the EPI Program.

    Data Collection

    We conducted semi-structured interviews with vaccinators and key informants (district managers) involved with EPI delivery (See Table 2). Data were collected and analyzed by trained researchers.

    Table 2. Themes and tools used to explore the user experiences with digitalized immunization tracking and the key enabling factors
    View this table
    Vaccinator Interviews

    Semi-structured interviews were conducted covering the following topics: frequency of use of the app features; operability of and issues using the digital app; perceptions of the data validity and comparison with that of manually collected data; engagement with the app; and enabling factors and barriers for data use. Interviews were conducted with all 26 vaccinators working in the 11 UCs in which the digital app was implemented. All the vaccinators used the immunization app for at least 24 months. Individual interviews were conducted with each vaccinator in the local language and the privacy of a separate room.

    District Stakeholder Interviews

    In-depth interviews with key informants (district stakeholders) included the following topics: acceptability of the app to track immunizations conducted by the district heath team, perception of the data validity, and use of the information by district health management. For each topic, we also asked about how their work had changed after using the app and the enabling factors and barriers for app use. The topic guide was written in English and translated into the local Sindhi language. Interviews lasted 45 to 60 minutes and were conducted in the local language. Stakeholders from the district administration and district health office were purposively selected based on their key roles in EPI oversight or delivery. A total of 7 district stakeholders were interviewed, including the Deputy Commissioner (administrative head of the district and chair of the Polio-EPI district committee), Assistant Deputy Commissioner, District Health Officer, Assistant District Health Officer, District EPI Focal Person, District Superintendent of Vaccination, and Coordinator of the Lady Health Worker Program (rural community-based female health workers that assist with the awareness provision for routine immunization and mobilization of households for routine immunization sessions provided by vaccinators).

    Ethical Considerations

    Ethical approval for the study was obtained from the Ethical Review Committee of Aga Khan University Karachi, Pakistan (ERC number: 2818-Ped-ERC-13). Informed consent was obtained from all participants. Interviews were conducted on a voluntary basis. Each participant was ensured of the confidentiality of personal information. Personal information was anonymized for analyses and reporting through a coded interviewee number. The recorded data were kept in locked drawers and encrypted laptops to ensure data confidentiality.

    Data Analysis

    The interview information was transcribed in Sindhi and translated back into English. These were then checked for errors and manually analyzed. Three a priori themes were identified in line with the research objectives and used as the basis for inductive analysis: acceptability and operability of the app, data validity, and use of the app data (Table 2). Content analysis of the transcripts was undertaken, developing main codes in line with the a priori themes and further detailed coding based on grounded findings emerging from the narrative.


    Results

    Interview guides were used to report the results from the in-depth interviews. Textboxes 1 and 2 highlight the important findings from the vaccinator and district management interviews, respectively.

    Vaccinators’ End-User Experience

    Table 3 lists specific responses provided by vaccinators during their interviews.

    Acceptability of the App
    Use of App Features

    All 26 vaccinators used the app to record their vaccination encounters. Nearly all vaccinators (n=24) took post-vaccination photos of the children. Although some parents initially refused the photos, they eventually agreed after the vaccinators explained that the photo would be used for immunization verification and recording. Only two vaccinators reported that parents would not allow a photo under any circumstances, in which case a photo of the vaccination card was uploaded instead. These vaccinators were assigned to UCs that had pockets of the population with reports of prior vaccine refusal.

    All the app features for routine immunization awareness including the awareness video, SMS alerts, and robotic calls were used by 18 vaccinators. Only the awareness videos were used by 6 vaccinators, and 2 vaccinators used only the SMS alert and robotic call features.

    Operability

    Of the 26 vaccinators, 25 reported that the app was easy to use and they could operate it without difficulty (Table 3). One vaccinator, who had a stroke-induced disability, needed his son’s help to enter the data in the app. However, he also required assistance to manually complete data entry. Real-time data display in the format of the government EPI report was considered helpful for reading and relating to the data.

    Vaccinators reported no instances of data loss from the app. Internet connectivity issues often occurred in the more remote villages, but the app allowed offline data entry. Then, vaccinators uploaded the data after they returned to the facility. The offline feature was particularly appreciated for its ability to prevent data loss (see Figure 4).

    However, a few vaccinators reported being uncomfortable using the offline option and would have preferred real-time information uploads to the central database. Their reported concern was they might forget to upload pending data or be unable to upload pending data if the phone was damaged or stolen. However, there were no reports of phone damage or loss during the study.

    Another reported issue was that, as the phones aged over the 24-month period, the phones would hang or freeze while using the app, making it difficult to enter data. Often, the phone had to be restarted, and there was a concern that unsaved data would be lost. The vaccinators wanted a replacement phone at least every 2 years to continue using the app.

    Validity of the App Data

    Most of the vaccinators (n=23) reported a high likelihood of error with manual documentation of outreach vaccination encounters. In practice, the data were entered based on recall in the EPI registers at the health care facilities rather than on site. Vaccinators recounted poorly recorded field data during prior measles outbreaks (before the Teeko app was available), resulting in an inability to target defaulters. They perceived that instant data entry through a mobile app generated a more accurate defaulter list.


    Textbox 1. Important findings from the vaccinator interviews.
    View this box

    Textbox 2. Important findings from the key informant interviews with district management.
    View this box
    Table 3. Specific responses from the vaccinator interviews, organized by theme.
    View this table

    The post-vaccination photograph was considered key to ensuring reliable data. Compared with the prior practice of checking the EPI records, the advantage of the photo feature was child verification. Vaccinators also reported that the GPS tracking of outreach visits resulted in greater vaccinator vigilance in conducting outreach sessions and improved the reliability of vaccination encounter reports. Vaccinators reported calls from their supervisors to check their location when the supervisors did not observe any field activity.

    Use of the App Data

    All vaccinators reported they mainly used the digital data to identify children overdue for their vaccinations. Vaccinators expressed ownership of the data and a feeling of empowerment when using the data to plan their monthly vaccination rounds. They also reported using the data to set target vaccination volumes.

    All vaccinators also mentioned the use of the GPS feature by the district health office and deputy commissioner for vaccinator supervision. Vaccinators reported being questioned by the district supervisors if they turned off their mobile location or could not be seen moving in the field.

    The vaccinators reported that the overriding motivator to use the app was the reporting of the performance level of each vaccinator. The vaccinators reported feeling gratified when the best performing vaccinators were publicly recognized during the review of individual performance levels in the monthly EPI meetings. All the vaccinators found it very satisfying to be noticed and praised by superior EPI officials and district officers. Most of the vaccinators reported that this motivated them to do their best. Others reported this was the first time they took their work seriously and found it a novel but satisfying experience to work toward targets rather than working to identify loopholes to avoid work.

    A few vaccinators reported that the app increased their workload because they were also expected to manually document the EPI records in addition to documenting via the app.

    Migration to digital records from manual recording was preferred by 23 of the 26 vaccinators, while 3 vaccinators wanted both manual and digital record keeping. All 26 vaccinators were willing to continue using the app in the future.

    The vaccinators also mentioned the conditional fuel support provided by the district health office based on the number of monthly visits reported by the app. The vaccinators reported that the lack of fuel support made it difficult to conduct outreach sessions.

    District Stakeholder Experiences

    Findings from the key informant interviews with district stakeholders were categorized into the following thematic areas: acceptability of the app for routine immunization supervision by the district heath team, perception of data validity, and use of the app information by the district health management. Within each thematic area, the narrative reports were analyzed for changes in work after using the app, experiences with implementing the app, and the enabling factors for and barriers to app use. Table 4 lists specific responses provided by the district stakeholders during their interviews.

    Table 4. Specific responses from the district stakeholder interviews, organized by theme.
    View this table
    Acceptability of the App

    Key informants from district management commonly reported that the vaccinators were comfortable using the app. District officials found it easy to retrieve routine immunization volume from their cell phones and computers, and the officials used the app to keep track of children who missed immunizations. The interviewees recounted poor vaccinator performance prior to using the app and attributed this to poor work attitudes, insufficient supervision from the supervisors, and a lack of fuel support and functional motorbikes for supervisory visits.

    The stakeholders reported that the app led to increased coordination between the EPI and Lady Health Worker Program—the two important community outreach resources within the district heath system—and closer interaction regarding routine immunization between the District Health Office and the Deputy Commissioner’s Office.

    The health officials expressed an eagerness to continue the tracking initiative and anxiety that routine immunization coverage would decline again after closure of the project. The stakeholders also requested that the project be expanded to the other sub-divisions within the district. The district leadership reported feeling as if they could serve as an example for other districts in the province and were gratified that other districts requested demonstrations and sharing of lessons learned.

    Validity of the App Data

    Key informants reported that tracking a vaccinator in the field was challenging in routine practice. Immunization work plans provided by the vaccinators were not always accurate; they could make changes to the plans without prior notice to the district health office. Moreover, even when EPI supervisors verified the vaccinators’ locations in the field, they needed assistance from local villagers familiar with the area. It was still possible that they would not successfully find the vaccinator. The district stakeholders appreciated that the app facilitated vaccinator tracking through its GPS feature. They were able to locate and reach vaccinators without delay.

    The EPI district supervisors reported that they relied on the app data to provide supervisory instructions for the vaccinators related to routine immunization coverage and defaulter reports. The app data were considered superior in quality to manually collected data and were particularly used for corrective actions.

    Use of the App Data

    The key informants commonly reported that the main uses of the app data were improving the number of routine immunization encounters and locating defaulting children. In the past, it was challenging to hold the vaccinator cadre accountable for routine immunization performance, making it difficult to improve vaccination volume. Furthermore, disciplinary actions such as a “notice of explanation,” for which employees must provide an explanation for an allegation of violation of company policies, rules, and procedures; “show cause letter,” which asks the employee to provide a reason why they should not face a disciplinary action for a conduct or capacity issue in the workplace; and termination of irregular, ghosting, or nonperforming vaccinators were not effective due to the political patronage of vaccinators by local legislators.

    The key informants mentioned that regular EPI review meetings were convened at the district health office, and the meetings used the real-time app data. These meetings were often attended by the district health commissioner and assistant commissioner. During these meetings, vaccinator performance was reviewed, including that of poor performers. Vaccinators with high performance were also recognized separately in quarterly public ceremonies attended by district legislators. The stakeholders reported that the public recognition was instrumental for increasing motivation. They added that the review meetings provided a platform for vaccinators to learn from each other’s experiences and promote healthy competition.

    District stakeholders also expressed a sense of empowerment from using the app to make operational decisions to improve routine immunization in the district. They used the data to direct activities to areas with incomplete vaccination coverage. Real-time immunization data were also used to provide support for vaccinator mobility based on GPS-verified outreach visits.

    Additionally, EPI motorbikes were provided by the district health office to the better performing vaccinators; these were co-financed by the project and the vaccinator. Both activities were undertaken during the monthly district EPI review meetings that were chaired by the District Health Officer.


    Discussion

    In LMICS, mHealth apps are increasingly used for immunizations and other public health concerns; however, recent evidence has largely focused on app development, dashboards, or the before-and-after effects on awareness or coverage [6,7,10,12,21-25]. Less evidence is available regarding end users’ and decision makers’ acceptance of and engagement with digital health technology. Bridging this evidence gap is critical to embed digital technology into mainstream health systems. In this paper, we report the qualitative experiences of frontline health staff and district managers in engaging with real-time, digital technology to improve the coverage of routine childhood immunization in an underserved rural district in Pakistan.

    Using an iterative approach, we gathered information about the acceptability of the app for data collection and tracking vaccine delivery, perceptions about data validity, the extent that app data were used, and the engagement of care. We also identified the key factors for frontline staff to engage with digital technology. The app was commonly accepted by vaccinators and district supervisors to track vaccination encounters; however, the app features aimed at increasing parents’ vaccination awareness were less commonly used. Lack of parental agreement for photograph verification of their children was not a major issue. Vaccinators found the app easy to operate, the fields were similar to those of EPI records, and the offline data recording feature was helpful in remote access areas. Double entry of both manual and digital records was considered time consuming, and aging phone technology created systems issues. District supervisors found the app data easy to access for quick verification of immunization activity and to provide supervisory direction. The ability for real-time tracking of both individual vaccinators and the entire team through the Web portal considerably reduced the time spent tracking vaccinators during outreach visits. This helped to not only accurately track the vaccinator but also identify vaccination coverage according to the geographical location. This drastically reduced the monitoring and tracking time during outreach. The increased availability of supervision time was then utilized for other monitoring and evaluation activities. In addition, the app-generated maps helped to finalize microplans and identify missed locations. The validity of the app data was considered superior to that of data from manual records by both the vaccinators and district supervisors; this perception was attributed to photo verification of the encounters. Opinions differed regarding what contributed to the improved data. The EPI supervisors attributed it to the transparency of vaccinator movements, while vaccinators felt that the ability to record immunization entries during the visits was better than using recall to manually record data later at the health facilities.

    Real-time immunization data were mainly used to monitor vaccination volume, track children with incomplete vaccinations, develop outreach visit plans, alter existing microplans, and provide a results-based fuel allowance. A significant outcome was the initiation of regular monthly EPI review meetings by the district leadership. During these meetings, real-time data were used to review immunization progress. Data were also used to recognize both good and poor immunization performance per individual and across UCs and vaccinators. These findings are supported by those from a study in Nigeria that used GPS and geospatial data to track vaccinators [26,27]. They reported improvements in microplanning, monitoring, and immunization coverage, which were attributed to the technology-based support for performance monitoring that reduced erroneous entries and data fabrication by vaccinators. They also used GPS data to provide feedback to the outreach team and target missed geographical areas.

    Our study also identified enablers that facilitated the use of digital immunization tracking. Recognition of individual performance, empowerment, and results linked to fuel support emerged as powerful motivators for the vaccinators to work towards target delivery rather than avoiding workload, as they had previously. For district management, key enablers included a sense of empowerment for reviewing and planning EPI delivery and enhanced transparency to counteract political patronage of vaccinators. A knock-on effect of the real-time verification data was improved coordination across different components of the district health systems.

    Reviews of digital technology use in other settings have drawn on mixed methods including interviews, focus group discussions, and systems analysis of the app data. The evidence indicates that instant data availability for monitoring purposes, as observed with neglected tropical diseases [28], and ease of use and digital literacy of end users, as observed with oral cholera vaccinations [29], are important parameters for the successful trialing of digital apps. Poorly designed devices and inadequate cellular infrastructure have been noted as barriers for integrating apps into health systems [30]. Lessons learned from immunization registries in Latin American and Caribbean countries indicate that digitalization must be useful to vaccinators for the data to be of good quality. Evidence on non-technological factors for health provider apps is still emerging. A study of a client data app for community health nurses in Ghana resulted in reasonably good acceptability due to the app’s capacity to facilitate client follow-up and data reporting; however, the feasibility and usability of the app were hindered by high client volumes, staff shortages, and software and device challenges [31]. A study on an mHealth app for midwives in Ghana found low levels of user acceptance and a software design that did not match the end‐user needs or work environment [32]. Another study that introduced an app for community health volunteers in Kenya found that acceptability improved after a period of initial anxiety; however, there were feasibility challenges related to battery drainage and difficulty keeping the phone charged [33]. Yet another study in South Africa with an app for mHealth workers to electronically track patients with multi-drug resistant TB found a high intent for use but low actual usage resulting from forgetfulness and low levels of responsibility for such work [34].

    Our study provides timely evidence on the successful integration of an immunization tracking app to be used by vaccinators as Pakistan prepares to upscale digitalized immunization tracking. It also provides lessons for LMICS on technological and non-technological factors that require attention to contextualize health provider apps within the local health system. We contend that immunization performance tracking through digital technology needs essential acceptance by frontline health force, and mere vertical enforcement is not enough. Technological ease of operability, personal recognition, results-oriented mobility support, and empowerment to improvise microplanning are powerful individual-level factors, whereas data transparency and empowerment for district planning are key organizational-level enablers to embed digital tracking in vaccine delivery systems. While there is a proliferation of digital technologies in countries such as Pakistan, dedicated investment in e-governance for continuous, independent, rigorous evaluation is required to assess actual use in the field and validity of the data.

    Strengths and Limitations

    This study used iterative qualitative research methods for in-depth exploration of the digital technology interface with end users and decision makers within the health system. This iterative approach helped identify themes in an unexplored area and identified details that would have been missed in a more prescriptive quantitative study. Triangulation of the findings across vaccinators, supervisors, and district managers strengthened the analysis and helped draw out common convincing narratives. To limit bias, researchers who were not part of the intervention team undertook tool development and data collection.

    A limitation of this study is the lack of quantitative assessment of the data generated by the app. A household vaccination coverage survey conducted as part of the larger health system strengthening immunization study found a significant difference in vaccination completion rates between the baseline and end timepoints across study control and intervention areas [35]. However, the app’s impact cannot be separately quantified from the other components of the larger health systems intervention package. While several of the findings are specific to the study context, the study provides the key parameters for consideration during the current rollout of digital immunization interventions in Pakistan as well as lessons for the introduction of a health provider app in LMICS. Digital technology investments must be accompanied by independent rigorous evaluation for relevance to health systems.

    Conclusions

    Embedding digital technology into mainstream health systems relies on acceptance by both end users and district stakeholders. Ease of operability, satisfaction with reliable data, personal recognition, results-based field support, and empowerment are powerful enablers. An important by-product of digitalization can be improved coordination as a result of transparent, easy data access. The findings are relevant to not only the current upscaling of digital technology to track immunization in Pakistan but also the application of digital technology for results-based delivery by frontline health workers.

    Acknowledgments

    We thank Dr Hassan Murad Shah Director General Health Sindh; Zahoor Baloch, EPI Programme Sindh; Dr Nazir Baloch, Department of Health Sindh,; Dr Hafeez, Pasha District Health Office, Tando Muhammad Khan; Agha Abdul Rahim, DCO, Tando Muhammad Khan; Abdul Muqeet; and Tehniat Sheikh, Aga Khan Development Network E-Health Resource Centre. This work was supported by GAVI-The Vaccine Alliance (PP33961213A1).

    Conflicts of Interest

    None declared.

    References

    1. World Health Organization. Decade of Vaccines―Global Vaccine Action Plan 2011-2020   URL: https://www.who.int/immunization/global_vaccine_action_plan/DoV_GVAP_2012_2020/en/ [accessed 2020-01-10]
    2. USAID. 2003 Oct. Immunization Essentials: A Practical Field Guide   URL: http://pdf.usaid.gov/pdf_docs/Pnacu960.pdf [accessed 2020-01-10]
    3. Stansfield SK, Walsh J, Prata N, Evans T. Chapter 54: Information to Improve Decision Making for Health. In: Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, editors. Disease control priorities in developing countries. New York, NY: The World Bank and Oxford University Press; 2006:1022.
    4. Barrington J, Wereko-Brobby O, Ward P, Mwafongo W, Kungulwe S. SMS for Life: a pilot project to improve anti-malarial drug supply management in rural Tanzania using standard technology. Malar J 2010 Oct 27;9:298 [FREE Full text] [CrossRef] [Medline]
    5. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Health Sci Pract 2013 Aug;1(2):160-171 [FREE Full text] [CrossRef] [Medline]
    6. Bangure D, Chirundu D, Gombe N, Marufu T, Mandozana G, Tshimanga M, et al. Effectiveness of short message services reminder on childhood immunization programme in Kadoma, Zimbabwe - a randomized controlled trial, 2013. BMC Public Health 2015 Feb 12;15:137 [FREE Full text] [CrossRef] [Medline]
    7. Domek GJ, Contreras-Roldan IL, O'Leary ST, Bull S, Furniss A, Kempe A, et al. SMS text message reminders to improve infant vaccination coverage in Guatemala: A pilot randomized controlled trial. Vaccine 2016 May 5;34(21):2437-2443 [FREE Full text] [CrossRef] [Medline]
    8. Haji A, Lowther S, Ngan'ga Z, Gura Z, Tabu C, Sandhu H, et al. Reducing routine vaccination dropout rates: evaluating two interventions in three Kenyan districts, 2014. BMC Public Health 2016 Feb 16;16:152 [FREE Full text] [CrossRef] [Medline]
    9. Kaewkungwal J, Singhasivanon P, Khamsiriwatchara A, Sawang S, Meankaew P, Wechsart A. Application of smart phone in "Better Border Healthcare Program": a module for mother and child care. BMC Med Inform Decis Mak 2010 Nov 03;10:69 [FREE Full text] [CrossRef] [Medline]
    10. Kazi A, Murtaza A, Kazi A, Khurram Z, Hussain K. Effect of mobile phone text messages reminders on uptake of routine immunization among children in Pakistan-study protocol for a randomized clinical trial. J Vaccines Vaccin 2016;7(333):2. [Medline]
    11. Kazi AM, Ali M, Zubair K, Kalimuddin H, Kazi AN, Iqbal SP, et al. Effect of Mobile Phone Text Message Reminders on Routine Immunization Uptake in Pakistan: Randomized Controlled Trial. JMIR Public Health Surveill 2018 Mar 07;4(1):e20 [FREE Full text] [CrossRef] [Medline]
    12. Lund S, Nielsen BB, Hemed M, Boas IM, Said A, Said K, et al. Mobile phones improve antenatal care attendance in Zanzibar: a cluster randomized controlled trial. BMC Pregnancy Childbirth 2014 Jan 17;14:29 [FREE Full text] [CrossRef] [Medline]
    13. Nguyen NT, Vu HM, Dao SD, Tran HT, Nguyen TXC. Digital immunization registry: evidence for the impact of mHealth on enhancing the immunization system and improving immunization coverage for children under one year old in Vietnam. Mhealth 2017;3:26 [FREE Full text] [CrossRef] [Medline]
    14. Schlumberger M, Bamoko A, Yaméogo TM, Rouvet F, Ouedraogo R, Traoré B, et al. [Positive impact on the Expanded Program on Immunization when sending call-back SMS through a Computerized Immunization Register, Bobo Dioulasso (Burkina Faso)]. Bull Soc Pathol Exot 2015 Dec;108(5):349-354. [CrossRef] [Medline]
    15. Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine 2016 Jan 4;34(2):276-283. [CrossRef] [Medline]
    16. Wakadha H, Chandir S, Were EV, Rubin A, Obor D, Levine OS, et al. The feasibility of using mobile-phone based SMS reminders and conditional cash transfers to improve timely immunization in rural Kenya. Vaccine 2013 Jan 30;31(6):987-993 [FREE Full text] [CrossRef] [Medline]
    17. Fadda M, Galimberti E, Fiordelli M, Romanò L, Zanetti A, Schulz PJ. Effectiveness of a smartphone app to increase parents' knowledge and empowerment in the MMR vaccination decision: A randomized controlled trial. Hum Vaccin Immunother 2017 Nov 02;13(11):2512-2521. [CrossRef] [Medline]
    18. LeFevre AE, Mohan D, Hutchful D, Jennings L, Mehl G, Labrique A, et al. Mobile Technology for Community Health in Ghana: what happens when technical functionality threatens the effectiveness of digital health programs? BMC Med Inform Decis Mak 2017 Dec 14;17(1):27. [CrossRef] [Medline]
    19. Zaidi S, Riaz A, Hussain SS, Omer SB, Ali A. Applying a governance barometer to vaccine delivery systems: Lessons from a rural district of Pakistan. Vaccine 2020 Jan;38(3):627-634. [CrossRef]
    20. Punjab Information Technology Board (PITB). Tracking Vaccinators (E-VACCS): Government of Punjab   URL: https://www.pitb.gov.pk/tracking_evaccs
    21. Ebile Akoh W, Ateudjieu J, Nouetchognou JS, Yakum MN, Djouma Nembot F, Nafack Sonkeng S, et al. The expanded program on immunization service delivery in the Dschang health district, west region of Cameroon: a cross sectional survey. BMC Public Health 2016 Aug 17;16(1):801 [FREE Full text] [CrossRef] [Medline]
    22. Braun R, Catalani C, Wimbush J, Israelski D. Community health workers and mobile technology: a systematic review of the literature. PLoS One 2013;8(6):e65772 [FREE Full text] [CrossRef] [Medline]
    23. Calderón TA, Martin H, Volpicelli K, Frasso R, Díaz Arroyo EC, Gozzer E, et al. Understanding potential uptake of a proposed mHealth program to support caregiver home management of childhood illness in a resource-poor setting: a qualitative evaluation. Mhealth 2017;3:19 [FREE Full text] [CrossRef] [Medline]
    24. JSI. Africa Routine Immunization System Essential. Improving the performance of routine immunization systems in Africa ? A case of the drivers of routine immunization system performance in Cameroon   URL: https:/​/3qvw4sd8qlr2o7003493odw1-wpengine.​netdna-ssl.com/​wp-content/​uploads/​2012/​11/​Arise_CameroonReport_final5081.​pdf
    25. JSI. 2018. Strengthening Immunization for the Urban Poor: Experiences and Lessons Learned from Five Country Studies   URL: https:/​/www.​jsi.com/​JSIInternet/​Resources/​publication/​display.​cfm?txtGeoArea=INTL&id=22079&thisSection=Resources [accessed 2020-01-10]
    26. Barau I, Zubairu M, Mwanza MN, Seaman VY. Improving polio vaccination coverage in Nigeria through the use of geographic information system technology. J Infect Dis 2014 Nov 01;210 Suppl 1:S102-S110. [CrossRef] [Medline]
    27. Gammino VM, Nuhu A, Chenoweth P, Manneh F, Young RR, Sugerman DE, et al. Using geographic information systems to track polio vaccination team performance: pilot project report. J Infect Dis 2014 Nov 01;210 Suppl 1:S98-101. [CrossRef] [Medline]
    28. Stanton M, Molineux A, Mackenzie C, Kelly-Hope L. Mobile Technology for Empowering Health Workers in Underserved Communities: New Approaches to Facilitate the Elimination of Neglected Tropical Diseases. JMIR Public Health Surveill 2016;2(1):e2 [FREE Full text] [CrossRef] [Medline]
    29. Teng JE, Thomson DR, Lascher JS, Raymond M, Ivers LC. Using Mobile Health (mHealth) and Geospatial Mapping Technology in a Mass Campaign for Reactive Oral Cholera Vaccination in Rural Haiti. PLoS Negl Trop Dis 2014 Jul 31;8(7):e3050. [CrossRef]
    30. Wallis L, Blessing P, Dalwai M, Shin SD. Integrating mHealth at point of care in low- and middle-income settings: the system perspective. Glob Health Action 2017 Jun;10(sup3):1327686 [FREE Full text] [CrossRef] [Medline]
    31. Rothstein JD, Jennings L, Moorthy A, Yang F, Gee L, Romano K, et al. Qualitative Assessment of the Feasibility, Usability, and Acceptability of a Mobile Client Data App for Community-Based Maternal, Neonatal, and Child Care in Rural Ghana. Int J Telemed Appl 2016;2016:2515420-2515414 [FREE Full text] [CrossRef] [Medline]
    32. Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Womens Health 2014 Jan 08;59(2):184-191 [FREE Full text] [CrossRef] [Medline]
    33. van Heerden A, Sen D, Desmond C, Louw J, Richter L. App-Supported Promotion of Child Growth and Development by Community Health Workers in Kenya: Feasibility and Acceptability Study. JMIR Mhealth Uhealth 2017 Dec 05;5(12):e182 [FREE Full text] [CrossRef] [Medline]
    34. Chaiyachati KH, Loveday M, Lorenz S, Lesh N, Larkan L, Cinti S, et al. A pilot study of an mHealth application for healthcare workers: poor uptake despite high reported acceptability at a rural South African community-based MDR-TB treatment program. PLoS One 2013 May 28;8(5):e64662 [FREE Full text] [CrossRef] [Medline]
    35. Gavi-The Vaccine Alliance. Annual Progress Report 2018   URL: https://www.gavi.org/results/gavi-progress-reports/


    Abbreviations

    EPI: Expanded Program on Immunization
    LMIC: low- and middle-income country
    mHealth: mobile health
    TB: tuberculosis
    TMK: Tando Muhammad Khan
    UC: union council


    Edited by G Eysenbach; submitted 18.09.19; peer-reviewed by M Fiordelli, N Jarour, D Felt, J Boegner; comments to author 26.10.19; revised version received 31.10.19; accepted 16.12.19; published 13.02.20

    ©Shehla Zaidi, Saqib Ali Shaikh, Saleem Sayani, Abdul Momin Kazi, Adeel Khoja, Syed Shahzad Hussain, Rabia Najmi. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 13.02.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.