This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Vaccination is a critical step in reducing child mortality; however, vaccination rates have declined in many countries in recent years. This decrease has been associated with an increase in the outbreak of vaccine-preventable diseases. The potential for leveraging mobile platforms to promote vaccination coverage has been investigated in the development of numerous mobile apps. Although many are available for public use, there is little robust evaluation of these apps.
This systematic review aimed to assess the effectiveness of apps supporting childhood vaccinations in improving vaccination uptake, knowledge, and decision making as well as the usability and user perceptions of these apps.
PubMed, Excerpta Medica Database (EMBASE), Web of Science, Cochrane Central Register of Controlled Trials, ClinicalTrials.gov, and Education Resources Information Center (ERIC) databases were systematically searched for studies published between 2008 and 2019 that evaluated childhood vaccination apps. Two authors screened and selected studies according to the inclusion and exclusion criteria. Data were extracted and analyzed, and the studies were assessed for risk of bias.
A total of 28 studies evaluating 25 apps met the inclusion criteria and were included in this analysis. Overall, 9 studies assessed vaccination uptake, of which 4 reported significant benefits (
There is little evidence to support the use of childhood vaccination apps to improve vaccination uptake, knowledge, or decision making. Further research is required to understand the dichotomous effects of vaccination-related information provision and the evaluation of these apps in larger, more robust studies. The methodology of studies must be reported more comprehensively to accurately assess the effectiveness of childhood vaccination apps and the risk of bias of studies.
RR2-10.2196/16929
In 2018, it was estimated that immunization prevented 2 to 3 million deaths each year, yet over 19 million children worldwide under the age of 1 year did not receive basic vaccines [
Despite the low mortality rate of vaccine-preventable diseases, various sociodemographic groups, including young children and elderly or immunocompromised individuals, are at risk of serious, sometimes fatal, complications [
With the increasing utilization and accessibility of mobile devices, digital technologies have shown promise in effectively disseminating information to diverse and diffuse populations and rolling out community-wide initiatives [
This study aimed to systematically review the evidence on the use of apps to support childhood vaccination uptake, information storage, and record sharing as well as to investigate the usability and user perceptions of these apps.
This systematic review was conducted following, where possible, the Cochrane collaboration [
Full methods for this review have been published in detail in a systematic review protocol [
This systematic review aimed to assess apps designed to support childhood vaccination uptake. As such, the search was limited to studies conducted during or after 2008, when the first smartphone was launched, thus reducing the number of irrelevant results. When searching ClinicalTrials.gov, the search was limited to studies first posted on or after January 1, 2008. Only studies published in English were included to ensure an accurate interpretation. Observational studies such as cross-sectional surveys, cohort studies, qualitative studies, economic studies, and intervention studies were included. Intervention studies were not required to have a specific comparator or any comparators. Studies were excluded if they were solely descriptive of the app.
To understand the latest developments in accessible technology supporting improvement in the uptake of childhood vaccinations, we restricted this review to apps hosted on mobile platforms. The app could provide any service related to the promotion of vaccination or vaccination decision making, including but not limited to information sharing and record storing or sharing, and appointment support. Studies that did not involve the use or study of an app or solely focused on other ways of delivering vaccination interventions such as text messaging, telephone calls, or web-based interventions were excluded. Owing to the specific nature of the intervention, the population was restricted to children, parents, guardians, and/or health care professionals involved in the management of children. Children were defined as individuals aged less than or equal to 18 years. Studies focusing on the vaccination of adults were excluded. The study could have been conducted in any geographical setting.
The primary outcome of this review was the uptake of vaccination. The secondary outcomes were the knowledge and decision making of parents; costs and cost-effectiveness; use of the app; measures of usability, for example, usefulness, acceptability, and experiences of different users (parents and health care professionals); and adverse events (eg, data leak and misinformation).
All studies retrieved from the databases were stored in Mendeley version 1.19.5 (Elsevier), a reference management software. This software automatically eliminated duplicates before screening the citations against the inclusion and exclusion criteria by 2 independent reviewers. When duplicates, or publications from the same study were identified, the more recent publication or the one with the most details was selected for inclusion in the review. Any disagreements were discussed, and if a consensus was not reached, a third reviewer was consulted.
Published results of trials that were retrieved from CENTRAL or ClinicalTrials.gov and that met the inclusion criteria were searched for and included if not already captured; trial designs or protocols were excluded. A total of 10 trials met the inclusion criteria; 8 had no published data at the date of screening, and the published results of the remaining 2 trials were already included. The titles of references of 5 relevant review studies that were retrieved with our search strategy were reviewed for inclusion; 4 additional references were identified and were included in the full-text review.
The full text of the abstracts that met the inclusion criteria was screened by one of the reviewers and validated by a second reviewer to determine the studies to be included in the final set. Overall, 10 of the screened studies eligible for inclusion were conference or meeting abstracts and did not have full texts available; hence, they were excluded.
Data were extracted by 1 reviewer, and key data points from the studies that were specified in the protocol and identified on further study of the publications were recorded in a spreadsheet. The data extraction form was based on the minimum requirements recommended by the Cochrane Handbook for Systematic Reviews [
Data extracted from the included studies.
Study information | Data extracted |
General study information | Title of publication, year of publication, authors, and journal of publication |
Study characteristics | Study design, country of study, analyzed sample size, key inclusion/exclusion criteria, and study arms |
Intervention characteristics | App name, device on which the app could be or was utilized, compatible platforms, intended user, aim of the app, vaccines covered by the app, and vaccine-related features of the app |
Evaluation | Number of users, impact on the uptake of vaccinations, impact on knowledge/learning, impact on vaccination decision making, perceived credibility, usability/user experiences, popular features, costs/cost-effectiveness, adverse events, and conclusions |
The quality assessment of the included studies was undertaken by 1 reviewer and validated by a second reviewer. Any disagreements were resolved by consensus or the opinion of a third reviewer, where required. The methods specified in the Cochrane collaboration tool for assessing the risk of bias were used. The Cochrane collaboration risk of bias tool was used to assess the quality of the randomized controlled trials [
Owing to the variability in populations, interventions, outcomes, and study designs, a meta-analysis of the studies was not possible; hence, we report a narrative overview of the findings to draw conclusions about the potential roles, value, and effectiveness of the apps to support childhood vaccinations. For the purpose of this review, the app was considered to provide significant benefit if there was a statistically significant (
Overall, 3415 studies were retrieved from the 7 databases; of these, 1243 were duplicates. Of the 2172 citations screened, 126 were selected for full-text review, and 4 additional studies were identified during the title screening of the references from 5 relevant review studies that had been retrieved in the database search. The primary reasons for exclusion at the screening stage were that the study was not vaccination-related (n=1171), did not include a mobile app (n=564), or was not health-related (n=89). Overall, 28 papers were included in the final review. The reasons for the exclusion of full-text review are detailed in
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of study selection.
The study characteristics of the 28 studies included in this review can be seen in
The 28 studies evaluated 25 unique apps; 3 papers evaluated the ImmunizeCA app [
The characteristics of the 25 apps investigated in the included studies are shown in
Characteristics of childhood vaccination apps.
App | Intended user | Technical specifications | Compatible platforms |
Conversational agent for HPVa vaccination [ |
Parent |
Graphical user interface tool on laptop used by an operator Bluetooth communication with iPad, the user-facing interface Text-to-speech capability Wizard of Oz agent architecture |
Apple iOSb and MacOS |
ImmunizeCA [ |
Women of childbearing age |
Generates customized vaccination schedules Vaccine information available Creates virtual immunization record Syncs with calendar for scheduling Embedded outbreak alert feature Basic security features Rotating banner in app used to display features and public health messages |
iOS and Android |
Tablet-based self-persuasion app [ |
Parent |
Voiceover narration of task Audio recording function to facilitate self-administration |
iOS |
ReadyVax [ |
Health care providers, pharmacists, parents, and patients |
Native app direct to smartphone Offline functioning Information updates automatically Browsable and searchable information Information updated through a web-based dashboard interface Alert notifications can be sent Links to multimedia |
iOS |
UberHealth [ |
Anyone |
Request and delivery of vaccines using geolocation software |
NRc |
EPId app [ |
Doctors |
Record vaccination status and upload data into the CIRSe CIRS sends daily updates on children for whom vaccination is overdue Contact details of families available |
NR |
Carrot Rewards [ |
Anyone |
In-app quiz about influenza vaccinations Geolocation-based push notification when in proximity to a pharmacist Loyalty points for completion of vaccination-related tasks |
NR |
MorbiQuiz [ |
Parent |
Daily quiz targeting vaccination literacy Vaccination empowerment videos Leaderboard for quiz results |
iOS and Android |
Tablet-based HPV educational module [ |
Patient and parent |
Educational videos on HPV Flashcard information on HPV |
NR |
RapidSMS [ |
Health worker |
Mobile alert system for vaccination tracking |
NR |
CHeITA [ |
Health care providers, parents, and guardians |
Stores health history of family and development statistics Vaccination tracking |
Windows, iOS, and Android |
Mobile technology supporting EPI coverage [ |
Health worker |
Stores personal and familial information Case identification via pictures Pronunciation of the name of the child in the mother’s ethnic language |
NR |
Mother and Child Care Module-EPI module [ |
Health worker |
Immunization status collected Connection with server module Generates appointment dates and SMS reminders |
NR |
Tailored interactive multimedia intervention [ |
Parent |
Tailored interactive health communication about HPV via videos |
NR |
Baby Care app [ |
Mothers |
Embedded FAQsf Upload child data Trend analysis Alert messaging Baby’s periodic health report generation |
NR |
EpiSurveyor [ |
Health worker |
Sources of information Basic demographics Consent to bring children for immunization |
Android |
ImTeCHO [ |
Health worker |
Registration of pregnant women and children aged under 2 years Generates daily appointment schedule Videos to emphasize key health messages |
Android and web |
Smartphone App for Premature Infants [ |
Mothers |
Electronic learning modules |
NR |
Call the shots [ |
Parent |
Reminders for vaccination Record keeping of child’s vaccinations Hosts latest immunization schedule Extensive toolkit embedded with FAQs Links to videos and resources |
Android |
FightHPV [ |
Teenagers |
Gamified narratives with connected text messages to convey HPV information Players able to share information with social network |
iOS, Java, and Android |
MomsTalkShots [ |
Mothers |
Videos with obstetricians and pediatricians of different ethnicities Intervention tailored to knowledge and beliefs |
NR |
VaccApp [ |
Parent |
Avatar requests vaccination information |
Android |
iCHRcloud [ |
Parent and doctor |
Mobile interface, doctor module, and cloud component Child health records stored, updated, and shared across network |
iOS and Android |
mTika [ |
Health worker |
Registration of pregnant women SMS birth notifications from mothers Automated SMS vaccination reminders to mothers and health workers EPI monitoring by supervisors |
Android |
iPhone app [ |
Parent |
Stores child’s vaccination information Hosts recommended vaccination schedule Generates customized vaccination schedule |
NR |
aHPV: human papillomavirus.
biOS: iPhone operating system.
cNR: not reported.
dEPI: Expanded Program on Immunization.
eCIRS: Child Immunization Register System.
fFAQs: frequently asked questions.
The investigated apps were most commonly designed for the primary purpose of education (11/25) [
There was no consistent reporting of the most popular features, perceptions, or usage of individual functions. A total of 5 studies reporting on the usage of the apps noted that the most commonly used/most popular features were those that helped manage vaccination records [
Capabilities of the apps described in the included studies.
App | Counseling | Self-persuasion | Management of records | Reminders | Vaccine-preventable disease breakout alert | Education | Frequently asked questions | Vaccine delivery | Total |
Conversational agent for HPVa vaccination [ |
Xb | N/Ac | N/A | N/A | N/A | N/A | N/A | N/A | 1 |
ImmunizeCA [ |
N/A | N/A | Xb | Xd | Xd | N/A | Xd | N/A | 4 |
Tablet-based self-persuasion app [ |
N/A | Xb | N/A | N/A | N/A | Xd | N/A | N/A | 2 |
ReadyVax [ |
N/A | N/A | N/A | N/A | Xd | Xb | Xd | N/A | 3 |
UberHealth [ |
N/A | N/A | N/A | N/A | N/A | N/A | N/A | Xb | 1 |
EPIe app [ |
N/A | N/A | Xb | Xd | N/A | Xd | N/A | N/A | 3 |
Carrot Rewards [ |
N/A | N/A | N/A | Xd | N/A | Xb | N/A | N/A | 2 |
MorbiQuiz [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
Tablet-based HPV educational module [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
RapidSMS [ |
N/A | N/A | N/A | Xb | N/A | N/A | N/A | N/A | 1 |
CHeITA [ |
N/A | N/A | Xb | N/A | N/A | N/A | N/A | N/A | 1 |
Mobile technology supporting EPI coverage [ |
N/A | N/A | Xd | Xb | N/A | Xd | N/A | N/A | 3 |
Mother and child care module-EPI module [ |
N/A | N/A | Xb | Xd | N/A | N/A | N/A | N/A | 2 |
Tailored interactive multimedia intervention [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
Baby Care app [ |
N/A | N/A | Xd | Xd | N/A | Xb | Xd | N/A | 4 |
EpiSurveyor [ |
N/A | N/A | Xd | N/A | N/A | Xb | N/A | N/A | 2 |
ImTeCHO [ |
N/A | N/A | Xb | N/A | N/A | N/A | N/A | N/A | 1 |
Smartphone App for Premature Infants [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
Call the shots [ |
N/A | N/A | Xd | Xb | N/A | N/A | Xd | N/A | 3 |
FightHPV [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
MomsTalkShots [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
VaccApp [ |
N/A | N/A | N/A | N/A | N/A | Xb | N/A | N/A | 1 |
iCHRcloud [ |
N/A | N/A | Xb | Xd | N/A | N/A | N/A | N/A | 2 |
mTika [ |
N/A | N/A | Xb | Xd | N/A | N/A | N/A | N/A | 2 |
iPhone app [ |
N/A | N/A | Xb | Xd | Xd | N/A | N/A | N/A | 3 |
Total | 1 | 1 | 12 | 11 | 3 | 14 | 3 | 1 | N/A |
aHPV: human papillomavirus.
bX: indicates primary functions.
cN/A: not applicable.
dX: indicates secondary functions.
eEPI: Expanded Program on Immunization.
The extracted outcomes and results are provided in
Of the remaining 5 studies, 2 reported no significant benefit [
A total of 10 studies reported on the impact of the vaccination apps on knowledge/learning, as shown in
The implications of the vaccination apps on decision making and evaluation of the risk-benefit of vaccinations were investigated in 8 studies, as shown in
Only 1 study reported on the costs or cost-effectiveness of a childhood vaccination app. The cost of developing a computerized, tailored, interactive multimedia intervention was found to be approximately double the cost of a print-based Photonovella intervention for HPV vaccine education (US $135,978 vs US $66,468, respectively). This difference was retained in amortized annual costs over a 7-year period (US $21,825 vs US $10,669 per year for the tailored, interactive multimedia intervention and Photonovella, respectively) [
Overall, 9 studies reported on the usability/ease of use (n=5), acceptability (n=1), or both (n=3) aspects of the vaccination apps. Furthermore, 8 of these studies reported high ease of use (average score for ease of use/usability >70%, or >70% of the participants rated the app easy to use). A total of 2 studies also reported high acceptance of the app (average score for acceptance >70%, or >70% of participants reporting acceptance).
A total of 11 studies reported on participants’ perceptions of childhood vaccination apps. Furthermore, 9 studies reported on the perceptions of parents [
Owing to the heterogeneity of the study types, a variety of quality assessment tools were employed to assess the risk of bias for the 28 included studies. The summary tables and figures are provided in
The critical appraisal skills program cohort checklist assessed confounding, completeness, and the duration of follow-up. The identification and mitigation of confounders was not found to be sufficient in any of the 14 studies assessed using this checklist [
Six cross-sectional studies assessed using the AXIS tool were found to lack justification of sample size, definition of target population (1/6), categorization of (1/6) [
In this systematic review, 28 studies evaluating 25 childhood vaccination apps were examined. Overall, there is little evidence to suggest that childhood vaccination apps are effective in improving vaccination coverage, with only 4 of the 9 studies assessing this outcome indicating significant benefit (
Similarly, 4 out of 10 studies assessing the impact on vaccination knowledge and 4 out of 8 studies assessing vaccination decision making reported significant benefit of the app (
The primary functionality of the apps described in the included studies varied; however, most had multiple functions, with the most common features being education, reminders, and record keeping. These were primarily for the use of parents or health care providers; only the iCHRcloud app facilitated the sharing of vaccination record information between parents and physicians [
There is insufficient evidence to draw any conclusions regarding the relationship between the function of the app and the efficacy in improving vaccination rates, knowledge, or positive decision making. The 25 investigated apps had diverse functionalities but were primarily designed for providing vaccine information and/or record keeping. Studies reporting on user statistics revealed that the most popular functions were record keeping, reminders, and information access. Overall, usability, acceptability, and user perceptions of the apps were positive.
The quality assessment of the included studies revealed that many were of poor to moderate quality, indicating an overall high risk of bias, which risks impairing the validity of the conclusions regarding the effectiveness of childhood vaccination apps. One study was determined to have a serious risk of bias. Most negative indicators were because of a lack of information about the criteria assessed. The risk of bias and inadequate robustness may be because of the nature of many of the included studies being pilot, early usability, and preliminary scoping studies. To draw valid and accurate conclusions on the quality of studies, study methods should be comprehensively reported. The infancy of these types of apps also had an impact on the assessment of the implications/value and the fit of the results in context, as many studies indicated that they were the first of their kind in their setting.
The strengths of this study lie in the comprehensive analysis of the available literature discussing apps for childhood vaccination. We investigated ClinicalTrials.gov and ERIC databases, which include gray literature, and we included letters and full-text conference proceedings [
Immunization is a simple and effective mechanism for reducing childhood mortality. Despite the insignificant findings of this review about the effect of apps on the uptake of vaccinations, the positive user perceptions, usability, and acceptability reported present a compelling opportunity to build on the successes of current apps and learn from their shortcomings. Individual studies included in this review reported the potential benefit of these apps on an individual, community, and nationwide level, highlighting the breadth of engagement that can be harnessed with the use of mobile apps [
Mobile apps will likely play a role in the storage and sharing of vaccination records, generation of reminders, and/or dissemination of vaccination education. Despite several publicly available apps and others in development, a lack of robust evidence remains regarding the effectiveness of vaccination apps in improving vaccination coverage. This systematic review, despite not reporting significant efficacy, indicates that many of the apps convey some degree of benefit with regard to improving vaccination uptake, knowledge, and/or decision making and are widely accepted by their users. For future research, it will be important to understand the priorities of different user groups in terms of app functionalities and the dichotomous effects of vaccination information. Many of the studies included in this review are early-stage investigations and were found to have a relatively high risk of bias. Further investigation of these apps in larger, more robust, controlled trials will allow greater granularity of evaluation and understanding of the role and implications of these apps for wider communities and various subpopulations. In addition, many of the included studies originated from developing countries; however, preventable childhood illnesses are increasing globally. Outcomes from similar studies in developed countries may present a different picture.
The objective of this systematic review was to evaluate the effectiveness of childhood vaccinations in improving vaccination uptake, knowledge, and decision making as well as to investigate the usability and patient perceptions of these interventions. Overall, 28 studies describing 25 apps were investigated. Although the apps were generally positively received and had high usability and acceptability scores, there was little evidence to suggest that they were effective in significantly improving uptake, knowledge, or decision making; however, most apps were seen to provide some benefit. This indicates that there is demand and engagement with apps supporting childhood vaccinations; however, further investigation is required.
The studies investigating these apps were considered to be of poor to moderate quality, likely because of the early phase nature of many of the apps and their respective studies. Only 5 studies were randomized. An additional concern raised by 3 studies was the potential for these apps to discourage vaccination among those who were initially undecided about infant vaccination and among those who had previously intended to vaccinate their children. Future research is warranted into the dichotomous effects of the provision of vaccination information, the outcomes of larger robust studies of these apps, and the needs and priorities of various user populations.
Search strategy.
Study characteristics.
Study outcomes and extracted effectiveness results.
Risk of bias assessment.
Appraisal tool for Cross-Sectional Studies
Cochrane Central Register of Controlled Trials
Expanded Program on Immunization
Education Resources Information Center
human papillomavirus
Medical Subject Headings
mobile health
risk of bias in nonrandomized studies of interventions
CC, EM, MI, and MV were supported in this study by EIT Health (grant 18654).
MM, MV, and EM conceived the study topic and designed the review protocol. CC and MI screened the studies. CC conducted the data extraction and risk of bias assessment, which were validated by MI. The systematic review was written by CC with revisions from MV, EM, MM, and MI.
None declared.