Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 18.05.20 in Vol 8, No 5 (2020): May

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/17371, first published Dec 10, 2019.

This paper is in the following e-collection/theme issue:

    Review

    Use of Apps to Promote Childhood Vaccination: Systematic Review

    1Digitally Enabled PrevenTative Health Research Group, Department of Paediatrics, University of Oxford, Oxford, United Kingdom

    2School of Nursing & Midwifery, Trinity College Dublin, Dublin, Ireland

    3Department of Primary Care and Public Health, Imperial College London, London, United Kingdom

    Corresponding Author:

    Edward Meinert, MA, MSc, MBA, MPA, PhD

    Digitally Enabled PrevenTative Health Research Group

    Department of Paediatrics

    University of Oxford

    Children's Hospital

    John Radcliffe Hospital

    Oxford, OX3 9DU

    United Kingdom

    Phone: 44 7824446808

    Email: e.meinert14@imperial.ac.uk


    ABSTRACT

    Background: Vaccination is a critical step in reducing child mortality; however, vaccination rates have declined in many countries in recent years. This decrease has been associated with an increase in the outbreak of vaccine-preventable diseases. The potential for leveraging mobile platforms to promote vaccination coverage has been investigated in the development of numerous mobile apps. Although many are available for public use, there is little robust evaluation of these apps.

    Objective: This systematic review aimed to assess the effectiveness of apps supporting childhood vaccinations in improving vaccination uptake, knowledge, and decision making as well as the usability and user perceptions of these apps.

    Methods: PubMed, Excerpta Medica Database (EMBASE), Web of Science, Cochrane Central Register of Controlled Trials, ClinicalTrials.gov, and Education Resources Information Center (ERIC) databases were systematically searched for studies published between 2008 and 2019 that evaluated childhood vaccination apps. Two authors screened and selected studies according to the inclusion and exclusion criteria. Data were extracted and analyzed, and the studies were assessed for risk of bias.

    Results: A total of 28 studies evaluating 25 apps met the inclusion criteria and were included in this analysis. Overall, 9 studies assessed vaccination uptake, of which 4 reported significant benefits (P<.001 or P=.03) of the implementation of the app. Similarly, 4 studies indicated a significant (P≤.054) impact on knowledge and on vaccination decision making. Patient perceptions, usability, and acceptability were generally positive. The quality of the included studies was found to be moderate to poor, with many aspects of the methodology being unclear.

    Conclusions: There is little evidence to support the use of childhood vaccination apps to improve vaccination uptake, knowledge, or decision making. Further research is required to understand the dichotomous effects of vaccination-related information provision and the evaluation of these apps in larger, more robust studies. The methodology of studies must be reported more comprehensively to accurately assess the effectiveness of childhood vaccination apps and the risk of bias of studies.

    International Registered Report Identifier (IRRID): RR2-10.2196/16929

    JMIR Mhealth Uhealth 2020;8(5):e17371

    doi:10.2196/17371

    KEYWORDS



    Introduction

    Background

    In 2018, it was estimated that immunization prevented 2 to 3 million deaths each year, yet over 19 million children worldwide under the age of 1 year did not receive basic vaccines [1]. Most of these children lived in developing countries, where access to vaccines and antenatal services is somewhat limited [1]. Nevertheless, an increase in vaccine-preventable disease outbreaks has also been identified in developed countries, which is associated with declining vaccination uptake [2-4]. Immunization coverage of 9 routine childhood vaccinations declined in England by 0.2% to 1% during 2018 to 2019, compared with the previous year, and 1.3% of children born in 2015 in the United States received no vaccinations by the age of 2 years, compared with 0.9% of those born in 2011 [4,5]. A number of studies have investigated the reasons for vaccine refusal among parents and caregivers and have revealed that religious, philosophical, and personal beliefs, coupled with safety concerns and insufficient information, were the most commonly cited reasons [6]. The now-refuted evidence linking measles-mumps-rubella with autism was seen to cause a 2% decrease in the uptake of measles-mumps-rubella vaccinations [7]. Furthermore, the widespread adoption of the human papillomavirus (HPV) vaccine has been thwarted by religious and cultural barriers [8,9].

    Despite the low mortality rate of vaccine-preventable diseases, various sociodemographic groups, including young children and elderly or immunocompromised individuals, are at risk of serious, sometimes fatal, complications [10]. The outbreaks of vaccine-preventable diseases can be minimized by maintaining herd immunity, which varies from 75% to 97% vaccination coverage, depending on the disease and setting in question [11]. As such, it is crucial that there is adequate provision of correct and comprehensive information, resources, and reminders to encourage parents and caregivers to obtain complete and timely vaccinations for their children. Many informational, behavioral, and environmental initiatives in various settings have been implemented to improve the uptake of childhood vaccinations [12-14]; however, the scalability and sustainability of these programs have been problematic [12].

    With the increasing utilization and accessibility of mobile devices, digital technologies have shown promise in effectively disseminating information to diverse and diffuse populations and rolling out community-wide initiatives [10,15]. A World Health Organization survey on mobile health (mHealth) revealed that 83% of member states offered at least one service in 2011 [16]. mHealth has been investigated by multiple private and public organizations to support the uptake of vaccinations, including vaccination information websites and mobile apps, hereafter referred to as apps. These apps have various functions designed to support health care providers, caregivers, and, in some cases, children to access vaccine-related information, recommended immunization schedules, store vaccination records, and receive appointment reminders. There are now over 200 vaccination-related apps available on the App Store [17]. A systematic review conducted in 2015 discussing the design of vaccination reminder apps reviewed 2 studies on mobile reminder apps [18]. However, a comprehensive review of the effectiveness and usability of childhood vaccination-related apps is yet to be conducted.

    Objectives

    This study aimed to systematically review the evidence on the use of apps to support childhood vaccination uptake, information storage, and record sharing as well as to investigate the usability and user perceptions of these apps.


    Methods

    This systematic review was conducted following, where possible, the Cochrane collaboration [19] and the Centre for Review and Dissemination [20] methodologies for conducting systematic reviews.

    Database Search

    Full methods for this review have been published in detail in a systematic review protocol [21]. This systematic review was registered with the International Prospective Register of Systematic Reviews (CRD42019156583). The Participant, Intervention, Comparison, and Outcome framework was used to develop the search strategy [22], which was performed following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols [23]. The search strategy was tweaked to ensure that a defined set of known references was returned without retrieving an unmanageably large number of studies. This primarily involved the selection and amendment of wildcard terms to ensure that irrelevant terms were not included; for example, immun* retrieved papers related to immunology; hence, this was amended to immuni* to make this more specific to immunizations. No study design filter was used as both quantitative and qualitative studies were included. The search strategy was finalized and tailored to different databases in consultation with a medical librarian. PubMed, Excerpta Medica Database (EMBASE, Ovid), Web of Science, Cochrane Central Register of Controlled Trials (CENTRAL), ClinicalTrials.gov, and Education Resources Information Center (ERIC) databases were searched. The search terms were grouped into 3 themes—vaccinations, mobile apps, and children—which were subsequently searched with the following structure: vaccinations (Medical Subject Headings, MeSH OR Keywords) AND mobile applications (MeSH OR Keywords) AND children (MeSH OR Keywords). The full search strategy is shown in Multimedia Appendix 1. The search took place on October 23, 2019.

    Inclusion and Exclusion Criteria

    This systematic review aimed to assess apps designed to support childhood vaccination uptake. As such, the search was limited to studies conducted during or after 2008, when the first smartphone was launched, thus reducing the number of irrelevant results. When searching ClinicalTrials.gov, the search was limited to studies first posted on or after January 1, 2008. Only studies published in English were included to ensure an accurate interpretation. Observational studies such as cross-sectional surveys, cohort studies, qualitative studies, economic studies, and intervention studies were included. Intervention studies were not required to have a specific comparator or any comparators. Studies were excluded if they were solely descriptive of the app.

    To understand the latest developments in accessible technology supporting improvement in the uptake of childhood vaccinations, we restricted this review to apps hosted on mobile platforms. The app could provide any service related to the promotion of vaccination or vaccination decision making, including but not limited to information sharing and record storing or sharing, and appointment support. Studies that did not involve the use or study of an app or solely focused on other ways of delivering vaccination interventions such as text messaging, telephone calls, or web-based interventions were excluded. Owing to the specific nature of the intervention, the population was restricted to children, parents, guardians, and/or health care professionals involved in the management of children. Children were defined as individuals aged less than or equal to 18 years. Studies focusing on the vaccination of adults were excluded. The study could have been conducted in any geographical setting.

    Outcome Measures

    The primary outcome of this review was the uptake of vaccination. The secondary outcomes were the knowledge and decision making of parents; costs and cost-effectiveness; use of the app; measures of usability, for example, usefulness, acceptability, and experiences of different users (parents and health care professionals); and adverse events (eg, data leak and misinformation).

    Screening and Selection of Studies

    All studies retrieved from the databases were stored in Mendeley version 1.19.5 (Elsevier), a reference management software. This software automatically eliminated duplicates before screening the citations against the inclusion and exclusion criteria by 2 independent reviewers. When duplicates, or publications from the same study were identified, the more recent publication or the one with the most details was selected for inclusion in the review. Any disagreements were discussed, and if a consensus was not reached, a third reviewer was consulted.

    Published results of trials that were retrieved from CENTRAL or ClinicalTrials.gov and that met the inclusion criteria were searched for and included if not already captured; trial designs or protocols were excluded. A total of 10 trials met the inclusion criteria; 8 had no published data at the date of screening, and the published results of the remaining 2 trials were already included. The titles of references of 5 relevant review studies that were retrieved with our search strategy were reviewed for inclusion; 4 additional references were identified and were included in the full-text review.

    The full text of the abstracts that met the inclusion criteria was screened by one of the reviewers and validated by a second reviewer to determine the studies to be included in the final set. Overall, 10 of the screened studies eligible for inclusion were conference or meeting abstracts and did not have full texts available; hence, they were excluded.

    Data Extraction

    Data were extracted by 1 reviewer, and key data points from the studies that were specified in the protocol and identified on further study of the publications were recorded in a spreadsheet. The data extraction form was based on the minimum requirements recommended by the Cochrane Handbook for Systematic Reviews [24]. The data extracted from the studies are shown in Table 1. This process was validated by a second reviewer, and disagreements were resolved by a third reviewer. There were no disagreements in the extracted data; however, there were 18 instances in which the second reviewer proposed the inclusion of additional data for greater clarity.

    Table 1. Data extracted from the included studies.
    View this table

    Risk of Bias

    The quality assessment of the included studies was undertaken by 1 reviewer and validated by a second reviewer. Any disagreements were resolved by consensus or the opinion of a third reviewer, where required. The methods specified in the Cochrane collaboration tool for assessing the risk of bias were used. The Cochrane collaboration risk of bias tool was used to assess the quality of the randomized controlled trials [25]; the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool was used to assess the nonrandomized intervention trials [26]; and the critical appraisal skills program tools for cohort, qualitative, and economic studies were used for pre-post and quantitative studies, qualitative studies, and economic studies, respectively, [27-29]. The quality of cross-sectional survey studies was assessed using the Appraisal tool for Cross-Sectional Studies (AXIS) tool [30]. The results of the Cochrane collaboration risk of bias tool and ROBINS-I evaluations were summarized using RevMan 5.3 [31] and robvis [32], respectively. The critical appraisal skills program scores were calculated using standard practice, yes=1, no=0, and cannot tell=0 for each question, following which the total score was summed for each study. AXIS scores were summarized tabularly, and the mean and SD were calculated.

    Data Analysis and Synthesis

    Owing to the variability in populations, interventions, outcomes, and study designs, a meta-analysis of the studies was not possible; hence, we report a narrative overview of the findings to draw conclusions about the potential roles, value, and effectiveness of the apps to support childhood vaccinations. For the purpose of this review, the app was considered to provide significant benefit if there was a statistically significant (P≤.054) improvement in a given outcome as compared with a comparator or control or over time. If no significance was reported or if the difference was nonsignificant or significantly worse among groups or over time, the app was considered to have no significant evidence supporting it. The limitations and future directions for research were also summarized.


    Results

    Study Selection

    Overall, 3415 studies were retrieved from the 7 databases; of these, 1243 were duplicates. Of the 2172 citations screened, 126 were selected for full-text review, and 4 additional studies were identified during the title screening of the references from 5 relevant review studies that had been retrieved in the database search. The primary reasons for exclusion at the screening stage were that the study was not vaccination-related (n=1171), did not include a mobile app (n=564), or was not health-related (n=89). Overall, 28 papers were included in the final review. The reasons for the exclusion of full-text review are detailed in Figure 1 [33].

    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of study selection.
    View this figure

    Study Characteristics

    The study characteristics of the 28 studies included in this review can be seen in Multimedia Appendix 2. These studies were published between 2010 and 2019. Overall, 3 of the 28 studies were randomized controlled trials [34-36], and 12 had a nonrandomized trial design; 9 were pre-post (before-and-after) studies [37-45], 2 were nonrandomized controlled trials [46,47], and 1 had an interrupted time series design [48]. A further 6 studies were cross-sectional survey studies [49-54], 4 were longitudinal observational studies [55-58], 2 were qualitative studies [59], and 1 was an economic design study [60].

    The 28 studies evaluated 25 unique apps; 3 papers evaluated the ImmunizeCA app [39,58,59] and 2 studies assessed MorbiQuiz [35,49]. The included studies varied in design, population, and geographical setting. The study populations generally included parents (15/28) [35-39,44,46-52,54,59], expectant mothers (5/28) [36,39,44,47,59], and/or children (4/28) [41-43,61]. Other populations included the general public (5/28) [45,53,55,57,58], households (1/28) [43], and village doctors (1/28) [34,43]. Some studies included multiple populations; hence, they were included multiple times in these statistics. The population size in the included studies ranged from 6 to 161,695 participants [43,51]. Overall, 12 of the 28 included studies were conducted in North American countries [37-40,44,45,51,53,57-60], 7 in Asia [34,36,41,42,47,52,56], 1 in the Middle East [54], 5 in Europe [35,46,49,50,61], 2 in Africa [43,48], and 1 included worldwide users [55]. Approximately, 39% (11/28) of the studies took place in deprived areas and/or developing countries [34,36,38,41-43, 47,48,52,54,56].

    App Characteristics

    The characteristics of the 25 apps investigated in the included studies are shown in Table 2. The apps were primarily delivered via a smartphone or tablet, with 4 apps using multimodal delivery methods [36,37,50,56]. Over 50% (15/25) of the apps were intended for use by parents [35,37,38,40,44,46, 50-52,54-58,60], with 4 designed specifically for use by mothers [44,52,54,58]. Furthermore, 3 apps were for use by multidisciplinary populations involved in the delivery of childhood vaccinations, for example, health care providers, pharmacists, and parents [50,55,56]. Overall, 7 of the 25 apps were designed solely for the use of health care providers/health workers [34,36,41-43,47,48]. The vaccines covered by these apps varied. Almost two-thirds (15/25) of the apps covered multiple vaccines [34,36,41,42,44,46-48,50-52,54-58], 5 focused solely on HPV [37,38,40,60,61], 2 on influenza [45,53], 1 on measles-mumps-rubella [35], and 1 on measles only [43]. Furthermore, 4 studies describing 4 unique apps reported an association with the Expanded Program on Immunization (EPI) [34,41,42,47]. All 25 apps served vaccination-related functions; however, 28% (7/25) had a focus beyond vaccinations [36,42,45,50,52,54,56]. The scope of these apps remained broadly within the field of antenatal, maternal, or child health, with the exception of the Carrot Rewards app, which was intended for general health initiatives [45].

    Table 2. Characteristics of childhood vaccination apps.
    View this table

    Functionality of Childhood Vaccination Apps

    The investigated apps were most commonly designed for the primary purpose of education (11/25) [35,40,43-46, 52,54,55,60,61], record keeping (8/25) [34,36,42,47,50,56-58], or reminder systems (3/25) [41,48,51], as shown in Table 3. Despite specifying distinct primary functions, the apps had multiple overlapping capabilities. On average, the apps performed 1.9 functions (range 1-4), with the most commonly occurring functions aligning with the primary functions: education (14/25), management of records (12/25), and reminders (11/25).

    There was no consistent reporting of the most popular features, perceptions, or usage of individual functions. A total of 5 studies reporting on the usage of the apps noted that the most commonly used/most popular features were those that helped manage vaccination records [39], provided vaccination information [55], supported appointment management [34,59], and generated summary reports [52]. These functions aligned with the primary functions of the app being investigated. A qualitative study supplemented with Google Analytics data for the ImmunizeCA app reflected the overall data. These researchers reported that 9 of the 10 women interviewed used the vaccination tracking function, 80% used the appointment reminders/calendar, and 80% used the information on vaccines [59]. This was echoed by the Google Analytics data that were reported in the same study, wherein 47.6% of all app sessions accessed the tracking function, compared with 9.5% and 4.9%, where the appointment reminders and vaccination information were accessed, respectively [59]. One study that asked participants about the helpfulness of specific features of the app reported that all 6 respondents found the date reminder system helpful, whereas 5 respondents found the vaccine information very helpful [51].

    Table 3. Capabilities of the apps described in the included studies.
    View this table

    Uptake of Vaccinations

    The extracted outcomes and results are provided in Multimedia Appendix 3. Overall, 9 of the 28 studies assessed the impact of an app on vaccination uptake. Furthermore, 4 studies reported a significant improvement in vaccination coverage after versus before the implementation of the app [34,42,45,47]. These studies reported a 17% (P=.03) [34], 5% (P<.001) [45], 9.7% (P<.001) [42], and 17.9% (rural) and 16.4% (urban; P<.001 for both) [42,47] increase in the vaccination rates after versus before the implementation of the app. In addition, 2 of the 4 studies included a control group. The study of the mTika app identified a significant difference-in-difference estimate of the difference between the intervention groups’ change from baseline to end line and the control groups’ change from baseline to end line of fully vaccinated children (21.6% rural and 23.2% urban difference; P<.05) [47]. Conversely, the study of the EPI app did not find a significant difference between the intervention and control groups at the end line (2.5% difference; P=.16).

    Of the remaining 5 studies, 2 reported no significant benefit [36,48] and 3 reported no significance level [41,43,53] regarding an increase in the vaccination rates between the intervention and control groups or after versus before intervention implementation.

    Vaccination Knowledge and Decision Making

    A total of 10 studies reported on the impact of the vaccination apps on knowledge/learning, as shown in Multimedia Appendix 3. Furthermore, 4 studies reported significant improvements in the knowledge or learning compared with a control group or after versus before the intervention (P≤.05) [35,40,46,61], 2 reported no significant improvement [39,41], and 4 did not indicate a significance level [43,47,49,54].

    The implications of the vaccination apps on decision making and evaluation of the risk-benefit of vaccinations were investigated in 8 studies, as shown in Multimedia Appendix 3. Furthermore, 7 of the 8 studies reporting this outcome indicated a positive impact of the apps on vaccination beliefs and intent to vaccinate [35,38-41,44,49]. The remaining study was unable to report improvements because of no vaccine hesitancy in participants at baseline [37]. Half of the studies (4/8) indicated significant improvements (P≤.05) in the intent to vaccinate children, positive attitudes toward vaccination, and/or confidence in their vaccination decision after interaction with the app versus before or compared with a control group [35,38,40,41]. The remaining 4 studies did not report a significance level [37,39,44,49]. Fadda et al [35] investigated the knowledge and empowerment functions separately and found that exposure to the knowledge intervention significantly improved the intent to vaccinate (P=.03) and confidence in the participants’ vaccination decision (P=.006) versus control, but empowerment and combined interventions did not show stronger intent to vaccinate. Furthermore, 3 studies indicated that as well as having the potential to promote vaccination, the apps also had the potential to discourage users from vaccinating their children [39,44,49].

    Costs/Cost-Effectiveness

    Only 1 study reported on the costs or cost-effectiveness of a childhood vaccination app. The cost of developing a computerized, tailored, interactive multimedia intervention was found to be approximately double the cost of a print-based Photonovella intervention for HPV vaccine education (US $135,978 vs US $66,468, respectively). This difference was retained in amortized annual costs over a 7-year period (US $21,825 vs US $10,669 per year for the tailored, interactive multimedia intervention and Photonovella, respectively) [60].

    Usability and Acceptability

    Overall, 9 studies reported on the usability/ease of use (n=5), acceptability (n=1), or both (n=3) aspects of the vaccination apps. Furthermore, 8 of these studies reported high ease of use (average score for ease of use/usability >70%, or >70% of the participants rated the app easy to use). A total of 2 studies also reported high acceptance of the app (average score for acceptance >70%, or >70% of participants reporting acceptance).

    Participant Perceptions

    A total of 11 studies reported on participants’ perceptions of childhood vaccination apps. Furthermore, 9 studies reported on the perceptions of parents [38-40,44,46,49,51,52,54], 1 reported on teenagers’ experiences [61], and 1 reported on the perceptions of mothers and vaccination service providers [47]. All 10 studies reporting quantitative results indicated positive user experiences, with participants considering the app to be helpful and/or trustworthy or reporting that they were satisfied, confident, and/or likely to recommend the app (average score of >70%, or >70% of the participants agreeing with relevant statements) [38-40,44,46,49,51,52,54,61]. The study reporting on the qualitative experiences of service providers and mothers with the mTika app revealed that the app was perceived as helpful, easily understood by mothers, user-friendly, time-efficient, and helpful in reducing the workload of vaccination service providers [47].

    Risk of Bias Assessment

    Owing to the heterogeneity of the study types, a variety of quality assessment tools were employed to assess the risk of bias for the 28 included studies. The summary tables and figures are provided in Multimedia Appendix 4. Overall, the quality of the studies assessed in this review ranged from moderate to poor. The studies assessed using the critical appraisal skills program cohort, qualitative, and economic checklists met on average 6.4 out of 12 (range 4-9), 6.5 out of 10 (range 6-7), and 9 out of 12 (range 9) criteria, respectively. Cross-sectional studies assessed using the AXIS tool had a mean score of 9.3 (SD 2.2) out of 20. Over 50% (15/28) of the studies were deemed to have inappropriate recruitment strategies [34,36,37,41,42,45,46,48,49,52,54,56,58,61], primarily owing to the lack of sufficient information. Risk of bias in exposure (performance bias) was identified in 13 out of 17 studies in which this was assessed [34-37,41,43-45,48,55-58]. Outcome bias (detection bias) was suspected in 5 out of 19 studies in which it was assessed [34,36,41,42,48]. The implications/value of the research and fit of the results in context was lacking in 11 out of 17 [37,39,42,44,48,55-60] and 7 out of 15 [37,40,43-45,56,60] studies assessed for these criteria, respectively.

    The critical appraisal skills program cohort checklist assessed confounding, completeness, and the duration of follow-up. The identification and mitigation of confounders was not found to be sufficient in any of the 14 studies assessed using this checklist [37-42,44,45,48,55-58]. A total of 13 studies gave no information on confounders, and the remaining study noted that despite the identification of potential confounders, they were unable to perform fixed or random-effects modeling to assess the impact of these factors [38]. Data on dropout rate were not recorded systematically (2/14) [45,58]; this was primarily because of a lack of substantiation of follow-up duration, information on the timing of follow-up, and/or no indication of whether discontinuations were significant.

    Six cross-sectional studies assessed using the AXIS tool were found to lack justification of sample size, definition of target population (1/6), categorization of (1/6) [50,51] and information on (0/6) nonresponders, validation of outcome measures (2/6) [52,54], repeatability (2/6) [49,51], and internal consistency of results (2/6) [51]. Most of these shortcomings were because of a lack of information. Moreover, 1 of the 2 nonrandomized controlled studies was deemed to have an overall serious risk of bias because of the serious risk of selection bias in determining the intervention groups [46].


    Discussion

    Principal Findings

    In this systematic review, 28 studies evaluating 25 childhood vaccination apps were examined. Overall, there is little evidence to suggest that childhood vaccination apps are effective in improving vaccination coverage, with only 4 of the 9 studies assessing this outcome indicating significant benefit (P≤.05) after versus before the app was introduced. This contrasts with a systematic review and meta-analysis conducted by Harvey et al [14], which revealed a significant benefit of reminder (P<.001), recall (P=.02), reminder and recall (P<.006), and educational initiatives on childhood vaccination rate (P=.02); however, these prompts were not delivered via the mobile app. Our findings also contrast with systematic reviews that have found mobile apps to be effective in eliciting health-related behavior change [62,63].

    Similarly, 4 out of 10 studies assessing the impact on vaccination knowledge and 4 out of 8 studies assessing vaccination decision making reported significant benefit of the app (P≤.05). Furthermore, 3 studies substantiated the dichotomous effect of information provision, thus illustrating the potential for apps to dissuade individuals from vaccination. This is in keeping with evidence from other studies [64,65]. Parental decision making regarding childhood vaccination is understood to be a specific scenario for health-related decision making where parents have been stipulated to put major weight on the subjective perception of the outcome [66]. It is, therefore, important to understand the likely interpretation of the information provided and how this may affect the parental risk-benefit analysis of vaccination.

    The primary functionality of the apps described in the included studies varied; however, most had multiple functions, with the most common features being education, reminders, and record keeping. These were primarily for the use of parents or health care providers; only the iCHRcloud app facilitated the sharing of vaccination record information between parents and physicians [56]. No apps had been designed for the use of school staff, and no apps had the functionality to share information between schools and parents. In the United Kingdom, many childhood and adolescent vaccinations are delivered at schools for convenience and to enhance delivery [67,68]. This could be an avenue for further investigation.

    There is insufficient evidence to draw any conclusions regarding the relationship between the function of the app and the efficacy in improving vaccination rates, knowledge, or positive decision making. The 25 investigated apps had diverse functionalities but were primarily designed for providing vaccine information and/or record keeping. Studies reporting on user statistics revealed that the most popular functions were record keeping, reminders, and information access. Overall, usability, acceptability, and user perceptions of the apps were positive.

    Quality of the Evidence

    The quality assessment of the included studies revealed that many were of poor to moderate quality, indicating an overall high risk of bias, which risks impairing the validity of the conclusions regarding the effectiveness of childhood vaccination apps. One study was determined to have a serious risk of bias. Most negative indicators were because of a lack of information about the criteria assessed. The risk of bias and inadequate robustness may be because of the nature of many of the included studies being pilot, early usability, and preliminary scoping studies. To draw valid and accurate conclusions on the quality of studies, study methods should be comprehensively reported. The infancy of these types of apps also had an impact on the assessment of the implications/value and the fit of the results in context, as many studies indicated that they were the first of their kind in their setting.

    Strengths and Weaknesses of the Study

    The strengths of this study lie in the comprehensive analysis of the available literature discussing apps for childhood vaccination. We investigated ClinicalTrials.gov and ERIC databases, which include gray literature, and we included letters and full-text conference proceedings [50,53]. The inclusion of gray literature minimizes publication bias and ensures that a comprehensive view of the latest literature is reported [69]. We also included all study types, thereby ensuring that apps at every stage of development were reported; a study limited to randomized controlled trials may omit studies of apps in their infancy. One limitation of this review is because of the heterogeneity of the studies and their reported outcomes; therefore, it was not possible to conduct a meta-analysis. A meta-analysis would enable the quantification of the heterogeneity of studies and allow us to quantify the effectiveness of childhood vaccination apps.

    Implications of the Study

    Immunization is a simple and effective mechanism for reducing childhood mortality. Despite the insignificant findings of this review about the effect of apps on the uptake of vaccinations, the positive user perceptions, usability, and acceptability reported present a compelling opportunity to build on the successes of current apps and learn from their shortcomings. Individual studies included in this review reported the potential benefit of these apps on an individual, community, and nationwide level, highlighting the breadth of engagement that can be harnessed with the use of mobile apps [42,45,47]. It is evident that more needs to be done to ensure that a positive change in vaccination knowledge and decision making is paralleled with the provision of sufficient, accessible resources to allow vaccinations to be easily obtained. There are important aspects to consider with regard to the needs of users and how apps will be implemented in health care service delivery.

    Unanswered Questions and Future Research

    Mobile apps will likely play a role in the storage and sharing of vaccination records, generation of reminders, and/or dissemination of vaccination education. Despite several publicly available apps and others in development, a lack of robust evidence remains regarding the effectiveness of vaccination apps in improving vaccination coverage. This systematic review, despite not reporting significant efficacy, indicates that many of the apps convey some degree of benefit with regard to improving vaccination uptake, knowledge, and/or decision making and are widely accepted by their users. For future research, it will be important to understand the priorities of different user groups in terms of app functionalities and the dichotomous effects of vaccination information. Many of the studies included in this review are early-stage investigations and were found to have a relatively high risk of bias. Further investigation of these apps in larger, more robust, controlled trials will allow greater granularity of evaluation and understanding of the role and implications of these apps for wider communities and various subpopulations. In addition, many of the included studies originated from developing countries; however, preventable childhood illnesses are increasing globally. Outcomes from similar studies in developed countries may present a different picture.

    Conclusions

    The objective of this systematic review was to evaluate the effectiveness of childhood vaccinations in improving vaccination uptake, knowledge, and decision making as well as to investigate the usability and patient perceptions of these interventions. Overall, 28 studies describing 25 apps were investigated. Although the apps were generally positively received and had high usability and acceptability scores, there was little evidence to suggest that they were effective in significantly improving uptake, knowledge, or decision making; however, most apps were seen to provide some benefit. This indicates that there is demand and engagement with apps supporting childhood vaccinations; however, further investigation is required.

    The studies investigating these apps were considered to be of poor to moderate quality, likely because of the early phase nature of many of the apps and their respective studies. Only 5 studies were randomized. An additional concern raised by 3 studies was the potential for these apps to discourage vaccination among those who were initially undecided about infant vaccination and among those who had previously intended to vaccinate their children. Future research is warranted into the dichotomous effects of the provision of vaccination information, the outcomes of larger robust studies of these apps, and the needs and priorities of various user populations.

    Acknowledgments

    CC, EM, MI, and MV were supported in this study by EIT Health (grant 18654).

    Authors' Contributions

    MM, MV, and EM conceived the study topic and designed the review protocol. CC and MI screened the studies. CC conducted the data extraction and risk of bias assessment, which were validated by MI. The systematic review was written by CC with revisions from MV, EM, MM, and MI.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategy.

    DOCX File , 14 KB

    Multimedia Appendix 2

    Study characteristics.

    DOCX File , 18 KB

    Multimedia Appendix 3

    Study outcomes and extracted effectiveness results.

    DOCX File , 20 KB

    Multimedia Appendix 4

    Risk of bias assessment.

    DOCX File , 1299 KB

    References

    1. World Health Organization. 2019. Immunization Coverage   URL: https://www.who.int/news-room/fact-sheets/detail/immunization-coverage [accessed 2019-11-25]
    2. Hotez P. America and Europe's new normal: the return of vaccine-preventable diseases. Pediatr Res 2019 Jun;85(7):912-914. [CrossRef] [Medline]
    3. Tanne JH. Measles: two US outbreaks are blamed on low vaccination rates. Br Med J 2019 Jan 21;364:l312. [CrossRef] [Medline]
    4. NHS Digital. Childhood Vaccination Coverage Statistics-England 2018-19   URL: https:/​/digital.​nhs.uk/​data-and-information/​publications/​statistical/​nhs-immunisation-statistics/​england-2018-19 [accessed 2019-11-25]
    5. Hill HA, Elam-Evans LD, Yankey D, Singleton JA, Kang Y. Vaccination coverage among children aged 19-35 months-United States, 2017. MMWR Morb Mortal Wkly Rep 2018 Oct 12;67(40):1123-1128 [FREE Full text] [CrossRef] [Medline]
    6. McKee C, Bohannon K. Exploring the reasons behind parental refusal of vaccines. J Pediatr Pharmacol Ther 2016;21(2):104-109 [FREE Full text] [CrossRef] [Medline]
    7. ScienceDaily. Vaccinations of US Children Declined After Publication of Now-Refuted Autism Risk   URL: https://www.sciencedaily.com/releases/2012/06/120604142726.htm [accessed 2019-11-25]
    8. Graham JE, Mishra A. Global challenges of implementing human papillomavirus vaccines. Int J Equity Health 2011 Jul 30;10:27 [FREE Full text] [CrossRef] [Medline]
    9. Larson HJ, Jarrett C, Eckersberger E, Smith DM, Paterson P. Understanding vaccine hesitancy around vaccines and vaccination from a global perspective: a systematic review of published literature, 2007-2012. Vaccine 2014 May 17;32(19):2150-2159. [CrossRef] [Medline]
    10. Greenlee CJ, Newton SS. A review of traditional vaccine-preventable diseases and the potential impact on the otolaryngologist. Int Arch Otorhinolaryngol 2018 Jul;22(3):317-329 [FREE Full text] [CrossRef] [Medline]
    11. Merrill R. Introduction to Epidemiology. Burlington, Massachusetts: Jones & Bartlett Learning; 2012.
    12. Nelson KN, Wallace AS, Sodha SV, Daniels D, Dietz V. Assessing strategies for increasing urban routine immunization coverage of childhood vaccines in low and middle-income countries: a systematic review of peer-reviewed literature. Vaccine 2016 Nov 4;34(46):5495-5503 [FREE Full text] [CrossRef] [Medline]
    13. Walling EB, Benzoni N, Dornfeld J, Bhandari R, Sisk BA, Garbutt J, et al. Interventions to improve HPV vaccine uptake: a systematic review. Pediatrics 2016 Jul;138(1):e20153863 [FREE Full text] [CrossRef] [Medline]
    14. Harvey H, Reissland N, Mason J. Parental reminder, recall and educational interventions to improve early childhood immunisation uptake: a systematic review and meta-analysis. Vaccine 2015 Jul 9;33(25):2862-2880 [FREE Full text] [CrossRef] [Medline]
    15. Springfield Centre. Springfield Centre. 2012. Making ICT Work For Bangladesh’s Farmers   URL: https://www.springfieldcentre.com/making-ict-work-for-bangladeshs-farmers/ [accessed 2019-11-25]
    16. World Health Organization. mHealth New Horizons for Health Through Mobile Technologies: Based on the Findings of the Second Global Survey on eHealth   URL: https://www.who.int/goe/publications/goe_mhealth_web.pdf [accessed 2019-11-25]
    17. Google Play. Vaccination Apps   URL: https://play.google.com/store/search?q=vaccination&c=apps&gl=GB [accessed 2019-11-25]
    18. Abahussin AA, Albarrak AI. Vaccination adherence: review and proposed model. J Infect Public Health 2016;9(6):781-789 [FREE Full text] [CrossRef] [Medline]
    19. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, New Jersey: Wiley; 2008.
    20. Centre for Reviews and Dissemination. Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care. York, England: University of York; 2009.
    21. van Velthoven M, Milne-Ives M, de Cock C, Mooney M, Meinert E. Use of apps to promote childhood vaccination: a systematic review protocol. JMIR Res Protoc 2019:- epub ahead of print. [CrossRef]
    22. Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak 2007 Jul 15;7:16 [FREE Full text] [CrossRef] [Medline]
    23. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. Br Med J 2015 Jan 2;350:g7647 [FREE Full text] [CrossRef] [Medline]
    24. Higgins JP. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, New Jersey, United States: Wiley; 2008.
    25. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane collaboration's tool for assessing risk of bias in randomised trials. Br Med J 2011 Oct 18;343:d5928 [FREE Full text] [CrossRef] [Medline]
    26. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J 2016 Oct 12;355:i4919 [FREE Full text] [CrossRef] [Medline]
    27. Critical Appraisal Skills Programme. CASP Checklist: 12 Questions to Help You Make Sense of a Cohort Study   URL: https://casp-uk.net/wp-content/uploads/2018/03/CASP-Cohort-Study-Checklist-2018_fillable_form.pdf [accessed 2019-11-20]
    28. Critical Appraisal Skills Programme. CASP Checklist: 10 Questions to Help You Make Sense of a Qualitative Research   URL: https://casp-uk.net/wp-content/uploads/2018/03/CASP-Qualitative-Checklist-2018_fillable_form.pdf [accessed 2019-11-20]
    29. Critical Appraisal Skills Programme. CASP Checklist: 12 Questions to Help You Make Sense of an Economic Evaluation   URL: https:/​/casp-uk.​net/​wp-content/​uploads/​2018/​03/​CASP-Economic-Evaluation-Checklist-2018_fillable_form.​pdf [accessed 2019-11-20]
    30. Downes MJ, Brennan ML, Williams HC, Dean RS. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016 Dec 8;6(12):e011458 [FREE Full text] [CrossRef] [Medline]
    31. Cochrane Training. 2014. Cochrane Review Manager (RevMan)   URL: https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman [accessed 2020-04-28]
    32. GitHub. A Package to Quickly Visualise Risk-of-Bias Assessment Results   URL: https://github.com/mcguinlu/robvis [accessed 2019-11-24]
    33. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009 Oct;62(10):1006-1012. [CrossRef] [Medline]
    34. Chen L, Du X, Zhang L, van Velthoven MH, Wu Q, Yang R, et al. Effectiveness of a smartphone app on improving immunization of children in rural Sichuan province, China: a cluster randomized controlled trial. BMC Public Health 2016 Aug 31;16:909 [FREE Full text] [CrossRef] [Medline]
    35. Fadda M, Galimberti E, Fiordelli M, Romanò L, Zanetti A, Schulz PJ. Effectiveness of a smartphone app to increase parents' knowledge and empowerment in the MMR vaccination decision: a randomized controlled trial. Hum Vaccin Immunother 2017 Nov 2;13(11):2512-2521 [FREE Full text] [CrossRef] [Medline]
    36. Modi D, Dholakia N, Gopalan R, Venkatraman S, Dave K, Shah S, et al. mHealth intervention 'ImTeCHO' to improve delivery of maternal, neonatal, and child care services-a cluster-randomized trial in tribal areas of Gujarat, India. PLoS Med 2019 Oct;16(10):e1002939 [FREE Full text] [CrossRef] [Medline]
    37. Amith M, Zhu A, Cunningham R, Lin R, Savas L, Shay L, et al. Early usability assessment of a conversational agent for HPV vaccination. Stud Health Technol Inform 2019;257:17-23 [FREE Full text] [Medline]
    38. Baldwin AS, Denman DC, Sala M, Marks EG, Shay LA, Fuller S, et al. Translating self-persuasion into an adolescent HPV vaccine promotion intervention for parents attending safety-net clinics. Patient Educ Couns 2017 Apr;100(4):736-741 [FREE Full text] [CrossRef] [Medline]
    39. Atkinson KM, Westeinde J, Ducharme R, Wilson SE, Deeks SL, Crowcroft N, et al. Can mobile technologies improve on-time vaccination? A study piloting maternal use of ImmunizeCA, a pan-Canadian immunization app. Hum Vaccin Immunother 2016 Oct 2;12(10):2654-2661 [FREE Full text] [CrossRef] [Medline]
    40. Gockley AA, Pena N, Vitonis A, Welch K, Duffey-Lind EC, Feldman S. Tablet-based patient education regarding human papillomavirus vaccination in colposcopy clinic. J Low Genit Tract Dis 2019 Jul;23(3):188-192. [CrossRef] [Medline]
    41. Kaewkungwal J, Apidechkul T, Jandee K, Khamsiriwatchara A, Lawpoolsri S, Sawang S, et al. Application of mobile technology for improving expanded program on immunization among highland minority and stateless populations in northern Thailand border. JMIR Mhealth Uhealth 2015 Jan 14;3(1):e4 [FREE Full text] [CrossRef] [Medline]
    42. Kaewkungwal J, Singhasivanon P, Khamsiriwatchara A, Sawang S, Meankaew P, Wechsart A. Application of smart phone in 'Better Border Healthcare Program': a module for mother and child care. BMC Med Inform Decis Mak 2010 Dec 3;10:69 [FREE Full text] [CrossRef] [Medline]
    43. Mbabazi WB, Tabu CW, Chemirmir C, Kisia J, Ali N, Corkum MG, et al. Innovations in communication technologies for measles supplemental immunization activities: lessons from Kenya measles vaccination campaign, November 2012. Health Policy Plan 2015 Jul;30(5):638-644 [FREE Full text] [CrossRef] [Medline]
    44. Salmon DA, Limaye RJ, Dudley MZ, Oloko OK, Church-Balin C, Ellingson MK, et al. MomsTalkShots: an individually tailored educational application for maternal and infant vaccines. Vaccine 2019 Oct 8;37(43):6478-6485. [CrossRef] [Medline]
    45. Dale LP, White L, Mitchell M, Faulkner G. Smartphone app uses loyalty point incentives and push notifications to encourage influenza vaccine uptake. Vaccine 2019 Jul 26;37(32):4594-4600. [CrossRef] [Medline]
    46. Seeber L, Conrad T, Hoppe C, Obermeier P, Chen X, Karsch K, et al. Educating parents about the vaccination status of their children: a user-centered mobile application. Prev Med Rep 2017 Mar;5:241-250 [FREE Full text] [CrossRef] [Medline]
    47. Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine 2016 Jan 4;34(2):276-283 [FREE Full text] [CrossRef] [Medline]
    48. Hategeka C, Ruton H, Law MR. Effect of a community health worker mhealth monitoring system on uptake of maternal and newborn health services in Rwanda. Glob Health Res Policy 2019;4:8 [FREE Full text] [CrossRef] [Medline]
    49. Fadda M, Galimberti E, Fiordelli M, Schulz PJ. Evaluation of a mobile phone-based intervention to increase parents' knowledge about the measles-mumps-rubella vaccination and their psychological empowerment: mixed-method approach. JMIR Mhealth Uhealth 2018 Mar 7;6(3):e59 [FREE Full text] [CrossRef] [Medline]
    50. Devos J, de Haes S. Ghent University Academic Bibliography. 2014. The Proceedings of The 8th European Conference on IS Management and Evaluation ECIME 2014   URL: https://biblio.ugent.be/publication/5697112/file/5697113.pdf [accessed 2019-11-21]
    51. Peck JL, Stanton M, Reynolds GE. Smartphone preventive health care: parental use of an immunization reminder system. J Pediatr Health Care 2014;28(1):35-42. [CrossRef] [Medline]
    52. Kuo M, Lu Y, Chang P. A newborn baby care support app and system for mhealth. NI 2012 (2012) 2012;2012:228 [FREE Full text] [Medline]
    53. Brownstein JS, Huston JE, Steingold L, Joyce MV. On-demand delivery of influenza vaccination. Ann Intern Med 2015 Dec 17;163(10):806-807. [CrossRef] [Medline]
    54. Nourani A, Ayatollahi H, Mirnia K. A smart phone application for the mothers of premature infants. Innov Res BioMed Eng 2019:269. [CrossRef]
    55. Bednarczyk RA, Frew PM, Salmon DA, Whitney E, Omer SB. ReadyVax: a new mobile vaccine information app. Hum Vaccin Immunother 2017 May 4;13(5):1149-1154 [FREE Full text] [CrossRef] [Medline]
    56. Singh H, Mallaiah R, Yadav G, Verma N, Sawhney A, Brahmachari SK. iCHRCloud: web & mobile based child health imprints for smart healthcare. J Med Syst 2017 Dec 29;42(1):14. [CrossRef] [Medline]
    57. Wilson K, Atkinson K, Pluscauskas M, Bell C. A mobile-phone immunization record in Ontario: uptake and opportunities for improving public health. J Telemed Telecare 2014 Dec;20(8):476-480. [CrossRef] [Medline]
    58. Wilson K, Atkinson KM, Penney G. Development and release of a national immunization app for Canada (ImmunizeCA). Vaccine 2015 Mar 30;33(14):1629-1632. [CrossRef] [Medline]
    59. Burgess K, Atkinson KM, Westeinde J, Crowcroft N, Deeks SL, Wilson K, Public Health Agency of Canada/Canadian Institutes of Health and Research Influenza Research Network (PCIRN) Program Delivery and Evaluation Group. Barriers and facilitators to the use of an immunization application: a qualitative study supplemented with Google analytics data. J Public Health (Oxf) 2017 Sep 1;39(3):e118-e126 [FREE Full text] [CrossRef] [Medline]
    60. Karanth SS, Lairson DR, Savas LS, Vernon SW, Fernández ME. The cost of developing a computerized tailored interactive multimedia intervention vs a print based Photonovella intervention for HPV vaccine education. Eval Program Plann 2017 Aug;63:1-6. [CrossRef] [Medline]
    61. Ruiz-López T, Sen S, Jakobsen E, Tropé A, Castle PE, Hansen BT, et al. FightHPV: design and evaluation of a mobile game to raise awareness about human papillomavirus and nudge people to take action against cervical cancer. JMIR Serious Games 2019 May 8;7(2):e8540 [FREE Full text] [CrossRef] [Medline]
    62. Han M, Lee E. Effectiveness of mobile health application use to improve health behavior changes: a systematic review of randomized controlled trials. Healthc Inform Res 2018 Jul;24(3):207-226 [FREE Full text] [CrossRef] [Medline]
    63. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015 Mar 26;3(1):e20 [FREE Full text] [CrossRef] [Medline]
    64. Wilson K, Mills EJ, Norman G, Tomlinson G. Changing attitudes towards polio vaccination: a randomized trial of an evidence-based presentation versus a presentation from a polio survivor. Vaccine 2005 May 27;23(23):3010-3015. [CrossRef] [Medline]
    65. Nyhan B, Reifler J, Richey S, Freed GL. Effective messages in vaccine promotion: a randomized trial. Pediatrics 2014 May;133(4):e835-e842. [CrossRef] [Medline]
    66. Damnjanović K, Graeber J, Ilić S, Lam WY, Lep Z, Morales S, et al. Parental decision-making on childhood vaccination. Front Psychol 2018;9:735 [FREE Full text] [CrossRef] [Medline]
    67. Lindley MC, Boyer-Chu L, Fishbein DB, Kolasa M, Middleman AB, Wilson T, Working Group on the Role of Schools in Delivery of Adolescent Vaccinations. The role of schools in strengthening delivery of new adolescent vaccinations. Pediatrics 2008 Jan;121(Suppl 1):S46-S54. [CrossRef] [Medline]
    68. O'Neill J, Newall F, Antolovich G, Lima S, Danchin M. The uptake of adolescent vaccinations through the school immunisation program in specialist schools in Victoria, Australia. Vaccine 2019 Jan 7;37(2):272-279. [CrossRef] [Medline]
    69. Paez A. Grey literature: an important resource in systematic reviews. J Evid Based Med 2017 Dec 21:- epub ahead of print. [CrossRef] [Medline]


    Abbreviations

    AXIS: Appraisal tool for Cross-Sectional Studies
    CENTRAL: Cochrane Central Register of Controlled Trials
    EPI: Expanded Program on Immunization
    ERIC: Education Resources Information Center
    HPV: human papillomavirus
    MeSH: Medical Subject Headings
    mHealth: mobile health
    ROBINS-I: risk of bias in nonrandomized studies of interventions


    Edited by G Eysenbach; submitted 10.12.19; peer-reviewed by C Fernández, M Das; comments to author 03.01.20; revised version received 13.03.20; accepted 22.03.20; published 18.05.20

    ©Caroline de Cock, Michelle van Velthoven, Madison Milne-Ives, Mary Mooney, Edward Meinert. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 18.05.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.