Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 15.04.19 in Vol 7, No 4 (2019): April

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11620, first published Jul 31, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Evaluation of Mobile Apps Targeted to Parents of Infants in the Neonatal Intensive Care Unit: Systematic App Review

    1School of Nursing, Dalhousie University, Halifax, NS, Canada

    2Faculty of Health, Dalhousie University, Halifax, NS, Canada

    3Centre for Pediatric Pain, Izaak Walton Killam Health Centre, Halifax, NS, Canada

    4School of Information Management, Dalhousie University, Halifax, NS, Canada

    5Division of Neonatal Perinatal Medicine, Department of Pediatrics, Faculty of Medicine, Dalhousie University, Halifax, NS, Canada

    6Division of Neonatal Perinatal Medicine, Department of Pediatrics, Faculty of Medicine, Izaak Walton Killam Health Centre, Halifax, NS, Canada

    Corresponding Author:

    Marsha Campbell-Yeo, NNP-BC, RN, PhD

    School of Nursing

    Dalhousie University

    PO Box 15000, Forrest Bldg

    5869 University Avenue

    Halifax, NS, B3H4R2

    Canada

    Phone: 1 9024944283

    Email: marsha.campbell-yeo@dal.ca


    ABSTRACT

    Background: Parents of preterm infants increasingly use their mobile phone to search for health information. In a recent review, websites targeted toward parents with infants in the neonatal intensive care unit (NICU) were found to have poor to moderate quality educational material; however, there is a dearth of literature regarding mobile apps for NICU parents.

    Objective: This study aimed to identify and evaluate apps targeting parents of infants in the NICU for quality of information, usability, and credibility.

    Methods: We systematically searched the Apple App Store and Google Play using 49 key terms (eg, “preterm infant”) from July 26 to August 18, 2017. English apps targeting NICU parents that cost less than $20 were included. Apps for health care professionals, e-books/magazines, or nonrelevant results were excluded. In total, 3 tools were used for evaluation: Mobile Application Rating Scale (MARS) to measure quality; Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV) to measure the app’s content usability; and Trust it or Trash It to measure credibility.

    Results: The initial search yielded 6579 apps, with 49 apps eligible after title and description screening. In total, 27 apps met the eligibility criteria with 9 apps available in both app stores; of those, the app with the most recent update date was chosen to be included in the analysis. Thus, 18 unique apps were included for final analysis. Using MARS, 7 apps (7/18, 39%) received a good score on overall quality (ie, 4.0 out of 5.0), with none receiving an excellent score. In addition, 8 apps (8/18, 44%) received a PEMAT-AV score between 51% and 75% on the understandability subscale, and 8 apps (8/18, 44%) scored between 76% and 100% on the actionability subscale. Trust It or Trash It deemed 13 apps (13/18, 72%) as trash for reasons including no identification of sources or lack of current information, with only 5 (5/18, 28%) deemed trustworthy. Reviewer’s expert evaluation found 16 apps contained content that matched information provided by multiple sources; however, most apps did not meet other objective measurement items to support credibility. When comparing the MARS overall quality and subjective quality scores with trustworthiness of apps, there was no statistically significant difference. A statistically significant difference was found between the 2 MARS quality scores, indicating that, on average, apps were ranked significantly lower on subjective quality compared with overall quality measures.

    Conclusions: This evaluation revealed that of the available apps targeting NICU parents, less than half should be considered as acceptable educational material. Over two-thirds of the apps were found to have issues regarding credibility and just over a quarter were considered good quality. The apps currently available for NICU parents are lacking and of concern in terms of quality and credibility.

    JMIR Mhealth Uhealth 2019;7(4):e11620

    doi:10.2196/11620

    KEYWORDS



    Introduction

    Background

    More than 1 in 10 babies are born preterm (ie, before 37 weeks gestation) worldwide and are often admitted to the neonatal intensive care unit (NICU) to help support their survival, growth, and development [1]. Parents of these infants requiring NICU care typically perceive this experience to be incredibly stressful, emphasized by the unfamiliar medical environment, appearance or behavior of their newborn, and interruptions in developing their role as a parent [2-7]. Unsurprisingly, parents strongly value being informed about their infant’s condition by the NICU care team; however, their communication needs often go unmet. This is often because of the amount of information, either too much or too little information, and timing that health care providers deliver the said information [8]. As a result, parents strive to regain a sense of control by seeking additional information from external sources, including other health care providers, written educational materials, and the internet [8]. Parents often consult the internet, even before a health care provider, using the search engine Google or social media to search for advice related to their child’s health and well-being [9-11]. More specifically, parents with infants in the NICU have also been found to prefer accessing the internet with their mobile phone when seeking information about their infant’s health, with a trend in younger NICU parents preferring to use mobile apps [8,12,13].

    Given the prevalence of mobile phone ownership worldwide, leveraging this medium to disseminate health information provides a great opportunity to better support overall health practices [14-16]. Not surprisingly, there has been a surge in the development of mobile health (mHealth) apps [17]. The use of mHealth apps by new parents has been increasing, often during the perinatal period, with mothers reporting an interest in or use of apps to monitor their health and their family’s well-being [18-21]. Although mHealth apps have great potential, there are currently no formal quality standards that are required when developing these resources; thus, little is known about the quality of mHealth apps [14,22].

    Broadly, concerns have been reported related to the quality of general health content on the internet [23] and considering this highly vulnerable population, it is imperative to evaluate the web-based health resources targeted to parents of infants in the NICU to ensure they are accessing current and evidence-informed information. To address this, we recently conducted a systematic review of websites available through Google that were targeted to NICU parents and found websites overall to be moderate to poor in terms of their reliability and quality of information [24]. A recent review evaluated parenting apps [25]; however, there was no emphasis on apps for parents with infants in the NICU. Building on our systematic review of Google, this review sought to evaluate the quality of current apps to provide further insight into how best to meet the informational needs of NICU parents.

    Objectives

    The primary objective of this review was to identify and evaluate apps available to parents of infants in the NICU for quality (ie, quality of health information and overall design), usability (ie, clarity and applicability of the health information), and credibility (ie, accuracy and reliability of the content). This study had the following research questions:

    1. What is the quality of mobile apps targeted to parents of infants in the NICU?
    2. What is the usability of the content within mobile apps targeted to parents of infants in the NICU?
    3. Is the information provided by mobile apps targeted to parents of infants in the NICU credible?

    In addition, a secondary objective was to explore common databases to determine if any peer-reviewed literature regarding the apps included for full review has been published and if so, what has been reported on them.


    Methods

    Study Design

    Although there are some differences from traditional review methods related primarily to the search strategy, this study followed systematic review methodology, adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standards [26]. A protocol for this review was developed a priori and registered through PROSPERO International prospective register of systematic reviews [27]. This study is consistent with numerous recent reviews of mHealth apps using the same methodological approach [25,28-33].

    Search Strategy

    To identify the appropriate records for this review, the Canadian Apple App Store and Google Play were used as databases to search for mobile apps. In 2017, Apple and Android mobile phones accounted for 99.7% of the new market share [15]. Although not all Android phones access Google Play, it has been rated the leading app store offering approximately 1.6 million apps available for download, with Apple App Store in second place offering 1.5 million apps [34]. Thus, the search was limited to Apple App Store and Google Play, given the substantial number of apps offered by these stores. This is consistent with other studies reviewing mobile apps [35].

    To ensure a comprehensive inquiry, this review conducted a systematic search using a 2-step approach developed in collaboration with a librarian (KR). In step 1, we searched both app stores using 21 relevant key terms (eg, “parenting,” “newborn,” “preterm infant,” “preemie,” “premature baby,” “neonatal intensive care unit,” “NICU,” and “neonatology”). This was followed by step 2 where each store was searched again using string keywords by inputting multiple forms of parent (eg, “parent,” “caregiver,” “guardian,” “mother,” and “father”) in combination with the following terms: “neonatal intensive care unit,” “NICU,” “special care nursery,” and “neonatal.” This 2-step approach was utilized because of differing search methods between app stores at the time of this review, with the search algorithm for Apple optimized by using single keyword and the algorithm for Google Play with string keywords [35-37].

    Selection Criteria

    For feasibility, the search was limited to the top 100 apps identified from each search term applied, which is consistent with previous studies reviewing apps [38]. This limitation was applied as searches within the Apple App Store will continually refresh with additional apps of increasingly less relevancy. Apps that met inclusion criteria and were available in both stores were evaluated separately at the screening stage to assess if there were any substantial differences across operating systems. However, to limit redundancy during the final stage of this review, the most recently updated app as identified in the store description was kept for analysis. Snowball searching through recommended apps was conducted; however, no new apps were found that met the criteria that were not identified in the original search.

    Inclusion Criteria

    Although the term parent will be used throughout this review, this term will be considered inclusive of guardians, additional family or individuals that provide care to infants in the NICU. We chose against using the term caregiver as it is often considered more suggestive of a health care provider role within this context. Apps targeting parents related to the NICU experience were eligible for inclusion. To ensure a broad reach of available apps, there were no restrictions on the app’s purpose. For example, apps could be for awareness, education, or tracking growth data, or reducing stress and enhancing coping. The following inclusion criteria was determined a priori: (1) apps targeted to parents of infants in the NICU; (2) available through Apple App Store or Google Play, accessible in Canada; (3) English language; (4) free or paid apps costing less than Can $20 (a consistent cut-off with similar studies as general users are unlikely to spend more than $10 per app [32]); (5) apps available in the following Apple App Store categories: Health & Fitness, Lifestyle, Medical, and Social Networking; and (6) apps available in the following Google Play categories: Communication, Education, Health & Fitness, Lifestyle, Medical, Parenting, and Social.

    Exclusion Criteria

    Apps were excluded if they were (1) general parenting apps, not related to the NICU experience; (2) intended for health care professionals; (3) classified as e-books by app store description or reviewers.

    Screening Process and Data Extraction

    After removal of duplicates, the title and store descriptions of all apps identified in the initial search were screened by 6 reviewers (BR, JD, KR, AO, JM, and KH) to determine eligibility for full review. For data extraction, 3 independent reviewers (BR, AO, and JM) were trained on how to use the measurement tools to ensure consistency when evaluating the apps. For full review, 1 reviewer was assigned to evaluate Android apps (AO), another for Apple apps (JM), and a third reviewer evaluated all apps (BR) using devices from both operating systems. Any disagreements were resolved by consensus or with a fourth reviewer (JD). Apps that were eligible for full review were downloaded and evaluated using 4 mobile phones, 2 Android (HTC One M7; LG G4) and 2 Apple (iPhone 5s; iPhone 6s). To address the primary objectives of this review, data were extracted using a structured data retrieval form compiled using the following 3 measurement tools:

    Mobile App Rating Scale

    The Mobile App Rating Scale (MARS) tool is designed to gather descriptive data and assess the quality of mHealth apps using an objective and reliable method [39]. This tool evaluates an average overall app quality through 4 core subscales: engagement, functionality, aesthetics, and information quality. The tool also offers 2 optional subscales to support the evaluation: app subjective quality and perceived impact of the app on user knowledge and behaviors. See Textbox 1 for the definition of each core subscale. Each item within the core subscales uses a 5-point scale (1=Inadequate, 2=Poor, 3=Acceptable, 4=Good, and 5=Excellent). Subscale scores can be isolated to determine strengths and limitations of the app under evaluation. Subscale mean scores are then summed and averaged to create an overall mean quality assessment. Mean scores are calculated for each subscale and scores classifications are consistent with item responses (ie, 1=Inadequate, 2=Poor, 3=Acceptable, 4=Good, and 5=Excellent). For this study, we report on each scale individually as well as report a total score.


    Textbox 1. Definitions of Mobile App Rating Scale core subscales.
    View this box

    Textbox 2. Example of Patient Education Materials Assessment Tool for Audiovisual Materials items.
    View this box
    The Patient Education Materials Assessment Tool for Audiovisual Materials

    This tool systematically evaluates content usability through the understandability and actionability of audiovisual patient education materials [40]. The Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-AV) assesses if materials are understandable and key messages are clear to a diverse population with varying levels of health literacy. By evaluating actionability, the PEMAT-AV assesses if a diverse population can identify what they can or need to do based on the information provided in the education material. Each item has 3 options for scoring (0=Disagree, 1=Agree, and NA=Not Applicable), and overall scores for each subscale are summed and divided into percentage quartiles, with a potential range of 0% to 100%. See Textbox 2 for an example of items to assess understandability and actionability.

    Trust It or Trash It

    To support assessment of credible and unbiased resources, Trust It or Trash It provides guidance on how to critically evaluate the quality of health information provided in health resources [41]. This tool uses 6 questions to help determine the validity and reliability of the resources: Who wrote the information you are reading? Who provided the facts/Where did the facts come from? Who paid for it? When was it written or updated? How do you know this information pertains to you? Does the information seem reasonable based on what you’ve read or know? Each question has an associated description to explain how to evaluate appropriately to either a trust or trash the resource. Resources will be considered as trash if receiving this option for at least 3 questions from this tool. Trust It or Trash It is a relatively simple tool that will complement the remaining measurement tools and strengthen the rigor of this review.

    The secondary objective, to identify the existence of peer-reviewed publications of included apps, was conducted by searching the app name included in the full review in PubMed and Google Scholar. This search was conducted in conjunction with data extraction in Fall 2017.

    Data Analysis

    Data analysis was conducted using IBM SPSS 24.0. Descriptive statistics and frequencies were used to summarize the results of the evaluations from each measurement tool. As both the MARS tool and Trust It or Trash It tool evaluated quality, we conducted independent t test to compare MARS overall quality scores and subjective quality scores on whether an app was found to be trustworthy, as per the Trust It or Trash It tool. Additionally, a dependent t test was conducted to compare between MARS overall quality scores and subjective quality scores.


    Results

    Screening Process

    After systematically searching both stores and manually inputting search findings into a Microsoft Excel spreadsheet over a month in 2017, our initial search yielded 6578 apps. After title and description screening, a total of 49 were assessed for eligibility. A total of 27 apps were included for full review and evaluated by 3 independent reviewers. Moreover, 9 apps were excluded as they were available on both stores; exclusion was based on oldest date as identified in the description, including 6 from Google Play and 3 from Apple App Store. There were 18 unique apps remaining that were included in the final analysis: 6 from Google Play and 12 from the Apple App Store. See Figure 1 for the PRISMA flow diagram outlining the screening process [26].

    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of search process. NICU: neonatal intensive care unit.
    View this figure

    Descriptive Characteristics of Apps

    Of the 18 unique apps, 89% (16/18) had a last update year of 2015 or later, of which 61% (11/18) were updated in 2017. Although 94% (17/18) of apps were free to download, the 1 app that required payment to download was last updated in 2015. Moreover, 3 apps used commercial advertisements and 1 had in-app purchases, costing users Can $20.99 for a subscription to access more content. The costs of in-app purchases were not identified in the app store description. Although this app was included for full review, only content that was freely accessible was evaluated. Half of the apps were developed in the United States (n=9), with the remaining coming from the United Kingdom (n=3), Australia (n=1), New Zealand (n=1), Canada (n=1), Greece (n=1), or were of unknown origin (n=2). The majority of apps appeared to have been developed by a reputable source including nongovernmental organizations (n=12), government agencies (n=3), or university (n=1). App user rankings are represented by star ratings (5-point scale) in the Apple App Store and Google Play. None of the apps from Apple had an overall star rating, stating that they have not received enough reviews or ratings to rank. Moreover, 4 apps from Google Play had enough user ratings to provide an overall score, ranging from 3 to 5 stars with a median of 4 stars. The number of users who rated the apps ranged from 1 to 127. Table 1 provides a complete list of the included apps and their characteristics (for full table, see Multimedia Appendix 1). Using theoretical strategies predefined by the MARS tool, the purpose of the apps was primarily for providing information or education (n=16), advice or tips (n=12), or monitoring or tracking data (n=10), as shown in Table 2. Although the apps covered a wide range of topics related to the NICU experience, those most commonly addressed were breastfeeding or feeding (n=18), growth and development (n=14), and illness or health issues (n=13). See Table 3 for a list of the topics identified in the apps.

    Table 1. Description of apps.
    View this table
    Table 2. Theoretical strategies classified by the Mobile Application Rating Scale (N=18).
    View this table
    Table 3. Full list of topics covered (N=18).
    View this table

    Quality Assessment Using Mobile Application Rating Scale

    Using the MARS tool, the average overall quality MARS score of the 18 apps ranged from 2.33 to 4.31, with an average of 3.37 (median 3.37). Less than half of apps (7/18, 39%) received an acceptable score (range: 3.26-3.72) on overall quality, with 28% (5/18) receiving a good score (range: 4.06-4.31), and no apps receiving an excellent score. Overall, apps scored low on engagement (1.00-4.6, mean=2.68) and moderate on functionality (2.75-5.00, mean=3.93), aesthetics (1.3-4.67, mean=3.21), and information quality (2.75-4.50, mean=3.65). See Table 4 for results across each core subscales and overall quality. Interestingly, the 5 apps to receive a good score in aesthetics were the only apps to receive a good score in overall quality. Within the engagement subscale, app’s capabilities for customization were evaluated with 50% of apps having no (n=7) or insufficient options customization (n=2) and the other 50% of apps having basic (n=5) or numerous (n=4) options for customization. The subjective quality of apps varied, with only 11% (2/18) of apps receiving a good score. As part of the subjective quality assessment, evaluators determined 72% (n=13) of apps would likely be used 3 to 10 times (n=6), 1 to 2 times (n=4), or not used at all (n=3) in the next 12 months. Apps varied in their perceived impact on parental knowledge, attitudes, and behavior, with 22% (4/18) of apps receiving an acceptable score, 11% (2/18) receiving a good score, and 6% (1/18) receiving an excellent score. See Multimedia Appendix 1 for the individual app results across each subscale and overall app quality. Quantum Caring for Parents received the highest overall quality score (mean=4.31), followed by MyPreemie App (4.16), NICU Companion (4.10), Babble (4.08), and Integrated Family Delivered Care (IFDC; 4.06).

    Patient Education Materials Assessment Tool for Audiovisual Materials Evaluation

    In terms of usability reflected by health literacy, using the PEMAT-AV, 44% (8/18) of the apps received a score between 51% and 75% on understandability and 44% (8/18) were within 76% to 100% on actionability. One app that was solely for tracking and monitoring data was removed from this analysis as the app contained no educational content. See Table 5 for PEMAT-AV results by quartiles.

    Table 4. Mobile App Rating Scale average scores.
    View this table
    Table 5. Patient education material assessment tool audiovisual scores.
    View this table

    Trust It or Trash It

    The credibility of apps appears lacking, with 72% (13/18) of apps receiving a trash score and only 28% (5/18) deemed as trustworthy. Over 80% (15/18) of apps received a trash score on the following questions: “Who provided the facts or Where did the facts come from?” and “When was it written or updated?” However, with the question “Does the information seem reasonable based on what you’ve read or know?,” nearly all apps (n=16) were considered to match the information found in multiple sources based on the reviewer’s expert evaluation.

    Overall Highest Ranking

    The app to rank the highest across all measurement outcomes was IFDC. We conducted a subanalysis comparing quality measures between the MARS tool and Trust It or Trash It tool. There was no statistically significant difference in overall quality of apps reported in the MARS between the apps that were deemed trustworthy (mean 3.65, SD 0.75) compared with those that were not (mean 3.26, SD 0.57) in the Trust It or Trash It tool, t16=−1.20, P=.25. Similarly, no statistically significant difference in the subjective quality of apps was found between apps that were trustworthy (mean 2.85, SD 1.22) compared with those that were not (mean 2.33, SD 0.89), t16=−1.01, P=.33. When comparing within the MARS quality scores of apps, on average apps had a higher overall quality score (mean 3.37, SD 0.63) than the subjective quality score (mean 2.47, SD 0.98), t17=−7.05, P<.001.

    Secondary Objective

    Only 1 app, MyPreemie App, had a relevant publication identified through a search in both PubMed and Google Scholar. The paper on MyPreemie App, published in 2013, was a description article that did not report any outcomes on feasibility, uptake, or impact on parent or newborn outcomes [42]. An abstract discussing MyChildren’s app was retrieved, but it provided no mention of the NICU-specific content [43].


    Discussion

    Principal Findings

    Parents in the NICU are utilizing the internet and mobile apps easily accessible to them through their mobile phones to specifically search for more information about their infant’s health and well-being [12]. To the authors’ knowledge, this is the first and only review of apps targeting parents of infants in the NICU. Consistent with our previous work that found websites targeted to parents with infants in the NICU to be of moderate to low quality[24], this review found the apps currently available for the same population also lacking in quality and credibility. Overall, just over a quarter of apps were considered good quality, less than half acceptable in terms of educational materials, and over two-thirds as having issues related to credibility.

    Over a third of the reviewed apps provided information on key topics regarding the NICU experience including breastfeeding or feeding, growth and development, and illness or health issues. However, information on topics that have been identified as prevalent concerns among NICU parents, such as attachment [44], infant pain [45], and death or loss [46,47], were found within only a small number of apps. Unsurprisingly, the topics covered and the proportion of apps related to these topics was comparable with our previous review of websites [24]. Although topics most commonly covered in the apps align with the increasing emphasis on family-centered or family-integrated care in the NICU, there are still gaps in addressing what parents identify as important. Many apps were updated in 2017, the same year when the review was conducted, and the “What’s New” statements in the app store descriptions included mainly technical aspects, such as fixing delays, as app version updates only occur when there are changes to the software code. Thus, it was difficult to determine if the content was updated within the apps, especially as most apps did not disclose their sources.

    Quality

    Similar findings have been identified from recent systematic reviews of various health-related apps [25,29,32,33] predominantly in terms of quality and credibility. Among other studies using the MARS tool, engagement scores of mHealth apps have been consistently low across reviews of mindfulness-based apps [32], cardiovascular disease symptom monitoring and self-management apps [29], and asthma self-management apps [28]. The focus of these reviews suggests that these apps should be inherently engaging to ensure target users believe they are valuable resources to improve their health and well-being. Additionally, our review found only half of apps had little to no options for customization limiting the user’s ability to engage and personalize the app. Using the MARS subjective quality subscale, our expert reviewers evaluated that on average the majority of apps included in this review would likely be used only 0 to 10 times within the next year by NICU parents, which may not be sufficient or valuable for supporting parents throughout their NICU experience. However, this is in contrast to findings that reported almost one hundred percent of NICU parents use the internet or their mobile phones daily [12]. Again, consistent with our evaluation, these reviews found apps to have high scores for their functionality, yet moderate in terms of aesthetics, information quality, and overall quality [28,29,32] This trend suggests that future apps should strive to improve engagement, aesthetics, and information quality to strengthen the overall quality of apps. In addition, a recent review of apps targeting adults with chronic lung disease specifically reported that none of the apps included in their evaluation provided source identification [33]. Although important across all mHealth apps, it is of particular importance for parents with infants in the NICU to have timely access to credible, yet also engaging and understandable information as they often experience feelings of vulnerability and stress during this highly sensitive period [8,48,49].

    Of the apps reviewed, most apps received at least an acceptable score in overall quality using the MARS tool. Although the Quantum Caring for Parents app had the highest overall quality MARS score, this app requires a paid subscription (ie, Can $20.99) to access all of the content, which may not be feasible for all parents. Although the other highest-ranking apps were free, it is important to note how the financial support model for these free apps (eg, grant funding) could impact quality over time if the developers are not able to sustain the app. Although outside of the scope of this review, it would be interesting to explore the relationship between app quality and financial support. Only 3 apps received a high score on perceived impact on parental knowledge, attitudes, and behavior change: IFDC, Quantum Caring for Parents, and Peekaboo ICU Preemie (see Multimedia Appendix 1). Interestingly, Peekaboo ICU Preemie was not ranked among the highest in overall quality, but based on expert evaluation, the information within this app was found to likely enhance parents learning and feelings to support their NICU experience, despite receiving lower scores in the MARS core subscales. Unsurprisingly, a statistically significant difference was found between overall quality scores and subjective quality scores, with subjective quality ranking lower on average. This finding is expected as in comparison with the overall quality criteria (ie, the 4 core subscales), the subjective quality items are particularly broad. For example, 4 questions ask for the reviewers’ opinion on star rating, recommending the app, anticipated time using the app, and paying for the app. Those receiving a good score in the aesthetics subscale scored highest overall, suggesting these apps may have been developed with greater attention or to be in line with a fundamental theory of design in which attractive things work better [50]. The design of web-based resources, both websites or apps, should be a pleasurable experience for users and stimulate an overall positive response when interacting with the resource [51]. Design experts have outlined certain harsh conditions that produce a negative response often apparent in the NICU environment, including bright lights, loud abrupt noises, bodies that appear abnormal, etc [50]. Thus, it is important for these apps to use design techniques to enhance aesthetics as to not add to an already stressful experience in the NICU.

    Usability and Credibility

    The information within many of the apps was found to be understandable and actionable by the general population as they provided content that used lay language or visual cues to identify important points. Interestingly, subscores related to quality were the most inconsistent across tools. Our subanalyses showed that the Trust It or Trash It tool was not correlated with MARS overall or subjective quality scores. On the basis of expert review using the MARS tool, apps received acceptable to good scores for the information quality subscale with most scoring high on item 15 (ie, is the app content correct, well written, and relevant to the goal/topic of the app) [39]. In contrast, when the same apps were measured using the Trust It or Trash It tool, the credibility of most apps was found to be questionable. There are several possible reasons for this difference. One reason may be that The Trust it or Trash Tool consists of 6 items, specific to quality, whereas the MARS tool is a single item. Additionally, the MARS tool does not include important questions related to credibility, such as identification of sources. Given the limitations of the MARS tool, a supplementary assessment of credibility should be considered. However, it is important to emphasize that although it is crucial to use criteria to assess credibility, current mobile apps may not weigh the criteria as strongly during development; that is, app developers may use evidence-based information in the development of content, yet not provide end users with where they obtained that information. Thus, it is possible the content may not necessarily be untrustworthy but simply does not provide a source and therefore, received a negative score using the Trust It or Trash It tool. However, this lack of provision of sources to parents would not be captured when solely evaluating using the MARS tool, resulting in the disconnect between these 2 scales. Parents are often limited in their ability to assess the reliability of health content, and mHealth apps that contain educational information should be held responsible to disclose important aspects related to credibility including original sources, author names, and bibliographies. Although not a focus of this review, it is important to note that 1 limitation of this review relates to the limitations of the tools currently available to evaluate the strength of health-related apps. The development of novel tools appears to be warranted.

    Secondary Objective

    Relative to our secondary objective, item 19 of the MARS tool determines to what degree the app has been trialed and is considered to be evidence based. At the time of evaluation, only 1 app had a peer-reviewed publication associated with it (MyPreemie App); however, the identified study provided a description of the app’s content and features with no indication of empirical testing [42]. There is a published abstract discussing MyChildren’s app; however, there is no mention of the NICU specific content that was evaluated in this review [43]. Although identified after this review had been completed, the IFDC app was included in a recent publication regarding the developer’s larger parent education project. However again, it appears the app was not tested; thus, although meeting our secondary objective, it would not impact the MARS score [39]. The low quantity of peer-reviewed literature is not unique to our review. Creber et al in 2016 found only 3 publications out of 34 identified apps related to cardiovascular disease management [29], whereas Owens et al’s review (2018) of apps regarding chronic lung disease [33]and Tinschert et al on asthma-related apps found no publications [28]. The lack of existing peer-reviewed literature may put the apps identified in this review at a disadvantage as health care providers are less likely to recommend use of apps to patients and families if there is no evidence supporting their benefit. Thus, it is important for mHealth apps to provide an assessment of app quality through effectiveness studies and peer-reviewed publications [29].

    Limitations

    Despite following a rigorous systematic approach with expert reviewers, synthesizing and evaluating mobile apps is still a relatively new concept and thus, there are some limitations to address. We did not include reviewers from outside the country and because of this, only apps accessible in Canada were included. Manually gathering apps during the initial search significantly prolonged this stage of the review. However, this has been resolved for future reviews as 1 author (MS) has since developed a method to retrieve all app data relevant for the search stage. Reviewers with different devices and operating system versions experienced differences in functionality and app store description information; however, this was resolved through discussion and briefly reviewing the apps again with each device to come to a consensus. This could be avoided in future reviews by using devices that are strictly for research purposes (ie, not the reviewer’s personal mobile phone). Due to the fact that we wanted to get an overview of all apps currently available to NICU parents, we not only included apps that were predominantly for monitoring and tracking data but also included other components such as a diary or a section to prompt questions to ask health care professionals. However, it was difficult to assess these apps using the PEMAT-AV as there was little health information within these apps. Moreover, we found that the PEMAT-AV may not be adequate to evaluate health literacy of mobile apps as the evaluation criteria have not yet been adapted to the unique variances within a mobile app platform. In addition, although the MARS tool was developed using rigorous methods, we would argue that most items in the core subscales elicit subjective responses, which could limit the objectivity and replicability of the measurement. The possibility for complete replicability of this review is limited because of the nature of mobile apps and various fluid components, including app ranking within stores, differences in devices, and recent updates of apps. Similarly, the findings of this review should be interpreted with some caution as the evaluation is based on the app version at the time of assessment, and thus, all future versions may differ in overall quality, usability, and credibility.

    Conclusions

    Despite the number of available apps for parents of infants in the NICU, this systematic review revealed that current mobile app resources vary in quality, usability, and credibility, with generally low scores. Additionally, peer-reviewed literature or empirical studies related to the identified apps are nearly nonexistent. Parents should be aware of the issues of quality and credibility identified in this review and be cautious when using an app for health information. This expert review was beneficial to provide a preliminary evaluation on the resources easily accessible by a parent’s mobile phone. Building on this, usability testing or content analysis of the top 5 apps could be warranted to further explore how parents interact with the apps and provide a thorough evaluation examining the impact of these resources on parent learning needs, parent engagement, and neonatal outcomes. In addition, further attention to the development of high quality, credible resources targeted to NICU parents is needed.

    Acknowledgments

    The authors gratefully acknowledge the Mechanisms, Outcomes, Mobilization of maternally Led Interventions in Newborn Care research lab at the IWK Health Centre with their support for this publication through the ACOA Atlantic Innovation Fund, Project Number: 207518 “Chez NICU Home: Transforming health care delivery in neonatal care through greater parental presence and involvement.”

    BR receives support by the Dalhousie University School of Nursing Doctoral Scholarship and Helen Watson Memorial Scholarship, the Nova Scotia Health Research Foundation Scotia Scholars Award, BRIC NS Student Research Award, and Chez NICU Home Doctoral Scholarship. MC-Y is supported by a Canadian Institutes of Health Research New Investigator Award. AO is supported through an IWK Health Centre Summer Studentship Award. JD is funded through a Canadian Institute of Health Research Doctoral Award (FRN154341) as a PhD in Health trainee at Dalhousie University.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Complete version of Table 1. Description of apps and Table 5. Mobile App Rating Scale average scores per app.

    PDF File (Adobe PDF File), 57KB

    References

    1. World Health Organization. 2018. Preterm birth   URL: http://www.who.int/en/news-room/fact-sheets/detail/preterm-birth [WebCite Cache]
    2. Ray JG, Urquia ML, Berger H, Vermeulen MJ. Maternal and neonatal separation and mortality associated with concurrent admissions to intensive care units. CMAJ 2012 Dec 11;184(18):E956-E962 [FREE Full text] [CrossRef] [Medline]
    3. Grosik C, Snyder D, Cleary GM, Breckenridge DM, Tidwell B. Identification of internal and external stressors in parents of newborns in intensive care. Perm J 2013;17(3):36-41 [FREE Full text] [CrossRef] [Medline]
    4. Leigh B, Milgrom J. Risk factors for antenatal depression, postnatal depression and parenting stress. BMC Psychiatry 2008 Apr 16;8:24 [FREE Full text] [CrossRef] [Medline]
    5. Carter BS, Carter A, Bennett S. Families' views upon experiencing change in the neonatal intensive care unit environment: from the 'baby barn' to the private room. J Perinatol 2008 Dec;28(12):827-829. [CrossRef] [Medline]
    6. Obeidat HM, Bond EA, Callister LC. The parental experience of having an infant in the newborn intensive care unit. J Perinat Educ 2009;18(3):23-29 [FREE Full text] [CrossRef] [Medline]
    7. Wigert H, Johansson R, Berg M, Hellström AL. Mothers' experiences of having their newborn child in a neonatal intensive care unit. Scand J Caring Sci 2006 Mar;20(1):35-41. [CrossRef] [Medline]
    8. De Rouck S, Leys M. Information needs of parents of children admitted to a neonatal intensive care unit: a review of the literature (1990-2008). Patient Educ Couns 2009 Aug;76(2):159-173. [CrossRef] [Medline]
    9. Sundstrom B. Mothers "Google It Up:" Extending Communication Channel Behavior in Diffusion of Innovations Theory. Health Commun 2016;31(1):91-101. [CrossRef] [Medline]
    10. Logsdon MC, Bennett G, Crutzen R, Martin L, Eckert D, Robertson A, et al. Preferred health resources and use of social media to obtain health and depression information by adolescent mothers. J Child Adolesc Psychiatr Nurs 2014 Nov;27(4):163-168. [CrossRef] [Medline]
    11. Logsdon MC, Mittelberg M, Myers J. Use of social media and internet to obtain health information by rural adolescent mothers. Appl Nurs Res 2015 Feb;28(1):55-56. [CrossRef] [Medline]
    12. Orr T, Campbell-Yeo M, Benoit B, Hewitt B, Stinson J, McGrath P. Smartphone and internet preferences of parents: information needs and desired involvement in infant care and pain management in the NICU. Adv Neonatal Care 2017 Apr;17(2):131-138. [CrossRef] [Medline]
    13. Gabbert TI, Metze B, Bührer C, Garten L. Use of social networking sites by parents of very low birth weight infants: experiences and the potential of a dedicated site. Eur J Pediatr 2013 Dec;172(12):1671-1677. [CrossRef] [Medline]
    14. Boulos MN, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform 2014;5(3):229 [FREE Full text] [CrossRef] [Medline]
    15. International Data Corporation. 2017. Smartphone OS market share, 2017 Q1   URL: https://www.idc.com/promo/smartphone-market-share/os [accessed 2018-06-06] [WebCite Cache]
    16. Cisco. 2016. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2017–2022 White Paper   URL: https://tinyurl.com/y2lwempd [accessed 2018-06-06] [WebCite Cache]
    17. Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med 2011 Mar;1(1):53-71 [FREE Full text] [CrossRef] [Medline]
    18. Waring ME, Moore ST, Xiao RS, Lombardini LM, Allison JJ, Rosal MC, et al. Pregnant women's interest in a website or mobile application for healthy gestational weight gain. Sex Reprod Healthc 2014 Dec;5(4):182-184. [CrossRef] [Medline]
    19. Goetz M, Müller M, Matthies LM, Hansen J, Doster A, Szabo A, et al. Perceptions of patient engagement applications during pregnancy: a qualitative assessment of the patient's perspective. JMIR Mhealth Uhealth 2017 May 26;5(5):e73 [FREE Full text] [CrossRef] [Medline]
    20. Lupton D, Pedersen S. An Australian survey of women's use of pregnancy and parenting apps. Women Birth 2016 Aug;29(4):368-375. [CrossRef] [Medline]
    21. Guerra-Reyes L, Christie VM, Prabhakar A, Harris AL, Siek KA. Postpartum health information seeking using mobile phones: experiences of low-income mothers. Matern Child Health J 2016 Dec;20(Suppl 1):13-21 [FREE Full text] [CrossRef] [Medline]
    22. Powell AC, Torous J, Chan S, Raynor GS, Shwarts E, Shanahan M, et al. Interrater reliability of mHealth app rating measures: analysis of top depression and smoking cessation apps. JMIR Mhealth Uhealth 2016;4(1):e15 [FREE Full text] [CrossRef] [Medline]
    23. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. J Am Med Assoc 2002;287(20):2691-2700. [CrossRef] [Medline]
    24. Dol J, Richardson B, Boates T, Campbell-Yeo M. Learning to parent from Google? Evaluation of available online health evidence for parents of preterm infants requiring neonatal intensive care. Health Informatics J 2018 Jan 1:1460458217752564. [CrossRef] [Medline]
    25. Davis DW, Logsdon MC, Vogt K, Rushton J, Myers J, Lauf A, et al. Parent education is changing: a review of smartphone apps. MCN Am J Matern Child Nurs 2017;42(5):248-256. [CrossRef] [Medline]
    26. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015 Jan 1;4:1 [FREE Full text] [CrossRef] [Medline]
    27. Richardson B, Dol J, Rutledge K, Monaghan J, Orovec A, Howie K, et al. PROSPERO International prospective register of systematic reviews. 2017 Jul 26. Evaluation of smart-device applications targeted to parents of infants in the neonatal intensive care unit: a systematic review   URL: http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017072548 [accessed 2019-03-06] [WebCite Cache]
    28. Tinschert P, Jakob R, Barata F, Kramer J, Kowatsch T. The potential of mobile apps for improving asthma self-management: a review of publicly available and well-adopted asthma apps. JMIR Mhealth Uhealth 2017 Aug 2;5(8):e113 [FREE Full text] [CrossRef] [Medline]
    29. Masterson CR, Maurer MS, Reading M, Hiraldo G, Hickey KT, Iribarren S. Review and analysis of existing mobile phone apps to support heart failure symptom monitoring and self-care management using the Mobile Application Rating Scale (MARS). JMIR Mhealth Uhealth 2016 Jun 14;4(2):e74. [CrossRef] [Medline]
    30. Mendiola MF, Kalnicki M, Lindenauer S. Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings. JMIR Mhealth Uhealth 2015 May 13;3(2):e40 [FREE Full text] [CrossRef] [Medline]
    31. Giunti G, Guisado FE, Dorronzoro ZE, Rivera RO. Supply and demand in mHealth apps for persons with multiple sclerosis: systematic search in app stores and scoping literature review. JMIR Mhealth Uhealth 2018 May 23;6(5):e10512 [FREE Full text] [CrossRef] [Medline]
    32. Mani M, Kavanagh DJ, Hides L, Stoyanov SR. Review and evaluation of mindfulness-based iPhone apps. JMIR Mhealth Uhealth 2015;3(3):e82 [FREE Full text] [CrossRef] [Medline]
    33. Owens OL, Beer JM, Reyes LI, Gallerani DG, Myhren-Bennett AR, McDonnell KK. Mindfulness-based symptom and stress management apps for adults with chronic lung disease: systematic search in app stores. JMIR Mhealth Uhealth 2018 May 15;6(5):e124 [FREE Full text] [CrossRef] [Medline]
    34. Furlan A. Business of Apps. 2017. The Ultimate App Store List   URL: http://www.businessofapps.com/guide/app-stores-list/ [accessed 2018-06-06] [WebCite Cache]
    35. Grundy QH, Wang Z, Bero LA. Challenges in assessing mobile health app quality: a systematic review of prevalent and innovative methods. Am J Prev Med 2016 Dec;51(6):1051-1059. [CrossRef] [Medline]
    36. Apple. Search Optimization - App Store - Apple Developer   URL: https://developer.apple.com/app-store/search/ [accessed 2018-07-09] [WebCite Cache]
    37. Zolotareva K. TheTool. 2018. Google Play Store ASO Guide for: How to Optimize Android Apps   URL: https://thetool.io/2017/how-to-optimize-google-play-aso-guide [accessed 2018-07-09] [WebCite Cache]
    38. Knight E, Stuckey MI, Prapavessis H, Petrella RJ. Public health guidelines for physical activity: is there an app for that? A review of android and apple app stores. JMIR Mhealth Uhealth 2015;3(2):e43 [FREE Full text] [CrossRef] [Medline]
    39. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    40. Shoemaker S, Wolf M, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns 2014 Sep;96(3):395-403 [FREE Full text] [CrossRef] [Medline]
    41. Trust It or Trash It?.   URL: http://www.trustortrash.org/TrustorTrash.pdf [accessed 2018-06-06] [WebCite Cache]
    42. Doron MW, Trenti-Paroli E, Linden DW. Supporting parents in the NICU: a new app from the US, ‘MyPreemie’. J Neonatal Nurs 2013 Dec;19(6):303-307. [CrossRef]
    43. Bourgeois FC, Mandl KD, Shaw D, Flemming D, Nigrin DJ. Mychildren's: integration of a personally controlled health record with a tethered patient portal for a pediatric and adolescent population. AMIA Annu Symp Proc 2009 Nov 14;2009:65-69 [FREE Full text] [Medline]
    44. Mäkelä H, Axelin A, Feeley N, Niela-Vilén H. Clinging to closeness: the parental view on developing a close bond with their infants in a NICU. Midwifery 2018 Jul;62:183-188. [CrossRef] [Medline]
    45. Franck LS, Oulton K, Bruce E. Parental involvement in neonatal pain management: an empirical and conceptual update. J Nurs Scholarsh 2012 Mar;44(1):45-54. [CrossRef] [Medline]
    46. Currie ER, Christian BJ, Hinds PS, Perna SJ, Robinson C, Day S, et al. Parent perspectives of neonatal intensive care at the end-of-life. J Pediatr Nurs 2016;31(5):478-489. [CrossRef] [Medline]
    47. Branchett K, Stretton J. Neonatal palliative and end of life care: what parents want from professionals. J Neonatal Nurs 2012 Apr;18(2):40-44. [CrossRef]
    48. De Rouck S, Leys M. Illness trajectory and Internet as a health information and communication channel used by parents of infants admitted to a neonatal intensive care unit. J Adv Nurs 2013 Jul;69(7):1489-1499. [CrossRef] [Medline]
    49. Ottosson C, Lantz B. Parental participation in neonatal care. J Neonatal Nurs 2017 Jun;23(3):112-118. [CrossRef]
    50. Norman D. Emotional Design. New York: Basic Books; 2004.
    51. Walter A. Designing for Emotion. New York: A Book Apart; 2011.


    Abbreviations

    IFDC: Integrated Family Delivered Care
    MARS: Mobile Application Rating Scale
    mHealth: mobile health
    NICU: neonatal intensive care unit
    PEMAT-AV: Patient Education Materials Assessment Tool for Audiovisual Materials
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses


    Edited by G Eysenbach; submitted 31.07.18; peer-reviewed by A Powell, E Dorronzoro, M Stuckey; comments to author 21.09.18; revised version received 10.10.18; accepted 29.10.18; published 15.04.19

    ©Brianna Richardson, Justine Dol, Kallen Rutledge, Joelle Monaghan, Adele Orovec, Katie Howie, Talia Boates, Michael Smit, Marsha Campbell-Yeo. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 15.04.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.