This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Pediatric cardiac arrest (PCA), although rare, is associated with high mortality. Deviations from international management guidelines are frequent and associated with poorer outcomes. Different strategies/devices have been developed to improve the management of cardiac arrest, including cognitive aids. However, there is very limited experience on the usefulness of interactive cognitive aids in the format of an app in PCA. No app has so far been tested for its usability and effectiveness in guiding the management of PCA.
To develop a new audiovisual interactive app for tablets, named PediAppRREST, to support the management of PCA and to test its usability in a high-fidelity simulation-based setting.
A research team at the University of Padova (Italy) and human–machine interface designers, as well as app developers, from an Italian company (RE:Lab S.r.l.) developed the app between March and October 2019, by applying an iterative design approach (ie, design–prototyping–evaluation iterative loops). In October–November 2019, a single-center nonrandomized controlled simulation–based pilot study was conducted including 48 pediatric residents divided into teams of 3. The same nonshockable PCA scenario was managed by 11 teams with and 5 without the app. The app user’s experience and interaction patterns were documented through video recording of scenarios, debriefing sessions, and questionnaires. App usability was evaluated with the User Experience Questionnaire (UEQ) (scores range from –3 to +3 for each scale) and open-ended questions, whereas participants’ workload was measured using the NASA Raw-Task Load Index (NASA RTLX).
Users’ difficulties in interacting with the app during the simulations were identified using a structured framework. The app usability, in terms of mean UEQ scores, was as follows: attractiveness 1.71 (SD 1.43), perspicuity 1.75 (SD 0.88), efficiency 1.93 (SD 0.93), dependability 1.57 (SD 1.10), stimulation 1.60 (SD 1.33), and novelty 2.21 (SD 0.74). Team leaders’ perceived workload was comparable (
The PediAppRREST app received a good usability evaluation and did not appear to increase team leaders’ workload. Based on the feedback collected from the participants and the preliminary results of the evaluation of its effects on the management of the simulated scenario, the app has been further refined. The effectiveness of the new version of the app in reducing deviations from guidelines recommendations in the management of PCA and its impact on time to critical actions will be evaluated in an upcoming multicenter simulation-based randomized controlled trial.
Pediatric cardiac arrest (PCA), although rare, is an important public health issue due to its high mortality and morbidity, its complex time-dependent management and emotional burden, its social and economic costs, and differences with adult cardiac arrest (CA) [
International guidelines by relevant societies are periodically updated to help health professionals provide the best evidence-based basic and advanced care to improve the management and outcome of PCA [
Different cognitive support tools have been developed and tested in order to improve adherence to guideline-recommended management of both adult CA and PCA, with variable results. Most of these tools are devices that provide real-time audiovisual feedback on the quality of chest compressions. Such tools have shown to be effective in improving the quality of compressions [
Research on apps developed and tested to guide the management of in-hospital PCA is very limited [
Based on the deviations from guidelines recorded on a prior study conducted by our research team in PCA simulation scenarios [
The primary aim of this pilot study was to refine the app and to test its usability and impact on team leader’s workload using high-fidelity simulation. As a secondary aim, we explored the trend in the occurrence of deviations from guidelines.
We designed and developed an app for tablet that we named PediAppRREST, which is the result of the collaboration between a pediatric research team, including physicians and researchers from the Pediatric Emergency Department and the Pediatric Intensive Care Unit of the University Hospital of Padova (Italy), and human–machine interface designers, human factor experts, and app/software developers of RE:Lab S.r.l., an Interaction Engineering company (Reggio Emilia, Italy).
The app was designed to guide the team leader to perform resuscitation interventions in the sequence/timing and modality reported by the American Heart Association (AHA) PALS 2015 guidelines [
We developed the app between March and October 2019. As a first step, the research team defined the actions to be displayed in separate screens, the flow/pathways, and the additional features that were deemed helpful to guide resuscitation and achieve a high-quality cardiopulmonary resuscitation (CPR), based on recommended PALS guidelines/algorithms. We then progressively refined and validated the cognitive aid following an iterative prototyping development approach [
Directions on recommended interventions, following the order reported in the PALS algorithms, are sequentially displayed in the app which has been designed as a
The main criteria applied in the user experience design phase of the app have been (1) timely information (each screen gathers only the necessary information for each phase of the PALS algorithm, communicating it both visually and acoustically, with the aim of reducing the load on the team leader’s working memory and relying on a multichannel communication); (2) priority (actions [ie, epinephrine administration] triggered by timers have priority on other actions displayed on the screen); and (3) sequential versus alternative choices (decisions that team leaders must take into consideration concern actions and choices to be performed sequentially or alternatively). Hence, sequential actions are displayed with rectangular buttons, aligned vertically on the page, whereas alternative choices are organized with square buttons, aligned horizontally on the screen (
Each screen is structured into 3 zones (
Sequential versus alternative choices. CPR: cardiopulmonary resuscitation; ROSC: return of spontaneous circulation.
User interface main areas. CPR: cardiopulmonary resuscitation; ROSC: return of spontaneous circulation; VF: ventricular fibrillation; VT: ventricular tachycardia.
The main area presents the actions to be performed (with buttons of the same shape and color) or a question with different choices (buttons with different colors). Once the user taps on a button, the flow of prompts will progress following the user’s choices. On the top bar a menu-log button, a 2-minute countdown clock for repeat rhythm check button, a button with countdown for medications, a metronome button, and a total counter are displayed. The metronome button can be activated by a tap: this is a sound guide to perform compressions at the recommended rate (100-120/minute). On the bottom bar, CPR and Return of Spontaneous Circulation (ROSC) buttons are available at any time: the CPR button opens up a recap of the characteristics of a high-quality CPR, whereas the “ROSC” button summarizes the recommended management when ROSC is achieved (
Cardiopulmonary resuscitation (CPR) and return of spontaneous circulation (ROSC) information. EKG: electrocardiogram; EtCO2: end-tidal CO2; FiO2: fraction of inspired oxygen; SaO2: oxygen saturation measured with pulse oximetry.
The flow of actions that pops up in the app main area follows 2 different pathways based on the identified cardiac rhythm on the monitor (shockable versus nonshockable rhythms), as per PALS algorithms.
The app provides assistance with shock delivery, in case of a shockable rhythm, and the preparation/administration of medications, prompting the correct doses (automatically calculated on patients’ weight) and time intervals of administration (
Epinephrine administration screens. CPR: cardiopulmonary resuscitation; ROSC: return of spontaneous circulation.
Every 2 minutes the app acoustically and visually reminds the user to check the rhythm and, in case of a shockable rhythm, to deliver a shock.
Finally, the app prompts to search/treat reversible causes of CA and to correctly manage the airway (
The app also gives audio prompts, suggesting the user to perform the actions shown on the main area of the screen. The user can navigate the app only using touch gestures. Voice interaction has not been integrated due to the characteristics of the resuscitation environment, which would impede accurate recognition of vocal commands.
All actions done by the user are sequentially saved on the device in the
Reversible causes and airway management screens. CPR: cardiopulmonary resuscitation; ROSC: return of spontaneous circulation.
We conducted a single-center simulation-based pilot nonrandomized controlled study in October–November 2019 at the University Hospital of Padova, Italy. Although our study is not a randomized controlled trial (RCT), we followed the guidelines for reporting simulation-based studies as far as applicable [
Further details regarding the study methodology and procedures are described in
Written informed consent for participation was obtained from all the participants. The study was approved by the Hospital Ethics Committee as an educational project.
The primary outcomes of our study were the usability of the app and the team leader’s workload. They were measured by 2 validated questionnaires, the User Experience Questionnaire (UEQ) [
Secondary outcomes were qualitative feedback on the app provided by participants, preliminary data on deviations in management from PALS guidelines recommendations, time to epinephrine administration, and resuscitation performance of the teams evaluated with the validated Clinical Performance Tool (CPT) [
The UEQ is a validated questionnaire which comprises 26 items. Each item is represented by 2 terms with opposite meanings that the user evaluates on a 7-point Likert-type scale (from –3 to +3). The 26 items are grouped into 6 scales that cover both classical usability aspects (efficiency, perspicuity, dependability) and user experience aspects (attractiveness, stimulation, novelty). The range of each scale is also between –3 and +3. The standard interpretation of the scale is that values between –0.8 and 0.8 represent a neutral evaluation of the corresponding scale, values over 0.8 represent a positive evaluation, and values less than –0.8 a negative evaluation [
The NASA RTLX is a simplified version of the NASA-Task Load Index which is a subjective multidimensional tool designed to assess workload. Six subscales represent different domains of the perceived workload: mental demand, physical demand, temporal demand, frustration, effort, and performance. Each domain is clearly defined and rated by participants through a 0 to 100 scale with 5-point steps. The ratings of the 6 subscales are simply averaged to create an estimate of overall workload, defined as low (<40), moderate (between 40 and 60), and high (>60) [
The qualitative feedback on user app interaction was collected through open-ended questions in the postscenario questionnaire (“What are the main difficulties you have encountered in the use of the app and/or tablet?”, “Do you have any suggestions to improve the app or its use?”) and through the postscenario debriefing. Feedback from participants was categorized by common themes.
Deviations from PALS guidelines recommendations were defined as delays and errors according to a novel checklist adapted to our intervention and scenario. We derived this new measure from the checklist, denominated c-DEV, published by Wolfe et al [
The CPT is a performance assessment tool and a validated scoring system designed based on PALS algorithms comprising different tasks. Each task is scored as follows: not performed (0 points), performed partially, incorrectly, or late (1 point); and performed completely, correctly, and timely (2 points). Thus, the tool assesses sequence, timing, and quality of specific actions during different simulated scenarios [
The characteristics of the study participants, stratified by group allocation (control vs intervention), and the outcome variables, were summarized using descriptive statistics, and compared between the 2 groups using Mann–Whitney
During the study period, 63 pediatric residents were assessed for eligibility, of whom 48 (16 for each one of the 3 years of residency program involved in the study) were included in the study and divided into teams of 3. Five teams managed the case following usual care (control group), whereas 11 teams (intervention group) conducted the scenario using the support of the PediAppRREST app (
Participants’ demographic characteristics, as well as training and clinical experience of resuscitation, were comparable between the 2 groups (
The PediAppRREST app attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty were on average evaluated positively; mean UEQ scale scores were substantially above the 0.8 cutoff. UEQ scales internal consistency varied from poor (perspicuity) to acceptable (efficiency and dependability), good (novelty), and excellent (attractiveness and stimulation); see
PediAppRREST app usability.
UEQa scale | Mean (SD) | 95% CI | Cronbach α coefficient |
Attractiveness | 1.712 (1.434) | 0.865-2.559 | .98 |
Perspicuity | 1.750 (0.880) | 1.230-2.270 | .55 |
Efficiency | 1.932 (0.929) | 1.383-2.481 | .72 |
Dependability | 1.568 (1.102) | 0.917-2.219 | .76 |
Stimulation | 1.598 (1.333) | 0.811-2.386 | .91 |
Novelty | 2.205 (0.740) | 1.767-2.642 | .86 |
aUEQ: User Experience Questionnaire.
Team leaders’ perceived workload was comparable between the 2 groups; median NASA RTLX score was 67.5 (IQR 65.0-81.7) for the control group and 66.7 (IQR 54.2-76.7) for the intervention group (
Based on the qualitative feedback provided by participants, the most frequently highlighted difficulties were (1) interacting with the screens flow because information delivery was unclear about recommendation on performance of an action versus suggestion to perform an action based on the team skillset, that is, advanced airway management (n=8 team leaders); (2) information overload in the reversible causes screen, which was perceived as too dense (n=5); (3) understanding whether the selection of an icon for a recommended action had to occur at the beginning of the action or after the action was completed (ie, users did not understand whether to select the epinephrine icon at the time of preparation or administration; n=5); and (4) interacting with the app while leading the teamwork (n=4). A less frequently reported difficulty was the lack of a traditional PALS algorithm embedded within the app (n=2). Lastly, it also emerged that longer training and familiarization with the app before the simulated scenario would have been beneficial to interact more efficiently and effectively with the app.
With respect to deviations from the guidelines, the frequency of (1) incorrect compressions-to-ventilations ratio during CPR, (2) prescription of incorrect doses/dilutions of epinephrine, and (3) lack of search/treatment of reversible causes of CA (ie, hypovolemia) were higher in the control group in comparison to the intervention group; however, these differences did not reach statistical significance (
Resuscitation performance of the teams.
Performance | Control group (N=5) | Intervention group (N=11) | |
Incorrect compressions-to-ventilation ratio, n (%) | 1 (20) | 0 (0) | .31 |
Incorrect dose or dilution of epinephrine, n (%) | 1 (20) | 0 (0) | .31 |
Lack of search and treatment of reversible causes of cardiac arrest, n (%) | 2 (40) | 2 (18) | .55 |
c-DEVplus score, median (IQR) | 6 (6-7) | 6 (4-7) | .27 |
Time (seconds) to first epinephrine administration, median (IQR) | 165 (139-173) | 254 (204-290) | .015 |
CPTa scores, median (IQR) | 9 (8-10) | 9 (9-10) | .77 |
aCPT: Clinical Performance Tool.
We developed a novel cognitive aid, an app for tablet, which aims to optimize the management of PCA by facilitating increased adherence to guideline recommendations. In our pilot study, the app showed a good usability profile and its use was not associated with increased team leaders’ workload. These findings are encouraging and in contrast to data on previously developed cognitive support tools which are shown to increase users’ workload [
Our preliminary results, although based on a very limited sample size, highlighted the potential benefits, as well as the drawbacks, of using the app to guide resuscitation. Nevertheless, we are confident that the refined version of the app based on the feedback received in this pilot study and a better familiarization with it prior to its use have the potential to significantly reduce deviations from guidelines, which correlate with clinical outcomes [
We also measured team performance by means of the validated CPT score [
Epinephrine preparation and administration screens. CPR: cardiopulmonary resuscitation; ROSC: return of spontaneous circulation.
The results of our pilot study are preliminary and are the first step of a larger project that aims at testing the effectiveness of the PediAppRREST app. As such, they cannot be considered definitive, as the sample size is very limited. However, this pilot experience was essential to refine the newly developed tool and to verify there were the premises for a larger comparative study.
Participants of our study were exclusively pediatric residents. Although this may limit the generalizability of study findings to other clinicians, trainees are in a unique process of learning and are more used to incorporate digital assistive tools in their clinical practice. While in our setting it is easier to get trainees involved in simulation projects, than experienced clinicians, the app could be seen as a training tool in itself and future studies will be designed to test its effectiveness in knowledge and skill retention.
Our resuscitation team composition differs from other settings, where a co-team leader, a respiratory therapist, or a CPR coach is often part of the team, which includes a higher number of members. We chose a team composition that reflects the actual management at our institution during night shifts, where trainees are in the front line in the management of the first few minutes of pediatric emergencies. We felt it was important to test the device in this highly stressful and staff-limited scenario where the app could be most useful.
The intervention and control groups were unbalanced with respect to the intervention group as we needed to test the device in the first place, and a few control teams were necessary to test the overall procedures for the RCT. Similarly, although we randomly allocated residents to each team and randomly selected the teams who were performing the scenario without the app, the timing of app availability was the main determinant of our pilot study procedure and a proper randomization process will be performed for the future RCT.
A single trained reviewer rated the videos as the preliminary evaluation of team performance and deviations from guidelines were a secondary aim of this study. Two independent and trained reviewers will be ensured for the RCT, and interrater reliability will be reported and monitored. Blinding of participants and research staff was not possible because of the nature of simulation-based study. Blinding of video reviewer was not applied as video recording of the team leader using the app and the tablet was necessary to detect possible difficulties with its use. Blinding of the statistician performing data analysis will be ensured for the RCT.
The high-fidelity simulation setting during the last decades has established itself as a way to investigate rare but high-risk medical conditions. Although it does not provide data on actual patient outcomes, it is the best available way to reproduce and study rare high-stake emergencies and test novel devices developed to improve their management without compromising patients’ safety.
Several researchers have tried to create and test software products and apps to improve the quality of resuscitation. Different products have been conceived, mostly dealing with OHCA and in-hospital CA (IHCA) in adults. For instance, to help lay rescuers to manage adult cases of OHCA, different tools have been developed, such as the M-AID (an app for mobile phones [
With regard to PCA, a mobile app was developed to help adolescent lay bystanders to manage an infant OHCA scenario, but an RCT showed that the participants who used the app only partially improved their performance [
An app to help nurses prepare and administer drugs for infusion during in-hospital pediatric resuscitation has been recently developed and tested in a simulation-based RCT. The app was effective in reducing errors and time to preparation/delivery of medications compared with conventional methods [
To our knowledge, no app similar to the PediAppRREST has been tested in a pilot study to be refined, and to evaluate its usability and related workload before being tested in an adequately powered RCT.
We developed and refined a novel interactive tablet app (PediAppRREST) for the management of PCA that has potential to reduce deviations from guidelines recommendations. The app showed a good usability profile and was not associated with higher team leaders’ workload. After this pilot testing its effectiveness will be evaluated in an adequately powered simulation-based RCT.
Pilot study methodology.
c-DEVplus score calculation grid.
Flowchart of participant recruitment and study group allocation.
Characteristics of participants: demographics, training and clinical experience on resuscitation.
American Heart Association
cardiac arrest
Clinical Performance Tool
cardiopulmonary resuscitation
in-hospital cardiac arrest
National Aeronautics and Space Administration
out-of-hospital cardiac arrest
Pediatric Advanced Life Support
pediatric cardiac arrest
randomized controlled trial
Return of Spontaneous Circulation
Raw Task Load Index
Usability Experience Questionnaire
The authors thank Professor Anna Chiara Frigo for her contribution in randomly assigning residents in teams of 3, the Department of Informatics and Telecommunications of University of Padova and its Director Dr Andrea Baraldo for supporting the project, and all pediatric residents who took part into this study. This work was funded and supported by the Department of Women’s and Children’s Health and the University of Padova, Italy.
FC and SB conceived the development of the app and the pilot testing study. ML, LS, and FT were responsible for designing and developing the app prototypes. FC, SB, VS, FM, MD, DS, and LDD tested the app prototypes, organized, and conducted the simulation sessions. FC reviewed the videotapes of the simulations and collected data. MA performed the statistical analysis. FC, SB, and MA drafted the manuscript. SB supervised the different stages of the study. All authors revised and approved the final version of the manuscript.
The authors affiliated with RE:Lab had no direct involvement in the project, other than the technical activities related to the development of the app. All the other authors have no conflict of interest to declare neither with the RE:Lab company nor with other companies.