Acceptability of a Mobile Clinical Decision Tool Among Emergency Department Clinicians: Development and Evaluation of The Ottawa Rules App

Introduction

Clinical decision rules attempt to reduce the uncertainty of medical decision making by standardizing the collection and interpretation of clinical data. These tools are derived from original research and incorporate 3 or more variables from clinical assessment or simple tests. These offer concrete yes/no answers and help clinicians with management decisions at the bedside [1-3].

The Ottawa Knee Rule [4], Ottawa Ankle Rules [5], and Canadian C-Spine Rule [6] are 3 internationally validated clinical decision rules that were developed to facilitate rapid detection of bone fractures upon entry into the emergency department (ED) and reduce unnecessary radiographic series. In various clinical trials, clinicians who applied the rules had a 28% reduction in ankle and 14% reduction in foot radiographic series [7]. In addition to reducing unnecessary use of diagnostic imaging services, appropriate application of the rules has been shown to improve standardization of practice and care and reduce emergency room wait times with significant health cost savings [8-10]. Although clinical decision rules such as The Ottawa Rules were developed to assist with bedside diagnostic or therapeutic decisions, some may have limited impact due to weak clinician uptake in jurisdictions [11,12].

Use of mobile technologies in the health care setting has provided clinicians with a means of rapidly and easily accessing hospital information systems and services. Mobile apps now assist clinicians with day-to-day tasks including health record maintenance, patient management and monitoring, and medical education and training. There exists an opportunity to leverage increasing use of mobile devices to support easy and efficient access to clinical decision rules.

In this study, we sought to develop and evaluate a mobile app housing 3 validated ED clinical decision rules, collectively known as The Ottawa Rules. Our primary objective was to determine acceptability of a mobile app format of The Ottawa Rules among ED clinicians (physicians, residents, nurses, and medical students) at The Ottawa Hospital (TOH), in Ottawa, Canada. The secondary objective was to evaluate the effect of publicity efforts on uptake of The Ottawa Rules app.

Methods

Mobile App Development

The Ottawa Rules app was developed natively for both iOS and Android operating systems and as a mobile-enhanced website at www.theottawarules.ca. App development was led by The Ottawa Hospital mHealth Lab located at the Ottawa Hospital Research Institute (OHRI). The Ottawa Knee Rule, Ottawa Ankle Rules, and Canadian C-Spine Rule, together known as The Ottawa Rules, were included in the app. Each rule was made available as a set of images with clear procedural guidelines. Instructional videos and links to academic resources for each rule were also included. The app was designed to include a mechanism for feedback and support where users were permitted to provide suggestions for app improvements, report bugs, and request technical assistance.

Pilot-Testing

Following internal alpha-testing, the prototype was beta-tested on 6 consenting ED clinicians at TOH. Beta testers were instructed to use the app as if they were experiencing the circumstances outlined in provided mock clinical scenarios. Scenarios involved a patient entering the ED with a suspected knee, foot, or neck injury for which the clinicians needed to use the app to determine whether the injury would require diagnostic imaging. Beta testers completed their mock scenario twice; once as new users and once after having familiarized themselves with the app for approximately 10 minutes. Semistructured interviews with beta testers were then conducted to establish further insight into what content would be appropriate for the app and any barriers to app access or perceived barriers for app use. Findings from beta-testing were then integrated into the app before its public release.

App Release

The app was publicly released on April 21, 2016, for iOS devices via the App Store, for Android devices via Google Play, through OHRI’s app portal, and on the Web at www.theottawarules.ca. A press release was circulated on the same day with local and national news sources, national and international emergency medicine media, and through social media messaging on affiliate Twitter accounts. Social media posts about app release were also disseminated via affiliate Twitter accounts and Facebook pages, and announcements were made at departmental rounds.

Study Enrollment

The study was advertised through poster displays in EDs at TOH Civic and General campuses and email distribution to ED staff and medical students at the University of Ottawa. Study staff also showcased the new app at a hospital-organized digital health networking event, resident rounds, and the hospital’s annual academic research day. Primary study recruitment was conducted in person by study staff. Participants were required to be over 18 years of age, work in or be on rotation in the ED, possess an institutional email (TOH, OHRI, or University of Ottawa), and own a personal or institutional iOS or Android mobile phone onto which they could download The Ottawa Rules app. Since 2011, TOH has been equipping clinicians with iPads—clinicians also had the option of downloading The Ottawa Rules app onto these institutional devices and enrolling in the study.

Inclusion criteria were assessed as part of the in-app informed consent process. Figure 1 provides screen shots of The Ottawa Rules app home screen with the “TOH Study” button for participants, informed consent screen, and 1 of the 3 in-app guidelines. From the app’s home screen interface, users could select 1 of the 3 rules according to the patient’s suspected injury. From the main menu, there were options to learn about The Ottawa Rules app, link to the lead physician’s Twitter account, provide feedback, and read the terms of use. With the goal of enhancing app usefulness, study participants had additional access to hospital resources within the app; these features were not available to general app users. Hospital resources included TOH’s antibiotic guidelines, nursing medical directives, and triage algorithms.

Once enrolled, consenting participants had to verify their institutional email before proceeding. Participants had in-app access to the consent documents and contact information of study staff throughout the duration of the study. Participants were instructed to explore the app features and use the app as reasonable when evaluating patients with suspected knee, foot, or neck injuries. This study was approved by the Ottawa Health Research Network Science Research Ethics Board (#20150405-01H).

Data Collection

Frequency of use of The Ottawa Rules app was measured via in-app analytics and user surveys as detailed below.

Participant Survey

One-month post-study enrollment, participants were prompted via their verified institutional email to complete a usability survey to assess their perceived acceptability and usability of the app. The survey consisted of multiple-choice, 5-point Likert scale, and open-ended questions designed to ascertain participant demographics, ease of use, and intention for future use and provide the opportunity for written feedback. Participants who completed the end-study survey received an electronic coffee gift card in the amount of Can $10 (US $8).

Technology Readiness Index

The Technology Readiness Index (TRI) 2.0 was also administered in the 1-month survey. The TRI 2.0 includes 16 questions that measure an individual’s innate “propensity to embrace and use new technologies for accomplishing goals in home, life, and at work.” TRI questions are measured on a 5-point Likert scale ranging from 1=strongly disagree to 5=strongly agree and capture 4 dimensions. Two motivating dimensions capture qualities of individual optimism and innovativeness, whereas questions pertaining to individual discomfort and insecurity are considered inhibiting dimensions. A higher individual TRI indicates a higher likelihood to adopt new technology: TRI 2.0 = (innovativeness + optimism + [6 – discomfort] + [6 – insecurity]) / 4). Questions related to each of the 4 TRI 2.0 dimensions are provided in Multimedia Appendix 1.

Participant In-App Activity

Analytics on participant use of the app were encrypted and sent to a secure cloud server in Canada administrated by The Ottawa Hospital mHealth Lab at OHRI. The following metrics were collected on an individual level, only identified by study ID on the server: date app was first opened, number of times each rule was accessed, number of consenting participants who did not reopen the app, frequency of rule use, number of app sessions, and number and content of submitted feedback reports.

Google Analytic Data

General anonymous usage data of The Ottawa Rules app were collected through Google Analytics and used to gauge global app uptake and success of promotional activities. Google Analytics was used to ascertain total downloads, geographical region by IP address, app screens accessed, and average time spent in the app.

‎View this figure

Results

Study Participants

A total of 155 consenting participants provided electronic verification of their enrollment into the study, and 119 (76.8%) participants submitted usability surveys 1 month after study enrollment. Due to data quality concerns, participants who enrolled under multiple email addresses and those who did not complete the usability survey in full were excluded from analysis. The final study cohort consisted of 108 participants (Figure 2).

Nurses constituted the largest number of study participants (42/108, 38.9%), followed by residents (33/108, 30.6%), physicians (20/108, 18.5%), and medical students (13/108, 12.0%). The majority of participants were 34 years of age or younger (73/108, 68.2%) and owned iOS devices (87/108, 80.6%). Table 1 summarizes the baseline characteristics of study participants.

Usability Survey

When asked if they encountered any issues using the app, all participants reported either strongly disagree (103/108, 95.4%) or disagree (5/108, 4.6%). Self-reported frequency of use revealed the majority of participants (92/108, 85.2%) used the app at least once during the course of the study. A total of 43.5% (43/108) of participants said they used the app weekly, 38.9% (42/108) said monthly, 14.8% (16/108) said never, and 2.8% (3/108) said they used the app daily (Figure 3).

Questions on app usability revealed favorable reception by ED clinicians. A total of 30.6% (33/108) of participants indicated strongly agree or agree when asked if they used the app for the majority of the cases that required application of the clinical decision rules, 67.6% (73/108) of participants found the app useful in applying the clinical rules (strongly agree, agree), 75.0% (81/108) indicated that they would recommend the app to colleagues, and 67.6% (73/108) would continue using the app in its current form. Figure 3 summarizes usability survey data.

In addition, 56.5% (61/108) of participants provided free-form written feedback. Feedback included recommendations based on functionality, acceptability, and available features. Participants reported a desire for access to more TOH-developed decision aids, particularly the addition of the Canadian CT Head Rule and Subarachnoid Hemorrhage Risk Score. Users also indicated wanting access to more institutional resources and directives. A number of participants suggested inclusion of more interactive features, including a drop-down menu for easier and quicker navigation. Some indicated their preference for other medical mobile apps that housed more comprehensive or interactive lists of medical directives and decision rules. Lastly, participants highlighted the potential benefits of the use of the app among early learners or community physicians who might be less familiar with The Ottawa Rules.

‎View this figure

Technology Readiness

After the exclusion of 6 incomplete surveys, the median TRI score was 3.56 (interquartile range [IQR] 0.62) out of a maximum score of 5, suggesting a slightly higher than average (3.02) propensity for technological adoption among participants. Examination of TRI scores across ED staff role (nurse, physician, resident, medical student), age (<35 or ≥35 years), by self-reported frequency of app use, and by response to usability questions revealed no association between TRI scores and patient demographics or user satisfaction and use of The Ottawa Rules app. The distribution of participant responses to survey questions ascertaining propensity to adopt new technologies overall and by participant subgroup are provided in Multimedia Appendix 1.

In-App Activity

Study participants accessed the app a total of 762 times between April 21 and August 30, 2016 (Table 2). Server data showed that of the 108 participants, 13 (12.0%) participants did not venture beyond the app home screen to use specific app features, confirming patterns of self-reported use. Nurses were the most active users of the app features (responsible for 37.8% [288/762] of activity), followed closely by ED residents (32.5% [248/762] of activity). Of the 3 Ottawa Rules hosted in the app, the Canadian C-Spine Rule was the most frequently accessed overall (20.1% [153/762] of app events), followed by the Ottawa Ankle Rules (134/762, 17.6%), and Ottawa Knee Rule (128/762, 16.8%). TOH guidelines and algorithms accounted for 45.5% (347/762) of app events overall and were among the most accessed tools across all participant subgroups.

‎View this figure

A summary of global app use and in-app activity since public release is shown in Table 3. As of June 1, 2017, The Ottawa Rules app had been downloaded nearly 4000 times across 89 countries. Users in Canada accounted for the largest number of downloads (1234/3863, 31.9%), followed by users in the United States (953/3863, 24.7%) and Spain (347/3863, 8.9%). The Ottawa Ankle Rules were accessed 2066 times, followed by the Ottawa Knee Rule (1991 times) and Canadian C-Spine Rule (1904 times).

Uptake of the app across 89 countries worldwide may reflect success of promotional efforts of our team for The Ottawa Rules app. Such efforts included media releases [13], a local morning televised news broadcast, and dissemination of the work through academic and clinical channels. We observed an increase in app downloads following circulation of a press release on May 9, resulting in 202 daily downloads. The period between September 23-30, 2016, saw 171 new downloads in Spain. Other potential reasons for global success may be app usefulness spread through word of mouth between clinicians and interactions or presentations at national and international conferences.

Discussion

Principal Findings

Mobile apps can be leveraged to improve the accessibility of clinical decision rules in the ED. Pilot-testing of The Ottawa Rules app among ED workers produced useful feedback that can be used to optimize the platform for our users. Overall, survey data suggest the app was useful in guiding clinical decision making and is a tool that clinicians would use in the future and recommend to others. A preference for single-source access to clinical resources and inclusion of additional decision rules and center-specific directives were among the most frequently cited feedback responses. Users also indicated that the platform could be improved by inclusion of interactive features.

The Ottawa Ankle Rules, Ottawa Knee Rule, and Canadian C-Spine Rule were developed to reduce unnecessary radiography for ankle/foot, knee, and cervical spine injuries without jeopardizing patient care. The rules have been widely validated across numerous international patient and hospital settings [14-17] and have been shown to reduce department crowding and patient length of stay [18] and save on hospital resources [19]. Dissemination, uptake, and implementation of the rules have not been optimal, however, even with educational strategies [12,20].

Health care has been impacted by advances in mobile technology. Whereas previously pagers and personal digital assistants were commonplace, mobile phones and tablets are now the preferred computing devices for health care professionals [21]. Ownership and use of smart devices is particularly high among the health care community, who report using medical apps on personal devices for both clinical activities and continued education [22,23]. Health care professionals increasingly rely on electronic resources to support patient management decisions, and there are numerous apps available that help provide information on diagnosis, treatment, and standard clinical formulas [24]. Other mobile apps, similar to the one evaluated in this study, have been designed to help clinicians identify the most appropriate scans or tests to order, thereby increasing efficiency of use of hospital resources [24]. Medical apps, although convenient, must be viewed with caution, however. Despite the high rate of adoption of health apps among young health care providers, medical professional involvement in the development of such apps is notably limited [23]. As a result there is a paucity of literature for reference on the rate of adoption, satisfaction, and use scores for apps used by health care providers. With the number of such tools becoming publicly available, the source of information upon which medical apps and electronic resources have been developed must be considered. Standards for the evaluation and validation of medical apps are warranted to ensure that the recommendations and outputs from these tools are reliable and safe [25].

Strengths and Limitations

This study has important strengths and weaknesses. Strengths include close involvement of clinicians in the development of The Ottawa Rules app, user testing among all levels of ED staff, the use of surveys to capture both closed and open-ended feedback on the app system, and a high response rate (119/155, 76.8%). The successful completion of this study with over 100 ED workers of varied educational training, duration of employment, and age demonstrated an openness of the clinical community to trying new technology in the workplace. Our ability to leverage 3 separate data collection mechanisms to capture user-level (study participants) and aggregate-level (all app users) app activity is also a notable strength. Self-reported use of The Ottawa Rules app by participants, validated by server analytics data, confirmed that 12% to 15% of participants did not actively use the app despite having it downloaded on their phones. Through server analytics, we were able to identify the most frequently accessed app features and resources. Google Analytics data further permitted review of in-app activity among all users of the app, beyond the study cohort. Through Google Analytics, patterns of download across geographies and by type of devices were available for review, and trends in app activity following promotional efforts could be assessed. The primary weaknesses of this study were the small sample size, self-selecting nature of participant recruitment, and our single-center approach. This study also took place at a tertiary care hospital in an urban center where there was a high degree of familiarity with The Ottawa Rules among ED workers. As highlighted by some participants, evaluation of receptiveness to the app among medical trainees, general practitioners, and community physicians may be warranted, as these groups are less likely to be familiar with the rules.

Conclusion

As we seek to optimize The Ottawa Rules app based on feedback to improve the user experience, interactive modalities will be provided. Importantly, inclusion of new clinical decision rules and other resources will be incorporated to provide a more comprehensive tool to our users. Work in this field would benefit from ongoing clinician involvement in the development and evaluation of health apps to ensure app quality, reliability, and user satisfaction. Future work should aim to assess the impact of health apps such as The Ottawa Rules on the quality and cost effectiveness of patient care and hospital resources. In sum, The Ottawa Rules app stands to provide health care professionals an efficient, reliable, and user-friendly means of accessing clinically validated decision rules shown to reduce health care costs and improve quality of patient care.