Background: Adverse drug reactions (ADRs) cause significant morbidity and mortality. Improved assessment of ADRs to identify the causal relationship, the severity, and the preventability will aid ADRs prevention or reduce patient burden.
Objective: The aim of this study was to develop mobile apps in assisting clinical decision in ADR assessments of causality, severity, and preventability using validated tools. The usability of the apps was assessed.
Methods: We designed mobile apps using validated assessment tools for ADRs. They are the Liverpool ADRs Causality Assessment Tool, Hartwig’s Severity Assessment Scale, and the Modified Schumock and Thronton Preventability Scale. The apps were named “Adverse Drug ReactionCausality,” “Adverse Drug ReactionSeverity,” and “Adverse Drug RxnPreventability.” A survey was conducted using the System Usability Scale (SUS) to assess the usability of the developed apps among health care professionals.
Results: These apps are available for download through Google Play Store for free since January 2015. From the survey, the mean SUS score was 70.9 based on 26 responses from the pediatric ward of Hospital Ampang, Malaysia.
Conclusions: The developed apps received an overall acceptable usability among health care professionals. The usage of these apps will improve detection, assessment, and avoidance of future ADRs. They will also contribute to future research on ADRs, thus increasing drug safety.
Adverse drug reactions (ADRs) cause significant mortality and morbidity in patients [- ]. The World Health Organization (WHO) defines an ADR as a response to a drug that is noxious and unintended and occurs at doses normally used in man for prophylaxis, diagnosis, or therapy of disease, or for modification of physiological function [ ]. Previous studies have shown that ADRs were the cause of 3% of all hospital admissions in the pediatric population and that 10% of children suffer an ADR while in hospital [ ]. Similar numbers are seen in adult patients [ ]. ADRs have been estimated to cause 3% of all deaths in the general population and up to 5% of deaths in hospitalized patients [ ].
Previous studies have shown that the combined use of mobile technology and mobile apps software in health care offer various benefits to many parties including health care professionals, patients, management, and even stakeholders [- ]. Mobile technology facilitates efficient delivery of services to patients. It also improves quality and effectiveness of services to the benefit of patients [ , ]. However, the use of apps in health care services does not seem as extensive as compared with other services such as social or public services [ , ]. Research in this area is needed to keep up with the increasing number of ADRs and integrate new knowledge with the rapidly advancing technology [ ].
According to the WHO, ADR causality assessment is “a method by which it estimates the relationship concerning the agent (which is the drug) and the adverse reactions” . It assesses the causal connection between the drugs and their adverse effects. Assessment of ADRs causality will give an advantage in the ability to classify the relationship, improve scientific evaluation for each individual ADR, and thus enable for an early warning system for clinicians, pharmacists, and health regulators [ - ].
The term severity in ADRs is used to describe the intensity of the adverse drug reaction [, ]. Similar to causality assessment, severity assessment of ADRs are also crucial in epidemiological studies. The ability to classify the severity of ADRs will provide a mechanism for the health care workers and authorities to identify the problem areas and improve the intervention for patient care that would reduce the burden of ADRs [ , , ].
Assessment of preventability is important for ADRs as it gives important information to improve prescription practice and enhance patient monitoring [, ]. Although the assessment of each ADR’s causality, severity, and preventability is crucial to provide important drug safety information, relatively few of these assessments are being performed [ ].
This study was designed to develop apps for the aforementioned ADR assessments using validated tools. The apps can be downloaded on a mobile phone or mobile devices, which are then adapted to improve knowledge on ADRs and ultimately drug safety in health care.
Assessment Tool for Adverse Drug Reaction (ADR) Causality, Severity, and Preventability
There is currently no operational tool that has been proven as a gold standard for each ADR assessment of causality, severity, and preventability; therefore, the most widely used or accepted operational tools were selected for the development of our apps. Each of these tools has been validated by previous studies [, - ].
The ADR causality app was developed using the Liverpool ADR causality assessment tool . This is a questionnaire-based classification for suspected ADRs using an algorithm built by a multidisciplinary team from the University of Liverpool in 2012. The algorithm classifies the suspected ADRs as definite, probable, possible, or unlikely. Results from a systematic review on assessment of ADR causality showed that the Naranjo algorithm was the most frequently used tool [ ]. However, the Liverpool ADR causality assessment tool showed full range of causality category and good interrater reliability (IRR) compared with Naranjo algorithm. Thus, this tool was used in developing the ADR causality app [ ].
The ADR severity app was developed based on the Hartwig’s Severity Assessment Scale , which is the most commonly used severity tool in ADR studies. It classifies the ADR into mild, moderate, or severe based on level of clinical outcomes [ ].
The ADR preventability app was developed using the Modified Schumock and Thornton Preventability Scale , which is the most frequently used scale in ADR studies in children [ ]. It is a questionnaire on the criteria for determining preventability of ADRs based on clinical circumstances surrounding the ADR. The category of preventability is either definitely preventable, probably preventable, or not preventable.
Development and Publishing the App Into Google Play Store
The apps were developed using the rapid application development (RAD) model . Using this model, the development processes are divided into three main phases which are preproduction, production, and postproduction.
For the development of the ADR assessment apps, Windows 8.1 by Microsoft was used as the operational system. MIT App Inventor Tool version 2.3.0  was used during the production process and aiStater emulator [ ] was then used to provide communication between App Inventor running in the browser and other parts of App Inventor. App Inventor is a free, cloud-based service accessed with a Google account.
After all the production phases were completed, the app was then saved in an APK file and then uploaded into the Google Play Developer Console. Once the ADR assessment app was published in Google Play Store, it could then be downloaded and installed for free by Android OS users.
|Assessment tool||Reference||App name|
|Causality assessment||Liverpool Adverse Drug Reaction Causality Assessment Tool ||Adverse Drug ReactionCausality|
|Severity assessment||Hartwig’s Severity Assessment Scale ||Adverse Drug ReactionSeverity|
|Preventability assessment||Modified Schumock and Thronton Preventability Scale ||Adverse Drug RxnPreventability|
The apps were designed without storage capacity to avoid issues regarding patient confidentiality or personal data. Therefore, information input into the system is not available to anyone. The apps are also accessible for offline use. The details for each ADR app and the references used are shown in.
Testing and Measure of App Usability
The System Usability Scale (SUS), a reliable and low-cost usability scale, was used to assess the usability of the ADR app. SUS is a 10-item scale presented with a 5-point Likert scale, which results in an overall score from 0 to 100 that indicates the perceived usability of the interface .
A survey was conducted among 26 health care professionals in the pediatric ward of Hospital Ampang, Malaysia, where they were asked to answer the SUS questionnaire. The survey was conducted 10 months after the introduction of the apps among staff at the pediatric ward of Hospital Ampang.
Results of the SUS questionnaire were recorded and normalized using SPSS version 20 (IBM Corp). The mean SUS score and the standard deviation (SD) were then recorded. Products with scores <70 were considered candidates for increased scrutiny, and continued improvement was judged to be marginal at best .
App Development and Publication
The developed apps were published in Google Play Store on January 22, 2015. All 3 apps were considered to have fulfilled the objective of the development. The apps were freely downloadable from Google Play Store from February 2015. Exemplar screenshots for each app are shown infor causality, for severity, and for preventability.
Up until January 20, 2017, a total of 609 users have downloaded the apps. The total installer, installer by country, and ranking statistics for each ADRs assessment app are shown in. The highest numbers of downloads were for the causality app followed by the severity and preventability apps. The installers were mainly from India and Malaysia for all the apps.
App Usability Among Health Care Professionals
Of the 26 health care professionals involved in the survey, 19 (73.1%) of the respondents were physicians, 6 respondents (23.1%) were nurses, and 1 respondent (3.8%) was a pharmacist. The mean SUS score was 70.9 (SD 12.86). The results showed that the SUS score was >70; thus, the app tested is within the acceptable range of usability . depicts the responses to the usability-related questions.
|Characteristics||Adverse Drug ReactionCausality||Adverse Drug ReactionSeverity||Adverse Drug RxnPrevenrability|
|Installer by country (%)||India 77 (28.1)||Malaysia 70 (35)||India 48 (35.5)|
|Malaysia 69 (25.0)||India 61 (30.5)||Malaysia 31 (23.0)|
|Saudi Arabia 43 (15.6)||Qatar 17 (8.5)||Qatar 16 (11.9)|
|South Africa 17 (6.3)||Saudi Arabia 17 (8.5)||Others 40 (29.6)|
|United States 17 (6.3)||South Africa 17 (8.5)|
|Others 51 (18.7)||Others 18 (9)|
|Answer option||Strongly disagree |
|Strongly agree |
|I think that I would like to use this system frequently||0 (0)||0 (0)||3 (11)||21 (81)||2 (8)|
|I found the system unnecessarily complex||5 (19)||10 (38)||8 (31)||3 (12)||0 (0)|
|I thought the system was easy to use||0 (0)||1 (4)||1 (4)||18 (69)||6 (23)|
|I think that I would need the support of a technical person to be able to use this system||6 (23)||11 (42)||3 (12)||6 (23)||0 (0)|
|I found the various functions in this system were well integrated||1 (4)||2 (8)||7 (27)||14 (53)||2 (8)|
|I thought there was too much inconsistency in this system||3 (12)||12 (45)||9 (35)||2 (8)||0 (0)|
|I would imagine that most people would learn to use this system very quickly||0 (0)||0 (0)||4 (15)||14 (54)||8 (31)|
|I found the system very cumbersome to use||5 (19)||12 (46)||5 (19)||4 (16)||0 (0)|
|I felt very confident using the system||0 (0)||0 (0)||7 (27)||16 (61)||3 (12)|
|I needed to learn a lot of things before I could get going with this system||5 (19)||15 (57)||3 (12)||3 (12)||0 (0)|
The causality app had the highest number of installers so far. This seems to be similar with previous 102 published ADR studies where causality was the most common assessment conducted in suspected ADR cases .
Based on the country of origin, the highest percentage of installers were from India for all of the ADR apps published. India is currently working to strengthen its pharmacovigilance program due to the rapidly growing number of ADR studies in the country [- ]. We expect that our ADR apps would be able to assist not only Indian researchers and clinical researchers but also any center conducting research on drug safety.
The apps usability among health professionals in the hospital was assessed using SUS, which consisted of 10 alternate statements of positive and negative items rated using a 5-point Likert scale. Our survey results show that the apps developed have a mean SUS of 70.9 (SD 12.86), thus demonstrating acceptable usability.
The health care professionals that used the apps concluded that the apps were convenient and they would choose them over conventional paper-based assessments. Previous studies have found that the use of apps in health care is cost-effective, faster, easier, and more interactive due to factors of mobility, convenience, and involvement of active touching of the screen to perform the assessment [, ]. The apps are also secure, as they do not store any information from the data inputted into the app [ ].
The use of medical apps by health care professionals and researchers, and the numbers of these apps are increasing rapidly. Apps can give additional advantages at the point of care such as in diagnosis, monitoring, reporting, or follow-up of treatment . The increased usage of mobile phone or mobile device apps warrant further studies evaluating their utility and effectiveness on a larger scale.
We have identified a few limitations of the apps. On a basic level, there is no assessment tool universally accepted or described as the gold standard for ADRs either for causality, severity, or preventability. We chose the most widely used and validated algorithms and scales to develop the apps; however, we recognize that not all researchers will agree with the algorithms chosen in development of ADR assessment.
Second, the apps have only been evaluated by health care professionals from the pediatric department in a hospital setting. Further evaluation is necessary to gain more feedback from a wider range of users.
Finally, the aesthetics of the app contents in terms of color, text letters, and pictures have been optimized; however, there is room for improvement to make the apps more attractive.
We are continuously working to update and upgrade the apps. Future research is needed to test the usability of the apps in varying populations and to add several other commonly used algorithms or tools in ADRs assessment. Research to highlight the context and content of the apps should also be designed specifically for health care professionals, researchers, and regulators.
These ADR assessment apps will aid health care professionals in determining the causality, severity, and preventability of ADRs. This is aimed to contribute toward efforts to reduce the burden of ADRs on patients. The SUS score data showed that the apps have acceptance usability among health care professionals. They will also support future research to enhance overall safety relating to drugs given to patients.
We would like to thank all the doctors, nurses, and pharmacists from Hospital Ampang who were involved in the survey. A special thanks to Mr Che Mat bin Muda for helping us to program the apps. This study was funded by Malaysian Ministry of Higher Education (MOHE), grant no. USIM/RAGS/FPSK/36/50314. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
KN, MDMR, and TAMTM conceived and supervised the study. MI and AS designed and published the app. They, together with ZAL and PK, tested the apps. MI wrote the paper, while KN and TAMTM edited the manuscript and gave the final approval. All authors contributed exclusively to the work presented in this paper.
Conflicts of Interest
- Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998;279(15):1200-1205. [Medline]
- Impicciatore P, Choonara I, Clarkson A, Provasi D, Pandolfini C, Bonati M. Incidence of adverse drug reactions in paediatric in/out‐patients: a systematic review and meta‐analysis of prospective studies. Brit J Clin Pharmaco 2001;52(1):77-83. [CrossRef]
- Le J, Nguyen T, Law AV, Hodding J. Adverse drug reactions among children over a 10-year period. Pediatrics 2006;118(2):555-562. [CrossRef] [Medline]
- Wester K, Jönsson AK, Spigset O, Druid H, Hägg S. Incidence of fatal adverse drug reactions: a population based study. Brit J Clin Pharmaco 2008;65(4):573-579. [CrossRef]
- Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse drug reactions: a systematic review of prospective observational studies. Ann Pharmacother 2008;42(7-8):1017-1025.
- World Health Organization. International drug monitoring. The role of the hospital. Geneva: World Health Organization; 1969.
- Smyth RM, Gargon E, Kirkham J, Cresswell L, Golder S, Smyth R, et al. Adverse drug reactions in children—a systematic review. PLoS One 2012;7(3):e24061. [CrossRef]
- Khan LM. Comparative epidemiology of hospital-acquired adverse drug reactions in adults and children and their impact on cost and hospital stay - a systematic review. Eur J Clin Pharmacol 2013;69(12):1985-1996. [CrossRef]
- Hameed K. The application of mobile computing and technology to health care services. Telemat Inform 2003;20(2):99-106 [FREE Full text]
- Dehlingher J, Dixon J. Mobileseworkshop. 2011. Mobile application software engineering: challenges and research directions URL: http://www.mobileseworkshop.org/papers/7_Dehlinger_Dixon.pdf [accessed 2017-05-26] [WebCite Cache]
- Holzinger A, Kosec P, Schwantzer G, Debevc M, Hofmann-Wellenhof R, Frühauf J. Design and development of a mobile computer application to reengineer workflows in the hospital and the methodology to evaluate its effectiveness. J Biomed Inform 2011;44(6):968-977. [CrossRef]
- Mosa AS, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis 2012;12(1):67 [FREE Full text] [CrossRef] [Medline]
- Ahtinen A, Mattila E, Välkkynen P, Kaipainen K, Vanhala T, Ermes M. Mobile mental wellness training for stress management: feasibility and design implications based on a one-month field study. JMIR mHealth and uHealth 2013;1(2):e11. [CrossRef] [Medline]
- Lu Y, Xiao Y, Sears A, Jacko JA. A review and a framework of handheld computer adoption in healthcare. Int J Med Inform 2005 Jun;74(5):409-422. [CrossRef] [Medline]
- Holzinger A, Searle G, Kleinberger T, Seffah A, Javahery H. Investigating usability metrics for the design and development of applications for the elderly. In: Miesenberger K, Klaus J, Zagler W, Karshmer A, editors. Computers Helping People with Special Needs. Berlin, Heidelberg: Springer; 2008:98-105.
- Holzinger A, Errath M. Mobile computer web-application design in medicine: some research based guidelines. Universal Access Inf 2007;6(1):31-41 [FREE Full text]
- World Health Organization. WHO. Uppsala: The Uppsala Monitoring Centre; 2005. The use of the WHO-UMC system for standardized case causality assessment URL: https://www.who-umc.org/media/2768/standardised-case-causality-assessment.pdf [accessed 2014-01-25] [WebCite Cache]
- Meyboom RH, Hekster YA, Egberts AC, Gribnau FW, Edwards IR. Causal or casual? The role of causality assessment in pharmacovigilance. Drug Saf 1997;17(6):374-389. [Medline]
- Arimone Y, Bégaud B, Miremont-Salamé G, Fourrier-Réglat A, Moore N, Molimard M, et al. Agreement of expert judgment in causality assessment of adverse drug reactions. Eur J Clin Pharmacol 2005;61(3):169-173. [CrossRef]
- Macedo AF, Marques FB, Ribeiro CF, Teixeira F. Causality assessment of adverse drug reactions: comparison of the results obtained from published decisional algorithms and from the evaluations of an expert panel. Pharmacoepidem Dr S 2005;14(12):885-890. [CrossRef]
- ICH Expert Working Group. Clinical safety data management: definitions and standards for expedited reporting ICH-2 EWG E2. 1994 Presented at: International conference on harmonisation of technical requirements for registration of pharmaceuticals for human use; October 27, 1994; Geneva, Switzerland URL: https://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E2A/Step4/E2A_Guideline.pdf
- Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. The Lancet 2000;356(9237):1255-1259. [CrossRef] [Medline]
- Doherty MJ. Algorithms for assessing the probability of an adverse drug reaction. Respiratory Medicine CME 2009;2(2):63-67 [FREE Full text] [CrossRef]
- Hartwig SC, Siegel J, Schneider PJ. Preventability and severity assessment in reporting adverse drug reactions. Am J Hosp Pharm 1992;49(9):2229-2232. [Medline]
- Agbabiaka TB, Savović J, Ernst E. Methods for causality assessment of adverse drug reactions. Drug Safety 2008;31(1):21-37. [CrossRef]
- Ferner RE, Aronson JK. Preventability of drug-related harms—part I. Drug Safety 2010;33(11):985-994. [CrossRef]
- Benkirane R, Soulaymani-Bencheikh R, Khattabi A, Benabdallah G, Alj L, Sefiani H, et al. Assessment of a new instrument for detecting preventable adverse drug reactions. Drug Safety 2015;38(4):383-393. [CrossRef]
- Gallagher RM, Kirkham JJ, Mason JR, Bird KA, Williamson PR, Nunn AJ, et al. Development and inter-rater reliability of the Liverpool adverse drug reaction causality assessment tool. PLoS One 2011;6(12):e28096. [CrossRef]
- Mrugank BP, Hareesha RP. Prospective observational, non-randomized, parallel sequence study for assessment of adverse drug reactions due to chemotherapeutic treatment in different types of cancer patients. Int J Pharm Sci Res 2013;4(1):386-391.
- Martin J. Rapid application development. Indianapolis, IN: Macmillan Publishing Co; 1991.
- Walter D, Sherman M. Learning MIT App Inventor: A Hands-on Guide to Building Your Own Android Apps. Crowfordsville, IN: Pearson Education; 2014.
- Massachusetts Institute of Technology. MIT App Inventor. 2012. Installing and running the emulator in AI2 URL: http://appinventor.mit.edu/explore/ai2/setup-emulator.html [accessed 2014-01-15] [WebCite Cache]
- Bangor A, Kortum P, Miller JA. The system usability scale (SUS): an empirical evaluation. Int J Hum Comput Interact 2008;24(6):574-594.
- Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Studies 2009;4(3):114-123.
- Priyadharsini R, Surendiran A, Adithan C, Sreenivasan S, Sahoo FK. Study of adverse drug reactions in pediatric patients. J Pharmacol Pharmacother 2011;2(4):277-280.
- Digra KK, Pandita A, Saini GS, Bharti R. Pattern of adverse drug reactions in children attending the department of pediatrics in a tertiary care center: a prospective observational study. Clin Med Insights Pediatr 2015;9:73-78. [CrossRef]
- Ponnusankar S, Tejaswini M, Chaitanya M. Assessment of Adverse Drug Reactions Based on Spontaneous Signals at Secondary Care Public Hospital. Indian J Pharm Sci 2015;77(4):490-493 [FREE Full text] [Medline]
- Kalaiselvan V, Kaur I, Singh S, Singh GN. Pharmacovigilance Programme of India: system put in place to report adverse drug reactions. Indian J Pharm Educ 2015;50(1):212-214 [FREE Full text] [CrossRef]
- Weaver A, Young AM, Rowntree J, Townsend N, Pearson S, Smith J, et al. Application of mobile phone technology for managing chemotherapy-associated side-effects. Ann Oncol 2007;18(11):1887-1892. [CrossRef]
- Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol 2015;6:27882. [CrossRef]
Edited by G Eysenbach; submitted 27.06.16; peer-reviewed by L Sheets, S Sahay, F Sanz, S Abideen, R Altman; comments to author 18.11.16; revised version received 27.01.17; accepted 15.03.17; published 30.05.17Copyright
©Muslimah Ithnin, Mohd Dzulkhairi Mohd Rani, Zuraidah Abd Latif, Paveethra Kani, Asmalita Syaiful, Khairun Nain Nor Aripin, Tengku Amatullah Madeehah Tengku Mohd. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 30.05.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.