There is currently no operational tool that has been proven as a gold standard for each ADR assessment of causality, severity, and preventability; therefore, the most widely used or accepted operational tools were selected for the development of our apps. Each of these tools has been validated by previous studies [16,25-29].

The ADR causality app was developed using the Liverpool ADR causality assessment tool [28]. This is a questionnaire-based classification for suspected ADRs using an algorithm built by a multidisciplinary team from the University of Liverpool in 2012. The algorithm classifies the suspected ADRs as definite, probable, possible, or unlikely. Results from a systematic review on assessment of ADR causality showed that the Naranjo algorithm was the most frequently used tool [7]. However, the Liverpool ADR causality assessment tool showed full range of causality category and good interrater reliability (IRR) compared with Naranjo algorithm. Thus, this tool was used in developing the ADR causality app [28].

The ADR severity app was developed based on the Hartwig’s Severity Assessment Scale [24], which is the most commonly used severity tool in ADR studies. It classifies the ADR into mild, moderate, or severe based on level of clinical outcomes [7].

The ADR preventability app was developed using the Modified Schumock and Thornton Preventability Scale [29], which is the most frequently used scale in ADR studies in children [7]. It is a questionnaire on the criteria for determining preventability of ADRs based on clinical circumstances surrounding the ADR. The category of preventability is either definitely preventable, probably preventable, or not preventable.

The apps were developed using the rapid application development (RAD) model [30]. Using this model, the development processes are divided into three main phases which are preproduction, production, and postproduction.

For the development of the ADR assessment apps, Windows 8.1 by Microsoft was used as the operational system. MIT App Inventor Tool version 2.3.0 [31] was used during the production process and aiStater emulator [32] was then used to provide communication between App Inventor running in the browser and other parts of App Inventor. App Inventor is a free, cloud-based service accessed with a Google account.

After all the production phases were completed, the app was then saved in an APK file and then uploaded into the Google Play Developer Console. Once the ADR assessment app was published in Google Play Store, it could then be downloaded and installed for free by Android OS users.

The apps were designed without storage capacity to avoid issues regarding patient confidentiality or personal data. Therefore, information input into the system is not available to anyone. The apps are also accessible for offline use. The details for each ADR app and the references used are shown in Table 1.

The System Usability Scale (SUS), a reliable and low-cost usability scale, was used to assess the usability of the ADR app. SUS is a 10-item scale presented with a 5-point Likert scale, which results in an overall score from 0 to 100 that indicates the perceived usability of the interface [33].

A survey was conducted among 26 health care professionals in the pediatric ward of Hospital Ampang, Malaysia, where they were asked to answer the SUS questionnaire. The survey was conducted 10 months after the introduction of the apps among staff at the pediatric ward of Hospital Ampang.

Results of the SUS questionnaire were recorded and normalized using SPSS version 20 (IBM Corp). The mean SUS score and the standard deviation (SD) were then recorded. Products with scores <70 were considered candidates for increased scrutiny, and continued improvement was judged to be marginal at best [34].

The developed apps were published in Google Play Store on January 22, 2015. All 3 apps were considered to have fulfilled the objective of the development. The apps were freely downloadable from Google Play Store from February 2015. Exemplar screenshots for each app are shown in Figure 1 for causality, Figure 2 for severity, and Figure 3 for preventability.

Up until January 20, 2017, a total of 609 users have downloaded the apps. The total installer, installer by country, and ranking statistics for each ADRs assessment app are shown in Table 2. The highest numbers of downloads were for the causality app followed by the severity and preventability apps. The installers were mainly from India and Malaysia for all the apps.

Of the 26 health care professionals involved in the survey, 19 (73.1%) of the respondents were physicians, 6 respondents (23.1%) were nurses, and 1 respondent (3.8%) was a pharmacist. The mean SUS score was 70.9 (SD 12.86). The results showed that the SUS score was >70; thus, the app tested is within the acceptable range of usability [34]. Table 3 depicts the responses to the usability-related questions.

The causality app had the highest number of installers so far. This seems to be similar with previous 102 published ADR studies where causality was the most common assessment conducted in suspected ADR cases [7].

Based on the country of origin, the highest percentage of installers were from India for all of the ADR apps published. India is currently working to strengthen its pharmacovigilance program due to the rapidly growing number of ADR studies in the country [35-38]. We expect that our ADR apps would be able to assist not only Indian researchers and clinical researchers but also any center conducting research on drug safety.

The apps usability among health professionals in the hospital was assessed using SUS, which consisted of 10 alternate statements of positive and negative items rated using a 5-point Likert scale. Our survey results show that the apps developed have a mean SUS of 70.9 (SD 12.86), thus demonstrating acceptable usability.

The health care professionals that used the apps concluded that the apps were convenient and they would choose them over conventional paper-based assessments. Previous studies have found that the use of apps in health care is cost-effective, faster, easier, and more interactive due to factors of mobility, convenience, and involvement of active touching of the screen to perform the assessment [11,39]. The apps are also secure, as they do not store any information from the data inputted into the app [40].

The use of medical apps by health care professionals and researchers, and the numbers of these apps are increasing rapidly. Apps can give additional advantages at the point of care such as in diagnosis, monitoring, reporting, or follow-up of treatment [12]. The increased usage of mobile phone or mobile device apps warrant further studies evaluating their utility and effectiveness on a larger scale.

We have identified a few limitations of the apps. On a basic level, there is no assessment tool universally accepted or described as the gold standard for ADRs either for causality, severity, or preventability. We chose the most widely used and validated algorithms and scales to develop the apps; however, we recognize that not all researchers will agree with the algorithms chosen in development of ADR assessment.

Second, the apps have only been evaluated by health care professionals from the pediatric department in a hospital setting. Further evaluation is necessary to gain more feedback from a wider range of users.

Finally, the aesthetics of the app contents in terms of color, text letters, and pictures have been optimized; however, there is room for improvement to make the apps more attractive.

We are continuously working to update and upgrade the apps. Future research is needed to test the usability of the apps in varying populations and to add several other commonly used algorithms or tools in ADRs assessment. Research to highlight the context and content of the apps should also be designed specifically for health care professionals, researchers, and regulators.

These ADR assessment apps will aid health care professionals in determining the causality, severity, and preventability of ADRs. This is aimed to contribute toward efforts to reduce the burden of ADRs on patients. The SUS score data showed that the apps have acceptance usability among health care professionals. They will also support future research to enhance overall safety relating to drugs given to patients.