Vula mobile is a mobile phone app for remote consultation and referrals between emergency doctors at point of care and specialists, which was developed by Vula Mobile, Mafami Pty Ltd. The app runs on both iPhone and Android operating systems, and it can be downloaded free of charge. The app was introduced to the Western Cape, South Africa, in 2014 and now allows for referrals to 15 specialties such as ophthalmology, orthopedics, dermatology, and burns. The app currently handles over 5000 referrals per month, with 62 specialist teams actively taking referrals on Vula. The ability to refer patients with burns was introduced in April 2016, and it handled around 250 referrals during 2017. The app provides a template for each specialty, including patient and clinical data (see Figure 1, screenshot 1). There are specific features related to each specialty, the ability to take photographs, and a chat function. The burns section of the app includes a feature to draw the burn on a depiction of a body, which will calculate burn size and fluid requirements (see Figure 1, screenshots 2, 3, and 4). This feature also allows the burns consultant to see the location, size, and depth of the burn. All the documented information including photos are only saved within the app and not saved elsewhere on the phone. The form and the features were established in collaboration with burns experts and emergency specialists before the development of the burns section of the app. When a referring clinician has completed the form, the information is uploaded to a secure server and shared with the burns specialist on call who will be notified. The specialist reviews the information on their mobile phone or computer and sends back treatment and referral advice either via predefined medical advice options or an instant messaging function. After the consultation is complete, the specialist can archive the referral, which will remove it from both the referring doctor’s and the specialist’s devices. Users are mandated to take the necessary precautions in accordance with the “South African Protection of Personal Information Act, 2013” and to act in accordance with national and local legislation. The user is prompted by the app to make sure that the patient has consented to data being stored electronically. As with any consultation, whether it is face-to-face or telephonic, both the referring doctor and the specialist have to make the appropriate medical documentation as mandated by the health care organization where they operate. In this study, participants assessed the app on their own device.

Cape Town has experienced rapid urbanization with a large part of its population living in suboptimal housing that is characterized by crowded living situations, lack of running water, lack of electricity, and houses built out of flammable materials [46]. It has a high burden of burn injuries, with an estimated mortality rate of 7.9 per 100.000 person-years [47]. In Cape Town, there are 2 hospitals with specialized burns units serving the Western Cape province. Depending on the burn severity and day-to-day capacity, patients can be treated at local clinics or referring hospitals. Furthermore, to improve referral of patients in the region, referral criteria have been implemented [48]. However, studies indicate that inappropriate or delayed referrals are still commonplace in this region [6].

This study was conducted in the emergency centers at 2 health facilities, 1 district hospital, and 1 health clinic in Cape Town. Participants were doctors sampled at the emergency center at Khayelitsha Hospital and Gugulethu Community Health Centre by convenience sampling. The primary focus of this study is on emergency doctors; however, an additional 4 of the burns consultants who were registered with the app were selected by convenience sampling and interviewed for the study to explore their view of the app. Although training is available upon request, no training is required to use the app. All participants had used the app for other specialties, and some participants used the app for burns consultations. Among the participants, 6 had never used the app for burns, 14 had used it less than 5 times, and 4 had used it more than 5 times. The burns consultants had all used the app for consultations at least 10 times before the interview.

This study was a mixed-methods study including a qualitative approach using the think-aloud method and a questionnaire measuring different usability metrics. As the app is already in use, the goal of this usability study was to identify usability problems related to the Vula app to improve the existing app.

Video recordings were analyzed with MAXQDA (Version 12, VERBI Software, Berlin, Germany), a software for analysis of qualitative data. The program allows for analysis of both video and text. First, the audio recordings from the video were transcribed verbatim. In MAXQDA, this can be done using timestamps to have easy access to each audio and video segment that correlates with the text segment in the transcript. The videos can be analyzed by coding directly into the timeline when viewing the video. Content analysis with predefined codes developed by Kushniruk and Borycki was used for coding the data [51]. The codes cover 3 themes: usability, usefulness, and safety- and technology-induced errors (see Table 3). The coding scheme was adapted to suit this study, and codes were also added if findings did not fit the predefined codes. Conversely, some codes were omitted if they did not apply to the material. AK and SCF independently coded all interviews and discussed the labeling of each coded event. PYY worked in the final stage with AK to ensure the codes were accurately assigned to each scenario. The data from the questionnaire were analyzed using descriptive statistics with SPSS (Version 23, IBM Corp, Armonk, NY).

All participants were presented with both verbal and written information about the study before the interview, and written consent was obtained. Consent forms with participant identifiers were separated from the collected data. Data are presented on an aggregated level. The study was approved by the University of Cape Town, Faculty of Health Sciences Human Research Ethics Committee.

Overall, 24 emergency doctors were included in the study, and all of them were working in the emergency department at the 2 different health facilities. Demographics of the study participants are presented in Table 4. Most participants were rotating, that is, they were on a short-term allocation to the emergency center. Experience in emergency care varied from 3 months to 7 years with a median time of 1 year. Participants rated their experience with burn care as either minimal (n=11), moderate (n=10), extensive (n=1), or none (n=2). The participants all used mobile phones for private and work purposes. The purpose of use was sending images, apps, or browsing the internet for reference on medical conditions, criteria, and drug dosage calculations. The mobile phones were also used to discuss cases with seniors using either instant messaging or voice calls.

All participants were able to complete all tasks with no major difficulties. However, several usability issues were identified. Usability issues were classified as either usability problems, usefulness of content, or safety- or technology-induced errors, as described in the Methods section (Table 3). Codes and frequency of problems encountered related to each domain in this analysis are presented in Table 5.

Questions and scores relating to each construct in the Health-ITUES questionnaire are presented in Table 6. In general, the app scored high on most of the constructs, with the construct “ease of use” scoring the highest. Usefulness also scored high; however, the questions relating to the ability to receive management advice in a timely manner tended to be lower (items 5, 10, and 12). The construct “user control” scored the lowest, especially the items relating to error prevention (item 18 and 19).

A total of 4 interviews were also conducted with burns consultants who used the app to give diagnostic support to emergency doctors. The first screen the consultants see displays all information about the patient sent by the referring doctor. From this page, they can either go into “Send advice,” “Chat,” or review photos. Some consultants did not use the app in the way it was intended to be used; mostly they found it easier to initiate a chat with the referring doctor. In one case, the consultant was unaware of the send advice function. One of the participants said, in general, it is good with predefined options, but in this app, the options were not very useful:

They all found the pictures to be very beneficial as they could then more accurately assess the burn depth and size. However, 1 consultant expressed that many doctors will not clean the burns or remove blisters before taking the pictures, which then made it harder for the consultant to be able to assess the burn. The consultant suggested there should be instructions in the app about this before the user takes a picture. One consultant seemed unaware that the referring doctor could take pictures within the app and would usually ask them to send pictures through WhatsApp instead.

Although the Health-ITUES questionnaire showed the usability to be satisfactory, the think-aloud evaluation revealed several important usability problems that should be considered for improvements for this particular app and for others planning to design apps for remote consultations.

Most issues occurred in the drawing feature, which is a central feature of the app that allows the users to describe the size, depth, and location of the burn. For the burns consultant, this information is of great importance when assessing a burn. Some users expressed frustration, and a few even said they thought this feature was too difficult to use. Conversely, many users said that this was one of the strengths of the app, suggesting it may be a central feature that is not present in other communication apps. The consultants also mentioned that being able to see the location of the burn visually is very important for them to do their assessment. This resonates with the findings by Blom et al in their study on expectations of burn specialists about image-based teleconsultation [52].

Given that the drawing function is one of the central features but at the same time where most users struggled, it calls for some reflection. Any extra added feature needs to be well justified, that is, increase the usefulness of the app, but at the same time, it needs to be easy to use, that is, free of effort. Rust et al argue that adding extra features may be attractive but could lead to a product that is overly complex with decreased usability, resulting in feature fatigue [53]. Considering these problems that the users encountered, we suggest that any extra feature be well justified (add value to the user) and be designed to be easy to use.

Other usability problems were related to the meaning of icons and their functionalities. Although design changes can improve these, it also demonstrates the importance of user testing with the intended users. Ehrler et al, who found similar issues assessing a mobile app to support nurses, suggest both design changes and also training to mitigate some of these problems [54].

Another finding related to the drawing feature was that some users did not agree with the calculation of the burn surface and said that they would have assessed the burn to be larger had they not used the app. It is well documented in the literature that nonburns specialists tend to overestimate burn surface area when visually assessing the burn [23,24]. One way to at least make it more apparent in the app would be to have the percentage indicated on each body part as a reminder of the burn surface area.

One critical part of usability testing of medical apps is to identify problems that could result in harm to the patient; for example, the fact that some users made mistakes that led to either a smaller or larger calculation of the percentage, which in turn could lead to adverse effects if this discrepancy is significant. This is one of the reasons why supplementing with photos is important so that the burns consultant can make their assessment of the burn.

Although it is important that new technologies are easy to use, the usefulness of the content of a system is equally or even more important to end users [55]. Although the participants found the content to be useful in the app, many users thought that the app lacked some information or options. First, many users felt limited by the predefined choices and often wanted to describe more about the injury. For example, the medical conditions that are asked for in the app are limited to a few that are of interest to the burn specialist. However, some users thought this list could have been extended. In a similar vein, many users were also not sure whether hot coffee should be classified as hot liquid, not water or hot water. Both these options are meant to indicate that the burn was a scald. Hot liquids, not water is meant to represent what can be referred to as a dense liquid burn that is caused by liquids such as milk and oil [56]. In contrary to less dense liquids such as water, dense liquids will retain more heat and will adhere to the skin longer because of their higher viscosity. Therefore, such burn injuries may result in deeper burns. Although coffee or tea, for example, is not just water, it still has the same properties, which many users also mentioned. However, as one participant pointed out, if milk is added, this will change. In terms of the usability of the app, this relates to the meaning of icons/terminology, as well as information needs. At best, this may only cause confusion and irritation, but there could be instances where misunderstandings might have implications for the patient.

Despite a field for additional comments at the bottom of the form, some users suggested that there should be a possibility to specify with free text under each subsection when the options were too limiting. We did not further explore why some participants felt like this, but one explanation could be that at this point in the user test, they were unaware of the comment box that is located at the end of the form. A study by Hysong et al of e-referral systems reported similar findings where the primary health practitioners felt constrained by the use of templates and that they were not able to communicate findings clearly [57]. Similarly, in the tests with the burn consultants, some of them said they did not like the predefined options in their present form and would rather use the chat function instead.  Hysong et al, on the other hand, found that specialists thought that more rigid templates with mandatory fields would enhance the quality of the referrals [57]. Other studies have found that feedback from consultants is more consistent and timely when referral templates are used [58,59]. One interesting finding regarding the comment box was that many participants used this to write a message to the specialist repeating much of the information already filled out. These messages would often be of a friendly nature including phrases such as “dear Dr” or “kind regards.” This highlights the fact that a consultation is not merely an exchange of information but also a collaboration that requires personal communication [60].

Our findings indicate that both emergency doctors and consultants wanted more flexibility within the system. Flexible systems can make it easier to transfer information about special cases, reducing the need for additional phone and text communication. However, when designing a system, the need for flexibility must be balanced with the need for its usability. Although it might be tempting to design a system for every scenario to maximize flexibility, one needs to take into account that when the flexibility of a system increases, so does its complexity and consequently its usability. This is often termed the flexibility-usability trade-off [61], and when doing this trade-off, it is important to understand the users’ needs, both present and future. This is, however, not always possible, especially with newer technologies. In addition, as acute burn injuries can manifest in different ways, it is difficult to design a template that will fit the clinical presentation of every single patient. Esquivel et al outlined 10 recommendations for improving the effectiveness of electronic referrals [3]. One recommendation that our findings support is to design and use standardized electronic referral templates that include both structured and free text fields. They argue that when designing electronic referral templates, there must be both structured fields to capture required information and free text fields for providers to freely expand on their clinical findings.

The usability tests with the consultants revealed that the 2 more experienced users found the ”Send advice” function cumbersome and would rather just chat with the referring doctor. The 2 less experienced consultants seemed to be unaware of some of the functionalities in the app, such as the “Send advice” function or how to access the photos, suggesting that the user interface may not be clear for new users.

Although the participants in our study perceived the burns section of the app to be useful and easy to use, it is still not used on a regular basis. However, there are other reasons for low uptake that are not related to the usability of the app. These include, but are not limited to, awareness of the app, resistance to new technologies and attitudes among colleagues [43]. Even though traditional means of discussing and referring patients with burns by telephonic consultations and paper-based referrals are still prevalent, recent reports from South Africa have described the use of WhatsApp as a mode of communication for clinical decision support [32,36]. The fact that WhatsApp is very prevalent may be a reason why a specific app for medical referrals is not widely adopted. For the referring doctor seeking advice, the most important aspect will inevitably be to receive advice back from the consultant in a timely manner. Therefore, the consultants have an important role to play as pioneers for apps such as the Vula app. Nikolic et al note in their recent study that the widespread use of WhatsApp may impede the introduction of other communication apps designed for medical consultations [62]. Future research should focus on identifying the barriers to using mobile phone-based referral and consultation systems.

The methods and equipment for data collection used in this study were simple, low cost, and allowed for quick data collection. None of the participants objected or showed any signs of being uncomfortable wearing the recording equipment. The video coding scheme was useful and covered most aspects. However, we changed the original definitions to cover both positive and negative aspects of the events and not only focus on problems. By doing this, the data generated more findings concerning things that the users liked about the app. We also added some codes that we thought were not covered in the original coding scheme. For example, Lack of user instructions was added as an extension to the code Not understanding user instructions.

During the think-aloud session, the interviewer tried not to interfere, except for encouraging the interviewee to keep talking. However, we found that the participants would mostly verbalize what they were doing but not what they were thinking and feeling. Consequently, during the second period of data collection, we extended the interview by going through the app once again where the interviewer asked about their thoughts on the different parts of the app.

Although the users rated the app to be easy to use, the think-aloud interviews found several problems related to the ease of use. This is not surprising as the purpose of a usability questionnaire is to assess the users’ overall perception of the app and not identify specific problems. It is, however, useful for comparing different user groups, different designs of the app, or how user perceptions change over time.

As this was a mixed-methods study spanning both emergency burn care and health informatics, we think it was a strength that the authors come from different disciplines, including public health, political science, sociology, emergency medicine, nursing, and health informatics. Another strength of the study was the large number of participants in the user tests. A limitation was the relatively homogenous sample, where all users were young junior doctors. Sonderegger et al found that age affects several usability measures such as speed and accuracy [63]. Cillessen et al studied the user satisfaction of a clinical notes app and only differences by medical specialty but no differences by sex, age, professional experience, or training hours [64]. Nevertheless, our sample does represent the typical user from our study population.

Mobile referral and communication systems are a relatively new concept that could simplify the consultation and referral process within burn care and other specialties. However, when a new system is competing with other technologies such as instant messaging apps or traditional phone calls, usefulness and ease of use become highly important. Although the Health-ITUES questionnaire showed high satisfaction with the system, the think-aloud interviews provided additional insights for further improvements. The users liked the ability to describe the burns through both the built-in drawing function as well as with photographs. For the burn consultants, the drawings and the photos were considered the most valuable information for their assessment. We also found that users had problems with navigation in the drawing section when the app did not act in the same way as other apps that were frequently used. There was also some confusion regarding terminology and the meaning of buttons and icons. Our study also resonates with previous findings that when using standardized electronic consultation and referral templates, the system should also allow the users to freely express their clinical findings in their own words.