Chemotherapy-induced nausea and vomiting are among the most common and burdensome side effects that are experienced by children who are undergoing cancer treatment [1,2]. The symptoms are often accompanied by profound physical and psychological consequences, including dehydration, electrolyte imbalance, weight loss, and disruption of normal childhood activities [1,2]. The prevention of chemotherapy-induced nausea and vomiting has been improved over the years with novel antiemetic drug combinations [3-6], but approximately 50% of pediatric cancer patients still suffer from chemotherapy-induced nausea and vomiting [7-9].

Correct assessment is an essential first step to manage nausea experienced by children with cancer, but it is difficult to assess subjective symptoms such as nausea in children. Clinicians and children express and interpret nausea differently, which makes it a challenge to score the symptoms with high validity and reliability; therefore, antiemetic efficacy in pediatric trials is often solely based on vomiting (an objective outcome), while nausea (a subjective outcome) is rarely included as a clinical outcome [7,8,10]. Nausea is an important outcome measure because it occurs more often than vomiting and is one of the most distressing side effects for children with cancer [2]. Clinical trials should, therefore, assess antiemetic efficacy in children receiving chemotherapy using both nausea and vomiting as clinical outcomes.

The side effects of chemotherapy are often underdocumented by clinicians during prospective clinical trials [11-13]. Patient-reported outcomes can give more valid and reliable information on the side effects of chemotherapy [14]. Patient-reported outcomes are defined as symptoms that are directly reported by the patient, without interpretation by a clinician or anyone else and are used as clinical trial outcomes [14]. One barrier to accurately reporting the side effects experienced by patients as patient-reported outcomes is that the accuracy can be affected by the patient’s memory [14]. It is extremely important to record patient-reported outcomes at assigned times during pediatric clinical trials as recall bias may be more pronounced among children than among adults [15]. Also, pediatric cancer patients must often commute frequently between hospital and home, and some children live alternately with different parents or caregivers; in these situations, paper diaries may get lost or forgotten.

Electronic diaries can collect patient-reported outcomes with more precision and effectiveness than paper diaries [16], and the increased number of smartphone users allows an opportunity to develop smartphone apps that collect real time patient-reported outcomes [17,18]. Even though pediatric patient-reported outcomes are advocated, parent-reported outcomes may be needed when the child is too young to complete a patient-reported outcome questionnaire [19].

There are several smartphone apps for adult patients with cancer that can monitor real time patient-reported outcomes and that are used in clinical trials and to support patients as they manage the side effects of chemotherapy [17,20,21]. In contrast, only a few smartphone apps have been developed to monitor patient-reported outcomes in children and adolescents with cancer [22]. Wang et al [23] developed a smartphone app where patients aged 8 years or older were able to record several symptoms such as pain, fatigue, depression, and anxiety. A user-centered design process has also been applied in the development of smartphone apps to assess and manage pain in children and adolescents [24,25]; Stinson et al [24] developed a game-based smartphone app to assess pain in adolescents with cancer [24]. The game-based nature and built-in reward system was appealing to users and promoted high adherence rates with a mean of 81% adolescents completing 100% of the tasks within a 2-week period [24].

In this study, we aimed to develop and evaluate a smartphone app to monitor nausea severity in children from 0 to 18 years of age undergoing chemotherapy. We used a user-centered design process where we engaged children who were undergoing chemotherapy and their parents during the design, development, and evaluation of the app. The evaluation process included an investigation of the usability of the app and of adherence to the app’s use.

This study was conducted in Copenhagen, Denmark and was part of a prospective observational study. The study was approved by the Danish Data Protection Agency (RH-2016-219, I-Suite: 04804) and the Danish National Research Ethics Committee (H-18020267). The app was approved by the regulatory authorities in Center for It, Medico og Telefoni in the Capital Region of Denmark.

We used a user-centered design process and the evolutionary prototype model to develop and evaluate the app [26,27], and we involved pediatric cancer patients and their parents throughout the process (Figure 1 and Figure 2). Medical researchers and software engineers worked synergistically in the development of the app. Pediatric oncologists were consulted throughout the process because they have an explicit understanding of the users (pediatric cancer patients). The app was developed in 4 phases: (1) The context of use was specified (ie, the users, the tasks that the users would perform, and the environment in which the users would use the app were characterized), (2) requirements were specified (ie, requirements and user goals that must be met for the app to be successful were identified), (3) design solutions were produced (ie, an app prototype was developed), and (4) the app was evaluated (ie, for data completeness and to further refine the app).

Researchers including pediatric oncologists (n=4), a clinical pharmacologist (n=1), and a medical researcher (n=1) participated in the first discussions. The advantages of developing an app and its content, as well as strategies to obtain complete data sets were discussed. The discussion was based on previous experience with pediatric antiemetic trials and a literature review that was used to identify the best validated tool for pediatric patient-reported nausea severity.

The requirements were discussed with 2 software engineers from ZiteLab in Copenhagen, Denmark [28]. A requirement was that the app should include a customized notification system to ensure high response rates. We aimed for a user-friendly and intuitive app, and the usability was refined during several rounds of patient feedback and modification (phases 3a and 3b).

Following the development of a fully functional app, we included patients to test usability of the app and adherence to its use. Patients (n=20) were recruited at the Department of Pediatric Oncology, Copenhagen University Hospital Rigshospitalet during March 2019 and April 2019. All parents provided written informed consent for their child to participate in the study. Eligible patients were children aged 0 to 18 years who would be receiving at least 3 cycles of chemotherapy. The child and at least one parent was required to be able to communicate fluently in Danish. At least one parent needed to have an Android or iPhone smartphone. Parents of children aged 0 to 3 years needed to be able to describe the nausea severity experienced by their child.

One or both parents downloaded and installed the app from Google Play or App Store and were given unique log-in information. It was emphasized that the data collected were to be used for research purposes, and that the parents should also inform their pediatric oncologist or nurse about their child’s symptoms. The questions in the app were delivered to each child by AE who also taught the parents to administer the app. The children (with assistance from their parents) were asked to record their own nausea severity in the app, and AE noted if they were able to naturally navigate through the app’s features. The parents could then assist the child to enter data about nausea severity for the day before and for the first 4 days after the start of chemotherapy during 3 chemotherapy cycles. The children and their parents were contacted in person before each cycle to promote adherence to use of the app. A semistructured interview was conducted at that time, and the participants were asked what they liked or disliked about the app, if the app was easy to use, and if the time spent on data entry was fair. AE provided continuous feedback to MKA who modified the app according to the feedback. This was continued until there were no further recommendations for changes in the app.

The web-based administrative portal was fully developed in this phase. AE tested if log-ins for new patients could be created and if the portal provided a user-friendly interface and overview of data completeness. MKA tested whether the back end received the entered data accurately and if data could be downloaded for further analysis.

We subsequently interviewed parents of patients aged 0 to 3 years in order to determine if the parents could grade the nausea severity experienced by their child.

A paired, two-tailed t test was used to determine if the response rates differed between morning and evening and if the response rates differed between chemotherapy cycles. P<.05 was deemed significant.

During phases 1 and 2, we specified in which context the app would be used, and we agreed on the overall requirements of the app (Figure 1 and Figure 2). Based on the functions and requirements that were described by the researchers, in phase 3, a beta-version smartphone app was created by software engineers from ZiteLab in Copenhagen, Denmark [28]. These functions and requirements were refined and modified according to feedback from children who were pediatric cancer patients, parents, and participants (phase 3a and phase 3b).

The demographic and disease characteristics of the children who were pediatric cancer patients (phases 3a and 4) and the demographic characteristics of clinical researchers (phase 3b) who participated in the user-centered testing are shown in Table 1.

The questions in the PeNAT were clear and easy to understand, and the patients and their parents had no comments relating to the meaning of words or phrases in the translation. The parents of one newly diagnosed patient (less than one week) reported that it would be demanding to enter data, which was in contrast with the feedback from parents of patients who had been through several rounds of chemotherapy, who expressed that they could easily overcome to enter data in the app.

One suggestion for the interface was to include emojis as pictorial expressions of the questions. Emojis were not included in the PeNAT part of the app to be true to the original PeNAT paper version.

Through interviews with parents, we found that the actual antiemetics that were given did not always correspond with the prescribed medications in the medical record, especially when the child was discharged from hospital. To capture the actual medication use, a medicine module was developed to let parents fill in administered antiemetics, numbers of administrations, and whether the patient used antiemetic rescue medications on any given day. The medicine module included an autofill function where the parents could copy answers from the previous day with a single tap and modify answers that had changed.

During the first test period, problems with the log-in function required that participants restarted the app. All test participants received push notifications informing them to enter data in the app, and they were subsequently able to do so.

The app was evaluated prospectively in patients (n=20) during a total of 60 chemotherapy cycles.

We described the process of developing and evaluating a smartphone app to collect pediatric patient-reported outcomes of nausea severity and episodes of vomiting and retching. Patients were able to self-report in the app the severity of nausea that they experienced with assistance from their parents, and clinicians and researchers had a good historical overview of their symptoms in the web-based administrative portal. Multidisciplinary group discussions ensured that the app and the web-based administrative portal met both medical and technical demands. It was essential that the app had a high usability, so patients and their parents were involved early in the development to get insight into their preferences.

Generally, patients prefer mobile data collection over more traditional methods such as paper diaries, because it is easier and faster to record data into a mobile device than it is to record in a paper diary [31]. Some children thought it was interesting to answer questions in the app because they liked to press the faces reflecting nausea severity. They easily understood the interactive component because the child’s own term for nausea was incorporated into the questions. These features are not possible in a paper diary or a web-based survey.

Nausea has a higher incidence than vomiting during administration of chemotherapy, but it is a challenge to recognize subjective symptoms such as nausea in children. Furthermore, children under 4 years of age are often excluded from studies that investigate the burden of chemotherapy-induced nausea and vomiting in children [32-34]. The target users of this app are children aged 0 to 18 years (and their parents) undergoing cancer treatment. Our subjective impression was that children aged 4 to 18 years were able to express their nausea severity in the app. One 17-year-old patient expressed (about nausea) that “it hurts in my stomach and I have no appetite,” whereas a 7-year-old patient expressed that nausea was “like an elevator moving up through the stomach and all the way to the throat.” Children aged 0 to 3 years are too young to complete a patient-reported outcome instrument, and the PeNAT is not validated for this age group; however, parents of patients aged 0 to 3 years expressed that they could easily grade the nausea severity experienced by their child. By including this age group, researchers were able to obtain data about parent-reported nausea severity and vomiting episodes for children aged 0 to 3 years.

It was important that the patients adhered to the use of the app, including patients with severe nausea who likely had reduced energy to perform extra tasks during chemotherapy. In a paper version of the PeNAT, 17% of the patients did not use the questionnaire for three or more days out of seven days [32]. The high response rates in our study were attributed to the individual patient reminders. Also, data were captured in real time, and we could identify and respond to nonadherence early by following the patient responses in the web-based administrative portal. The reasons for nonadherence were mainly technical difficulty–based (mainly in the first version of the app) and not that the tasks were too demanding. The response rates did not decrease over three chemotherapy cycles; therefore, the app has the advantage that it could be used in antiemetic trials using a crossover design, where patients undergo two chemotherapy cycles.

Real time patient-reported outcomes reduce the risk of recall bias and improve data quality. After requests from patients and parents, patients were allowed to return to answer questions from within the previous 2 days. This allowed patients with severe symptoms on one day to respond the following day. The survey time was recorded in the administration system which permitted analysis of data accuracy and the ability to exclude data that was not registered in real time. This is in contrast to a paper-based approach, where the researchers are not aware of whether patients are back filling or completing the paper diary just prior to returning it to the researchers [24].

The digital revolution offers exciting new tools to support research, but along with these new opportunities come some challenges. We conducted a preliminary assessment of the suitability of apps in the target population, and every participant in this study had a smartphone. Other target populations may not have access to digital technology, and the target population should, therefore, be assessed to determine whether digital data collection is appropriate.

The development of an app is time consuming compared to the time required to develop a paper diary or a simple electronic web-based survey. Simple electronic diaries for use on smartphones can be created at online research platforms without extensive programming skills [35,36]. More advanced options, such as individual patient reminders and interactive components, require assistance from experienced software engineers. Also, data storage in these platforms may not adhere to general data protection regulations. The cost of developing an app depends on the complexity of the interface and functions, and additional costs must be considered for data storage and distribution in Google Play and App Store; however, electronic data collection reduces the time used for data handling because researchers do not need to manually record data from paper diaries [31]. This more precise and efficient data collection method could also outweigh the time and price spent on preparing the trial [16-18].

Technical support needs to be available to maintain and upgrade the app over time in clinical trials using apps for data collection. Technical problems mainly arose when the software was updated or when a new version of the app was launched. The major technical problems were that patients could not log-in to the app and that push notifications were not sent. Other barriers that could prevent efficient data entry are limited access to internet or if a patient signs out and forgets the username and password. These problems require that investigators of the clinical trial pay attention to both the function of the app and to feedback from patients throughout the clinical trial so that technical problems can be solved immediately. In our study, one researcher was signed into the app with a test user log-in to capture technical difficulties which were then directly forwarded to a software engineer. A researcher also directly transcribed patient-reported outcomes from the patients at any time they were not able to report patient-reported outcomes in the app.

The data entered by the patients will be used to determine patient-related risk factors of chemotherapy-induced nausea and vomiting, and the app will be used to collect pediatric patient-reported outcomes in upcoming antiemetic trials. The overview of the patient-reported outcomes can be used in the future by clinicians to better refine antiemetic treatment for forthcoming chemotherapy.

The app and the web-based administrative portal demonstrated good usability for patients and researchers, and pediatric patient-reported outcomes of nausea severity were collected efficiently. The app can be used to facilitate future research regarding antiemetic efficacy in pediatric cancer patients and to better refine antiemetic treatment in upcoming chemotherapy cycles.