RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 20.02.18 in Vol 6, No 2 (2018): February

Preprints (earlier versions) of this paper are available at, first published Nov 24, 2017.

This paper is in the following e-collection/theme issue:

    Original Paper

    Mobile App Delivery of the EORTC QLQ-C30 Questionnaire to Assess Health-Related Quality of Life in Oncological Patients: Usability Study

    1Department of Radiation Oncology, Technical University of Munich, Munich, Germany

    2Institute for Innovative Radiotherapy, Helmholtz Zentrum München, Neuherberg, Germany

    Corresponding Author:

    Kerstin A Kessel, Dr Sc Hum

    Department of Radiation Oncology

    Technical University of Munich

    Ismaninger Straße 22

    Munich, 81675


    Phone: 49 0894140 ext 4502

    Fax:49 08941404477



    Background: Mobile apps are evolving in the medical field. However, ongoing discussions have questioned whether such apps are really valuable and whether patients will accept their use in day-to-day clinical life. Therefore, we initiated a usability study in our department.

    Objective: We present our results of the first app prototype and patient testing of health-related quality of life (HRQoL) assessment in oncological patients.

    Methods: We developed an app prototype for the iOS operating system within eight months in three phases: conception, initial development, and pilot testing. For the HRQoL assessment, we chose to implement only the European Organization for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire-Core 30 (QLQ-C30; German version 3). Usability testing was conducted for three months. Participation was voluntary and pseudonymized. After completion of the QLQ-C30 questionnaire using iPads provided by our department, we performed a short survey with 10 questions. This survey inquired about patients’ opinions regarding general aspects, including technical advances in medicine, mobile and app assistance during cancer treatment, and the app-specific functions (eg, interface and navigation).

    Results: After logging into the app, the user can choose between starting a questionnaire, reviewing answers (administrators only), and logging out. The questionnaire is displayed with the same information, questions, and answers as on the original QLQ-C30 sheet. No alterations in wording were made. Usability was tested with 81 patients; median age was 55 years. The median time for completing the HRQoL questionnaire on the iPad was 4.0 minutes. Of all participants, 84% (68/81) owned a mobile device. Similarly, 84% (68/81) of participants would prefer a mobile version of the HRQoL questionnaire instead of a paper-based version. Using the app in daily life during and after cancer treatment would be supported by 83% (67/81) of participants. In the prototype version of the app, data were stored on the device; in the future, 79% (64/81) of the patients would agree to transfer data via the Internet.

    Conclusions: Our usability test showed good results regarding attractiveness, operability, and understandability. Moreover, our results demonstrate a high overall acceptance of mobile apps and telemedicine in oncology. The HRQoL assessment via the app was accepted thoroughly by patients, and individuals are keen to use it in clinical routines, while data privacy and security must be ensured.

    JMIR Mhealth Uhealth 2018;6(2):e45




    Since the first smartphone was introduced by IBM in 1995 [1], the development of cell phones and mobile apps has been world-changing. The success of medical and health apps (labeled under mobile health [mHealth] or electronic health [eHealth]) is undeniable, with 165,000 programs in the respective leading app stores (Apple and Google) [2]. Apps for registering heart rate, blood pressure, and blood glucose are forthcoming, and apps for depression, body weight reduction, and diabetes are widely accepted. However, most of the apps are not scientifically validated [3-5]. To date, only a few apps in an oncological context exist, which allow for quality of life (QoL) assessment.

    An essential key to everybody’s well-being is QoL, which the World Health Organization defined in 1946 as an, “individual's perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards and concerns” [6]. This broad definition includes a holistic approach to QoL, which is rarely used in medical research. In the 1980s the concept of health-related quality of life (HRQoL) evolved [7]. HRQoL is a critical criterion in the therapeutic decision-making process undertaken by health care professionals (HCPs) who are torn between the patients’ well-being, outcome, and economic considerations. Especially in oncology, where treatment is often only life-prolonging and not curative, HRQoL assessment is crucial to identify therapeutic benefit and need. The European Organization for Research and Treatment of Cancer (EORTC) developed a wide range of standardized and validated questionnaires to assess HRQoL in oncological patients. In this study we used the EORTC Quality of Life Questionnaire-Core 30 (QLQ-C30) [8], which is the core questionnaire that can be extended by cancer-specific modules.

    In earlier series, we asked HCPs [9] and patients [10] about their attitudes regarding telemedicine and mobile apps in oncology. A total of 84.3% of HCPs supported the idea of an oncological app, and 97.8% found the HRQoL assessment very useful or useful. Approximately half of the patients asked were willing to use an oncological app; 75.3% of those patients were keen to send HRQoL data. In general, younger patients were more in favor of such an app (P=.032, r=-0.12).

    Our previous results show an existing demand for oncological apps. Hence, we developed an app for HRQoL assessment in oncological patients. This work aims to present our results of the first prototype, the design/development, and patient testing.


    For the first version of the app, we decided to develop the prototype for the iOS operating system (version 8 and later) optimized for iPads (generation 2 and later), but the app also worked on iPhones (generation 4s and later). Further operating systems and device options were scheduled after the first patient testing and validation results were obtained. The development was completed within eight months in three phases: conception, initial development, and pilot testing.

    App Design and Development

    Considering our previously published work on app use during cancer treatment [9-11], we specified primary functional requirements for the prototype listed in Table 1 with the following four main characteristics:

    1. Clear interface and user-friendly design
    2. The prototype should implement only the EORTC QLQ-C30 questionnaire (German version 3; Multimedia Appendix 1 [German], Multimedia Appendix 2 [English])
    3. Data is locally stored on the device; no Internet connection or online data transfer were intended in this version
    4. Login information (user identification [ID] and password) will be generated and provided by the department; this method ensured pseudonymization

    The app was developed using Xcode 7.0 and Swift 2.2.

    Table 1. Functional requirements for the prototype app.
    View this table


    As an empirical usability evaluation method, we used a questionnaire. The survey was conducted for three months at the Department of Radiation Oncology, Klinikum rechts der Isar, Munich, Germany. Participation was voluntary and pseudonymized. Inclusion criteria for participation were: age older than 18 years, German-speaking, and being physically and mentally able to fill out a structured questionnaire on a mobile device. Research assistants supervised the patients during app use and while completing the questionnaire. The Ethics Committee of the Technical University of Munich approved the nature and content of the study with the project number 321/16 S.

    After completion of the QLQ-C30 questionnaire using iPads provided by our department, we conducted the short usability survey with 10 questions (Multimedia Appendix 3). This survey inquired about patients’ opinions regarding general aspects, including technical advances in medicine, mobile and app assistance during cancer treatment, and the app-specific functions (eg, attractiveness, operability, and understandability). Statistical calculations were performed using SPSS Statistics v23 (IBM, Armonk, NY, USA) in a primarily descriptive way.


    App Design and Development

    After launching the app, the login page appears (Figure 1, left). If accessing the app for first time, a user ID and password need to be registered on the device (Figure 1, right). After login, the start page appears (Figure 2, left). We decided to use a slide bar for the menu, as it is easily extendable for future additional functions of the app. Users can click the button in the top left corner or swipe left on the screen to open the menu. For the prototype, the functions for starting a questionnaire, reviewing answers (administrator only), and logging out were implemented (Figure 2, right).

    Figure 3 (left) shows the start page of the EORTC QLQ-C30 questionnaire. We displayed the same information that is written on the paper-based version of the questionnaire. Likewise, all 30 questions were copied from the original QLQ-C30 sheet. No alterations in wording or answers were made. Figure 3 (right) shows the first question. On the bottom of each page, the user can go back to the previous question or skip the current question. After the last question, the user is asked to save the completed questionnaire. These answers can be reviewed by the administrator (Multimedia Appendix 4).

    Figure 1. Screenshot of the login (left) and registration (right) page.
    View this figure
    Figure 2. Screenshot of the start page (left) and slide bar of the menu (right). Administrator view is displayed.
    View this figure
    Figure 3. Screenshot of the start page of the European Organization for Researchand Treatment of Cancer Quality of Life Questionnaire-Core 30 (EORTC QLQ-C30; left) and the first question (right).
    View this figure
    Figure 4. Patient opinion about mobile technologies in medicine (scale from 1 = “I like the development” to 6 = “I don’t like the development”).
    View this figure
    Figure 5. Patient opinion about the app design in terms of color, text and button size, content, and the amount of content per page (scale from 1 = “I like it” to 6 = “I don’t like it”).
    View this figure


    The usability of the app was tested with 81 patients (44 male, 37 female); median age was 55 years (range 21-80 years). When using the app, older patients (>60 years old) distinguished themselves from younger patients (<60 years old). The research assistants observed that while the elderly users had to ask questions about where to click and swipe on the screen, the younger users intuitively knew how to navigate through the app.

    The median time for completing the HRQoL questionnaire on the iPad was 4.0 minutes (range 1.5-8.0 minutes). Of all participants, 84% (68/81) owned a mobile device. Similarly, 84% (68/81) would prefer the mobile version of the HRQoL questionnaire instead of the paper-based version that they usually get during a clinical visit. Using the app in daily life during and after cancer treatment would be supported by 83% (67/81) of participants, which corresponds with their general opinion about mobile technologies in medicine (described in Figure 4). Patients were satisfied with the current development and the introduction of the app into clinical life. In the prototype version of the app, data were stored on the device; in the future, 79% (64/81) of the patients would agree to transfer data via the Internet.

    The operability and navigation of the app were rated as intuitive by 95% (77/81) and 93% (75/81) of participants, respectively. Six participants stated opportunities for improvements regarding the HRQoL questions. However, these cannot be changed as they follow a standardized structure and wording that has been established by the EORTC. Regarding the attractiveness, two patients wished for a setting option to change the font size; five had difficulties finding the menu at first. Figure 5 shows the patients’ opinions on the app interface and design.


    The long-term aim of app-assisted care is to generate higher treatment quality and HRQoL for our patients. In this study, we present our first app-prototype for regular HRQoL assessment, along with the patient usability testing. Most of the participants (84%, 68/81) owned a mobile device and therefore were appropriately equipped to use an app. Similarly, 84% (68/81) of participants would prefer a mobile version of HRQoL assessment instead of a paper-based version, while 83% (67/81) would use it in daily life. Such an app would increase clinical efficiency by reducing paperwork and costs, and enhance patient empowerment. Our results show that mobile technologies are widely accepted; 96% (78/81) of the patients scale their opinion about telemedicine as positive (scale 1-3; Figure 4).

    The users in this study were cancer patients, and the concept of the prototype was to implement a clean and simple app, with the primary goal to be user-friendly. The oncological patient population includes individuals of young and old age. While implementing apps in clinical routine, age and technical skills always seem to be a problem. In our study, patients >60 years old needed more assistance than younger patients. However, Smith et al showed a growing trend of smartphone use amongst elderly people (>65 years) in the United States, with 18% in 2014 [12] and 27% in 2015 [13]. Older individuals incrementally adapt to mobile technology. Nevertheless, it is essential to develop apps for older patients with a focus on operability and understandability. Intensive preuse teaching might be necessary in some cases.

    Although we recently showed a higher usage of Android phones (52.9%) than iOS phones (37.2%) in our patient population [10], we rationally decided to develop the first version of the app for iPhones and iPads. iOS development is cheaper [14] and more comfortable, as apps only need to meet the criteria of two model types. However, a version of our app for Android is planned in the near future.

    The interface was designed in light, cool colors and we omitted any animation or gamification. Overall, we used hues of the blue and green family, which tend to psychosocially calm down the already anxious cancer patients [15]. We chose to use dark characters on white background. Piepenbrock et al [16] showed an advantage in positive display polarity for young and old adults. Font size 12 was chosen to be readable by older patients with visual impairment. Darroch et al [17] showed that font size does not affect reading accuracy in both young and old adults, but young adults preferred size 8 and 10 while older adults preferred slightly bigger sizes of 8 to 12. Darroch et al recommend offering a setting option for font size, which was requested by two of our patients.

    Button size for the questionnaire was chosen to be 20 pixels high. Anthony [18] recommends a target width of 45-57 pixels wide to allow the user’s finger to fit in the target while the edges are visible when tapping. We chose the width to be the full size of the screen, as our app needs to be comfortable for older patients with visual impairment and lower touch accuracy. After pretesting, we decided to implement a button for skipping questions, as patients can do the same in the paper-based version. Patients who do not want to (or cannot) answer one of the questions can still complete the survey.

    Overall, patients were in favor of the presented app design (Figure 5); operability (95%) and navigation (93%) were rated excellent. These factors play a critical role in patient compliance, as poor user-friendliness will automatically lead to reduced usage. Completion time of the paper-based EORTC QLQ-C30 questionnaire was reported by Aaronson et al [8] as approximately 11 minutes, while our mobile version only took median 4 minutes. Both patient populations were different (median age 55 vs 63 years, international multicenter study vs in-house study) [8]. However, our study shows that patients with cancer disease are compliant with an app-based HRQoL assessment, and it is an efficient method.

    To date, data are only stored locally on the device and can be transferred via a cable. In the future, patient data will be sent online, and the treating physicians will have access to the data (eg, at aftercare appointments). Here, it is crucial to guarantee data protection and security. We showed previously that 85.2% patients marked pseudonymization and data security as very important/important [10]. As health data is always highly sensitive, data security was the most crucial requirement for using an oncological app [11,19].

    The first version of the app only contains the EORTC QLQ-C30 questionnaire. Future functions are planned and in progress. A multilingual approach (English, French, Italian, Russian, Arabic) is necessary, as our department is frequented by many international patients, and furthermore German is not the mother tongue of some local patients (eg, refugees). Moreover, cancer-specific modules such as the EORTC QLQ-BR23 [20] for breast cancer or QLQ-PR25 [21] for prostate cancer will be implemented next. HRQoL data from cancer patients can be used to adjust the individuals’ treatment or offer supportive therapy. Surveys on therapy satisfaction will help to improve the departments’ workflow and the patients’ contentment.

    Another function of the app will be the documentation of treatment-related side effects and symptoms. The course of a disease will be monitored and chronicled. This feature can also be used to register study parameters in app-assisted randomized controlled trials (smartRCTs). SmartRCTs can reduce study duration, costs, and subject bias, as well as collect a broader range of data [11]. Certainly, a workflow regarding how to check patient answers regularly and how to handle severe entries by the patients must be developed. However, we could recently show that 94.3% of HCPs were willing to contact the patient [9], which may lead to a quicker detection of progress, as recently stated by Denis et al [22,23]. This group demonstrated that using a Web-based app resulted in a significant improvement in overall survival (12 months vs 19 months) in patients with high-risk lung cancer, and relapses were detected five weeks earlier that the control group [23].

    Our usability test showed good results for the presented app. Moreover, our results demonstrate a high overall acceptance of mobile apps and telemedicine in oncology, which is in line with our previous results [9,10]. The HRQoL assessment via the app was accepted thoroughly by patients, and individuals are keen to use it in clinical routine, while data privacy and security must be ensured. Digital medicine (medicine 4.0) is an unstoppable trend and will play a significant role in the future of clinical health care.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    QLQ-C30 questionnaire paper based version (German).

    PDF File (Adobe PDF File), 141KB

    Multimedia Appendix 2

    QLQ-C30 questionnaire paper based version (English).

    PDF File (Adobe PDF File), 20KB

    Multimedia Appendix 3

    Patient usability survey (German).

    PDF File (Adobe PDF File), 303KB

    Multimedia Appendix 4

    Screenshot of the review screen with completed questionnaires (administrator view).

    PNG File, 399KB


    1. Sager I. Bloomberg. 2012 Jun 29. Before IPhone and Android came Simon, the first smartphone   URL: https:/​/www.​​news/​articles/​2012-06-29/​before-iphone-and-android-came-simon-the-first-smartphone [accessed 2017-11-21] [WebCite Cache]
    2. The Economist. 2016 Mar 10. Things are looking app. Mobile health apps are becoming more capable and potentially rather useful   URL: https:/​/www.​​news/​business/​21694523-mobile-health-apps-are-becoming-more-capable-and-potentially-rather-useful-things-are-looking [accessed 2017-11-21] [WebCite Cache]
    3. Brouard B, Bardo P, Bonnet C, Mounier N, Vignot M, Vignot S. Mobile applications in oncology: is it possible for patients and healthcare professionals to easily identify relevant tools? Ann Med 2016 Nov 27;48(7):509-515. [CrossRef] [Medline]
    4. Collado-Borrell R, Escudero-Vilaplana V, Ribed-Sánchez A, Ibáñez-García S, Herranz-Alonso A, Sanjurjo-Sáez M. Smartphone applications for cancer patients; what we know about them? Farm Hosp 2016 Jan 01;40(1):25-35 [FREE Full text] [CrossRef] [Medline]
    5. de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012;12:CD007459. [CrossRef] [Medline]
    6. WHO: Constitution of the World Health Organization. In: WHO: Basic Documents Forty-Seventh Edition. Geneva: World Health Organization; 2009:1.
    7. National Center for Chronic Disease Prevention and Health Promotion. Division of Population Health, Prevention. 2016 May 23. Health-Related Quality of Life (HRQOL)   URL: [accessed 2017-11-21] [WebCite Cache]
    8. Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993 Mar 3;85(5):365-376. [Medline]
    9. Kessel KA, Vogel MM, Schmidt-Graf F, Combs SE. Mobile apps in oncology: a survey on health care professionals' attitude toward telemedicine, mHealth, and oncological apps. J Med Internet Res 2016 Nov 24;18(11):e312 [FREE Full text] [CrossRef] [Medline]
    10. Kessel KA, Vogel MM, Kessel C, Bier H, Biedermann T, Friess H, et al. Mobile health in oncology: a patient survey about app-assisted cancer care. JMIR Mhealth Uhealth 2017 Jun 14;5(6):e81 [FREE Full text] [CrossRef] [Medline]
    11. Vogel MME, Combs SE, Kessel KA. mHealth and application technology supporting clinical trials: today's limitations and future perspective of smartRCTs. Front Oncol 2017;7:37 [FREE Full text] [CrossRef] [Medline]
    12. Smith A. Pew Research Center. 2014 Apr 03. Older adults and technology use   URL: [WebCite Cache]
    13. Smith A. Pew Research Center, Internet & Technology. 2015 Apr 1. U.S. smartphone use in 2015   URL: [WebCite Cache]
    14. Car T. Infinum. 2015 Oct 27. Android development is 30% more expensive than iOS. And we have the numbers to prove it!   URL: [accessed 2017-11-21] [WebCite Cache]
    15. Cao J. The Next Web. 2015 Apr 7. Web design color theory: how to create the right emotions with color in web design   URL: [WebCite Cache]
    16. Piepenbrock C, Mayr S, Mund I, Buchner A. Positive display polarity is advantageous for both younger and older adults. Ergonomics 2013;56(7):1116-1124. [CrossRef] [Medline]
    17. Darroch I, Goodman J, Brewster S, Gray P. The effect of age and font size on reading text on handheld computers. In: Human-Computer Interaction - INTERACT 2005. Berlin, Heidelberg: Springer; 2005 Presented at: INTERACT 2005; September 12-16, 2005; Rome p. 253-266   URL: [CrossRef]
    18. Anthony T. Smashing Magazine. 2012 Feb 21. Finger-friendly design: ideal mobile touchscreen target sizes   URL: https:/​/www.​​2012/​02/​finger-friendly-design-ideal-mobile-touchscreen-target-sizes/​ [accessed 2017-11-21] [WebCite Cache]
    19. Rumbold JMM, Pierscionek B. The effect of the general data protection regulation on medical research. J Med Internet Res 2017 Feb 24;19(2):e47 [FREE Full text] [CrossRef] [Medline]
    20. Sprangers MA, Groenvold M, Arraras JI, Franklin J, te Velde A, Muller M, et al. The European Organization for Research and Treatment of Cancer breast cancer-specific quality-of-life questionnaire module: first results from a three-country field study. J Clin Oncol 1996 Oct;14(10):2756-2768. [CrossRef] [Medline]
    21. van Andel G, Bottomley A, Fosså SD, Efficace F, Coens C, Guerif S, et al. An international field study of the EORTC QLQ-PR25: a questionnaire for assessing the health-related quality of life of patients with prostate cancer. Eur J Cancer 2008 Nov;44(16):2418-2424. [CrossRef] [Medline]
    22. Denis F, Lethrosne C, Pourel N, Molinier O, Pointreau Y, Domont J, et al. Overall survival in patients with lung cancer using a web-application-guided follow-up compared to standard modalities: Results of phase III randomized trial. J Clin Oncol 2016 Jun 20;34(18_suppl):LBA9006-LBA9006 [FREE Full text] [CrossRef]
    23. Denis F, Viger L, Charron A, Voog E, Dupuis O, Pointreau Y, et al. Detection of lung cancer relapse using self-reported symptoms transmitted via an internet web-application: pilot study of the sentinel follow-up. Support Care Cancer 2014 Jun;22(6):1467-1473. [CrossRef] [Medline]


    eHealth: electronic health
    EORTC: European Organization for Research and Treatment of Cancer
    HCP: health care professional
    HRQoL: health-related quality of life
    ID: identification
    mHealth: mobile health
    QLQ-C30: Quality of Life Questionnaire-Core 30
    QoL: quality of life
    smartRCT: smart randomized controlled trial

    Edited by G Eysenbach; submitted 24.11.17; peer-reviewed by R Collado-Borrell, J Calero; comments to author 21.12.17; revised version received 27.12.17; accepted 02.01.18; published 20.02.18

    ©Kerstin A Kessel, Marco M E Vogel, Anna Alles, Sophie Dobiasch, Hanna Fischer, Stephanie E Combs. Originally published in JMIR Mhealth and Uhealth (, 20.02.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.