Published on in Vol 6, No 4 (2018): April

Preprints (earlier versions) of this paper are available at, first published .
Perceptions of Adolescents With Cancer Related to a Pain Management App and Its Evaluation: Qualitative Study Nested Within a Multicenter Pilot Feasibility Study

Perceptions of Adolescents With Cancer Related to a Pain Management App and Its Evaluation: Qualitative Study Nested Within a Multicenter Pilot Feasibility Study

Perceptions of Adolescents With Cancer Related to a Pain Management App and Its Evaluation: Qualitative Study Nested Within a Multicenter Pilot Feasibility Study

Original Paper

1School of Nursing, Faculty of Health Sciences, University of Ottawa, Ottawa, ON, Canada

2Evidence-to-Practice Program, Children's Hospital of Eastern Ontario Research Institute, Ottawa, ON, Canada

3Child Health Evaluative Sciences Program, Hospital for Sick Children, Toronto, ON, Canada

4Lawrence S Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON, Canada

5Division of Hematology/Oncology, Hospital for Sick Children, Toronto, ON, Canada

6Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

7eHealth Innovation, University Health Network, Toronto, ON, Canada

8Division of Hematology/Oncology, Children's Hospital of Eastern Ontario, Ottawa, ON, Canada

9Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada

10Think Research Corporation, Toronto, ON, Canada

11Department of Anesthesia and Pain Medicine, Hospital for Sick Children, Toronto, ON, Canada

Corresponding Author:

Lindsay A Jibb, RN, PhD

School of Nursing

Faculty of Health Sciences

University of Ottawa

Roger Guindon Hall

451 Smyth Rd

Ottawa, ON, K1H8M


Phone: 1 613 562 5800 ext 4253

Fax:1 613 562 5443


Background: Pain in adolescents with cancer is common and negatively impacts health-related quality of life. The Pain Squad+ smartphone app, capable of providing adolescents with real-time pain management support, was developed to enhance pain management using a phased approach (ie, systematic review, consensus conference and vetting, iterative usability testing cycles). A 28-day Pain Squad+ pilot was conducted with 40 adolescents with cancer to evaluate the feasibility of implementing the app in a future clinical trial and to obtain estimates of treatment effect.

Objective: The objective of our nested qualitative study was to elucidate the perceptions of adolescents with cancer to determine the acceptability and perceived helpfulness of Pain Squad+, suggestions for app improvement, and satisfaction with the pilot study protocol.

Methods: Post pilot study participation, telephone-based, semistructured, and audio-recorded exit interviews were conducted with 20 adolescents with cancer (12-18 years). All interviews were transcribed and independently coded by 2 study team members. Content analysis was conducted to identify data categories and overarching themes.

Results: Five major themes comprising multiple categories and codes emerged. These themes focused on the acceptability of the intervention, acceptability of the study, the perceived active ingredients of the intervention, the suitability of the intervention to adolescents’ lives, and recommendations for intervention improvement.

Conclusions: Overall, Pain Squad+ and the pilot study protocol were acceptable to adolescents with cancer. Suggestions for intervention and study improvements will be incorporated into the design of a future randomized clinical trial (RCT) aimed at assessing the effectiveness of Pain Squad+ on adolescents with cancer health outcomes.

JMIR Mhealth Uhealth 2018;6(4):e80




Up to 96% of adolescents with cancer experience pain related to the disease or associated invasive procedures and treatment [1]. Pain negatively impacts adolescent health-related quality of life (HRQoL) [2], impedes cancer recovery [3], interferes with activities of daily living [4], and results in long-term morbidity [5,6]. Pain may also represent a significant cost burden to the health care system and families [7], with pain being the most common reason for adults with cancer to utilize emergency health services [8-11]. Treatment advancements and health care system transformations have resulted in adolescents with cancer receiving much of their care on an outpatient basis [12]. This change may mean that a significant proportion of adolescent cancer symptoms (including pain) are experienced in an environment where management options may be suboptimally applied (eg, at home). In addition, adolescents with cancer have emerged as a childhood cancer subgroup with unique developmental characteristics and a unique cancer epidemiology [13-15]. These characteristics suggest that the cancer pain experience and appropriate management techniques may be different for adolescents when compared with younger children and adults [15-17].

In response to these issues, our team has developed a smartphone-based app, called Pain Squad+, that is capable of providing adolescents with real-time pain management support (Multimedia Appendix 1) [18,19]. The app integrates automated valid and reliable pain assessments [1] and personalized pain self-management advice with centralized decision support via a pediatric oncology-trained registered nurse to enable adolescents to better manage their pain in real time, regardless of their location (eg, hospital, home, school).

The development of Pain Squad+ has followed the phased approach to designing and evaluating complex interventions outlined by the UK Medical Research Council (MRC) [20]. In particular, we established a pain management evidence base for the intervention using a systematic review and expert consensus conference [21,22]. We then developed a Pain Squad+ software prototype and adapted the software to the needs of adolescents with cancer using iterative cycles of usability testing [18]. Next, the feasibility of implementing Pain Squad+ for a future randomized controlled trial (RCT) and preliminary intervention effect estimates were determined using a multicenter pre-post test pilot study [19]. The pilot study demonstrated a 77% (40/52) accrual rate, a 5% (2/40) withdrawal rate, technical issues to be experienced by 15% (6/40) of participants, 69 ± 38% adherence to pain reporting, and that Pain Squad+ was highly acceptable to adolescents with cancer, based on the quantitative Acceptability E-Scale (AES) [23]. Small to moderate effect sizes showed the potential impact of Pain Squad+ on key clinical outcomes, including pain intensity, HRQoL, and pain interference.


The pilot study used a nested design with a qualitative exit interview conducted with adolescent participants once they completed or withdrew from the study [24]. The nesting of these qualitative interviews within the pilot was informed by the UK MRC framework and was intended as an in-depth process evaluation assessing our capacity to implement the intervention and clarifying mechanisms by which the intervention may work [20]. Our specific study aim was to elucidate the perceptions of adolescents with cancer who participated in these exit interviews as they related to Pain Squad+ acceptability, perceived helpfulness of Pain Squad+, and suggestions for app improvement, as well as satisfaction with the pilot study protocol. The ultimate goal of this research was to use these perceptions to inform changes to the Pain Squad+ app and study protocol before conducting an adequately powered RCT.

Reporting in this paper is in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) tool [25] (Multimedia Appendix 2).

Study Design

Guided by phenomenological inquiry [26], this semistructured interview-based qualitative study was nested within a pilot study. The pilot used a one-group pre-post study design examining the feasibility and preliminarily effectiveness of Pain Squad+ ( NCT02901834). The methods and results of the pilot study have been previously published [19].

Setting and Participants

Forty participants were recruited for the pilot from two hematology/oncology divisions at Canadian pediatric tertiary care centers. Adolescents who met the following inclusion criteria were eligible: 12-18 years old, English-speaking and reading, undergoing cancer treatment, at least 2 months from diagnosis, and having self-reported pain of >3/10 at least once in the week before recruitment. Exclusion criteria were as follows: major comorbid conditions or receiving end-of-life care. Adolescents were primarily being treated on an outpatient basis, although some participants were hospitalized for periods during the study for disease, treatment, or complication-related reasons. We did not record the amount of hospitalized time. Enrollment in the pilot study was the only eligibility criterion for participation in qualitative interviews. To include the perspectives of adolescents who could have potentially varied in terms of their satisfaction with and commitment to the study, adolescents who withdrew from the pilot were eligible for participation in this study. A purposive maximum variation sampling strategy was used to recruit adolescents for the interviews. Specifically, adolescents who varied in terms of age, sex, diagnosis, and pilot-related outcomes (ie, quantitatively rated app acceptability, adherence, and pre- to poststudy change scores on health-related study outcome measures) were selected to participate. The number of adolescents interviewed was determined by the number of interviews needed to reach data saturation [27].


The Pain Squad+ app is an mHealth technology aimed at supporting the management of pain by adolescents with cancer and clinicians. The study team loaned adolescents participating in the pilot an Apple iPhone 6 loaded with the app. Using the app, adolescents completed a 22-item valid and reliable pain assessment [1] twice daily in the morning and evening for 28 days at times they specified. Three audible notifications occurring within a 30-min window signaled adolescents to complete each assessment and if a pain assessment was not completed within this time frame, it was considered missed. Adolescents were also able to complete an 8-item short-form pain assessment anytime between the morning and evening reports if pain was experienced. If an adolescent reported pain on any assessment, they received real-time pain self-management advice from the app according to an evidence-based investigator-developed algorithm. Recommendations could be pharmacological (eg, reminders to adhere to prescribed medications), psychological (eg, distraction techniques), or physical (eg, yoga instruction). In all cases, if pain advice was given, adolescents received a notification to complete another pain assessment in 1 hour and were provided with additional advice as appropriate. In addition to self-management advice from the app, email alerts related to clinically significant pain [28] (ie, 3 consecutive pain reports of ≥3/10) were sent to a study pediatric oncology-trained registered nurse. The nurse then contacted the adolescent and their medical team to discuss the case and initiate provider-driven intervention such as medication changes. To encourage adherence to pain assessment reporting and to using management advice, the Pain Squad+ app was gamified with adolescents playing the role of law-enforcement officers who receiving rewards for engagement with the app.


Ethics approval was obtained from the Research Ethics Boards at both study sites. Consent was obtained from interested and eligible adolescents. Parental consent for research involvement is not required in our jurisdiction. At enrollment, adolescents completed questionnaires on demographic information and comfort with smartphone devices. Disease-related data were collected from the adolescent’s health record. Adolescents also completed prestudy measures for the following preliminary effectiveness outcomes: (1) pain intensity (2) HRQoL, (3) pain interference, and (4) and pain management self-efficacy. Pain intensity was assessed using the Brief Pain Inventory, which assesses current pain and worst, least, and average pain in the preceding week on a 0 to 10 numerical rating scale [29,30]. Health-related quality of life was assessed using the Pediatric Quality of Life Inventory (PedsQL) 4.0, which is a valid and reliable 23-item instrument assessing general physical, emotional, social, and school function [31]. Higher scores represent better HRQoL. Pain interference was assessed using the Patient-Reported Outcomes Measurement Information System Pediatric Pain Interference Short-Form scale (PROMIS PPI-SF), which is a valid 8-item scale assessing the impact of pain on function. Participants scores are standardized and interpreted such that 60 represents high pain interference, 70 represents very high pain interference, 40 represents low pain interference, and 30 represents very low pain interference [4,32]. Pain management self-efficacy was assessed using an investigator-developed question as there were no appropriate psychometrically sound scales to assess this construct at the time of the study.

Adolescents were then asked to use the Pain Squad+ app for the 28-day period. Adherence to Pain Squad+ was defined as 100% when 56 reports (2 reports per day for 28 days) were completed. On day 29, each adolescent who did not formally withdraw participation was asked to complete poststudy outcome measures and the AES. The AES collected data on the degree to which Pain Squad+ was satisfactory to participants in terms of how difficult, helpful, enjoyable, and understandable it was to use, and how tolerable the amount of time required to complete it was. The possible total score range for the AES is 6 to 30, and greater scores indicate higher acceptability.

A semistructured, telephone-based, and audio-recorded interview was conducted with a sample of pilot study adolescents. These adolescents had completed the pilot between 1 day to 2 weeks before the interview. This time frame was selected to help ensure that adolescents could accurately recall the use of the app and participation in the study, while also being flexible to adolescents’ individual schedules. An interview guide based on a version that was previously and successfully used with adolescents with cancer [33] guided interviews, and field notes were taken. All interviews were conducted by one investigator (LJ) who is a pediatric oncology nurse and was familiar with many of the interviewed adolescents before the study. The interviewer was not blinded to participant characteristics or Pain Squad+ outcomes but used a bracketing procedure during interviews and analyses to minimize the impact of subjectivity on findings [34]. No other individuals besides the interviewer and adolescent were present for interviews. Interview topics included app acceptability, app ease of use and understanding, recommendations for app changes, the perceived clinical value of the app, and the acceptability of study involvement. All interviews were audio-recorded.

Data Analysis

A trained transcriptionist transcribed interview audio-recordings verbatim. Data analyses began once the first interview was transcribed, allowing issues identified in early interviews to inform later interviews using constant comparative analyses [35]. The transcribed data were managed using NVivo 10.0 software (QRS International, Australia). Field notes were also included in the analysis. All data were read several times by 2 authors (LJ and VH) to obtain an overall understanding and identify data codes. The authors then independently coded the data using a line-by-line technique based on the study objectives. Using content analysis, codes were grouped into categories based on between-code relationships. Categories were generated until all data were classified under the existing categories [26,36,37]. Categories were then grouped into themes. Quantitative data were integrated into the analysis process to illustrate or clarify qualitative results using a mixed methods matrix approach [38]. It was intended that discrepancies in opinion regarding categories and themes would be resolved using group discussion with a third party; however, no discrepancies occurred. An audit trail consisting of analytical decisions was kept as a means to enhance validity [34].

Participants and Interview Process

All 20 adolescents who were approached to participate in poststudy interviews agreed to do so. Nineteen out of 20 (95%) of these interviewed adolescents had completed the entire pilot study, and 1 out of 20 adolescents (5%) withdrew from participation after completing 5 days of the intervention due to worry about damaging the loaned phone. Multimedia Appendix 3 presents adolescent characteristics, Pain Squad+ adherence, AES scores, and health outcome change scores (from pre- to poststudy). Interview length ranged from 7 to 20 min, and no adolescents had any issues recalling the use of the app or study participation. Interview data were categorized into 5 main themes: acceptability of the intervention, potential active ingredients of the intervention, suitability of the intervention, recommendations for improvement, and acceptability of the study. Each theme comprised several categories and codes as shown in Table 1.

Acceptability of the Intervention

General Impressions

All adolescents enjoyed using Pain Squad+. Adolescents stated the app was “fun” (female, 12 years), “really neat” (female, 15 years), and engaging.

One of the adolescents stated:

It’s really appealing to the eye. The color, the theme is good. The font. And it’s not really that hard to understand. The vocabulary is really straightforward and all of the things on it, like you know, the multiple-choice questions and the [visual analogue scale] sliders are really easy to use. And for cons? Don’t really think I can really think of any.
[Male, 16 years]

Every adolescent endorsed the ease of use of Pain Squad+:

It was really easy. It was very straightforward. It wasn’t really complicated. It was just like simplified so it was easy to use for little kids.
[Male, 16 years]

One adolescent specifically reported that because of her familiarity with smartphone apps, use of Pain Squad+ was not problematic:

I’m used to that stuff so it made it easy.
[Female, 16 years]

Thirteen participants discussed ease of understanding related to the pain assessment questions and pain management advice. For instance, one adolescent stated:

It was really straightforward. I think all the questions were worded well so you could like understand what they were getting at.
[Female, 16 years]

Two participants endorsed the app as efficient to complete:

It was good because it was really fast and easy.
[Male, 14 years]
Specifically Endorsed App Elements

Specific elements of Pain Squad+ that were well-liked or helpful to adolescents were the pain management advice, the design and gamification mechanics (endorsed by all participants), and the ability to use the app in real time and in any environment, especially when at home. Speaking about the pain self-management advice, one participant stated:

I thought the pain help ideas were really awesome. When they suggested like different things that you could do? Those were really helpful. And they had some [pieces of pain management advice] where they would suggest like relaxation and breathing, and how do I do that. And then you click on it and there is someone talking to you, walking you through it. Like how to relax. So that’s helpful because someone can tell you to relax, but you can just be sitting there like, “I don’t know how.”
[Female, 16 years]

Referring to the ability of the app to provide pain management advice at any time and in any environment, adolescents said:

I think overall it was good. Probably one of my favorite parts of it was that you could do the 8 questions in the middle (short-form pain assessment). That helps a lot.
[Female, 12 years]
There was stuff you could try at home and like do yourself. So I liked that.
[Female, 15 years]

Sixteen adolescents discussed challenges they experienced with the app. Twelve participants stated that the 22-item morning and evening pain assessments included too many questions

It was okay and too much. Because sometimes people don’t want to like the same thing over again...22 questions every time.
[Male, 16 years]
Table 1. Post-pilot study interview themes, categories, and codes.
Theme and categoryCodes
Acceptability of the intervention

General impressions
  • Enjoyed use

  • Easy to use
  • Easy to understand
  • Quick to complete

Specifically endorsed app elements
  • Pain management advice
  • Design and gamification mechanics
  • Real-time any-environment reporting

  • Number of assessment questions
  • Notifications
  • Other technical problems
Potential active ingredients of the intervention

Self-management support
  • Engagement in self-management
  • Self-monitoring of pain
  • Patient-provider communication

Study nurse support
  • Value of nurse
  • Timing of nurse support
Suitability of the intervention

Impact on daily activities
  • Perceived burden

Recommended usage
  • Appropriateness when ill
  • Appropriateness when symptom-free
  • Recommended usage
Recommended improvements
  • Additional self-management advice
  • Additional gamified mechanics
  • Tutorials on use
  • Audio-visual assets
  • Review data
Acceptability of the study

General impressions and specific likes
  • General impression
  • Specific likes

Motivation for participation
  • Altruism
  • Novel experience
  • Gamification

  • Study phone

Nine adolescents described challenges related to the frequency of pain assessment notifications. The notification issue appeared to be related to a technical problem with the app software server, where repeated notifications to complete the same pain assessment questionnaire were sent to some study phones, which was “annoying” (Female, 17 years) for adolescents:

Even if I did do my case (pain assessment), it would still just keep on giving notifications. And I know that after you say, “yes” to the case, (the app will) follow-up (on the severity of pain one hour later). But even if I would do the follow-ups, it would just keep on giving more and more (notifications).
[Female, 15 years]

Technical problems were the final app-related challenge cited. One adolescent described an issue likely related to the app being unable to connect to the internet:

Sometimes it was hard for me to do it. Like 2 or 3 times I couldn’t (use the app), because like as soon as I clicked on the app, the screen would just go white.
[Male, 14 years]

Two adolescents reported an issue related to not receiving scheduled reminders to complete pain assessments:

I think it was a problem with my version [of the app] but I wasn’t getting any reminders (to complete pain assessments)...So I set up just a regular alarm on the clock the phone has itself.
[Male, 17 years]

Potential Active Ingredients of the Intervention

Self-Management Support

All participants reported that the ability of Pain Squad+ to support pain self-management was of therapeutic benefit. For example, one participant stated:

Yeah because being like an out-patient, you’re not at the hospital all the time. And you don’t want to call the doctor every time you have something as simple as a stomach ache when you know you got tips from the app to help.
[Female, 14 years]

The ability for adolescents to self-monitor their pain through routine assessments was also considered a valuable self-management feature of the app:

Yeah especially because it really helps me to track my pain and remember everything.
[Male, 14 years]

Improved awareness of pain was also a perceived benefit of the app:

Some of the questions were things that I didn’t really consider when I was thinking about my pain. So (Pain Squad+) helped with (recognizing how pain) affected me and all that.
[Female, 16 years]

Pain Squad+ helped to facilitate adolescents’ communication about pain and pain treatments with their medical team. Highlighting this finding, one adolescent said:

I think when I had a problem at home, like experiencing some kind of pain, after inputting it into the app and coming back to the hospital, talking to my doctor was easier.
[Male, 17 years]
Study Nurse Support

Ten participants discussed interaction with the Pain Squad+ study nurse as being of therapeutic benefit. Most adolescents appreciated knowing that the nurse was a component of the intervention even if they did not require pain management support from the nurse during the study. In particular, these adolescents viewed nurse involvement in Pain Squad+ as an extra layer of pain treatment support in the event that self-management strategies were ineffective:

I liked that when you answer questions if you go higher (in pain intensity rating), a nurse actually calls you and like asks about your pain. It’s actually a good thing because like if you actually have pain and you don’t know what to do, she can help you.
[Male, 15 years]

Three adolescents, however, reported not believing the nurse was essential to the Pain Squad+ intervention and that the recommended self-management strategies were sufficient to support pain treatment:

I had it pretty controlled on my own.
[Female, 16 years]

One adolescent commented that he experienced some difficulty in connecting with the nurse when she attempted to contact him:

The part where you get the advice from the nurse was good but then sometimes I would just miss her if I was out or my phone was on silent. So, it might be better if she left you a (text) message so that you could check what she was telling you to do.
[Male, 14 years]

Suitability of the Intervention

Impact on Daily Activities

Seven participants commented that use of the app fits well with their usual daily activities, for example:

Oh it was good, it was good. It didn’t really take that much time and effort.
[Female, 17 years]
Recommended Usage

Nine participants stated that completing morning and evening pain assessments when not experiencing pain was not appropriate. In particular, one adolescent said:

Yeah, I didn’t really have pain all of the time. So, I think [the number of notifications] just wasn’t for me, but I’m sure for someone who’s going through the pain it’s going to be really helpful.
[Male, 16 years]

When asked about his lack of interaction with Pain Squad+, another adolescent said:

Like it got a little tiring sometimes. Because especially there are days I didn’t feel any problems, like have any problems, and I felt less inclined to actually finish the surveys.
[Male, 17 years]

Recommendations for Improvement

Six adolescents suggested that the app include additional pieces of self-management advice. For example:

I think if there were more ways to help you manage your pain added on, then it would be more helpful.
[Male, 14 years]

These adolescents did not, however, have specific recommendations regarding which pieces of management advice should be added. Four participants recommended enhancements to the gamification mechanics of the app. One adolescent stated:

Um, maybe like something a little more than just like how you “level up” (by completing pain assessments and management advice). Just...more to do with that. A little more interactive kind of even more, fun.
[Female, 14 years]

Three participants recommended adding tutorials to Pain Squad+ on how to use the app. Another two adolescents recommended converting text-based content into an audio- or video-recorded format. For example:

Umm I think probably like uh a few videos from actual professionals, healthcare professionals, like doctors. Or like maybe like, err therapists, like massage therapists. If there was like maybe, for example like a massage therapist showing someone how to relieve pain in a certain area.
[Male, 14 years]

Finally, three adolescents suggested incorporating a capacity to visually review previously logged pain reports in order to track pain treatment progress.

Acceptability of the Study

General Impressions and Specific Likes

Overall, adolescents enjoyed participating in the pilot, stating that the study was “really good” (Male, 15 years) and they “didn’t mind” (Female, 14 years) participating. Specifically, adolescents endorsed the ease of study participation and the ability to use the intervention for a time:

And I don’t know, I really liked...I just liked the app. I liked being able to record my pain, like without just writing it down in like a journal or something you know? I really liked how they gave me suggestions for like what I could do. I just liked it all, overall.
[Female, 16 years]

Two adolescents specifically cited completion of the outcome measures as a well-liked study component, with one participant stating:

I liked the questionnaires at the beginning and the end. Just to like compare sort of how you’re doing before and after.
[Female, 15 years]
Motivation for Participation

Two participants discussed the chance to potentially help other adolescents with cancer as their rationale for study involvement. For example:

I just liked, you know, contributing to the development of this app. It will be a huge help to other little kids going through cancer.
[Female, 16 years]

Another adolescent mentioned the novel experience of being a research participant as her motivation for participation. Finally, 6 participants stated that they were motivated to continue participation because of the gamification mechanics, for instance:

Yeah but it was pretty cool. It made you sort of want to do it more. “Okay I’ve got to do this because I want to get the next level.”
[Female, 16 years]

The use of the study phone loaned to participants was a challenge for 5 adolescents. These participants already owned a smartphone and considered care and use of a second phone to be a burden:

Half the time, I wouldn’t even hear it because it would be in a different room or something and I just totally forget about it. If (Pain Squad+) was on my actual cell phone I probably would have done it more. But it wasn’t on my actual phone that I have on me all the time.
[Female, 14 years]

Principal Findings

Pain is a problematic issue for adolescents with cancer [1,2] that may be mitigated with the rapid self-management support made possible by mHealth interventions [19]. Adolescents with cancer who participated in a pilot of the Pain Squad+ smartphone-based real-time pain self-management app reported that they generally liked the app and considered it helpful. In particular, adolescents reported that the app was easy to use and understand, supported the self-care of pain, and simplified patient-provider communication. The suitability of the app to adolescents’ lives and the acceptability of participating in the pilot were also shown. Finally, challenges related to app use and pilot participation that will inform intervention and study protocol changes were described.

Adolescents with cancer considered the intervention satisfactory, despite variations in age, sex, diagnosis, level of interaction with the app, and change scores on study outcome measures. As indicated by high levels of engagement with the app [19], adolescents liked the capacity of the app to support pain treatment via the advice provided (especially when they were outside the hospital), pain self-monitoring, and facilitation of communication with health care providers. These findings suggest that mHealth-based self-management of cancer symptoms in real time is amenable to young patients, echoing results from other mHealth-based disease self-management research conducted with youth [39-43]. Additionally, in accordance with previous research [33,44], all interviewed adolescents found the design and gamification mechanics attractive and acceptable. Gamification mechanics are a relatively novel addition to mHealth interventions, and there is still limited literature on benefits with respect to patient behavior change. However, the role these mechanics may play in incentivizing self-management behaviors in chronic conditions [45-47], combined with their acceptability to patients, makes the evaluation of their application to mHealth interventions an area for future exploration.

Challenges adolescents experienced with Pain Squad+ suggested that real-time symptom assessments questionnaires should be brief and notifications directing patients to interact with apps should be minimized. These findings are in contrast with previous studies conducted by our group that showed adolescents to report high acceptability with a 22-item pain questionnaire and the same notification schedule employed presently [1,33]. However, adolescents in the previous studies completed pain assessments only twice-daily for 14 days rather than for 28 days. Acceptability may have been adversely impacted by the increased length of pain reporting used in this study, as well as the technical issues we experienced related to the server sending some adolescents numerous notifications. Additionally, a number of adolescents suggested that it would be preferable to interact with the app only when symptomatic and that the scheduled morning and evening pain reports were not necessary. Creating an optimal balance between improving user engagement with symptom reporting and limiting alert fatigue [48] should be considered in developing real-time mHealth interventions. The technical problems experienced by adolescents also represented a challenge to intervention use. mHealth technical malfunctioning limits intervention usability [49] and also may represent a safety issue for users if the device does not collect, interpret, and deliver medical information to patients and health care providers as intended [50]. Pain Squad+ software modifications to ensure sound functioning and improve the user experience will be made before a subsequent Pain Squad+ RCT research is conducted.

Generally, adolescents suggested that both pain self-management advice and study nurse involvement contributed to the therapeutic benefit of the app. Previous qualitative research with adult cancer patients engaged in studies of similar real-time symptom management interventions also showed positive feedback related to the role of a health care provider in remotely monitoring symptoms [51,52]. Acceptability of the nurse suggests the role may be a valuable component of real-time symptom management interventions for cancer patients. During the pilot trial, however, only 38% (15/40) of participants actually interacted with the nurse [19], who represents a cost addition to the intervention. A planned Pain Squad+ RCT will, therefore, use a 3-arm design (ie, Pain Squad+ with nurse interaction, Pain Squad+ without nurse interaction, and control) to elucidate the therapeutic benefit and cost implications of clinician involvement.

On the basis of thematic analyses of interview data, satisfaction with the pilot study protocol in general, was high. Engagement in the study was considered easy and the chance to use the intervention was liked. Participant reports of altruism as the motivation for study participation agree with previous studies of adolescents with cancer engaged in research [53,54]. Other motivations for participation included the novelty of being involved in research and the gamification of the app. The cited challenge associated with study participation related to the loaned study phone. Fear of damaging the phone was the reason for the single episode of study attrition, and using the study phone if an adolescent owned their own phone was considered a burden. Given that 80% to 87% of Western adolescents currently own a smartphone [55,56], our future RCT will involve installing Pain Squad+ on adolescent’s personal devices and loaning phones only as necessary.


A potential limitation of this study involved the introduction of potential cognitive biases into study findings. In particular, experimenter bias may have been introduced as the study member who conducted the interviews and data analyses was a PhD student whose dissertation focused on the development of Pain Squad+. To minimize the effect of a bias whereby the student could be overly positive when interpreting participants’ remarks about the app and study, an additional study member independently conducted coding and analysis, and an audit trail detailing analyses decisions was kept. Bracketing or mentally suspending biases also minimized the impact of researcher subjectivity during data collection and analysis [34]. A second limitation relates to social desirability response bias whereby to provide a socially desirable response to the interviewer, adolescents could have denied negative thoughts and feelings about the app [57]. To minimize this effect, at the onset of interviews, adolescents were told that both positive and negative feedback was equally important. A final limitation relates to the pilot study design. Because we used a pre-post study pilot, we could not explore the experience of adolescents being randomized to treatment arms or the acceptability of participating in the control condition.


This study has demonstrated the acceptability and perceived helpfulness of the Pain Squad+ intervention and pilot study protocol to adolescents with cancer, albeit with a few caveats and suggestions for improvement. This study also shows that both self-management and clinician support are generally considered as helpful ingredients in mHealth real-time symptom management interventions for adolescents. However, because not all adolescents agreed that the nurse involvement in the intervention was necessary, future research will elucidate the value of clinician support in improving pain-related outcomes for adolescents. Using the UK MRC framework, this qualitative research provides evidence that will be used to refine the Pain Squad+ intervention and study protocol to design a successful RCT. Specifically, we will modify the Pain Squad+ software by truncating the pain assessments, minimizing the number of notifications received by adolescents, and addressing technical problems related to accessing the app through the internet. Satisfaction with the protocol was generally demonstrated, but modifications to improve acceptability by including the capacity to install adolescents’ personal phones will be added before an RCT. Study findings have applicability to other researchers engaged in the design and development of mHealth- and internet-based interventions for youth with chronic and life-limiting health conditions.


Funding for this study was provided by an eHealth Catalyst Grant from the Canadian Institutes of Health Research and a Nurse Researcher Grant from Alex’s Lemonade Stand Foundation to JS. LJ was supported by a Doctoral Fellowship from the Pediatric Oncology Group of Ontario and a Pain Scientist Fellowship from the University of Toronto Centre for the Study of Pain. The results and conclusions reported presently are those of the authors and are independent from the funding sources.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Pain Squad+ pain management app screenshots. Top to bottom, left to right: (a) pain management screen; (b) sample of pain self-management recommendations; (c) detailed view of "Mental Games" pain management recommendation; (d) sample of "gamification" reward for adherence (ie, advancement through law-enforcement ranks); and (e) view of user placement within law-enforcement ranks to encourage adherence.

PDF File (Adobe PDF File), 243KB

Multimedia Appendix 2

Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist.

PDF File (Adobe PDF File), 44KB

Multimedia Appendix 3

Characteristics of interviewed Pain Squad and participants.

PDF File (Adobe PDF File), 52KB

  1. Stinson JN, Jibb LA, Nguyen C, Nathan PC, Maloney AM, Dupuis LL, et al. Construct validity and reliability of a real-time multidimensional smartphone app to assess pain in children and adolescents with cancer. Pain 2015;156(12):2607-2615. [CrossRef] [Medline]
  2. Sung L, Klaassen R, Dix D, Pritchard S, Yanofsky R, Dzolganovski B, et al. Identification of paediatric cancer patients with poor quality of life. Br J Cancer 2009 Jan 13;100(1):82-88 [FREE Full text] [CrossRef] [Medline]
  3. Shepherd E, Woodgate R, Sawatzky JA. Pain in children with central nervous system cancer: a review of the literature. Oncol Nurs Forum 2010 Jul;37(4):E318-E330. [CrossRef] [Medline]
  4. Hinds P, Nuss S, Ruccione K, Withycombe J, Jacobs S, DeLuca H, et al. PROMIS pediatric measures in pediatric oncology: valid and clinically feasible indicators of patient-reported outcomes. Pediatr Blood Cancer 2013 Mar;60(3):402-408. [CrossRef] [Medline]
  5. Lu Q, Krull KR, Leisenring W, Owen JE, Kawashima T, Tsao JC, et al. Pain in long-term adult survivors of childhood cancers and their siblings: a report from the Childhood Cancer Survivor Study. Pain 2011 Nov;152(11):2616-2624 [FREE Full text] [CrossRef] [Medline]
  6. Chordas C, Manley P, Merport MA, Chen B, Liptak C, Recklitis C. Screening for pain in pediatric brain tumor survivors using the pain thermometer. J Pediatr Oncol Nurs 2013;30(5):249-259. [CrossRef] [Medline]
  7. Abernethy A, Samsa G, Matchar D. A clinical decision and economic analysis model of cancer pain management. Am J Manag Care 2003;9(10):651-664 [FREE Full text] [Medline]
  8. Tsai S, Liu L, Tang S, Chen J, Chen M. Cancer pain as the presenting problem in emergency departments: incidence and related factors. Support Care Cancer 2010;18(1):57-65. [CrossRef] [Medline]
  9. Barbera L, Taylor C, Dudgeon D. Why do patients with cancer visit the emergency department near the end of life? CMAJ 2010;182(6):563-568 [FREE Full text] [CrossRef] [Medline]
  10. Walker AJ, Gedaly-Duff V, Miaskowski C, Nail L. Differences in symptom occurrence, frequency, intensity, and distress in adolescents prior to and one week after the administration of chemotherapy. J Pediatr Oncol Nurs 2010;27(5):259-265. [CrossRef] [Medline]
  11. Kuo K, Saokaew S, Stenehjem D. The pharmacoeconomics of breakthrough cancer pain. J Pain Palliat Care Pharmacother 2013;27(2):167-175. [CrossRef] [Medline]
  12. Fortier M, Sender L, Kain Z. Management of pediatric oncology pain in the home setting: the next frontier. J Pediatr Hematol Oncol 2011;33(4):249-250. [CrossRef] [Medline]
  13. Baxter N, Daly C, Gupta S, Pole J, Sutradhar R, Greenberg M, et al. The Initiative to Maximize Progress in Adolescent and Young Adult Cancer Therapy (IMPACT) Cohort Study: a population-based cohort of young Canadians with cancer. BMC Cancer 2014 Nov 03;14:805 [FREE Full text] [CrossRef] [Medline]
  14. Smith MA, Seibel NL, Altekruse SF, Ries LA, Melbert DL, O'Leary M, et al. Outcomes for children and adolescents with cancer: challenges for the twenty-first century. J Clin Oncol 2010 May 20;28(15):2625-2634. [CrossRef] [Medline]
  15. Erickson J, Macpherson C, Ameringer S, Baggott C, Linder L, Stegenga K. Symptoms and symptom clusters in adolescents receiving cancer treatment: a review of the literature. Int J Nurs Stud 2013;50(6):847-869. [CrossRef] [Medline]
  16. Baggott C, Dodd MA, Kennedy C, Marina N, Matthay K, Cooper B, et al. Changes in children's reports of symptom occurrence and severity during a course of myelosuppressive chemotherapy. J Pediatr Oncol Nurs 2010;27(6):307-315. [CrossRef] [Medline]
  17. Hinds P, Quargnenti A, Bush A, Pratt C, Fairclough D, Rissmiller G, et al. An evaluation of the impact of a self-care coping intervention on psychological and clinical outcomes in adolescents with newly diagnosed cancer. Eur J Oncol Nurs 2000;4(1):6-17. [CrossRef] [Medline]
  18. Jibb LA, Cafazzo JA, Nathan PC, Seto E, Stevens BJ, Nguyen C, et al. Development of a mHealth real-time pain self-management app for adolescents with cancer: an iterative usability testing study. J Pediatr Oncol Nurs 2017;34(4):283-294. [CrossRef] [Medline]
  19. Jibb LA, Stevens BJ, Nathan PC, Seto E, Cafazzo JA, Johnston D, et al. Implementation and preliminary effectiveness of a real-time pain management smartphone app for adolescents with cancer: a multicenter pilot clinical study. Pediatr Blood Cancer 2017;64(10). [CrossRef] [Medline]
  20. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008 Sep 29;337:a1655 [FREE Full text] [Medline]
  21. Jibb LA, Nathan PC, Stevens BJ, Seto E, Cafazzo JA, Stephens N, et al. Psychological and physical interventions for the management of cancer-related pain in pediatric and young adult patients: an integrative review. Oncol Nurs Forum 2015;42(6):E339-E357. [CrossRef] [Medline]
  22. Jibb LA, Stevens BJ, Nathan PC, Seto E, Cafazzo JA, Stinson JN. A smartphone-based pain management app for adolescents with cancer: establishing system requirements and a pain care algorithm based on literature review, interviews, and consensus. JMIR Res Protoc 2014;3(1):e15 [FREE Full text] [CrossRef] [Medline]
  23. Wu W, Johnson R, Schepp K, Berry D. Electronic self-report symptom and quality of life for adolescent patients with cancer: a feasibility study. Cancer Nurs 2011;34(6):479-486. [CrossRef] [Medline]
  24. Creswell J, Klassen A, Plano CV, Clegg SK. 2011. Best practices for mixed methods research in the health sciences Internet   URL: [accessed 2018-03-07] [WebCite Cache]
  25. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19(6):349-357. [CrossRef] [Medline]
  26. Patton M. Qualitative research and evaluation methods. 3rd ed. California: SAGE; 2002.
  27. Bowen G. Naturalistic inquiry and the saturation concept: a research note. Qual Res 2008;8(1):137-152. [CrossRef]
  28. Hirschfeld G, Zernikow B. Cut points for mild, moderate, and severe pain on the VAS for children and adolescents: what can be learned from 10 million ANOVAs? Pain 2013;154(12):2626-2632. [CrossRef] [Medline]
  29. Cleeland C. The Brief Pain Inventory: User Guide. Houston: MD Anderson Cancer Center, University of Texas; 2009.
  30. Mathias S, Crosby R, Qian Y, Jiang Q, Dansey R, Chung K. Estimating minimally important differences for the worst pain rating of the Brief Pain Inventory-Short Form. J Support Oncol 2011;9(2):72-78. [Medline]
  31. Varni J, Burwinkle T, Katz E, Meeske K, Dickinson P. The PedsQL in pediatric cancer: reliability and validity of the Pediatric Quality of Life Inventory Generic Core Scales, Multidimensional Fatigue Scale, and Cancer Module. Cancer 2002;94(7):2090-2106 [FREE Full text] [Medline]
  32. DeWalt D, Rothrock N, Yount S, Stone A, PROMIS Cooperative Group. Evaluation of item candidates: the PROMIS qualitative item review. Med Care 2007;45(5 Suppl 1):S12-S21 [FREE Full text] [CrossRef] [Medline]
  33. Stinson JN, Jibb LA, Nguyen C, Nathan PC, Maloney AM, Dupuis LL, et al. Development and testing of a multidimensional iPhone pain assessment application for adolescents with cancer. J Med Internet Res 2013 Mar 08;15(3):e51 [FREE Full text] [CrossRef] [Medline]
  34. Morrow S. Quality and trustworthiness in qualitative research in counseling psychology. J Couns Psychol 2005;52(2):250-260. [CrossRef]
  35. Lingard L, Albert M, Levinson W. Grounded theory, mixed methods, and action research. Br Med J 2008;337:a567. [Medline]
  36. Sandelowski M. What's in a name? Qualitative description revisited. Res Nurs Health 2010;33(1):77-84. [CrossRef] [Medline]
  37. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health 2000;23(4):334-340. [Medline]
  38. O'Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. Br Med J 2010;341:c4587. [Medline]
  39. Stinson JN, Sung L, Gupta A, White M, Jibb LA, Dettmer E, et al. Disease self-management needs of adolescents with cancer: perspectives of adolescents with cancer and their parents and healthcare providers. J Cancer Surviv 2012;6(3):278-286. [CrossRef] [Medline]
  40. Linder L, Erickson J, Stegenga K, Macpherson C, Wawrzynski S, Wilson C, et al. Symptom self-management strategies reported by adolescents and young adults with cancer receiving chemotherapy. Support Care Cancer 2017;25(12):3793-3806. [CrossRef] [Medline]
  41. Stinson J, Jibb LA, Greenberg M, Barrera M, Luca S, White M, et al. A qualitative study of the impact of cancer on romantic relationships, sexual relationships, and fertility: perspectives of Canadian adolescents and parents during and after treatment. J Adolesc Young Adult Oncol 2015;4(2):84-90. [CrossRef] [Medline]
  42. Macpherson C, Linder L, Ameringer S, Erickson J, Stegenga K, Woods N. Feasibility and acceptability of an iPad application to explore symptom clusters in adolescents and young adults with cancer. Pediatr Blood Cancer 2014;61(11):1996-2003. [CrossRef] [Medline]
  43. Goyal S, Nunn C, Rotondi M, Couperthwaite A, Reiser S, Simone A, et al. A mobile app for the self-management of type 1 diabetes among adolescents: a randomized controlled trial. JMIR Mhealth Uhealth 2017;5(6):e82. [CrossRef] [Medline]
  44. Stinson JN, Gupta A, Dupuis F, Dick B, Laverdière C, LeMay S, et al. Usability testing of an online self-management program for adolescents with cancer. J Pediatr Oncol Nurs 2015;32(2):70-82. [CrossRef] [Medline]
  45. Miller AS, Cafazzo JA, Seto E. A game plan: gamification design principles in mHealth applications for chronic disease management. Health Informatics J 2016;22(2):184-193. [CrossRef] [Medline]
  46. AlMarshedi A, Wills G, Ranchhod A. Guidelines for the gamification of self-management of chronic illnesses: multimethod study. JMIR Serious Games 2017;5(2):e12 [FREE Full text] [CrossRef] [Medline]
  47. King D, Greaves F, Exeter C, Darzi A. 'Gamification': influencing health behaviours with games. J R Soc Med 2013;106(3):76-78 [FREE Full text] [CrossRef] [Medline]
  48. Muench F, Baumel A. More than a text message: dismantling digital triggers to curate behavior change in patient-centered health interventions. J Med Internet Res 2017;19(5):e147 [FREE Full text] [CrossRef] [Medline]
  49. Makai P, Melis R, Olde-Rikkert M. Technical difficulties and evaluating e-health interventions. JAMA Intern Med 2014;174(2):304-305. [CrossRef] [Medline]
  50. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012;Suppl:49-56. [CrossRef] [Medline]
  51. Maguire R, Ream E, Richardson A, Connaghan J, Johnston B, Kotronoulas G, et al. Development of a novel remote patient monitoring system: the advanced symptom management system for radiotherapy to improve the symptom experience of patients with lung cancer receiving radiotherapy. Cancer Nurs 2015;38(2):E37-E47. [CrossRef] [Medline]
  52. Sundberg K, Eklöf AL, Blomberg K, Isaksson A, Wengström Y. Feasibility of an interactive ICT-platform for early assessment and management of patient-reported symptoms during radiotherapy for prostate cancer. Eur J Oncol Nurs 2015;19(5):523-528. [CrossRef] [Medline]
  53. Barrera M, D'Agostino N, Gammon J, Spencer L, Baruchel S. Health-related quality of life and enrollment in phase 1 trials in children with incurable cancer. Palliat Support Care 2005;3(3):191-191. [Medline]
  54. Miller VA, Baker JN, Leek AC, Hizlan S, Rheingold SR, Yamokoski AD, et al. Adolescent perspectives on phase I cancer research. Pediatr Blood Cancer 2013;60(5):873-878 [FREE Full text] [CrossRef] [Medline]
  55. eMarketer. 2016. Teens' Ownership of Smartphones Has Surged Internet   URL: [accessed 2018-03-07] [WebCite Cache]
  56. eMarketer. 2016. In Canada, children and teens have an affinity for digital tools   URL: [accessed 2018-03-07] [WebCite Cache]
  57. Logan D, Claar R, Scharff L. Social desirability response bias and self-report of psychological distress in pediatric chronic pain patients. Pain 2008;136(3):366-372. [CrossRef] [Medline]

AES: Acceptability E-Scale
HRQoL: health-related quality of life
UK MRC: United Kingdom Medical Research Council
RCT: randomized controlled trial

Edited by G Eysenbach; submitted 31.10.17; peer-reviewed by J Erickson, K Birkeland; comments to author 30.11.17; revised version received 08.01.18; accepted 26.01.18; published 06.04.18


©Lindsay A Jibb, Bonnie J Stevens, Paul C Nathan, Emily Seto, Joseph A Cafazzo, Donna L Johnston, Vanessa Hum, Jennifer N Stinson. Originally published in JMIR Mhealth and Uhealth (, 06.04.2018.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.