JMU JMIR Mhealth Uhealth JMIR mHealth and uHealth 2291-5222 JMIR Publications Toronto, Canada v7i3e11568 30924789 10.2196/11568 Original Paper Original Paper Development and Long-Term Acceptability of ExPRESS, a Mobile Phone App to Monitor Basic Symptoms and Early Signs of Psychosis Relapse Eysenbach Gunther Rotondi Armando Jacob Christine Chaudhry Beenish Rodríguez Iyubanit Eisner Emily BA, MRes 1
Division of Psychology and Mental Health, School of Health Sciences Manchester Academic Health Science Centre University of Manchester S42, Zochonis Building (Second Floor) Brunswick Street Manchester, M13 9PL United Kingdom 44 161 306 042 44 161 306 0406 emily.eisner@manchester.ac.uk
http://orcid.org/0000-0001-5164-2407 Drake Richard James BSc, MB, ChB, MRC, PhD 1 2 http://orcid.org/0000-0003-0220-4835 Berry Natalie BSc, MSc, PhD 1 http://orcid.org/0000-0001-7414-1541 Barrowclough Christine BA, MSc, PhD 1 http://orcid.org/0000-0001-9037-6201 Emsley Richard BSc, PhD 3 http://orcid.org/0000-0002-1218-675X Machin Matthew BEng, CEng, MIET 4 http://orcid.org/0000-0001-6600-9508 Bucci Sandra BSc, DClinPsy 1 2 http://orcid.org/0000-0002-6197-5333 Division of Psychology and Mental Health, School of Health Sciences Manchester Academic Health Science Centre University of Manchester Manchester United Kingdom Greater Manchester Mental Health National Health Service Foundation Trust Manchester United Kingdom Department of Biostatistics and Health Informatics King's College London Institute of Psychiatry, Psychology and Neuroscience De Crespigny Park London United Kingdom Division of Informatics, Imaging & Data Sciences Manchester Academic Health Science Centre University of Manchester Manchester United Kingdom Corresponding Author: Emily Eisner emily.eisner@manchester.ac.uk 032019 29 03 2019 7 3 e11568 15 7 2018 7 10 2018 25 10 2018 10 12 2018 ©Emily Eisner, Richard James Drake, Natalie Berry, Christine Barrowclough, Richard Emsley, Matthew Machin, Sandra Bucci. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 29.03.2019. 2019

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.

 Background

Schizophrenia relapses are common, have profound, adverse consequences for patients and are costly to health services. Early signs interventions aim to use warning signs of deterioration to prevent full relapse. Such interventions show promise but could be further developed. This study addresses 2 developments: adding basic symptoms to checklists of conventional early signs and using a mobile phone app ExPRESS to aid early signs monitoring.

 Objective

This study aimed to (1) design a pool of self-report items assessing basic symptoms (Basic Symptoms Checklist, BSC); (2) develop and beta test a mobile phone app (ExPRESS) for monitoring early signs, basic symptoms, and psychotic symptoms; and (3) evaluate the long-term acceptability of ExPRESS via qualitative feedback from participants in a 6-month feasibility study.

 Methods

The BSC items and ExPRESS were developed and then adjusted following feedback from beta testers (n=5) with a schizophrenia diagnosis. Individuals (n=18) experiencing a relapse of schizophrenia within the past year were asked to use ExPRESS for 6 months to answer weekly questions about experiences of early signs, basic symptoms, and psychotic symptoms. At the end of follow-up, face-to-face qualitative interviews (n=16; 2 were uncontactable) explored experiences of using ExPRESS. The topic guide sought participants’ views on the following a priori themes regarding app acceptability: item content, layout, and wording; app appearance; length and frequency of assessments; worries about app use; how app use fitted with participants’ routines; and the app’s extra features. Interview transcripts were analyzed using the framework method, which allows examination of both a priori and a posteriori themes, enabling unanticipated aspects of app use experiences to be explored.

 Results

Participants’ mean age was 38 years (range 22-57 years). Responses to a priori topics indicated that long-term use of ExPRESS was acceptable; small changes for future versions of ExPRESS were suggested. A posteriori themes gave further insight into individuals’ experiences of using ExPRESS. Some reported finding it more accessible than visits from a clinician, as assessments were more frequent, more anonymous, and did not require the individual to explain their feelings in their own words. Nevertheless, barriers to app use (eg, unfamiliarity with smartphones) were also reported. Despite ExPRESS containing no overtly therapeutic components, some participants found that answering the weekly questions prompted self-reflection, which had therapeutic value for them.

 Conclusions

This study suggests that apps are acceptable for long-term symptom monitoring by individuals with a schizophrenia diagnosis across a wide age range. If the potential benefits are understood, patients are generally willing and motivated to use a weekly symptom-monitoring app; most participants in this study were prepared to do so for more than 6 months.

 Trial Registration

ClinicalTrials.gov NCT03558529; https://clinicaltrials.gov/ct2/show/NCT03558529 (Archived by WebCite at http://www.webcitation.org/70qvtRmZY).

 schizophrenia psychotic disorders recurrence telemedicine mobile health mHealth eHealth mental health Introduction

A total of 80% of those with first episode psychosis relapse within 5 years [1,2], which often leads to unplanned admissions [3-5], increased personal distress [6], vocational disruption [7], worse residual symptoms [8], and risk of suicide [9-11]. Interventions using early signs of deterioration (eg, increased anxiety and insomnia) to prompt timely preventative action can forestall relapse [12-14] but could be further developed by improving relapse prediction and increasing engagement with long-term early signs monitoring [15]. This study addresses 2 such developments: (1) adding basic symptoms to checklists of conventional early signs and (2) using mobile phone technology to aid early signs monitoring.

Basic symptoms are subtle, subjective changes in individuals’ experiences of themselves (eg, difficulty managing attention) and the world around them (eg, more vivid colors) that predict first episodes of psychosis [16,17] and might predict relapses [18-20]. The most comprehensive studies to date have been retrospective [18,19]. One prospective study [21] examined 10 items resembling basic symptoms, but the authors did not explicitly state that these were basic symptoms or specify how they were measured [19]. There is a clear need for a well-powered, prospective study to establish whether combining basic symptoms with conventional early signs improves relapse prediction. We plan to carry out such a study using a phone app to facilitate monitoring, in line with recent developments in mobile health (mHealth) for psychosis.

In this study, we describe the development and long-term acceptability of ExPRESS, a phone app that monitors basic symptoms and conventional early signs of psychosis relapse on a weekly basis, and the basic symptoms checklist (BSC), the pool of self-report items used to assess basic symptoms in the app. Both as a research tool and in clinical practice, an app potentially has numerous advantages over face-to-face [22], postal [23], or text message-based [24] assessments of early signs of relapse. An app can be accessed at times and places convenient to the patient, as mobile phones tend to be carried around from place to place and their use is often integrated within daily life [25]. Monitoring early signs with an app is less resource intensive, intrusive, time consuming, and burdensome than weekly visits from a researcher or clinician. People with a schizophrenia diagnosis find apps acceptable for short-term self-monitoring [26] and prefer them to text message–based systems [27]. Native phone apps are preferable to Web-based systems, being less dependent on a good data or Wi-Fi connection and, therefore, more accessible in rural locations and for low-income users [25]. Furthermore, apps can include automated features such as reminders, generation of graphs, and secure upload of data, which might enhance user experience and increase engagement.

We know of 9 published studies that have prospectively assessed symptom course using a phone app [26-34] but none testing an app to monitor early signs of psychosis relapse. Only 1 study to date has evaluated a symptom monitoring app for longer than 6 months [34], and none in a sample with established psychosis. Much current literature relies on satisfaction ratings of apps for psychosis [25,33-35]. Only a few studies include qualitative feedback from people with psychosis regarding actual [28,36,37] or hypothetical [38-41] acceptability of apps to monitor or ameliorate psychosis symptoms. Most had relatively young samples, likely to be more au fait with this technology (digital natives) than is typical of those with chronic psychosis, and none examined psychosis patients’ experiences of long-term native phone app use.

Integrating user feedback into the design of mobile phone apps and psychological interventions is best practice [28,36,42-44] and improves engagement with digital tools for psychosis [45]. Ben-Zeev et al [25] recommend that researchers publish descriptions of app development, including specific ways the design was influenced by user feedback. Accordingly, we provide details of the ExPRESS app design and how it was changed in response to feedback during beta testing. We then describe a qualitative analysis of in-depth feedback from longitudinal feasibility study participants regarding the actual acceptability of the app (a priori themes), as recommended in a recent systematic review [46]. Framework analysis [47] allowed us to also consider a posteriori themes to further understand patients’ experiences of long-term symptom monitoring with a phone app. The rich qualitative data provided valuable information on how people with psychosis use symptom monitoring apps, whether they perceive any value from doing so and what challenges they might encounter.

We describe 3 stages of the study (Figure 1), with the following 3 aims:

Stage 1: To design a pool of self-report items assessing basic symptoms (BSC);

Stage 2: To develop and beta test a mobile phone app ExPRESS for monitoring early signs, basic symptoms, and psychotic symptoms; and

Stage 3: To evaluate the long-term acceptability of ExPRESS by gathering qualitative feedback from participants in a 6-month feasibility study.

Flow diagram of study design. BSC: Basic Symptom Checklist; SPI-A: Schizophrenia Proneness Instrument, Adult Version.

 Methods Stage 1: Basic Symptom Item Pool

A pool of 56 self-report basic symptoms items was developed to form the BSC. Item wording was based on qualitative interviews in which participants described basic symptom experiences before psychosis relapse [19] and descriptions and examples given in the Schizophrenia Proneness Instrument, Adult Version manual (SPI-A) [48]. Our retrospective study [19] is one of the few to collect quotations from English-speaking participants regarding basic symptoms, making it an ideal foundation for an English-language self-report measure. Items were specifically designed for mobile phone app completion and had a 7-point visual analog scale with 2 anchors (Not at all; A great deal).

 Stage 2: ExPRESS App Development Overview of ExPRESS

The software was adapted from ClinTouch [26], a symptom-monitoring app assessing 12 Positive and Negative Syndrome Scale (PANSS) [49] and 2 Calgary Depression Scale (CDS) [50] items. ClinTouch and ExPRESS are compared in Table 1. They share their look, feel, and general functionality, but features such as item content and alert frequency differ, as does the period of monitoring studied.

ExPRESS includes a pool of 5 PANSS and 2 CDS items, plus items from the BSC, the early signs scale (ESS) [51], fear of recurrence scale (FoRSe) [52], and optional personalizable items. All assessments relate to the past week, with PANSS, CDS, BSC, and personalizable items answered by moving a sliding bar along a 7-point visual analog scale and ESS and FoRSe items scored on a 4-point ordinal scale (Figure 2). Individuals do not use all items but select a subset of relevant items (≤5 from BSC; ≤5 from ESS or FoRSe; and ≤5 personalizable items) for monitoring, together with all 7 PANSS and CDS items. PANSS and CDS items include additional follow-up questions (eg, This stopped me from doing things) contingent on the user’s answer to an initial question (eg, I have heard voices). The total number of questions to be answered per week ranges from 26 to 53, with the average number asked in this study being 42.

Comparison of ClinTouch and ExPRESS app design.

App feature ClinTouch app ExPRESS app
Items
Items scored on a 7-point visual analog scale, using a sliding bar with 2 anchors 12 PANSSa items: delusions, hallucinations, suspiciousness, grandiosity, anxiety, depression, guilt, somatic concern, passive apathetic social withdrawal, hostility, excitement, and conceptual disorganization; and 2 Calgary Depression Scale items: depression and hopelessness 5 PANSS items: delusions, hallucinations, suspiciousness, grandiosity, and anxiety; 2 Calgary Depression Scale items: depression and hopelessness; Basic Symptoms Checklist (56 items); and optional personalizable items
Items scored on a 4-point Likert scale None Early Signs Scale (34 items) and Fear of Recurrence Scale (3 items)
Items that can be personalized (eg, based on baseline interview) PANSS delusions items (≤2) Basic Symptoms Checklist (≤5), Early Signs Scale or Fear of Recurrence Scale (≤5), personalizable items (≤5), and PANSS delusions items (≤2)
Extra features
Wallpaper Yes Yes, with new photos
Daily diary Yes Yes
Graphs of delusions and hallucinations Yes Yes
Useful numbers Yes Yes
Alerts
Frequency of alerts 6 pseudo-random occasions per day Once per week
Alert day(s) Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday Wednesday, 1.30 pm
Time window for answering the questions 15 min 24 hours
Snooze options Snooze for 5 min First snooze for half an hour and second snooze for 22.5 hours
Data upload
Data upload Wireless upload using mobile internet Wireless upload using mobile internet
Data storage Secure server at the University of Manchester Secure server at the University of Manchester

aPANSS: Positive and Negative Syndrome Scale.

Screenshots of an example basic symptom item (left) and early signs item (right) displayed on the ExPRESS (Experiences of Psychosis Relapse: Early Subjective Signs) app.

ExPRESS alerts the participant to answer questions once a week (1.30 pm on a Wednesday), using a beep and visual notification. The participant has 24 hours to answer. They have the option to snooze the app and receive additional alerts after 0.5 and 22.5 hours. Using the mobile phone’s internet connection, ExPRESS automatically uploads participants’ responses to a secure server maintained by the University of Manchester, where responses are accessible to the research team via a password-protected Web interface.

As in ClinTouch, participants could choose to personalize the app by changing the wallpaper and could view automatically generated graphs of recent responses to delusions, hallucinations, and mood items. There was also a daily diary for the individual’s own reference (not uploaded) and a section containing useful numbers (eg, Samaritans).

 Beta Testing

ExPRESS was beta tested on a variety of mobile phone handsets by psychology researchers (n=5) and lay people (n=6) for 1 week each. Separate, one-off beta tests were then conducted by patients (n=5) with a schizophrenia-spectrum diagnosis (Diagnostic and Statistical Manual of Mental Disorders, 4th Edition; DSM-IV) and experience of basic symptoms. Participants were given an overview of ExPRESS and were asked to perform a series of in-app tasks (observed by the researcher) and to provide comments using a structured app-evaluation form (Multimedia Appendix 1). Their feedback and the researcher’s observations were collated, and proposed changes to ExPRESS were discussed by the research team. Given the study’s finite resources, namely, computer programmer time, changes that improved the app’s functionality and were in line with its overall purpose were prioritized (eg, adding in the past week to all questions to clarify which period they refer to).

 Stage 3: 6-Month Feasibility Study and Qualitative Interviews Feasibility Study

Patients (n=18; convenience sample) were recruited from 3 NHS Mental Health Trusts in the North West of England to participate in a 6-month, single-arm, open longitudinal feasibility study (see study protocol for full details [53]; Research Ethics Committee reference 14/NW/1471). They were trained on ExPRESS (on their own phone, if compatible, or on a study phone) and asked to use it weekly for 6 months or until relapse, whichever was sooner. Participants received supportive telephone calls from the researcher (weekly for the first 4 weeks and monthly thereafter) to encourage participation and troubleshoot any difficulties with app use. Those using ExPRESS on a study phone received weekly text messages to their own phone number to remind them to use the app. For participants giving additional consent, app assessments were sent to a named clinician. Notes on all telephone calls and face-to-face meetings with participants during the follow-up period were included in the study feasibility diary. At the end of follow-up, all participants were invited (via telephone) to take part in a qualitative interview, including those who had dropped out of the app use phase (n=4, of whom 2 were uncontactable). Quantitative findings from the longitudinal feasibility study are reported elsewhere.

Inclusion criteria for the longitudinal feasibility study were as follows: schizophrenia-spectrum diagnosis (DSM-IV); at least 1 acute psychotic episode in the past year (exacerbation of psychotic symptoms lasting at least 2 weeks and requiring a management change, including admission) or at least 2 acute episodes in the past 2 years, including index episode; reporting basic symptoms (assessed using SPI-A), which began or increased before a previous psychotic episode; currently prescribed antipsychotic medication; age over 18 years; fluent in English; fixed abode; sufficiently stable to take part; no current alcohol or drug dependence (Structured Clinical Interview for DSM-IV, [54]); and providing informed consent.

 Qualitative Interviews

Qualitative interviews were conducted at the end of the 6-month feasibility study by 2 researchers (EE and NB) using a topic guide (Multimedia Appendix 2; piloted with 1 lay participant) covering: item content, layout, and wording; app look and feel; length and frequency of assessments; worries about app use; how app use fitted with participants’ routines; the app’s extra features; and other experiences of using the app. Both interviewers were female PhD researchers with Master’s degrees and experience conducting qualitative interviews; EE had met all interviewees 6 months earlier, whereas NB had spoken to them via telephone only. Participants knew that EE had developed the app as part of her PhD research and that NB was covering maternity leave.

Interviews were conducted in participants’ homes or a nearby NHS service, ranged from 7 to 53 min (mean 20 min), and were audio-recorded. Immediately afterward, the interviewer made notes in a reflective journal (eg possible themes, topic guide additions, and contextual details). In 3 cases, a nonparticipant was also present in the interview, in accordance with the study’s lone-worker policy. The researcher telephoned 1 participant shortly after the interview to clarify a point they had made. However, because of time limitations, transcriptions were not returned to participants for comment and participants were not asked to give feedback on the findings. Data saturation was not reached, despite all consenting, eligible individuals (feasibility study participants) taking part.

 Framework Analysis

The research team conducted a framework analysis (Table 2) of interview transcripts and the study feasibility diary. The framework method allows a combination of deductive and inductive coding to be used [47]. This was appropriate as there were specific issues that we set out to investigate (a priori themes, eg, participants’ views on the item wording), while also allowing space to explore unanticipated aspects of participants’ experiences (a posteriori themes).

 Reflexivity

EE is a PhD researcher investigating measurement of basic symptoms using a mobile phone app; SB (a clinical psychologist) and RD (a psychiatrist) have worked extensively with individuals with psychosis, both in a clinical and research capacity; and SB and NB are involved in several studies investigating digital health interventions (DHIs) for this population. Although the multidisciplinary nature of the research team promoted consideration of a variety of interpretations, we inevitably brought certain expectations and assumptions to the interview and analysis process, for example, an attachment to the app’s features that we had designed and hoped would be well received. Several steps were taken to minimize potential bias. Questions in the topic guide were worded in an open, neutral manner that avoided indicating researchers’ opinions. Interviewers were careful not to put pressure on participants to answer in a certain way. All ideas, insights, discussions, and decisions during data collection and analysis were documented in a reflective journal. The first author also avoided reading any closely related literature (qualitative studies of app interventions) during the analysis process to avoid biasing her interpretation.

Framework analysis method with details regarding its use in this study.

Stage Description Procedure Details of this stage in this study
1 Transcription Audio-recorded interviews transcribed verbatim EEa transcribed all 16 interviews verbatim.
2 Familiarization Read transcript and listen to audio recording SB, RD, and NB each read 2 of the first 3 interview transcripts, and EE listened to audio-recordings and read transcripts of all 16.
3 Coding Read transcript line by line and apply a code describing why that section is important EE, SB, RD, and NB independently coded ≥2 of the 3 transcripts. A priori codes (eg, item wording and clinician access) were predefined by EE based on the topic guide (deductive coding), and inductive coding was used for other potentially relevant topics.
4 Developing a working analytical framework Researchers compare codes and agree a set of codes for subsequent transcripts EE, SB, RD, and NB met and discussed their coding; differences were resolved and a working analytical framework agreed. The discussion was audio-recorded for reference and noted in the reflective journal.
5 Applying the analytical framework The working analytical framework is applied to subsequent transcripts EE recoded the first 3 transcripts and then coded the remaining 13 transcripts and feasibility diary using the working analytical framework; NVivo was used to manage this process. The framework was updated where necessary (eg, new code needed), with changes discussed periodically with SB, RD, and NB. A final framework was agreed, and EE recoded all 16 transcripts and the feasibility diary for consistent coding across the dataset.
6 Charting data into the framework matrix Data are summarized in a framework matrix. Illustrative quotations are included EE charted the data into 2 framework matrices: 1 containing a priori codes and the second containing a posteriori codes. Both matrices consisted of 1 row per participant and 1 column per code, with codes grouped into provisional themes and subthemes. Draft framework matrices were discussed by EE, SB, RD, and NB and updated as necessary for the final versions.
7 Interpreting the data Researchers keep notes on analytical insights during the analysis process. The team discusses these insights periodically and works toward an interpretation of the data EE kept notes in a reflective journal during analysis. Analysis meetings were audio-recorded, with notes taken. EE used the reflective journal to revisit previous ideas and consolidate these with new insights during analysis. Once the final themes and subthemes had been agreed by the whole team, EE drafted a write-up of the findings. The team critiqued the draft before agreeing a final version.

aAll initials used in this table are author initials.

 Results Stage 1: Basic Symptom Item Pool

The final wording of the 56 self-report basic symptom items is presented in Table 3, with corresponding SPI-A item numbers. Beta testers and longitudinal feasibility study participants generally reported that the items had face validity and the item wording was acceptable. A total of 6 changes in response to feedback from participants in this study and piloting in nonclinical samples are indicated in Table 3.

 Stage 2: ExPRESS App Development Beta Testing

Demographic and clinical characteristics of patient beta testers and changes made to ExPRESS in response to their feedback are shown in Tables 4 and 5, respectively. Aside from the comments summarized in Table 5, beta testers were extremely positive about the ExPRESS app.

 Stage 3: 6-Month Feasibility Study and Qualitative Interviews Participants

The demographic and clinical characteristics of participants are shown in Table 4, with basic characteristics of individual participants shown in Table 6 and retention shown in Figure 1. Participants were mostly white British, single, unemployed, and living alone, with a diagnosis of schizophrenia and an average age of 38 years. In total, two-thirds of the sample was male and almost half had accessed further or higher education. Although average PANSS positive scores suggest mild to moderate symptoms, these were nonnormally distributed with some participants reporting virtually no positive symptoms and others reporting much higher symptom levels.

 A Priori Themes: Long-Term Acceptability of App Use

A priori themes and subthemes are outlined in Multimedia Appendix 3. Participants found ExPRESS acceptable: they were generally very positive about the specific aspects of the app discussed in qualitative interviews and most would be willing to use it weekly for more than 6 months as part of their day-to-day life.

Basic symptoms checklist item pool, with corresponding Schizophrenia Proneness Instrument Adult Version item numbers.

Schizophrenia Proneness Instrument Adult Version item number Basic symptom checklist item wording
A1.1 Doing new things has been more stressful than usual
A1.2 Crowds or people have been more stressful than usual
A1.3 Doing things in a hurry has been more stressful than usual
A2.2 I have felt empty and flat
A3 I seemed to care less about people than I usually do
B1 I have found it very hard to do two things at once
B2 I have found myself more easily distracted than usual
B3 My concentration has been worse than usual
B4 I have been forgetting things I’ve done less than an hour before
B5 My thoughts have been slower than usual
B6 I haven’t had the energy for thinking
C1 Even simple choices have been difficult
C2 Random thoughts have popped into my head
C3 My mind has sometimes gone blank
C4 It’s been hard to follow what people say
C5 I have found it hard to say what I mean
C6 I forgot things I was told almost immediately
D1 It’s been hard to decide what I’m feeling
D2 I have felt more emotional about everyday things
D3 My head’s been buzzing with lots of thoughts
D4 Random things seemed to have a personal meaning for a moment
D5 People have looked somehow different than usual
E1-E5 I’ve had some unusual feelings in my body
E6 It sometimes felt like part of my body had swollen up or shrunk
F1 Light has seemed very bright
F2 I have sometimes seen flashes of light
F3 Things have looked the wrong size
F4 I have noticed sounds more than usual
F5 Sounds have seemed different than usual
F6 My body has sometimes felt like it didn’t belong to me
O1a Sometimes thoughts and images that are unimportant and have no special meaning keep repeating over and over in my mind and I can’t push them away
O2.1 Sometimes I have mixed up real and imaginary things
O2.2 Sometimes I have mixed up real and imaginary memories
O3a Sometimes I take things literally when they are not meant that way. For example, I sometimes misunderstand sayings or metaphors
O4.1 Things seemed closer or further away than they actually were
O4.2 Things have seemed to change shape
O4.3 Colors have seemed different than usual
O4.4 Sometimes when I looked in the mirror I looked different
O4.5 Things that I saw sometimes seemed to move
O4.6b Things have looked wonky or like there was more than one
O4.7 Judging distance or size has been hard
O4.8 Lines have looked somehow wrong
O4.9 If I stared at something and then looked away I could still see it afterwards
O4.10 I have had “tunnel vision”
O5.1 I could sometimes hear sounds that didn’t seem quite real
O5.2a Sometimes I hear sounds which I heard a few minutes ago or even hours before
O5.2 Sounds have sometimes seemed to continue after I know they have stopped
O6.1 Things have smelt different from usual
O6.2 Things have tasted different from usual
O6.3 Objects have felt different from usual
O7 At times I could not take my eyes off something
O8 Everything around me has seemed somehow not real
O9.1a Sometimes I make certain movements even though I had no intention to, like I have lost control of my body
O9.2a I have sometimes spoken without meaning to
O10 My body has sometimes got “stuck” for a short time
O11 I have had to think about things that I usually do automatically

aWording updated in response to piloting in nonclinical sample.

bWording updated in response to feedback from beta tester in this study. Original wording: Things have looked crooked or like there was more than one.

Clinical and demographic characteristics of the 2 patient samples.

Characteristics Beta testers (n=5) 6-month feasibility study (n=18)
Age (years), mean (SD) 45.8 (21.0) 37.9 (9.9)
Gender (male), n (%) 2 (40) 12 (67)
Positive and Negative Syndrome Scale positive, mean (SD) —a 15.4 (5.4)
Diagnosis, n (%)
Schizophrenia 3 (60) 14 (78)
Schizoaffective 2 (40) 4 (22)
Education, n (%)
Secondary 2 (40) 10 (56)
Further 1 (20) 5 (28)
Higher 2 (40) 3 (17)
Employment, n (%)
Employed 1 (20) 2 (11)
Voluntary work 0 (0) 1 (6)
Retired 2 (40) 1 (6)
Unemployed 2 (40) 14 (78)
Ethnicity, n (%)
Asian or Asian British 0 (0) 1 (6)
Black or black British 1 (20) 2 (11)
White British 4 (80) 15 (83)
Marital status, n (%)
Single 5 (100) 14 (78)
Married 0 (0) 2 (11)
Separated 0 (0) 2 (11)
Living arrangement, n (%)
Alone 2 (40) 12 (67)
With family 2 (40) 4 (22)
Supported accommodation 1 (20) 2 (13)

aBeta testers were not assessed using the Positive and Negative Syndrome Scale.

Changes made to the app or training protocol in response to feedback from patient beta testers.

Comment or suggestion [participant number] Change made to the app Added to the app training protocol
Item wording
PANSSa follow-up questions are confusing; it seems like items are repeated [P1] Change not possible; PANSS items validated Explanation of branching follow-up questions
Items phrased in own words may be invasive [P1] —b Check participant is happy with item wording before use in the study
Alter crooked to wonky, basic symptom item O4.6 [P3] Altered the wording of the item as suggested —
Did not realize that PANSS follow-up questions are contingent on previous answers [P5] — Explanation of branching items; follow-up items are contingent on response to initial item
Item content
I had to generalize in some cases. Specifics occurred to me later [P2] — Reassurance not to worry if cannot remember everything
Items did not seem very closely connected [P2] — Description of item types and how they are connected
Participant would not want to answer items about certain symptoms once unwell [P5] Added to instructions: If you don’t feel comfortable answering, please feel free to ignore the alert If you do not feel comfortable answering, please feel free to ignore the alert
When items are not applicable, might wonder why you asked about them [P5] Added to instructions: Explanation that some are standard items; they might not apply at the moment Explanation that some are standard items; they might not apply at the moment
Sliding bar
One PANSS item is scored in the opposite direction [P1] — Explanation that 1 item is scored in the opposite direction
1-7 scale felt odd; a 1-10 scale might be better [P2] Change not possible; PANSS items validated —
Tended to answer to the extreme (1 or 7) [P2] — Suggestion to leave room for improvement or deterioration when answering items
4-point scale
Change anchor wording so that the number of times a week is first [P1] Anchor wording changed as suggested —
Previous selection sometimes stays for next item [P3] This bug was fixed —
Alerts
Afternoon best as gets up late because of medication [P3] Moved the alert time to afternoon (1.30 pm) —
Reminder when the 24 hours is nearly finished [P3] Added extra reminder after 23 hours —
During work lunch break would be best [P5] Moved the alert time to 1.30 pm —
Would like to be able to set own alert time [P5] Not changed; consider for future app version —
Would like to be able to set the snooze duration [P5] Not changed; consider for future app version —
Worries
Worried that the items might be negative or might tell him to do something negative [P3] — Reassurance that the items are the same each time and will never tell you to do anything
Worry that other people might get hold of the answers [P4] — Reassurance that only the research team can see the uploaded answers
Worry that someone picking up phone might see psychosis mentioned in the app [P5] — Explanation that answers are not visible after upload; guidance on putting a lock on the phone
General
Make more accessible to standard mobile phones [P2] — Participants can borrow a study smartphone
Smartphones take some getting used to [P2] — Participants without smartphone experience will need more training
Prefer menu button to be in the top left corner [P3] No change; not an issue for most participants —
A back button would be helpful [P3] Not changed; consider for future app version —

a Positive and Negative Syndrome Scale.

bCells in this table are empty in cases where the column is not applicable, ie, no changes were made to the app or training protocol.

Basic characteristics of individual participants.

Participant Age (years) Gender Baseline Positive and Negative Syndrome Scale positive subscale scorea
P204 51 Female 20
P205 28 Female 15
P206 36 Male 7
P207 35 Male 26
P208 42 Female 11
P209 51 Male 18
P211 28 Male 16
P214 40 Female 22
P215 37 Male 18
P223 37 Male 7
P224 22 Male 21
P225 48 Male 16
P227 41 Male 19
P230 57 Male 16
P231 22 Female 9
P235 39 Male 12
P236 41 Male 16
P239 28 Female 8

aPossible range: 7 to 49.

Summary of a posteriori themes and subthemes.

 A Posteriori Themes: Benefits of and Barriers to Long-Term App Use Overview of A Posteriori Themes

A total of 5 a posteriori themes were derived from inductive coding of interview transcripts: connection and accessibility, self-reflection, therapeutic value, response style, and barriers to app engagement. Overall, these themes relate to the perceived benefits of, and possible barriers to, app use in this population. Participants were not directly asked about these themes in the topic guide but disclosed them spontaneously when discussing their experiences of using the app during the previous 6 months. A posteriori themes and subthemes are summarized in Figure 3.

 Connection and Accessibility Overview

This theme brings together participants’ comments that relate to the importance of their connection with the researcher and the extent to which they found the app easy to access. Participants compared their interactions with both the researcher and the app to their interactions with mental health staff.

 Connection With the Researcher

The connection with the researcher and a sense of belonging to the study were very important for some participants who found reminder texts and regular telephone calls positive and encouraging:

You’ve really done a lot because you are taking part every day [laughs] just reminding me every day, is part of it. It’s as if we are together!

P206

This participant completed the app assessment every week during the 6-month study period; therefore, it might be that participants’ degree of engagement with the app is dependent on the rapport with the person facilitating delivery, in this case the researcher. One individual felt it was important to know who was receiving her app answers, not just from a confidentiality point of view but also so she knew who she was talking to:

So sometimes, you’d be constantly thinking “well who are you telling all this to and what’s the point?”

P239

Filling in the app was not just a self-reflective exercise for this participant but one involving communication with a specific person.

For some participants, it took time to build a rapport with the researcher, with 2 commenting that they found the initial meeting somewhat stressful, but participating got easier once they built a rapport. One individual who was generally anxious about telephone calls was happy to receive calls during the study. The researcher noted in the reflective journal (reflective journal, data analysis phase):

Phone calls were acceptable. It probably helps that I had met all participants in person and discussed symptoms in detail face-to-face before the phone assessments. Rapport was already established and they had already shared significant info with me without me freaking out or causing them any problems

Several participants commented that it was reassuring to know that somebody was looking at their app responses:

It was just nice to know that there would be somebody looking over you...so that was quite comforting in a way. [P208]

Some seemed to view the study as almost an extension of clinical care:

I also think it depends on the individual person, how they effectively view the phone calls cos, if you have a care coordinator and your care coordinator checks up on you it’s just like that so... I don’t see it kind of different.[P205].

Conversely, 1 participant emphasized that her care coordinator did more than simply assess mental state:

She helps me with other things...so it’s nice that you’ve got someone there for you.

P239 App Accessibility

The majority of participants (83%; 15/18) owned a smartphone and a few mentioned in the interview that they used apps for other things, which made using an app to communicate about their mental health a natural thing [P205]. For example, 1 participant reported using apps such as Facebook and WhatsApp so included the app as part of his social network:

But it benefitted more than Facebook cos it’s really talking about my health.

P206

Some individuals seemed to enjoy the app because they experienced it as similar to a person:

It’s almost like somebody’s speaking to you...asking me questions like “how’s your day been?” or “how’s your week been?”

P205

It becomes like a...little friend.

P208

However, others liked it because they did not have to talk to a person, reporting:

You feel... you can open up more. I know it sounds weird.

P214

Strikingly, those who liked the app because they did not have to talk to a person had high symptoms during follow-up, whereas those who described the app as conversation-like all had low symptoms. It is possible that certain features of an app make it nonthreatening to someone who is wary of disclosing high symptoms; for example, an app might be experienced as more neutral and less judgmental than a person as it does not give any verbal or nonverbal feedback.

Participants reported other ways in which they found the app more accessible than mental health staff. One individual noted that the app asked questions more frequently and regularly than he typically saw his care coordinator, which he found useful:

They [the questions] come every week, she [care coordinator] come like four weeks, three weeks, two weeks, so, but this is reminding me of what I’m supposed to do, all the time.

P206

Another tended not to seek help from his care coordinator when he was having a bad week because he was a bit stoic [P236] and did not want to bother her but he still answered the app at these points, which enabled the care coordinator to offer additional help when necessary. Similarly, it was noted in the study feasibility diary that 1 participant filled in the app once during relapse at a point that he was not willing to do a telephone interview, suggesting that, for some, an app could be more accessible in cases where a patient is unwell and unwilling to talk.

 Self-Reflection Overview

Participants spontaneously reported a number of ways in which the app helped them to reflect on their own mental health. Not only did completing the app provide a time to take stock of things but it also enabled them to reflect on their experiences more objectively and to notice patterns in their experiences over time.

 Taking Stock

The process of answering questions on the app prompted participants to take time to think about their mental health each week (It makes you realize, you know, take sort of stock of where you’re at [P214]), which they believed they would not have done otherwise (Like I had to say to myself “think about your week” cos sometimes I don’t think about the week [P239]). By eliciting regular self-reflection, the app might serve a similar function to talking to a confiding person, as sharing experiences often provides an opportunity to reflect and process them:

My mum asks me like how’s my day been or I just automatically tell her so by the time I’m coming to do the questions it’s like well I’ve already spoke to mum about this and that so it’s just a reminder of that.

P205 Objectivity

The process of answering the questions gave participants the opportunity to translate their subjective experiences into an objective format:

I found that the more I used it, the more I started looking objectively at my life

P205

This provided some individuals with a sense of the realness of their experiences, especially with the help of the graphs:

When you look at the analysis of the data collected...you can objectively see.

P205

Participants theorized that having access to objective data representing their symptoms, particularly in the graph form, might enable a shared understanding of their experiences, both with the care team (Maybe help themselves to understand a little bit better [P227]) and potentially with the general public (If it can help people in the future understand a little bit better about what we go through [P227]). Developing a shared understanding with the care team might facilitate help seeking. One participant who had sought help for an emerging relapse in the past but had not been taken seriously suggested:

If you were to answer the questions and go to the doctor and say “look, these are my results, you can see clearly there’s a change, and these are my experiences,” that would be substantial evidence for the doctor to then sit up and take note.

P205 Recognizing Symptom Ebb and Flow

Taking time to reflect every week and to articulate their experiences in an objective format helped participants notice the level of their symptoms and to recognize ways in which this varied over time, providing a long-term perspective on the ebb and flow of symptoms:

They come and pass, they come and pass away

P205

Several participants acknowledged the potential value of the app in spotting fluctuations in their symptoms:

If it’s something could spot my mood going up and down and things it might be useful.

P207

It would gauge how well you were.

P230 Therapeutic Value Overview

Although a few people could not think of anything that they found helpful about the app when asked directly and 1 participant stated that he did not find the app particularly useful as he is already aware of everything [P207], the majority of participants reported finding the app beneficial.

 Early Detection and Staying Well

If the app was used in a clinical setting, the intention would be to help patients and clinicians spot predefined early signs and intervene to prevent a full relapse from occurring. Several participants spontaneously identified this as a potentially valuable outcome of using the app (It can be like a toolkit for you...A toolkit for like er staying well really...A wellness reaction plan you know. A “wrap” [P230]), with 1 participant suggesting that using the app would help her and others recognize changes in their symptoms in response to stressors:

It could help people identify, cos like for me, now that I’ve used an app, I’m thinking when I know my stressors are coming I’d like to have the app there available so that I can do it and then see “oh is it like my thoughts are changing.”

P205

Recognizing patterns in their symptoms allowed some participants to take action to help themselves stay well during the study. For example, 1 participant reported that using the app meant that she recognized that she felt unwell the week before her monthly depot was due:

Cos it gets you into a pattern of realizing when you’re a bit down, when you’re a bit, you know, dodgy type thing.

P208

The realization that she was feeling down because her depot was due reassured her (I thought that the levels in my depot was probably going down a bit and I was ready for another depot. That’s what that helped me to realize. And that helped to keep me calm by using the phone app [P208]) and meant that she was able to use PRN (pro re nata) medication (medication that can be taken when required) at this time. Thus, using the app could help patients to take more responsibility in managing their mental health in conjunction with their care team.

 Learning Benefit, Normalization, and Hope of Recovery

For many, the opportunity for self-reflection provided by the app was a valuable experience that increased their understanding of psychosis:

I think it probably made me understand my illness a little bit better. The voices and all.

P211

It’s been eye opening.

P208

Although, for some, this brought the more uncomfortable realization that:

I’m not as well as I thought I was.

P227

Although not formally assessed in the study, 1 participant appeared to have a sealing-over recovery style, a form of avoidance coping common in people recovering from psychosis [55]. This individual did not wish to continue using the app for longer than 6 months as he did not like being reminded of his symptoms:

I just don’t feel like looking back on myself in a way if you see what I mean.

P223

In contrast, another participant initially worried that thinking about her experiences every week would remind her that she was ill but actually, it had the opposite effect (normalization):

I thought oh it’d make you feel like oh you’re ill, you know. It’s just a constant reminder: you answer the questions and the results show that you’re ill. But, if anything, it showed that you have this illness but you’re normal, you have normal feelings and emotions and anybody can have them at any time.

P205

There were several ways in which this participant found the app normalizing: (1) it helped her to discuss symptoms in a normalizing way (Cos when I was talking to mum about some of the questions it was like – yeah, she feels that, at times. Ok she doesn’t hear voices but she feels down and things like that and just that normalization of talking to somebody about the study and then explaining to them how it was. [P205]); (2) the fact that there was an app for tracking symptoms was normalizing in itself because of the availability of apps for lots of everyday things; and (3) taking part in the study was normalizing as there must be other people with similar experiences (It was like a normalization so, cos I knew I was taking part and many others but I don’t know to what degree their psychosis is but like it’s just a normalization of like well we have this psychosis. [P205]).

This participant found that the normalizing effect of the app facilitated a sense of acceptance of her experiences, enabling integration rather than sealing over (It was just an acceptance it’s part of me and it’s normal life. Like I didn’t differentiate it from “oh this is a part of me that I don’t know about and I don’t like.” It was just a part of me that I could just carry on living with. [P205]), and gave her hope that she could recover (Afterwards I’d just be like “I can get better. I can get better!” there was just, there was this hope that I had when I was using it. [P205]). She speculated that, had she been filling in the app for a few years, seeing the pattern would have helped her to see the extent of her recovery to date:

My answers to the questions three years ago... would have been completely negative and so to see the change in that, then I would have been like “Yeah actually...I can see I am getting better.”

P205 Value in Acute Psychosis

Two participants commented that the app would be useful during the acute phase of psychosis:

I think it’s good for maybe someone that’s just gone into hospital.

P239

One thought that using the app during an initial episode of psychosis might help people to be more recovery-focused (I think like if you got most people with psychosis to do it when they’re like at their worst illness... I think it would, it’d have a bearing on them being more positive about the illness and being more positive about how they could get through it, with it rather than against it. [P205]) and that it might help mitigate memory loss associated with the acute phase:

Like for me when I was talking about my illness, the more I got ill, the more I couldn’t remember. And I, I think that’s to do with psychosis sometimes, your memory’s not so good.

P205 Value in Conjunction With Other Interventions

All participants in the study were prescribed a maintenance antipsychotic during the follow-up period, but 1 participant in particular [P206] found the app complemented this well in that it increased his medication adherence by providing a sense of accountability. As this participant felt that he should be honest on the app and he did not want the researcher or his care coordinator to see that he had not taken his medication, this motivated him to continue taking it:

With this app, it’s, even when I want to stop my medication, you know, I won’t tell lie that I didn’t take my medication, so it will now tell me “ah, you need to take your medication because tomorrow when you are doing this... you have to be sincere... so you should take your medication!” [P206].

Participants also noted the value of the app in conjunction with other interventions, for example, cognitive behavioral therapy (CBT) or occasional use of PRN medication.

 Response Style Overview

This theme relates to participants’ comments regarding the way they completed the app: whether they did so reluctantly or willingly, how much thought they put into their responses to the app’s questions, and whether they were honest in these.

 Reluctance, Commitment, and Enjoyment

Participants reported that the app was sometimes annoying but sometimes helpful. Although a few people seemed to genuinely enjoy the process of completing app entries (I did enjoy answering the questions... I didn’t see it as a chore. [P230]), most others viewed it as a chore (Not really anything to enjoy actually doing the app... It’s not like you’re playing a computer game or you know. [P236]), but one which hopefully had some benefit (The questions do help you a bit. [P214]). Several participants expressed some reluctance at using the app at times but usually did it anyway because of finding it easier than talking about how they felt [P224] or because they were committed to taking part in the study (I thought no I’ve agreed to do it so I’ll carry on doing it. [P227]). Moreover, 2 participants described a process of motivating self-talk to persuade themselves to complete the app:

I just kept saying to myself “you’ve nearly finished, you’ve nearly finished, it’s not forever.”

P214

I had to say to myself “think about it...just answer the question” you know.

P239

One participant who had enjoyed the app was extremely committed to it, even describing 1 week that he returned early from staying overnight in another city specifically to do the app:

I didn’t go away with the phone so, on Thursday when I woke up in the morning I just told my friend... “I have to go back to Manchester cos I’m supposed to do this thing yesterday”... so I rushed down.

P206 Honesty

A number of participants commented that they answered the app honestly, despite, in some cases, feeling some paranoia about it or finding it tricky to force themselves to think about their week:

I just had to say to myself “tell the truth, think”... I had to say to myself “think about your week.”

P239

One participant reported during a telephone call with the researcher (noted in the study feasibility diary) that he had begun to answer the app more honestly: although his app responses appeared to indicate that he was getting unwell, he was in fact just being more honest.

 Second Guessing Answers Versus Answering Without Deep Thought

Some participants appeared to put a lot of thought into answering the questions, for example, by second-guessing their own answers:

Even as I was doing it I was thinking well was that right, maybe I should have put something else, you know, different number.

P236

Conversely, others reported that they answered without deep thought:

I didn’t really think about it [laughs slightly] you know.

P235

One participant commented that he did not record small fluctuations in his mental health on the app:

I don’t allow it to stay long, so that’s why I don’t put it there.

P206 Barriers to App Engagement Overview

Participants noted a number of barriers, which prevented them from engaging with the app more fully. Mental health symptoms and phone-related barriers interfered with app use in some cases, though these rarely prevented app engagement entirely.

 Mental Health Symptoms Interfered

Positive symptoms interfered with app engagement in several cases, but the extent of this interference varied. One participant had very high levels of positive symptoms that he did not believe were symptoms of an illness; he did not want to continue with the app because he believed it was irrelevant to him, making him feel like an experimental rabbit [P215]. Other participants with positive symptoms were happy to carry on with the study but they occasionally delayed answering questions or missed them for a week and sometimes they took longer to answer:

Sometimes it takes a lot longer cos I’ve got the voices going on at me so it, it takes longer cos I have to argue with them, and I have to really concentrate on what’s on the phone...and then it’s absolutely knackering. I know it sounds weird cos all I’m doing is answering questions.

P214

Overall, 2 participants relapsed during the study. One of these individuals stopped using the app the week before he met study criteria for relapse, having used it virtually every week beforehand. Thus, for some, stopping the app might be a sign of imminent relapse, an observation that is consistent with findings of a previous study in which early signs of relapse were monitored using postal questionnaires [23]. The second relapsing participant stopped using the app the week after meeting relapse criteria, meaning that the app assessments in the previous few weeks could be used to predict relapse. Encouragingly, both participants were happy to continue with the app once the relapse had resolved:

It was only that time when er I was really unwell ...I was just so unwell that I couldn’t do it but then after that I went back into it. [P230].

Mood symptoms affected a few participants’ ability to engage with the app (when I was really low... I just didn’t communicate with anyone anyway [P227]). Although 1 commented that even when things were not so good for me, it’s still easy to answer it [P239], she pointed out that mental health staff can take into account how you are feeling when asking questions in a way that the app cannot:

When you’ve got people coming to you...they can perk you up or get through it and eventually answer the questions.

P239

She also suggested that her mental state at the time of answering the app might affect how she answered; for example, if she was feeling happy when answering them, she might forget that she was feeling down at other points in the week.

 Phone-Related Barriers

A total of 13 people used a study phone during follow-up. Some found the study phone a barrier, stating that they did not tend to look at it regularly, did not take it out with them, sometimes struggled to use the unfamiliar handset, were worried about it being broken or stolen, and that they might forget to charge it. This issue was mitigated to some extent by the researcher sending a weekly reminder text message to participants’ own phones.

Lack of smartphone experience was a barrier in some cases, with 1 participant accidentally deleting the app from the phone and 2 others commenting that their lack of smartphone experience prevented them from accessing the app's extra features. Unlike a digital native, who would probably have explored the app’s features of their own accord, they only used its basic functions. One participant with little smartphone experience described a general anxiety about using phones, including for telephoning and text messaging; another commented:

Sometimes I get tired of looking at a phone.

P239 Other Barriers

Some participants expressed skepticism regarding certain aspects of the study such as whether the research would find anything useful (You don’t gonna find patterns. I’m pretty sure. [P215]), the validity of translating experiences into numbers (You get a more of an insight I would say with erm [pause] people's opinions and thoughts and that rather than turning them into numbers and statistics. [P236]), and how the researcher would work out what the answers on the app meant (I just think about the other person on the other side having to sit there and read the questions. How do they work it out? [P239]). If not addressed, such skepticism might hinder app engagement as beliefs and attitudes predict intentions and behavior [56].

Lack of literacy was a significant barrier for 1 participant and prevented him from continuing with the study:

Basically I’m not very good at reading and spelling...that’s the reason really...I didn’t really want to say last time I seen you...I felt a bit embarrassed...

P225

A number of more minor barriers were recorded in the study feasibility diary. Participants missed 1 or 2 weeks of app use because of being physically unwell, lending their phone to someone else, being busy, oversleeping, being on vacation in the United Kingdom, or being abroad. The study policy was for participants not to complete the questions when they were abroad (because of lack of local support in the destination country). However, allowing participants to do so might increase their access to help at such times as life events, even positive ones, are associated with relapse [57,58].

 Discussion Overview

This study describes the development and testing of ExPRESS, the first mobile phone app to monitor basic symptoms and conventional early signs of psychosis relapse. It also outlines the development of the BSC, a pool of self-report items that can be used to monitor basic symptoms as putative relapse predictors. It has the longest duration of any study testing a symptom monitoring app in a sample of individuals with established psychosis and is the first study to qualitatively examine psychosis patients’ experiences of using an app over a 6-month period. Framework analysis of interviews with patients enabled us to examine both a priori (acceptability) and a posteriori themes (wider app experiences).

 Principal Findings

A priori themes related to the actual acceptability of the ExPRESS app and its use in day-to-day life. Participants found the app acceptable in terms of the way it looked, length of assessments, item content, item wording, and response format. As assessments were only weekly and the 24-hour response window meant they could answer when convenient, most (including those with jobs) reported that the app fitted well with their routine and they would be happy to use it for longer than 6 months as part of their day-to-day life. This contrasts with feedback from participants using Clintouch [36], an app with several symptom assessments per day, who were unwilling to use it for longer than 2 weeks and thought it would not fit well with employment. The authors of the latter study suggest that symptom monitoring for relapse prevention might require less frequent assessments than Clintouch [36], which is consistent with recommendations of minimum fortnightly monitoring during early signs interventions [59]. This study shows that with less frequent monitoring, people are willing to use a symptom monitoring app for significantly longer.

In contrast with clinicians’ hypothetical views [39,60,61], most participants were happy with a clinician having automatic access to their responses. The absence of difficulties while using ExPRESS, which automatically uploads data to the researcher, might have mitigated concerns about automatic uploads in our sample. Indeed, a recent systematic review found that actual acceptability of various aspects of DHIs tended to be higher than hypothetical acceptability [46]. Similarly, after using a mood monitoring app for 3 months as part of the Automated Monitoring of Symptoms Severity study, participants with bipolar disorder were generally willing for clinicians to access the data [62]. However, although clinician access was viewed by participants as potentially valuable, the practical and legal implications of such access require further consideration if apps are to include this feature.

Participants in this study did not report significant worries about the app, except for 2 reporting some paranoia about it and 2 who worried about hospitalization, neither of which prevented engagement. The fact that these worries did not hinder app engagement might be because the research study and the app itself came from a trusted source, namely, a university. Previous qualitative studies with participants with early psychosis [41] have reported that endorsement by a trusted source, such as a university or health service, is likely to allay participant concerns about privacy and security issues.

Virtually no one used the app’s extra features such as the daily diary, graphs, or helpful numbers because they did not remember being told about them. Given that participants would have used the graphs had they known about them, more emphasis should be placed on these features during the app training session to ensure that participants are aware of them.

The a posteriori themes encapsulate participants’ wider experiences of using the app, including participants’ connection with the researcher, accessibility of the app, self-reflection, the therapeutic value of the app, participants’ response styles, and barriers to app engagement. A number of observations and inferences can be made from the first theme, connection and accessibility. The person who the patient sees as administering the app appears to have a key role; taking time to build up a rapport in face-to-face sessions might increase app engagement. Regular telephone calls and text messages from the researcher were important to participants, consistent with findings that the acceptability of DHIs is higher when patients have access to remote support [46,63-65] and that using technology can help patients and clinicians maintain a connection between appointments [37,66]. Reviews indicate that telephone support is acceptable to those with severe mental illness [67,68]. In this study, even a participant who was generally anxious about telephone calls found them acceptable as he had met the researcher previously. This is an important finding for researchers and clinicians seeking to assess symptoms remotely via telephone. Although it is unlikely that patients will be willing to discuss their symptoms over the telephone with a stranger, 1 or 2 meetings in which symptoms are discussed face-to-face might be sufficient for them to feel comfortable discussing them over the telephone.

Although participants found it reassuring to know that someone could see their weekly symptom reports, and some viewed this as an extension of clinical care, others emphasized that clinicians do more than simply monitor symptoms. Thus, as in previous studies with both patients and staff, participants in this study suggested that self-monitoring apps should enhance rather than replace face-to-face appointments [36,60]. Nevertheless, as participants in a recent study hypothesized [41], some found an app more accessible than mental health staff, reporting that they are more open about their symptoms on an app, they can complete it at points when they are unwilling to speak to someone, it is more frequent than their usual contact with a clinician, and it removes the feeling of inconveniencing someone to report symptom increases. Furthermore, several participants with low symptoms described the app in positive, anthropomorphic terms (eg, as a friend), supporting previous findings that individuals can develop strong connections with mobile phones [69].

Some of these qualities (eg, openness, acceptance, friendliness) overlap with descriptions of therapeutic alliance with respect to DHIs [70,71]. Those with psychosis [71] or depression and anxiety [70] have reported a positive therapeutic alliance with DHIs in the absence of a therapist. In the latter case, higher levels of alliance were associated with greater engagement in self-monitoring tasks [70], implying that therapeutic alliance might be an important consideration when evaluating symptom-monitoring apps such as ExPRESS. However, it is often difficult to disentangle the therapeutic alliance attributable to the DHI and to the person administering it [70]. Future studies using ExPRESS could examine therapeutic alliance with the app and the researcher using self-report measures such as the Mobile Agnew Relationship measure [71] and the Agnew Relationship measure [72], respectively.

The second theme, self-reflection, suggests that using a symptom monitoring app prompts patients to reflect regularly on their experiences and to express them in an objective way. Doing so enables them to recognize the ebb and flow of their symptoms and to share their experiences with others. This can have surprisingly therapeutic results, at least for some people, as summarized in the third theme, therapeutic value. Participants recognized the potential value of an app such as ExPRESS for early detection of deterioration, with many reporting that using such an app increased their understanding of their psychosis. This facilitated self-management in some cases, which can be empowering for patients [28,73,74]. It also appeared to have a normalizing effect for some, which seemed to aid an integrative recovery style.

Conversely, patients with a sealing-over recovery style might not be willing to use a symptom monitoring app as it provides regular reminders of past or current symptoms. Previous studies have framed participants’ discomfort with being regularly reminded of their symptoms as a side effect of the biographical disruption experienced by individuals coming to terms with a diagnosis of a chronic disorder [36]. These 2 explanations tap into the idea that regular symptom monitoring reminds patients of their symptomatic status, which can threaten their already fragile view of themselves. They are consistent with findings that a sealing over recovery style predicts low engagement with mental health services more generally [75] and with findings from patients with bipolar disorder [63,76].

Particularly for early psychosis patients, an app is more like their usual communication with others making it potentially destigmatizing compared with usual care [41]. Participants emphasized the potential value of the app for those in the acute phase of psychosis. However, such individuals might require additional support with app use as acute patients do not engage with symptom monitoring apps as well as remitted patients [26]. Nevertheless, with support, using an app during the early stages of a first episode of psychosis might prompt a more recovery-focused approach.

Participants also underlined the potential value of using a symptom monitoring app such as ExPRESS in conjunction with other interventions such as maintenance antipsychotics and CBT. Indeed, self-monitoring is by no means a new concept and has long been used in the context of CBT as a means of fostering collaborative empiricism, with its therapeutic effects reported across a wide variety of psychological disorders [77]. Although often used clinically, there was a decline in self-monitoring research since the 1980s [78]. Nevertheless, the advent of mHealth has seen this revived to some extent [79]. For example, a recent randomized controlled trial comparing app-based symptom monitoring (Clintouch app) with usual care found a significant reduction in positive symptoms in the early psychosis subsample, though not in a sample with chronic psychosis [80]. Similarly, studies using mHealth to prompt medication adherence have had promising results [81,82].

The fourth theme, response style, indicates that patients are generally willing to use a symptom monitoring app, if they can see the benefit, despite it being a chore. Most participants in our study reported answering honestly, despite concerns raised by patients and staff in previous studies that people might underplay or overplay their symptoms on the app to elicit or reduce care [36,60]. Nevertheless, this study also suggests that, if a patient reports increased symptoms, it is worth considering the possibility that they have begun to answer more honestly rather than that their symptoms have actually increased.

The final theme, barriers to engagement, summarizes several factors, which might hinder app engagement; it is useful to identify these so that they can be addressed to increase app engagement in future studies and potentially in clinical practice. Although both psychotic symptoms and mood symptoms interfered with app use to some extent, especially in patients lacking insight, most individuals with high symptoms engaged well with the app, albeit more slowly than others. Similarly, a recent study [33] found that baseline psychotic or depressive symptoms did not predict app completion rates, although negative and agitation symptoms did in a subgroup of study completers, although Ben-Zeev et al [29] reported that app completion was not associated with cognitive functioning, negative symptoms, or persecutory ideation. These findings contrast with surveys of digital health experts, who suggested that poor cognitive functioning and severe symptoms might negatively impact engagement with DHIs [83].

Use of a study phone was reported as a barrier by some, as participants did not tend to integrate this with their daily life as they would their own phone. Although previous qualitative study participants have also expressed a preference for their own phones [27,36], a recent study in an early psychosis sample found no difference in completion rates between those using study phone and those using their own phone [33]. Other key barriers that could be addressed to increase app engagement included lack of smartphone experience, skepticism regarding the app, the study policy to not use the app while abroad, and lack of literacy. Although the latter was also reported as a potential barrier by a third of participants in a recent qualitative study, they nevertheless found mobile phones more accessible than paper-based alternatives [41].

 Strengths and Limitations

The range of clinical histories within the sample is an important strength of the study; there was a substantial proportion of participants with longstanding illness and high levels of residual symptoms and a wide age range. Precautions were taken by the researchers to avoid biasing their interpretation during the interview and analysis process.

A number of limitations should be borne in mind when considering the results of this study. The sample size was modest, and despite the range of participants, this is not a representative sample, so findings should be interpreted with caution. Participants had all consented to use an app as part of a research study so they might be an unusual subset of patients who are particularly interested in mHealth. We did, however, seek to interview all 4 people who dropped out of the study; 2 were uncontactable but the remaining 2 were interviewed. It is possible that socially desirable responding might have biased reported views as half the participants were interviewed by the researcher who conducted app training and all other study procedures. This said, half the interviews were conducted by a new researcher, potentially reducing this bias. The mean interview length was relatively short for a qualitative study, but there was a wide range. Nevertheless, as illustrated by the reported participant quotations, the study was still able to gather rich, detailed information about patient experiences of using ExPRESS.

 Future Research

Having demonstrated the long-term acceptability (reported here) and feasibility (reported elsewhere) of the ExPRESS app, the next step will be to conduct a longitudinal study with sufficient power to examine the hypothesis that adding basic symptoms to conventional early signs improves relapse prediction. If such a study shows that basic symptoms do predict relapse, the app can then be tested as a clinical tool.

 Conclusions

Symptom monitoring apps have the potential to help services move toward more preventative strategies, in which symptom deteriorations are tackled early, rather than reactionary strategies, in which deteriorations are only addressed once they warrant crisis intervention. Although high caseloads often limit the frequency of face-to-face appointments, this study suggests that participants find it acceptable to use technology (phone calls, texts, and a symptom-monitoring app) to maintain contact between face-to-face meetings. Although undoubtedly many clinicians already adopt similar strategies with younger patients, this study suggests that it is possible and acceptable to do so across a wide age range, including with older patients who are less familiar with technology. As long as the potential benefits are understood, patients are generally willing and motivated to use a weekly symptom monitoring app; virtually, all participants in this study were prepared to do so for more than 6 months.

 Multimedia Appendix 1

App evaluation form used during Stage 2 beta testing.

 Multimedia Appendix 2

Topic guide used during Stage 3 qualitative interviews.

 Multimedia Appendix 3

A priori themes and subthemes: long-term acceptability of Experiences of Psychosis Relapse: Early Subjective Signs.

 Abbreviations BSC

Basic Symptoms Checklist

 CBT

cognitive behavioral therapy

 CDS

Calgary Depression Scale

 DHI

digital health intervention

 DSM-IV

Diagnostic and Statistical Manual of Mental Disorders, 4th Edition

 ESS

Early Signs Scale

 FoRSe

Fear of Recurrence Scale

 mHealth

mobile health

 NHS

National Health Service

 PANSS

Positive and Negative Syndrome Scale

 PRN

As needed (from Latin, pro re nata)

 SPI-A

Schizophrenia Proneness Instrument, Adult Version

The authors thank the study participants and the staff who facilitated recruitment, including those from the Greater Manchester Clinical Research Network (National Institute for Health Research, UK). Funding was provided by the Medical Research Council (EE, grant numbers MR/J500410/1, MR/K500823/1, MR/K501311/1). The funder did not have a role in the study design; collection, analysis, and interpretation of data; in the writing of the paper; or in the decision to submit the paper for publication.

SB is a director of Affigo CIC, a not-for-profit social enterprise company spun out of the University of Manchester in December 2015 to enable access to social enterprise funding and to promote ClinTouch, a symptom-monitoring app, to the NHS and public sector.

 Robinson D Woerner MG Alvir JM Bilder R Goldman R Geisler S Koreen A Sheitman B Chakos M Mayerhoff D Lieberman JA Predictors of relapse following response from a first episode of schizophrenia or schizoaffective disorder Arch Gen Psychiatry 1999 03 56 3 241 7 10078501 Alvarez-Jimenez M Priede A Hetrick SE Bendall S Killackey E Parker AG McGorry PD Gleeson JF Risk factors for relapse following treatment for first episode psychosis: a systematic review and meta-analysis of longitudinal studies Schizophr Res 2012 08 139 1-3 116 28 10.1016/j.schres.2012.05.007 22658527 S0920-9964(12)00259-9 Almond S Knapp M Francois C Toumi M Brugha T Relapse in schizophrenia: costs, clinical outcomes and quality of life Br J Psychiatry 2004 04 184 346 51 10.1192/bjp.184.4.346 15056580 Andrew A Knapp M McCrone P Parsonage M Trachtenberg M London School of Economics and Political Science 2012 2018-07-12 London Rethink Mental Illness Effective interventions in Schizophrenia: The economic case http://www.lse.ac.uk/LSEHealthAndSocialCare/pdf/LSE-economic-report-FINAL-12-Nov.pdf Wu EQ Birnbaum HG Shi L Ball DE Kessler RC Moulis M Aggarwal J The economic burden of schizophrenia in the United States in 2002 J Clin Psychiatry 2005 09 66 9 1122 9 16187769 Maclean C University of Glasgow 2008 2018-07-12 Glasgow An interpretative phenomenological analysis of service users' perspectives and experiences of relapse in psychosis http://theses.gla.ac.uk/452/1/2008Maclean1DClinSci.pdf Gumley A Schwannauer M Staying Well After Psychosis: A Cognitive Interpersonal Approach to Recovery and Relapse Prevention 2006 Chichester John Wiley and Sons Wiersma D Nienhuis FJ Slooff CJ Giel R Natural course of schizophrenic disorders: a 15-year followup of a Dutch incidence cohort Schizophr Bull 1998 24 1 75 85 9502547 Appleby L Suicide in psychiatric patients: risk and prevention Br J Psychiatry 1992 12 161 749 58 10.1192/bjp.161.6.749 1483160 Iqbal Z Birchwood M Chadwick P Trower P Cognitive approach to depression and suicidal thinking in psychosis. 2. Testing the validity of a social ranking model Br J Psychiatry 2000 12 177 522 8 11102327 S0007125000156041 Birchwood M Iqbal Z Chadwick P Trower P Cognitive approach to depression and suicidal thinking in psychosis. 1. Ontogeny of post-psychotic depression Br J Psychiatry 2000 12 177 516 21 11102326 S000712500015603X Gumley A O'Grady M McNay L Reilly J Power K Norrie J Early intervention for relapse in schizophrenia: Results of a 12-month randomized controlled trial of cognitive behavioural therapy Psychol Med 2003 33 3 419 31 PMID: 16719982 Herz MI Lamberti JS Mintz J Scott R O'Dell SP McCartan L Nix G A program for relapse prevention in schizophrenia: a controlled study Arch Gen Psychiatry 2000 03 57 3 277 83 10711914 Lee SH Choi TK Suh S Kim YW Kim B Lee E Yook KH Effectiveness of a psychosocial intervention for relapse prevention in patients with schizophrenia receiving risperidone via long-acting injection Psychiatry Res 2010 02 28 175 3 195 9 10.1016/j.psychres.2008.06.043 20022120 S0165-1781(08)00234-5 Eisner E Drake R Barrowclough C Assessing early signs of relapse in psychosis: review and future directions Clin Psychol Rev 2013 07 33 5 637 53 10.1016/j.cpr.2013.04.001 23628908 S0272-7358(13)00051-2 Fusar-Poli P Bonoldi I Yung AR Borgwardt S Kempton MJ Valmaggia L Barale F Caverzasi E McGuire P Predicting psychosis: meta-analysis of transition outcomes in individuals at high clinical risk Arch Gen Psychiatry 2012 03 69 3 220 9 10.1001/archgenpsychiatry.2011.1472 22393215 69/3/220 Schultze-Lutter F Klosterkötter J Picker H Steinmeyer E Ruhrmann S Predicting first-episode psychosis by basic symptom criteria Clin Neuropsychiatry 2007 02 4 1 11 22 Bechdolf A Schultze-Lutter F Klosterkötter J Self-experienced vulnerability, prodromal symptoms and coping strategies preceding schizophrenic and depressive relapses Eur Psychiatry 2002 11 17 7 384 93 12547304 S0924933802006983 Eisner E Drake R Lobban F Bucci S Emsley R Barrowclough C Comparing early signs and basic symptoms as methods for predicting psychotic relapse in clinical practice Schizophr Res 2018 02 192 124 130 10.1016/j.schres.2017.04.050 28499766 S0920-9964(17)30259-1 PMC5821684 Schultze-Lutter F Schimmelmann Bg Early detection and treatment of psychosis: the Bern child and adolescent psychiatric perspective Adv Psychiatry 2014 09 2014 1 16 10.1155/2014/365283 Gaebel W Riesbeck M Are there clinically useful predictors and early warning signs for pending relapse? Schizophr Res 2014 02 152 2-3 469 77 10.1016/j.schres.2013.08.003 23978776 S0920-9964(13)00438-6 Gaebel W Riesbeck M Revisiting the relapse predictive validity of prodromal symptoms in schizophrenia Schizophr Res 2007 09 95 1-3 19 29 10.1016/j.schres.2007.06.016 17630253 S0920-9964(07)00270-8 Tait A McNay L Gumley A O'Grady M The development and implementation of an individualised early signs monitoring system in the prediction of relapse in schizophrenia J Ment Health 2009 07 06 11 2 141 53 10.1080/096382301200041551 Španiel F Hrdlička J Novák T Kožený J Höschl C Mohr P Motlová LB Effectiveness of the information technology-aided program of relapse prevention in schizophrenia (ITAREPS): a randomized, controlled, double-blind study J Psychiatr Pract 2012 07 18 4 269 80 10.1097/01.pra.0000416017.45591.c1 22805901 00131746-201207000-00005 Ben-Zeev D Kaiser SM Brenner CJ Begale M Duffecy J Mohr DC Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia Psychiatr Rehabil J 2013 12 36 4 289 96 10.1037/prj0000019 24015913 2013-32035-001 PMC4357360 Palmier-Claus JE Ainsworth J Machin M Barrowclough C Dunn G Barkus E Rogers A Wykes T Kapur S Buchan I Salter E Lewis SW The feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application BMC Psychiatry 2012 12 172 10.1186/1471-244X-12-172 23075387 1471-244X-12-172 PMC3502449 Ainsworth J Palmier-Claus JE Machin M Barrowclough C Dunn G Rogers A Buchan I Barkus E Kapur S Wykes T Hopkins RS Lewis S A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations J Med Internet Res 2013 15 4 e60 10.2196/jmir.2328 23563184 v15i4e60 PMC3636800 Bucci S Barrowclough C Ainsworth J Machin M Morris R Berry K Emsley R Lewis S Edge D Buchan I Haddock G Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis Schizophr Bull 2018 03 16 44 5 1070 80 10.1093/schbul/sby032 29566206 4939353 Ben-Zeev D Brenner CJ Begale M Duffecy J Mohr DC Mueser KT Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia Schizophr Bull 2014 11 40 6 1244 53 10.1093/schbul/sbu033 24609454 sbu033 PMC4193714 Meyer N Kerz M Folarin A Joyce DW Jackson R Karr C Dobson R MacCabe J Capturing rest-activity profiles in Schizophrenia using wearable and mobile technologies: development, implementation, feasibility, and acceptability of a remote monitoring platform JMIR Mhealth Uhealth 2018 10 30 6 10 e188 10.2196/mhealth.8292 30377146 v6i10e188 PMC6234334 Ben-Zeev D Brian R Wang R Wang W Campbell AT Aung MS Merrill M Tseng VW Choudhury T Hauser M Kane JM Scherer EA CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse Psychiatr Rehabil J 2017 09 40 3 266 275 10.1037/prj0000243 28368138 2017-14914-001 PMC5593755 Barnett I Torous J Staples P Sandoval L Keshavan M Onnela J Relapse prediction in schizophrenia through digital phenotyping: a pilot study Neuropsychopharmacology 2018 07 43 8 1660 6 10.1038/s41386-018-0030-z 29511333 10.1038/s41386-018-0030-z PMC6006347 Kumar D Tully LM Iosif A Zakskorn LN Nye KE Zia A Niendam TA A mobile health platform for clinical monitoring in early psychosis: implementation in community-based outpatient early psychosis care JMIR Ment Health 2018 02 27 5 1 e15 10.2196/mental.8551 29487044 v5i1e15 PMC5849797 Niendam TA Tully LM Iosif A Kumar D Nye KE Denton JC Zakskorn LN Fedechko TL Pierce KM Enhancing early psychosis treatment using smartphone technology: a longitudinal feasibility and validity study J Psychiatr Res 2018 01 96 239 46 10.1016/j.jpsychires.2017.10.017 29126059 S0022-3956(17)30545-9 Kim S Lee G Yu H Jung E Lee J Kim S Kim J Shin I Yoon J Development and feasibility of smartphone application for cognitive-behavioural case management of individuals with early psychosis Early Interv Psychiatry 2017 05 18 12 6 1087 93 10.1111/eip.12418 28516480 Palmier-Claus JE Rogers A Ainsworth J Machin M Barrowclough C Laverty L Barkus E Kapur S Wykes T Lewis SW Integrating mobile-phone based assessment for psychosis into people's everyday lives and clinical care: a qualitative study BMC Psychiatry 2013 13 34 10.1186/1471-244X-13-34 23343329 1471-244X-13-34 PMC3562160 Forchuk C Reiss JP O'Regan T Ethridge P Donelle L Rudnick A Client perceptions of the mental health engagement network: a qualitative analysis of an electronic personal health record BMC Psychiatry 2015 10 14 15 250 10.1186/s12888-015-0614-7 26467210 10.1186/s12888-015-0614-7 PMC4606496 Aref-Adib G O'Hanlon P Fullarton K Morant N Sommerlad A Johnson S Osborn D A qualitative study of online mental health information seeking behaviour by those with psychosis BMC Psychiatry 2016 07 16 232 10.1186/s12888-016-0952-0 27400874 10.1186/s12888-016-0952-0 PMC4940927 Huerta-Ramos E Escobar-Villegas MS Rubinstein K Unoka ZS Grasa E Hospedales M Jääskeläinen E Rubio-Abadal E Caspi A Bitter I Berdun J Seppälä J Ochoa S Fazekas K M-RESIST Group Corripio I Usall J Measuring users' receptivity toward an integral intervention model based on mHealth solutions for patients with treatment-resistant schizophrenia (m-RESIST): a qualitative study JMIR Mhealth Uhealth 2016 09 28 4 3 e112 10.2196/mhealth.5716 27682896 v4i3e112 PMC5062002 McClelland GT Fitzgerald M A participatory mobile application (app) development project with mental health service users and clinicians Health Educ J 2018 06 05 001789691877379 10.1177/0017896918773790 Bucci S Morris R Berry K Berry N Haddock G Barrowclough C Lewis S Edge D Early psychosis service user views on digital technology: qualitative analysis JMIR Ment Health 2018 10 31 5 4 e10091 10.2196/10091 30381280 v5i4e10091 PMC6236205 Yardley L Morrison L Bradbury K Muller I The person-based approach to intervention development: application to digital health-related behavior change interventions J Med Internet Res 2015 17 1 e30 10.2196/jmir.4055 25639757 v17i1e30 PMC4327440 Bhugra D Tasman A Pathare S Priebe S Smith S Torous J Arbuckle MR Langford A Alarcón RD Chiu HF First MB Kay J Sunkel C Thapar A Udomratn P Baingana FK Kestel D Ng RMK Patel A Picker LD McKenzie KJ Moussaoui D Muijen M Bartlett P Davison S Exworthy T Loza N Rose D Torales J Brown M Christensen H Firth J Keshavan M Li A Onnela J Wykes T Elkholy H Kalra G Lovett KF Travis MJ Ventriglio A The WPA-Lancet Psychiatry Commission on the future of psychiatry Lancet Psychiatry 2017 10 4 10 775 818 10.1016/S2215-0366(17)30333-4 28946952 S2215-0366(17)30333-4 Biagianti B Hidalgo-Mazzei D Meyer N Developing digital interventions for people living with serious mental illness: perspectives from three mHealth studies Evid Based Ment Health 2017 11 20 4 98 101 10.1136/eb-2017-102765 29025862 eb-2017-102765 PMC5750413 Killikelly C He Z Reeder C Wykes T Improving adherence to web-based and mobile technologies for people with psychosis: systematic review of new potential predictors of adherence JMIR Mhealth Uhealth 2017 07 20 5 7 e94 10.2196/mhealth.7088 28729235 v5i7e94 PMC5544896 Berry N Lobban F Emsley R Bucci S Acceptability of interventions delivered online and through mobile phones for people who experience severe mental health problems: a systematic review J Med Internet Res 2016 05 31 18 5 e121 10.2196/jmir.5250 27245693 v18i5e121 PMC4908305 Gale NK Heath G Cameron E Rashid S Redwood S Using the framework method for the analysis of qualitative data in multi-disciplinary health research BMC Med Res Methodol 2013 09 18 13 117 10.1186/1471-2288-13-117 24047204 1471-2288-13-117 PMC3848812 Schultze-Lutter F Addington J Ruhrmann S Klosterkötter J Schizophrenia Proneness Instrument: Adult Version (SPI-A) 2007 Rome, Italy Giovanni Fioriti Editore Kay SR Fiszbein A Opler LA The positive and negative syndrome scale (PANSS) for schizophrenia Schizophr Bull 1987 13 2 261 76 3616518 Addington D Addington J Schissel B A depression rating scale for schizophrenics Schizophr Res 1990 3 4 247 51 2278986 0920-9964(90)90005-R Birchwood M Smith J Macmillan F Hogg B Prasad R Harvey C Bering S Predicting relapse in schizophrenia: the development and implementation of an early signs monitoring system using patients and families as observers, a preliminary investigation Psychol Med 1989 08 19 3 649 56 2798634 Gumley AI MacBeth A Reilly JD O'Grady M White RG McLeod H Schwannauer M Power KG Fear of recurrence: results of a randomized trial of relapse detection in schizophrenia Br J Clin Psychol 2015 03 54 1 49 62 10.1111/bjc.12060 25040487 Eisner E Drake RJ Berry N Barrowclough C Bucci S Emsley R 2018-07-12 Feasibility and Acceptability of a Smartphone App to Assess Early Warning Signs of Psychosis Relapse https://clinicaltrials.gov/ct2/show/NCT03558529 First MB Spitzer RL Williams JB Structured Clinical Interview for DSM-IV Axis I Disorders, Clinician Version (SCID-CV) 1996 Washington, DC American Psychiatric Press, Inc Tait L Birchwood M Trower P Adapting to the challenge of psychosis: personal resilience and the use of sealing-over (avoidant) coping strategies Br J Psychiatry 2004 11 185 410 5 10.1192/bjp.185.5.410 15516550 185/5/410 Ajzen I Fishbein M Attitude-behavior relations: a theoretical analysis and review of empirical research Psychol Bull 1977 84 5 888 918 10.1037/0033-2909.84.5.888 Brown GW Birley JL Crises and life changes and the onset of schizophrenia J Health Soc Behav 1968 09 9 3 203 14 5676853 Ventura J Nuechterlein KH Lukoff D Hardesty JP A prospective study of stressful life events and schizophrenic relapse J Abnorm Psychol 1989 11 98 4 407 11 2592674 Birchwood M Spencer E McGovern D Schizophrenia: early warning signs Adv Psychiatr Treat 2018 01 02 6 2 93 101 10.1192/apt.6.2.93 Berry N Bucci S Lobban F Use of the internet and mobile phones for self-management of severe mental health problems: qualitative study of staff views JMIR Ment Health 2017 11 01 4 4 e52 10.2196/mental.8311 29092809 v4i4e52 Hetrick SE Robinson J Burge E Blandon R Mobilio B Rice SM Simmons MB Alvarez-Jimenez M Goodrich S Davey CG Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-harm JMIR Ment Health 2018 01 23 5 1 e9 10.2196/mental.9041 29362208 v5i1e9 PMC5801516 Saunders KE Bilderbeck AC Panchal P Atkinson LZ Geddes JR Goodwin GM Experiences of remote mood and activity monitoring in bipolar disorder: a qualitative study Eur Psychiatry 2017 03 41 115 121 10.1016/j.eurpsy.2016.11.005 28135594 S0924-9338(16)30171-7 Nicholas J Proudfoot J Parker G Gillis I Burckhardt R Manicavasagar V Smith M The ins and outs of an online bipolar education program: a study of program attrition J Med Internet Res 2010 12 5 e57 10.2196/jmir.1450 21169169 v12i5e57 PMC3057316 Proudfoot J Parker G Manicavasagar V Hadzi-Pavlovic D Whitton A Nicholas J Smith M Burckhardt R Effects of adjunctive peer support on perceptions of illness control and understanding in an online psychoeducation program for bipolar disorder: a randomised controlled trial J Affect Disord 2012 12 15 142 1-3 98 105 10.1016/j.jad.2012.04.007 22858215 S0165-0327(12)00251-0 Simon GE Ludman EJ Goodale LC Dykstra DM Stone E Cutsogeorge D Operskalski B Savarino J Pabiniak C An online recovery plan program: can peer coaching increase participation? Psychiatr Serv 2011 06 62 6 666 9 10.1176/appi.ps.62.6.666 21632737 62/6/666 PMC4176611 Bjerke TN Kummervold PE Christiansen EK Hjortdahl P “It Made Me Feel Connected”—an exploratory study on the use of mobile SMS in follow-up care for substance abusers Journal of Addictions Nursing 2008 19 4 195 200 10.1080/10884600802504735 Baker AL Turner A Beck A Berry K Haddock G Kelly PJ Bucci S Telephone-delivered psychosocial interventions targeting key health priorities in adults with a psychotic disorder: systematic review Psychol Med 2018 05 25 1 21 10.1017/S0033291718001125 29798730 S0033291718001125 Kasckow J Felmet K Appelt C Thompson R Rotondi A Haas G Telepsychiatry in the assessment and treatment of schizophrenia Clin Schizophr Relat Psychoses 2014 04 8 1 21 27A 10.3371/CSRP.KAFE.021513 23428781 9684Q297P151LW04 PMC4132656 Fullwood C Quinn S Kaye LK Redding C My virtual friend: A qualitative analysis of the attitudes and experiences of Smartphone users: Implications for Smartphone attachment Computers in Human Behavior. 2017 Vol.75 347 55 2017-32485-036 Clarke J Proudfoot J Whitton A Birch M Boyd M Parker G Manicavasagar V Hadzi-Pavlovic D Fogarty A Therapeutic alliance with a fully automated mobile phone and web-based intervention: secondary analysis of a randomized controlled trial JMIR Ment Health 2016 02 25 3 1 e10 10.2196/mental.4656 26917096 v3i1e10 PMC4786687 Berry K Salter A Morris R James S Bucci S Assessing therapeutic alliance in the context of mhealth interventions for mental health problems: development of the Mobile Agnew Relationship Measure (mARM) questionnaire J Med Internet Res 2018 04 19 20 4 e90 10.2196/jmir.8252 29674307 v20i4e90 PMC5934536 Agnew-Davies R Stiles WB Hardy GE Barkham M Shapiro DA Alliance structure assessed by the Agnew Relationship Measure (ARM) Br J Clin Psychol 1998 05 37 Pt 2 155 72 9631204 Stevens S Sin J Implementing a self-management model of relapse prevention for psychosis into routine clinical practice J Psychiatr Ment Health Nurs 2005 08 12 4 495 501 10.1111/j.1365-2850.2005.00840.x 16011506 JPM840 McAllister M Dunn G Payne K Davies L Todd C Patient empowerment: the need to consider it as a measurable patient-reported outcome for chronic conditions BMC Health Serv Res 2012 12 157 10.1186/1472-6963-12-157 22694747 1472-6963-12-157 PMC3457855 Tait L Birchwood M Trower P Predicting engagement with services for psychosis: insight, symptoms and recovery style Br J Psychiatry 2003 02 182 123 8 12562739 Proudfoot JG Parker GB Benoit M Manicavasagar V Smith M Gayed A What happens after diagnosis? Understanding the experiences of patients with newly-diagnosed bipolar disorder Health Expect 2009 06 12 2 120 9 10.1111/j.1369-7625.2009.00541.x 19538647 HEX541 PMC5060482 Cohen J Edmunds J Brodman D Benjamin C Kendall P Using self-monitoring: implementation of collaborative empiricism in cognitive-behavioral therapy Cogn Behav Pract 2013 11 20 4 419 428 10.1016/j.cbpra.2012.06.002 Korotitsch WJ Nelson-Gray RO An overview of self-monitoring research in assessment and treatment Psychol Assess 1999 11 11 4 415 25 10.1037/1040-3590.11.4.415 Bell IH Lim MH Rossell SL Thomas N Ecological momentary assessment and intervention in the treatment of psychotic disorders: a systematic review Psychiatr Serv 2017 11 01 68 11 1172 81 10.1176/appi.ps.201600523 28669284 Lewis S Fraccaro P Ainsworth J Machin M Sanders C He Z Whelan P Stockton-Powdrell C Hopkins R Wykes T A randomised controlled trial of smartphone active symptom monitoring in psychosis Schizophr Bull 2018 03 44 Suppl 1 S106 10.1093/schbul/sby015.261 4939353 Depp CA Moore RC Perivoliotis D Granholm E Technology to assess and support self-management in serious mental illness Dialogues Clin Neurosci 2016 06 18 2 171 83 27489457 PMC4969704 Naslund JA Marsch LA McHugo GJ Bartels SJ Emerging mHealth and eHealth interventions for serious mental illness: a review of the literature J Ment Health 2015 08 19 24 5 321 32 10.3109/09638237.2015.1019054 26017625 Hatch A Hoffman JE Ross R Docherty JP Expert consensus survey on digital health tools for patients with serious mental illness: optimizing for user characteristics and user support JMIR Ment Health 2018 06 12 5 2 e46 10.2196/mental.9777 29895514 v5i2e46