The app was systematically integrated into the daily clinical workflow in a FEPP at the University Clinic Hospital of Valencia, Spain. This FEPP started in 2010 with the objective of improving early detection, evaluation, and personalization of treatment. It covers a total of 330,000 inhabitants included in Area 5 of Valencia city. The incidence of novel psychotic disorders in this area has gradually increased during the 10 years since the program started. Currently, the FEPP in the clinic hospital has a mean of 30 to 35 new patients with psychosis per year.

The implementation of the ReMindCare app into the FEPP and into clinical practice started in October 2018 and is still in use today. In this study, we present the results from the first 19 months of use of the app.

Neither patients nor physicians received any remuneration or compensation for participating in the program or using the app. The use of the app was offered as an extra free service to the patients in the program.

Once patients enrolled in the FEPP, after an interview with their psychiatrist of reference, they were asked to complete some baseline assessments. Subsequently, they were offered the use of the ReMindCare app. The ReMindCare app was described as an extra tool developed by the FEPP that could help them manage their symptoms and help clinicians better understand their illness evolution. The main characteristics of the app were listed. After receiving this information, patients decided whether they were willing to use the app. If they were not interested, they were placed in the TAU group. If patients were interested, they were informed in more detail by an expert clinician about the installation process, characteristics of the app, and ethics and data privacy information.

Patients could use the ReMindCare app to contact their psychiatrist of reference directly in case of symptoms worsening by using the urgent consultation request tab on the home screen of the app. If they clicked the urgent consultation request, their clinician would contact them by phone within 48 hours (patients who did not use the app could call the department of psychiatry at the hospital and be referred to their psychiatrist or attend an urgent care unit). In addition, clinicians contacted patients by phone in response to preset alarms. As a result of these phone calls and the information that patients provided to the clinician, urgent care visits could be scheduled if necessary. With these services, we aimed to improve the detection of early psychotic symptoms and reduce the visits to urgent care units at the hospital, as these prodromal symptoms will be primarily treated by a phone call or in the outpatient services. If patients did not make an urgent consultation request and no preset alarms were set off, they continued with their scheduled clinical appointments.

Furthermore, the use of the ReMindCare app changed the dynamics of the clinical appointment at the outpatient services. Once patients arrived at the clinical appointment, physicians accessed their profile on the ReMindCare’s physician dashboard and used the information provided for patients to guide them through the interview. Clinicians used shared decision making with patients and discussed their responses.

Data were analyzed with the statistical program SPSS Statistics version 22 (IBM Corp). The cohort was divided into two groups: ReMindCare group patients agreed to use the app and used it for at least 1 month; the TAU group patients did not use the app or used it for less than 1 month. To consider that patients in the ReMindCare group had a relapse while using the app, patients had to be actively using the app. Relapses of patients who did not use the app for more than 2 months were not considered as relapses while using the app. Descriptive statistics (mean, standard deviation, frequency, and percentage) were determined, and chi-square test analysis was performed to compare the differences between the ReMindCare group and the TAU group.

The ReMindCare app project received approval from the research ethics committee of the faculty of medicine at the University of Valencia and from the research ethics committee of the Sanitary Research Institute of the University Clinic Hospital of Valencia, Spain.

To protect the data sent by patients, communications to the platform were encrypted with a transport layer security certificate from the Generalitat Valenciana and were sent through the https protocol. The hospital infrastructure is protected through a reverse proxy, which enhances security by establishing a single access point to it and hiding all inner infrastructures. Moreover, the integration of the app into the hospital systems was subjected to Organic Law 3/2018: protection of personal data and digital rights guarantee, December 5th, the Spanish organic law adaptation of the General Data Protection Regulation.

The mean age of the sample was 32.8 (SD 9.4) years, 73% (66/90) were males, 91% (83/90) were White, and 81% (74/90) were single. No significant differences were found between the ReMindCare and TAU groups in any of the sociodemographic information analyzed except for the native country. We found that nearly every immigrant considered for inclusion agreed to use the app (ReMindCare group 19% [10/11], TAU group 3% [1/11]; χ²=4.2, P=.04). Further information regarding sociodemographic analysis of the data is displayed in Table 1.

Significant differences were found between the ReMindCare group and TAU group in some clinical factors. With regard to injectable medication, 32% (10/31) of TAU patients were taking injectable medication, while only 14% (8/59) of the ReMindCare took it (χ²=4.4, P=.04). Every new patient in the FEPP (length of illness: 0-1 year) agreed to use the app (13/90, 22%), and 58% (18/31) of the TAU group had their illness for 6 or more years (χ²=12.3, P=.002). Moreover, the TAU patients showed higher scores on the PANSS N5 and N6 negative subscales and G5 in the general subscales (χ²=12.8, P=.03; χ²=12.9, P=.02; χ²=9.9, P=.01, respectively).

Considering medication, 20% (18/90) of patients were taking injectable medications, 32% (29/90) of the patients suffered from another illness, and 18% (17/90) had a prior suicidal attempt. The mean of the CGI-SI was 4.2 (SD 0.9), the GAF mean=60.7 (SD 10.9), PANSS mean 65.9 (SD 18.8), and PAS mean 10.5 (SD 2.8). Finally, 12% (11/90) of patients were discharged from the FEPP. No significant differences were found between the groups in any of these factors. Moreover, no significant differences were found between the ReMindCare group and TAU group in terms of the number of relapses (χ²=4.3, P=.12), visits to urgent care units (χ²=0.9, P=.61), or the number of hospitalizations (χ²=4.6, P=.10) at baseline. Further clinical information is available in Table 2.

The mean of days using the app was 352.2 (SD 191.2), which corresponds to 11.6 months. The mean of compliance was 84.5 (16.04), and 61.1% of the ReMindCare group had a compliance rate between 85% and 100%.

Of the 59 ReMindCare patients, 31% (18/59) requested an urgent consultation, 20% (12/59) had a relapse while using the app, and 8% (2/59) developed a delusion involving the app and the research group.

After 19 months of intervention, 63% (37/59) of patients continued using the app, while 12% (7/59) stopped using the app because they were discharged from the FEPP and 25% (15/59) opted to stop using ReMindCare. Reasons for discontinuation: 33% (5/15) of patients felt suspicious about technology (among these patients, 4 had a relapse while using the app); 40% (6/15) perceived the app as boring and did not perceive any benefit; and 27% (4/15) of patients left treatment and did not continue in the program. This information is shown in Table 3.

With regard to the clinical outcomes, after 19 months of ReMindCare’s integration into the clinical workflow, only 20% (12/59) of patients from the ReMindCare group had a relapse, while 58% (18/31) of TAU patients had one or more relapses (χ²=13.7, P=.001). Moreover, ReMindCare patients had fewer visits to urgent care units (χ²=7.4, P=.006) and fewer hospitalizations than TAU patients (χ²=4.6, P=.03). Information regarding these clinical outcomes is displayed in Table 4.

The results obtained from these analyses of the first 19 months of ReMindCare use highlight the potential benefits of this eHealth intervention for patients with early psychosis. Patients who used the app not only had fewer relapses than the TAU group, but they also had fewer visits to the urgent care unit and fewer hospitalizations.

Results related to the efficacy of the app are in line with previous results obtained in clinical trials [14-16]. However, as far as we know, this is the first study to identify the benefits of the use of an app as a tool systematically integrated into daily clinical practice in a FEPP.

With regard to the feasibility of the app, no significant differences were found between the ReMindCare group and the TAU group in terms of sociodemographic characteristics except for native country. The feasibility of this intervention aligns with the results obtained in our previous study [8], where we found no differences in terms of sociodemographic characteristics and interest in using eHealth interventions.

With regard to the clinical characteristics of the samples and their impact on the effect of ReMindCare, there were some differences between groups. We found that patients who did not use the app were more likely to be taking injectable medication, have a longer history of illness, and have higher scores on the PANSS N5 and N6 negative subscales and G5 in the general subscales. These results might suggest that the use of ReMindCare was not indicated for chronic patients. However, we did not find differences in other clinical scales such as the CGI-SI, GAF, and PAS scales or even on the PANSS total scale. More importantly, we did not find any differences between groups in terms of baseline relapses, hospitalizations, or visits to urgent care units.

These results are in line with the ones we obtained in our previous study [8], where we found that interest in using eHealth apps was equivalent between chronic and early psychosis patients. In this regard, we suggest that differences obtained in terms of the clinical characteristics of the patients could be more related to the history of treatment than to clinical characteristics. As we found, every new patient who joined the FEPP (length of illness less than 1 year) was interested in using the app (22% of users), while patients who had a longer history of treatment (length of illness more than 6 years) were more likely to reject its use (58% of TAU group). This could highlight the relevance of introducing these new technologies at the very beginning of treatment so early psychosis patients consider these apps to be just another tool included in their daily clinical treatment and not an extra service, especially since our results suggested that use of the app had a significant impact in improving the course of the illness.

Finally, with regard to compliance and engagement with the app, we found that 61% of patients had compliance rates between 85% to 100%. Rates of engagement were also high, as 63% of patients still use the app after almost 1 year. These results of compliance and long-term engagement are contrary to previous studies [20,21] and suggest that the use of an app in a long-term approach is feasible and beneficial.

However, we would like to highlight that 20% of patients had a relapse while using the app and 8% developed a delusion involving the use of the app and the research group. These negative results should be cautiously considered.

Technology could be a major resource to improve the quality of treatments, but as we found in a previous study [8], it can also play an important role as a trigger for psychotic symptoms. In this regard, in a 3-case study in 2011 conducted by Nitzan et al [32], they stated that the use of the internet and computers might contribute to a gradual break with reality and development of psychotic symptoms. They suggested that given that patients with psychotic diagnoses have greater difficulties in filtering and understanding signals and symbols, they are also more likely to misinterpret digital messages. However, no specific studies regarding the potential harms of the use of new technologies have been undertaken until the present.

In our study, we found that the ReMindCare app was related to beneficial clinical effects for the vast majority of patients who used it. However, despite the general positive effects found in this study, there are still some barriers and negative effects that must be taken into consideration. The main barrier found in our study relates to the 34% of the approached patients who did not want to use the app and who also tended to be the more chronic patients. Moreover, the main negative effect we found related to the 8% of patients who developed a delusion involving the app. As a result, we would like to point out that this app is not a panacea to prevent relapses. However, it is clear that the app positively affected the course of the illness, as only 5% of those who relapsed required hospitalization compared with 19% of patients who relapsed in the TAU group.

There were some limitations that must be taken into consideration. First, not every outpatient from the FEPP was eligible for inclusion, as some patients did not have their own smartphone with an internet connection or did not have the ability to use the app or understand it due to language barriers. Developing strategies to prevent digital exclusion should be a priority to ensure that every patient could benefit from these technologies [33]. Second, as a real-world study, this study was not randomized. Despite the groups not differing in the vast majority of clinical or demographic characteristics, there were some factors such as personality that could influence our results.

The main strength of our study was the fact that ReMindCare is the first app that has been systematically integrated into the clinical FEPP workflow. To our knowledge, there are no previous studies that used an app as a tool to improve the daily treatment of patients with early psychosis. All the studies we found were conducted in academic research settings that did not emulate real-world environments [17,34].

Another strength is in regard to the development of the ReMindCare app. First, it was based on two previous studies [2,8] and co-designed with patients [27]. Second, we conducted a pilot study and focus groups to ensure the involvement of both patients and care providers [27] in the design and improvement process of the app.

Finally, we would like to highlight the long-term approach of this intervention. As stated before, ReMindCare is now integrated into clinical practice and it was used for 19 months. These results align with previous studies [16] that found that people with psychosis have the abilities and interest required to engage in long-term eHealth interventions.

As a result of these analyses, we highlighted the benefits that the use of ReMindCare app produced on early psychosis patients in a FEPP. Our aim is to continue improving the app in response to the needs and suggestions provided by patients and clinicians. As Ross et al [22] claimed in their meta-review, in order to ensure the use of these eHealth technologies over time, there are three challenges that should be overcome. First, the apps must be able to adapt to the characteristics of the environment and patients. Second, the apps should be easy to use. Third, the apps should be integrated into clinical practice, adjusting the characteristics of the app in order to ensure it is user-friendly and efficient for patients and clinicians. It is our aim to address these issues to maintain the positive results obtained in this study.

However, we would like to point out a major issue that must guide future eHealth interventions. As stated before, 8% of patients developed a delusion related to the use of the app, 25% of patients deliberately stopped using the app, and 34% of patients approached did not want to use the app in the first place. These results suggest that there are still significant numbers of patients not willing to use eHealth interventions, and there are some patients who could be adversely affected by the use of these technologies. Studying the characteristics of these patients should guide future research in order to ensure that the use of digital technologies only provides benefits to the patients [8].

Finally, we would like to underline that given the exceptional situation that the world is facing at the moment with COVID-19 and in order to address the requirements of interventions that could improve the telematic treatment of patients and prevention of hospital collapses [4,35], ReMindCare could be used as an effective and efficient tool. Since quarantining in Spain began March 13, 2020, patients have not been permitted to come in person to their clinical appointments and have received their clinical evaluations by phone. Since that moment, the use of ReMindCare has been extremely useful to improve the evaluation and adherence of early psychosis patients. However, future analysis will be conducted in regard to this aspect.

As the conclusion of this study, we would like to point out that, to the best of our knowledge, ReMindCare is not only the first app to be integrated into the clinical practice, it is the first eHealth intervention with evidence that it improves the outcomes of early psychosis patients in a real-world care setting.