Borderline personality disorder (BPD) is a psychological disorder that influences all domains of life. It is characterized by a pervasive pattern of unstable relations, a distorted self-image, and profound difficulties in regulation of one’s emotions [1]. Self-harming behaviors are common [2-5]. A lack of emotional awareness or the ability to timely recognize the onset of emotions and their increasing or decreasing intensity plays a role in emotional instability and dysregulated and self-harming behaviors in BPD [6-11]. Patients with BPD seem to especially have less focus on the level of emotional arousal than controls [12].

Perception of internal bodily states is found to be of significant importance to subjective experience, awareness, labeling, and understanding of emotional processes [13-16]. Interestingly, people with low emotional awareness, in general, do respond to emotional triggers physiologically, and to a certain extent, behaviorally, but lack the experience of the emotion—the feeling [14,17,18]. Therefore, available evidence suggests that treatment of low emotional awareness in BPD should focus on bodily signals and could very well involve biofeedback [6,12-23]. Although there are indications that emotional awareness can improve with psychological interventions [24], development and testing these interventions is in its infancy [25,26]. From literature and information gathered from patients and professionals, we identified a need to improve the treatment of low emotional awareness in BPD [19,24,27].

We started a project to develop a biosensor-informed e-coaching app on emotional awareness in the challenging environment of a psychiatric ward for patients with severe BPD. We gave the app the name Sense-IT, as it refers to both its intended purpose for patients (sense it) and its technological nature (IT: information technology). Importantly, it should help its users to learn to better recognize changes in their physiological and, with that, emotional arousal and thus increase emotional awareness. To the best of our knowledge, this project is one of the first in providing ambulatory biofeedback to this group of users. We previously developed an initial prototype [28]. The aim of this study was to complete the next step and deliver a working version of the Sense-IT app that is deemed useful and usable by 3 important groups of stakeholders: patients with BPD and low emotional awareness, mental health professionals working with these patients, and experts on user-centered design (UCD).

We decided to design our app for wearable technology that comes equipped with essential biosensor technology that is widely available for consumers, is affordable, and runs on a mature operating system (OS) that offers easy-to-use app programming interfaces (APIs) by which one can develop native apps and access sensors directly. After a small preliminary study with main stakeholders and users on acceptable, usable, and nonstigmatizing hardware (unpublished), we decided to use a smartwatch and mobile phone. Next, we decided on developing an app for Google’s Android (including Wear) OS. As a mature, easy-to-access OS, it has a broad base of support by manufacturers, software developers, and users alike. Most smartwatches (at the time) came equipped with a photoplethysmogram (PPG) sensor and accelerometer. The PPG sensor is used for measuring heart rate (HR). After studying the literature on the use of biosignals for affect detection, we built an algorithm that takes the HR data of the user to calculate a physiological correlate of emotional arousal (PCEA). HR, similar to HR variability (HRV) and electrodermal activity (EDA), is triggered by the sympathetic nervous system [29,30]. Typical consumer wrist-based HR monitors provide a fairly accurate measurement of HR even when deployed during physical activity and movement [31].

The first version of the app was built by one of the authors (RK) and 2 graduate students in Human Media Interaction. The outcomes of previous design cycles served as the main sources of input for programming (refer to the study by Derks et al [28]). Visual design was kept low in complexity so that the app’s graphical user interface (GUI) is easy to interpret by the user yet remains nondescript to the environment (see Figure 1). To cater to the expressed need for discreet and unobtrusive coaching, audio signals were deliberately not implemented. Instead, the app was programmed to give tactile feedback via the vibration motor in the smartwatch. Users have direct access to a proxy of the current level of the PCEA via continuous HR measurements on both the smartwatch and mobile phone. On the smartwatch, users have the option to choose from 4 different watch faces. The mobile phone app provides an overview of the recent measurements. At higher levels of arousal (stages 4 and 5 out of 5), the user receives a textual prompt, intended to stimulate deliberate reflection on one’s current status of emotional arousal. See Figure 2 for an impression.

An option to manually export recorded data to a server was also implemented. With this option, a primary user is able to transfer his or her data to a server. These data can then be accessed via an internet browser through a secured gateway. The data can be presented in raw form or plotted to provide a graph depicting the changes in PCEA over time. This option was built in to increase the number of ways the data could be used to support therapy. For example, it could provide an opportunity for the patients to get a better overview of the changes in their arousal over time or to let their therapists have a view (if they were granted access to the secured data by the patient). Although this functionality was built in, it could not be tested at the clinic because of limitations in the technical infrastructure at site and ethical limitations concerning the sending of personal data to servers outside the clinic. Figure 3 gives an overview of the setup of the app.

The first 2 cycles took place within an inpatient psychotherapeutic setting of Scelta, GGNet, a mental health care provider situated in the east of the Netherlands. Testing during the third cycle took place in an office setting within the facilities of the University of Twente.

Scelta is a specialized division within GGNet, one of the larger mental health care providers in the Netherlands. Scelta provides psychiatric treatment of severe personality disorders. At their inpatient clinic, Scelta offers several psychotherapeutic treatment programs for personality disorders. The testing of the app with patients and therapists was within the dialectical behavioral treatment (DBT) [35] unit of the clinic. Here, multidisciplinary group DBT [35] is given to a maximum of 27 inpatients at a time. All patients are diagnosed with 1 or more personality disorders—the majority of them having BPD diagnosed as main disorder. Four days a week, during working hours, patients receive multiple forms of psychological treatment. Medical treatment can be part of the treatment but serves a subsidiary role. On average, duration of treatment within the DBT program is 9 to 12 months. After that, most patients are referred for further (outpatient) care.

Data from interviews, questionnaires, and task scenarios were transcribed via a transcription program (F4). Additional remarks made by participants during or in between hands-on testing were also registered. Qualitative analysis consisted of a combination of content and thematic analysis [36]. Comments were categorized by applying constant comparison, comparing statements numerically and content wise. Usability problems and needs were listed and subsequently grouped, after which a central theme was allocated to each group. This was done by 2 of the authors (YD and RK), of which one has a background in psychology and the other in computer science. Final categorization reflects consensus after comparison and discussion of the interpretation processes of both authors. The following main themes were identified: (1) technology, (2) user interface and interaction, and (3) functionality (and ethics). Next, within each set of results per group, inductive thematic analysis was used to identify whether, and, if so, which specific values regarding the acceptance and adoption of the app were present. Next to the qualitative analyses, individual and overall scores of the system usability scale (SUS) [37] (used in cycles 1 and 2) and the cognitive walkthrough (used in cycle 3) were quantitatively analyzed.

The study was granted approval by both a nationwide operating ethical commission and a local ethical commission associated with the university. Participation in the study was on a voluntary basis and only after informed consent. Patients undergoing clinical treatment at the treatment center and therapists working in the center were eligible if they were willing to actively participate. All participants could withdraw from the study at any time with no further obligations. In addition, therapists of the patients held the right to anytime exclude or withdraw patients from participating in the study if they judged a patient’s participation potentially detrimental for his or her well-being or unwanted or inappropriate in any other way. There was no financial reward or other incentive for participating.

As the cycles were carried out as three consecutive studies, each with its own participants, procedure, and materials, the remainder of the method section will be discussed per cycle. Next, results will also be presented per cycle, in paragraphs representing the main themes that resulted from the thematic analysis. Multimedia Appendix 1 pprovides a schematic overview of the thematic ordering. It lists all usability issues and user needs that were identified by 1, 2, or all user groups grouped by the main theme and presents the applied—or planned—solutions and adjustments to the Sense-IT app after the 3 cycles of usability testing. The result section concludes with the presentation of the current version of the Sense-IT app.

The SUS is a commonly used questionnaire that quickly and reliably assesses the usability of a product [37-40]. The SUS [39,40] comprised 10 statements that are scored on a 5-point scale, ranging from totally disagree to totally agree. It contains statements such as “I thought the system was easy to use” and “I thought there was too much inconsistency.” The SUS yields an overall score of the system usability ranging between 0 and 100, where higher scores indicate better usability. For interpretation of scores, we used the guideline provided by Bangor et al [38].

A total of 2 short self-constructed questionnaires were administered to the patients during the first cycle of testing. The first questionnaire (26 items) contained questions on the use context; the patient’s level of experience with technology; general UX of the app; UX of the interface on mobile phone and smartwatch; the use of prompts; and additional questions regarding privacy, perceived risks, and missing/desired features. See Multimedia Appendix 2. The second questionnaire (16 items) contained questions on the patient’s general UX with the mobile phone and smartwatch interface, the use of the diary/note keeping function, and questions about the option to share data with therapists in the future. See Multimedia Appendix 3.

See the description under cycle 1.

A total of 4 scenarios were written by the research team, based on the functions of the app. Each scenario consisted of a task to be performed on either the mobile phone or smartwatch: (1) add a comment to your latest measurement, (2) change the watch face to another Sense-IT watch face, (3) add a general comment to the timeline (not to a measurement), and (4) let the app stop measuring your HR. With each scenario, the user was asked which steps he or she took to perform the task, what difficulties were encountered while performing the task, and whether the user had further comments or suggestions.

The UCD experts were asked to complete several tasks while thinking out loud about what a primary user (ie, a patient with BPD and low emotional awareness) would do and evaluate if the task at hand would be easily achievable. The tasks were the same as the scenarios for the therapists.

There were 4 additional questions on usability to be answered with yes or no. These questions were as follows: (1) Will the primary user try to achieve the correct effect?, (2) Will the primary user notice that the correct action is available?, (3) Will the primary user associate the correct action with the desired effect?, and, if the user performed the right action, (4) Will the primary user notice that progress is being made toward accomplishment of her goal? As an indication of usability error [41], the average number of yes answers were added up to a score between 0 and 1, with higher scores indicating less errors.

All patients experienced the app as discreet and inconspicuous. All would have liked more data visualization. They stressed it was important that the data visualization should remain neutral for reasons of discreetness while still being intuitively understandable. Regarding the Watch face GUI, users found it important that they could choose from several designs.

Patients stressed that the app should be compatible with their therapies. This need was already addressed, as physiological arousal was represented in a two times five stage process that is compatible with the current user setting (DBT and systems training for emotional predictability and problem solving [3,42]) as well as cognitive behavior therapy (CBT). Patients indicated that other ways of breaking down in levels would still be welcome for the use of the app in other forms of therapy. One of the patients believed that the levels of the physiological scale were representing the subjective emotional states that were used in the main model on emotions in therapy.

The timing and frequency of feedback given by the app (via the vibration motor in the smartwatch) yielded mixed reactions. One patient said the notifications matched the perceived moments of arousal well. For 2 patients, the frequency of notifications was acceptable. One patient received a bit too many notifications, especially since they were accompanied by a vibration of the smartwatch that was noticed by other patients during group therapy. In contrast, another patient received too little notifications in regard to her experienced moments of heightened emotional arousal. She also believed the notifications were sometimes lagging behind with her actual feeling of increases in arousal. The reliability of the app was perceived as average to slightly unreliable. One patient encountered a problem in HR measurement, which was discovered and adjusted the second day.

To prevent confusion of actual, subjective emotional arousal and the PCEA provided by the Sense-IT app, the numerical denoting of the scales was changed to icons consisting of spheres. Here, 1 sphere is equivalent to level 1, and 5 spheres are equivalent to level 5. One watch face was replaced by a new one, as the original one was judged as little attractive.

Next to adjustments in graphical layout, the app was extended with 4 new features. Of 4 features, 2 were based on the feedback received from the primary users, and the other 2 were based on the review of the app by the research team. The first new feature was the option for the user to provide feedback to the system by adding personal notes to each notification of change in PCEA by the Sense-IT app.

The second new feature was the option to add personal notes to a diary that is unrelated to recorded changes in PCEA level. As this is a common element in regular psychological treatments such as CBT, the idea for adding a diary was to enable patients to comment on their experiences over longer periods of time, for example, their morning, whole day, week, instead of adding comments to momentary situations captured in the app. This is also in line with recommendations from a recent review study of mobile health (mHealth) mobile phone apps [43]. Both features were added in response to an expressed need for a memory aid. Such an aid could be used, for example, during appointments with professional caregivers.

The third feature was a settings tab in which several personal values/preferences could be set. The addition of this feature was based on a review of the app by the research team. It concerned the option to manually adjust the mean HR and SD used by the system, as well as an option to alter the sensitivity of the system, that is, decrease or increase the threshold of when a new level was reached. We implemented 3 levels of sensitivity: normal (every change in HR of 1 SD adds or subtracts a PCEA level), low (change of 1.5 SD), and high (change of 0.5 SD). The option to set the sensitivity of the app was implemented to better adjust to the preferred level of feedback received by the user. The option to manually alter the values of mean HR and SD was primarily added to let the researchers override the values set by the system based on the baseline measurement in case the set values were unrepresentative for the user (eg, the user turned out to have an uncommonly low or high average HR during baseline measurement). The perceived lagging of feedback that 1 primary user reported could not be addressed in this stage of development but was scheduled as future work.

A fourth new feature was the ability of the app to also measure and represent PCEA states up to (theoretically) 5 steps below the user’s average HR. The decision to add this feature was made by the research team and was based on current expert theory on emotional (under)arousal in the field of affective neuroscience and psychotherapy [44]. These states were visualized in the app as hollow and/or blue-colored spheres (see Figure 6).

The adjusted version of the app was used in the second iteration. All patients indicated that they experienced the option to add notes to detected changes in PCEA as useful. Of 5 patients, 3 had used the feature on more than 1 occasion. The diary for miscellaneous notes was also perceived as useful by all patients, although none of them had actually used this function.

The representation of PCEA states with below the personal baseline was also regarded as useful: “By adding this feature, the therapeutic goal to reach or maintain an overall lower state of arousal is supported by the app.” Patients suggested that they could practice mindfulness or do an exercise from their relaxation training and see if it had a lowering effect on their level of arousal.

All patients indicated that the use of notifications increased their self-awareness. Although 2 of them indicated that the app initially induced some more stress, for example, by making them become too focused on the feedback from the app and reacting to that, all indicated they quickly got used to being monitored continuously. All indicated they would like to see this app being integrated into their therapy, preferably as an adjunct that could be used in regular face-to-face meetings with their therapists. They all thought the app had positively affected their awareness of their emotions. One patient indicated she got notifications about increased physiological arousal during group therapy, which led her to observe her emotional arousal more closely. She thought this had really helped her to make more out of the session. All participants answered they would have liked to continue wearing the devices and using the app all day.

The general layout of various buttons proved less then optimally intuitive for 2 of the 4 therapists. During the scenarios, they found the option to add a miscellaneous note confusing when asked to add a comment to the latest registered change in PCEA. All indicated that the self-report tab was easy to find and leaving a note was easy to do. Also, ending the measurement of HR was not clear to the 2 previously mentioned therapists. Although the other 2 immediately correctly pressed the yellow switch to stop the measurement, 1 of the other 2 did not manage to find the on/off switch at all.

Furthermore, 2 of the therapists found the graphical representation in spheres convenient and logical. The other 2 therapists, however, did not figure out the meaning of the spheres by themselves. In addition, for both these therapists, it was not clear what the buzzing of the device intended to convey. They did understand how the app worked after it was explained to them by the researchers. One of the 2 therapists who understood the app by intuition commented that the app should be even more graphical: it should present the user with more graphs and figures.

Overall, UCD experts considered it to be easy for users to access information and perform actions but also gave several suggestions for improvement of the app.

On the basis of the analysis of the identified usability problems and user needs (and proposed solutions) gathered after completing all 3 cycles, the Sense-IT app was revised. The app as a whole still consists of a wearable app on a smartwatch and a mobile app on a mobile phone, all implemented using the Android ecosystem. Below, we describe its components and functionalities.

This paper describes the development and usability testing of Sense-IT, a wearable biofeedback app for Android-based devices that can be used in the daily practice of a mental health clinic for personality disorders. The app is meant to support BPD patients in increasing their level of emotional awareness. The app was tested on usability by patients, therapists, and UCD experts.

The concept of a biosensor-informed app for emotional awareness enhancement was appreciated by both patients and therapists; the prototype was judged as promising by all user groups. After 1 iteration in the first cycle of testing with patients, basic functionality of the Sense-IT system was rated as acceptable. This waived the need for a major revision before starting the consecutive rounds of testing with mental health care professionals and UCD experts. Still, results after finishing all 3 testing cycles made clear that the app should be considered a candidate for increased scrutiny and continued improvement [38]. In total, 30 usability problems and/or needs were identified (Multimedia Appendix 1). All 3 user groups brought up usability problems and suggestions for improvement. UCD experts identified most of these usability problems (20 in total). Patients identified 11 issues, and therapists identified 9 issues. There was some overlap between them: 8 issues were mentioned by at least 2 groups. The UCD experts brought to our attention 14 themes or problems that were not mentioned by the other groups, the patients 6, and the therapists 2. Most usability problems and needs could be addressed in the software revision that followed after the 3 cycles were finished. This resulted in the version of the Sense-IT app presented at the end of the results section.

Results of the 3 cycles of testing favor the use of ambulatory biofeedback to improve emotional awareness in patients suffering from BPD and low emotional awareness. To our knowledge, the Sense-IT is one of the first scientifically grounded apps that can be used in clinical research and/or clinical settings for longer periods of time without requiring extensive support by researchers and/or developers. Although it may be a quite simple app from a technological perspective, from a mental health perspective, it is a real innovation. This applies to both the way it was developed as to it being a new way of delivering treatment to patients. The use of consumer technology to enable always available indices of physiological changes could prove to be a relevant addition to existing therapeutic interventions, even if the measurements and algorithms used are relatively crude and simple, and the integrated sensors have limited resolution and less than perfect accuracy. There is an ever-growing number of publications that introduce and discuss concepts of biosensor-informed mHealth interventions on emotional/psychological awareness [45-47]. However, at present, feeling the changes in the body is something patients with mental health problems still mainly have to learn by purely subjective methods. We found 1 recent project in which—nonwearable—technology was used in a mental health setting that aimed to “direct alexithymic persons to reflect on their internal, somatic experiences as a source of information for interpreting and labeling emotional experiences” [15]. Although there are numerous commercial companies and startups that sell products and/or apps that claim to help to raise awareness on—or even directly measure—emotion, emotional arousal, or stress, they generally lack research that validates or supports their relevance for users and/or validity (see the study by Peake et al [48] for just a select number of examples).

Of course, both the app and the study have their limitations. Regarding the choice of hardware, research-oriented hardware that gives access to more advanced or potentially better sources of physiological data such as EDA and/or HRV exists [49]. However, the use of HR as the main cardiovascular parameter for physiological arousal is a defendable option. HR, such as HRV and EDA, is triggered by the sympathetic nervous system [29,30]. Typical consumer wrist-based HR monitors provide a fairly accurate measurement of HR even when deployed during physical activity and movement [31]. Measuring HRV is much more susceptible for producing artifacts under real-world conditions. Even producers of wearable technology such as the E4, which is claimed to be able to measure HRV from PPG, stress that this is only feasible in short scenarios (ie, several minutes) that are free of movement.

We believe that with time, better (consumer) hardware with more and more advanced sensors will become available, as will be the case for more accurate signal processing algorithms that can be used in (validated) wearable devices [50]. Improving hardware or data processing algorithms is not what is at the core of our project. However, the Sense-IT is first in providing a new, stable platform that can be considered a new type of intervention in mental health practice.

Regarding the OS for which the app was built, the app is currently only available on devices running Android and Wear OS. We did not develop a version for any other OS, such as iOS, because in this study, the hardware was provided to the participants. In addition, Android roughly has had a 75% market share worldwide over the last years in contrast to 22% market share for iOS [51].

More participants could have been included to ensure saturation of feedback and overall group representativeness. To gather a relatively high number of relevant remarks and comments without too many duplicates, we included 3 to 5 users per group per iteration. On the basis of the literature, these numbers seem proportionate, although more could have been better [52,53].

Regarding the selected use case scenario, it could be considered a limitation of this study that it is exclusively focused on the use of the app with patients with BPD and low emotional awareness. Of course, use of the app by other patient and nonpatient groups with low emotional awareness seems feasible after context-dependent tailoring [54]. However, if an app works for one of the most challenging groups of users in terms of emotional regulation, it could very well work well for others too [55]. Since the start of the project, researchers from 2 other Dutch health care institutions have joined our group and set up studies with the Sense-IT app within their own settings and their specific patient groups. These studies concern the usefulness, usability, and effect on clinical outcome measures of Sense-IT for patients in forensic psychiatric care with aggression regulation problems, and for adolescents in residential care who have many conflicts because they struggle to detect increasing levels of stress.

Considerable effort was made to ensure trustworthiness of this qualitative research project—as, for example, discussed by Shenton [56]. We believe this to be one of the strengths of this study. Shenton mentions 4 criteria for trustworthiness that were originally formulated by Guba for assessing trustworthiness in naturalistic inquiries [57]: credibility (in preference to internal validity), transferability (in preference to external validity/generalizability), dependability (in preference to reliability), and confirmability (in preference to objectivity). To ensure credibility, we adopted a design science paradigm [58,59] to construct and simultaneously test a scientifically informed approach in designing an mHealth app, using well-recognized research methods. We developed early familiarity with the setting, patients, and therapists. We used different methods and different types of informants. We stimulated honesty in interviewing the participants and used UCD experts to also assess the app. We previously published a detailed report on how this was done and provided a description of the backgrounds, qualifications, and experience of the researchers [28]. To ensure transferability, we have provided ample background data to establish the context of study and gave a detailed description of the phenomenon of interest. As mentioned in this section, we started collaborations with other researchers from other settings to study the app in different environments. To ensure dependability, we applied several overlapping methods and used an iterative design approach when testing with the patients in which the results of the second cycle served as a test of the correct interpretation of the results of the first one. In addition, with the introduction of the EMP framework, we delivered an in-depth methodological description that should allow others to repeat our study. Confirmability should be evident from this and previous publication, in which our work was put up for thorough peer review.

What this study added to the literature is an example of how development of an mHealth app within a clinical mental health setting can be challenging, yet feasible, and that it can result in a stable working prototype of an app. It also shows how the use of the multiple user groups is of added value during design and realization. In this study, the app was not yet tested for clinical effectiveness. Although this is perhaps not so much a limitation of the study, as the usability and stability of the app should be tested before using and testing it as a clinical intervention, it is a question that has to be addressed before use of the app can really be recommended for use as an adjunct in psychotherapies. Such is planned later this year in the same setting as this study took place. Furthermore, a second, graphically more advanced GUI for Sense-IT is currently developed by a dedicated graphic/UX designer to further optimize the UX.

In this study, mHealth development within a clinical mental health setting proved to be challenging, yet feasible and welcomed by targeted users. The Sense-IT app was met with enthusiasm and openness by both patients with BPD and therapists, groups that are both known to be reluctant to embrace technological innovations. The use of the EMP framework and the involvement of multiple user groups proved to be of added value during design and realization, as evidenced by the complementary requirements and perspectives. If the app proves to be effective after further clinical testing, Sense-IT would be one of the first broadly applicable technological interventions in the treatment of BPD—and probably in general mental health care—that is actually new as it is not the next form of talking cure (ie, psychotherapy), medical treatment, or traditional skills or behavioral training. It would support the treatment of BPD by directly addressing one of the most important factors in BPD, namely limited emotional awareness [6]. In general, it could enable patients to take therapy out of the therapist’s office into their lives far easier.