Published on 15.01.20 in Vol 8, No 1 (2020): January
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/12516, first published Oct 16, 2018.
Mobile Health Technology Interventions for Suicide Prevention: Systematic Review
Background: Digital interventions are proposed as one way by which effective treatments for self-harm and suicidal ideation may be improved and their scalability enhanced. Mobile devices offer a potentially powerful medium to deliver evidence-based interventions with greater specificity to the individual when the intervention is needed. The recent proliferation of publicly available mobile apps designed for suicide prevention underlines the need for robust evidence to promote safe practice.
Objective: This review aimed to examine the effectiveness of currently available mobile health (mHealth) technology tools in reducing suicide-specific outcomes.
Methods: The following databases were searched: Cochrane Central Register of Controlled Trials (The Cochrane Library), MEDLINE, EMBASE, PsycINFO, and relevant sources of gray literature. All published and unpublished randomized controlled trials (RCTs), pseudo-RCTs, and pre-post observational studies that evaluated the effectiveness of mHealth technology in suicide prevention delivered via mobile computing and communication technology were included. Studies were included if they measured at least one suicide outcome variable (ie, suicidal ideation, suicidal intent, nonsuicidal self-injurious behavior, and suicidal behavior). A total of 2 review authors independently extracted data and assessed study suitability, in accordance with the Cochrane Collaboration Risk of Bias Tool, on July 31, 2018. Owing to the heterogeneity of outcomes found across studies, results were not amenable for pooled synthesis, and a meta-analysis was not performed. A narrative synthesis of the available research is presented here.
Results: A total of 7 studies met criteria for inclusion . Four published articles that reported on the effectiveness of the following mobile phone apps were included: iBobbly, Virtual Hope Box, BlueIce, and Therapeutic Evaluative Conditioning. Results demonstrated some positive impacts for individuals at elevated risk of suicide or self-harm, including reductions in depression, psychological distress, and self-harm and increases in coping self-efficacy. None of the apps evaluated demonstrated the ability to significantly decrease suicidal ideation compared with a control condition. In addition, 3 unpublished and recently completed trials also met criteria for inclusion in the review.
Conclusions: Further research is needed to evaluate the efficacy of stand-alone mHealth technology–based interventions in suicide prevention. The small number of studies reported in this review tentatively indicate that such tools may have a positive impact on suicide-specific outcomes. Future mHealth intervention evaluations would benefit from addressing the following 3 main methodological limitations : (1) heterogeneity of outcomes: a lack of standardized measurement of suicide outcomes across studies; (2) ecological validity: the tendency to exclude potential participants because of the elevated suicide risk may reduce generalizability within clinical settings; and (3) app regulation and definition: the lack of a standardized classification system for mHealth intervention type points to the need for better definition of the scope of such technologies to promote safe practice.
Trial Registration: PROSPERO CRD42017072899; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=72899
International Registered Report Identifier (IRRID): RR2-10.2196/resprot.8635
JMIR Mhealth Uhealth 2020;8(1):e12516
More than 800,000 people die by suicide every year globally, accounting for 1.4% of all deaths worldwide . Suicide occurs throughout the lifespan and was the second leading cause of death among people aged 15 to 29 years globally in 2012. In addition, it is estimated that 25 suicide attempts (100-200 for youth) occur for every death by suicide [ ], resulting in more than 400,000 emergency department visits annually in the United States [ ]. Prior suicidal behavior increases the risk of subsequent death by suicide 10- to 60-fold [ - ]. Adolescents with depressive disorders and a history of suicidal behavior are a particularly high-risk group for repeated suicide attempts and suicide [ ]. Prospective studies have attempted to predict which individuals will attempt or die by suicide [ ], and a diverse range of risk factors that correlate with suicidal behavior has been proposed to support the identification of those at elevated risk, such as sleep disturbances [ ], emotion regulation deficits [ ], family history of suicide [ ], and chronic pain and illness [ ].
In a meta-analysis of studies that have attempted to longitudinally predict suicidal thoughts or behavior-related outcomes, Franklin et al  found that prediction was only slightly better than chance for all outcomes, and they highlighted several fundamental changes required in future research. They point toward the proliferation of mobile health (mHealth) technologies as a means by which to capture large datasets and to support the expansion of the research base from a focus on risk factors to risk algorithms. Furthermore, in an attempt to improve the accuracy of suicide estimates, Kristoufek et al [ ] found that estimates drawing on Google search data are significantly better than estimates using previous suicide data alone.
In parallel, suicidology researchers have argued that Ecological Momentary Assessment (EMA)—high-frequency data collection in an individual’s usual environment—provides the potential for the development of temporal, individualized prediction of risk states. Thompson et al  tested the ability of EMA to predict individual symptom change in suicidal ideation in a sample of 35 adults diagnosed with interepisode bipolar disorder. The results showed that EMA with functional linear models substantially increased the accuracy of prediction of study-emergent suicidal ideation. Advances in mHealth technologies provide potential opportunities to operationalize EMA research to support the sensitive and timely identification of those at risk of suicide.
Mobile Health and Suicide Prevention
mHealth is a component of electronic health (eHealth). The Global Observatory for eHealth defines mHealth as “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants (PDAs) and other wireless devices” . According to the World Health Organization (WHO), “mHealth involves the use and capitalization on a mobile phone’s core utility of voice and short messaging service (SMS) as well as more complex functionalities and applications including general packet radio service (GPRS), third and fourth generation mobile telecommunications (3G and 4G systems), global positioning system (GPS), and Bluetooth® technology” [ ]. mHealth programs and interventions use mHealth technology for a range of functions from data collection tools for health care professionals and clinical decision support systems to supporting health behavior change and disease management by patients in the community.
Although effective face-to-face treatments for self-harm and suicidal ideation are available [- ], access to effective psychotherapeutic options can be limited. Indeed, negative associations have been found between the availability of mental health services per capita and suicide rates in a number of countries [ ]. Stigma and geographical isolation are 2 of the major barriers to help seeking for individuals at risk of suicide [ ]. Recent advances in mHealth technology could address these barriers by directing individuals at risk of suicide, who would not otherwise seek help, to access appropriate evidence-based online programs or traditional mental health services [ ]. The use of digital technology has been found to be beneficial in the delivery of Web-based suicide prevention interventions [ ]. Furthermore, a survey in a psychiatric outpatient setting reported that 69% of respondents and 80% of those aged 45 years or younger indicated a desire to use a mobile app to track their mental health [ ]. Public health services are being encouraged to harness digital technology to enhance and support psychological health [ , ]. Digital interventions, in general, have been proposed as one way by which the scalability of effective treatments for self-harm and suicidal ideation may be improved [ , ].
However, the regulation of such technologies and the identification of boundaries in terms of their clinical utility require further attention . In the United States, the Food and Drug Administration (FDA) has announced a precertification program, where app makers are preapproved to release FDA-approved health apps based on fulfilling certain quality control criteria. With more than 250,000 mHealth apps currently available, at least 10,000 of which target mental health conditions [ ], the development of this technology is occurring at a faster rate than the evidence base needed to inform it. Mental health was the most common focus of disease-specific mobile apps in 2015, constituting 29% of all chronic condition management. Recent meta-analyses of apps aimed at the management of depression [ ], anxiety disorders [ ], and self-harm [ ] report similar results, with a small evidence base derived from often heterogeneous pilot studies. With thousands of mental health apps readily available through Apple or Google marketplaces, finding a useful tool supported by robust evidence presents a considerable challenge to the individual at risk [ ] and to clinicians wishing to use such technology as part of their work.
Despite the motivation to use mHealth technologies, there is a dearth of specific outcomes data on the efficacy of mHealth technology interventions on suicide outcomes. In 2014, Christensen et al  conducted a review of the literature on eHealth and suicide. Most eHealth interventions identified in their search were Web based as opposed to mobile based. The researchers concluded that there is some evidence to suggest that suicide interventions via the Web may be effective, but only if they target suicidal content specifically, as opposed to the associated symptoms of depression through cognitive behavioral therapy (CBT). Larsen et al [ ] carried out a systematic assessment of mobile phone tools for suicide prevention, screening, and reviewing app content. The researchers concluded that many suicide prevention apps were available, some of which provided comprehensive evidence-based support. Apps with potentially harmful content were also identified.
As digital technology may be particularly attractive for young people, it is essential that mHealth technology and mobile phone apps, in particular, are subject to research evaluation  and co-designed with people who have lived experience [ ]. Donker et al [ ] found that mental health apps evaluated in randomized controlled trials (RCTs) were not publicly available, whereas those with no research evidence were publicly available. In addition, mobile apps that presented harmful content were also identified. Perry et al [ ] conducted a systematic review of online and mobile psychosocial suicide prevention interventions for adolescents and young adults. The researchers searched 4 major psychological databases for interventions that explicitly targeted suicidality using a mobile, computer, or Web-based app for individuals aged between 12 and 25 years. However, only 1 study met the author’s inclusion criteria. Witt et al [ ] reviewed the effectiveness of online and mobile apps (digital interventions) for the self-management of suicidal ideation and self-harm. They identified 14 nonoverlapping studies, the majority of which described online as opposed to mHealth technologies and concluded that overall digital interventions were associated with reductions in suicidal ideation scores at postintervention. There was no treatment effect for self-harm or attempted suicide. Building on the work of Perry et al [ ], this review used a broader search strategy to include unpublished studies and ongoing trials. Although Witt et al [ ] examined digital interventions more broadly, this review focused on the research evidence examining mHealth technology specifically.
Why Is It Important to Do This Review?
The objective of this review was to examine the effectiveness of currently available mHealth technology tools in reducing suicide-specific outcomes in individuals taking part in a suicide prevention intervention delivered via mHealth technology.
Following the rise in the number of publicly available mHealth technology tools for suicide prevention, a review of the efficacy of this modality on suicide-specific outcomes is required. A review of the content and usability of available tools has been undertaken , and empirical data on their effectiveness in reducing suicide outcomes are warranted.
This review aimed to address the following research question: do suicide prevention mHealth technology tools effectively reduce suicide-specific outcomes?
The study procedure has been developed in line with the author’s proposed protocol , the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) statement [ ], and was registered with the International Prospective Register of Systematic Reviews database (systematic review number: CRD42017072899). In accordance with the PRISMA checklist recommendations, this review used the participants, interventions, comparisons, and outcome(s) (PICO) process for framing and reporting the review criteria; as such, the PICO and study design of the included studies were reported.
No restrictions were placed on diagnoses or any clinical or demographic characteristics of eligible samples. Studies using active or inactive control groups were eligible for inclusion. Inactive control groups were those in which participants received no intervention during the trial period (or were placed on a waiting list). Active control groups were those that used apps (not aimed at suicide outcomes), face-to-face interventions, or other forms of patient contact to control for the time/attention given to those in the intervention condition. Studies comparing mHealth-based interventions with antidepressants were eligible for inclusion.
Types of Studies
The types of studies included were all published and unpublished RCTs, pseudo-RCTs, and pre-post observational studies, which evaluated the effectiveness of mHealth technology in suicide prevention. Studies were included if the full report was accessible in English. Only studies that evaluated mobile tools relating specifically to suicide prevention or where suicidality is explicitly mentioned were included.
Types of Participants
Participants were individuals who took part in a suicide prevention intervention via mHealth technology. No restriction was placed on the age or gender of participants included in the studies reviewed. mHealth technology represents a modality that is accessible across the lifespan. However, the age of participants included in each study was noted in this review, and, where this information was available, it was used to draw conclusions regarding the efficacy of this modality for specific age groups.
Types of Interventions
Included studies reported on a suicide prevention intervention delivered via mHealth technology, that is, interventions aimed to reduce suicide risk by employing mobile communication or mobile computing technology. The review included studies with psychological and nonpsychological interventions (eg, psychoeducation, diaries, mood monitors, and self-management programs). As defined by Slattery et al  in a protocol for a systematic review on eHealth interventions for chronic pain, psychological treatments are those that explicitly deliver a psychological component (eg, psychotherapy for suicidal thoughts). Studies were included regardless of treatment intensity or duration.
Included studies comprised at least one suicide-specific outcome. This could include suicidal behavior, nonsuicidal self-injurious behavior, suicidal ideation, and suicidal intent.
Secondary outcomes were symptoms of depression or anxiety, as measured using administered or self-reported scales, where this information was available.
All databases were searched from their start date. Studies were included if a full-text paper in English was made available, either through databases or through contact with the study authors. The following databases were searched: MEDLINE, EMBASE, PsycINFO, and Cochrane Central Register of Controlled Trials (The Cochrane Library). Although the same search strategy was used for each database, appropriate changes were made to accommodate the different interfaces. Details of the search strategy are provided in. Medical Subject Headings and text word terms were used.
Clinical trial registries were searched to identify completed and in-progress trials. This included ClinicalTrials.gov, the metaRegister of controlled trials, and the WHO International Clinical Trials Registry Platform. Grey literature was searched using the OpenGrey database, which included technical or research reports, doctoral dissertations, and conference papers from the previous 5 years.
The reference lists of relevant systematic reviews and of included studies were also searched to identify additional studies that may be relevant. A handsearch of the Journal of Medical Internet Research and a separate search in the PubMed database were also conducted to identify any additional relevant studies.
The Review Team
The review team managed and conducted the review and had experience in systematic review methods, information retrieval, and statistics. Furthermore, 2 independent investigators (KF and EH) judged article eligibility, with any disagreements resolved through discussion with a third reviewer (RM). In addition, 3 researchers were involved so that measures to minimize bias and error were implemented at all stages of the review.
In addition to the review team, a broader research group was consulted at various stages, including individuals (JD, JB, MO’S, and KY) with expertise in the areas of computer science, clinical psychology, and suicide prevention policy.
Selection of Studies
Studies that were identified by the search strategy were managed using EndNote X8 . Members of the research team initially screened the titles and abstracts of publications for any duplicates of studies. They then screened for any studies that were not relevant to the review and exported them to a global exclusion folder. All remaining publications were retrieved for further scrutiny. Overall, 2 reviewers independently assessed the full-text articles of the remaining studies for inclusion. Papers that did not meet the inclusion criteria were systematically excluded via the exclusion categories, and the reason for exclusion was recorded. Disagreements between reviewers were discussed with a third reviewer until resolved. A PRISMA flowchart was created to graphically depict the inclusion and exclusion of studies ( ).
Data Extraction and Management
A data extraction form was created before data extraction. Data were extracted independently by 2 reviewers and verified by another reviewer using a customized form. Where the necessary outcome data were unavailable, the study authors were contacted. The authors were not blind to the study author, institution, or journal. Data were extracted relevant to the following categories: (1) study population and design, (2) intervention, and (3) outcome. Tables were created to present the characteristics of the studies included, containing the following information, where available: participant characteristics, geographic location, assessment periods, assessment/screening measures, description of intervention and comparison interventions, primary and secondary outcomes, theoretical basis, therapeutic content, mode of delivery (mobile phone app, telephone, and text), suicide prevention strategies, behavior change techniques, control condition, intensity and frequency of use, and treatment engagement (retention and attrition).
Assessment of Risk of Bias in Included Studies
The reviewers independently assessed the risk of bias using the recommended Cochrane Collaboration’s Risk of Bias Tool  to assess randomization procedures, bias, allocation, outcome assessor, reporting of findings, and losses to follow-up. Studies were classified as being of low, high, or an unclear risk of bias. The Risk of Bias in Nonrandomized Studies of Interventions was used to assess the risk of bias for controlled before/after designed studies.
The level of heterogeneity was taken into account when considering the suitability of the data for a meta-analysis. A small number of studies were identified with a large amount of heterogeneity present. The data were not amenable to synthesis, and a meta-analysis was not conducted. A full narrative review was undertaken using the Narrative Synthesis in Systematic Reviews tool .
The narrative synthesis involved the following elements:
- Developing a theory—the theoretical basis of the evaluated interventions was identified.
- Tabulation of data—extracted data from included studies included data on participants, interventions, outcome measures, country of origin, duration, delivery of the intervention, number of participants in each group, context in which intervention was delivered, results, and author comments.
- Textual descriptions summarizing each study were extracted to capture additional findings not identified during earlier stages of the synthesis.
- Vote counting as a descriptive tool—identified studies where the effect of the intervention was positive and statistically significant.
The PRISMA flowchart of the study selection process is presented in. Electronic searches identified 868 publications. The clinical trials database search identified 3 further completed unpublished clinical trials. After excluding duplicates, 568 studies remained. After non-English studies were removed, 525 abstracts and titles were screened for suitability, with 422 records excluded. A total of 103 full-text articles were then assessed for eligibility via a full-text screening.
Any disagreements regarding study eligibility were resolved following consensus discussions between the 3 reviewers (KF, EH, and RM). The lead authors of the 3 unpublished studies were contacted for further information on program design, study design, data analysis, and methodology, as required. Overall, 4 published studies were included in this review (one of which described 3 RCTs), reporting on a total of 624 participants. A summary of the published studies included in this review is presented in.
The studies were conducted in the United States , Australia [ ] and the United Kingdom [ ], with 1 paper reporting on 3 separate studies encompassing data from participants based in Canada, the United States, Australia, and Europe [ ]. All included studies reported on samples with a current suicide risk or a history of self-harming and suicidal behaviors. The studies reported on veterans [ ], indigenous youth in rural Australia [ ], and individuals with a recent and severe history of self-injurious thoughts and behaviors recruited from Web forums focused on self-harm [ ] and Child and Adolescent Mental Health Services (CAMHS) patients aged 12 to 17 years [ ]. Participants in the Franklin study [ ] were recruited from online forums (n=12) that focused on discussions of self-injury and related phenomena. Advertisements did not explicitly describe the study as a treatment study. The informed consent form made the treatment-related aspects of the study clear but did not provide details about Therapeutic Evaluative Conditioning (TEC) that would have allowed participants to discern whether or not they were in the active or control group.
A total of 4 studies met the criteria for inclusion in the review, including 2 RCTs [, ], 1 pilot RCT [ ], and 1 open-phase pre-post trial [ ]. Although all studies were inclusive of participants with current or previous self-harm or suicidal ideation, some studies excluded participants who were seriously contemplating or planning a suicide attempt [ ]. Of the 4 studies, 3 included a control group—only 1 provided a control version of the app [ ], whereas the other 2 studies provided face-to-face meetings with study staff and safety checks with the same frequency as participants in the intervention condition [ , ].
In terms of therapeutic modalities, app content was informed by acceptance and commitment therapy (ACT) , dialectical behavior therapy [ ], CBT [ , ], and TEC [ ]. Suicide prevention–specific interventions included asking users to identify reasons for living [ ], activities to help develop coping skills [ ], emotional regulation strategies [ , ], providing emergency contact details [ , ], and developing an action plan [ ]. The iBobbly mobile app [ ], based on ACT, was the only of the 4 published studies that followed a module-based approach, whereby participants were asked to complete 3 ACT-based modules within a 6-week period. The Virtual Hope Box, BlueIce, and TEC mobile apps encouraged participants to use the app as often as they felt necessary. Participants in 3 of the 4 studies rated the mobile app with high acceptability [ , ] or high frequency of app use [ ].
Suicide-specific outcomes were reported in each study. These included suicide ideation [, ], self-injurious thoughts [ ], and self-injurious behavior or self-harm [ , ]. Outcomes and their definitions differed across studies. A diverse range of measures was used to assess suicide-specific outcomes with no 2 studies using the same measure. Bush et al [ ] used the Beck Scale for Suicide Ideation (BSS) [ ] and the Columbia Suicide Severity Rating Scale (C-SSRS) [ ]. Tighe et al [ ] used the Depressive Symptom Inventory Suicidality Subscale [ ] and the Patient Health Questionnaire (PHQ; item 9 of the PHQ-9 evaluates passive thoughts of death or self-injury within the last 2 weeks) [ ]. Franklin [ ] used the Self-Injurious Thoughts and Behaviors Interview [ ], and Stallard et al [ ] used self-reported changes in self-harming behavior.
In terms of efficacy, 2 studies described a statistically significant positive effect of the mobile app intervention on 1 or more suicide outcomes. Franklin’s analyses of 3 studies reported that TEC produced moderate reductions for all self-injurious thoughts and behaviors except suicidal ideation. Reductions were reported for self-cutting episodes (32%-40%), suicide plans (21%-59%), and suicidal behaviors (33%-77%) . TEC effects were not maintained at the 1-month posttreatment follow-up. Self-reported self-harm behavior reduced for 73% of participants in the intervention condition, who had used the BlueIce app over a 12-week period [ ]. All of those who reported not self-harming in the 4 weeks before baseline assessment maintained their status and had not self-harmed over the course of the 12-week trial. Of the 26 participants who had self-harmed at baseline, 4 (15%, 4/26) had completely stopped, with a further 15/26 (58%) reporting less frequent acts of self-harm at follow-up. Tighe [ ] reported that although pre- and postintervention changes were significant in the iBobbly arm (t58.1=2.40; P=.02), the interaction of intervention arm by time (pre- vs postintervention) was not significant (t57.8=1.05; P=.30). Estimated marginal means show that any difference between change in the 2 arms arose because of a slight but nonsignificant difference (t59.0=0.84; P=.40) in mean baseline status between the 2 arms. Bush [ ] measured the presence and intensity of suicidal ideation (BSS and C-SSRS); importance of reasons for living (Brief Reasons for Living Inventory); feelings of thwarted belongingness (Interpersonal Needs Questionnaire); and how unpredictable, uncontrollable, and overloaded individuals found their lives (Perceived Stress Scale). They found no statistically significant advantage of treatment augmented by the Virtual Hope Box app compared with the control condition for any of these outcomes.
All studies reported significant efficacy of the app interventions on secondary outcomes. Bush et al  found that Virtual Hope Box users reported a significantly greater ability to cope with unpleasant emotions and thoughts (Coping Self-Efficacy Scale) at 3 time points. Franklin et al [ ] reported that the active group (mean −0.05, SD 0.27) showed a significantly smaller drop in positive affect toward self-related words compared with the control group (mean −0.17, SD 0.24; t49=−1.77; P=.04; Cohen d=0.47). Diminished aversion toward the self was associated with less self-cutting (B=−2.49; SE=1.10; incidence rate ratio [IRR]=0.08; P=.02), nonsuicidal self-injury (B=−.77; SE=0.17; IRR=0.46; P=.001), suicidal ideation (B=−1.02; SE=0.20; IRR=0.36; P=.001), and suicide plans (B=−.92; SE=0.36; IRR=0.40; P=.01). Stallard [ ] reported a statistically significant mean difference of 4.91 (t31=2.11; P=.04; 95% CI 0.17-9.64) on postuse symptoms of depression (Mood and Feelings Questionnaire) and 13.53 on symptoms of anxiety (Revised Child Anxiety and Depression Scale; t30=3.76; P=.001; 95% CI 6.17-20.90), which was evident across all anxiety subscales. provides a summary of published results for each study.
|Study||Summary of results|
|Tighe et al, 2017 |
|Bush et al, 2017 |
|Franklin et al, 2016 |
|Stallard et al, 2018 |
aVHB: Virtual Hope Box.
bTEC: Therapeutic Evaluative Conditioning.
cSITB: Self-Injurious Thoughts and Behaviour.
Risk of Bias in Included Studies
Risk of bias in the 4 published studies was evaluated using the Cochrane Collaboration Risk of Bias Tool . Overall, 3 of the studies had high risk, and 1 had unclear risk, with biases most apparent for the domains of participant, clinical personnel, and outcome assessor blinding. Performance and detection bias, therefore, cannot be ruled out. Attrition bias was reported and was an important limitation for studies with small sample sizes [ , ]. Selection bias was apparent in 2 studies; in particular, Franklin et al [ ] recruited participants online; therefore, their motivation and ability to engage may not be reflective of those recruited in the community. In the Stallard study [ ], participants were identified by their CAMHS clinician for inclusion, and the study did not include a control group. Participants were also paid to complete the study [ ]. Performance bias was evident in studies where allocation concealment was not described [ ], and personnel were not blind to the intervention allocation [ ], or where no method of blinding was reported in the study [ ]. Sensitivity of outcome measures was poor in some studies because of a low number of items [ ]. The amount of app usage was not specified in some studies, meaning that the level of engagement was not controlled for [ ], and only 1 study controlled for the impact of a digital intervention by providing a control version of the app [ ]. provides a summary of risk of bias for each study.
|Study||Risk of bias|
|Bush et al, 2017 |
|Tighe et al, 2017 |
|Franklin et al, 2016 |
|Stallard et al, 2018 |
In all, 3 completed but unpublished trials that met the criteria for inclusion are described in further detail in[ - ].
However, the evaluation studies included here indicate that mHealth tools show promise for individuals at elevated risk of suicide and self-harm and can be a useful adjunct to face-to-face therapy. This review confirms and updates the findings of Witt et al  who identified only 2 studies describing the efficacy of mobile apps, with the majority of digital intervention studies evaluating the effectiveness of online programs, and adds to the review conducted by Larsen et al [ ], which identified that 24 apps for the prevention of suicidal behavior were available for download. This review identified 4 published and 3 unpublished studies evaluating the efficacy of mHealth technology in suicide prevention, indicating that the pace of development of such tools is not matched by evaluative research. Byambasuren et al [ ] proposed a concept of prescriptible mHealth apps that are currently available, proven effective, and preferably stand-alone. The paucity of robust evidence evaluating the efficacy of mHealth interventions on suicide outcomes indicates that there is not yet sufficient research to support the prescribing of stand-alone mHealth tools for suicide prevention.
Overall, the evaluated mHealth technology interventions demonstrated some positive results for individuals at elevated risk of suicide or self-harm, including reductions in depression, psychological distress, and self-harm, and increases in coping self-efficacy. Mobile apps are one mechanism by which the scalability of effective interventions for suicide prevention can be improved . The use of mobile apps has the capacity to increase accessibility to therapeutic interventions for at-risk individuals who may not otherwise seek help. Digital interventions have the potential to address major barriers to help-seeking behaviors, such as geographical location and stigma [ ]. They also provide opportunities for evidence-based intervention to be accessed several times a day and at the time when it is most needed. Participants in 3 of the 4 published studies were encouraged to use the mobile app as often as they felt necessary [ , , ]. The findings revealed positive results for individuals who have decreased help-seeking behaviors [ ] and who were at elevated risk of suicide and self-harm [ - ]. The utilization of mobile apps and other digital interventions is in line with the stepped-care treatment approach [ ], which offers the least intensive and most accessible intervention, that has the best chance of delivering a positive outcome as the first line of treatment.
However, neither the iBobbly nor Virtual Hope Box or the TEC apps demonstrated the ability to significantly decrease suicidal ideation compared with a control condition. The studies included reported a positive impact on secondary outcomes, such as measures of depression and anxiety. This confirms parallel findings within face-to-face interventions. Insufficient evidence exists to suggest that CBT focusing on mental illness reduces suicidal cognitions and behaviors. CBT specifically focusing on suicidal cognitions and behaviors has been found to be effective . Such interventions are likely to be most effective if they target the prominent risk factors that exist during acute suicidal crises [ ]. The findings of this review may be reflective of the relatively small number of key suicide prevention strategies included in the apps described, with content focused instead on symptoms of depression and anxiety. Similarly, Larsen et al [ ] found that although all apps evaluated contained at least 1 component that was broadly consistent with known evidence or best practice guidelines, there was limited concordance with high-quality, evidence-based practice.
In terms of clinical utility, suicide prevention mobile apps can be most beneficial for individuals at risk of suicide, particularly those with decreased help-seeking behaviors, when delivered as an adjunct to therapy, and when they deliver suicide-specific interventions. Future researchers could leverage the current and existing research findings by combining the delivery via mobile app of EMA with evidence-based psychological interventions. In-built EMA in suicide prevention apps has the potential to provide individualized time-stamped data spanning biological, social, and psychological variables, alongside behavioral measures of app usage and engagement to facilitate a greater understanding of suicide processes. The current evidence points to the need to focus on more dynamic intervention approaches, such as Just-In-Time Adaptive Interventions (JITAIs).
Limitations of the Studies Included
A significant limitation found across all studies pertained to the heterogeneity of the outcomes studied. This hampered the extent to which data could be compared and generalized. Diversity in nomenclature continues to challenge research in this area. Although in the United States, distinction is frequently made between nonsuicidal self-injury and suicidal behavior , for example, outside of the United States, these terms are yet to receive widespread acceptance [ ]. The need for a standardized and shared labeling of suicidal ideation and behaviors has long been recognized. The International Association of Suicide Prevention Nomenclature Special Interest Group has formed a task force to generate an international standardized nomenclature on all terms within the area of suicidology, inclusive of death wishes, assisted suicide, and bereavement, which may render research more comparable around the world.
Similarly, no 2 studies used the same suicide outcome measure in this review. Studies used combinations of suicide-specific measures, single suicide-specific items, including measures assessing mental health more broadly, and self-report. Heterogeneity of outcomes evident across studies may be reflective of difficulties within research and clinical practice in identifying homogenous subgroups of self-harming patients . It is imperative that to leverage previously completed work and maximize the available data, suicide prevention researchers work to develop and use standard practices in relation to suicide outcome measures.
A further methodological issue, which is likely to reduce ecological validity, is the practice of excluding individuals at increased risk of suicide from participation. Importing empirically supported treatments for depressed adolescents or suicidal adolescents may not be appropriate because the trials in which efficacy was established excluded suicidal teens . Similarly, this review identified studies that excluded participants because of current suicide risk [ ], a practice that may contribute to a sample bias and reduce generalizability of findings within clinical settings.
The findings highlight broader issues in relation to regulation and policy pertaining to mHealth technology. Some have argued that it is “now critical, even if paradoxical, to draw new boundaries in the seemingly boundless world of digital health” . Indeed, the specific focus and purpose of the mHealth tools being evaluated in this review were not generally apparent without a full-text review and may underline a lack of standard categorization of tools to guide clinicians in their decision to use or recommend an app as part of their work. Torous and Hsin [ ] argue that there is an urgent need to unite the potential of digital health with the fundamental ethics of clinical practice and to encourage innovation while protecting both the future of digital health and the trust it requires to engage patients and clinicians. They propose a practical taxonomy accessible to clinicians, comprising 3 categories of digital health use to illustrate these boundaries: (1) treatment and diagnosis, (2) care enhancement, and (3) resources. It is argued that adopting this rubric at the level of clinical judgment in direct patient care may facilitate empowerment of digital health technology by harnessing the utility of decision-guiding frameworks that clinicians already use in practice. Future research would also benefit from following the guidance of established standards of reporting on eHealth evaluations [ ].
Strengths and Limitations of This Review
This review extends the reviews undertaken by Larsen et al  and Witt et al [ ] by (1) not restricting the modalities reviewed to mobile phone apps and including other mHealth technology–delivered interventions and (2) evaluating efficacy using outcomes research to complement Larsen’s comprehensive assessment of content. However, the small number of studies identified hindered our ability to synthesize data and conduct a meta-analysis. There were not sufficient data to provide a comparison of mobile technology tools across outcome measures such as mobile phone apps, texting, and gaming, which could help identify the most effective modes of delivery. This may be, in part, contributed to by the restrictive inclusion and exclusion criteria employed within this study. For example, 21 studies were excluded for not being an RCT, pseudo-RCT, or observational pre-/posttest design, and an additional 7 studies were excluded for not having a suicide-specific measure as a primary outcome. Therefore, had the inclusion criteria been less stringent, more studies would have been included within the review, which may have allowed for a meta-analysis to be completed. However, the current eligibility criteria were decided on for this review to capture the results of scientifically rigorous studies.
Risk of bias assessment indicated methodological issues across studies, which could be addressed in future research. The small sample sizes across studies and attrition during studies limited the generalization of findings. Blinding procedures were problematic across studies, and the small number of research-validated suicide prevention apps available makes comparisons with similar interventions difficult. These difficulties may signal a need for a shift in focus from traditional research methods investigating unidirectional cause-effect relationships to examining dynamic systems by applying machine learning approaches to big data.
The 4 completed and published studies included in this review evaluated the iBobbly, Virtual Hope Box, BlueIce, and TEC apps. Together, results evaluating the apps show some positive results for individuals at elevated risk of suicide or self-harm, including reductions in depression, psychological distress, and self-harm and increases in coping self-efficacy. Despite the continued growth in the development and availability of suicide prevention mHealth tools, there is currently limited evaluative research on the efficacy of such tools in reducing suicide-specific outcomes.
However, neither the use of iBobbly nor Virtual Hope Box or the TEC apps demonstrated the ability to significantly decrease suicidal ideation compared with a control condition. Further evaluation studies would benefit from addressing 3 main methodological issues, which arose across studies: (1) heterogeneity of outcomes, (2) exclusion of individuals at higher risk of suicide, and (3) performance biases arising because of blinding procedures. Although the evidence available indicates some progress, the pace of suicide prevention app development needs to be matched by a greater focus on empirically supported mHealth technology–based interventions. Future research endeavors would benefit from leveraging the current findings with existing EMA research to support the development and evaluation of dynamic interventions (such as JITAIs) delivered via mobile apps to provide effective intervention while simultaneously enhancing our understanding of suicide processes.
Conflicts of Interest
Published studies included in the review.DOCX File , 23 KB
Unpublished completed studies identified.DOCX File , 17 KB
- World Health Organization. Public Health Action for the Prevention of Suicide: A Framework. Geneva: WHO Press; 2012.
- Maris RW. Suicide. Lancet 2002 Jul 27;360(9329):319-326. [CrossRef] [Medline]
- Doshi A, Boudreaux ED, Wang N, Pelletier AJ, Camargo CA. National study of US emergency department visits for attempted suicide and self-inflicted injury, 1997-2001. Ann Emerg Med 2005 Oct;46(4):369-375. [CrossRef] [Medline]
- Shaffer D, Gould MS, Fisher P, Trautman P, Moreau D, Kleinman M, et al. Psychiatric diagnosis in child and adolescent suicide. Arch Gen Psychiatry 1996 Apr;53(4):339-348. [CrossRef] [Medline]
- Brent DA, Baugher M, Bridge J, Chen T, Chiappetta L. Age- and sex-related risk factors for adolescent suicide. J Am Acad Child Adolesc Psychiatry 1999 Dec;38(12):1497-1505. [CrossRef] [Medline]
- Bridge JA, Goldstein TR, Brent DA. Adolescent suicide and suicidal behavior. J Child Psychol Psychiatry 2006;47(3-4):372-394. [CrossRef] [Medline]
- Lewinsohn PM, Rohde P, Seeley JR. Psychosocial risk factors for future adolescent suicide attempts. J Consult Clin Psychol 1994 Apr;62(2):297-305. [CrossRef] [Medline]
- Goldston DB, Daniel SS, Reboussin DM, Reboussin BA, Frazier PH, Kelley AE. Suicide attempts among formerly hospitalized adolescents: a prospective naturalistic study of risk during the first 5 years after discharge. J Am Acad Child Adolesc Psychiatry 1999 Jun;38(6):660-671. [CrossRef] [Medline]
- Hawton K, van Heeringen K. Suicide. Lancet 2009 Apr 18;373(9672):1372-1381. [CrossRef] [Medline]
- Bernert RA, Kim JS, Iwata NG, Perlis ML. Sleep disturbances as an evidence-based suicide risk factor. Curr Psychiatry Rep 2015 Mar;17(3):554 [FREE Full text] [CrossRef] [Medline]
- Moskowitz A, DeWitt K, White N, Melia R, Zhao D, Bernert R. Nightmares and emotion regulation deficits as predictors of risk for suicidal ideation and history of self-directed violence among military veterans. Sleep 2017;40(1):413. [CrossRef]
- O'Connor RC, Smyth R, Ferguson E, Ryan C, Williams JM. Psychological processes and repeat suicidal behavior: a four-year prospective study. J Consult Clin Psychol 2013 Dec;81(6):1137-1143 [FREE Full text] [CrossRef] [Medline]
- Cassidy F. Risk factors of attempted suicide in bipolar disorder. Suicide Life Threat Behav 2011 Feb;41(1):6-11. [CrossRef] [Medline]
- Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull 2017 Feb;143(2):187-232. [CrossRef] [Medline]
- Kristoufek L, Moat HS, Preis T. Estimating suicide occurrence statistics using Google Trends. EPJ Data Sci 2016 Nov 8;5(1):32. [CrossRef]
- Thompson WK, Gershon A, O'Hara R, Bernert RA, Depp CA. The prediction of study-emergent suicidal ideation in bipolar disorder: a pilot study using ecological momentary assessment data. Bipolar Disord 2014 Nov;16(7):669-677 [FREE Full text] [CrossRef] [Medline]
- Kay M, Santos J, Takane M. mHealth: New Horizons for Health Through Mobile Technologies. Geneva, Switzerland: World Health Organization; 2011.
- World Health Organization. Preventing Suicide: A Global Imperative. Geneva: WHO Press; 2014.
- Gould MS, Greenberg T, Velting DM, Shaffer D. Youth suicide risk and preventive interventions: a review of the past 10 years. J Am Acad Child Adolesc Psychiatry 2003 Apr;42(4):386-405. [CrossRef] [Medline]
- Rathus JH, Miller AL. Dialectical behavior therapy adapted for suicidal adolescents. Suicide Life Threat Behav 2002;32(2):146-157. [CrossRef] [Medline]
- Wood A, Trainor G, Rothwell J, Moore A, Harrington R. Randomized trial of group therapy for repeated deliberate self-harm in adolescents. J Am Acad Child Adolesc Psychiatry 2001 Nov;40(11):1246-1253. [CrossRef] [Medline]
- Tondo L, Albert MJ, Baldessarini RJ. Suicide rates in relation to health care access in the United States: an ecological study. J Clin Psychiatry 2006 Apr;67(4):517-523. [CrossRef] [Medline]
- Christensen H, Batterham P, O'Dea B. E-health interventions for suicide prevention. Int J Environ Res Public Health 2014 Aug 12;11(8):8193-8212 [FREE Full text] [CrossRef] [Medline]
- Ybarra ML, Eaton WW. Internet-based mental health interventions. Ment Health Serv Res 2005 Jun;7(2):75-87. [CrossRef] [Medline]
- Lai MH, Maniam T, Chan LF, Ravindran AV. Caught in the web: a review of web-based suicide prevention. J Med Internet Res 2014 Jan 28;16(1):e30 [FREE Full text] [CrossRef] [Medline]
- Torous J, Friedman R, Keshavan M. Smartphone ownership and interest in mobile applications to monitor symptoms of mental health conditions. JMIR Mhealth Uhealth 2014 Jan 21;2(1):e2 [FREE Full text] [CrossRef] [Medline]
- Assets: how they work. 2015. Future in Mind: Promoting, Protecting and Improving Our Children and Young People's Mental Health and Wellbeing URL: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/414024/Childrens_Mental_Health.pdf [accessed 2019-10-01]
- European Commission. 2012. eHealth Action Plan 2012-2020: Innovative Healthcare for the 21st Century URL: http://ec.europa.eu/information_society/newsroom/cf/dae/document.cfm?doc_id=4188 [accessed 2019-10-01]
- de Beurs D, Kirtley O, Kerkhof A, Portzky G, O'Connor RC. The role of mobile phone technology in understanding and preventing suicidal behavior. Crisis 2015 Jun;36(2):79-82. [CrossRef] [Medline]
- Witt K, Spittal MJ, Carter G, Pirkis J, Hetrick S, Currier D, et al. Effectiveness of online and mobile telephone applications ('apps') for the self-management of suicidal ideation and self-harm: a systematic review and meta-analysis. BMC Psychiatry 2017 Aug 15;17(1):297 [FREE Full text] [CrossRef] [Medline]
- Torous J, Hsin H. Empowering the digital therapeutic relationship: virtual clinics for digital health interventions. NPJ Digit Med 2018;1:16 [FREE Full text] [CrossRef] [Medline]
- Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. J Nerv Ment Dis 2017 Jan;205(1):4-8. [CrossRef] [Medline]
- Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017 Oct;16(3):287-298 [FREE Full text] [CrossRef] [Medline]
- Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord 2017 Aug 15;218:15-22 [FREE Full text] [CrossRef] [Medline]
- Shen N, Levitan M, Johnson A, Bender JL, Hamilton-Page M, Jadad AA, et al. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth 2015 Feb 16;3(1):e16 [FREE Full text] [CrossRef] [Medline]
- Larsen ME, Nicholas J, Christensen H. A systematic assessment of smartphone tools for suicide prevention. PLoS One 2016;11(4):e0152285 [FREE Full text] [CrossRef] [Medline]
- Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res 2017 May 25;19(5):e176 [FREE Full text] [CrossRef] [Medline]
- Hawton K, Witt KG, Salisbury TL, Arensman E, Gunnell D, Townsend E, et al. Interventions for self-harm in children and adolescents. Cochrane Database Syst Rev 2015 Dec 21(12):CD012013. [CrossRef] [Medline]
- Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):e247 [FREE Full text] [CrossRef] [Medline]
- Perry Y, Werner-Seidler A, Calear A, Christensen H. Web-based and mobile suicide prevention interventions for young people: a systematic review. J Can Acad Child Adolesc Psychiatry 2016;25(2):73-79 [FREE Full text] [Medline]
- Melia R, Francis K, Duggan J, Bogue J, O'Sullivan M, Chambers D, et al. Mobile health technology interventions for suicide prevention: protocol for a systematic review and meta-analysis. JMIR Res Protoc 2018 Jan 26;7(1):e28 [FREE Full text] [CrossRef] [Medline]
- Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015 Jan 1;4:1 [FREE Full text] [CrossRef] [Medline]
- Slattery BW, Haugh S, Francis K, O'Connor L, Barrett K, Dwyer CP, et al. Protocol for a systematic review with network meta-analysis of the modalities used to deliver eHealth interventions for chronic pain. Syst Rev 2017 Mar 3;6(1):45 [FREE Full text] [CrossRef] [Medline]
- Endnote X9. URL: http://endnote.com/ [accessed 2019-10-01]
- Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product From the ESRC Methods Programme Version. Lancaster: Institute of Health Research; 2006.
- Tighe J, Shand F, Ridani R, Mackinnon A, de la Mata N, Christensen H. Ibobbly mobile health intervention for suicide prevention in Australian Indigenous youth: a pilot randomised controlled trial. BMJ Open 2017 Jan 27;7(1):e013518 [FREE Full text] [CrossRef] [Medline]
- Stallard P, Porter J, Grist R. A smartphone app (BlueIce) for young people who self-harm: open phase 1 pre-post trial. JMIR Mhealth Uhealth 2018 Jan 30;6(1):e32 [FREE Full text] [CrossRef] [Medline]
- Franklin JC, Fox KR, Franklin CR, Kleiman EM, Ribeiro JD, Jaroszewski AC, et al. A brief mobile app reduces nonsuicidal and suicidal self-injury: evidence from three randomized controlled trials. J Consult Clin Psychol 2016 Jun;84(6):544-557. [CrossRef] [Medline]
- Beck A. Manual for Beck Scale for Suicidal Ideation. New York: Pearson Education; 1991.
- Bush NE, Smolenski DJ, Denneson LM, Williams HB, Thomas EK, Dobscha SK. A virtual hope box: randomized controlled trial of a smartphone app for emotional regulation and coping with distress. Psychiatr Serv 2017 Apr 1;68(4):330-336. [CrossRef] [Medline]
- Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011 Dec;168(12):1266-1277 [FREE Full text] [CrossRef] [Medline]
- Metalsky GI, Joiner TE. The hopelessness depression symptom questionnaire. Cogn Ther Res 1997;21(3):359-384 [FREE Full text] [CrossRef]
- Spitzer R, Williams J, Kroenke K. Instructions. Instructions for Patient Health Questionnaire (PHQ) and GAD-7 Measures URL: https://phqscreeners.pfizer.edrupalgardens.com/sites/g/files/g10016261/f/201412/instructions.pdf [accessed 2019-10-01]
- Nock MK, Holmberg EB, Photos VI, Michel BD. Self-Injurious Thoughts and Behaviors Interview: development, reliability, and validity in an adolescent sample. Psychol Assess 2007 Sep;19(3):309-317. [CrossRef] [Medline]
- Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane collaboration's tool for assessing risk of bias in randomised trials. Br Med J 2011 Oct 18;343:d5928 [FREE Full text] [CrossRef] [Medline]
- Clinical Trials. ReliefLink: A Preventative Mobile Toolkit for Follow-Up Care of Psychiatric Patients URL: https://clinicaltrials.gov/ct2/show/NCT02691221 [accessed 2019-09-19]
- Clinical Trials. The Efficacy of a Mobile Application for Treating Depression and Anxiety Symptoms URL: https://clinicaltrials.gov/ct2/show/NCT03032952 [accessed 2019-09-29]
- Clinical Trials. Mobile Phone Intervention to Reduce Youth Suicide in Rural Communities URL: https://clinicaltrials.gov/ct2/show/NCT03145363 [accessed 2019-10-01]
- Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digit Med 2018;1:12 [FREE Full text] [CrossRef] [Medline]
- Mewton L, Andrews G. Cognitive behavioral therapy for suicidal behaviors: improving patient outcomes. Psychol Res Behav Manag 2016;9:21-29 [FREE Full text] [CrossRef] [Medline]
- Stanley B, Brown G, Brent DA, Wells K, Poling K, Curry J, et al. Cognitive-behavioral therapy for suicide prevention (CBT-SP): treatment model, feasibility, and acceptability. J Am Acad Child Adolesc Psychiatry 2009 Oct;48(10):1005-1013 [FREE Full text] [CrossRef] [Medline]
- Franklin J, Nock M. Non suicidal self-injury and its relation to suicidal behavior. In: Kleespies P, editor. The Oxford Handbook of Behavioral Emergencies and Crises. New York, NY: Oxford University Press; 2017.
- Kapur N, Cooper J, O'Connor RC, Hawton K. Non-suicidal self-injury v. attempted suicide: new diagnosis or false dichotomy? Br J Psychiatry 2013 May;202(5):326-328. [CrossRef] [Medline]
- Wilkinson P. Non-suicidal self-injury. Eur Child Adolesc Psychiatry 2013 Feb 1;22 Suppl 1(S1):S75-S79.
- Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
|ACT: acceptance and commitment therapy|
|BSS: Beck Scale for Suicide Ideation|
|C-SSRS: Columbia Suicide Severity Rating Scale|
|CAMHS: Child and Adolescent Mental Health Services|
|CBT: cognitive behavioral therapy|
|eHealth: electronic health|
|EMA: Ecological Momentary Assessment|
|FDA: Food and Drug Administration|
|JITAI: Just-In-Time Adaptive Intervention|
|mHealth: mobile health|
|PHQ: Patient Health Questionnaire|
|PICO: participants, interventions, comparisons, and outcome(s)|
|PRISMA: Preferred Reporting Items for Systematic Review and Meta-Analysis|
|RCT: randomized controlled trial|
|TEC: Therapeutic Evaluative Conditioning|
|WHO: World Health Organization|
Edited by G Eysenbach; submitted 16.10.18; peer-reviewed by M Subotic-Kerry, K Witt, M Larsen, D Paolotti, N Shen; comments to author 23.03.19; revised version received 17.05.19; accepted 02.08.19; published 15.01.20
©Ruth Melia, Kady Francis, Emma Hickey, John Bogue, Jim Duggan, Mary O'Sullivan, Karen Young. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 15.01.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.