Mental Health Apps Implemented in the Workplace: Scoping Review of Trends and Gaps in Evaluation Research

Introduction

Mental health issues have a significant human and financial cost for society and workplaces. Poor mental health and loss of productivity are estimated to cost the global economy approximately US $1 trillion each year [1], and this figure is projected to rise, given the COVID-19 pandemic’s impact on workplace mental health. Rapid transitions and changes in workplace demands linked to the pandemic have led to an increase in reports of perceived stress, depression, and anxiety among working adults, particularly for working mothers, people of minority ethnicities, and those working in frontline positions [2-4]. Limited access to traditional mental health services, along with the longstanding stigma associated with seeking support, has led to a growing need to expand access to a range of mental health supports to address the unmet needs of the workforce and reduce the burden of mental health issues.

In recent years, there has been a growing interest in digital health technologies, including web-based and smartphone-based interventions as tools to address mental health problems. One of the main advantages of digital mental health interventions is their ability to overcome traditional barriers to accessing mental health care by providing on-demand access to quality mental health information and support [5,6]. For example, many people are reluctant to seek help due to the stigma associated with mental illness. Digital interventions, such as mental health apps, offer a private and anonymous way to access support without the fear of being judged. Digital interventions can also help overcome geographical barriers by providing access for people who live in remote or rural areas, where mental health services may be limited or nonexistent [5]. In addition, digital interventions can be accessed at any time and from anywhere, which can help overcome the time constraints that many working adults face when trying to access traditional mental health services. Mental health apps can be used across the full spectrum of mental health care, from prevention to management. For example, there are apps that help people assess and manage stress and anxiety, apps that provide support for people with depression, and apps that help people improve their well-being.

There is a growing body of research that supports the effectiveness of mental health apps in both the general and clinical populations [4,7]. A review of app-based mental health interventions found them to be effective in reducing symptoms of depression and anxiety [4,7], as well as improving mental health literacy and well-being [8] in both clinical and non-clinical populations [4,7-9]. However, a review of publicly available popular mental health apps found that only 2 out of the 6 mental health apps identified were empirically evaluated [10], underscoring the gap between mental health apps available publicly and those that are evaluated in empirical studies. There is an urgent need for high-quality studies in this field to explore mental health apps that are evidence-based, empirically supported, and scalable to broader users.

There is a growing interest in the use of mental health apps across different settings such as clinical, postsecondary, and community settings, and more recently in workplaces. Several reviews provide evidence to support that digital mental health interventions implemented in workplaces have positive effects on improving general mental health and workplace-related outcomes [11-13]. Studies have explored the effects of these interventions in improving well-being, symptoms of stress, and several workplace-related outcomes, such as work effectiveness or productivity and job satisfaction [11-13]. However, the majority of the studies in these reviews consisted of web-based interventions, with only a small percentage of them exploring mental health apps.

As the availability and use of mental health apps in workplace settings become more prevalent, there is a growing need to identify and synthesize the existing evidence related to their systematic evaluation with workers. In this paper, we aim to conduct a scoping review to synthesize and critically reflect on the body of evidence in this emerging field. Specifically, we will summarize and identify trends and gaps in the evaluation of mental health apps in workplace settings.

Methods

Study Design

We followed the 5 steps of scoping review methodology [14]: specifying the research question, identifying relevant literature, selecting studies, extracting data, and synthesizing the findings. The protocol for this review is registered on the Open Science Framework. We also adhered to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines (Checklist 1) to ensure consistent and transparent reporting [15]. To examine and summarize the breadth of evidence regarding the evaluation of mental health apps for working adults, we developed the following research questions based on the population, intervention, comparison, and outcome (PICO) framework:

1. How are mental health apps being evaluated with workers and/or in the context of a workplace?
2. What types of mental health apps are studied?
3. With whom are the applications being evaluated (workers or workplace sectors)?
4. What key variables are being assessed, and what tools are being used to track progress and impact?

Inclusion Criteria

Studies were included in this review if they met the following inclusion criteria: (1) the primary study intervention was a mobile mental health app that was tested by employees; (2) the study evaluated the effectiveness, implementation, impact, or use of the mental health app; (3) study participants included workers currently employed in any field and/or implemented in the context of a workplace; (4) the study included any quantitative or qualitative evaluation design; (5) the study was published in a peer-reviewed journal; and (6) the study was available in English. Studies were excluded if: (1) the intervention in the study was primarily a web-based or internet-based intervention (ie, not a mobile app) and (2) the study design only focused on app development (not evaluation) of a mental health app for workers. Conference proceedings and gray literature were not included, as the key focus of this review was to present a summary of published peer-reviewed studies that systematically evaluated mental health apps.

Search Strategy

We searched the following electronic databases: PsycINFO, Embase, MEDLINE, Cochrane Database, OVID Healthstar, and OVID Emcare for peer-reviewed studies from January 2000 to August 2023. A comprehensive search strategy, including search terms, subject headings, and syntax requirements for the databases, was adapted from a previous systematic review of digital mental health interventions for the workplace [11] and was modified in consultation with a health science librarian. Search terms included a combination of constructs such as “Mental Health,” “Workplace,” and “Mobile Application” (Multimedia Appendix 1). The search was conducted from May 2023 to August 2023.

Study Screening

Data screening and extraction were carried out by 2 independent reviewers (SA and ML) using Covidence systematic review software (Veritas Health Innovation), an online tool that supports screening and data extraction for reviews, to facilitate a systematic screening process. All studies were screened independently by 2 reviewers, and any discrepancies in study selection were discussed in a review meeting. A third reviewer, SM, was consulted to resolve any disagreements between the 2 primary reviewers and to improve clarity in decision-making when necessary.

Data Extraction and Analysis

Data extraction was an iterative process designed to develop and refine a data charting sheet in Microsoft Excel to ensure all relevant data were captured comprehensively. The fields of data extracted included study characteristics such as the year, country of publication, study design, workplace context or industry where the study was conducted, intervention characteristics, timeframe, and evaluation approaches including both assessment tools and outcomes explored in the study. The initial extraction chart was piloted with 5 studies and was later revised to capture additional information relevant to our research questions. Analysis of these data included a summary of the study characteristics, types of app interventions, study populations, and approaches to evaluation (variables and evaluation tools). Descriptive summaries of the trends in each category will be outlined.

Results

Overview

We searched 6 electronic databases, identifying 288 studies after the removal of duplicates. After screening titles and abstracts, 85 studies were deemed eligible for full-text screening. Subsequently, 54 studies met our inclusion criteria for being included in this scoping review (Figure 1). Data from these studies were extracted and charted to generate a comprehensive summary of the breadth and trends in the body of evidence of mental health app evaluation in the workplace. Of these 54 studies, 11 were protocols for studies in progress; hence, no results were available for these studies at the time of the review. Upon careful consideration of the protocols and clinical trial registrations, we were able to gather relevant information related to the key outcomes being evaluated and the processes by which the studies were being conducted, which was deemed sufficient to answer our main research question: “How are mental health apps being evaluated with workers and/or in the context of the workplace?” As a result, we decided to report these ongoing studies in our review.

‎

Study Characteristics

Among the 54 studies included, approximately two-thirds were randomized controlled trials (RCTs; protocols or trial registrations n=11 [16-26]; completed RCTs n=21 [27-48]; pilot RCTs n=3 [49-51]). A smaller subset of studies adopted nonrandomized methods such as pre-post nonexperimental designs (n=13) [52-63], with even fewer studies using mixed methods (n=5) [64-67] and qualitative designs (n=1) [68]. Over two-thirds of the studies (n=37) were published after the onset of the COVID-19 pandemic, reflecting accelerated remote work and rapid digital transformation in workplaces, with over 26% (n=14) of the studies on this topic published in the year 2022 alone. Table 1 outlines a summary of the key characteristics of all included studies.

Types of Mental Health App Interventions

A total of 44 unique apps were evaluated across the 54 studies included in this review. There was considerable variability across these apps in terms of their therapeutic approach, goals, functionalities, target population, and intervention duration. Over half of the apps (23/44, 52%) included a programmatic structure with predetermined guidelines or prescribed instructions on how to use them, while 34% (15/44) were open-ended and user driven without a fixed guideline for use. Based on these distinctions, we broadly classified these apps as: (1) structured self-guided, (2) unstructured self-guided, and (3) an adjunct to other interventions. See Table 2 for an overview of the identified apps categorized by type, including proposed purpose and target population noted in the study.

Structured self-guided apps (n=23 apps) had a structured pathway in the form of programs or modules that users needed to complete, often in the form of modules, lessons, exercises, or activities. For instance, the HeadGear app (cited in 8 studies) is a 30-day intervention informed by mindfulness-based techniques that include daily challenges users are instructed to complete [17,38,45,51,54,60,67]. Headspace included daily guided 10-minute mindfulness practices or techniques [26,27,40,68]. Several studies noted that the app content was informed by evidence-based therapies, including behavioral activation and mindfulness [24,38,48,53,62,70], acceptance and commitment therapy [37,42,43,68], and principles of cognitive behavioral therapy [32,59], but varied widely in terms of the extent to which they adhered to evidence-based guidelines.

Unstructured self-guided apps (n=15) were more open-ended and provided users with the choice to select and engage with intervention components on demand. For instance, the Oiva app [66] consisted of information sessions on stress management and well-being based on the principles of acceptance and commitment therapy but had no prescribed guidance for engaging with the app. Many apps included meditation or relaxation exercises [28,31,35,44,48,49], self-monitoring tools (eg, mood, physical activity, and stress) [52,57,65,69], and/or links to resources [55,57].

Finally, there were 5 apps that were designed to be implemented in conjunction with other interventions. Three were used to supplement in-person training [21,34,42], and three included wearable devices linked to the mental health app [34,35,58]. One study, for example, combined a brief in-person mindfulness training followed by monitoring of respiratory patterns through a wearable device, with real-time biofeedback received through the app [34].

In terms of intervention duration, most studies (n=38) implemented a fixed intervention period, typically ranging from 2 to 8 weeks, with 4 weeks being the most common. Outcomes were generally assessed at baseline and at the end of the intervention, although some studies included longer follow-up periods, with the longest extending to 12 months [38].

Study Populations

Participant demographics varied significantly across the studies. Some studies targeted the implementation of the intervention with specific gender groups, among individuals with symptoms of mental health issues and/or in specific workplace sectors. For instance, one study targeted male participants experiencing mild to moderate symptoms of stress and/or depression to assess the feasibility of a multicomponent, technology-based (P4Well) intervention [50]. Two other studies specifically recruited individuals from male-dominated industries to evaluate the effectiveness of a therapeutic app (HeadGear) [38,45]. One study focused on evaluating the effectiveness of a well-being and stress management app for working women [30].

Study participants worked in a range of industries, including health and social services (n=23), which largely aimed at implementing therapeutic apps, large corporations or consumer-based industries (n=5), information and technology (n=5), education (n=2), agriculture (n=1), apprenticeships (n=1), and construction (n=1). A subset of studies implemented their interventions across multiple industries (n=7), and a few studies did not explicitly specify the contextual setting (n=10) in which the research was being conducted.

Approach to Evaluation

We identified three primary ways (Figure 2) in which studies evaluated mental health apps with workers: (1) app engagement and utilization; (2) impact evaluation (mental distress, mental health, and workplace outcomes); and (3) implementation process evaluation. The variables and tools within each approach are summarized in Multimedia Appendix 2.

‎

App Engagement and Utilization

Most of the studies reported on users’ engagement with the intervention by tracking app usage patterns. App use ranged from simply tracking the number of downloads or logins to tracking the number of times specific features were used, completion of specific tasks, and tracking the amount of time spent in the app. In the structured interventions, app usage included tracking adherence to the prescribed protocols. The data was typically gathered through built-in analytics that highlighted aggregate usage patterns of the app, although some studies gathered subjective reports of app usage through survey or interview questions. There were at least a dozen studies that did not report app usage data.

User feedback was another common way of tracking engagement, noted in 22 (41%) of the studies. Typically, the follow-up evaluation included exploring perceptions of the app’s key features, such as usability, acceptability, quality, motivation to use the app, and overall satisfaction, and/or providing an opportunity to suggest improvements. Standardized measures, such as the Mobile App Rating Scale [70], were used in 5 studies [18,25,41,51,67]; the Systems Usability Scale [71] was used in 2 studies [22,32]; and the eHealth Impact Questionnaire was used in 1 study [16]. Other studies used more informal feedback approaches.

Impact Evaluation

Most of the research on the impact of mental health apps for workers focuses on multiple primary and secondary outcomes, ranging from mental distress to mental health and well-being, as well as specific work-related outcomes for individual app users. Table 3 highlights the most common outcomes that were evaluated, along with an overview of the tools that were used to track impact.

Impact on mental distress was the most common outcome, evaluated in 42 of the 54 (78%) studies (Table 3). Symptoms of depression were evaluated in 30 (56%) of the studies, often using standardized tools such as the Patient Health Questionnaire-9 [73]. Symptoms of anxiety were evaluated in 24 (44%) of the studies, many using the Generalized Anxiety Disorder (GAD-2 or GAD-7) [74]. Perceived stress was another common outcome measure used in 18 of the 54 (33%) studies, with tools such as the Perceived Stress Scale [75]. It is interesting to note that four of the studies incorporated biological indicators of stress [49,58,62]. Other measures of change in mental distress included tools to track symptoms of trauma, alcohol use, and sleep or insomnia.

Impact on mental health and well-being was another common outcome, evaluated across 34 (63%) studies. The primary outcome was the assessment of impact on well-being (n=24, 44%), using standardized tools such as the World Health Organization–5 (WHO-5) Well-Being Index [76] or the 14-item Warwick-Edinburgh Wellbeing Scale [77]. Other well-being outcomes included resilience (n=10), life satisfaction (n=7), quality of life (n=5), mindfulness (n=5), self-compassion (n=3), and social connection (n=2). These measures are congruent with many of the tools that focused on mindfulness and psychoeducation for coping with stress.

Impact on work-related outcomes was examined in 37 (68%) studies. Common work-related outcomes included work productivity and performance, burnout, job satisfaction, work engagement, and subjective perceptions of work stress and working conditions. Work performance outcomes, noted in at least 15 (30%) studies [17,23,32,38,39,41-43,45,47,51,56,60,63,67], included standardized tools such as the Work Limitations Questionnaire, the WHO Health and Work Performance Questionnaire, and the Work Productivity and Activity Impairment Questionnaire. Many of these tools included presenteeism or absenteeism measures, but several studies evaluated these separately [23,40,43,45,63,67], with some noting the productivity cost for organizations [23,47,67]. Burnout was another relatively common outcome noted in 12 (22%) of the studies [25-27,33,40,44,45,49,50,61,62,68], and evaluated using tools such as the Maslach Burnout Inventory or Bergen Burnout Inventory. Positive measures of job satisfaction were noted in four studies, often tracked through self-report [16,19,22,63]. Work engagement was another outcome noted in 9 (17%) studies [22,24,26,29,36,46,56,63,66], typically measured by tools such as the Utrecht Work Engagement Scale. In addition to work outcomes, 14 (26%) studies evaluated whether using a mental health app could change users’ perceptions of their working conditions including perceived work stress [25,26,31,35,51,57], job demands or strain [16,31,36], psychosocial climate [21,23], and perceived social support, including support from leaders and co-workers [39,43,63,64].

Other outcomes used to track impact included changes in mental health knowledge [21,32], cognitive function [28,50,58], physical function [49], or engagement in healthy behaviors related to diet and exercise [23,57]. Help-seeking or health care utilization was another outcome noted in 7 (13%) studies [21,23,42,43,47,54,60], particularly with apps that promoted connection to mental health resources. It is interesting to note that very few studies evaluated behavior change.

Process of Implementation

While most studies evaluated the overall effectiveness of the technology on individual or work-related outcomes, only a limited number of studies (n=5) focused on evaluating how the process of implementation affected these outcomes. Protocol studies included in this review proposed examining the relationship between the extent of app usage [19] in terms of its influence on the overall effectiveness of the intervention and understanding strategies needed to initiate and promote the use of the app [22]. Implementation outcomes in 3 other studies involved the evaluation of individual, contextual, and implementation-related barriers and facilitators that influenced the uptake, use, and overall impact of the mental health intervention [66,68,69]. Process measures were incorporated in order to generate recommendations for optimizing the utilization and effectiveness of the mental health technology.

Discussion

Principal Findings

This scoping review identified 54 studies (43 completed RCTs and 11 protocols) evaluating mobile health (mHealth) applications in workplace contexts. The main trends in the research were (1) research on workplace mental health apps has expanded rapidly in recent years, particularly following the pandemic; (2) the majority of studies were conducted in health and social services, with few in male-dominated industries; (3) about half of the 44 distinct apps adopted a structured, self-guided approach with modules or activities for the user to complete, another third were unstructured apps to be used as needed, and 11% of the apps were designed to be used in conjunction with a wearable sensor or web-based program; and (4) commonly evaluated outcomes included mental distress (including perceived stress, anxiety, and depression), mental wellness (well-being and resilience), and workplace-related outcomes (burnout, workplace engagement, work productivity, and performance), with limited studies on the process of implementation. Together, these findings highlight important gaps in both the design and evaluation of workplace mental health apps, underscoring the need for greater use of implementation science frameworks to guide future research.

Study Characteristics  

The results of this review profiled 43 completed studies and 11 study protocols that evaluate the impact of mHealth interventions on a working population. A previous systematic review, published in 2019, on e–mental health interventions in the workplace identified 22 studies, but only 3 studies included apps as a primary intervention [78]. The pandemic and post-pandemic era have precipitated a rapid increase in app-related research with workers (41 studies in the past 5 years). This may reflect a shift to mHealth approaches to support the growing mental health needs of workers. The number of completed randomized clinical trials (24/54, 44%) is promising, reflecting high-quality research designs to evaluate impact. Furthermore, this review included 11 trial registrations or published protocols, reflecting that the research is ongoing. The rise of digital technology and associated research on mental health apps for workers speaks to the importance of summarizing current evaluation practices and gaps in the evidence.

Types of Mental Health Applications

The review also highlighted diversity in the nature of mHealth interventions, with 44 different types of unique applications across the mental health spectrum, from promoting wellness to managing illness. In fact, several studies included wearable devices linked to the app, which may reflect a new technology trend. As noted in other review studies of mobile apps in the workplace, many were based on principles of cognitive behavioral therapy or mindfulness, combining information sharing with self-monitoring and self-management activities [11-13,79]. It is interesting to note, however, that even in the structured apps, the mechanisms of action were not consistently explained. There were a few studies that explicitly cited theories such as behavioral activation and cognitive behavioral approaches, but many of the app tools and techniques were not necessarily embedded within a psychotherapeutic protocol. This is a significant gap since explicit links between app design and expected outcomes are important aspects of evidence-based interventions [80].

Over half of the studies featured structured, self-guided apps with specific modules or activities that users were directed to complete, often over a 2 to 8-week period. Outcomes related to engagement, including completion of the activities, were more amenable to tracking “success” than apps that were less structured and available to use “as needed.” Given high attrition rates in app use, recommendations for optimizing engagement noted in the literature include providing “challenges” within the technology over short periods to build habits, providing regular “push notifications,” and connecting with an online coach or therapist [81]. Additional research is needed, however, to systematically evaluate the effect of these implementation strategies, and longitudinal research is needed to track the impact of these strategies over time [79,81]. Only a few apps (Headspace, Headgear, or Shift) had more than 2 studies examining their effectiveness, reflecting the need for more robust evaluation.

Study Participants

Although studies in this review were conducted across a range of sectors, the highest proportion (23/54, 42%) was in health and social services. Given the mental health crisis noted in health and social services that is leading to significant shortages in staffing [44,68], these studies indicate promising efforts to evaluate innovative strategies to support the mental health of this group of workers. Studies in other sectors were more limited, although there were several in service-based sectors (education and information technology), with comparatively fewer studies in male-dominated industries such as construction. Unfortunately, 10 of the studies did not specify the workplace context where the app was implemented, making it difficult to identify relevant recommendations for specific workplace sectors. Given the importance of the workplace context, future comparative research is needed to explore how mobile apps function differently across industry sectors and organizations. Furthermore, many studies included in this review lacked sufficient reporting details regarding the intervention, research setting, and evaluation plan, making it challenging for future reproducibility, comparability, and dissemination. There are guidelines, such as the 16-item mERA (mHealth Evidence Reporting and Assessment) checklist, that can be used to improve the reporting quality of interventions and support replicability of research [82]. This checklist provides comprehensive recommendations for study authors on adequately describing the content, context, and technology infrastructure to implement the mobile health technology with a specific population group [82].

Key Evaluation Variables and Tools

The review highlights 3 different categories of evaluations conducted by studies exploring mental health apps. User engagement and feedback were important outcomes tracked across many studies, either objectively through app metrics or subjectively through a recall of usage patterns. Feedback about usability, acceptability, and relevance was tracked qualitatively but also through standardized measures such as the Mobile Application Rating Scale [70] and the System Usability Scale [71]. Tracking user perceptions, engagement, and interaction with mental health apps is critical in establishing a relationship between usage (dose) and desired outcomes (response) [79]. Individual studies varied in their operational definitions of these constructs along with how they were measured, ranging from objective measures of usage (frequency and duration of use) that are passively collected from within the app to subjective measures of user ratings of usefulness and satisfaction, making it difficult to conduct cross-study comparisons.

Currently, there is no consensus on what constitutes optimal engagement for a mental health app to produce desired effects [83,84]. One study exploring different types of app use, features, and their effect on outcomes found that the right combination of different features (ie, learning, goal setting, and self-tracking) and their use at a moderate level could lead to desired outcomes, thereby refuting the common belief that higher usage always leads to better outcomes [85]. Collectively, each of the identified measures could be valuable in determining whether users who reported higher levels of usage, found the app useful, and were satisfied were more likely to engage with the intervention optimally, subsequently leading to significant improvements in outcomes. However, only 1 study protocol proposed assessing the link between the dose (the extent to which an app needs to be used) and the response (a significant improvement in outcomes) [19].

In addition to assessing usage and engagement, user experience and feedback were typically assessed only in feasibility studies. Since apps require iterative updates and maintenance, assessing user experience and feedback over time is important when attempting to deploy them in a large-scale, real-world setting [84].

In terms of impact, it is interesting to note that there was a range of approaches to evaluation, ranging from impact on reducing mental distress (78%) to improving mental well-being (63%) or work-related outcomes (68%). Since all of the studies were conducted in the context of the workplace, these workplace-related outcomes may be important for organizational decision-makers who are considering the value of investing in mHealth technology. Although there were 37 studies that considered workplace outcomes, there was a wide range of tools to evaluate these outcomes, ranging from downstream measures of productivity and burnout to more upstream measures of work engagement, perceived working conditions, and job satisfaction. Although there is a growing body of evidence indicating that digital mental health interventions implemented in workplaces can improve work effectiveness and productivity [13], tracking these outcomes can be complex, considering that studies need to speculate when changes in these workplace performance outcomes will take place and embed appropriate follow-up time points to measure them [86]. Study designs need to clearly outline the components of the intervention and the mechanisms of change to define what constitutes proximal (short-term or immediate) outcomes and distal (long-term) outcomes and determine appropriate time points for the measurement of these outcomes.

Even when apps are designed to be evidence-based, this does not guarantee that the expected outcomes will occur. Evaluating a complex mental health intervention in the context of the workplace requires a multilevel approach, considering the implementation process as well as its impact [81,86]. While several studies in this review acknowledged the complexities of implementing a digital intervention in a workplace setting, very few systematically evaluated the process of implementation, and there appears to be a gap in the application of existing theories to systematically guide this inquiry.

Several implementation science theories and frameworks can be useful to guide further research in understanding and unpacking the complexities of implementing technology across different contexts and achieving large-scale adoption and sustained effects. For example, the Consolidated Framework for Implementation Research (CFIR) provides a summary of evidence-based constructs across 5 domains: the intervention, the individual user, the inner context, the outer context, and the process of implementation [87]. These constructs can help identify and explain the facilitators and barriers to the uptake and impact of the intervention [87]. Another similar framework, called the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework [88], highlights six key domains that can be used to guide researchers in evaluating the success of an mHealth intervention implemented in an organizational context. It considers the characteristics of (1) the condition being addressed (eg, mental health), (2) the technology-based intervention, (3) the adopters of the technology, (4) the value proposition for the users, (5) the organization where the technology is being implemented, and (6) the wider system or society. The NASSS framework also reflects on the complex and dynamic interaction between the 6 domains over time, which can help answer questions related to users’ engagement with apps over time [88]. Systematic evaluation of the implementation process can facilitate exploration of key facilitators and barriers that influence the uptake and effectiveness of the intervention. While traditional health care interventions typically emphasize evaluation of their effects on key outcomes, mental health app–based interventions necessitate the examination of the different ways in which users engage with the intervention as well as exploring other implementation-related and contextual factors that influence the uptake and overall impact [79-81,83,86].

Limitations

To our knowledge, this review is one of the first to summarize current research regarding approaches to evaluating mental health apps specifically in the context of the workplace. While a broad objective was established at the onset of the review in order to sufficiently capture all relevant studies, we restricted our search to studies that were published in English to ensure feasibility in terms of timelines and resources. Another limitation was the focus only on descriptions of the evaluation approaches outlined in the studies. There was no formal assessment of the study quality or synthesis of study findings. Future systematic reviews are needed to critically evaluate the current research evidence to identify the impact of mental health apps, and to profile what works for whom and in what context. This is particularly important to assist organizational leaders, policymakers, and stakeholders in deciding how to invest in and optimize the use of mental health apps with frontline workers.

Recommendations

Results from this study informed the following recommendations for future research regarding app-based mental health interventions in the workplace. First, future research should improve transparency and consistency in the reporting of studies by incorporating guidelines such as the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) [89]. Second, evaluation studies need to identify and measure appropriate outcomes by examining the congruence between intervention goals and outcomes when exploring the effectiveness of mental health apps. Third, there is a need for longitudinal studies to assess the long-term effects of mental health apps, particularly when evaluating their impact on workplace outcomes that may require a longer duration to detect meaningful changes. Fourth, long-term effects of mental health apps, particularly when evaluating their impact on workplace outcomes that may require longer duration to detect meaningful changes. Fifth, there is a need to investigate strategies for integrating mental health apps into existing workplace settings by considering organizational factors and fostering an implementation science lens. Applying implementation theory can generate new insights about what works for whom and in what context. Sixth, there is a need to investigate strategies for integrating mental health apps into existing workplace settings by considering organizational factors and fostering an implementation science lens. Applying implementation theory can generate new insights about what works for whom and in what context. Finally, a future systematic review is needed to synthesize and appraise current evidence regarding the effectiveness of mental health apps for workers.

Conclusions

In conclusion, the past decade has seen tremendous advancements in the field of mHealth innovation, including its application in the context of the workplace. Traditional experimental approaches to evaluating mHealth interventions, however, may not be sufficient to capture the complexities of the facilitators and barriers that shape the adoption of the intervention and the sustainability of its effects within diverse workplaces. This methodological gap calls for research in this field to embrace new implementation science methodologies to truly capture the nuances of technology adoption and reliably measure its effectiveness over time.