We followed the 5 steps of scoping review methodology [14]: specifying the research question, identifying relevant literature, selecting studies, extracting data, and synthesizing the findings. The protocol for this review is registered on the Open Science Framework. We also adhered to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines (Checklist 1) to ensure consistent and transparent reporting [15]. To examine and summarize the breadth of evidence regarding the evaluation of mental health apps for working adults, we developed the following research questions based on the population, intervention, comparison, and outcome (PICO) framework:

Studies were included in this review if they met the following inclusion criteria: (1) the primary study intervention was a mobile mental health app that was tested by employees; (2) the study evaluated the effectiveness, implementation, impact, or use of the mental health app; (3) study participants included workers currently employed in any field and/or implemented in the context of a workplace; (4) the study included any quantitative or qualitative evaluation design; (5) the study was published in a peer-reviewed journal; and (6) the study was available in English. Studies were excluded if: (1) the intervention in the study was primarily a web-based or internet-based intervention (ie, not a mobile app) and (2) the study design only focused on app development (not evaluation) of a mental health app for workers. Conference proceedings and gray literature were not included, as the key focus of this review was to present a summary of published peer-reviewed studies that systematically evaluated mental health apps.

We searched the following electronic databases: PsycINFO, Embase, MEDLINE, Cochrane Database, OVID Healthstar, and OVID Emcare for peer-reviewed studies from January 2000 to August 2023. A comprehensive search strategy, including search terms, subject headings, and syntax requirements for the databases, was adapted from a previous systematic review of digital mental health interventions for the workplace [11] and was modified in consultation with a health science librarian. Search terms included a combination of constructs such as “Mental Health,” “Workplace,” and “Mobile Application” (Multimedia Appendix 1). The search was conducted from May 2023 to August 2023.

Data screening and extraction were carried out by 2 independent reviewers (SA and ML) using Covidence systematic review software (Veritas Health Innovation), an online tool that supports screening and data extraction for reviews, to facilitate a systematic screening process. All studies were screened independently by 2 reviewers, and any discrepancies in study selection were discussed in a review meeting. A third reviewer, SM, was consulted to resolve any disagreements between the 2 primary reviewers and to improve clarity in decision-making when necessary.

Data extraction was an iterative process designed to develop and refine a data charting sheet in Microsoft Excel to ensure all relevant data were captured comprehensively. The fields of data extracted included study characteristics such as the year, country of publication, study design, workplace context or industry where the study was conducted, intervention characteristics, timeframe, and evaluation approaches including both assessment tools and outcomes explored in the study. The initial extraction chart was piloted with 5 studies and was later revised to capture additional information relevant to our research questions. Analysis of these data included a summary of the study characteristics, types of app interventions, study populations, and approaches to evaluation (variables and evaluation tools). Descriptive summaries of the trends in each category will be outlined.

We searched 6 electronic databases, identifying 288 studies after the removal of duplicates. After screening titles and abstracts, 85 studies were deemed eligible for full-text screening. Subsequently, 54 studies met our inclusion criteria for being included in this scoping review (Figure 1). Data from these studies were extracted and charted to generate a comprehensive summary of the breadth and trends in the body of evidence of mental health app evaluation in the workplace. Of these 54 studies, 11 were protocols for studies in progress; hence, no results were available for these studies at the time of the review. Upon careful consideration of the protocols and clinical trial registrations, we were able to gather relevant information related to the key outcomes being evaluated and the processes by which the studies were being conducted, which was deemed sufficient to answer our main research question: “How are mental health apps being evaluated with workers and/or in the context of the workplace?” As a result, we decided to report these ongoing studies in our review.

Among the 54 studies included, approximately two-thirds were randomized controlled trials (RCTs; protocols or trial registrations n=11 [16-26]; completed RCTs n=21 [27-48]; pilot RCTs n=3 [49-51]). A smaller subset of studies adopted nonrandomized methods such as pre-post nonexperimental designs (n=13) [52-63], with even fewer studies using mixed methods (n=5) [64-67] and qualitative designs (n=1) [68]. Over two-thirds of the studies (n=37) were published after the onset of the COVID-19 pandemic, reflecting accelerated remote work and rapid digital transformation in workplaces, with over 26% (n=14) of the studies on this topic published in the year 2022 alone. Table 1 outlines a summary of the key characteristics of all included studies.

A total of 44 unique apps were evaluated across the 54 studies included in this review. There was considerable variability across these apps in terms of their therapeutic approach, goals, functionalities, target population, and intervention duration. Over half of the apps (23/44, 52%) included a programmatic structure with predetermined guidelines or prescribed instructions on how to use them, while 34% (15/44) were open-ended and user driven without a fixed guideline for use. Based on these distinctions, we broadly classified these apps as: (1) structured self-guided, (2) unstructured self-guided, and (3) an adjunct to other interventions. See Table 2 for an overview of the identified apps categorized by type, including proposed purpose and target population noted in the study.

Structured self-guided apps (n=23 apps) had a structured pathway in the form of programs or modules that users needed to complete, often in the form of modules, lessons, exercises, or activities. For instance, the HeadGear app (cited in 8 studies) is a 30-day intervention informed by mindfulness-based techniques that include daily challenges users are instructed to complete [17,38,45,51,54,60,67]. Headspace included daily guided 10-minute mindfulness practices or techniques [26,27,40,68]. Several studies noted that the app content was informed by evidence-based therapies, including behavioral activation and mindfulness [24,38,48,53,62,70], acceptance and commitment therapy [37,42,43,68], and principles of cognitive behavioral therapy [32,59], but varied widely in terms of the extent to which they adhered to evidence-based guidelines.

Unstructured self-guided apps (n=15) were more open-ended and provided users with the choice to select and engage with intervention components on demand. For instance, the Oiva app [66] consisted of information sessions on stress management and well-being based on the principles of acceptance and commitment therapy but had no prescribed guidance for engaging with the app. Many apps included meditation or relaxation exercises [28,31,35,44,48,49], self-monitoring tools (eg, mood, physical activity, and stress) [52,57,65,69], and/or links to resources [55,57].

Finally, there were 5 apps that were designed to be implemented in conjunction with other interventions. Three were used to supplement in-person training [21,34,42], and three included wearable devices linked to the mental health app [34,35,58]. One study, for example, combined a brief in-person mindfulness training followed by monitoring of respiratory patterns through a wearable device, with real-time biofeedback received through the app [34].

In terms of intervention duration, most studies (n=38) implemented a fixed intervention period, typically ranging from 2 to 8 weeks, with 4 weeks being the most common. Outcomes were generally assessed at baseline and at the end of the intervention, although some studies included longer follow-up periods, with the longest extending to 12 months [38].

Participant demographics varied significantly across the studies. Some studies targeted the implementation of the intervention with specific gender groups, among individuals with symptoms of mental health issues and/or in specific workplace sectors. For instance, one study targeted male participants experiencing mild to moderate symptoms of stress and/or depression to assess the feasibility of a multicomponent, technology-based (P4Well) intervention [50]. Two other studies specifically recruited individuals from male-dominated industries to evaluate the effectiveness of a therapeutic app (HeadGear) [38,45]. One study focused on evaluating the effectiveness of a well-being and stress management app for working women [30].

Study participants worked in a range of industries, including health and social services (n=23), which largely aimed at implementing therapeutic apps, large corporations or consumer-based industries (n=5), information and technology (n=5), education (n=2), agriculture (n=1), apprenticeships (n=1), and construction (n=1). A subset of studies implemented their interventions across multiple industries (n=7), and a few studies did not explicitly specify the contextual setting (n=10) in which the research was being conducted.

We identified three primary ways (Figure 2) in which studies evaluated mental health apps with workers: (1) app engagement and utilization; (2) impact evaluation (mental distress, mental health, and workplace outcomes); and (3) implementation process evaluation. The variables and tools within each approach are summarized in Multimedia Appendix 2.

This scoping review identified 54 studies (43 completed RCTs and 11 protocols) evaluating mobile health (mHealth) applications in workplace contexts. The main trends in the research were (1) research on workplace mental health apps has expanded rapidly in recent years, particularly following the pandemic; (2) the majority of studies were conducted in health and social services, with few in male-dominated industries; (3) about half of the 44 distinct apps adopted a structured, self-guided approach with modules or activities for the user to complete, another third were unstructured apps to be used as needed, and 11% of the apps were designed to be used in conjunction with a wearable sensor or web-based program; and (4) commonly evaluated outcomes included mental distress (including perceived stress, anxiety, and depression), mental wellness (well-being and resilience), and workplace-related outcomes (burnout, workplace engagement, work productivity, and performance), with limited studies on the process of implementation. Together, these findings highlight important gaps in both the design and evaluation of workplace mental health apps, underscoring the need for greater use of implementation science frameworks to guide future research.

The results of this review profiled 43 completed studies and 11 study protocols that evaluate the impact of mHealth interventions on a working population. A previous systematic review, published in 2019, on e–mental health interventions in the workplace identified 22 studies, but only 3 studies included apps as a primary intervention [78]. The pandemic and post-pandemic era have precipitated a rapid increase in app-related research with workers (41 studies in the past 5 years). This may reflect a shift to mHealth approaches to support the growing mental health needs of workers. The number of completed randomized clinical trials (24/54, 44%) is promising, reflecting high-quality research designs to evaluate impact. Furthermore, this review included 11 trial registrations or published protocols, reflecting that the research is ongoing. The rise of digital technology and associated research on mental health apps for workers speaks to the importance of summarizing current evaluation practices and gaps in the evidence.

The review also highlighted diversity in the nature of mHealth interventions, with 44 different types of unique applications across the mental health spectrum, from promoting wellness to managing illness. In fact, several studies included wearable devices linked to the app, which may reflect a new technology trend. As noted in other review studies of mobile apps in the workplace, many were based on principles of cognitive behavioral therapy or mindfulness, combining information sharing with self-monitoring and self-management activities [11-13,79]. It is interesting to note, however, that even in the structured apps, the mechanisms of action were not consistently explained. There were a few studies that explicitly cited theories such as behavioral activation and cognitive behavioral approaches, but many of the app tools and techniques were not necessarily embedded within a psychotherapeutic protocol. This is a significant gap since explicit links between app design and expected outcomes are important aspects of evidence-based interventions [80].

Over half of the studies featured structured, self-guided apps with specific modules or activities that users were directed to complete, often over a 2 to 8-week period. Outcomes related to engagement, including completion of the activities, were more amenable to tracking “success” than apps that were less structured and available to use “as needed.” Given high attrition rates in app use, recommendations for optimizing engagement noted in the literature include providing “challenges” within the technology over short periods to build habits, providing regular “push notifications,” and connecting with an online coach or therapist [81]. Additional research is needed, however, to systematically evaluate the effect of these implementation strategies, and longitudinal research is needed to track the impact of these strategies over time [79,81]. Only a few apps (Headspace, Headgear, or Shift) had more than 2 studies examining their effectiveness, reflecting the need for more robust evaluation.

Although studies in this review were conducted across a range of sectors, the highest proportion (23/54, 42%) was in health and social services. Given the mental health crisis noted in health and social services that is leading to significant shortages in staffing [44,68], these studies indicate promising efforts to evaluate innovative strategies to support the mental health of this group of workers. Studies in other sectors were more limited, although there were several in service-based sectors (education and information technology), with comparatively fewer studies in male-dominated industries such as construction. Unfortunately, 10 of the studies did not specify the workplace context where the app was implemented, making it difficult to identify relevant recommendations for specific workplace sectors. Given the importance of the workplace context, future comparative research is needed to explore how mobile apps function differently across industry sectors and organizations. Furthermore, many studies included in this review lacked sufficient reporting details regarding the intervention, research setting, and evaluation plan, making it challenging for future reproducibility, comparability, and dissemination. There are guidelines, such as the 16-item mERA (mHealth Evidence Reporting and Assessment) checklist, that can be used to improve the reporting quality of interventions and support replicability of research [82]. This checklist provides comprehensive recommendations for study authors on adequately describing the content, context, and technology infrastructure to implement the mobile health technology with a specific population group [82].

The review highlights 3 different categories of evaluations conducted by studies exploring mental health apps. User engagement and feedback were important outcomes tracked across many studies, either objectively through app metrics or subjectively through a recall of usage patterns. Feedback about usability, acceptability, and relevance was tracked qualitatively but also through standardized measures such as the Mobile Application Rating Scale [70] and the System Usability Scale [71]. Tracking user perceptions, engagement, and interaction with mental health apps is critical in establishing a relationship between usage (dose) and desired outcomes (response) [79]. Individual studies varied in their operational definitions of these constructs along with how they were measured, ranging from objective measures of usage (frequency and duration of use) that are passively collected from within the app to subjective measures of user ratings of usefulness and satisfaction, making it difficult to conduct cross-study comparisons.

Currently, there is no consensus on what constitutes optimal engagement for a mental health app to produce desired effects [83,84]. One study exploring different types of app use, features, and their effect on outcomes found that the right combination of different features (ie, learning, goal setting, and self-tracking) and their use at a moderate level could lead to desired outcomes, thereby refuting the common belief that higher usage always leads to better outcomes [85]. Collectively, each of the identified measures could be valuable in determining whether users who reported higher levels of usage, found the app useful, and were satisfied were more likely to engage with the intervention optimally, subsequently leading to significant improvements in outcomes. However, only 1 study protocol proposed assessing the link between the dose (the extent to which an app needs to be used) and the response (a significant improvement in outcomes) [19].

In addition to assessing usage and engagement, user experience and feedback were typically assessed only in feasibility studies. Since apps require iterative updates and maintenance, assessing user experience and feedback over time is important when attempting to deploy them in a large-scale, real-world setting [84].

In terms of impact, it is interesting to note that there was a range of approaches to evaluation, ranging from impact on reducing mental distress (78%) to improving mental well-being (63%) or work-related outcomes (68%). Since all of the studies were conducted in the context of the workplace, these workplace-related outcomes may be important for organizational decision-makers who are considering the value of investing in mHealth technology. Although there were 37 studies that considered workplace outcomes, there was a wide range of tools to evaluate these outcomes, ranging from downstream measures of productivity and burnout to more upstream measures of work engagement, perceived working conditions, and job satisfaction. Although there is a growing body of evidence indicating that digital mental health interventions implemented in workplaces can improve work effectiveness and productivity [13], tracking these outcomes can be complex, considering that studies need to speculate when changes in these workplace performance outcomes will take place and embed appropriate follow-up time points to measure them [86]. Study designs need to clearly outline the components of the intervention and the mechanisms of change to define what constitutes proximal (short-term or immediate) outcomes and distal (long-term) outcomes and determine appropriate time points for the measurement of these outcomes.

Even when apps are designed to be evidence-based, this does not guarantee that the expected outcomes will occur. Evaluating a complex mental health intervention in the context of the workplace requires a multilevel approach, considering the implementation process as well as its impact [81,86]. While several studies in this review acknowledged the complexities of implementing a digital intervention in a workplace setting, very few systematically evaluated the process of implementation, and there appears to be a gap in the application of existing theories to systematically guide this inquiry.

Several implementation science theories and frameworks can be useful to guide further research in understanding and unpacking the complexities of implementing technology across different contexts and achieving large-scale adoption and sustained effects. For example, the Consolidated Framework for Implementation Research (CFIR) provides a summary of evidence-based constructs across 5 domains: the intervention, the individual user, the inner context, the outer context, and the process of implementation [87]. These constructs can help identify and explain the facilitators and barriers to the uptake and impact of the intervention [87]. Another similar framework, called the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework [88], highlights six key domains that can be used to guide researchers in evaluating the success of an mHealth intervention implemented in an organizational context. It considers the characteristics of (1) the condition being addressed (eg, mental health), (2) the technology-based intervention, (3) the adopters of the technology, (4) the value proposition for the users, (5) the organization where the technology is being implemented, and (6) the wider system or society. The NASSS framework also reflects on the complex and dynamic interaction between the 6 domains over time, which can help answer questions related to users’ engagement with apps over time [88]. Systematic evaluation of the implementation process can facilitate exploration of key facilitators and barriers that influence the uptake and effectiveness of the intervention. While traditional health care interventions typically emphasize evaluation of their effects on key outcomes, mental health app–based interventions necessitate the examination of the different ways in which users engage with the intervention as well as exploring other implementation-related and contextual factors that influence the uptake and overall impact [79-81,83,86].

To our knowledge, this review is one of the first to summarize current research regarding approaches to evaluating mental health apps specifically in the context of the workplace. While a broad objective was established at the onset of the review in order to sufficiently capture all relevant studies, we restricted our search to studies that were published in English to ensure feasibility in terms of timelines and resources. Another limitation was the focus only on descriptions of the evaluation approaches outlined in the studies. There was no formal assessment of the study quality or synthesis of study findings. Future systematic reviews are needed to critically evaluate the current research evidence to identify the impact of mental health apps, and to profile what works for whom and in what context. This is particularly important to assist organizational leaders, policymakers, and stakeholders in deciding how to invest in and optimize the use of mental health apps with frontline workers.

Results from this study informed the following recommendations for future research regarding app-based mental health interventions in the workplace. First, future research should improve transparency and consistency in the reporting of studies by incorporating guidelines such as the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) [89]. Second, evaluation studies need to identify and measure appropriate outcomes by examining the congruence between intervention goals and outcomes when exploring the effectiveness of mental health apps. Third, there is a need for longitudinal studies to assess the long-term effects of mental health apps, particularly when evaluating their impact on workplace outcomes that may require a longer duration to detect meaningful changes. Fourth, long-term effects of mental health apps, particularly when evaluating their impact on workplace outcomes that may require longer duration to detect meaningful changes. Fifth, there is a need to investigate strategies for integrating mental health apps into existing workplace settings by considering organizational factors and fostering an implementation science lens. Applying implementation theory can generate new insights about what works for whom and in what context. Sixth, there is a need to investigate strategies for integrating mental health apps into existing workplace settings by considering organizational factors and fostering an implementation science lens. Applying implementation theory can generate new insights about what works for whom and in what context. Finally, a future systematic review is needed to synthesize and appraise current evidence regarding the effectiveness of mental health apps for workers.

In conclusion, the past decade has seen tremendous advancements in the field of mHealth innovation, including its application in the context of the workplace. Traditional experimental approaches to evaluating mHealth interventions, however, may not be sufficient to capture the complexities of the facilitators and barriers that shape the adoption of the intervention and the sustainability of its effects within diverse workplaces. This methodological gap calls for research in this field to embrace new implementation science methodologies to truly capture the nuances of technology adoption and reliably measure its effectiveness over time.