Access to mental health care by essential workers and the people unemployed due to COVID-19–related business closures and social distancing policies has been challenging [1-3]. To address this problem, health care organizations have created free mobile apps for stress related to COVID-19. Although overall app use during COVID-19 has been low (16%) [4,5], technology companies report substantial increases in the use of their tools.

There is limited information about the effectiveness of mental health apps, particularly free, self-guided commercial apps. Research on self-guided apps is mixed, with some studies finding them to be minimally effective [6-8] and others reporting beneficial effects; we note here that most evidence points to the superiority of coach-based apps for depression and anxiety outcomes, but effect sizes for self-guided apps are still notable [9] and offer an opportunity for stress management in populations that do not have the financial or time resources to avail themselves of coaching services [10]. It is important to also note here that most studies that find positive effects use research grade tools with a paid participant pool and are typically not available to the public. Many commercial apps do include principals and features that are similar to research grade tools; however, there remains skepticism about the effectiveness of these derivations [11]. This has led to the need to create app review resources, such as One Mind PsyberGuide [12] and the American Psychiatry Association’s App Advisor [13], which provide ratings of app effectiveness, transparency, and usability. Still, evidence for free commercial apps is limited, and calls for additional research [14,15], particularly in the context of COVID-19 [16], have been made.

We previously reported on a large-scale survey of essential workers and people unemployed due to COVID-19 for their preferences for mobile apps for mood management [4]. In this study, we found that participants had strong preferences for apps that focus on mindfulness approaches, information about coping with COVID-19, symptom tracking, and connection with others. In this pragmatic clinical trial, we randomized 838 of these participants who indicated they were depressed or anxious or had suicidal thoughts in order to use 1 of 4 commercial apps for 4 weeks. We selected the 4-week time frame because in our past research, we found that this is an optimal dose of digital mental health in a distressed sample [17] and other research has found that this is the length of time participants tend to engage with these tools [18,19]. Thus, we are interested in addressing the issues of app use and outcomes pragmatically, as it would occur in actual practice. The main objectives of this study were:

Participants were recruited nationally via Prolific, an online research platform that includes several safeguards to preserve data quality [20-22] and minimize bad actors and has been shown to be reliable, efficient, and affordable for remote data collection for behavioral research [23]. Participants provided electronic informed consent prior to study completion. Additional survey safeguards were an attention check [24] and a review of open-ended items to screen out autofilled and nonsensical responses.

The study received ethical approval from the University of Washington Institutional Review Board (STUDY00010842). In the consent, participants were explained the purpose of the study, that it would be randomized to 1 of 4 mobile apps, and that they would be asked to complete surveys before treatment began and 4 weeks later. Participants were also told how data were stored and managed and approximately how long each survey would take.

Participants for this study were recruited from a larger study [4], which included a convenience sample of approximately 2000 participants that self-identified as COVID-19–designated essential workers or unemployed due to COVID-19 social distancing policies or COVID-19–related business closures. To identify as 1 of these 2 groups, participants responded to the following 2 questions: (1) Are you considered an essential worker during the COVID-19 pandemic? (2) Have you become unemployed as a result of the COVID-19 pandemic?

To be eligible for this study, inclusion criteria included (1) previously granting permission to be recontacted for future research; (2) age≥19 years, living in the United States, and English speaking; (3) access to a mobile device; and (4) report of depression (Patient Health Questionnaire [PHQ]-2 score≥3) [25], anxiety (Generalized Anxiety Disorder Scale [GAD]-2 score≥3) [26], risk for suicidal behaviors (Suicide Behaviors Questionnaire-Revised [SBQ-R] score≥7) [27], or a history of past suicide attempt [28]. Participants were offered crisis management resources when they endorsed the ninth item of the PHQ-9 or were over the cut-off for the SBQ-R.

Participants were recruited from October through December 2020, during the middle of the initial COVID-19 variant, and shortly after vaccines were available to the public. Additionally, most states (with few exceptions) were continuing to institute public closures of restaurants, gyms, and other enclosed public places, meaning the unemployment rate due to COVID-19 was still quite high. Hospital censuses were at historically high rates, and essential workers were still mandated to wear protective gear. Thus, the sample is representative of people living under peak pandemic conditions. Participants were randomized after completing a web-based baseline assessment of mood and paid US $1 (see the Measures section). Participants were randomized to 1 of 4 apps and asked to use their assigned app as instructed by the developers. Participants completed a web-based posttreatment survey at 4 weeks postrandomization and app assignment. After completing follow-up, participants were compensated US $4.

This remote pragmatic clinical trial used simple randomization with parallel assignment comparing 3 active apps to an attention control app. This study meets the definition of a pragmatic trial in that the study was designed to test the effects of mobile apps for depression, anxiety, emotion regulation, and suicide as they are typically used by the general public [29]. In pragmatic trials, the intent is to determine the effect of existing treatments in the context of real-world use compared to existing treatment options. In such trials, the control condition is not a placebo, which is not usually part of standard care [30]. Although a waitlist may be appropriate for a pragmatic trial, waitlist controls are appropriate only when this is part of usual practice and if they are ethically sound; however, previous research has found internal validity issues with waitlist controls, and in the context of self-guided commercial digital mental health, there is no waitlist control [31]. In our sample, which consisted of participants at risk for suicide, neither a placebo nor waitlist controls were ethical choices [32]. Thus, our decision to use an attention control app was based on what is considered appropriate for pragmatic trials of this nature in potentially high-risk populations [33,34].

We selected apps based on the following criteria: (1) they were free; (2) reflected desired app features during COVID-19, as identified in the survey study [4]; and (3) had good ratings on PsyberGuide [4]. The 3 active app interventions included (1) meditation (Calm), (2) COVID-19 coping (COVID Coach), and (3) chat and positive psychology (7 Cups of Tea). The attention control app used only mood tracking (Beautiful Mood) and did not include any intervention elements the other apps possessed (mindfulness meditation, emotional coping skills, social connection, or positive psychology approaches). Participants were randomized by study staff using random allocation functions in Microsoft Excel and received their app assignment through a URL to Google Play Store or Apple App Store. Participants confirmed app download prior to receiving compensation. Participants were blinded to the study hypotheses but not condition.

We feel it is important to note that although mental health apps are based on evidence-based treatment approaches, people use apps differently than the way they use traditional mental health services [35]. The optimal dose of mobile mental health apps is measured in the frequency of use rather than the number of weeks of use, and research shows that considerable improvement in mood and function can occur rapidly with digital mental health tools and as early as after 2 weeks of use [17,36,37]. We acknowledge that although other randomized clinical studies do show the greatest impact at 8 weeks [38], the general population tends to initially engage with digital mental health apps frequently over the course of 2 weeks, with notable disengagement by 4 weeks [11,18,19]. Based on the literature from the informatics field on typical engagement patterns with digital health tools in general, this is a common pattern of engagement and may mean the user has met their goal [18].

All data collected for this study are considered sensitive. We did not collect or store names, addresses, locations, IP addresses, or other digital identifiers. All survey data, including demographics, were immediately stored behind secure firewalls on servers at the University of Washington School of Medicine. The survey was developed by the study’s lead investigators (authors PAA and KAM), measures were selected for their validity and reliability, and we selected those measures that had been validated for online use. The survey was programmed into REDCap, a web-based survey program developed by Vanderbilt University [39]. It has been used extensively for clinical research and is Health Insurance Portability and Accountability Act (HIPAA) compliant, highly secure, and intuitive to use. After the survey was built, we tested it with research group members naive to the study for readability, programming bugs, and time to completion.

The survey was completed by volunteers identified through our initial sample [4]. Participation in the study, including survey completion, was voluntary. After volunteering on Prolific and providing consent, participants completed all measures in a web-based REDCap interface. The survey included 14 measures, each on a separate page, with 4-18 questions per measure. Participants were able to review and change their responses on each measure before proceeding to the next measure. The survey items were not randomized, as each scale used must be delivered in the order it was validated. We did not use skip patterns or other survey logic; participants were asked to complete all survey questions and had the option to not answer certain questions.

Prior to analyses, we examined the data and eliminated participants for not meeting inclusion or good-actor criteria. Good-actor criteria required participants to correctly answer an attention check item, answer at least 50% of the items on the survey, complete the survey faster than 33% of the median length of time, and not have any problems with the Prolific ID (not being an approved ID, being a duplicate ID, or having a missing ID). We performed t tests and cross-tabulations with chi-square tests to compare demographic and baseline clinical outcomes by missing data status at the follow-up time point. Chi-square tests examined the association between condition assignment and compliance (whether a participant used the app they were assigned or used an alternative app). All analyses were of the intention-to-treat (ITT) type. Analyses of variance were used to compare conditions on AIM, IAM, and IUS scores at follow-up, with Tukey honestly significant difference (HSD) tests making pairwise post hoc comparisons among all apps. Mixed effects models using restricted maximum likelihood estimation were built to test the linear time change on the PHQ-9, GAD-7, SBQ-R, and DERS-SF and to test for condition differences at follow-up and change slope. We applied mixed effects models with 2 time points with within-person nesting. We used this rather than alternatives, such as a simple regression, because mixed effects models efficiently (1) simplified simultaneous testing of within-person changes via testing slope coefficients and between-condition differences via testing condition coefficients; (2) facilitated testing as we built models progressively adding time trends, condition effects, and dosage effects; and (3) permitted the inclusion of random intercepts to account for variance at baseline, which was particularly important for testing models of dosage. Models were built and tested in an outwardly nested fashion, such that an initial null model was computed, followed by models that added a random intercept, random time component, condition assignment with Beautiful Mood as the reference variable (the attention control condition), and condition × time interaction effects. To test whether there was a dose-response relationship such that the app use frequency was associated with the rate of change on the PHQ-9, GAD-7, SBQ-R, and DERS-SF scores, we computed another series of mixed effects models for each outcome. An initial model included variables for time and frequency of use, a second model added condition terms for each app using Beautiful Mood as the reference, a third model added condition × frequency interaction terms for each app, and a fourth model included time, frequency, and a time × frequency interaction term. Model comparisons applied –2 log likelihood (–2LL), the Akaike information criterion (AIC), and Bayesian information criterion (BIC) deviance statistics. To test the impact of app use frequency on change over time, similar nested model testing was applied using an initial null model, followed by models that added dosage, dosage × time interaction, and condition assignment.

Figure 1 presents the Consolidated Standards of Reporting Trails (CONSORT) diagram. There were 3486 individuals assessed for eligibility, 2130 (61.1%) were excluded for not meeting good-actor criteria (n=988, 46.4%) or not meeting randomized clinical trial (RCT) inclusion criteria (n=1142, 53.6%). A total of 1356 (38.9%) individuals were randomized to Beautiful Mood (n=330, 24.3%), COVID Coach (n=355, 26.2%), Calm (n=336, 24.8%), or 7 Cups of Tea (n=335, 24.7%). Among those allocated to a condition, 838 (61.8%) participants completed the RCT baseline assessment, while 643 (47.4%) participants completed the follow-up assessment. In addition, 581 (90.4%) participants reported using the assigned app and 62 (9.6%) reported using a nonassigned app. For this ITT trial, all randomized participants were included in the primary analysis.

Table 1 presents individual-level demographic data. The analytic sample consisted of 838 adults, with a mean age of 31.1 (SD 9.5) years. Most patients identified as women (467/833, 56.1%) and White (616/838, 73.5%). Participants self-identified as unemployed due to COVID-19 (428/838, 51.1%) or COVID-19–designated essential workers (410/838, 48.9%). There were no significant differences in demographics by condition.

A total of 643/838 (76.7%) participants completed the follow-up assessment. There were no significant differences between those missing or not missing the follow-up assessment in the demographic data in Table 1 or clinical measures at baseline.

At follow-up, 62/643 (9.6%) participants reported being nonadherent to condition assignment and reported that they use a different app than the one they were randomly assigned to use. Participants who were randomized to Beautiful Mood were less likely to use their assigned app, while individuals randomized to Calm were more likely to use their assigned app (P<.001).

A cross-tabulation with the chi-square test found significant differences between the apps in the amount of use the participants reported; participants used Beautiful Mood more frequently and COVID Coach and 7 Cups of Tea less frequently (see Table 2).

ANOVA found a significant difference on the IUS between the conditions (mean_{Beautiful Mood} 72.9, SD 16.7; mean_{COVID Coach} 71.2, SD 15.4; mean_Calm 66.8, SD 17.3; mean_{7 Cups} 65.2, SD 17.7). Tukey HSD post hoc tests indicated Beautiful Mood is significantly more usable than Calm (mean difference 6.0, 95% CI 1.2-10.8, P=.01) and 7 Cups of Tea (mean difference 7.7, 95% CI 2.9-2.5, P<.001). COVID Coach was significantly more usable than 7 Cups of Tea (mean difference 6.1, 95% CI 1.2-10.9, P=.01). We found no significant differences in app acceptability (overall AIM mean 3.5, SD 1.0, 95% CI 3.4-3.6, P=.22) or appropriateness (overall IAM mean 3.6, SD 0.9, 95% CI 3.6-3.7, P=.48).

Table 3 displays the reporting sample size, mean scores, and SDs at each time point for the PHQ-9, GAD-7, SBQ-R, and DERS-SF for each app.

Examining the –2LL, AIC, and BIC deviance statistics for each of the 4 analyses revealed that the best-fitting model was a random intercept model with a linear time slope in 2 cases: the PHQ-9 (–2LL=9118, dev₁=63.2, P<.001; AIC=9128, dev₁=61.2, P<.001; BIC=9137, dev₁=71, P<.001; parameters=5) and GAD-7 (–2LL=8686, dev₁=71.7, P<.001; AIC=8696, dev₁=69.7, P<.001; BIC=8720, dev₁=64.9, P<.001; parameters=5). From baseline to follow-up, participants improved by an estimated –1.5 points on the PHQ-9 (SE 0.2, 95% CI –1.1 to –1.8, P<.001) and –1.3 points on the GAD-7 (SE 0.2, 95% CI –1.0 to –1.6, P<.001). Models that included condition main effects centered at the follow-up time point and condition × time were not significantly better fitting than the random intercept and time model. For the other 2 analyses, the SBQ-R and DERS-SF, the best-fitting models were the null models, with no random terms or interaction variables (SBQ-R: –2LL=6695, AIC=6703, BIC=6722, parameters=4; DERS-SF: –2LL=11,078, AIC=11,086, BIC=11,105, parameters=4). Thus, there were significant mean improvements in the PHQ-9 and GAD-7 scores but not the SBQ-R and DERS-SF scores, and the app condition was not associated with differences in any of the 4 analyses.

Mixed effects models were computed to examine the relation of frequency of app use with change over time on the PHQ-9, GAD-7, SBQ-R, and DERS-SF, over all conditions, controlling for the condition, for a condition × app use interaction, and for condition × time. The best-fitting model for the PHQ-9 included time, frequency of app use, and frequency × time interaction, as indicated by 2 of the 3 fit indices (–2LL=7868, dev₁=8.9, P=.002; AIC=7882, dev₁=6.9, P=.01; BIC=7914, dev₁=2.4, P=.12; parameters=7, P<.001). The BIC statistic, which penalizes for model complexity, was not statistically significant; therefore, these results should be viewed with some caution. All 3 fit statistics indicated that this same model structure was the best fit for the GAD-7 (–2LL=7481, dev₁=14.2, P<.001; AIC=7495, dev₁=12.2, P<.001; BIC=7527, dev₁=7.8, P=.01; parameters=7). None of the models that included condition or condition × frequency of app use was a significantly better fit, meaning we found no differences between the treatment groups on the impact that app use frequency had on change over time. For the SBQ-R and DERS-SF, none of the more complex models improved on the fit of the initial model that included the time and frequency of app use.

Salient model parameters for the best-fitting models were as follows. For the PHQ-9, when time, frequency of app use, and time × frequency interaction were included in the best-fitting model, the frequency of app use was not significant (estimate=0.05, SE 0.3, 95% CI 0.5-0.6, P=.86) and time was not significant (estimate=0.1, SE 0.6, 95% CI 1.0-1.3, P=.80), but the time × frequency interaction was significant (estimate=–0.6, SE 0.2, 95% CI 1.0 to –0.2, P=.003). There were similar findings for the GAD-7 such that when time, frequency of app use, and time × frequency interaction were included in the best-fitting model, the frequency of app use was not significant (estimate=0.1, SE 0.3, 95% CI 0.4-0.6, P=.78) and time was not significant (estimate=0.4, SE 0.5, 95% CI 0.6-1.3, P=.43), but the time × frequency interaction was significant (estimate=–0.7, SE –0.7, 95% CI 1.0 to –0.3, P<.001). Figure 2 depicts the actual mean score for each condition by frequency. For the PHQ-9 and GAD-7, those who did not use the app had no significant change on that measure over time; those who used the app more frequently improved more quickly than those who used the app less frequently. By the 4-week follow-up, however, there were no significant differences in outcome by frequency of app use (dose).

For the SBQ-R, the best-fitting model indicated that the frequency of app use was not associated with lower scores overall (estimate=–0.03, SE 0.2, 95% CI 0.4-0.3, P=.16), although time was significant (estimate=–0.1, SE 0.1, 95% CI 0.3 to –0.003, P=.05); interaction terms were not included. Therefore, when statistically controlling for the frequency of app use, SBQ-R scores decreased over time but there was no association between app frequency and change on the SBQ-R.

For the DERS-SF, the best-fitting model indicated that the frequency of app use was associated with lower scores overall (estimate=–1.5, SE 0.6, 95% CI 2.6 to –0.4, P=.01), but time was not significant (estimate=–0.1, SE 0.3, 95% CI 0.8 to –0.6, P=.84), and interaction terms were not included. Those who used their app frequently had lower scores on the DERS-SF at baseline and follow-up, with no change on the DERS-SF over time and no association between frequency of app use and change on the DERS-SF.

To the best of our knowledge, this is 1 of the first pragmatic trials of free commercial apps among essential workers or those unemployed due to COVID-19 experiencing emotional distress and suicide risk. Our primary findings were that commercial mobile mental health apps are found to be usable and acceptable and have a positive impact on depression and anxiety but not emotional regulation or suicide risk. Although we did not find any significant difference between the 3 active apps on outcomes, nor between the active apps compared to the control app, we did find that the frequency of app use during the 4 weeks had a significant and positive impact on depression and anxiety outcomes. However, we offer here that 4 weeks of engagement may not be sufficient to show changes in emotional regulation or suicidal behavior or that online interventions may not be potent enough to manage these mental health challenges, given that those with greater emotion dysregulation used the apps less throughout the period. Indeed, a recent study offering online interventions aimed at suicide prevention not only found no effects but also demonstrated more adverse events in those offered online care versus those offered care as usual [78]. We also cannot rule out regression to the mean, as people who used the attention control app had outcomes similar to participants who used the active apps.

Our previous work suggests that apps that focus on mindfulness, pandemic information, mood tracking, and connection with others are an acceptable means of managing stress during COVID-19 [4]. Our findings on the lack of differential clinical impact between apps is not surprising, given the data from other pragmatic trials of research-grade mobile apps. In large-scale remote, pragmatic clinical trials of mobile apps for depression, all apps found significant improvements in mood over an 8-week period but no differences between groups [17]. Our findings regarding the importance of app use on clinical outcomes have also been found in previous studies on research-grade mobile apps, where frequent use of a mobile mood app early in care resulted in better depression outcomes for those who were more severely distressed [17,79]. Although smaller, controlled trials of self-guided apps in a research context do find small but statistically significant differences in outcomes compared to waitlist controls or no treatment [76], this study, and other pragmatic trials to date, have not demonstrated that active apps are more effective than attention control apps [7,78].

The strength of this study is in its design: It is 1 of the first studies to evaluate free commercial mental health apps prospectively and independently in a large-scale, pragmatic RCT and to assess their impact on emotional distress, emotional regulation, and suicide risk in 2 suicide-vulnerable populations at the peak of the COVID-19 pandemic. The lessons learned from this study can be useful to people seeking free and readily accessible help for emotional distress. For the field, more work is needed to understand what role commercial apps can play for emotional distress, and the data from this study serve as a good starting point for understanding what is acceptable and effective and what optimal engagement should be.

Study limitations include the following:

There are several papers calling for more research to study the effectiveness of commercial mental health app interventions [7,10,83,84], specifically for COVID-19 [85-88], but published studies to date report only app downloads, aesthetics, and app use [6,83,89]. Our data suggest that essential workers and those unemployed who want self-guided mental health care found 4 commercially available apps both acceptable and usable and might receive emotional benefit from a variety of self-guided mental health apps, particularly if they use the apps frequently, but that regression to the mean cannot be ruled out, so improvement in symptoms may not be attributable to app use.