A randomized controlled trial (RCT; trial registration: ClinicalTrials.gov NCT01738256) was organized to study the effectiveness of 3 low-intensity lifestyle interventions against a no-intervention control. The interventions were (1) a face-to-face, group-based ACT intervention, (2) an ACT-based mobile intervention, and (3) a Web-based educational intervention with cognitive behavioral therapy components. The RCT consisted of a 10-week period during which the interventions were delivered, followed by a 26-week follow-up period. The recruitment began in August 2012 and ended in February 2013. The last follow-up measurement was performed in December 2013. The RCT was performed in 3 study centers in Finland—Jyväskylä, Kuopio, and Helsinki. The purpose of the lifestyle interventions was to improve the participants’ well-being and activate them to make beneficial changes in their everyday life. The design of the RCT was described in detail by Lappalainen et al [34]. See Multimedia Appendix 1 for the CONSORT-EHEALTH checklist [35].

The primary outcome of the RCT was psychological flexibility (measured with the Acceptance and Action Questionnaire II, AAQ-II [36]). The main analysis of the RCT found that although AAQ-II did not change significantly, psychological flexibility related to weight issues (Acceptance and Action Questionnaire for Weight-Related Difficulties, AAQW) improved significantly in both ACT intervention groups (in manuscript in preparation by Marjukka Kolehmainen and colleagues). This paper presents a secondary analysis, which focused solely on the mobile intervention group and investigated the usage of the mobile intervention and the dose-response relationship between usage and the change in psychological flexibility (AAQ-II).

Participants were recruited through newspaper advertisements seeking overweight individuals suffering from psychological stress. The inclusion criteria were (1) age between 25 and 60 years, (2) body mass index (BMI) of 27-34.9 kg/m², (3) psychological stress (at least 3/12 points in the General Health Questionnaire (GHQ-12 [37]), and (4) the possibility to use a computer and Internet connection. Out of the 645 individuals who responded to the advertisements, 339 fulfilled the inclusion criteria and consented to participate. After baseline examinations, further 41 participants declined to participate or were excluded because of findings in baseline measurements. Thus, 298 participants started the actual study. The participants consisted of 48 males and 250 females. Their mean age was 49.0 (SD 7.6) years and BMI was 31.1 (SD 3.0) kg/m².

In this study, only the mobile intervention participants were analyzed. Altogether, 85 of 339 (25.1%) participants were randomized to the mobile intervention group. Of these, 78 (92%) received the intervention, and 75 (88%) participated in postintervention measurements. One participant did not provide postintervention results on psychological flexibility and thus was excluded, leaving 74 (87%) participants for the analyses.

The mobile intervention group received their ACT intervention through a mobile application, called Oiva Figure 1, which has been previously described by Ahtinen et al. [37]. The aim of the app was to improve psychological flexibility by teaching the users the 6 core processes of ACT. The processes are (1) Being present, that is, having a nonjudgmental contact with psychological and environmental events; (2) Self as context, that is, being aware of one’s flow of experiences without attachment to or investment in them; (3) Acceptance, that is, actively embracing feelings and inner events without trying to change them; (4) Cognitive defusion, that is, changing the undesirable functions of thoughts and feelings, for example, by observing them without attachment to them; (5) Values, that is, identifying things that are truly important to an individual and that help determine life directions; and (6) Committed action, that is, committing to concrete goals and actions based on personal values [19].

The content was provided as 45 exercises, divided into 4 main modules. Each exercise was coded according to the ACT process or processes it addressed. There were 14 exercises related to the Being present process including, for example, a mindful sitting exercise and a mindful eating exercise [39]. The Self as context process had 5 exercises, for example, the “Floating leaves on a moving stream” metaphor [39]. The Cognitive defusion process had 9 exercises, for example, the “Passengers on a bus” metaphor [40]. The Acceptance process had 8 exercises, for example, the “Tug of war with a monster” metaphor [40]. The Values process had 8 exercises, for example, the “Attending your own funeral” exercise [39]. The Committed action process had 10 exercises, for example, a goal-setting exercise [39]. Of the exercises, 9 were associated with 2 processes.

Most exercises were short, taking about 1-3 minutes to complete, and were provided both in text and in audio format in Finnish. The participants were free to use the exercises in any order while the app provided subtle guidance by highlighting the next recommended exercise and module. The aim was to guide the users to proceed from easier exercises and basic skills to more challenging ones. A feasibility study was conducted to ensure the app’s readiness for the RCT, reported by Ahtinen et al [38]. Currently, mobile and Web versions of the app are available in Finnish [41-43].

In the beginning of the study, the participants were invited to a 1.5-hour group meeting where a trained psychologist gave a 30-minute presentation about the principles of ACT. The features of the mobile app were introduced by a researcher. The participants were given Android mobile phones (ZTE Blade or ZTE Skate; ZTE Corporation, Shenzhen, China) with the app preinstalled along with printed user instructions. The participants were allowed to use the phones as their personal phones, but as the intervention period was only 8 weeks, it was not expected of them. The phones and the app were briefly tested in the group to make sure everyone knew how to use them. The participants were encouraged to use the app independently and actively, a few times a week, for the following 8 weeks. The participants did not get any feedback or support during the course of the intervention. The suggested number of exercises to be performed per session was 1-3, but the participants were encouraged to find personally appropriate ways of use. At the end of the intervention period, the participants attended postintervention measurements and returned their phones. Researchers had no access to the app during the intervention, and the app and its contents remained constant during the intervention.

This study analyzed the data collected during the baseline measurements and postintervention measurements of the RCT as well as the usage log data collected by the mobile app. The psychological measurements and prior experience in using technologies were collected through Web-based questionnaires.

The main outcome measure of the study was psychological flexibility—or, more precisely, psychological inflexibility—measured with the AAQ-II [36]. The AAQ-II is a 7-item questionnaire that assesses experiential avoidance and psychological inflexibility. The statements of the questionnaire (eg, “I’m afraid of my feelings” or “Worries get in the way of my success”) are rated on a scale from 1 (=never true) to 7 (=always true). Higher scores on the AAQ-II reflect lower levels of psychological flexibility. Thus, improvement in psychological flexibility is defined as a decreased score on the AAQ-II.

Weight-specific psychological flexibility was measured using the AAQW [44], which is a 22-item questionnaire measuring acceptance of weight-related thoughts and feelings and their interference with valued actions. The statements include, for example, “I try hard to avoid feeling bad about my weight and how I look” and “If I’m overweight, I can’t live the life I want” and are rated on a scale from 1 (=never true) to 7 (=always true) [44]. Higher scores on the AAQW reflect lower psychological flexibility regarding weight.

Participants’ prior experience in using smart mobile phones, mobile wellness apps, and wellness devices was assessed in the beginning of the study as part of the baseline questionnaire. Each participant was assigned 3 binary attributes based on his or her prior experience in using technologies: (1) Smart mobile phone owner (if the participant owned a smart mobile phone before joining the study), (2) mobile wellness user (if the participant had used wellness-specific apps on the mobile phone), and (3) wellness device user (if the participant had used wellness devices, such as pedometers or heart rate monitors).

The mobile app recorded usage log files locally in the mobile phone. The log files were obtained from the phones at the end of the intervention period. The log files were analyzed to extract altogether 15 usage metrics, detailed in Table 1. The usage metrics can be divided into 3 categories: metrics describing the intensity of use (metrics 1-7 in Table 1), metrics describing the usage of content (metrics 8-13), and metrics describing the ways of use (metrics 14 and 15).

First, individual usage sessions were identified and their durations were calculated. Then, individual exercises performed, and whether they were performed by reading or listening, were identified. The usage of exercises related to the 6 processes of ACT was studied based on the coding of exercises. First, the number of exercises performed was calculated for each process and normalized by the number of exercises belonging to the process, to account for there being a different number of exercises related to the processes (varying between 5 and 14). Then, a percentage of use compared with the total number of exercises performed by the participant was calculated for each process, indicating what the participants focused on the most.

The dose-response relationship between the usage metrics and the change in psychological flexibility was analyzed using linear regression analyses on continuous variables. First, univariate linear regression models were used, and then potential confounders (age, gender, baseline GHQ-12 score, and baseline AAQ-II and AAQW scores) were added to the models. Regression coefficients (B’s) with their 95% confidence intervals and significance levels were reported for both unadjusted and adjusted models. In addition, the R² values were reported for the unadjusted models to assess the magnitude of the effect.

To assess the effects of usage between those whose psychological flexibility improved and those whose flexibility did not improve, binary logistic regression analyses were conducted. The participants were divided into 2 groups based on the change in psychological flexibility during the intervention. Those whose psychological flexibility increased (ie, the change in the AAQ-II score was negative) were labeled “improvers” and those whose psychological flexibility remained unchanged or decreased (ie, the change in the AAQ-II score was zero or positive) were labeled “nonimprovers.” The cutoff of zero was used because AAQ-II does not have reference values for clinically significant change. First, univariate logistic regression models were used to test the independent effect of each usage metric and then, multiple regressions were employed to adjust for the potential confounders. The odds ratios (ORs) with their 95% confidence intervals and significance values were reported both for the unadjusted and for adjusted models. In addition, medians and interquartile ranges (IQRs) of usage metrics were reported for both groups.

The usage metrics found to be significant in the regression analyses were further investigated for associations with baseline variables. As the distributions of the usage metrics were skewed, Spearman correlation was used for continuous variables and Mann-Whitney U test for cases where 1 variable was categorical.

The statistical analyses were performed using the IBM SPSS Statistics version 22 (IBM Corp, Armonk, NY, USA). Statistical significance was set at P<.05.

The RCT was approved by the ethics committee of the Central Finland Health Care District, and written informed consent was obtained from all participants. The approval and consent also covered the secondary analyses performed. The trial was registered at ClinicalTrials.gov with the identifier NCT01738256.

The mean level of psychological flexibility in the mobile ACT intervention group was 20.4 (95% CI 18.3-22.5) at baseline and 18.5 (95% CI 16.4-20.7) at postintervention. The mean change was −1.9 (95% CI −3.2 to −0.5) and the within-group effect size (Cohen’s d) was 0.2, indicating a small improvement in psychological flexibility.

Table 2 presents the participants’ baseline characteristics. The baseline score of psychological flexibility (AAQ-II) was significantly correlated with the change in psychological flexibility during the intervention (ρ=−.33, P=.004). Also, the correlation between weight-related psychological flexibility (AAQW) and the change in psychological flexibility indicated slight association (ρ=−.20, P=.09).

When the participants were divided into 2 groups (improvers, n=38; and nonimprovers, n=36), the mean change from baseline to postintervention in psychological flexibility was −6.3 (95% CI −7.6 to −5.0) for improvers and 2.8 (95% CI 1.6-3.9) for nonimprovers (P<.001). The within-group effect size was 0.8 for improvers and −0.3 for nonimprovers.

The mobile app was available to the mobile intervention participants for a median of 58 days (IQR 55-60). During this period, they used the app as follows. The median number of usage sessions was 21 (IQR 12-35), the number of usage days was 15 (IQR 9.0-24), and the number of usage weeks was 7.0 (IQR 4.0-8.0). Less than half of the participants, 31/74 (42%), used the app for 8 weeks or more.

The median total duration of use was 4.7 (IQR 3.2-7.2) hours and the number of exercises performed was 63 (IQR 46-98). The median session duration was 13.5 (IQR 9.8-17) minutes. The median completion percentage was 91% (IQR 64%-96%), that is, the participants completed most exercises. The most used exercise types were Being present (26%, IQR 20%-33%) and Self as context (24%, IQR 17%-30%).

The usage was mostly focused on the first half of the intervention period, as the median usage pattern was 2.4, IQR 1.7-4.1. Most exercises (81%, IQR 63%-92%) were performed by listening instead of reading.

The usage metrics that were found to be significant predictors of changes in psychological flexibility (Tables 3 and 4) were analyzed for associations with the participants’ baseline characteristics (Table 2).

Older participants used the app more actively, as indicated by the number of exercises (ρ=.25, P=.03) and the total duration of use (ρ=.25, P=.04). Older participants used the Self as context exercises (ρ=−.33, P=.005) less than younger participants.

Women used the app more than men in terms of the number of usage sessions (median 23, IQR 13-38 vs median 14, IQR 8.8-19; P=.04), usage days (median 16, IQR 10-25 vs median 11, IQR 5.8-16; P=.03), and usage weeks (median 7, IQR 5.0-8.0 vs median 4, IQR 3.0-8.0; P=.04).

Participants who owned a smart mobile phone before the study were less likely to use the app actively than those who did not own a smart mobile phone, based on the number of usage days (median 11, IQR 6-18 vs median 16, IQR 12-25; P=.02) and usage weeks (median 5.0, IQR 3.0-8.0 vs median 7.0, IQR 5.5-8.0; P=.005), the number of exercises (median 52, IQR 25-71 vs median 79, IQR 50-108; P=.02), and the total duration of use (median 220 minutes, IQR 131-324 vs median 319 minutes, IQR 227-461; P=.01). Smart mobile phone owners also used Cognitive defusion exercises (median 11%, IQR 7%-16% vs median 14%, IQR 11%-18%; P=.04) and Acceptance exercises (median 11%, IQR 3%-13% vs median 13%, IQR 11%-19%; P=.003) less.

This study focused on metrics describing the usage of a mobile ACT intervention and their associations with the change in psychological flexibility during an 8-week intervention. The purpose of the ACT intervention was to increase psychological flexibility and thereby support value-based approach in life. The aims of this study were to investigate the dose-response relationship between usage and the change in psychological flexibility and to identify the usage metrics most strongly associated with improvement. The importance of psychological flexibility has been highlighted in a review by Kashdan and Rottenberg [45] describing flexibility as a fundamental aspect of health. There is ample evidence on the value of psychological flexibility. For example, in many forms of psychopathology, the processes associated with flexibility are absent. It is also proposed that psychological flexibility may be helpful not only to people suffering from different pathologies but also to highly functioning people in finding greater efficacy and fulfillment in their daily lives [45].

On the group level, the mean change in psychological flexibility was −1.9 points, indicating a small improvement with an effect size of 0.2. The psychological flexibility of about half of the participants (38/74, 51%) increased and the mean change in this group was −6.3, indicating a large improvement with an effect size of 0.8. In comparison, Ly et al [29] reported a within-group effect size of 0.5 for psychological flexibility for an ACT-based mobile self-help program.

A dose-response relationship between usage and the change in psychological flexibility was found. The usage metrics that were the most strongly associated with the change in psychological flexibility were related to the intensity of use, that is, the number of usage sessions, the number of usage days, the number of usage weeks, the number of exercises performed, and the total duration of use. Logistic regression analyses confirmed these findings. However, the R² values of individual usage metrics, ranging from .04 to .12, as well as the R² of the combined model (.05) consisting of all significant usage metrics showed that the predictive power of the usage metrics was relatively low. Interestingly, the session duration was not associated with improvement of psychological flexibility. This may be due to the fact that as the exercises were relatively short, the session duration was naturally rather limited.

A dose-response relationship was found also for 2 metrics describing content usage. Active usage of exercises related to the Acceptance process and less active usage of Self as context related exercises predicted increased psychological flexibility. Logistic regression analyses confirmed these findings and also indicated that more active usage of Cognitive defusion related exercises was associated with increased flexibility. Hayes et al [19] found that, based on a series of small-scale dismantling studies, Acceptance and Cognitive defusion related exercises had the strongest evidence of effectiveness.

When comparing participants whose psychological flexibility increased (improvers) and whose did not increase (nonimprovers), we saw that the total duration of use was more than 6 hours for improvers compared with 4 hours for nonimprovers. This result highlights that, even when an intervention is delivered through a mobile app, the participants need to commit substantial time and effort to the intervention in order to gain benefits. In comparison, in the study by Bricker et al [23] the average total usage time of the Web-based smoking cessation intervention was about 2.9 hours, and in the study by Carlbring et al [24] a depression intervention was used, on average, for 3.5 hours.

Older age, female gender, and not owning a smart mobile phone before the study were associated with more active use. The reason why not owning a smart mobile phone before the study affected usage may be that participants who already had a smart mobile phone were more reluctant to take a secondary phone into use. Another factor may be that those who did not own a smart mobile phone before were actually motivated by receiving a new phone for a few weeks.

Donkin et al [11] investigated the associations between several usage metrics and improvement in depression during a 12-week Internet intervention. Similar metrics and associations with outcome as in our study were found for the following usage metrics: (1) the total number of minutes spent in the program and (2) the total number of activities completed. There was also an interesting similarity in the total usage time in these 2 studies, despite the fact that one of the interventions was delivered through the Internet whereas the other was a mobile app. Donkin et al [11] reported an average total usage time of 5.9 hours for those whose depression improved compared with 4.9 hours for those who did not achieve improvements. In our study, the median total usage time was 6.0 hours for improvers versus 4.0 hours for nonimprovers. Ly et al [32] reported a study of an ACT-based mobile intervention for stress management. In their study, 16/36 (44%) participants adhered to the study for all of the 6 weeks of the intervention. In our study, using a similar criterion, 31/74 (42%) participants used the mobile app weekly for the entire 8-week intervention period.

Studying the usage of content in digital interventions is a relatively new area of research, made possible by automatic and detailed tracking of usage. Only a few studies have attempted to investigate the associations between the usage of content and improvement in mobile interventions. Heffner et al [33] studied the usage of a mobile ACT intervention for smoking cessation and found that only 2 of the 10 most popular features were associated with smoking abstinence and, furthermore, that there were several features among the less popular ones that were associated with smoking abstinence. Our study provided preliminary evidence that focusing on Acceptance and Cognitive defusion related exercises was associated with improvement of psychological flexibility, whereas focusing on Self as context related exercises actually decreased the likelihood of improvement. These results suggest that Acceptance and Cognitive defusion related exercises are needed in addition to Self as context related exercises in order to gain a significant effect on flexibility. Identifying the most effective components of interventions is important because it enables better design of interventions and making sure that users are exposed to those components instead of or in addition to the less effective ones. For example, in our app, most Self as context related exercises were presented earlier than Acceptance and Cognitive defusion related exercises. As many participants proceeded in the app in the suggested order, those who used the app less may not have been exposed to the more advanced exercises appearing later in the app as extensively.

Many studies have tried to identify demographic predictors of usage and depending on the type of intervention, different and often conflicting predictors have been found. Female gender has often predicted better adherence [8,46]. This was the case also in our study, but as there were very few men involved in the study, strong conclusions cannot be drawn from this result. Sometimes older [8,17,47] and sometimes younger age [8] have been found to predict better adherence. In our study, older participants were more active users. Owning a smart mobile phone before the study was associated with less active usage, which is in line with the finding by Christensen et al [8] who found that participants who were less experienced with using computers adhered better. Also, lower symptom levels have been found to predict adherence, especially in psychological disorders [8]. In our study, no such associations were found.

Other important predictors of adherence to interventions reported in the literature include, for example, outcome expectancy, social support, and autonomous motivation [3,48,49]. The importance of these factors probably increases in the case of self-help and digital interventions where users need to find the time and initiative to use the intervention on their own. Although we did not measure motivation, we found that the baseline level of psychological flexibility correlated with the changes in flexibility, that is, those with a lower baseline level of psychological flexibility improved more. This may mean that those who gained more benefits may have had more room for improvement and may have felt a greater need—and motivation—to use the intervention. Also, RCT as the study setup and having counselor contact have been found to predict usage [9,15,16]. Although in our study the study procedures included only 1 face-to-face meeting and only a short presentation by a trained psychologist, it may have contributed to active usage along with all the study procedures related to the RCT setup.

The study population consisted of participants who volunteered to take part in the RCT. Although they were randomly allocated to different interventions and therefore did not know which intervention, if any, they would receive, the group is likely to be biased and the results cannot be generalized. Volunteering in a research study often requires a lot of effort from the participants and they may therefore be highly motivated. Also, the large proportion of female participants, which is not uncommon in health trials, limits making generalizations.

The baseline level of psychological flexibility correlated significantly with the change in psychological flexibility. This means that those who had more room for improvement also experienced larger effects. It is not possible to evaluate whether this is due to higher motivation, greater perceived need, or features of the app.

We acknowledge that we performed a large number of statistical tests without correcting the statistical significance level of .05, which means that some of the low P values may have occurred by chance. However, this study was exploratory by nature and any results should be confirmed in future studies.

Regarding the design of the RCT, the duration of the mobile intervention (8 weeks) was arbitrary and does not correspond to real-life use of mobile apps. Because the participants had to return the phones at the end of the intervention period, there is no way of knowing how the usage would have evolved over time and whether additional benefits would have been gained as a result. On the other hand, the knowledge of the limited availability may also have motivated the participants for such active use. Also, the fact that the app did not run on the participants’ personal mobile phones probably affected the ways they used the app.

There were also some limitations related to the mobile app logs that were recorded automatically by the mobile phone. As the app did not require logging in and out for session, it could be running constantly in the background without being used, or the app could be closed accidentally during a usage session, which made the log files challenging to analyze. Therefore, the usage metrics may not be absolutely accurate. However, the rules for determining the usage metrics were verified by manually inspecting several random log files and therefore we can be fairly confident that the metrics describe actual usage satisfactorily.

The results indicated that active usage of a mobile ACT intervention was associated with improved psychological flexibility. Usage metrics related to the intensity of use and the usage of content were found to be the strongest predictors of the change in psychological flexibility. This study showed that rather intensive usage was required in order to gain benefits, and therefore the study highlights the need to measure and optimize the intensity of usage in mobile interventions. The study also implies that, to ensure effectiveness, the components known to be the most effective should be prioritized to make sure all users are exposed to them.

Future research should strive to study the usage of mobile apps in a more natural way, including using the participants’ personal mobile phones and not limiting usage time. The results provided in this study about the associations between usage of different types of exercises and outcomes highlight the capabilities of digital interventions to enable detailed analyses of what the participants are actually exposed to during interventions.