In recent years, mobile apps addressing health and fitness issues have grown at a remarkable rate. Although there are significant advantages to health apps, such as lower health care costs and increased accessibility to health advice [1], their full potential is still untapped, mainly because of a failure to engage users in terms of sufficient and effective use [2,3].

User engagement has been recognized as a key factor in determining the success of an app [4-6] because it is linked to the user’s intention to continue using a mobile app [7,8]. Continued use, a subset of behavioral engagement, is a critical issue because of the highly competitive nature of the mobile app market [8,9]. However, it is important for researchers to move beyond user continuance behavior and examine user engagement as a broader concept [10].

Originally influenced by the user experience movement in human-computer interaction (HCI), user engagement has become a buzzword in various areas, including mobile health apps [11]. However, there is no general agreement as to what constitutes user engagement or how it is operationalized and measured [11,12].

To answer the aforementioned research question in the best possible way, we first need to consider the scale. The recommendation when applying a self-reported measurement is to rely on previous questionnaires [24]. In the context of digital health, an eHealth engagement scale was developed [43]. We chose the eHealth scale because its characteristics are similar to ours. Both studies have an HCI setting [43], but they measured engagement with eHealth content on a website and in a laboratory context. The mobile aspect of our study is probably not as important because spatial data do not matter in both contexts. However, we still propose that through the smaller window of a phone, user engagement might be different. Therefore, this scale was not found to be an appropriate fit for this research context also because no other studies could be found that utilized this scale, causing us to question its robustness and validity.

A better fit was the UES by O’Brien et al [40], a psychometric tool used to measure user engagement with a digital system. Using empirical observations and theoretical elements of the flow theory, as well as John Dewey’s philosophy of experience, the original scale has in recent years been applied to over 40 published studies in various HCI settings, such as Web news, educational technologies, social networking, or information search [13]. Few studies have used this scale in its entirety, probably because of its length and insufficient data on how to administer it; moreover, the 6-factored solution was questioned in various studies [44,45]. As a consequence, another study conducted in 2018 presented a refined scale consisting of 30 items intended to measure the 4 dimensions of user engagement in HCI settings. Table 1 describes the 4 dimensions (based on [14,19,38)]. Researchers may use only subscales of the scale; however, user engagement as a holistic construct can then not be measured [40].

Therefore, we proposed the following hypothesis for our study:

Using the full-length UES proposed by O’Brien et al [40], we designed an online survey using EFS Survey, a software package created by Questback. We had to make 1 major adaptation to the existing UES: a translation of the questions from English to German as we conducted the survey in German-speaking countries. Not all items are suitable or compatible in another language [40]. Nonetheless, we did not make any item selections before data collection; we translated and included all the items. We slightly modified some of the wording to adapt them to the context of Ada.

We pretested the online survey under field conditions on 4 participants, using the available pretest feature of the EFS Survey. This allows pretest participants to attach comments to individual questions. After the pretest, small changes and refinements to the wording and layout were made. In addition, the data export feature was tested to confirm that the collected data could be exported.

Data collection took place for 2 weeks in April 2018. A convenience sample as a sampling type was used. Accordingly, the link to the online survey was distributed to selected people. In addition, the snowball system was applied, whereby the participants were asked to share the survey link with their circle of friends and acquaintances. We asked participants to first download the health app Ada, use it at least once, and then relate the questions to their experience with Ada in completing the survey. We assessed the survey questions using 5-point Likert scales. The 30 questions of the UES were randomized, and information about user engagement dimensions was hidden. The English items and their corresponding translations into German can be found in Multimedia Appendix 1.

When using the UES, it is suggested to perform factor analysis [40]. As the aims of this study were to understand the concept of user engagement and test the applicability of the UES to a mobile health app, a principal component analysis (PCA) with varimax rotation was performed. The main reason for choosing PCA instead of factor analysis was that the UES had originally been developed in English and that there is no German version to date.

Before conducting the main analysis, we considered several conceptual and statistical issues. These initial evaluations are of high importance because of the dependence of the quality of the data on the results of factor analysis [46]. We examined the sample size, communalities, and correlations between the items. In addition, the Kaiser-Meyer-Olkin (KMO) measure and Bartlett test of sphericity were considered.

Once this is checked, the number of extracting factors can be determined. All 3 criteria (scree test criterion, Kaiser criterion, and an a priori criterion) were discussed. Next, the factor structure was evaluated. To do so, the values of the factor loadings, which is the correlation between the original variable and its factor, were considered [47,48]. We followed the recommendation by Backhaus et al [46] and only assigned variables with loadings higher than 0.5.

After the factor structure had been defined, sum scores for each subscale were calculated. Sum scores are calculated as means for each subscale [40]. As some of the variables were reverse phrased (v_08, v_10-v_13, and v_23; see Multimedia Appendix 1), they were first recoded and transferred to the same variable so that values could be compared. Furthermore, because of the multidimensionality character of the UES, reliability in the form of internal consistency of the subscales was examined separately for each subscale using Cronbach alpha. Statistical data analysis was conducted using the statistics program SPSS version 23 (IBM).

In total, the survey link was viewed 363 times, which translated to 73 responses. As all the questions were programmed as mandatory questions, there are no missing values; thus, none of the participants needed to be excluded from further analysis. Out of 73 participants, 36 (49%) were female and 37 (51%) were male. Their average age was 39 years (SD 15.4 years), with the youngest participant being 18 years and the oldest participant being 73 years. Multimedia Appendix 2 contains a summary of participant demographics.

A PCA was conducted on the 30 items using varimax rotation. PCA was determined to be suitable, as verified by multiple criteria. First, the sample size of this study, 73, was above the minimum absolute sample size of 50 [48]. Second, there were correlations below 0.3. It is recommended to exclude variables that correlate below 0.3 or correlate above 0.9 with any other variable [47,49]. However, as no variables had zero correlations below 0.3, all variables were retained for further analysis. Furthermore, there were no correlations above 0.9. Third, communalities were mostly within the 0.7 range. With a sample size of 73, communalities of around 0.7 were deemed sufficient [49]. In addition, the factorability of the items was verified by the sampling adequacy of KMO=0.858 (meritious according to Kaiser and Rice [50]) and a significant Bartlett test of sphericity (χ²₃₀₀=1127.1; P<.001). In conclusion, based on the consideration of these criteria, factor analysis was determined to be suitable.

The next step was to obtain the factors. Overall, 4 factors had eigenvalues over Kaiser criterion of 1 and, in combination, explained 65.1% of the variance. The scree plot was ambiguous and showed inflections that would justify retaining both 2 and 4 factors. The a priori criterion indicated 4 factors [40]. Thus, 4 factors were retained because they accord with Kaiser criterion, scree plot, and the a priori criterion.

Furthermore, 5 items showed factor loadings below 0.5 (v_07, v_09, v_14, v_15, and v_26) and were therefore removed from PCA as their correlation with other variables was not strong enough. One variable, v_16, showed cross-loadings, as it loaded on factor 1 (0.533) and on factor 3 (0.582). Such variables either need to be excluded or interpreted with both factors, unless strong theoretical reasons speak against this [46]. In this case, this variable was theoretically expected to load on factor 3, and indeed, the higher factor loading of this variable was on factor 3. This variable was, therefore, kept and assigned to factor 3.

The results of the PCA are illustrated in Multimedia Appendix 3, in which the variables are listed within the respective factors in a descending order. Overall, this PCA yielded consistent subscales and a 4-factor solution as suggested by O’Brien et al [40]. Therefore, the factor labels were taken over. Only 1 of the 25 items, v_29, loaded on another factor (focused attention), as suggested by the original UES.

Factor 1, focused attention, accounted for 34% of the variance and consisted of items v_01-06 and v_29. This factor was the original UES’s focused attention subscale, with the addition of 1 reward element, v_29. Factor 2, perceived usability, accounted for 14% of the variance and consisted of items v_08 and v_10-13. Factor 3, aesthetic appeal, accounted for 9% of total variance and consisted of item v_16-20. This factor was the original aesthetics subscale. Factor 4, reward, accounted for 8% of variance and consisted of v_21-25, v_27-28, and v_30.

In addition, sum scores were calculated for each factor. The factor perceived usability had the highest sum, with a value of 294; second was the factor aesthetic appeal, with a value of 275.4; third was the factor reward, with a value of 259.5; and the lowest subscale score was for the factor focused attention, with a value of 198.1. In other words, users of Ada showed the highest engagement level with perceived usability and the lowest engagement level with focused attention.

Focused attention, aesthetic appeal, and reward subscale of the UES all had high reliabilities, with Cronbach alpha=.912, alpha=.852, and alpha=.910, respectively. However, the perceived usability subscale had a lower reliably, with Cronbach alpha=.693 (see Multimedia Appendix 4).

The results of this study confirm the proposed hypothesis by demonstrating that the UES developed for measuring user engagement with digital technology can be used to assess user engagement with health apps. We had to remove 5 items because of factor loadings below 0.5. Possible reasons for removing these factors are suggested in Multimedia Appendix 5. The modified 25-item version of the scale accounted for 65% of the variance in user engagement. Overall, 4 factors emerged: focused attention, perceived usability, aesthetic appeal, and reward.

In O’Brien et al [40]’s and our study, all items loaded on the same factors except for v_29 (“I felt involved in this experience”). In our study, it loaded on focused attention rather than reward. This can be explained by taking into consideration our research context of health apps and the flow theory. This item indicates absorption in the experience with Ada, and absorption is a major characteristic of the flow theory, therefore making it a good fit for the focused attention factor. Aside from this, the suggested factor structure could be replicated in this study.

Although our findings support the use of the German UES with health apps, this study is subject to certain limitations, and results should be interpreted with caution.

First, data were collected from a sample that was not selected randomly, and the characteristics of the population were not adequately represented in the sample. In addition, the number of participants, 73, is rather small, although it is sufficient for conducting factor analysis [48].

Second, the UES was administrated and adapted to the context of this study. All items were taken from English and translated by the authors. Therefore, some items may have a slightly different meaning to respondents than the items in the original instrument. In addition, some of the items did not make sense in a non-English context and had to be removed because of low factor loadings during analysis.

The third limitation concerns the UES itself. In this study, no items were added to the original scale, so construct validity is not threatened. However, this tool might not capture all the important determinants of user engagement. In addition, adding another research method, for example, interviewing the participants, would have contributed to the interpretation of our findings.

Fourth, engagement was only measured once during a short period and not across multiple sessions, which may represent a limited view on user engagement. Further research could examine user engagement by having participants complete the scale more than once as part of the same study. Researchers could then compare engagement among participants and between iterations [40] and take a long-term view on engagement.

Furthermore, the question arises if participants engaged fully and intrinsically with the health app by being initially asked to use it at least once. Other user engagement studies in different fields have had similar problems (eg, [43,51]). Future research should try to address this issue.

This study is among the first to investigate the use of the UES as a whole in the area of health apps and in the German language, so the findings cannot be compared with those of other studies. However, patterns of the UES may be compared with those of other app user engagement studies. Still, direct comparison of attributes and factors within the area of health app user engagement has to be treated with caution because of the different approaches used. This, finally, comes back to various perceptions of what constitutes user engagement and what does not.

Furthermore, the high context dependence of the defining features of user engagement are reported [11], and in the context of mobile apps, it is probably also a question of the mobile app type examined. For example, mobile app types might be divided into experiential and informational app types [52]. According to this distinction, Ada, the example of this study, falls into the category of informational apps and focuses on goal-oriented and utilitarian benefits rather than on social and hedonic benefits, as is the case for experiential app types. It has been discovered that the effect of time convenience on mobile app engagement is greater for informative mobile apps than for experiential apps [9]. Applying this to user engagement with Ada might lead to the conclusion that utilitarian benefits, such as time convenience, are better suited to explain the factor of reward than other mobile app types would have.

User engagement is a complex concept, and there is no general agreement as to what constitutes the phenomenon or how it is operationalized and measured [11,12]. This paper contributes in several ways to the growing literature concerning user engagement with health apps.

One major contribution of this study is the applicability of a German version of the UES [40] with a health app. On the basis of our findings, researchers and practitioners can now further investigate the user engagement concept. Future research could build on our findings and interpret the data gathered with the UES by applying a multiple method design. Owing to its scope and complexity, researchers have taken a multidimensional view of user engagement [5,13]. In addition, further research could investigate the use of the UES with apps in general and not be limited to health apps.

Given the importance of user engagement, the findings of this study could benefit practitioners, mainly marketers and app developers, in 2 ways.

First, marketers and app developers can gain a better understanding of user engagement. Evidence suggests that one of the main challenges for successful companies in mobile environments is to find ways to keep their users engaged [4-6]. Knowing what keeps users engaged has important implications for strategic retention management [9].

Second, findings based on the self-reported measurement of the German version of the UES with Ada suggest that the attribute that drives the highest engagement among users is perceived usability. This is followed by aesthetic appeal, reward, and focused attention.

An improved understanding of the attributes that drive user engagement should help app developers and marketers in creating and marketing attractive health care apps that will be used and appreciated in the long term. This could be a vital factor in increasing health literacy among users and, therefore, a contribution toward improving public health in general.