In modern society, many people live a fast-paced, high-stress lifestyle and do not engage in regular physical activity. A study showed that 80% of adults in America lack physical exercise [1]. A lack of exercise is the main cause of most chronic diseases [2] and the leading cause of death worldwide [3,4]. Research also suggests that regular exercise improves the mood and self-reported performance of white-collar workers [5]. With the advancement of mobile and wearable technologies, many mobile health (mHealth) apps have been developed to enhance exercise adherence and facilitate a healthy lifestyle. mHealth apps are defined as “mobile applications that assist consumers in self-management of overall wellness, disease prevention, and disease management” [6]. The mHealth app market has been steadily growing over the past few years. According to an industry report [7], more than 318,000 mHealth apps are available in the top app stores worldwide, and more than 200 apps are added daily.

Despite the popularity of mHealth apps, questions remain regarding their sustained effectiveness. According to a recent survey [8], 87% of patients used mHealth apps, but more than one-quarter of them stopped using these apps because of ineffectiveness in helping achieve their health goals or yielding tangible results. Prior research has also noted that many mHealth apps lack theory-based motivational techniques, rendering it difficult for these apps to sustain over the long term [9]. Moreover, mHealth apps are often designed and treated as black boxes; the design is not evidence-based [10,11]. These findings suggest that mHealth app developers must design tools to be more engaging for users. However, few studies have examined the quality of mHealth apps in terms of their effectiveness in motivating health-related behaviors or the design principles of these apps from the user perspective [12-14].

This study’s purpose was to improve the design of mHealth apps for exercise self-management by using an innovative research approach that combines field experiments with action design research (ADR). The literature examining adherence to exercise plans has defined and measured adherence as the percentage of completion of an exercise plan—often pertaining to plans agreed upon by patients and care providers in the case of medical studies [68,69]. We adopted this definition and measurement method by defining adherence to an exercise plan as the percentage of the exercise plan completed, and adherence was used as the dependent variable in this study. Notably, in this study, participants established their own plans. As they had very different health conditions and physical capabilities, it was not realistic for the researcher to create a universal plan. As the study participants likely had the desire to exercise, they set goals that they deemed beneficial to their health.

Specifically, our research objectives included (1) investigating whether sending motivational messages could improve adherence to exercise plans, (2) considering whether the motivational effect was impacted by personality, (3) testing message type (logical vs emotional) and repetition impact, and (4) exploring the possibility of involving a field experiment in the design process, learning new design principles from the results.

We followed an ADR approach to conduct this study. ADR, proposed by Sein et al [70], conceptualizes the research process as comprising the inseparable and inherently interwoven activities of building the information technology artifact, intervening in the organization, and concurrently evaluating the artifact. The process includes the following stages:

To examine the effectiveness of an mHealth app in promoting adherence to an exercise plan, we enhanced the ADR process by adding a field experiment step. In this stage, we used artifacts to examine the specific design principles. By understanding how users interact with the artifact, we were able to link the behavior study with design research and then verify and correct the design principles. We demonstrate the research process in Figure 1 and detail the process in the following section.

On the basis of a literature review, review of web-based comments for mHealth apps, and focus groups with users, we identified mHealth apps as lacking theoretical guided design and found that few mHealth apps are focused on motivation, which is one of the main problems, along with the lack of evaluation of effectiveness, as detailed in the Introduction section. We identified app developers, users, and health care professionals (ie, gym trainers in our research settings) as stakeholders. We recruited 3 gym trainers and their trainees during the process of forming our initial research scope. While working with the researchers (among them, a participant was also the developer of the research app), all stakeholders provided input during the process. The gym trainers contributed their thoughts based on their prior experience with exercise management, that is, persuasive principles, such as daily progress reports of diet and exercise plans to supervise performance of trainees. We also interviewed trainees to learn about their prior experience using mHealth apps and summarized several necessary functions, such as event schedulers, alarm reminders, and progress reports. By understanding the mechanisms motivating the trainees, we and the developer discussed the executability of certain mHealth functionalities and how to maximize the effectiveness of motivational messages. The stakeholders can benefit from using our designed mHealth app as a tool to monitor and manage their exercise plans and learn persuasive principles. We defined the scope of the project as follows: develop an mHealth app to remind and motivate users to adhere to their exercise plans and examine the impact of personality and message characteristics on adherence. We intended to generate design principles for health self-management and other motivational applications.

In this research, we designed a field experiment and received approval from the institutional review board of the University of Illinois. We conducted a field experiment using the mHealth app ActiveTrack. ActiveTrack is an exercise planning mobile app designed based on our theory-based message tailoring method that aims to examine the effects of (1) tailored communication, (2) motivational messages, and (3) repetition of messages on users’ adherence to their goal settings. ActiveTrack was available to anyone for download from the Apple store. We also promoted our app through gyms, where many members participated in weight loss programs. Our partnered trainers introduced ActiveTrack to their trainees but were not authorized to monitor their trainees’ behavior on the app. We targeted users who used the app for at least two months. The experiment included the following steps:

Therefore, we had a 2×3×2 design with 12 groups. Participants remained blinded to the assigned intervention to prevent them from being influenced by such knowledge;

In total, 160 users downloaded the app during the 2-month period; after excluding users who stopped using the app in the middle of the process, 89 participants remained. The attrition rate was 44.4% (71/160). We compared the characteristics of those who dropped out (inactive users) and those who remained (active users) and did not find any significant differences. Of these active participants, 55 were female and 31 were male (3 did not indicate their sex). Participants’ ages ranged from 19 to 56 years, with a mean age of 28.8 (SD 8.3) years. Among all active users, results showed no significant difference in adherence level across sexes and ages. In addition, we found that white-collar workers had 17% higher adherence level to exercise plans than blue-collar workers (t₆₃=−2.045, two-tailed; P=.04). Moreover, higher exercise frequencies per week were associated with a higher adherence level (t₆₃=2.341, two-tailed; P=.02). We further tested the difference in adherence between those who downloaded the app following their trainers’ recommendations and those who downloaded the app voluntarily, and the results showed no significant differences between the 2 groups (t₆₃=−0.254, two-tailed; P=.80).

We also compared the characteristics of those who dropped out (inactive users) and those who remained (active users). We found no significant difference in age and exercise frequency between the active and inactive groups. We used a chi-square test to examine any differences between active and inactive users in sex, job type, and working hours. There were no significant differences in sex and working hours between the 2 groups. However, in the active group, we found that white-collar workers had a higher adherence rate than blue-collar workers. In the inactive group, white-collar workers were more likely to quit. The job type was a critical factor that affected users’ retaining behavior.

The sample size was relatively small because of the difficulty of recruiting active users; however, the sample size was similar to that in research in the health care domain, which investigated the effectiveness of using mobile phones for health management. In a review of the effectiveness of mHealth and technology-based health behavior management interventions, all 7 studies related to physical activity behaviors included 17-150 participants [72]. Payne et al [13] systematically searched and described the literature on mobile apps used in health behavior interventions; 17 of the 24 studies reviewed had a sample of fewer than 100 participants. In addition, there was a limited impact of the low power caused by the small sample size in this study, as detailed in the following sections. The number of active participants in each group and the mean and SD of plan achievement are presented in Table 1.

Our results suggest that mHealth apps can be effective in promoting adherence. People with a feeling personality type were more likely to adhere to exercise plans than those with a thinking personality type. When receiving repeated reminders and messages, the adherence rates of people with a thinking personality type significantly improved and showed no significant difference from those of people with a feeling personality type. These results are consistent with prior studies that suggest that different personality traits are linked to certain behavioral tendencies, such as proneness to addiction [73] and excessive use of the internet [74]. Our study suggests that personality can contribute to the persistence of adherence to exercise plans.

Although the purpose of this research was to examine the effects of tailored messages based on personality and repetition, other tailoring criteria might be selected to motivate exercise behaviors. For example, demographic tailoring has been found to have effects that are independent of theory-based tailoring [75,76]. To further enhance our understanding of the effects of message tailoring and justify the applicability of our model, we conducted an additional analysis to investigate the interactions between sex and message type. We found that emotional messages to women and logical messages to men had a 35.56% higher adherence level (t₄₆=3.278; P=.002) than the other way around (emotional messages to men and logical messages to women). These findings warrant further analysis.

In the behavior research field, the issue of violation of parametric tests’ statistical assumptions is rather common, such as skewed distribution, heteroskedasticity, and violation of the independence of errors. Consequently, nonparametric tests have been proposed and widely used to address these issues because of their advantage of not being limited to the assumptions of distribution or homogeneity and because they can be applied to a small sample size [77]. Resampling methods, permutation (randomization test), and bootstrapping are common nonparametric methods.

Permutation tests use all possible permutations of a treatment variable or dependent variable, whereas all other independent variables are fixed to construct the exact null distribution using the available data to determine how extreme the observed test statistic of research interest is against the null distribution. The randomization test [78] relies on the same idea as permutation tests; however, these tests compare the observed statistic against the approximate null distribution generated by repeating a large number of permutations (eg, 10,000-time default in the R package) rather than all possible permutations. As the null distribution is generated empirically from the observed samples and makes no assumptions regarding the population, permutation (randomization test) is beneficial for evaluating the statistical significance or treatment effect of any variable of interest. Another resampling method, which is bootstrap, repeats the available sample and draws from the same sample with replacement to calculate the test statistics and construct an empirical distribution. In experiments, the bootstrap method is a useful way to examine the treatment effects of a designed experiment [79,80].

We used a randomization test and bootstrap resampling method to assess the main effects of the 3 factors and their interactions. By permuting 10,000 times in the randomization test and resampling with replacement 10,000 times for the bootstrap test, our robustness check of the treatment effects shows that the 2 main effects of personality and repetition and the interaction effect between personality and repetition were significant. The results are summarized in Table 4.

We also used a randomization test and bootstrap resampling method to examine the effect of sex-message tailoring. The effect was significant, with a 95% CI between 17.07% and 55.74% and P=.004 in the randomization test. Both resampling methods show consistent results in terms of the treatment effect.

According to our research process indicated in Figure 1, the evidence collected from the field experiment allowed us to summarize the insights presented in Figure 5. The design principles were revised based on the knowledge from the design and experimental effort and are described in Figure 5. These design principles were used to generate a series of app development problems, such as motivation for other behavioral changes and apps adapted to individual differences. These generalized principles are also presented in Figure 5.

In this research, we enhanced the ADR framework by developing the ActiveTrack iOS app and incorporating a behavior experiment into the process. When participants downloaded the app, they were randomly assigned to one of the groups for the behavioral experiment. On the basis of the users’ real-time responses, we revised the functions and designs of the app. As a result of the field experiment, we updated the design principles through the ADR process. Our field experiment results suggested that (1) the mobile app was effective in motivating adherence to exercise plans among users with both thinking and feeling personality types, (2) personality type was associated with the likelihood of adherence to exercise plans, and (3) repetition improved the exercise plan adherence rate of people with a thinking-type personality, and with repeated reminders, the adherence rate of people with a thinking personality type becomes similar to that of people with a feeling personality type. Our additional findings further imply that users’ behavior may depend on their (1) job types, (2) exercise habits, and (3) gender. By adding the behavior experiment as a component of ADR, we were able to learn design principles from the behavior experiment results and generalize these principles to a class of design problems: how to effectively motivate users through the personalized design of apps.

This study makes several contributions to the field. First, this research contributes to the literature on mHealth by investigating the effectiveness of mobile apps in motivating health behaviors. Second, this research improves our understanding of the impact of individual personality, message type, and repetition on the effectiveness of motivation. Specifically, through the field experiment, we were able to follow the participants and capture actual behavior change over time, which has been a challenge in behavior-related studies [81]. Third, field surveys, behavior intervention experiments, and the design science approach were normally used separately in the research. In this study, we used all of the above approaches to address the research questions. Combining these methods enabled us to actively design the app by observing usage data and using the design artifact to carry out the research. Thus, by incorporating a field experiment into the research process, we propose an enhanced ADR process. This research process enables us to better connect research with practical and relevant problems. Finally, the study provides several design principles that can be applied to the design of other mHealth and motivational apps, such as apps for lifestyle management, energy-saving behaviors, and adherence-to-treatment plans.

In this research, we used the Apple ResearchKit to build artifacts. Apple released ResearchKit, an open-source software framework for medical research, in March 2015. The ResearchKit framework comes with some predefined modules commonly used in research procedures, such as informed consent, surveys, and active tasks, to make the development of mobile apps feasible and more convenient for research. This framework has gained much attention from research institutions and companies owing to its global release [82-84]. However, using ResearchKit and similar development frameworks is still a novel method for scientific research projects, and there are many challenges in conducting research using this method. Through this project, we learned that some of those challenges include difficulty promoting the app because of the number of apps available in the same category in the Apple store, verifying the validity of inputs, and ensuring data security and privacy. In future research, some of these challenges can be overcome during the study design phase. For example, using the heart rate module, the app could more accurately estimate exercise completion. If a study targets a specific group of users, such as people with certain types of disease, it would be a good practice to promote the use of the app among targeted audiences, such as online patient support groups. It is convenient to download the app from the app store, so the barrier for a targeted population to use the app is low regardless of their physical locations.

This study has several limitations that can be addressed and improved in future studies. First, the sample size of this research was relatively small because of the difficulty in recruiting active users over the 2 months. However, the sample size was in line with that of a similar study. To address this issue, we used both a randomization test and bootstrap resampling methods to test our treatment effects. The consistent results showed that the main effects of personality and repetition and the two-way interaction between personality and repetition were statistically significant. Future researchers can collaborate with developers, health care providers, and health social networks to increase the potential of participant recruitment. Second, self-reported adherence might be biased. Although we implemented strategies such as reminding participants that the data would not be shared with anyone and honest responses would improve the design of the messages, the validity of self-reported exercise levels could not be verified. Some smartphone features, such as the tracking of heartbeats, can be used for more accurate estimates of exercise in future studies. Third, we attempted to limit the impacts of other confounding factors that may influence the behavior of the users through random assignment to groups; however, there might still have been influences from confounding factors. This is a difficulty that similar research has always faced, as it is difficult to make behavior change in such experiments totally independent of external factors [81].

Potential topics for future research include other strategies for personalized app design and motivation, such as using different theory-based tailoring methods, testing strategies to retain users by analyzing their churn behavior, and identifying ways to use the data collected from these apps to promote health and well-being (ie, different metrics to measure users’ lifestyle change). For example, we found that 14% (12/84) of the users achieved higher average exercise frequencies than their original exercise habits after 2 months. This finding implies that users may change their lifestyles using our mHealth app. Future research can also focus on the clinical validation of the experimental results, ways of using economic incentives to promote healthy lifestyles, new business models for health care providers and insurance companies to motivate adherence, and the use of other wearable devices, such as smartwatches, to motivate health behaviors.