Noom (Noom Inc) is a mobile app that provides various lifestyle-related logs and is available for Android and Apple devices. The app provides interventions based on (1) food logging, (2) exercise logging, (3) weight logging, (4) in-app group activities, (5) article reading, and (6) communication with coaches through messaging [20]. Once users sign up and install the app, they are required to record their initial status information such as weight, BMI, and target weight. The coach encourages users to record their food intake, daily exercise, and weekly weight [20]. Furthermore, passive data, such as the number of steps taken, are automatically collected and saved in order to track users’ behaviors.

We obtained anonymous and deidentified log data of Noom users in the United States from August 8, 2013 to August 8, 2019 for a retrospective study. This study adhered to the TRIPOD [21] statement on reporting predictive models.

From a total of 93,696 users in the coaching program, we excluded users who did not take part in the 16-week weight loss program. From this group, we ruled out users who had not entered weight or meal records for the entire 16-week program. Although the coach encouraged users to enter their meal and weight log daily, those with incomplete records were considered users who did not successfully receive the intervention; moreover, we could not evaluate their outcomes in the model for weight loss at 16 weeks. We included users who had a target weight that was below the initial weight to exclude users with the purpose of weight gain. In addition, we excluded those whose height did not fall between 125 cm and 230 cm (n=19,129).

In users with a valid profile, we included only adult users (range: 18 to 65 years old) and excluded older adults over 65 years old and users under 18 years old. Furthermore, only overweight and obese users (BMI ≥25 kg/m² based on the definitions provided by the Center for Disease Control and Prevention [22]) were included in our prediction model to target users with a need for weight loss (n=67,301). Finally, we identified 17,867 eligible users with 16 weeks of weight records for analysis to evaluate the outcome, presented as predicted weight (Figure 1).

We developed an interpretable RNN model by modifying the reverse time attention model (RETAIN) [16}. As the purpose of the RETAIN model is to predict the risk of cardiac disease using sequential data on categorical data such as diagnosis (International Statistical Classification of Diseases, Tenth Revision), we improved the model to (1) give leverage to continuous variables such as weight input frequency and time-fixed variables such as user profile and (2) convert this into a regression model for the prediction of weight after 16 weeks while preserving the linearity of the model for interpretability.

We modified the prediction model to (1) consider time-fixed variables by concatenating initial status as auxiliary inputs, (2) use continuous variables by removing the embedding layer, and (3) convert this into a regression model by not using the activation function for classification (Figure 2).

Assuming we have N timeseries related to app usage with i length, where and k time-fixed variables where and a target series y of length i, where y=[y₁,…,y_i], and . By stacking N timeseries, we define a multivariate input series and contemporary multivariate input vector is denoted at time i by . Given X_i and x^(F), we leverage 2 types of data for our model to predict contemporary target values using deep learning , namely . The concatenation operation is denoted by ⊕ and element-wise multiplication is denoted by ⊙. A_:i is denoted by slice 1-dimensional tensor A from first index to ith index (Multimedia Appendix 1).

We attempted to predict the participant’s weight after a 16-week intervention using the output vector y_i ∈ {0, ∞} by concatenating c_i from time-variant data and x^(F) and applying a linear transformation with w weight vector, where w ∈ . This is expressed as follows:

where c_i denotes the context vector c_i ∈ and is the sum of the time-level variables weighted by attention α_i, β_i. Time-level attention weighting α_i is a scalar value that reflects the relative importance of the data at time i, and it ranges from 0 to 1. β_i is a vector with N length that explains the importance of each value of within i time.

Consistent with RETAIN, interpretation of the weight prediction model involves getting the time-level attention weighting and variable-level attention weighting from each RNN g_i, h_i [16]. Therefore, the equation can be rewritten, and the contributions of the predicted value of the model can be calculated as follows:

Therefore, the contribution of the Nth time-variant variable ω(ŷ_i, x_jN)^(T) and that of the mth time-fixed variable ω(ŷ_i, x_m)^(F) can also be written as

where each contribution coefficient of the time-variant variable is , and each contribution coefficient of the time-fixed variable is .

We subsequently generated 3 dimensions of time-variant variables based on app usage patterns (window, week, variables) to provide immediate feedback if users performed behaviors that infringed on weight loss from week to week, and to improve model performance, and for real-time prediction of future target value, we fixed all elements of y as the weight at 16 weeks.

Furthermore, we leveraged information on the shape of the weight loss trend week to week. We used shape-based timeseries clustering on the weight logs; individuals’ weights were assigned into clusters at each time for real-time prediction. The weight loss trajectory can be labeled at each time point and can represent the historical shape of weight loss, not actual weight (kilograms) from the first week to each subsequent week. This trajectory does not require the outcome of the model. Therefore, the real-time model can leverage a user’s historical shape of weight loss at each time. To exclude the fact that weight (the outcome of the model) is directly used as a feature, we normalized each weight log by converting their means to 0, and the variance to 1. Then, we used the clustering number in the prediction process. Dynamic time warping was used to measure the distance between user weight loss trajectories; we used k-means clustering with this metric. We found 5 to be the optimal number of clusters. For individuals, a weight loss cluster was assigned for each time. An example of weight loss is included in Multimedia Appendix 2.

The performance of the regression model was evaluated using the mean absolute percentage error (MAPE) [23]. MAPE was calculated for the entire intervention (from week 1 to week 16) to identify model accuracy. Furthermore, to evaluate the consistency of the model’s performance, 5-fold cross-validation (training set: 70% and validation set: 30%) was performed.

We identified the contribution coefficients by exploring the time-level attention weighting and variable-level attention weighting for a window of 16 weeks to ascertain the contributions of week 1 to week 16 in predicting weight. Furthermore, we explored the attention weighting of the predictive model by calculating the average of the attention weightings used for predicting weight in test data.

The study was approved by the institutional review board at Advarra (CR00123125). The anonymous and deidentified nature of the retrospective log data made obtaining informed consent unnecessary.

In this study, a transparent RNNs model was constructed utilizing both timeseries and time-fixed data for weight loss prediction after a mobile-based intervention targeting obesity. Currently, the explanation methods based on attention mechanisms are referred to as explaining individual predictions because attention can be generated in each person by providing a contributing role in prediction. However, our research first demonstrated the connection between individuals and a global pattern with homogenous length timeseries. Through the identification of time-level and variable-level attention weighting, we identified that weight, meal, and exercise inputs are important factors for weight loss in the context of mobile apps. Furthermore, the findings regarding time-level attention weighting may suggest that constant use of the app is important in mobile-based interventions. In individual profiles, we identified that being male and being overweight were relatively negative contributing factors to weight after the mobile program intervention.

In addition, this predictive model considered both time-variant variables, such as weekly adherence to the app, and time-fixed variables, such as the initial status of users to predict week-to-week weight change. Recently, interpretable RNN architecture has been proposed in machine learning areas [16,24-26]. However, the simple application of this architecture may be difficult in research problems involving classification or regression, such as those in medical fields where demographic information is important for prediction. To address this limitation, we modified the RETAIN model while preserving model transparency using conditional RNNs by adding auxiliary inputs such as basic patient information. Our results, which predicted weight based on interpretation of both demographic status and app usage patterns, showed how each variable was important in predicting weight change each week.

Finally, based on the lifelog data of approximately 17,000 users, we found that a deep learning model that takes advantage of the sequential nature of lifelog data was able to perform well in the general population. Research on models for weight loss prediction has been conducted in fields such as bariatrics or pharmacology on small populations; therefore, it is difficult to generalize the findings of these previous studies [12,27] to the general population. Since we used a general population for this study, these findings may be applied to a general and heterogeneous group.

It is important for end-users such as the patient or clinician to interpret and understand the AI system because erroneous predictions are especially risky and expensive [28]. As black-box machine learning models are frequently employed due to the importance of prediction in critical contexts (such as within the medical field), the demand for transparency in AI, from end-users to research staff, is increasing [29,30]. This has led to a strong interest in explainable AI, which can provide details or reasons to make its functioning clear or easy to understand [17]. Explainable AI research to make RNNs transparent has been conducted [16,24-26]; however, it is difficult to directly adopt these algorithms for problems in medical fields without patient profiles because time-fixed variables have not been properly handled. Therefore, we modified the interpretable AI model to handle both types of data while preserving transparency, and our work can be a cornerstone of medical research in discovering new insights. In addition, contribution coefficients from the model can provide intuitive insight into why the model’s predicted values differed compared to actual data of users’ behaviors and weight change from week to week [16].

We found that weighting was equally distributed each week with a value of 0.0625 (Figure 4); however, this indicated that the initial usage of the app was slightly more important than later usage (from 0.0626 to 0.0624). Given that time-level attention weighting is relative, and the sum of these is 1, the importance of each week is almost equivalent because the weight loss program continued for 16 weeks (0.0625≒1/16). Therefore, this implies that each week in the diet program is nearly equally important, but early usage is relatively more important. In addition, timely feedback may be important if a user behaves in a way that is detrimental to weight loss for a week. To address this, our model was developed to cover 3 types of sequence data (window, week, variables) to forecast changes from any week in the mobile program. Therefore, our model can provide timely feedback by predicting weight change from week to week.

Most of the variable-level attention weightings shown were either 1 or –1 (Figure 5). Each attention weighting needs to be understood to infer why the variable-level attention is represented as –1 or 1. The time-level attention weighting was always positive as it is derived from the softmax function, and the variable-level attention weighting ranged from –1 to 1 as it is derived from the hyperbolic tangent function, which is multiplied with the parameter w in linear transformation; therefore, we inferred that variable-level attention represented the sign of weighting at each time step in the model. Instead, parameter w in linear transformation seemed to represent the magnitude of weightings, and its values were constant regardless of the time step. Therefore, to at least understand both the magnitude and sign of the weightings (eg, regardless of the time step), variable-level attention weighting and W should be multiplied together.

Weight and meal inputs core behavior components of self-monitoring showed results consistent with those of previous studies [31-33]. A previous study [34] that did not consider timeseries identified factors contributing to weight loss using multivariate and univariate regression consistent with our findings; overall weight, meal input, and exercise input had a significant correlation with weight loss [34]. Similarly, calories were also a contributing factor to weight gain [34]. In our study, the weight, meal, and exercise inputs had consistent contribution coefficients of –0.021, –0.023, and –0.016, respectively, while caloric intake had a contribution coefficient of 0.100; these were found using interpretable RNNs, though our model cannot provide the statistical significance of contribution coefficient.

Furthermore, the yo-yo weight loss trajectory had a negative contribution coefficient of –0.006 for 8 weeks, then changed to a positive value of 0.06 at 8 weeks. To determine whether the weight loss trajectory is classified as a yo-yo or decreased pattern, the eighth week may be a critical week. In other words, to avoid a yo-yo trajectory, self-monitoring using the mobile app may be important especially for the initial 8 weeks. Meanwhile, because this model uses the historical shape of weight loss, there may be a concern that the model is weak because it indirectly learns the weight trajectory shape. Thus, we also conducted an experiment where the model without weight loss trajectories was retrained, and the result showed that the model maintained the robustness of performance (Multimedia Appendix 3). From the user’s weight entry, we derived and leveraged 2 variables—weight loss trajectory and weight input. In the real world, although coaches encourage users to enter their weight, there may be users without weight records. For users without any weight records, the accuracy and interpretation of the model’s prediction are limited. From the comparison between the model with and without weight loss trajectory as the feature (Multimedia Appendix 3), the trajectories may not be a high-contributing factor for predicting weight loss because the model compensates. Despite this, adding weight loss trajectory may provide a hint for early intervention with cluster assignment. For example, it is possible to coach the user by considering the user’s weight loss trajectory compared to the weight loss plateau or rebounding shape. Therefore, the selection of feature needs to be considered with the interpretability of each feature in mind.

Unexpectedly, drinking frequency and overall caloric intake did not make a consistent contribution to the model. This may be due to low retention, causing the number of drinks and overconsumption of calories not to be recorded because the records related to meal logs were not entered consistently within the 16 weeks.

When AI is used as a decision aid (eg, for a coach), an explanation can improve its trustworthiness [35]. For example, a coach (or another health provider) may highlight user’s behaviors that cause weight gain based on the interpretation of the model. Based on the feedback of intervention theory, feedback focused on the behavior task itself introduces the importance of timing in feedback delivery [36,37]. Also, a feedback strategy based on such a theory should be personalized and goal oriented. From this point of view, users may modify their behavior to increase weight loss based on the results of a real-time prediction and contribution factors, which improves their motivation [36,37].

Our study has some inherent limitations due to its implementation of interpretable AI using lifelog data. The individuals included in this study had purchased a subscription to a weight loss program and were selected if they had weight records for 16 weeks. This may result in selection bias wherein the chosen individuals were those with high adherence or the intention to lose weight. However, given that most of the participants who used other weight loss apps were in their forties, and a high rate of users were female in previous studies, characteristics similar to those in our study, our study is able to show robust findings with respect to population group that uses these apps [20,38-42]. Furthermore, given that the average 30-day retention rate is 3.4% for health and fitness apps in the United States, and that some researchers have reported a 3% to 8% 30-day retention rate for mental health apps, this app has an inherently higher retention rate (completion of 16-week program) compared to peer health care apps [43-45].

Despite this, there is limited research on predictive models for weight change using lifelog data and identification of contributing factors to weight loss. These results can make up for the lack of evidence on the prediction of weight change in an electronically delivered intervention setting and contribute to better outcomes. Furthermore, the implementation of the model was based on a single mobile app for obesity management. Therefore, for more generalized evidence, research using lifelogs from various apps is needed.

In conclusion, we have demonstrated that an interpretable RNN can consider app usage behavior and the users’ demographic characteristics while maintaining model transparency.