Ecological momentary assessment (EMA), also known as experience sampling or in situ (situated) self-reporting, is a powerful method for capturing real-time insights into an individual’s health and well-being. Leveraging the convenience and ubiquity of mobile devices, EMA has been particularly effective in longitudinally monitoring conditions such as depression and mental well-being [1,2], mobility [3], physical activity [4], and fatigue [5].

The strength of EMA lies in its ability to minimize recall bias [6,7] and provide more fine-grained longitudinal data compared with traditional observation methods or retrospective reporting [8,9]. Participants can answer questions without traveling to a research lab, reducing a burden that is already great for individuals with Alzheimer disease and related dementias. Data collected through EMA reveal both within-day and between-day fluctuations in health and well-being [10], laying a foundation for timely, in-the-moment interventions. Furthermore, these responses provide a rich source of patient-reported outcome measures. When combined with objective, sensor-based data representing digital phenotypes, EMA enables the comparison of digital behavior markers with self-reported data.

Despite these advantages, EMA implementation faces challenges, especially in the variability, completeness, and accuracy of participant responses to prompts. Factors such as distraction, self-awareness, boredom, time of day, and interruption burden [11] can impact participant responses. Addressing these issues is essential for maintaining the integrity of research findings. Furthermore, the design of notification strategies may dramatically impact response compliance and quality [12,13].

In this paper, we conduct an examination of EMA response completeness and quality across a spectrum of settings. By leveraging data from multiple studies, we enhance the robustness and generalizability of our conclusions. This broad approach also allows us to consider variables previously unexplored in EMA research. By correlating EMA responses with sensor-derived biomarkers, we will better understand the contextual factors influencing participant engagement with EMA prompts. The findings from this study will inform refinements in EMA methodology and guide future research to improve the reliability of EMA-derived data and the effectiveness of just-in-time interventions.

In-the-moment sampling builds a foundation for designing effective just-in-time interventions [14]. McDevitt-Murphy et al [15] used such a technique to combat anxiety by prompting individuals to write about their worries and initiate physical activity. Similarly, Mair et al [16] found such interventions improved physical activity in older adults, while Perski et al [17] constructed a technology-driven in-the-moment intervention to reduce substance abuse and promote healthy behaviors. Brenner and Ben-Zeen [18] found that letting individuals see the difference between their predicted future EMA reports of mood and actual EMA responses improved the effectiveness of their intervention.

EMA benefits must be weighed with the cost of an increased interruption load. Frequent requests for self-report often lower responsiveness [13]. Many factors have been considered that impact responsiveness. These include the impact of EMA prompts while a person is performing physical activities [19,20] or activities that require concentration [21]. Jeong et al [22] and Rintala et al [23] studied the relationship between a person’s location (eg, at home vs away from home) and responsiveness, while Pizza et al [24] and Ziesemer et al [25] found that engagement in social activity was an important consideration. Several studies consider whether prompt time of day influences likelihood of response [12,26-30]. Aminikhanghahi et al [31] found that prompting during transitions between activities improved participant responsiveness. Recently, investigators observed a relationship between the complexity of an EMA interaction and responsiveness [32]. They responded by shortening the questions [33], simplifying the interface [21], and making use of voice input [34].

Researchers also observed that the quality of responses decreases over time [35], becoming bistable or skewed [36], more habitual [37], and less variable [38-40]. Such trends threaten the reliability of research results. For example, Verbeij et al [41] observed decreased convergent validity between reported and objective social media use over 3 weeks of reporting [41].

These findings allow researchers to better design EMA-based studies and control for such factors in their analysis. One difficulty, though, is that some of the findings are contradictory. While Jeong et al [22], McMillan et al [27] and Ziesemer et al [25] reported higher response rates (RRs) when a person was alone, Pizza et al [24] observed no significant impact of social setting and Ponnada et al [26] reported a higher RR when a person was with family or friends. Regarding time of day, some studies found that participants responded more often during afternoons and evenings [26,28,42], while others [43,44] observed more responses early in the day. Khanshan et al [19] found RRs to be higher while participants were active, while Boukhechba et al [29] found higher activity was correlated with missing more responses. These discrepancies may indicate the influence of other population and study parameters. We address the need to consider these factors by performing a multistudy analysis.

Furthermore, Wrzus and Neubauer [45] emphasized the need to employ sensors in the understanding of the interplay between participant behavior, EMA responsiveness, and response quality. Stach et al [46] observe that mobile notification responsiveness is influenced not only by the app delivering the notification but also by user demographics. In our analysis, we leveraged continuously collected sensor data to examine the relationship between EMA responses (as patient-reported outcome measures), digital markers extracted from sensor data (digital phenotyping), and demographics across multiple studies. This comprehensive approach enabled us to uncover patterns and develop recommendations for optimizing EMA methodologies.

In this study, we performed statistical analysis of EMA responses and sensor data from 9 clinical studies. The overall goal was to provide insights into EMA response patterns that can be used to design and improve future in-the-moment monitoring and intervention studies. We analyzed data from 454 participants to validate the following hypotheses:

First, EMA RR will vary with prompt time of day, number of questions, and sensor-observed activity.

Second, EMA response quality will decrease as a function of the number of days that have elapsed in the study.

Third, both RR and response quality will be impacted by a participant’s behavior context, as modeled by markers extracted from mobile data.

Fourth, differences in EMA RRs and response patterns will exist between studies, influenced by participant demographics (ie, participant age) and study purpose (ie, cognitive intervention vs observational).

Daily EMA responses were collected between October 2015 and December 2023 for a total of 454 participants across 9 distinct studies. Reporting of the EMA steps is consistent with the STROBE CREMAS (Strengthening the Reporting of Observational Studies in Epidemiology Checklist for Reporting Ecological Momentary Assessment Studies) guidelines [47]. In each study, participants received prompts to answer EMA questions using an in-house app running on a smartwatch or tablet. Participants were trained on how to interact with the app by a research assistant and completed practice prompts and tasks before data collection commenced.

The EMA prompts were randomly distributed within predefined time blocks throughout the day. We adjusted these time blocks when necessary to fit participant schedules. When a prompt was issued, an audio tone was played to alert the participant. The participant would then read the question and select their response from options on a mobile screen. If there was no response within 5 minutes, the prompt was reissued, up to a maximum of 5 attempts (2 attempts for dyad study). If the participant still did not respond, we recorded this as a nonresponse and did not issue further prompts until the next designated time block. Parameters for the studies are summarized in Table 1. In total, these comprise 145,853 EMA prompts with 115,283 sets of responses.

Well-being has been defined multiple ways in the literature [48-51]. Many studies rely on 1-time questionnaires, like the Satisfaction With Life Scale [52], to measure well-being at a single time point. However, well-being is dynamic, fluctuating across different times and contexts [53]. EMA allows us to capture individuals’ daily behaviors within their natural environments alongside real-time self-reports on well-being. In this analysis, we focus on three dimensions of well-being: mental sharpness, physical fatigue, and stress.

Each study included subjective questions that examined (1) Qsharp = the person’s mental sharpness, (2) Qfatigue = the person’s physical fatigue, and (3) Qstress = the person’s level of stress, all self-reported and relevant to the point in time (eg, immediately and up to the previous 2 h) when they received the query. The Likert scales varied between studies, so we normalized the values to a 1‐5 range. Where needed, we inverted responses for negative questions so that a higher-valued response is more positive in each case (eg, Qsharp=5 indicates feeling greater amounts of mental sharpness; Qfatigue=5 indicates feeling less physically fatigued; Qstress=5 indicates feeling less stressed. Multimedia Appendix 1 plots the relative frequency of each response to the 3 question categories. The studies are ordered primarily by the cognitive health status of participants and secondarily by age. As the plots demonstrate, most of the responses are high but do vary among studies. Notably, the similarity in response between stress and fatigue is higher than other question pairs. This finding is consistent with the correlation that exists between responses to the 3 questions, shown in Figure 1.

These studies were reviewed and approved by the institutional review boards (IRB protocol 14460) at Washington State University and the University of California at Davis (No. 19784). To participate in any of the studies, participants needed to understand English and sign an informed consent; each person received compensation between US $20 and US $125 for their participation. All data were anonymized before performing analyses.

RR across all participants was 79.95%. This is within 0.76% of the 79.19% average rate Wrzus and Neubauer [45] reported in a survey of 417 EMA studies. Among the prompts that received participant responses, the response completion rate was 88.37%.

Figure 2 plots RR for the studies, ordered from largest to smallest. Response variance is quite different among the studies. Notably, response variance increased as the mean RR decreased. Table 2 shows the overall most popular time of day for responding to EMA prompts. However, these results vary based on participant age. Only considering studies with older adults (age 50 y or older), participants were more responsive in the evening than morning (by 3.95) or afternoon (by 3.58). In contrast, younger adult RR remained fairly uniform throughout the day, with afternoon showing the highest rate. Weekday and weekend prompts yielded very similar RR overall (a difference of 0.04). However, this varied by study and age group. Specifically, while weekend response was higher by 3.25 for younger adults, the rate decreased for older adults by 0.27. These findings highlight the role that participant demographics play in understanding EMA responsiveness; prompt time alone may not be a strong indicator of responsiveness.

Figure 3 (left) plots EMA RR as a function of the number of days that elapsed since the beginning of the study for each participant. The linear fit line has a positive slope of 0.078, indicating that the duration does not negatively impact RR. At the same time, the fraction of participants still collecting data is much smaller at the end of this time span. A nonsignificant small positive correlation exists between the study duration and RR (r=0.100, P=.29). This may be due to the EMA prompts becoming more integrated into the participant’s routine and their gaining familiarity with the questions and methods of responding.

Figure 3 (right) shows RR as a function of the number of questions asked at the EMA prompt, together with the fitted line. Correlation between the number of questions asked during an EMA prompt and RR is moderate and negative (r=−0.433, P<.001). The line fitting this trend has a slope of −1.90. This finding is consistent with previous work indicating that individuals are less likely to respond when each EMA prompt requires a large time interruption and corresponding task burden [37].

Table 3 summarizes participant responses and response quality. The average response value across all studies is relatively high, as is the percentage of responses near the mode. There was a noticeable shift toward response uniformity over time, evidenced by a significant increase in carelessness and decrease in variance. Results from individual studies are shown in Multimedia Appendix 2. Although the earlier analysis showed that RR did not decline over the study, the increase in careless responses and decrease in variance indicates a possible increase in habitual responses to EMA questions [37] as participants got bored or felt time pressure to respond to EMA notifications. Some studies inserted a break of 1+ months during data collection. For these studies, we calculated changes within the period before the break and the period after the break. The trends remain the same: carelessness increased and variance decreased over time. These changes are smaller, possibly due to the decreased time (typically 7 d) being examined.

Relationships between RRs and sensor-detected states were also examined. A correlation of r=0.045 (P<.001) exists between RR and activity level over the previous 30 minutes, r=0.174 (P<.001) between RR and time spent at home over the previous 30 minutes, and r=0.124 (P<.001) between RR and nearness to detected activity change points. Multimedia Appendix 3 summarizes the results for individual studies.

RR correlation with activity level is positive but small. This finding, considered together with the discrepancies observed in previous studies [19,29], may indicate that the nature of the person’s activity needs to be examined to understand the impact on prompt response. Correlations with at-home time and change points were larger, although still small. Individuals may not want to interrupt their activities, particularly in social situations, to answer lengthy EMA questions. In these contexts, they may ignore the prompt and wait until the next time block. As with our previous study examining RR and change points in smart homes [31], we observe that prompting near a change point improve responsiveness for all studies.

Results from the previous analyses reveal that differences in EMA response compliance and quality may occur as a function of study and participant characteristics. Here, we delve further into EMA response differences between studies. First, we measure the distance (inverse similarity) between EMA time series using DTW. Across all studies and participants, the average within-study DTW distance was 14.141 (SD 11.957) overall (sharp 15.357, highest dyad=36.111, lowest bb=11.486; fatigue 15.014, highest dyad=36.489, lowest bb=10.299; stress 12.051; and highest dyad=33.511, lowest bb=7.069).

In contrast, the average between-study DTW distance was 33.246 (SD 4.971) overall (sharp 44.848, highest dyad/bb=81.448, lowest mc/bb=16.262; fatigue 29.365, highest mn/func=54.927, lowest mc/bb=12.434; stress 25.526, highest mn/func=52.05, lowest mc/bb=8.759). This represents a 135% distance increase over the within-study distances. The large increase indicates that while response variability exists between participants, an even larger difference exists between studies. Table 1 lists some of the parameters that may influence these differences; further investigation may identify more contributing factors.

Plots of the aggregated EMA responses for the 3 question types are shown in Figure 4. The shapes of the aggregated time series are different between the 3 question categories. The Qsharp trend is consistently positive, while there are less noticeable trends for the other questions.

Because 2 of the studies (bb and mn) represent interventions, we examine differences in the response trends between these studies and the 2 observational studies with cognitively healthy older adults (dyad and cs). Aggregated over all 9 studies, the slope of the response fitted line is 0.018, with a y-intercept of 3.181. The fitted line for the intervention studies has a slope of 0.023 (an increase of 0.005) and y-intercept of 2.745 (a decrease of 0.436), while for observational studies with cognitively healthy older adults, the fitted line has a slope of 0.005 (a decrease of 0.013) and y-intercept of 3.449 (an increase of 0.614). Here, we observe that older adults receiving interventions start at a lower self-reported mentally sharp value but indicate that the sharpness increases over the study duration. In comparison, the cognitively healthy adults start at a higher sharp reported value and have a lesser amount of change in reported values during the study.

Table 4 presents the ARIMA parameters for the response categories sharp, fatigue, and stress. The log-likelihood values are high for all models, indicating a good fit. Similarly, the negative Akaike Information Criterion and Bayesian Information Criterion values suggest that the models strike a balance between fitting the data well and avoiding unnecessary complexity. The P values for the Moving Average of order 1 terms (P<.001 for all categories) indicate that errors at lag-1 are significant for predicting the next value in the time series. In addition, the Ljung-Box test results, with values greater than 0.05, suggest that the residuals are not significantly autocorrelated, meaning the models capture relevant patterns beyond mere autocorrelation.

Early detection and intervention, crucial for reducing health care costs and enhancing quality of life, depend on timely reports of health status. This cross-study analysis highlights the pivotal role of EMA design in obtaining frequent, accurate EMA responses. In particular, we observed that participant demographics and study parameters markedly influence EMA response values, compliance, and quality. In particular, timing of EMA prompts varied in effectiveness depending on participant age. On the other hand, the negative impact of lengthy EMA interactions was consistent across all of the studies. Supporting eyes-free or hands-free interactions [21,34] may reduce interruption burden. Similarly, employing more frequent but shorter EMA sessions, or μEMA [26], may improve responsiveness. These findings support the first hypothesis and highlight the need to tailor EMA prompt strategies to specific participant and study characteristics.

A consistent observation was an increase in habitual, careless responses over the study duration, supporting the second hypothesis. In this study, 2087 of the 17,939 sessions (11.63%) were started but not completed, so researchers may consider strategies such as randomizing question or task order [58], gamifying the data collection process [8], or allowing selected questions to be skippable [59], as some may hold more interest. Additional factors, such as time pressure from concurrent activities or individual personality traits [55], may also influence responses. Ultimately, these considerations can be included in the study design.

Integrating sensor-based behavioral context is key to improving EMA studies. As hypothesized in the third hypothesis, our analysis revealed a significant relationship between sensor-derived biomarkers and EMA responsiveness. Analyzing these biomarkers could benefit personalized EMA design. In addition, timing EMA prompts close to detecting activity transitions may improve RR and quality.

Finally, our results showed that consistent with the fourth hypothesis, EMA response dynamics varied according to study parameters such as participant age and study type. We observed that response values increased more dramatically for intervention than observational studies, perhaps due to participants viewing their intervention involvement as beneficial, as noted in previous literature [60]. The response series were well fit by statistical models such as ARIMA. Such models provide a basis to compare EMA trajectories between groups and predict future response patterns.

This study had several limitations. Although we examined factors like study purpose and participant age, other variables, such as education, race or ethnicity, employment status, environment, and cognitive or physical health, were not examined. In addition, we did not consider the impact of breaks of 1 or more days in data collection. Restarting data collection after a break could affect EMA compliance and response quality.

Another limitation involves the duration of the studies. While some studies spanned 6 months or more, others lasted only 2 weeks. Extending the duration of all studies would enhance the generalizability of our time series findings. The sample sizes also varied considerably, with smaller young adult samples compared with older adult studies. In addition, slight differences in EMA question wording between studies, though normalized, may have influenced responses. Greater uniformity in future studies would provide more consistent data.

A further limitation was the lack of diversity in medical conditions. While we focused on individuals with varying levels of cognitive decline, other physical or psychiatric conditions were not represented. Including a wider range of medical conditions in follow-up studies would provide a more comprehensive understanding of EMA responses across different health conditions.

While the studies were consistent in their data collection hardware and software, technology influences results through factors like accessibility, battery life, ease of use, device familiarity, and extractable digital markers. Future studies can systematically consider alternative technologies.

Finally, this analysis concentrated on a subset of well-being factors. Since well-being encompasses broader emotional, social, and physical domains, future research can include additional dimensions for a more holistic analysis. In particular, a next step can be to evaluate the impact of rural versus urban residence as well as the quality of the build environment on EMA responses.

This study offers insights into the factors that influence EMA responses and lays the groundwork for further exploration. Future studies should broaden the range of participant, study, and interaction characteristics to better inform EMA design strategies. In addition, expanding the range of sensor-based behavioral markers will enhance understanding of how changes in routine and behavior can enrich EMA data collection. Addressing these areas will help refine EMA methodologies, leading to more personalized and effective health monitoring and intervention technologies.