Participants were recruited from studies of aging and dementia at the Knight Alzheimer Disease Research Center at Washington University School of Medicine in St. Louis. The Ambulatory Research in Cognition (ARC) study began in March of 2020. ARC is a custom-built smartphone app that assesses cognition with repeated brief assessments up to 4 times per day for one week [13]. Participants were invited to enroll after completing their annual cognitive and clinical assessment and began ARC testing shortly after enrolling. As data collection is still ongoing, we used a data freeze from June 2024 for the current analyses.

For the current analyses, participants were included if they (1) completed at least 10 sessions during their baseline ARC testing to ensure adequate engagement in the study and enough observations in different environments for comparisons and (2) had a clinical assessment (see below) within a year of starting their ARC testing to ensure an accurate classification of cognitive status.

Clinical status was assessed using the Clinical Dementia Rating (CDR) [27]. The CDR rates cognitive and functional performance on a 5-point scale across 6 domains (memory, orientation, judgment and problem solving, community affairs, home and hobbies, and personal care). Participants and a collateral source (a close family member or friend) both underwent semistructured interviews to determine CDR scores, with CDRs of 0 indicating cognitive normality, 0.5 indicating very mild dementia, 1 indicating mild dementia, 2 indicating moderate dementia, and 3 indicating severe dementia. For this study, we grouped participants as either cognitively normal (CDR 0) or diagnosed with very mild dementia (CDR 0.5). The parent study was designed to characterize the transition from cognitive normality to the very first signs of clinical symptoms of AD; therefore, most participants were cognitively normal (CDR 0), with the final sample consisting of 385 cognitively normal participants and 37 participants with a CDR of 0.5. Of the 37 participants with CDR 0.5, a total of 26 (70%) were clinically diagnosed with AD, and the rest with a dementia of different etiology, including Parkinson disease, Lewy body dementia, or uncertain etiology. CDR assessments were completed 7.64 (SD 7.78) weeks from their ARC assessment.

Methods for the procedures of ARC are detailed elsewhere, so are only briefly described here (for full details on the ARC platform, see studies by Nicosia et al [13] and Wilks et al [18]). Participants completed all ARC assessments on a smartphone (either their own or a study-provided device). ARC assessments were sent to participants using the native iOS (Apple Inc) or Android (Google LLC) notification systems pseudorandomly. Participants were instructed to complete the assessment as soon as possible within a 2-hour window. Participants completed 3 cognitive tasks (described below) up to 4 times per day over the course of a week. Figure 1 provides screenshots of each task presented on ARC.

During this processing speed task, participants were randomly shown 3 pairs of abstract shapes and were asked to select which of 2 possible responses matched 1 of the 3 targets [17]. At each assessment, participants completed 12 trials lasting approximately 20‐60 seconds. Consistent with our previous work [13,18,28], we analyzed symbols’ performance in 2 ways. The first was the median RT of correct trials, which we use to capture differences in processing speed. We also calculated a measure of RT variability, the coefficient of variation (CoV). For both median RT and RT CoV, higher scores reflect poorer performance (ie, slower responding or greater variability).

This associate memory task presented subjects with 2 phases: a learning phase and a recognition phase. During learning, participants were instructed to learn and remember 10 item-price pairs that were presented on screen for 3 seconds per pair. The items were everyday shopping items (eg, foods or household supplies), and prices were randomly assigned 3-digit prices. During the recognition phase, participants were presented with an item they encountered during learning and 2 prices and were asked to choose which price was originally paired with the item. Each administration of this test requires approximately 60 seconds. The primary dependent measure for this task was the error rate during recognition, such that higher scores indicated poorer recognition.

In this spatial working memory task, participants were presented with images of three everyday objects (key, smartphone, and pen) on a 5x5 grid. Participants were told to remember the location of each item on the grid. After the encoding phase, participants were given a distractor task (locating Fs among a grid of Es). Following the distractor task, participants were asked to recall the location of each item on a blank grid. Participants completed 2 trials at each session (total time 30‐40 s). The primary dependent measure for this task was the Euclidean distance for each item. Higher scores indicated that participants placed the items further away from the initially encoded location and suggested worse performance [17].

At each ARC session, participants answered several questions about their experiences or situations at the time of the notification. The first question, which we used to operationalize testing location, asked, “Where are you right now?” Participants selected one of the following response options: my home, my work, school, another person’s home, vehicle, outside, or other. We categorized participants’ responses into either home (if they answered “my home”) or away (all other responses).

The second question, which we used to operationalize social context, asked, “Who is with you right now?” Participants could select all that apply from the following options: nobody, partner or spouse, family, friends, coworkers, pet, and other. As above, we categorized participants’ responses into either testing alone (if they answered “nobody”) or not alone (all other responses). These environment and social context categorizations were consistent with a previous study [26]. As displayed in Table 1, most participants completed their tests at home and alone, with few reports in other contexts. As there was little variation within each category, we chose not to treat these as separate variables in analyses. Additionally, there is some debate in the field of EMA on whether “pets” should be counted as alone or not alone. We considered responses of “pets” as not alone, as it is likely that pets might also be a source of distraction for participants during remote cognitive assessments (eg, a dog barking at a passing car might be a salient distraction). The results were nearly identical when including “pet” responses as alone.

Upon completion of each session, participants were asked if they experienced any interruptions during the testing period. This item was used in additional analyses (described below) to account for sessions where participants were acutely aware of potential distractions.

Our primary analyses addressed how the environmental factors affect the relationship between CDR status and performance in remote unsupervised testing. In each linear mixed effects model, the outcome measure was either symbols median RT, symbols CoV, prices error rate, or grids Euclidean distance. We included fixed effects of age at ARC testing, years of education, self-reported gender (male or female), and number of completed sessions (to account for practice effects). We also included a fixed effect term for the proportion of sessions completed at home and the proportion of sessions completed alone to account for between-person differences in the testing location or context. That is, some people might complete more ARC assessments at home, and some might complete more assessments alone than others. To account for within-person variance, we included a term for the reported location (ie, home or not home; coded 0 and 1, respectively) and social context (ie, alone or not alone; coded 0 and 1, respectively) at each ARC session. We included all 2-way interactions between CDR status, within-person location, and within-person social context, and the higher-order 3-way interaction as fixed effects. These terms allowed us to test the differential impact that environmental distractions had on both cognitively normal older adults and those with very mild dementia. All models included a random intercept for the participant.

In our second set of analyses, we removed sessions where, upon completion of each assessment session, participants reported being interrupted. We tested the hypothesis that if participants were aware of their distractions and reported them, then removing these sessions should greatly reduce or eliminate any effects of location or social context on cognitive performance. Conversely, if location and social context effects are still apparent, then we would demonstrate that environmental distractions impact remote cognitive performance beyond instances in which participants are overtly interrupted.

All mixed-effect models were conducted using R software (version 4.4.0; R Foundation for Statistical Computing) including the lmer package [29] and P values for mixed-effect models were estimated using the lmerTest [30] package. Follow-up comparisons were conducted using emmeans [31].

All participants provided informed consent, and all procedures were approved by the Human Research Protections Office at Washington University in St. Louis (IRB# 201907164), and the research was conducted in accordance with the Helsinki Declaration. Participants’ data were fully anonymized during the data collection process, and researchers cannot access any identifiable participant information without requesting access to the data via a formal data request to the Knight Alzheimer Disease Research Center Administrative Core. As described in Nicosia et al [13], participants were paid US $0.50 per completed assessment session. Participants also received bonus payments for completing all 4 sessions on any given day (US $1.00 per occurrence, max of US $7.00), completing at least 2 assessments per day for 7 days (US $6.00), and completing at least 21 assessments over 7 days (US $5.00). The maximum compensation possible for one 7-day assessment visit was US $32.00.

The final sample consisted of 417 participants (see Table 1). Table 2 provides demographic and ARC task information on the full sample, grouped by clinical status. Participants were highly educated, most self-identified their race as White, and were a majority of females. Adherence was high with participants completing, on average, 23.11 (SD 4.41) out of a possible 28 sessions (82.54% adherence).

As expected, participants with very mild dementia performed worse on the ARC assessments when aggregated across the week of testing. Further, there was a small difference in the proportion of tests completed at home, such that individuals with very mild dementia completed a greater proportion of tests at home compared to cognitively normal participants (n=704, 86% vs n=7131, 81%, respectively). There was no difference between cognitively normal participants and those with very mild dementia in the proportion of tests completed alone. Table 1 provides more detailed information regarding the response frequencies to both the location and social surrounding questions for both cognitively normal participants and those with very mild dementia.

Consistent with our other publications in this sample [13,18,28,32], older age at the ARC assessment predicted worse performance on all ARC measures (Table S1 in Multimedia Appendix 1). Male participants made more errors than females on the prices test but were slightly better at the grids test. More years of education were also associated with better performance on prices and grids (but not symbols). Participants who completed more sessions performed better on grids and had faster processing speed and lower variability over time on symbols indicating evidence of practice effects on these tasks.

As seen in Table 3, neither the proportion of tests completed at home (ß’s=–0.131 to 0.072, P’s>.31) nor the proportion of tests completed alone (ß’s=–0.309 to 0.020, P’s>.06) was related to performance on any of the ARC tasks. When considering the session-level effects of testing location, there were no significant effects of testing location on ARC performance. Participants showed similar performance on all ARC measures regardless of testing location: prices (M_home 0.327, SE 0.007 vs M_{not home} 0.354, SE 0.011, t₉₃₉₉=–2.773, P=.006, Cohen d=–0.06, 95% CI –0.10 to –0.02), grids (M_home 0.816, SE 0.02 vs M_{not home} 0.813, SE 0.03, t₉₂₁₇=–0.118, P=.91, d < –0.01, 95% CI –0.04 to 0.04), symbols processing speed (M_home 3.52, SE 0.09 vs M_{not home} 3.43, SE 0.10, t₉₃₉₉=1.925, P=.054, d=0.04, 95% CI 0.00 to 0.08), symbols variability in processing speed (M_home 0.361, SE 0.007 vs M_{not home} 0.372, SE 0.01, t₉₃₆₆=–1.362, P=.17, d=–0.03, 95% CI –0.07 to 0.01). The main effect on prices performance is significant when calculating the simple effect; however, caution should be used when interpreting this effect, as it is likely driven by the interaction between testing location and social context.

This was also largely the case for session-level effects of social context. Participants showed similar performance, regardless of social context, on prices (M_alone 0.336, SE 0.01 vs M_{not alone} 0.345, SE 0.009, t₉₅₀₄=–0.871, P=.38, d=–0.02, 95% CI –0.06 to 0.02]), grids (M_alone 0.755, SE 0.03 vs M_{not alone} 0.874, SE 0.03, t₉₃₃₃=–4.398, P<.001, d=–0.09, 95% CI –0.13 to –0.05), and symbols processing speed (M_alone 3.43, SE 0.10 vs M_{not alone} 3.52, SE 0.09, t₉₂₂₃=–1.878, P=.06, d=–0.04, 95% CI –0.08 to 0.00). Again, this seemingly large effect is likely driven by higher-order interactions, and so caution should be used when interpreting this effect.

However, there was a significant effect on the variability measure from the symbols task, such that when participants completed sessions in the presence of others (mean 0.377, SE 0.008), they demonstrated increased variability in RTs on the symbols task compared to when they completed sessions alone (mean 0.356, SE 0.009; t₉₄₆₉=–2.563, P=.04, d=–0.05, 95% CI –0.09 to –0.01).

Performance on 3 ARC measures was significantly impacted by both CDR status and testing location or social context (ie, significant 2-way interactions; Table 3). For grid performance, cognitively normal older adults performed better when they completed tests at home compared to when they completed tests away from home (M_home 0.714, SE 0.01 vs M_{not home} 0.755, SE 0.02, t₉₀₃₆=–3.317, P=.001). However, older adults with very mild dementia did not differ in their grids performance in different testing environments (M_home 0.918, SE 0.04 vs M_{not home} 0.871, SE 0.06, t₉₀₃₆=–3.317, P=.001). For symbols processing speed, cognitively normal older adults did not differ in either testing location (M_home 3.22, SE 0.05 vs M_{not home} 3.23, SE 0.06, t₉₁₅₂=–0.157, P=.88). However, older adults with very mild dementia showed a significant, albeit small, difference in processing speed based on their testing location (M_home 3.81, SE 0.17) vs M_{not home} 3.63, SE 0.19, t₉₂₅₄=0.925, P=.36).

As for social context, there was a significant interaction with CDR status on RT variability on the symbols task. When cognitively normal participants completed the test alone, they were significantly less variable compared to when they completed the tests in the presence of others (M_alone 0.337, SE 0.004 vs M_{not alone} 0.346, SE 0.05, t₉₂₆₁=–4.650, P=.02). This effect was larger for participants with very mild dementia who completed tests (M_alone 0.375, SE 0.02 vs M_{not alone} 0.409, SE 0.01, t₉₅₁₀=–2.085, P=.04).

Finally, across all participants, there was evidence that the combination of taking tests away from home and in the presence of others significantly impacted performance on prices and grids. Specifically, when tests were taken in the most suboptimal locations (ie, away from home and in the presence of others), participants, on average, made significantly more errors on prices (mean 0.359, SE 0.01) compared to when they completed the test in “optimal” contexts (ie, at home and alone; mean 0.323, SE 0.008; t₉₅₃₅=–3.218, P=.001). This was also evident for grids as participants were, on average, worse when completing both away from home and in the presence of others (mean 0.917, SE 0.03) compared to when they completed tests at home and alone (mean 0.801, SE 0.02; t₉₁₇₀=2.099, P=.04).

These lower-order interactions were qualified by 3-way interactions on grids (Figure 2) and symbols median RT (Figure 3). Decomposing the interaction suggested that when those with very mild dementia completed tests both away from home and in the presence of others, they had an average grids score of 1.049 (SE 0.06), which was significantly higher than when they completed tests at home and alone (mean 0.894, SE 0.05; t₉₃₉₉=–2.754, P=.006, d=–0.05, 95% CI –0.09 to –0.01]). For processing speed, however, the difference between the optimal (mean 3.85 s, SE 0.18) and suboptimal (mean 3.83 s, SE 0.19) environments was not statistically significant (P=.88).

Interruptions were reported on 1194 of 9633 completed sessions (approximately 12% of all sessions). Removing these sessions resulted in an overall similar pattern of effects (Table 4), albeit an attenuated effect. The previously described 3-way interactions between CDR status, testing location, and social context were still evident on the grids and symbols median RT. Thus, even after removing sessions during which participants reported experiencing an overt interruption, the effects of environmental distractions were still apparent, especially for those with very mild dementia.

The overall results of this study illustrate the importance of considering environmental factors in remote studies of cognition where participants may complete assessments in different locations and social contexts. The effect sizes of distractions on cognition were generally small in magnitude and likely would not invalidate remote testing, but distractions may be most important to consider in populations where cognitive impairment is evident.

Across participants, in-the-moment effects of testing location and social context were evident on measures of associative memory and visual working memory but not processing speed. When participants completed tests in the most distracting environments (both away from home and around others), they performed worse. However, for cognitively normal older adults, the impacts of these environmental factors were minimal, if at all apparent. On the other hand, participants with very mild dementia were the most sensitive to distractions, specifically when completing tests away from their homes and in the presence of others. These findings indicate that an inability to maintain performance in the face of distractions may be a novel indicator of very early neurodegenerative disease and emphasize the need for remote studies to carefully consider testing environments. While a strength of EMA is that it can allow for the measurement of cognition in a variety of environments and situations, the potential for distractions should at least be evaluated and, if necessary, included in statistical models. This is particularly true for studies enrolling individuals with very mild dementia symptoms.

The descriptive results from this study revealed that older adult participants engaged in remote cognitive assessments outside of their homes during 81% of assessments and in the company of others during 44% of assessments, both of which are environments that are likely to produce distractions. Our estimates of tests completed in these environments closely match that of Cerino et al [26], who found that older adults in their study reported testing at home in 78% of assessments and testing alone for 44%. Interestingly, Cerino et al [26] reported that participants with MCI completed slightly more tests at home compared to cognitively normal participants (P=.08). Results from our sample replicated these patterns as we found a significant difference between the frequency of tests completed at home between cognitively normal participants and those with a CDR of 0.5, indicating the presence of a very mild dementia. A CDR of 0.5 can be considered very similar to a classification of MCI [33,34]. This suggests that older adults with cognitive difficulties may spend more time at home or may avoid taking tests outside of the home, perhaps due to the impacts of environmental distractions.

Participants in our study have the option to report if they experienced a significant interruption during testing. Interrupted sessions were not uncommon for our participants, as nearly 12% of sessions were reported as having some form of interruption. Interestingly, the effects of environment, while reduced, remained significant after removing sessions where participants specifically flagged interruptions. Thus, aspects of environmental context that may not rise to the level of an overt interruption may have deleterious effects on cognition, and these effects appear to be magnified in those with dementia symptoms.

The results of this study have several implications for EMA studies of cognition in older adults at risk for AD. First, the descriptive results showed that a large proportion of assessments are completed at home. Thus, it is possible that we are not fully sampling participants’ everyday contexts as well as we can or that participants are more likely to only take tests when they are at home. Due to the age of our cohort, most of our participants are retired and therefore may spend most of their time at home. Although overall adherence was high in this study, it is also possible that our participants chose not to complete tests when they were away from home, which may have led to an underestimation of the effects we observed. In other words, it might appear that older adults enrolled in EMA studies prefer home environments over completing tests “in the wild.” While EMA has a clear strength of being able to capture cognition (and other processes) in different environments, future studies should be aware that EMA studies of cognition in older adults might have limited sampling outside the home or in different social contexts because participants tend to complete assessments in minimally distracting and more controlled environments.

A second important implication of this work stems from our findings regarding the impact of environmental factors on cognition. Our participants frequently completed assessments in environments that could be considered distracting. Additionally, participants reported overt interruptions much more frequently than anticipated, during 12% of testing occasions. However, even after accounting for testing location, social context, and overt interruptions, the effects of environmental distractions had very little impact on performance for cognitively normal older adults. However, for those experiencing the earliest symptoms of dementia—in what could be considered the MCI stage, the effects of distractions were stronger, although still at a small effect size. EMA studies should routinely sample for location and social context and allow participants to report interruptions so that these can be included in statistical models. This is particularly important for studies that enroll participants with cognitive impairment.

These findings also have implications for understanding how distractions and interruptions impact functioning during everyday cognitively demanding tasks. Everyday tasks such as driving, managing money and medications, preparing meals, and general housekeeping require several different processes and are important for maintaining functional independence [35,36]. Being interrupted or distracted during these activities might lead to severe outcomes such as accidents or taking the wrong medicine. Our results suggest that techniques or interventions that teach individuals how to minimize distractions, especially during cognitively demanding tasks, might be useful to minimize adverse outcomes.

While the results of this study are intriguing and provide insight into the impact of environmental distractions on unsupervised cognitive assessments in older adults, it is worth noting some limitations. The sample used in this study consisted of older adults who are largely retired and therefore, may spend more time at home (which was reflected in the high proportion of tests completed at home, roughly 80%). This also likely contributed to the large standard errors for our estimates, especially among older adults with very mild dementia. Examining these effects in a cohort of younger participants, who are likely still employed and dealing with more day-to-day activities that require them to be out of the house, may more effectively examine the impacts of environmental distractions.

This study also relied on self-reports of testing location and social context, which may not capture objective levels of environmental distractions. Future research could consider incorporating continuous passive monitoring of environmental factors to better contextualize the effects. For example, GPS tracking could be used to detect differences in cognitive performance as a function of participants’ distance from home. Likewise, passive monitoring of noise levels from participants’ testing devices could be used to objectively quantify distractions in their environment. Additionally, our question regarding “interruptions” was not open-ended, and so we could not capture differences in the types of interruptions participants experienced. For example, notifications from other applications that popped up during the testing may differ in their distraction and impact on performance compared to interruptions from a family member calling for the participant. Future research should consider gathering more detailed information about the types of distractions or interruptions experienced by participants. Finally, this study focused only on the baseline assessment from ARC and how participants’ self-reported environmental factors were related to cognitive performance. Future work could investigate if there are longitudinal changes in participants’ environmental contexts (frequency of testing at home or alone) and how that is related to cognitive changes over time.

Unsupervised EMA cognitive assessments are a unique data collection method that more closely represents how participants use cognition in their daily lives, but tests are often completed in distracting environments. Our results suggest that effects of environmental distractions, such as overt interruptions during testing and the location and social context in which tests are completed can have small impacts on cognition, and these effects were most apparent for older adults showing early symptoms of dementia. The effect sizes attributable to distractions in cognitively normal participants were very small and are not likely to threaten the validity or reliability of cognitive testing acquired in natural environments on mobile devices; thus, our results continue to support the validity of unsupervised ambulatory cognitive assessments.