This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Acquired Brain Injuries (ABIs) can result in multiple detrimental cognitive effects, such as reduced memory capability, concentration, and planning. These effects can lead to cognitive fatigue, which can exacerbate the symptoms of ABIs and hinder management and recovery. Assessing cognitive fatigue is difficult due to the largely subjective nature of the condition and existing assessment approaches. Traditional methods of assessment use self-assessment questionnaires delivered in a medical setting, but recent work has attempted to employ more objective cognitive tests as a way of evaluating cognitive fatigue. However, these tests are still predominantly delivered within a medical environment, limiting their utility and efficacy.
The aim of this research was to investigate how cognitive fatigue can be accurately assessed in situ, during the quotidian activities of life. It was hypothesized that this assessment could be achieved through the use of mobile assistive technology to assess working memory, sustained attention, information processing speed, reaction time, and cognitive throughput.
The study used a bespoke smartphone app to track daily cognitive performance, in order to assess potential levels of cognitive fatigue. Twenty-one participants with no prior reported brain injuries took place in a two-week study, resulting in 81 individual testing instances being collected. The smartphone app delivered three cognitive tests on a daily basis: (1) Spatial Span to measure visuospatial working memory; (2) Psychomotor Vigilance Task (PVT) to measure sustained attention, information processing speed, and reaction time; and (3) a Mental Arithmetic Test to measure cognitive throughput. A smartphone-optimized version of the Mental Fatigue Scale (MFS) self-assessment questionnaire was used as a baseline to assess the validity of the three cognitive tests, as the questionnaire has already been validated in multiple peer-reviewed studies.
The most highly correlated results were from the PVT, which showed a positive correlation with those from the prevalidated MFS, measuring 0.342 (
The results show that the PVT mobile cognitive test developed for this study could be used as a valid and reliable method for measuring cognitive fatigue in situ. This test would remove the subjectivity associated with established self-assessment approaches and the need for assessments to be performed in a medical setting. Based on our findings, future work could explore delivering a small set of tests with increased duration to further improve measurement reliability. Moreover, as the smartphone assessment tool can be used as part of everyday life, additional sources of data relating to physiological, psychological, and environmental context could be included within the analysis to improve the nature and precision of the assessment process.
Cognitive fatigue can be caused by multiple different conditions, including Acquired Brain Injuries (ABIs) [
Evaluation of cognitive fatigue has traditionally been performed using self-assessment in the form of easy-to-comprehend questionnaires that are often delivered within a clinical setting [
More recently, research into computerized cognitive fatigue assessment has sought to adopt cognitive testing methods by either adapting paper-based methods or repurposing cognitive testing batteries [
Originally designed for a static desktop computer-based evaluation, PVT has since been modified for use on a mobile-based platform (to improve the utility of on-the-go assessment [
The study presented in this paper aimed to address the research question, “
As highlighted, the assessment of cognitive fatigue commonly takes place within a clinical environment, which does not allow for accurate in situ assessment. This study primarily explored the adaptation of cognitive fatigue tests for delivery using a smartphone. During the study, the validity of the smartphone-based tests was compared to self-reported cognitive fatigue levels, as measured using the MFS. Accordingly, the MFS version employed had to be adapted to allow for delivery on a smartphone. Three cognitive tests were chosen from those identified in the literature. The Spatial Span Test was selected for inclusion to assess working memory. The PVT was also selected, as it has been proven as a cognitive fatigue evaluation method [
A multi-disciplinary, iterative approach (as illustrated in
Smartphone app iterative design process.
Requirements for each Development Stage.
Development Stage | Targeted Outcome | Specific Processes | Modifications |
Expert Review | Overall workflow of app finalized. Requirements list and features identified. | Research into appropriate cognitive testing methods was carried out alongside mobile workflow and design research. Expert opinion was used for each requirement inclusion. | Sequencing of cognitive tests were discussed and finalized based upon perceived enjoyment and difficulty. |
Prototype | Prototype app finalized, based upon requirements set up from Expert Review. | Storyboarding of smartphone app screens to create a structured guide for design. Color scheme and visual design was finalized. Front and backend development finalized to create a stable, secure, and usable app. | Through storyboarding and visual planning, complexity of the cognitive tests was reduced to improve their usability. The Mental Fatigue Scale was adapted for easier use on a smartphone. |
Pilot Study | Deployment of the prototype app to a small group of participants in a pilot study. Gather usability feedback from participants. | Carried out a two-week pilot study using the developed app. Participants did not have any prior acquired brain injuries (to validate the app on a healthy population first, as using a vulnerable population is unethical). | Feedback from the pilot study informed the team of usability issues and bugs, and participants suggested improvements to make the app more enjoyable. |
Deploy | Deployment of the finalized app to a larger number of participants. Data collection to allow for analysis of results. | Targeted study was used to validate the cognitive fatigue measures that were used within the app. | Targeted study deployment allowed for data collection through participant use of the finalized app. This data was then analyzed to establish the accuracy and validity of proposed cognitive testing methods. |
The initial phase of the design process primarily focused on using the research team’s multidisciplinary expertise to inform the overall nature and workflow of the app. The order in which the cognitive tests would be delivered was considered an important design decision, as retaining participant engagement (particularly when cognitively fatigued) is crucial. Through Expert Review, the order of the selected tests was based on a combination of the perceived level of difficulty and expected level of participant enjoyment. These considerations subsequently lead to potentially less-engaging tests being sequenced earlier in the workflow of the app, whereas more engaging tests were sequenced later. Accordingly, the cognitive tests presented to participants would become increasingly stimulating over the course of a session. As a result, the ordering of the tests presented by the app was: MFS, Spatial Span Test, PVT, and Mental Arithmetic Test. In addition, the Expert Review also helped to ensure that any changes made to the MFS, specifically its redesign for optimal presentation on a smartphone screen, did not compromise the validity of the corresponding results.
Following the initial Expert Review stage, a Pilot Study was delivered to a small set of participants (n=5), using a first-iteration prototype app. This stage facilitated an evaluation of the efficacy of each of the selected cognitive tests to help ensure that they would challenge participants sufficiently, and facilitate collection of a dataset with appropriate variability to permit subsequent analyses. The Pilot Study also provided an opportunity to gain feedback on information design and visual design choices made during the Expert Review, which might affect the usability of the app.
The iterative nature of the design process allowed for further Expert Review to inform a response to findings from the Pilot Study, which helped to further refine and optimize the design of the app and the planned larger-scale study. A notable instance of this process is the design of the Spatial Span Test, which was revised from a 5x5 grid to a 4x4 grid layout, as the original design proved difficult to accurately tap when displayed on a smartphone screen. Additionally, a countdown timer and progress bar were added to inform participants of both the remaining time during the test and their ongoing progress.
The design of the PVT was also revised to introduce a randomly positioned stimulus, to prevent participants from preempting a response. Immediate visual feedback was also incorporated to encourage concentration and participation during the test. In terms of the Mental Arithmetic Test, initial concerns regarding the layout of the on-screen number keypad used for participant’s responses was validated, with a telephone style layout proving acceptable to participants. Moreover, changes to the MFS element of the app required much more care, as the purpose of including it was to provide an established cognitive fatigue assessment tool to help evaluate how effective the mobile cognitive tests were at assessing cognitive fatigue. To maintain the validity of the MFS, it was important that it differed as little as possible from the original paper-based questionnaire. However, the presentation of each question as a separate element marks a departure from the original paper-based version; this change was made in response to feedback from the design process that found a single continuous onscreen set of questions to be awkward from a user-interaction viewpoint.
Upon completion of different iterations of Expert Review, Pilot Study, and subsequent iterations of Prototype development, the smartphone app was ready to
Screenshots of the smartphone app’s four main screens from left to right: Mental Fatigue Scale, Spatial Span Test, Psychomotor Vigilance Task, and Mental Arithmetic Test.
As shown on the left side of
The second test presented was the Spatial Span Test, which required each participant to recreate a sequence of flashing boxes that appeared on the screen one after the other in a grid formation. If the participant correctly repeated the presented sequence, they were then presented with a new sequence that contained one additional box. Conversely, if the participant entered an incorrect sequence, then the next sequence was one box shorter, with a minimum sequence of three boxes being presented. The use of adaptive difficulty in this manner was intended to help maintain engagement regardless of the participant’s level of performance. The Spatial Span Test lasted for a total of 90 seconds, during which time participants attempted to complete as many sequences as possible.
Upon completion, the PVT was presente: the mobile implementation of PVT required the participant to attempt to achieve the quickest reaction times possible in response to the presentation of onscreen stimuli 20 times; each time a large block of color was randomly displayed on one half of the smartphone screen.
The final test presented was the Mental Arithmetic Test, as illustrated on the right side of
Data collected from mobile fatigue assessment app.
Test | Factors Measured | Data Collected | Type of Data |
Mental Fatigue Scale | Overall fatigue assessment | Date and time test was started | Date/time |
Questionnaire results | Numerical array | ||
Spatial Span Test | Cognitive attention | Total number of sequences complete | Numerical |
Working memory | Longest sequence achieved | Numerical | |
Total number of correct sequences | Numerical | ||
Total number of incorrect sequences | Numerical | ||
Time to complete each full sequence | Numerical array | ||
Psychomotor Vigilance Test | Alertness | Reaction times | Numerical array |
Reaction time | Total number of premature reactions (wrong) | Numerical | |
Total number of timely reactions (right) | Numerical | ||
Mental Arithmetic Test | Cognitive throughput | Total number of correct answers | Numerical |
Total number of incorrect answers | Numerical | ||
Total number of questions presented | Numerical array | ||
Correct answer | Numerical array | ||
User’s answer | Numerical array |
Each specific test captured a range of different data points to assess relative performance. These data included correct and incorrect responses and time it took to respond during each task. Data that was collected via the app is detailed in
Upon conclusion of the study, all participants were invited to provide feedback on the smartphone app using an online version of the System Usability Scale (SUS) [
Participants (n=21) were recruited within Ulster University to undertake the study over a two-week period. Participants with no prior brain injuries were explicitly chosen, as it was considered unethical to test a newly developed method on a clinical population without first understanding how it might be of benefit. Moreover, the experience of cognitive fatigue, while a characteristic of ABI, is not limited to the condition. The mean age of participants recruited was 22 years (standard deviation=4). In addition, participants were required to own an iPhone to ensure that they were familiar with using iOS-based smartphone apps, which removed the need for additional training in device use prior to undertaking the study. Participants were required to use the app once daily to self-assess their cognitive fatigue levels. During the study, 81 individual testing instances were recorded, resulting in an overall daily adherence by participants of 28%.
Data were analyzed using IBM SPSS version 22. Descriptive statistics were used to provide information on average reaction time, MFS questionnaire total score, total number of timely reactions, total number of correct Spatial Span sequences, total number of correct answers to Mental Arithmetic questions, and longest sequence achieved in the Spatial Span Test (
Correlations of testing variables to the self-reporting MFS scale.
Mental Fatigue Scale Score | Average Psychomotor |
Average Psychomotor |
Average Spatial |
Average Arithmetic |
Average Highest |
Correlation | .342a | .159 | -.141 | -.016 | -.064 |
Significance (P-value) | .008 | .157 | .209 | .884 | .568 |
aCorrelation is significant at the 0.01 level (2-tailed).
To identify outliers of extreme and incorrect testing instances, data were analyzed using box plots. Analysis of outliers used the standard measure of 1.5 times the interquartile range to find outliers that were viewed to be too far from the central values to be reasonable. A likely cause of outliers is inaccurate self-assessment (eg, assessment scores were abnormally high or low in comparison to testing scores), in conjunction with a simplistic pattern of responses observed (ie, all responses were the lowest or highest possible choices). Removal of outlying instances of this type helped to ensure only reliably accurate tests were included during analysis.
Overall, 81 individual testing instances were recorded by the participant population, resulting in a daily adherence by participants of 28%. Daily reminder notifications resulted in a participant adherence rate of 23% within the first two hours of receiving the notification; this finding indicates that a large proportion of app usage instances occurred due to the daily reminders.
Initial analysis of the results obtained indicated the average reaction time during the PVT had the strongest correlation with the MFS for evaluating cognitive fatigue, measuring 0.342 (
The best correlation to the MFS was from the average reaction times in the PVT. The significance of this correlation shows that the PVT may be a valid way to assess cognitive fatigue as a replacement for self-reporting. Average reaction time, total number of correct reactions, total number of correct Spatial Span sequences, and total number of correct Mental Arithmetic questions were entered into a multiple linear regression model to determine which factors were predictive of fatigue (see
Correlations of testing variables against the self-reported MFS scale.
Mental Fatigue Scale Score | Average Psychomotor |
Average Psychomotor |
Average Spatial |
Average Arithmetic |
Average Highest |
|||||
Correlation | .342a | .159 | -.141 | -.016 | -.064 | |||||
Significance (P-value) | .008 | .157 | .209 | .884 | .568 |
aCorrelation is significant at the 0.01 level (2-tailed).
Standardized and unstandardized regression coefficients for variables entered into the model.
Variable | B | Standard Error | Βeta Coefficient | T-statistic | P-value | 95% CI |
Mental Fatigue Scale | -6.990 | 27.042 | N/A | -0.258 | .797 | -60.848 to 46.869 |
Average Reaction Time | 46.610 | 18.898 | .312 | 2.466 | .016 | 8.971 to 84.249 |
Average Reactions Correct | 1.442 | 1.654 | .099 | 0.872 | .386 | -1.853 to 4.737 |
Average Spatial Span Correct | -3.505 | 2.991 | -.138 | -1.172 | .245 | -9.462 to 2.452 |
Total Mental Arithmetic Correct | 0.696 | 0.488 | .179 | 1.427 | .158 | -.276 to 1.668 |
Average reaction time performance over the duration of the study.
Self-reported cognitive fatigue measured by the MFS was also shown to correlate with a slower average reaction time, which supports the hypothesis that information processing speed is a predictor of fatigue. From
As previously mentioned, results data were grouped into three epochs throughout the day (
It may be appropriate for future work to further analyze the time of day that tests were undertaken. Consequently, this data could support the hypothesis that a reduced level of performance would be seen in participants with higher self-reported fatigue (see
After completion of the study, participants were further invited to evaluate the smartphone app using an online version of the SUS and to comment on changes to the app that could encourage higher levels of participation. Subsequently, results obtained from this evaluation indicated that the primary reason for nonadherence was the MFS, which was considered
Mean amount of correctly answered arithmetic questions was 4.34% lower during the morning epoch than the afternoon epoch and 4.58% lower than the evening epoch
The longest sequence achieved in the Spatial Span test was 1.05% lower in the morning epoch than the afternoon epoch and 1.57% lower than the evening epoch
The total number of correct Spatial Span sequences achieved was 0.41% lower in the morning epoch than the afternoon epoch and 1.63% lower than the evening epoch
Descriptive statistics for morning, afternoon, and evening epochs.
Epoch | N | Minimum | Maximum | Mean | Standard Deviation | |
Reaction time (seconds) | 15 | 0.34846 | 0.60071 | 0.41786a | 0.07409 | |
Mental Fatigue Scale result | 15 | 24 | 84 | 40.93 | 17.123 | |
Total number of timely reactions | 15 | 12 | 15 | 14.53 | 0.834 | |
Total number of correct Spatial Span sequences | 15 | 4 | 5 | 4.87 | 0.352 | |
Total number of correct Mental Arithmetic answers | 15 | 15 | 28 | 23.47 | 3.461 | |
Longest Spatial Score sequence reached | 15 | 4 | 7 | 5.67 | 0.9 | |
Reaction time (seconds) | 45 | 0.2950 | 0.65199 | 0.386a | 0.05733 | |
Mental Fatigue Scale result | 45 | 14 | 78 | 30.2 | 12.836 | |
Total number of timely reactions | 45 | 12 | 15 | 13.89 | 0.959 | |
Total number of correct Spatial Span sequences | 45 | 4 | 6 | 4.89 | 0.573 | |
Total number of correct Mental Arithmetic answers | 45 | 14 | 31 | 24.51 | 3.501 | |
Longest Spatial Score sequence reached | 45 | 3 | 9 | 5.73 | 1.372 | |
Reaction time (seconds) | 21 | 0.3069 | 0.7600 | 0.37417a | 0.1443 | |
Mental Fatigue Scale result | 21 | 14 | 53 | 30.81 | 10.731 | |
Total number of timely reactions | 21 | 12 | 15 | 14 | 0.894 | |
Total number of correct Spatial Span sequences | 21 | 4 | 6 | 4.95 | 0.590 | |
Total number of correct Mental Arithmetic answers | 21 | 19 | 31 | 24.57 | 3.682 | |
Longest Spatial Score sequence reached | 21 | 3 | 9 | 5.76 | 1.411 |
aDenotes the median reaction time
While it is clear from the results obtained that all cognitive fatigue tests showed a degradation in the participant’s performance when there is a higher level of self-reported fatigue (as hypothesized prior to the study), analysis indicates that only the PVT is significant enough to make claims on its validity. Subsequently, both the Spatial Span Test and Mental Arithmetic Test do not provide a strong correlation with the MFS for measuring cognitive fatigue. Although a correlation may be observed from all three of the tests as hypothesized, currently only the PVT has the potential to accurately assess cognitive fatigue using a mobile device, as it is able to determine the current level of mental fatigue concordant with the levels reported using the MFS. Furthermore, PVT is less demanding for participants to undertake than the other cognitive tests. Feedback from a usability viewpoint also suggested that the PVT was considered by participants to be more engaging than the MFS. From a participant perspective, assessment of cognitive fatigue is feasible via a smartphone app; however, adherence to regular testing is crucial to gaining an understanding of the condition over time.
Adaptation of the smartphone app to increase the overall time on the cognitive tests could potentially lead to more accurate evaluation results across all tests, although this is only possible if participants are willing to take part in a longer daily task. Tailoring time-on-task to promote continued adherence, while also collecting sufficient data to accurately assess the condition, is a critical balance that requires further research. Such studies will enable a greater understanding of the testing duration required to permit accurate mobile-based cognitive fatigue evaluation. Future work may benefit from increased time-on-task for each of the three cognitive tests.
When considering the scores of the PVT from individual participants, it can be observed that there was no improvement in scores over time. It is important that the participants find the test equally taxing over time, so that the same indicators of fatigue are present, rather than a participant learning from previous sessions and becoming increasingly expert in undertaking the assessment. Such learning would consequentially mask any underlying cognitive fatigue.
The approach of interpreting fatigue using a mobile device has been demonstrated to have validity. However, a number of improvements could be made to the process, in addition to changes to the specific data that is collected. Such changes could include collecting data on environmental factors such as the quantity and quality of sleep, daily exercise levels, current location and social environment, and current emotional status. Based on the study performed, participant feedback indicated that carrying out the MFS was one of the least enjoyable parts of the overall process, which potentially lead to a reduction in engagement. The use of PVT has been shown to provide a similar capability in the assessment of cognitive fatigue, meaning that future work can potentially exclude the use of MFS. Subsequently, this exclusion may further increase user participation rates, which in turn may potentially increase the accuracy of the cognitive fatigue evaluation. An abridged variation of PVT was employed within the smartphone app; however, it is anticipated that a higher degree of accuracy could be achieved by using a longer test session. Accordingly, future work should increase the length of the PVT utilized, which could produce a more precise indicator of fatigue.
A limitation of the study is that daily participation was not enforced. This enforcement could have increased the amount of data obtained, thereby improving the confidence of the findings. Given the real-world approach employed throughout the study, the only available (and appropriate) participation encouragement mechanism was seen to be mobile device notifications. Future work will aim to improve participation rates, in conjunction with increasing the amount of data collected to potentially obtain a more accurate assessment of daily fatigue levels. Furthermore, all cognitive test metrics that were employed alongside the MFS were constrained to a relatively short timeframe, in order to aid participation rates. However, this limitation may have had an adverse impact on the effectiveness of the tests in identifying small discrepancies in performance. Other limitations of the current study include the short duration of the overall study; future work will seek to address all of these limitations.
The validity of assessing fatigue using both the Spatial Span Test and Mental Arithmetic Test has been proven to be effective by Van Dongen et al [
Work carried out by Thomson et al [
However, in terms of the efficacy of mobile-based self-assessment, Swendeman et al [
One key concern with cognitive tests (such as PVT) that have been previously adapted for a mobile platform [
Previous studies have confirmed the use of the MFS and PVT as a feasible way to assess cognitive fatigue in a clinical setting. This study demonstrated that smartphone-based adaptations of these proven methods are internally compatible methods of assessing cognitive fatigue in situ. The smartphone app presented in this research provides a potentially effective tool for the individual evaluation of cognitive fatigue levels in situations where formal intervention and assessment approaches are neither feasible nor available. Furthermore, the use of smartphone app-based fatigue assessment permits evaluation to be carried out on a continual daily basis.
All three of the cognitive tests employed within the smartphone app produced positive participant feedback, with some participants even indicating that they would like personal scores, as it would further encourage them to participate more frequently. Consequently, by introducing a competitive aspect to the cognitive tests, participant effort and daily participation rates could potentially be improved. Future work may additionally permit the provision of real-time data to relevant medical professionals, so effective and timely interventions can be arranged if extreme fatigue becomes apparent. Correspondingly, there exists three main areas that in situ fatigue assessment could benefit from sensor data and contextual factors: (1) to improve notifications so that daily participation can be increased; (2) to provide additional data that may give a more precise indication of the occurrence of cognitive fatigue; and (3) to track a participant’s daily activity and advise them on appropriate steps to further combat cognitive fatigue.
Based on the three episodic epochs, results from both the MFS and PVT indicated that the morning epoch produced higher levels of fatigue. Accordingly, this knowledge could be employed on an individual basis to help tailor the time of delivery of the test session, to pinpoint higher fatigue levels throughout the day. Adaptation of notifications (based on collected data over time) would further facilitate assessment when fatigue is usually at its highest. Correspondingly, identifying the time of day when fatigue levels are high offers an increased ability to prevent mental fatigue by preempting its arrival and suggesting steps to take to reduce its severity. Assessing the time of an individual’s participation could help inform notifications for the most likely time of adherence.
Acquired Brain Injury
Fatigue Severity Scale
Mental Fatigue Scale
Psychomotor Vigilance Task
System Usability Scale
Visual Analogue Scale for Fatigue
This research was funded and supported by the Department of Learning and Employment, Northern Ireland. This publication was supported by the Smart Environments Research Group, within the Computer Science Research Institute, Ulster University.
None declared.