Published on 26.05.20 in Vol 8, No 5 (2020): May
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/17804, first published Jan 14, 2020.
Augmented Reality–Based Rehabilitation of Gait Impairments: Case Report
Background: Gait and balance impairments are common in neurological diseases, including stroke, and negatively affect patients’ quality of life. Improving balance and gait are among the main goals of rehabilitation. Rehabilitation is mainly performed in clinics, which lack context specificity; therefore, training in the patient’s home environment is preferable. In the last decade, developed rehabilitation technologies such as virtual reality and augmented reality (AR) have enabled gait and balance training outside clinics. Here, we propose a new method for gait rehabilitation in persons who have had a stroke in which mobile AR technology and a sensor-based motion capture system are combined to provide fine-grained feedback on gait performance in real time.
Objective: The aims of this study were (1) to investigate manipulation of the gait pattern of persons who have had a stroke based on virtual augmentation during overground walking compared to walking without AR performance feedback and (2) to investigate the usability of the AR system.
Methods: We developed the ARISE (Augmented Reality for gait Impairments after StrokE) system, in which we combined a development version of HoloLens 2 smart glasses (Microsoft Corporation) with a sensor-based motion capture system. One patient with chronic minor gait impairment poststroke completed clinical gait assessments and an AR parkour course with patient-centered performance gait feedback. The movement kinematics during gait as well as the usability and safety of the system were evaluated.
Results: The patient changed his gait pattern during AR parkour compared to the pattern observed during the clinical gait assessments. He recognized the virtual objects and ranked the usability of the ARISE system as excellent. In addition, the patient stated that the system would complement his standard gait therapy. Except for the symptom of exhilaration, no adverse events occurred.
Conclusions: This project provided the first evidence of gait adaptation during overground walking based on real-time feedback through visual and auditory augmentation. The system has potential to provide gait and balance rehabilitation outside the clinic. This initial investigation of AR rehabilitation may aid the development and investigation of new gait and balance therapies.
JMIR Mhealth Uhealth 2020;8(5):e17804
Many neurological diseases lead to impairments of gait and balance . Approximately 80% of persons who have a stroke experience such deficits [ ]. The key characteristics of impaired gait after stroke are shortened stance phase of the affected side, reduced knee and hip flexion during swing, slower walking speed, and shorter stride length [ ]. Balance impairments are characterized by reduction in maximum weight shift toward the affected side [ ], delayed postural reactions [ ], shift of the center of mass toward the non-affected side, and decreased ability to avoid obstacles [ ]. These gait and balance impairments are increased when patients are required to perform a cognitive task in parallel [ ] and in patients who have attention or vision deficits [ , ]. Impaired gait and balance poststroke can have profound consequences for patients, as these impairments are strongly related to increased fear of falling [ ] and reduced quality of life [ ].
Therefore, improving gait and balance is one of the main goals of stroke rehabilitation. Repetitive practice is an essential ingredient of established evidence-based interventions  such as speed-dependent treadmill training, postural control with visual feedback training, and task-specific training [ ]. These interventions can be delivered as part of inpatient or outpatient programs. In all these settings, training lacks context specificity (i.e., mobility in the daily life environment of the individual patient). In the last decade, virtual reality (VR) training systems have been found to effectively improve gait and balance after stroke [ , ]. VR systems provide challenging training situations and many different training environments. However, these VR systems still do not reflect the real-world environment of the patient; also, they require expensive stationary equipment. In addition, these systems are often limited in their variety of exercises to improve balance during gait training [ ]. To interact with the real-world environment, augmented reality (AR) is an option to provide multiple sensory feedback enhanced by computer-generated perceptual information.
Considering the importance of performance feedback for motor skill learning , AR combined with sensor-based kinematic measurements can deliver fine-grained visual and auditory feedback on gait and balance parameters; this feedback can provide higher specificity and continuity and lower delay than the feedback delivered by a therapist [ ]. Sensor-based motion capture systems, including inertial measurement units, can measure a patient’s gait kinematics and center of mass outside the laboratory and clinic [ ], although laboratory-based optical tracking systems remain the gold standard with respect to the sensitivity and accuracy of these systems [ ].
Here, we propose a new method for gait and balance rehabilitation in patients who have had a stroke that has potential to provide challenging and personalized gait and balance therapy with auditory and visual performance feedback based on gait kinematics in an environment that is adjusted to the patients’ needs. We developed the Augmented Reality for gait Impairments after StrokE (ARISE) system, in which we combined a development version of a head-mounted system for real-time visual and auditory feedback and a commercially available sensor-based motion capture system. Subsequently, we evaluated the usability of the AR feedback prototype in a chronic stroke subject with minor gait and balance impairments. We hypothesized that this system is more capable of modifying gait kinematics than walking without the system.
Ethical clearance to execute the experiment with the presented subject was provided by the Cantonal Ethics Commission of the Canton of Zurich, Switzerland (BASEC-Nr. Req-2019-00758). The participant received information about the experiments. Written informed consent in accord with the Declaration of Helsinki was obtained from the participant prior to performing the experiments.
The patient was a right-handed man aged 74 years who had a right-hemispheric ischemic stroke in the thalamus, capsula interna, and right temporal lobe 7 years before participating in the experiment. Acutely after stroke, he had mild motor deficits in the left leg (National Institutes of Health Stroke Scale leg item score 1/4). At the time of this experiment, he had minor limitations of sitting and standing balance (Berg Balance Scale 54/56); he also had mild limitations in motor function (Fugl-Meyer Motor Assessment lower extremity subscale 29/34) and strength (Motricity Index lower extremity subscale 88/100) of the affected left side. The patient walked independently without a walking aid (Functional Ambulation Categories 5/5), with a comfortable walking speed of 1.0 m/s and a step length of 0.56 meters as measured by the 10-Meter Walk Test. The patient had slight risk of falling (Dynamic Gait Index 19/24). He reported numbness in the feet and a feeling of wearing socks when he was not. In addition, a mild cognitive impairment was present (Montreal Cognitive Assessment 25/30).
The ARISE system consisted of two essential components: an optical see-through head-mounted display (OST-HMD) and a sensor-based motion capture system (, Supplementary Material A).
We identified the HoloLens 2 (Microsoft Corporation) as the most suitable OST-HMD, as it provides a wider field of view (43×29 degrees) compared to other devices and thus can display more virtual objects in a real-world environment. In addition, the HoloLens 2 is able to track head movements with an inertial measurement unit and has an intuitive hand-interaction user interface that is enabled through fully articulated hand tracking. The OST-HMD was used to visualize the AR parkour course (A-C). The state-of-the-art parkour course had an area of approximately 14×4 meters; it consisted of visualizations of real-life obstacles and barriers, such as blocks and floor mats, that forced the patient to perform certain leg movements. A trail of arrows indicated the walking direction, and the parkour course changed dynamically depending on the position of the patient.
In addition, the OST-HMD provided visual and auditive feedback based on real-time gait kinematic performance. An inertial measurement unit–based motion capture system, the Xsens MVN (Xsens Technologies B.V.), was chosen to track the kinematics of the lower limbs during gait. We strapped seven inertial measurement units on the pelvis and the lower extremity of the patient (). The Xsens software (MVN version 2019.2) converted the rotational data from the inertial measurement units into a fully articulated virtual mannequin. This provided translational and rotational data of every large humanoid joint. With this combination, the patient was able to walk longer distances (more than 10 meters).
The HoloLens 2 and Xsens systems were integrated by streaming movement performance data to the OST-HMD through a user datagram protocol client. To adjust for translational drift of the motion capture system in the ARISE system, the captured motion data were rigidly attached to the pose of the HoloLens 2. To synchronize the forward directions of both systems, we performed a short calibration phase where the user was required to align his head with his hips and feet. Once the 2 systems were coupled, we were able to provide subtle positive reinforcement of successful knee flexion (>45 degrees) by playing an ambient bird song sound through the HMD.
To increase the difficulty of the tasks the subject was asked to complete, simple math calculations were presented visually (dual task procedure,D). The subject responded by pressing a virtual button, which was detected by the hand-tracking capabilities of the HoloLens 2. At the end of the parkour course, a knowledge of results display was shown, including the time to run the parkour course, the correct answers to the math calculations, and the percentage of time that the condition of knee flexion >45 degrees was fulfilled.
We provided the participant with information about the experimental setup. After clinical assessments of the participant were performed, the sensors were fitted to his legs and pelvis, and he donned the HoloLens 2. First, a patient-personalized height scaling model was applied to the motion capture software. The motion capture system was calibrated. The participant performed a 10-meter walk test and then completed the AR parkour course three times. During walking, the 10-meter walk test and the AR parkour, the patient’s lower extremity kinematics and center of mass were tracked with the motion capture system. After the AR parkour, the System Usability Scale , the Virtual Reality Symptom Questionnaire [ ], and a semi-structured interview were used to assess the patient’s experience using the AR system.
All datasets generated for this publication are included in the manuscript.
The kinematic parameters of the knees and the center of mass while walking during the 10-meter walk test and the AR parkour are listed in detail inand , respectively. The patient adapted his gait performance and increased his knee flexion angles during the gait cycle when performing the AR parkour compared to when walking a straight line during the 10-meter walk test ( ). His walking speed in the 10-meter walk test did not change (pre 9 seconds, post 9 seconds).
During the AR parkour, the patient perceived the virtual objects, stepped over the obstacles and barriers, and reported a feeling of being in a real-world parkour course. Despite the positive effects on knee flexion, he did not consciously perceive the bird songs.
The patient overlooked the math problems during the first run but solved them in subsequent trials after being reminded to do so. This increased the time to complete the AR parkour from 75 seconds without the second task to 100 seconds with the second task.
The patient reported that wearing the HoloLens 2 felt like wearing a hat. He criticized the limited vertical field of view, which forced him to lean forward to see the obstacles directly beneath him (see the video in). However, he ranked the usability of the AR parkour system as excellent (System Usability Scale 87.5/100). He stated that he would like to use the system more often for self-training to complement his current conventional outpatient therapy program. No adverse events were measured with the Virtual Reality Symptom Questionnaire, with the exception of the symptom of exhilaration.
|Parameter||Stancea||Swingb||At foot strikec||At foot released|
|Minimum||Maximum||Minimum||Maximum||Mean (SD)||Mean (SD)|
|Ten-meter walk test|
|Hip angle flexion|
|Lefte||–7.17||31.06||–3.20||33.57||28.34 (2.02)||–0.78 (0.54)|
|Right||–11.18||29.63||–8.01||30.21||26.74 (0.84)||–4.00 (2.79)|
|Knee angle flexion|
|Lefte||7.18||37.82||3.05||61.37||8.44 (1.59)||41.84 (0.38)|
|Right||2.53||41.45||–2.50||61.34||3.04 (0.73)||38.23 (4.77)|
|Ankle angle flexion|
|Lefte||–4.92||19.49||–10.85||9.07||0.86 (1.19)||–4.05 (2.50)|
|Right||–8.11||21.58||–26.09||7.63||–3.14 (2.25)||–4.62 (4.02)|
|Augmented reality parkour|
|Hip angle flexion|
|Lefte||–2.09||29.77||–2.06||49.26||23.77 (3.42)||3.82 (4.80)|
|Right||–3.55||33.90||–3.30||50.98||23.49 (4.68)||1.83 (5.40)|
|Knee angle flexion|
|Lefte||5.26||28.28||5.63||79.74||10.22 (2.84)||23.06 (4.39)|
|Right||3.00||43.83||–2.57||82.93||8.57 (5.42)||28.08 (6.04)|
|Ankle angle flexion|
|Lefte||–6.70||23.22||–4.49||21.33||3.60 (3.67)||15.28 (4.60)|
|Right||–6.32||31.45||–16.95||29.63||2.13 (3.41)||17.48 (6.35)|
aPeriod of time from a foot strike to the following release of the same foot.
bPeriod of time from a foot release to the following strike of the same foot.
cTime at which the foot contacts the ground after a swing phase.
dTime at which the foot stops being in contact with the ground after a stance phase.
eThe patient’s affected side.
|Ten-meter walk test|
|Augmented reality parkour|
aPeriod of time from a foot strike to the following release of the same foot.
bPeriod of time from a foot release to the following strike of the same foot.
cThe patient’s affected side.
Gait and balance training requires training systems that are mobile and preferably wearable. The ARISE system proposed here combines flexible training with performance feedback in a wearable form by integrating an OST-HMD and a sensor-based motion capture system. It enables context-specific training and can display virtual objects via the HMD in a real-world space. The AR rehabilitation system can be used for overground walking, while gait kinematics are measured with the motion capture system. The combination of these technologies is new in gait rehabilitation. Most AR and VR systems consist of stationary equipment, including a treadmill, cameras, and projection devices . Hence, they offer fewer options for adaptation [ ] and are restricted to the unreal training environment of treadmill walking; moreover, a minimum of 6 minutes (around 400 strides) is required to achieve stable performance on a treadmill [ ].
The ARISE system was tested during a single session experiment with one patient who had a stroke. The patient perceived the system to be comfortable, and it did not restrict his movements. The limited vertical field of view (29 degrees) of the HoloLens 2 should be improved to increase the usability of the system. The field of view forced the patient to bend forward; this can induce adverse events, such as near-falls or neck pain, when using the system over longer periods of time.
Through the coupling of the OST-HMD and the motion capture system, the patient perceived auditive feedback based on the kinematics of the knee (flexion angle). The OST-HMD provided feedback in the form of bird songs and a knowledge of results display. When the patient was given multimodal feedback, he changed his gait pattern. This is in line with previous results [, ]. A transfer of training from the ARISE system to overground walking can be expected when increasing the duration of the intervention [ ]. Consolidation may be further improved by adding monetary rewards [ ].
The current AR parkour course has a size of 14×4 meters; therefore, it requires a large open space to operate. In future versions, the ARISE system will be able to generate a parkour course with an adaptable area depending on the local environment. This will provide the advantages of flexibility in choosing the test grounds and a variety of parkour designs. In future research, the impacts of different virtual obstacles or even obstacle themes (e.g., street, garden, supermarket) on the gait kinematics can be evaluated. Due to the high mobility of all involved components, our system can be used virtually anywhere, both indoors and outdoors and with different surfaces and distractors. The mobility of the system is also ensured through the web interface of the OST-HMD and the motion capture system, which allows remote access. Therefore, a telerehabilitation approach is conceivable.
The present ARISE system is a prototype, and it requires technical support for setup and calibration. Therefore, a reduced sensor set and simplified calibration procedure would increase its usability and applicability. Furthermore, movement sensors are limited by orientation drift when they are used over a long period of time . In this study, we did not use the system long enough to observe this drift. These issues of technical support and drift currently limit the use of the system in telerehabilitation and should be addressed in future studies [ ].
The ARISE system combines an adjustable and personalized training environment with a feedback and monitoring system for gait and balance rehabilitation. This case study showed that use of the system by patients who have had a stroke and who have mild gait and balance impairments is promising. Compared to existing AR systems, the system utilizes the real physical environment to place virtual obstacles that are immediately and intuitively treated as real-world objects. This system can be used to provide feedback on a variety of gait parameters and to implement personalized (dual-task) gait training environments.
This research was supported by the P & K Pühringer Foundation.
JH and AL conceived the presented idea. JH designed the study, recruited the patient, conducted the clinical assessments, and analyzed the patient data. KY, CP, and JH developed the methodology and conducted the instrumented experiments during Medical Augmented Reality Summer School 2019 at University Hospital Balgrist, Zurich, Switzerland. JH, KY, and JV drafted the manuscript. All authors revised the manuscript and approved the final version.
Conflicts of Interest
AL is a scientific advisor for Hocoma AG (Volketswil, Switzerland), which develops rehabilitation technology. The ARISE system, which includes HoloLens 2 (Microsoft) in combination with the Xsens MVN, is a research prototype that was developed during the Augmented Reality Summer School at University Hospital Balgrist, Zurich, Switzerland, and is not commercially available.
Augmented Reality parkour for gait Impairments after StrokE.MP4 File (MP4 Video), 31316 KB
- Nonnekes J, Goselink RJM, Růžička E, Fasano A, Nutt JG, Bloem BR. Neurological disorders of gait, balance and posture: a sign-based approach. Nat Rev Neurol 2018 Mar;14(3):183-189. [CrossRef] [Medline]
- Duncan PW, Zorowitz R, Bates B, Choi JY, Glasberg JJ, Graham GD, et al. Management of Adult Stroke Rehabilitation Care: a clinical practice guideline. Stroke 2005 Sep;36(9):e100-e143. [CrossRef] [Medline]
- Perry CJ, Kiriella JB, Hawkins KM, Shanahan CJ, Moore AE, Gage WH. The effects of anterior load carriage on lower limb gait parameters during obstacle clearance. Gait Posture 2010 May;32(1):57-61. [CrossRef] [Medline]
- van Dijk MM, Meyer S, Sandstad S, Wiskerke E, Thuwis R, Vandekerckhove C, et al. A cross-sectional study comparing lateral and diagonal maximum weight shift in people with stroke and healthy controls and the correlation with balance, gait and fear of falling. PLoS One 2017;12(8):e0183020 [FREE Full text] [CrossRef] [Medline]
- Tasseel-Ponche S, Yelnik AP, Bonan IV. Motor strategies of postural control after hemispheric stroke. Neurophysiol Clin 2015 Nov;45(4-5):327-333. [CrossRef] [Medline]
- van Ooijen MW, Heeren A, Smulders K, Geurts ACH, Janssen TWJ, Beek PJ, et al. Improved gait adjustments after gait adaptability training are associated with reduced attentional demands in persons with stroke. Exp Brain Res 2015 Mar;233(3):1007-1018. [CrossRef] [Medline]
- Bowen A, Wenman R, Mickelborough J, Foster J, Hill E, Tallis R. Dual-task effects of talking while walking on velocity and balance following a stroke. Age Ageing 2001 Jul;30(4):319-323. [CrossRef] [Medline]
- Bonan IV, Guettard E, Leman MC, Colle FM, Yelnik AP. Subjective visual vertical perception relates to balance in acute stroke. Arch Phys Med Rehabil 2006 May;87(5):642-646. [CrossRef] [Medline]
- Bonan IV, Hubeaux K, Gellez-Leman MC, Guichard JP, Vicaut E, Yelnik AP. Influence of subjective visual vertical misperception on balance recovery after stroke. J Neurol Neurosurg Psychiatry 2007 Jan;78(1):49-55 [FREE Full text] [CrossRef] [Medline]
- Schinkel-Ivy A, Inness EL, Mansfield A. Relationships between fear of falling, balance confidence, and control of balance, gait, and reactive stepping in individuals with sub-acute stroke. Gait Posture 2016 Jan;43:154-159. [CrossRef]
- Park J, Kim T. The effects of balance and gait function on quality of life of stroke patients. NeuroRehabilitation 2019;44(1):37-41. [CrossRef] [Medline]
- French MA, Moore MF, Pohlig R, Reisman D. Self-efficacy Mediates the Relationship between Balance/Walking Performance, Activity, and Participation after Stroke. Top Stroke Rehabil 2016 Apr;23(2):77-83 [FREE Full text] [CrossRef] [Medline]
- Veerbeek JM, van Wegen E, van Peppen R, van der Wees PJ, Hendriks E, Rietberg M, et al. What is the evidence for physical therapy poststroke? A systematic review and meta-analysis. PLoS One 2014 Feb;9(2):e87987 [FREE Full text] [CrossRef] [Medline]
- Heeren A, van Ooijen M, Geurts ACH, Day BL, Janssen TWJ, Beek PJ, et al. Step by step: a proof of concept study of C-Mill gait adaptability training in the chronic phase after stroke. J Rehabil Med 2013 Jul;45(7):616-622 [FREE Full text] [CrossRef] [Medline]
- de RIJM, van DPIGL, Meijer JG. Effect of Virtual Reality Training on Balance and Gait Ability in Patients With Stroke: Systematic Review and Meta-Analysis. Phys Ther 2016 Dec;96(12):1905-1918. [CrossRef] [Medline]
- van Duijnhoven HJR, Heeren A, Peters MAM, Veerbeek JM, Kwakkel G, Geurts ACH, et al. Effects of Exercise Therapy on Balance Capacity in Chronic Stroke: Systematic Review and Meta-Analysis. Stroke 2016 Oct;47(10):2603-2610. [CrossRef] [Medline]
- Veerbeek JM, van Wegen E, van Peppen R, van der Wees PJ, Hendriks E, Rietberg M, et al. What Is the Evidence for Physical Therapy Poststroke? A Systematic Review and Meta-Analysis. PLoS ONE 2014 Feb 4;9(2):e87987. [CrossRef]
- Stanton R, Ada L, Dean CM, Preston E. Biofeedback improves activities of the lower limb after stroke: a systematic review. J Physiother 2011;57(3):145-155 [FREE Full text] [CrossRef] [Medline]
- Reenalda J, Maartens E, Buurke JH, Gruber AH. Kinematics and shock attenuation during a prolonged run on the athletic track as measured with inertial magnetic measurement units. Gait Posture 2019 Feb;68:155-160. [CrossRef] [Medline]
- Karatsidis A, Richards RE, Konrath JM, van den Noort JC, Schepers HM, Bellusci G, et al. Validation of wearable visual feedback for retraining foot progression angle using inertial sensors and an augmented reality headset. J Neuroeng Rehabil 2018 Aug 15;15(1):78 [FREE Full text] [CrossRef] [Medline]
- Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. Int J Hum-Comput Int 2008 Jul 30;24(6):574-594. [CrossRef]
- Ames SL, Wolffsohn JS, McBrien NA. The development of a symptom questionnaire for assessing virtual reality viewing using a head-mounted display. Optom Vis Sci 2005 Mar;82(3):168-176. [Medline]
- Cavalcanti V, Santana M, Gama A, Correia WF. Corrigendum to “Usability Assessments for Augmented Reality Motor Rehabilitation Solutions: A Systematic Review”. Int J Comput Games Technol 2019 Jan 21;2019:1-1. [CrossRef]
- Meyer C, Killeen T, Easthope CS, Curt A, Bolliger M, Linnebank M, et al. Familiarization with treadmill walking: How much is enough? Sci Rep 2019 Mar 26;9(1):5232 [FREE Full text] [CrossRef] [Medline]
- Widmer M, Held JP, Wittmann F, Lambercy O, Lutz K, Luft AR. Does motivation matter in upper-limb rehabilitation after stroke? ArmeoSenso-Reward: study protocol for a randomized controlled trial. Trials 2017 Dec 02;18(1):580 [FREE Full text] [CrossRef] [Medline]
- Roetenberg D, Luinge H, Slycke P. Xsens MVN: full 6DOF human motion tracking using miniature inertial sensors.: Xsens Motion Technologies BV. Tech Rep; 2009. URL: https://www.researchgate.net/publication/239920367_Xsens_MVN_Full_6DOF_human_motion_tracking_using_miniature_inertial_sensors [accessed 2020-04-20]
- Held JPO, Klaassen B, Eenhoorn A, van Beijnum BF, Buurke JH, Veltink PH, et al. Inertial Sensor Measurements of Upper-Limb Kinematics in Stroke Patients in Clinic and Home Environment. Front Bioeng Biotechnol 2018 Apr;6:27 [FREE Full text] [CrossRef] [Medline]
|AR: augmented reality|
|ARISE: Augmented Reality for gait Impairments after StrokE|
|OST-HMD: optical see-through head-mounted display|
|VR: virtual reality|
Edited by G Eysenbach; submitted 14.01.20; peer-reviewed by J Salisbury, M Ozkan; comments to author 03.03.20; revised version received 06.03.20; accepted 22.03.20; published 26.05.20
©Jeremia Philipp Oskar Held, Kevin Yu, Connor Pyles, Janne Marieke Veerbeek, Felix Bork, Sandro-Michael Heining, Nassir Navab, Andreas Rüdiger Luft. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 26.05.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.