AME Mixed Reality System for Stroke Rehabilitation

Foundational Principles and Structure

Our team has developed a novel mixed reality system that aims to extend the benefits of mediated rehabilitation through the use of perceptual and interactive arts principles, real-time motion analysis, hybrid (physical-digital) training and computational adaptation of system parameters.

Our system utilizes a nested network representation of key functional features of reach and grasp movement. Our representation of movement is based upon phenomenological approaches to interaction and rehabilitation approaches to functional restitution, and the related understanding of activity recovery versus compensation. This representation of action forms the basis of the system's multimodal feedback environment. It also provides a novel approach to facilitating the stroke survivor's knowledge of the highly complex action space of reach and grasp activities.

The interactive, multimodal feedback environment intuitively communicates to the stroke survivor the state and dynamics of key features of his/her movement performance in an integrated manner. Our feedback derives its intuitive communication strength from foundational perceptual and design principles used within film, music, dance, and media arts. Through the feedback, we direct the subject's attention to key aspects of his/her action, which allow for the subject to actively assess multiple features of his/her performance in terms of how each component contributes to task accomplishment. Our feedback environment maps to any type of reaching task and thus encourages development of generalizable movement strategies.

Our system integrates adaptive virtual feedback with adaptable physical elements, which increases the customization possibilities for each individual subject. It further allows us to (1) increase the virtual elements in order to recontextualize the reaching task within the feedback environment, and (2) decrease (fade) feedback to facilitate transfer of performance knowledge gained through virtual elements to unassisted physical action. Thus the digital-physical continuum ultimately facilitates transference of knowledge gained through the interactive media to activity performed in the physical space.

Throughout the five-year development of the system, small pilot studies were conducted with unimpaired subjects and stroke survivors to test key elements and concepts. These studies have successfully demonstrated a proof of principle that an adaptive mixed reality rehabilitation system can provide a customized reaching and grasping training program to chronic stroke patients and elicit improvements in important movement parameters. Further details on the system and its original development work can be found in the list of papers given below. Information on current trials and ongoing research is given in the paragraphs below.

Computational Assessment and Adaptation

We are also developing computational methods of assessing and adapting the training procedures within the mixed reality rehabilitation system. Our system includes over 250 control variables (approximately 100 kinematic parameters linked to the multi-joint coordinated arm movement, and over 130 adaptable virtual and physical parameters that control the feedback type and presentation). Sequence and intensity of tasks also vary per subject. The control of a system with such high dimensionality requires computational assistance. We therefore have developed customized computational algorithms and tools to assist the therapist in adjusting the feedback and task to better address each subject's needs.

recovery of physical space

We have developed a kinematic deficit index, which is a unified and subject-independent deficit measure for evaluating subject performance during a reaching and grasping task. The key aspects of the deficit measure are that (1) it is a subject-independent computational measure of kinematic deficit that allows for comparison of progress across patients, (2) the deficit measure is bounded, and thus it is indicative of the room for improvement, and (3) it allows for quantitative understanding of the rehabilitation progress, and hence provides a computational indicator for the rehabilitation team on decisions for adapting the therapy.

We address the computational performance evaluation problem by presenting an algorithm to compute the kinematic deficit index and deficit-training-improvement (DTI) correlation. The deficit-training-improvement (DTI) correlation communicates the effect of the therapy by reflecting the patient's progress from pre-therapy to post-therapy and the correlation between the improvement and training. The DTI correlation is based upon calculating the correlations between the patient's initial movement deficit (D), the training implemented through our system (T), and the improvement in the patient's movement at the end of the therapy (I). With this framework, we can evaluate and compare the different training procedures implemented for different subjects who engaged in mixed reality rehabilitation.

Current Research

Clinical Study At Banner Baywood

Last year, we began a partnership with the Rhodes Rehabilitation Institute at Banner Baywood Medical Center. Our system was installed at the Institute and we initiated a clinical research study that compares our mediated therapy to traditional physical therapy. This study will further inform development of a home based training system that will be used in partnership with Banner Baywood Medical Center as well. As of October 2009, we have recruited 9 stroke survivors to participate in the study and aim to recruit a total of 30 patients. The patients in the intervention group experience 4 weeks of mediated training and the patients in the control group experience four weeks of traditional therapy. The mediated system includes a low-cost motion tracking system, full feedback paradigm, and various tangible objects. Results from the clinical study to-date show that our system has significantly improved performance of a reaching and grasping task for stroke survivors with different severity of stroke, after one month of short-term training. Improvement of patients using our system also compares favorably to improvements of our control group that is undergoing traditional physical therapy.

Home System

homesystem

Our team is also developing a scaled-down version of the system for use within a stroke survivor's home. The home rehabilitation system attempts to bring the training and assessment of physical therapist-led therapy sessions to the patient's home in an affordable and low cost manner. The first version consisted of a table top game that featured three fixed targets of varying difficulty for the patient to reach towards and grasp. The 3D position of the user's wrist was tracked by using the infrared technology of two Nintendo Wii Remotes. This preliminary assessment tool was used by two patients, from which basic kinematic calculations and patient usage patterns were collected. The overall response from both users was very positive, as both participants utilized the system well above the minimum requirements.

At this time, a second version of the home system is in development. The overall goal is to advance beyond a pure assessment tool and develop an active and engaging training device that can integrate within the home environment. Some of the main innovative features include (1) development of interactive components that can be moved outside of a motion capture space, thus integrating more with daily living activities, (2) presentation of a more engaging interactive environment with intuitive information provided to the patient about progress, and also (3) an improved tele-rehabilitation component for the therapist to interact with the patient remotely as needed. The second version of the home system will be in development through the spring of 2010.

Brain Imaging

fmri
Experiments are currently underway to incorporate electroencephalography (EEG) and functional MRI to observe changes in neural activity associated with training in a mixed reality rehabilitation system. The fMRI studies look at long-term effects on the brain from training in a mixed reality system and are performed before and after training. These scans are conducted at Barrow Neurological Institute of St. Joseph's Health and Medical Center. The EEG experiments will occur at ASU using a 32 channel EEG system and will examine the effect of different types of feedback environments on neural activity.

Parkinson's Disease

Finally, we are also scaling our approach and system for stroke rehabilitation to be used for the rehabilitation of Parkinson's patients. This aspect of the mixed reality project is being realized in collaboration with the University of Arizona Medical School. A small design study of the pilot version of the Parkinson's rehabilitation system will be completed in Spring 2009.

Project Publications

  • Yinpeng Chen, Nicole Lehrer, Margaret Duff, Jiping He, Hari Sundaram, Thanassis Rikakis, Computationally Assisted Adaptive Stroke Rehabilitation using a Mixed Reality System, In preparation, 2010
  • Nicole Lehrer, Suneth Attygalle, Jiping He, Hari Sundaram, Thanassis Rikakis, Theoretical Foundations, Representation Models and Implementation Structures for a Mixed Reality Rehabilitation System, In preparation, 2009
  • Yinpeng Chen, Hari Sundaram, Thanassis Rikakis, Computational Assessment of Movement and Functional Improvement for Adaptive Stroke Rehabilitation, In preparation, 2009
  • Margaret Duff, Yinpeng Chen, Suneth Attygalle, Janice Herman, Hari Sundaram, Gang Qian, Jiping He, Thanassis Rikakis, An Adaptive Mixed Reality Training System for Stroke Rehabilitation, In Review, 2009
  • Yinpeng Chen, Weiwe Xu, Hari Sundaram, Thanassis Rikakis, SM. Liu , A Dynamic Decision Network Framework for Online Media Adaptation in Stroke Rehabilitation, ACM Trans. On Multimedia Computing, Communications and Applications. Oct. 2008
  • Suneth Attygalle, Margaret Duff, Thanassis Rikakis, Jiping He. (2008). "Low-cost, at-home assessment system with Wii Remote based motion capture". Virtual Rehabilitation 2008, Vancouver, Canada, 2008. [second, best student paper award] PDF
  • Margaret Duff, Suneth Attygalle, Jiping He, Thanassis Rikakis. (2008). "A Portable, Low-cost Assessment Device for Reaching Times". Engineering in Medicine and Biology Conference (EMBC2008), Vancouver, Canada, 2008. PDF
  • Yinpeng Chen, Hari Sundaram, Thanassis Rikakis, Loren Olson, Todd Ingalls and Jiping He (2008), "Experiential Media Systems - The Biofeedback Project, in Multimedia Content Analysis: Theory and Applications", A. Divakaran (eds.), Springer Verlag, Oct. 2008. PDF
  • Yufei Liu and Gang Qian (2007). Projector-Camera Based Fast Environment Restoration of a Biofeedback System for Rehabilitation, Projector and Camera Systems Workshop, at IEEE Conference on Computer Vision and Pattern Recognition (CVPR'07), Minneapolis, 2007 PDF
  • Isaac Wallis, Todd Ingalls, Thanassis Rikakis, Loren Olson, Yinpeng Chen, Weiwei Xu, Hari Sundaram. "Realtime Sonification of Movement for an Immersive Stroke Rehabilitation Environment." International Conference on Auditory Display (ICAD 2007), Montreal, Canada, 2007 PDF
  • Weiwei Xu and Hari Sundaram (2007). Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (paper), ACM Press, Sep. 2007, Augsburg, Germany. PDF
  • Weiwei Xu and Hari Sundaram (2007). Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (demo paper), ACM Press, Sep. 2007, Augsburg, Germany. [best paper finalist] PDF
  • Yinpeng Chen, Weiwei Xu, Hari Sundaram, Thanassis Rikakis, Sheng-Min Liu, "Media Adaptation Framework in Biofeedback System for Stroke Patient Rehabilitation" to appear in ACM Multimedia 2007 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, Jiping He; "The Design of a Real-Time, Multimodal Biofeedback System for Stroke Patient Rehabilitation." ACM Mulitmedia 2006 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, Jiping He; "A Real-Time, Multimodal Biofeedback System For Stroke Patient Rehabilitation" (demo paper) ACM Multimedia 2006 [best demo award] PDF
  • Weiwei Xu, Yinpeng Chen, Hari Sundaram, Thanassis Rikakis, "Multimodal Archiving, Real-Time Annotation and Information Visualization in a Biofeedback System for Stroke Patient Rehabilitation.", CARPE'06, 2006 PDF
  • He Huang, Todd Ingalls, Loren Olson, Kathleen Ganley, Thanassis Rikakis, Jiping He; “Interactive, Multimodal Biofeedback System for Task-Oriented Neural Rehabilitation”; IEEE-EMBC 2005, Shanghai, China; PDF
  • Huang H, He J, Rikakis T, Ingalls T, Olson L. “A new framework of biofeedback system for neural rehabilitation.” Biomedical Engineering Society Fall meeting. 2004
  • Huang H, He J, Rikakis T, Ingalls T, Olson L. “Design of biofeedback system to assist the robot-aided movement therapy for stroke rehabilitation.” Proceeding of Society for Neuroscience's 34th Annual Meeting, 2004.