Goals

The primary goal of this project is the development of a multimedia based system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching and grasping tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results.

biofeedback system

Impact

The development of a portable immersive multimodal environment will reduce rehabilitation time, promote more extensive recovery, alleviate rehabilitation monotony and also be used in the home.

Challenges

Activation of conscious sensorimotor integration during the therapy promotes neural plasticity for recovery of motor and cognitive function, especially in neural trauma patients, such as those with stroke and spinal cord injury

The effectiveness of inducing neural plasticity for functional recovery from any therapeutic system is based upon the active participation of the patient. This consideration is critical for repetitive exercise type of therapy because it is a challenge for the subjects to maintain attentive and motivated during a long and tedious session and they easily become physically and mentally tired. Furthermore, conscious sensorimotor integration requires participation and coordination of multitude sensory systems in addition to the motor systems and necessitates a system that holds attention through engagement of the subject. As Noe [6] argues, perception is not something that happens to us, but is something that we do.

recovery of physical space

Recent research indicates that the use of interactive feedback in conjunction with movement therapy has a significant impact on the functional recovery of stroke patients with sensorimotor deficits. However, there are some significant limitations affecting the outcome and validation of existing studies: (1) Most of the systems used relied on unimodal feedback (such as using sound alone), (2) these systems are not adaptive (e.g. it is not possible to change the therapy, based on an specific individual’s abilities and rate of progress) and can not perform complex, model-based evaluation in real time, and (3) the test population was small.

Current project activities

We are extending prior work and our current research by addressing these shortcomings through the ongoing development of multimodal systems in which individual feedback streams are fused to create a unified environment.

The multimodal sensing of the patients actions is performed through an optical motion capture system, an EMG system and a video camera array. Sensing data is synchronized and archived. We are working on improving sensing and sensor fusion algorithms through modeling of arm and hand kinematics. User-context models are used for real-time and off-line evaluation of key parameters indicative of the patient’s performance. Integrated visual and auditory feedback is used to provide the patient with real time performance evaluation. As stroke survivals often have cognitive deficits in addition to motor deficits, we address the issue of “information overload” by carefully coupling feedback mechanisms with apposite tasks and through gradual, evaluation-based, introduction of feedback streams. We are making the system engaging through individualizing the environment based on user context models and preferences, making the system adaptive to meet individual performance measures and by adding story telling, game like, paths that help maintain interest.

biofeedback system

Results and validation

Preliminary results of a pilot study involving ten stroke patients demonstrate the potential of the system to improve patients’ reaching performance. We will be seeking validation for this approach through long term trials with a large (~50) number of subjects. This study will be done in conjunction with Dr. Mark K. Lyons, M.D. and Dr. Joseph I. Sirven, M.D. from the Mayo Clinic, Scottsdale; Dr. Richard Herman, M.D. at Banner Good Samaritan, Phoenix and several physical and occupational therapists.

Project Publications

  • He Huang, Todd Ingalls, Loren Olson, Kathleen Ganley, Thanassis Rikakis, Jiping He; “Interactive, Multimodal Biofeedback System for Task-Oriented Neural Rehabilitation”; IEEE-EMBC 2005, Shanghai, China; PDF
  • Huang H, He J, Rikakis T, Ingalls T, Olson L. “A new framework of biofeedback system for neural rehabilitation.” Biomedical Engineering Society Fall meeting. 2004
  • Huang H, He J, Rikakis T, Ingalls T, Olson L. “Design of biofeedback system to assist the robot-aided movement therapy for stroke rehabilitation.” Proceeding of Society for Neuroscience's 34th Annual Meeting, 2004.
  • Weiwei Xu, Yinpeng Chen, Hari Sundaram, Thanassis Rikakis, "Multimodal Archiving, Real-Time Annotation and Information Visualization in a Biofeedback System for Stroke Patient Rehabilitation." PDF
  • Yufei Liu and Gang Qian (2007). Projector-Camera Based Fast Environment Restoration of a Biofeedback System for Rehabilitation, Projector and Camera Systems Workshop, at IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07), Minneapolis, 2007
  • Isaac Wallis, Todd Ingalls, Thanassis Rikakis, Loren Olson, Yinpeng Chen, Weiwei Xu, Hari Sundarum. "Realtime Sonification of Movement for an Immersive Stroke Rehabilitation Environment." International Conference on Auditory Display (ICAD 2007), Montreal, Canada, 2007.
  • Weiwei Xu and Hari Sundaram (2007). Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (paper), ACM Press, Sep. 2007, Augsburg, Germany. PDF
  • Weiwei Xu and Hari Sundaram (2007). Information Dense Summaries for Review of Patient Performance in Biofeedback Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia (demo paper), ACM Press, Sep. 2007, Augsburg, Germany.PDF
  • Yinpeng Chen, Weiwei Xu, Hari Sundaram, Thanassis Rikakis, Sheng-Min Liu, "Media Adaptation Framework in Biofeedback System for Stroke Patient Rehabilitation" to appear in ACM Multimedia 2007 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, Jiping He; "The Design of a Real-Time, Multimodal Biofeedback System for Stroke Patient Rehabilitation." ACM Mulitmedia 2006 PDF
  • Yinpeng Chen, He Huang, Weiwei Xu, Richard Isaac Wallis, Hari Sundaram, Thanassis Rikakis, Todd Ingalls, Loren Olson, Jiping He; "A Real-Time, Multimodal Biofeedback System For Stroke Patient Rehabilitation" (demo paper) ACM Multimedia 2006 [best demo award]

Future Work

We are currently extending our work in the following directions

  • validation of system through long term trials with large (~30) of subjects
  • modeling of arm and hand kinematics
  • task/exercise scenarios for better integration of reaching and grasping
  • development of sophisticated user context models
  • enrichment of types and scenarios of feedback
  • generalized models for optimized correlation of feedback modes to task parameter (i.e. contribution of music structures on temporal features of movement)
  • story telling structures in biofeedback for increased engagement
  • sensing through video based portable system for home or clinic use

References

  1. D. Jack, R. Boian, et al. (2001). Virtual reality-enhanced stroke rehabilitation. IEEE Trans Neural Syst Rehabil Eng 9: 308-318.
  2. M. L. Dombovy (2004). Understanding stroke recovery and rehabilitation: current and emerging approaches. Curr Neurol Neurosci Rep 2004 4(1): 31-35.
  3. E. Todorov, R. S. E., et al. (1997.). Augmented Feedback Presented in a Virtual Environment Accelerates Learning of a Difficult Motor Task. J Mot Behav 29: 147-158.
  4. H. Woldag, G. Waldmann, et al. (2003). Is the repetitive training of complex hand and arm movements beneficial for motor recovery in stroke patients? Clin Rehabil 2003 Nov 17(7): 723-730.
  5. L. Zhang, B. C. Abreu, et al. (2003). A virtual reality environment for evaluation of a daily living skill in brain injury rehabilitation: reliability and validity. Arch Phys Med Rehabil 84: 1118-1124.
  6. A. Noë (2004).Action in perception. Cambridge, Mass., MIT Press.
  7. H. Sveistrup (2004). Motor rehabilitation using virtual reality. J Neuroengineering Rehabil 1: 10.