Sensing

Multiple types of electronic sensors record key aspects of human activity and the physical world.

• Movement Sensing and Analysis conducts research, ranging from multimodal movement sensing and fusion, to movement modeling, multi-level extraction, tracking and recognition of movement features. [Gang Qian]
• Audio Sensing obtains a complete picture of the sonic environment using microphone arrays. Sub themes include tracking/separation of multiple sources, event segmentation and semantic information retrieval. [Harvey Thornburg and Andreas Spanias]
• Physiological Sensing researches the design and use of technologies ranging from electromyography and pressure-sensitive floors to affective systems, leveraging them within multimodal-sensing frameworks to create new computing interfaces. [Winslow Burleson, David Birchfield and Jiping He]

Perception and Modeling

Computational modeling allows the extraction of structure and meaning from sensed activity and the optimization of ecological coupling between users and systems.

• Perception and Cognition focuses on scientific theories and behavioral methods for understanding how people experience and interact with their environment. [Ellen Campana, Michael McBeath and Clark Presson]
• Modeling and Fusion explores knowledge representation, multi-sensory context and mathematical models for multimodal fusion. [Hari Sundaram, Harvey Thornburg and Gang Qian]
• Annotation and Archiving develops frameworks for the storage, retrieval and summarization of real-time multimodal data as well as frameworks for collaborative annotation within a social network. [Hari Sundaram and Aisling Kelliher]

Interaction and Feedback

Dynamic, multimodal systems are developed for sensitizing, engaging and connecting users to the experiential media environment.

Interaction Architectures leverages the sensing of activity, context and affect, along with scalable media management to design and promote optimal interactions for each user’s and system’s goals. [Winslow Burleson]

Visual Feedback researches and develops responsive and adaptive visual interfaces, employing technologies such as programmable LEDs, mobile devices, projection surfaces and mounted screens. [Aisling Kelliher, Loren Olson and Mary Bates Neubauer]

• Audio Feedback develops interactive and generative sound models that are context-sensitive, user-adaptive and integrate principles from computer music composition, interactive performance, sound anthropology and music cognition. [David Birchfield , Todd Ingalls and Thanassis Rikakis]
• Interactive Movement utilizes movement and physical sensing, Laban Movement Analysis principles and dance composition tools to allow for meaningful, movement-based, human-computer interaction. [Jodi James]

Experiential Construction and Knowledge Creation

Formal frameworks are developed for creating compelling, user-oriented, physical-digital experiences. They seamlessly integrate sensing, perception, modeling, feedback and formalized procedures for knowledge understanding. [All AME faculty]