Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration

You-Jin Kim, Myungin Lee, JoAnn Kuchera-Morin, and Tobias Höllerer:

Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration. In: ACM Conference on Human Factors in Computing Systems, ACM CHI 2024 Interactivity

Presented: ACM CHI 2024 - [Interactivity Sessions] May 11–16, 2024, Honolulu, Hawaiʻi

▪️  DOI: 10.1145/3613905.3648659    ▪️  arXiv: 10.48550/arXiv.2510.23848 

Abstract

Spatial Orchestra demonstrates how easy it is to play musical instruments using basic input like natural locomotion, which is accessible to most. Unlike many musical instruments, our work allows individuals of all skill levels to effortlessly create music by walking into virtual bubbles. Our Augmented Reality experience involves interacting with ever-shifting sound bubbles that the user engages with by stepping into color-coded bubbles within the assigned area using a standalone AR headset. Each bubble corresponds to a cello note, and omits sound from the center of the bubble, and lets the user hear and express in spatial audio, effectively transforming participants into musicians. This interactive element enables users to explore the intersection of spatial awareness, musical rhythm that extends to bodily expression through playful movements and dance-like gestures within the bubble-filled environment. This unique experience illuminates the intricate relationship between spatial awareness and the art of musical performance.

Research Contributions

• Maps human gait to audio synthesis, allowing users to achieve musical expression and creation through intuitive physical movement and walking.

• Implements a dual-sided volumetric rendering shader to maintain visual clarity and spatial orientation whether the user is inside or outside.

• Establishes safety and accessibility guidelines for diverse mobility needs, specifically supporting wheelchair users and children in immersive environments.

• Proves that interactive spatial objects increase natural walking distances for patients in therapeutic and rehabilitation settings that require walking.

• Explores how participants reconcile physical bodily movement with localized audio cues to trigger spontaneous rhythmic and artistic expressions.

Citation IEEE Format

[1] Y-J. Kim, M. Lee, M. Peljhan, J. Kuchera-Morin, and T. Höllerer, "Spatial Orchestra: Locomotion Music Instruments through Spatial Exploration," in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2024, Art. no. 420, doi: 10.1145/3613905.3648659. (5 Pages)

Citation APA Format

Kim, Y-J., Lee, M., Peljhan, M., Kuchera-Morin, J., & Höllerer, T. (2024). Spatial orchestra: Locomotion music instruments through spatial exploration. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Article 420. https://doi.org/10.1145/3613905.3648659. (5 Pages)