Audience Amplified: Virtual Audiences in Asynchronously Performed AR Theater

You-Jin Kim, Misha Sra, and Tobias Höllerer:
Audience Amplified: Virtual Audiences in Asynchronously Performed AR Theater. In: IEEE International Symposium on Mixed and Augmented Reality, IEEE ISMAR 2024
Presented: IEEE ISMAR 2024 - [Avatar 2 Session] October 24, 2024, Bellevue, Greater Seattle area
▪️  DOI: 10.1109/ISMAR62088.2024.00062    ▪️  arXiv: 10.48550/arXiv.2511.02807

Abstract

Audience reactions can considerably enhance live experiences; conversely, in anytime-anywhere augmented reality (AR) experiences, large crowds of people might not always be available to congregate. To get closer to simulating live events with large audiences, we created a mobile AR experience where users can wander around naturally and engage in AR theater with virtual audiences trained from real audiences using imitation learning. This allows us to carefully capture the essence of human imperfections and behavior in artificial intelligence (AI) audiences. The result is a novel mobile AR experience in which solitary AR users experience an augmented performance in a physical space with a virtual audience. Virtual dancers emerge from the surroundings, accompanied by a digitally simulated audience, to provide a community experience akin to immersive theater. In a pilot study, simulated human avatars were vastly preferred over just audience audio commentary. We subsequently engaged 20 participants as attendees of an AR dance performance, comparing a no-audience condition with a simulated audience of six onlookers. Through questionnaires and experience reports, we investigated user reactions and behavior. Our results demonstrate that the presence of virtual audience members caused attendees to perceive the performance as a social experience with increased interest and involvement in the event. On the other hand, for some attendees, the dance performances without the virtual audience evoked a stronger positive sentiment.

Research Contributions

• DNN AI avatar agents can replicate human interaction behaviors for specified physical spaces through imitation learning.

• Player height impacts immersion, presence, and enjoyment, requiring body-conscious calibration in immersive play design.

• AI-trained virtual audiences support live concert-like experiences, enabling "always-ready" social experiments in AR.

• This provides a proof-of-concept for capturing and reliving historical, human-involved events through the deployment of DNN avatar audiences.

• Research empirically proves that the deployment of DNN virtual audiences can increase social involvement and the sense of a live event.

Citation IEEE Format

[1] Y-J. Kim, M. Sra, and T. Höllerer, "Audience amplified: Virtual audiences in asynchronously performed AR theater," in 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2024, pp. 475–484, doi: 10.1109/ISMAR62088.2024.00062. (9 Pages)

Citation APA Format

Kim, Y-J., Sra, M., & Höllerer, T. (2024). Audience amplified: Virtual audiences in asynchronously performed AR theater. 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 475–484. https://doi.org/10.1109/ISMAR62088.2024.00062. (9 Pages)