Digital Culture & AudioVisual Challenges

Dance is a field that has experimented with technology quiet a lot in the past decades. Even since the 90’s choreographers used motion capture techniques not just for presenting a performance, but as part of the inspiration and creation procedure(Downie 2005). Coming to our days, we can still find plenty of choreographers that keep on finding ways to connect dance with digital technology and that is something to be expected, if we take into consideration the common ground between the way digital artists and choreographers think and work (Downie 2005). Nevertheless, a closer look on these cases, show limited use of Mixed Reality techniques. Digital content is limited in graphics and background visuals, while digital characters, when present, usually follow predefined actions. Moreover, these cases are more popular in fields such as education or movement rehabilitation, while performance arts are still hesitant in using such content. Real-time motion capture in performances is rarely used, and in most of the cases as a visuals generator(Palacio and Bisig 2017). The use of digital AR/MR content, that is being created during the performance is, still, kept outside of the theatrical stage.

This paper studies the design and implementation of a mixed reality dance performance and its impact on the audience. Specifically, we have prepared a performance where motion capture techniques are used not just for creating visuals, as part of the set on stage, but also as part of the creation process, with the digital content acting as the dancer’s partner. Gerridae is an interactive dance and acrobatics performance for a single dancer. Looking at the visual part of the performance, we distinguish two acts, based on the way the digital content is created and presented. On the first act, we see the dancer with a video background, performing dance and aerial acrobatics acts. On the second act, the audience watches a choreography made for a human and digital dancer, with the digital dancer’s avatar changing form, from a humanoid avatar, to the dancers spiritual animal and finally to an abstract body of glowing particles

The video, on the first act, is made using a Kinect camera, creating visual feedback from the dancers move, with either graphics or live video feed. On the second act, the live dancer is leading the digital character not into a prerecorded dance sequence, but instead the avatar follows the choreography of the dancer as it is captured by the Kinect camera and translated on Unity3D on the fly. Besides the visual content, the dancer’s moves also control the audio part of the performance. More specifically the dancer’s hands and feet position act as digital potentiometers, changing the volume balance of 4 different audio channels. The result is a completely different soundscape on every performance, allowing the dancer to keep control of the final soundtrack at the same time.

This performance is a first experiment over the audience satisfaction between two types of interactive performances. On the first part of the performance, we examine the use of motion capture for creating background visuals. On the second part, motion capture is used for creating 3D MR content, such as a digital dancer that does not follow a prerecorded dance, but creates its movements live by following the choreography performed. The performance is also part of a research on the ways that digital art can communicate with the dancer and become the medium through which a performer is connected to the space around him on stage, and eventually to the audience. After the performance, the audience is interviewed, to evaluate its satisfaction, and compare the MR parts of the performance to the parts with prerecorded video in the background. Moreover, part of the audience is equipped with a wearable device that keeps track of the heartbeat and the GSR. The data collected is also examined, searching for a possible connection between the audience satisfaction and the visual stimulation, comparing two situations, with and without the use of MR content.

One of the most important aspects when creating an interactive dance piece, is to determine the connection between motion capture and the digital content created (El Raheb et al. 2019). For that reason, the results of this experiment will be used for preparing the next experiment on interactive MR dance performances, where the focus will be on the dancer and the choreographer, researching on the ways that the dancer’s movement is connected to the digital content creation.


Bibliography

Beira, João, Rodrigo Carvalho, and Sebastian Kox. 2013. “Mixed Reality Immersive Design: A Study in Interactive Dance.” ImmersiveMe 2013 - Proceedings of the 2nd International Workshop on Immersive Media Experiences, Co-Located with ACM Multimedia 2013, no. May: 45–50. https://doi.org/10.1145/2512142.2512147.

Bokaris, Panagiotis-Alexandros, Michele Gouiffès, Véronique Caye, Jean-Marc Chomaz, and Christian Jacquemin. 2020. “Gardien Du Temple: An Interactive Installation Involving Poetry, Performance and Spatial Augmented Reality.” Leonardo 53 (1): 31–37. https://doi.org/10.1162/leon_a_01569.

Clay, Alexis, Gael Domenger, Julien Conan, Axel Domenger, and Nadine Couture. 2014. “Integrating Augmented Reality to Enhance Expression, Interaction & Collaboration in Live Performances: A Ballet Dance Case Study.” ISMAR 2014 - IEEE International Symposium on Mixed and Augmented Reality - Media, Arts, Social Science, Humanities and Design 2014, Proceedings, no. September: 21–29. https://doi.org/10.1109/ISMAR-AMH.2014.6935434.

Downie, Marc. 2005. “Choreographing the Extended Agent: Performance Graphics for Dance Theater.”

Palacio, Pablo, and Daniel Bisig. 2017. “Piano & Dancer - Interaction between a Dancer and an Acoustic Instrument.” ACM International Conference Proceeding Series Part F1291. https://doi.org/10.1145/3077981.3078052.

Raheb, Katerina El, Marina Stergiou, Akrivi Katifori, and Yannis Ioannidis. 2019. “Dance Interactive Learning Systems: A Study on Interaction Workflow and Teaching Approaches.” ACM Computing Surveys 52 (3). https://doi.org/10.1145/3323335.