Digital Culture & AudioVisual Challenges

1. Introduction

    This research examines the capability of incorporating interactive media into the dramaturgical process of theatre storytelling. Instead of focusing on the control mechanism of the interactive system, we are more interested in how the technology contribute to delivering dramaturgical information in the play. With a focus on sound and music, it draws on cross-disciplinary research in theatrical sound design, projection design, and interactive media, and culminates in a large-scale theatrical performance entitled Pangu.

    The practical research of launching Pangu follows research method guidelines from auto-ethnography [1] model and research through design model [2]. It investigates interactive storytelling in the context of Chinese dance drama, with the design of three projection layers and surround sound system to accomplish multiple interactive storytelling approaches including sound visualisation projection and sensor-based motion tracking.

2. Motivation: Technology vs content debate

Within the realm of “digital performance”[3], according to [4], the attraction to high technology in performing arts is mythically high. However, there exists a debate in digital performance that concerning “technology versus content”. When the artists’ interest in technological experimentations surpasses how technology connects to the storytelling content of the performance, the spectators may find technology excessive over the substance of theatre art. Guardian critic [5] comments on contemporary British theatre that “I keep seeing productions in which it appears as if playing with the technologies is the prime interest of the theatre-makers [where] the technology has become the show, rather than being in service of the show.”

Despite the critique, researchers and practitioners investigate extensively on the technical domain itself, such as discussions on movement expression [6] and/or embodied sound [7], within the form of performance. Researchers elaborate how they utilise technology to create various types of interactive content[8,9,10], yet few research analyse how the content situates in the dramaturgical process of theatre making. It seems the interest of incorporating the technology comes directly from the technology itself, just as [11] points out that “there has been little inquiry into why such interaction [is] necessary other that technical novelty”.

On the other hand, academics and practitioners who are aware of this issue are encountering challenges in solving it. Martin, one developer of the Isadora software, admits that even after years of working with motion tracking technology, he still struggles to build a satisfying dramaturgically deep working process [12]. When producing the telematic performance Phone Home, although [13] was aware of this challenge from the beginning, dramaturgical discussions after the performance revealed that it still “falls prey to the ‘technology’s sake’ criticism of digital performance”. The relationship between emerging technology and content in theatre remains as a challenging and underdeveloped research area.

3. Integrated design Process and Multimodal storytelling

    We first attempted to integrate interactive media as a design element into typical theatre productions following design framework similar to other design disciplines (e.g., sound design or projection design). However, having discussed with several directors and theatre artists, it is realized that the integration of interactive media requires an innovative workflow.  Accordingly, we established a design principle Integrated Design. Compared to Integrated Process [14], our experience requires more departments, which includes set design, lighting design, projection design, sound design, and choreography, all work on certain aspects of the interaction collaboratively. At the same time, the first author works as the interactive media director to oversee and lead the integrated design process.

A design sheet was created to outline the design ideas for lighting, sound, and projections in all scenes, which turned out to be straightforward and helpful to determine the detailed design goal among all the design departments. In details, we break down the entire play as a structured list of acts, a cluster of events. Within each event, we specify interactive media design, from which we established the second design principle Multimodal Storytelling. Instead of relying on single interactive media approach to foster the narrative development, we use various approaches to create an interactive narrative sequence. Moreover, the interactive media is designed with a collaborative consideration of the set, music, lighting and choreography, so these elements alternatively take main responsibility in delivering significant storytelling information at certain scene, and collectively constitute the entire representation of the narrative on stage.

4. Conclusion

    In this abstract, we describe our investigation of the design space of interactive media in the dance drama Pangu. Through the process, we propose two design principles to serve as guidelines: Integrated design and Multimodal storytelling. We suggest to introduce a new role of the Technical Director and create a design sheet to facilitate the communication among all involved departments. We believe that multimodal storytelling where various artistic elements alternatively take the main responsibility in delivering significant storytelling information, and collectively constitute the entire representation of the story on stage can create theatre production with diverse artistic expression. We hope our exploration offers insights to researchers and practitioners working on theatre performance with interactive media.

REFERENCES

[1]   Skains, R. Lyle. 2018. ‘Creative Practice as Research: Discourse on Methodology’. Media Practice and Education 19 (1): 82–97. https://doi.org/10.1080/14682753.2017.1362175.

[2]   Zimmerman, John, Jodi Forlizzi, and Shelley Evenson. 2007. ‘Research through Design as a Method for Interaction Design Research in HCI’. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 493–502. San Jose California USA: ACM. https://doi.org/10.1145/1240624.1240704.

[3]   Dixon, Steve. 2007. Digital Performance: A History of New Media in Theater, Dance, Performance Art, and Installation. London United Kingdom: The MIT Press.

[4]   Wechsler, Robert. 2006. ‘Artistic Considerations in the Use of Motion Tracking with Live Performers: A Practical Guide’. In Performance and Technology, edited by Susan Broadhurst and Josephine Machon, 60–77. London: Palgrave Macmillan UK. https://doi.org/10.1057/9780230288157_5.

[5] Gardner, Lyn. 2008. ‘Modern Theatre Relies Too Much on Technology’. The Guardian, 17 April 2008, sec. Stage. https://www.theguardian.com/stage/theatreblog/2008/apr/17/theatretechnology.

[6]   Fdili Alaoui, Sarah. 2019. ‘Making an Interactive Dance Piece: Tensions in Integrating Technology in Art’. In Proceedings of the 2019 on Designing Interactive Systems Conference, 1195–1208. San Diego CA USA: ACM. https://doi.org/10.1145/3322276.3322289.

[7]    Peters, Deniz, Gerhard Eckel, and Andreas Dorschel. 2012. Bodily Expression in Electronic Music: Perspectives on Reclaiming Performativity. Routledge.

[8]  Cheok, A.D., Wang Weihua, Xubo Yang, S. Prince, Fong Siew Wan, M. Billinghurst, and H. Kato. 2002. ‘Interactive Theatre Experience in Embodied + Wearable Mixed Reality Space’. In Proceedings. International Symposium on Mixed and Augmented Reality, 59–317. Darmstadt, Germany: IEEE Comput. Soc. https://doi.org/10.1109/ISMAR.2002.1115073.

[9]  Honauer, Michaela, and Eva Hornecker. 2015. ‘Challenges for Creating and Staging Interactive Costumes for the Theatre Stage’. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition - C&C ’15, 13–22. Glasgow, United Kingdom: ACM Press. https://doi.org/10.1145/2757226.2757242.

[10]  Nicholas, Molly Jane, Stephanie Claudino Daffara, and Eric Paulos. 2021. ‘Expanding the Design Space for Technology-Mediated Theatre Experiences’. In Designing Interactive Systems Conference 2021, 2026–38. Virtual Event USA: ACM. https://doi.org/10.1145/3461778.3462123.

[11]    Baalman, Marije A. J., Daniel Moody-Grigsby, and Christopher L. Salter. 2007. ‘Schwelle: Sensor Augmented, Adaptive Sound Design for Live Theatrical Performance’. In Proceedings of the 7th International Conference on New Interfaces for Musical Expression - NIME ’07, 178. New York, New York: ACM Press. https://doi.org/10.1145/1279740.1279774.

[12]   Martin, Montgomery C. 2022. ‘Performing Technology: Mapping Interface Metaphors and Interactive Dramaturgies’. Toronto: University of Toronto.

[13]  Beck, Sarah. 2018. ‘Digitalising the Shared Experience: Interconnected Dramaturgy and the Role of Media in the Tri-National Performance Phone Home’. Critical Arts 32 (3): 60–76. https://doi.org/10.1080/02560046.2018.1447593.

[14] Gonzalez, Berto, Erin Carroll, and Celine Latulipe. 2012. ‘Dance-Inspired Technology, Technology-Inspired Dance’. In Proceedings of the 7th Nordic Conference on Human-Computer Interaction Making Sense Through Design - NordiCHI ’12, 398. Copenhagen, Denmark: ACM Press. https://doi.org/10.1145/2399016.2399078.Conference Name:ACM