Digital Culture & AudioVisual Challenges

Abstract
In this paper, we examine the conjunction between (1) the mythologic concepts of auditory echo and reflection, (2) their extension to visible phenomenon, (3) a new digital technology and a new interactive art.

We introduce a new concept, called “interactive simulated matter”allowing to render wide variety of echoes, reflections or more or less permanent absorbing effects in response of actions. We present 3 artistic creations experiencing this concept with a software called GravDyn, for Dynamic Engraving, developed by ACROE.

1. Introduction
In this paper, we examine the conjunction between (1) the mythologic concepts of auditory echo and reflection, (2) their extension to visible phenomenon, (3) a new digital technology and a new interactive art. We introduce a new concept, called “interactive simulated matter”allowing to render wide variety of echoes, reflections or more or less permanent absorbing effects in response of actions. We present 3 artistic creations experiencing this concept with a software called GravDyn, for Dynamic Engraving, developed by ACROE.

2. Presentation of the concept
In Greek mythology, Echo is a nymph punished by Hera who can only repeat the last words she hears. In its usual sense, echo is what is heard by a listener, facing a petrified world, which hears the last words spoken. In both cases it is a phenomenon which manifests itself in the acoustic dimension. The myth of Narcisse, mirroring himself in the water is a similar kind of phenomenon, but transposed optical phenomenon.

However, if we consider, not the point of view of the nymph, of the listener or of the viewer, but that of the world they are facing with, such perceptual phenomena are perceptual expressions of the interaction between someone and the world. Thus, by shifting from both acoustic and optical situations to a more global interaction with the world, we generalize the echo phenomenon as a kind of response of the world to our actions. Taking the case of a petrified world (a mountain for example), “echo”reflects - mimics - emitted sounds.

Suppose we can simulate this world through physical computer simulations, we will be able to generate not only acoustical echoes, but several other types of how this world reflects our actions. The world’s feedback could be acoustic but also visual and gestural and more generally multisensory. More generally, we could generalize“echoes”or reflections, not only to“imitate”our actions but to transform them.

We have called this new point of view: “real gestures on simulated matter”and we will experience how it opens up new avenues for the concepts of echo, propagations, absorbtions, etc.

3. Scientific and technical developments
We developed a software called“Dynamic Engraving”. It consists on the modeling and simulating a physical surface that exhibits dynamic behaviors such as programmable reflection, propagation, engraving, according to its physical parameters. The parameters can be changed allowing to create various degrees of such phenomena: from perfect echo and reflection, to perfect absorbsion as more or less permanent engraving. We interact gesturally with such physically-based surface, and so doing, the react of the surface is such as a dynamic answer to our actions.

4. Artistic Works
We experienced such concept and technics in three artistic visual, musical and poetic creations:
•“EtOndes..Particules”(Image 1), designed and created by Kevin Sillam, scientific PhD researcher during his doctoral internship and Melanie Lesbats, young artist during her internship, working at ACROE with Annie Luciani.
•“Somnambule”(Image 2), designed and created by Kevin Sillam, scientific PhD researcher during his doctoral internship and Noé Guirand, young composer during his studies at Music Conservatory in Grenoble.
•“Wanderings”(Image 3), created and designed by Annie Luciani, who designed the software “Dynamic Engraving”.

In“EtOndes..Particules”, we interact with a simulated physical surface, causing propagations playing like an echo of the movement of his body, felt by a camera. The simulated physical surface reacts to the body gestures by creating multiple and varied dynamic echoes more or less transformed. The process can be collective: several people can interact simultaneously with the same simulated world. Behaviors such as multiple echoes of one action can emerge generating multiple ghost echoes.

In ”Somnambule”, a pianist interacts through his musical gesture with a physically-based puppet simulation. The puppet dynamically engraves the simulated physical surface, creating dynamic traces as in more or less fluid water. The dynamic trace of the puppet also controls another - its ghosts - similar but different from the first one, also engraving the same simulated material. Such a cascade of dynamic traces of the pianistic gesture aims to represent a dream of dreams as successive echoes of an initial gesture.

In “Wanderings”, the simulated physically-based surface presents several zones, with physical parameters just a little bit different. Thus, at each frontier, the fluid propagation is more or less disrupted. By managing the physical parameters on the frontiers, echo and propagation are able to cross more or less the frontier, from total echo refection to waves propagation, and between both cases, we obtain a large panel transformed echoes, propagations, ephemeral or remanent absorbsions, etc.

5. Conclusion
Such simulated phenomena enlarge the realistic ones - acoustical echo or optical mirroring - by creating intermediate situations of propagation, echoes and permanent or ephemeral absorbsions and engraving. The concept of interactive and dynamic simulated matter illustrates the concept of heterotopia with an intimate intertwining of space and time.

CV
Annie Luciani is a researcher and a visual artist at ACROE. She developed physically-based modeling software for arts: MIMESIS for computer animation and GravDyn for physically based visualization. She headed the research and artistic works of many students and artists, among them, Kevin Sillam, PhD Student at ACROE in Art-Science-Technology at Grenoble University (France) and Mélanie Lesbats, Master in Visual Arts at EESI (Ecole Européenne Supérieure de l’Image) in Angoulême-Poitiers (France).

Hari Marini : Encountering counter-sites through live performance and video-work: The practice-based project Spirals

Abstract
Live performance is experienced in the present in front of the physical presence of an audience, as Peggy Phelan puts it:‘Performance’s only life is in the present […] Performance implicates the real through the presence of living bodies’(Phelan 1993, 146-148). Video and audio recordings capture a moment, an action, a voice, a body in time and space, and they can be repeated and re-experienced via the projection of the material. However, the deployment of digital technology in performance can challenge the temporality of performance, as well as its spatiality, and can generate meanings that expand audience experience beyond the performance space and the live act. This paper examines the encounter between live performance and digital technology, especially through video and audio recordings and projections. I focus on the multidisciplinary project Spirals (2013-ongoing) by PartSuspended, which embraces a variety of art forms (poetry, video, sound, walking, performance interventions, live art, collage, workshops), and has been filmed and performed in a variety of settings and cities in Europe, engaging female artists and poets.

The project Spirals seeks to articulate the female experience of time, movement, memory, nature and sense of belonging in a poetic and innovative way. It crosses geographical borders and unites European female voices in an exchange of languages, cultures, personal narratives and modes of expression. It employs leftover spaces in Europe where interventions and spatial performative gestures based on the symbol of the spiral are filmed; work has been filmed in London, Broadstairs, Barcelona, Belgrade, Coventry, Agost, Devon and Athens. The project consists of a series of performances, poetry, exhibitions, online incarnations, workshops, videography, recorded material, music and movement; therefore, different aspects can be analysed in diverse ways. For the purposes of this paper, I focus on the live performances of the project, and I examine the ways in which the deployment of video-poems in live performance invite audience to experience the multiplicity of time and spaces.

The video projections echo actions happening live on stage, whilst the spirals shaped in live performance reflect the spirals created in other city-spaces that become present via video projections. Through this process of reflection, the experience of space and time is multiplied. Considering Michel Foucault’s ‘heterotopia’as‘counter-sites’,‘a kind of effectively enacted utopia in which the real sites, all the other real sites that can be found within the culture, are simultaneously represented, contested, and inverted’ (Foucault 1986: 24), I argue that the experience of the space produced by the intersection of live act and recorded material create a ‘counter-site’in which spaces and time exist simultaneously and their traces act as a reminder or the world interconnectedness and the cyclical movement of time and space. The interaction of live performance and audio-visual material generate a creative intervention and invite audience to rethink and reimagine their engagement with spaces, time, texts otherwise, and opens a common space for dialogue and sharing.

The spiral is connected with time and space, challenging the idea of productive-linear time and opposing values that often govern contemporary human action and interaction. The symbol of the spiral can be found in nature and in countless ancient and contemporary artefacts, often representing evolution, transformation, rebirth, growth, lifecycles, fertility, cyclical forces and patterns of nature. It can also make reference to a movement from internal concepts and the inner self to the outer world and vice-versa. The spiral acts as a sign of becoming, transforming and awareness. Through the symbol of spiral, the project emphasises how opposite‘lands’meet; how curves challenge the straight lines and ordered environment; how time is linked to cyclical movement of exploration, migration, discovery, path and nature. Through the use of voice, sound, poetry, movement and recorded material, the project seeks to re-discover the accidental, the contingent and the improvisatory. The project’s form and qualities (dynamic, non-linear, open ended) as well asthe engagement with digital technologies produce space beyond the static and welldefined systems that endeavour to organise space. An open and ‘unfinished’space (Massey 2005: 107) that offers a great potential for imagining and acting otherwise. A space that is dynamic and fluid, open to accidental discoveries, and which produces and is produced by a multiplicity of trajectories.

1. References
Foucault, Michel. ‘Of Other Spaces’. Trans. Jay Miskowiec. Diacritics. 16.1 (1986): 22-27.
Massey, Doreen. For Space. London, Thousands Oaks, New Delhi: Sage, 2005.
Phelan, Peggy. Unmarked: The Politics of Performance. London and New York: Routledge, 1993.

Ricardo Climent : “Games4Good: at the intersections of videogaming and digital philanthropy, building new ways to give back”

Prof. Ricardo Climent (ASPECT Fellow 2021), Professor of Interactive Music at University of Manchester will discuss his spin-out company Keep•It•Human (2020), a social venture that harnesses the power of videogaming to transform society for the better. He will describe how this tool can enable charities, foundations and NOGs to reach a new generation of donors eager to own social challenges, and will conclude explaining why aligning his academic research with social goals was key to demonstrate added value.

Rui Penha : Expressive interactivity

Abstract
The conventional way of producing sound with musical instruments implies some degree of interaction between the human agent and the instrument, mostly present in the struggle to control the forces and reactions, one that is informed by an embodied knowledge of the instruments’ particular resonances and idiosyncrasies. The duality of gesture-as-movement and gesture-as-intention is therefore clearly presented in the process that transforms a physical performance gesture into the conveyed musical or sonic gesture.

Digital instruments and systems are often used to further expand the reach of these gestures, taking them beyond the physical capabilities of the performer. Traditional approaches favoured pre-conceived prosthetic gestures that were triggered during live performance, whereas the current paradigm is mostly based on the digital expansion of the regular capabilities of acoustic instruments. Whilst this often implies a decoupling of the aforementioned duality ―favouring the musical or sonic gesture over the physical performance ―, performers can still embody this expansion in order to exert and display a given degree of control over the musical outcome.

But what happens when the interactive capabilities of a digital system increase and the performer can no longer reasonably expect to fully control the outcome of his or her actions? Is the expressivity of the system compromised or are we facing a new kind of expressive potential? Who are the agents behind that expression? How can interactive musical systems expand our current notions of musical expressivity and musical agency? What does that bring for the composer, the sound designer, the performer or the audience? These are some of the questions that will be addressed in this talk, based on the implications that different concepts of interactivity have in our understanding of interactive music.

CV
Composer, media artist, and performer of electroacoustic music, Rui Penha was born in Porto in 1981. He completed his PhD in Music (Composition) at the University of Aveiro. His music is regularly recorded and played in festivals and concert halls around Europe and North America, by musicians such as Arditti Quartet, Peter Evans, Remix Ensemble, or the Gulbenkian Orchestra. He was a founder and curator of Digitópia (Casa da Música, Porto) and has a deep interest on the relationship between music and its technology.

His recent production includes interfaces for musical expression, sound spatialisation software, interactive installations, musical robots, autonomous improvisers, and educational software. More recently, Rui has focused his attention on the problems of defining and guiding artistic research. He taught at several Portuguese institutions, in both music, art and engineering faculties, and is currently an assistant professor at ESMAE and researcher at CESEM. More info at http://ruipenha.pt

Pavlos Antoniadis, Aurélien Duval, Jean-François Jégo, Makis Solomos, Frédéric Bevilacqua : Merging symbolic, physical and virtual spaces: Augmented reality for Iannis Xenakis’ Evryali for piano

Abstract
The proposed paper will present interactive systems for the visualisation and optimisation of extreme scorebased piano performance. The systems are founded on an ecological theory of embodied interaction with complex piano notation, under the title embodied navigation (Antoniadis, 2018a; Antoniadis and Chemero, 2020). The theory has materialised in a modular, sensor-based environment for the analysis, processing and real-time control of notation through multimodal recordings, called GesTCom (Antoniadis, 2018b; Antoniadis and Bevilacqua, 2016). The motion capture modeling is based on an one-shot learning Hidden Markov Model developed at Ircam and called Gesture Follower (Bevilacqua et al., 2010). At a later stage, mixed reality applications have been developed on the basis of existent visualisation methodologies for motion capture (Jégo, Meyrueis and Boutet, 2019), seeking to create a virtual concert environment. Drawing on music performance analysis, embodied cognition, movement modeling and augmented reality, we consider the concert experience as embodied navigation of performers and listeners in a hybrid environment. This environment capitalises on the isomorphisms and decouplings of physical, virtual and symbolic spaces, which merge in static and dynamic relationships: the performer’s gesture shapes music notation, music notation becomes an integral part of the concert space, a virtual avatar of the performer allows the audience to experience multimodal aspects of the performance which usually remain private, and so on. The main focus of this presentation will be on a recent performance of Iannis Xenakis’solo piano work Evryali employing live motion capture and augmented reality1. This particular work problematises usual notions of virtuosity and performability, bears extra-musical references and is encoded in a unique graphic design. These features justify the task’s characterisation as extreme and demand a rethinking of technology-enhanced performance that combines sensorimotor learning, symbolic interpretation and multimodal feedback in novel ways.

1. References:
Antoniadis, Pavlos (2018a). Embodied Navigation of Complex Piano Notation: Rethinking Musical Interaction From A Performer’s Perspective, PhD thesis. Strasbourg: Université de Strasbourg
–IRCAM, 2018. http://theses.unistra.fr/ori-oai-search/notice/view/2018STRAC007, accessed 14.04.2021

Preliminary documentation of this project may be found in the following link: https://youtu.be/D-vhOX88NfM

Antoniadis, Pavlos and Chemero, Anthony (2020). “Playing without mental representations: embodied navigation and the GesTCom as a case study for radical embodied cognition in piano performance”, in the Journal of Interdisciplinary Music Studies, special issue“Embodiment in Music”following CIM19 conference in Graz, Austria (eds. Andrea Schiavio and Nikki Moran), season 2020, volume 10, art. #20101207, pp. 126-174. http://musicstudies.org/wp-content/uploads/2021/01/10Antoniadis_Chemero.pdf, accessed 14.04.2021

Antoniadis, Pavlos (2018b). “GesTCom: A sensor-based environment for the analysis, processing and real-time control of complex piano notation through multimodal recordings”. Invited talk at Séminaires Recherche et Technologie, IRCAM, 15.10.2018. https://medias.ircam.fr/x2253e1, accessed 14.04.2021

Antoniadis, Pavlos and Bevilacqua, Frédéric (2016). “Processing of symbolic music notation via multimodal performance data: Ferneyhough’s Lemma-Icon-Epigram for solo piano, phase 1”. Proceedings of the TENOR 2016 conference, 127-136. Cambridge: Anglia Ruskin University 2016. http://tenor2016.tenor-conference.org/TENOR2016-Proceedings.pdf, accessed 14.04.2021

Bevilacqua, F., Zamborlin, B., Sypniewski, A., Schnell, N., Guedy, F., and Rasamimanana, N. (2010). “Continuous realtime gesture following and recognition”. In Lecture Notes on Computer Science, Gesture Workshop, pages 73–84. Springer.

Jégo, Jean-François, Meyrueis, Vincent, and Boutet, Dominique (2019). “A Workflow for Real-time Visualization and Data Analysis of Gesture using Motion Capture”. In: Proceedings of the 6th

CV
Dr Pavlos Antoniadis (PhD University of Strasbourg-IRCAM, MA University of California, San Diego, MA University of Athens) is a pianist, musicologist and technologist from Korydallos, Athens, Greece, currently based in France. He performs complex contemporary and experimental music, studies embodied cognition and develops tools for technology-enhanced learning & performance. He is currently researcher at EUR -ArTeC, Paris 8 and following up at the Berlin Institute of Technology (TU-Audiokommunikation) as a Humboldt Stiftung scholar. He collaborates with the Interaction-Son-Musique-Mouvement team at IRCAM and he is a member of the Laboratory of Excellence GREAM, Université de Strasbourg, where he also taught seminars on computer music and contemporary performance practice.