Digital Culture & AudioVisual Challenges

Sound in cinema constitutes a means of non-linguistic narration, which serves the aesthetic needs of a film and complements the visual and linguistic narration (Dalianoudi 2021: 283-309). The representations that result from the interconnection of image and sound create fixed and predetermined musical and cultural associations which contribute to a film’s overall meaning. In other words, cinema provides an intertextual context in which visual, verbal and auditory narratives interact and are transformed into cultural and emotional associations. As for music especially, its primary function is to provide emotional support to the story (Kivy, 1997: 322), while the whole interplay between speech, image and sound/music becomes artistically expressive (Monaco, 1981).

Yet, not all spectators have access to the auditory narrative and to the interplay between the visual, the verbal and the auditory narratives given that increasingly many people around the world –466 million people according to 2022 WHO data– have hearing problems or are d/Deaf. People who are deaf or hard of hearing identify themselves with the mainstream hearing society, while Deaf people are members of the linguistic and cultural minority of the Deaf community (Zárate, 2021: 21). However, medical and sociological reports (Gouge 1990; Darrow 1993) prove that the world of the d/Deaf is all but silent. In fact, music is for them far more important than it is given credit for (Neves, 2010: 123). The experience d/Deaf people have when “feeling” music is similar to the experience other people have when hearing music, since both musical vibrations (perceived by the d/Deaf), and sound are processed in the same part of the brain (Shibata, 2001). Furthermore, music “plays an important role in landmarking significant experiences and spaces in people’s lives”, since songs and melodies “have a shared life which goes beyond their compositional existence” (Neves, 2010: 124). It becomes clear that it is crucial for all those working in the sphere of accessibility services for d/Deaf people –and subtitling for the d/Deaf and the hard of hearing (SDH) in particular– to pay special attention to the conveyance of music. Yet, conveying/transferring non-verbal auditory elements, like sound effects and paralanguage (i.e., intonation, accent, and information about who is speaking, e.g. an identified character, or whether the voice is male or female) and above all music pose many challenges to subtitlers.

As Neves (2010: 125) points out, in order to proceed with the linguistic transfer of acoustic messages, translators have to be “sensitive to the interplay between images, speech, sound and music to be able to decode their inherent messages and to find adequate and expressive solutions to convey such sensations verbally” and ultimately create subtitles that “serve the purpose of the acoustic component of the audiovisual text in all its effects”, yet in a way that is meaningful to people who might have never perceived sound before. Naturally, this balance is hard to achieve. Translating contextually occurring sound and music into written language will demand transcoding expertise that will pull the translator between the intended meaning of the acoustic messages, their function in the text and the effect any rendering may produce on the deaf viewer, while bearing in mind the spatiotemporal constraints of subtitling and the need to also convey the spoken messages. As it is expected, guidelines and codes of good practice around the world contain recommendations on how to convey sound effects and music in the form of subtitles. Yet, actual practice shows that even when the topic is explicitly addressed on paper, the actual presentation of information on screen is less systematic, while there are very few reception studies that shed light into the way d/Deaf viewers perceive the transcoding of music and sound into subtitles and into the way the transcoding affects their immersion or enjoyment (Revuelta et al., 2020).

The present study seeks to address this shortcoming by adopting a novel approach to SDH whereby two subtitlers, a music expert and a deaf viewer work together to create the set of subtitles including the transcoding of music and sound and by then conducting a reception study to establish how different viewers, i.e., hard of hearing, deaf and Deaf receive the subtitles and react to the transcoding used. The film chosen for the study is “Dunkirk”, a 2017 history film directed by Christopher Nolan, which has more music and sound (by Hans Zimmer) than speech, and thus the verbal subtitling conveys both the diegetic sounds (environmental sounds, alarms, cries usually in battle fields, heard by the actors) and the non-diegetic music (the music itself, intended only for the audience)[ Gorbman, 1980: 185-202. Genette, 2007. Δαλιανούδη, 2021, σελ. 293-4 ] while taking into account the different genres of music: classical music which merges into electracoustic music and conventional instruments which co-exist with other non-conventional sound sources. The findings are used to produce best practices and inform the subtitling process with a view to optimising SDH and meeting the needs of the intended audience.

References
Dalianoudi, R. (2021) “History-Music-Film. From the national epos of 1821 to the musical epos of commercial cinema” [In Greek Ιστορία-Μουσική-Κινηματογράφος. Από την εθνική εποποιία του 1821 στη μουσική εποποιία του εμπορικού κινηματογράφου]. Proceedings of the 10th International Conference of the Holy Synod of the Church of Greece The impact of 1821 on literature and art, 18-20 November 2020, Athens, Greece: Archontariki, p. 283-309.
Genette, Gérard (2007), Σχήματα ΙΙΙ. Ο λόγος της αφήγησης. Δοκίμιο μεθοδολογίας, μτφρ. Μπάμπης Λυκούδης, Αθήνα: Πατάκης.
Gorbman, Claudia (1980), “Narrative Film Music”, Yale French Studies 60, Cinema/ Sound, pp. 183-203.
Darrow, A. A. (1993) “The role of music in deaf culture: Implications for music education”, Journal of Research in Music Education, 41(2), 93-110.
Gouge, P. (1990) “Music and profoundly deaf students”. British Journal of Music Education, 7(3), 279-281.
Kivy, P. (1997) “Music in the movies: A philosophical enquiry”. In R. Allen and M. Smith (eds) Film Theory and Philosophy. Oxford: Claredon Press, 308-328.
Monaco, J. (1981) How to Read a Film. The Art, Technology, Language, History, and Theory of Film and Media. New York and Oxford: Oxford University Press.
Revuelta P., Ortiz T., Lucía M.J., Ruiz B. and Sánchez-Pena J.M. (2020) “Limitations of Standard Accessible Captioning of Sounds and Music for Deaf and Hard of Hearing People: An EEG Study”. Frontiers in Integrative Neuroscience. 14:1. doi: 10.3389/fnint.2020.00001
Shibata, D. (2001), “Brains of deaf people rewire to ‘hear; music”. https://www.sciencedaily.com/releases/2001/11/011128035455.htm [last accessed 15/3/2023]
Zárate, S. (2021) Captioning and Subtitling for d/Deaf and Hard of Hearing Audiences. UCL Press. https://doi.org/10.2307/j.ctv14t478b