Abstract:

Ode to My Frayed Nerves is a composition for technology, extended vocal techniques such as overtone singing and growling, and gesture. The work sonically explores the sensations that occur in my left hand due to a physical trauma. This paper discusses how these sensations are realised sonically, implementation of the MiMu system (MI.MU Gloves Ltd, 2010) and accompanying performance gestures, and how it has shaped the creative output and immersion in the activity and sound environment. The piece explores how a physical trauma may be expressed through an immersive sound performance.

The exploration is grounded in embodiment studies, which discuss the intertwining perceptions of the brain, body, and objects in actions, and performative autoethnography to document the interactions. Preliminary conclusions yield that the glove-based gestural sound manipulation of voice can encourage or enable an embodied experience, but it can be fleeting due to sensor drift, rigorous preparation demands, and performance variabilities. However, the ability to paint and malleate a sound environment without physically touching a computer is powerful and absorbing activity.

1 Introduction

Ode to My Frayed Nerves (Rose, 2022) is a composition for technology, extended vocal techniques such as overtone singing and growling, and gesture. The work sonically explores the sensations that occur in my left hand due to a physical trauma. This paper discusses how these sensations are realised sonically, implementation of the MiMu system (MI.MU Gloves Ltd, 2010) and accompanying performance gestures, and how it has shaped the creative output and immersion in the activity and sound environment.

The work was composed for my doctoral thesis which investigates the bridge between voice and technology through embodiment, which in simple terms is the intertwining perceptions of the brain, body, and objects in actions. The compositions for the larger body of work centre on traumatic experience(s) and how they may be expressed and (re)negotiated through sound, voice, and gesture. The aim for the larger study is to enhance the connection between the mind and body and exploit the psychological feedback loop that this creates in composition and performance.

2 Framing

The piece explores how a physical trauma may be expressed through an immersive sound performance. The hands are the site of the injury, and the hands shape the vocal output through looping and effects. The MiMu interface is hand-based and thus aligns with the focus on the hands. The near universal possession of hand(s) gives the potential for empathetic connections with the artwork due to the hands’ sensitive structure and intertwinement with the human experience. Languageless voice is used because in traumatic situations the function of Broca’s and Wernicke’s areas (which control speech production and comprehension (Kolk, 2014; Levine, 2015)) are impaired. By utilising the internal experience of my nerve damage, I simulate an experience based on a cumulation of first-person experienced data and then express it externally using voice.

Practice-based research methods fuel this project and other research methods are used to support and develop the given problem, be it aesthetic, or technical (Candy, 2006; Sullivan, 2009). As with the work of Höök (2018) and Mainsbridge (2018), this exploration is rooted in embodiment studies and attempts to holistically deepen the user’s engagement in a technologically mediated activity. I frame the work through an performative autoethnographic, and therefore also écriture féminine, lens to document the interactions and assist in fleshing out the academic canon to normalise and diversify non-masculine perspectives and experiences (Spry, 2011; Stojanović, 2015).

Making is an essential part of practice and research as an artist (Mäkelä, 2007). Research in the arts melds practice with abstraction of knowledge and meaning contained in the art and/or the process of making. Donald Schön (1983) proposed that research ought to be geared towards an understanding of the nature and origin of knowledge, which is tied to the practice from knowing through making. The artist, or practitioner, learns and understands the world through the artefacts they create and the process of making. In exposing one’s trauma for an audience and including the bodily knowledge within both the practice and performance of a work I operate in performative autoethnography. The body, the music, and the writing are linked aspects of knowledge. The inclusion of implicit knowledge and explication of personal experiences is analysed through society in a critical narrative.

Embodiment studies discuss the intertwining perceptions of the brain, body, and objects in actions. The inclusion of bodily knowledge is essential to experientially focused artwork and experimentation. If an experience is the cumulation of sensory data resolving into a unity, then we must include past experiences in our exploration in the present and perception of an experience. Dewey likened experiencing to breathing in the rhythmic intake and output and conscious experience requires engagement with the body in combination with this rhythmic process and fulfilment of an aesthetic event (Dewey, 1934). In this discussion I look at the interplay of unified mind-body absorption in the activity, technology, and the artwork.

3 Slice

Seemingly small traumas may have diverse and long-lasting effects. They may be far deeper and more significant than even a medical professional could realise, though seeming superficial.

3.1 Storytime

I was already in shock when I got to the doctors last June. I had been cleaning up my electronics workstation and had slipped catastrophically with the utility knife. I remember cleaning. I remember having it in my hand. And then I was looking down at blood pouring out of my hand. I called to Cloud (partner) and said I need to go to the doctors. Now. I made Cloud put my mask on me while I tried to hold the skin together and put pressure on the area. All the while blood continued to seep between my fingers.

The doctor rushed me in, cleaned the external wound, and injected my hand with an anesthetic. I did not feel the needle. This should have been a clue. The doctor proceeded to clean out the wound and then held his breath and went still. He asked me to move my hand, and then each of my fingers. I had missed the tendon by around a millimeter. Functioning seemed normal, so he continued cleaning. At one point the tugging of the cleaning process caused an electric shock through my hand and I gasped and jerked. The doctor stitched me up and sent me on my way.

I could not use my hand for about a month. Over time I discovered that I could no longer feel most of my pinky finger and I had a pins and needles sensations on the outer edge of my palm above the cut. To this day I have loss of sensation in parts of my hand. I no longer feel the texture of things, just a confusing mishmash of nerve static that refers to unlikely places. When the area is touched, electrical signals fire in my hand in the wrong places. If the touch is unexpected, I go momentarily blind, in that I can only “see” the sensation. My pinky finger is weaker, easily fatigued, and no longer as dexterous. It’s worse on hot days when the body retains water (almost impossibly so given the sweltering Melbourne heat). Most of the time it is merely annoying, but when the cat licked my scar... It was akin to old science fiction movies and warp speed, but with less hyper-speed travel and no crew to strap in with.

3.2 Make It Sound

Ode to My Frayed Nerves¹ (Rose, 2022) uses the technologically mediated female voice and extended vocal techniques drawn from contemporary, experimental, and non-Western influences and manipulate my voice with a glove-based gestural control interface (MiMu). This piece can be performed by other singers, however, due to the inaccessibility and inflexible sizing of gesture-based wearable controllers, I am the sole performer currently. Gesture data is fed through proprietary software called Glover (MI.MU Gloves Ltd, 2022) and into Ableton Live (Ableton, 2021) to process the voice in a variety of ways, including looping, distorting, granularising, and spatialising around a quadraphonic, binaural, or other surround sound format immersive sound environment. I wrote, recorded, and performed the work in a quadraphonic environment using the Max for Live addon Surround Panner (Ableton, 2020) with four channels of audio sent through the auxiliary channels, thus the audio format configuration is flexible to suit different venues. Hand gestures manipulate the audio and gestural relationships with sound were selected through a mixture of practicality and symbolism. For example, the fist gesture grabs a sound and horizontal arm movements change its position in the sonic environment. This draws on the well understood concept of grabbing and moving something with one’s hand.

The piece aims to reflect the current physical situation of my hand injury and intends to convey a simplified version of the journey from normal functioning to injury. The piece does this primarily through the vocal line which builds by layering live vocal loops which ascend chromatically. There are 14 total loops. At the start, only the live acoustic voice is audible. An automated filter on all audio output gradually opens the audible frequencies over time from inaudible (100Hz for most female voices) to audible frequencies. As the pitch rises overtones and undertones, and other solo multiphonic singing techniques are incorporated. A pre-recorded voice sings a G with a tongue trill and functions as a timekeeping device. This sound can be heard around the five, ten, 15, and 20 minute marks. The pitches stack chromatically with specified extended vocal techniques as shown in Table 1. With the addition of effects, the sound becomes a wall of noise over time due to the dense layering of processed vocals.

After 14 minutes, the filter opens past 10,000Hz and the audio quality is degraded through fuzz and grain delay effects, becoming a fuzzy noise, like dialing into television static. Closed fist hand gestures “pick up” the sound which can then be moved, effected, and placed in the quadraphonic audio environment. A button cycles through which fist gesture is controlling what vocal loop. This control provides instability and unpredictability, as well as a sense of movement and being surrounded through the sound moving around the speaker setup. I use this to allude to the physical damage which my body has tried to knit back together and failed. The threads go together in unexpected ways and the body does not mend correctly. Further gesture routing is contained in Table 2.

After the loops have been recorded, there is space for improvisation. First by affecting the sound through the gesture and placing sounds in the quadraphonic environment. Secondly through two melodic phrases (rhythm is at the discretion of the performer) using harmonic minor and double harmonic scale in any comfortable range, with and without overtone singing. The phrases are 1, 3, b2, 1/8, b6, 5, 7, 8/1 and 8, 5, b7, 7, 4, 3, 8, b9.

The larger context of this piece is ways of expressing trauma, which is a deeply personal experience, to an audience in a way that is evocative and honest. It is also to demonstrate the affordances of technology to enhance and extend the voice and express instances of trauma where language does not ordinarily preside or take precedence.

3.3 Assigning Meaning

For this work, Ableton Live, MiMu, and Glover capture and interpret data to create the electronic sound components. The MiMu gloves use flexible sensors, a button, accelerometer, gyroscope, and magnetometer sensor to capture movement data and transmits data over WIFI where it is translated by Glover and output to one's digital audio workstation of choice. It can provide user feedback through LED’s and/or vibration feedback on the gloves.

The overarching filter is automated with several steps in the steepness of the curve going from a low pass filter from 200Hz to 20,000Hz. There are seven scenes, and so fourteen possible loops. The loops pair in twos, so 1 & 2, 3 & 4, et cetera. The performer records one loop at a time and then places it within the surround (or binaural, for headphone performance) sound environment by the spatial location of the hand. See Table 2 for basic gesture routing.

Easily remembered gestures that are pre-laden with meaning have been chosen for preference. This makes it easier for the performer to switch between different gesture interfaces with similar abilities or configuration and communicate the function of a gesture to the audience. Some gestures are commonplace, or conducting gestures, for example, volume up/down is taken from conducting gestures (it may also be seen in education settings for shushing excited children). Other assignments were more challenging, for example, loop start/stop. These mapping assignments have a personal logic, but it is more contrived. The logical thought for my practice is: “I have an idea!” (index finger pointing upward) to “Stop” (open palm + up).

The yaw (side to side on a flat plane), pitch (up/down as an angle in relation to the floor), and roll (side to side around the front to back axis of the microcontroller) movement whilst holding a closed fist hand position manipulates audio effects and spatialization of each loop. This grants a tactile and visual representation of grabbing a sound to twist and shape it. The manipulation and effects routing leave room for improvisation within a set piece, enabling free flow between choreographed and improvised sections, to allow the performer freedom to immerse themselves in the sound.

4 Theatre versus “Real Life”

Thembi Soddell (2019) notes that trauma is felt well after the event is in the past, often without a direct connection to the initial event. This piece is an artistic interpretation of the sensations that arise when an area of the body with nerve damage is touched, not specifically of the accident (the slicing and dicing, see Storytime). The narrative is the physical healing journey; the progression from no feeling to confused, sporadic feelings, swelling, and random electrical noise circulating through the area. This pitch splitting of the voice through extended techniques and the automated filter are used to reference the split skin and the frayed nerves. Over time, the wall of noise diminishes and there are small gaps where only the live acoustic voice is audible. And as the final swelling recedes, the electrical noise dims and silences. The hands tell the story, which is apt given the hands are the site of the injury. As a musician and maker, my hands are strongly connected to my identity as a person and what I do as an artist. It is also interesting from the perspective of healing. We ordinarily think of healing as being a journey from broken or ill, to “back to normal”. With trauma that may not be possible, and instead an unfair expectation on the experiencer. Even considering the incredible plasticity of the brain, sometimes the best a traumatized person might hope for is to craft a life that is satisfying whilst embracing the chaos to find peace.

Placing the sound in space by the act of grasping by a closing of the fist is something that can be easily understood by an audience. Movements need to be performed in a more exaggerated way to ensure the message is clear for the audience. Having one neutral position (where the sound was not being actively altered) was an important lesson to learn as a composer-performer. My neutral position was open palm any position except “open palm + up”. This neutral stance provides space for me to listen to the soundscape and locate the next compositional input and/or prepare the next fixed sequence. Without a neutral position, every movement must be deliberate as all movement changes the sound. A lack of neutral posture hinders the likelihood of the performer becoming immersed in the activity which obstructs feelings of embodiment.

We do not yet have a generalized performance gesture language or ingrained understanding of how movement follows sound in this performance context (Mainsbridge, 2016). Additionally, as Jane Davidson notes there are three streams of movement used in performance – biomechanical, individual, and culturally-specific (Davidson & Salgado Correia, 2001). A performer may wish to draw from all or any of these categories to frame their performance. Layman experience with (technologically mediated) gestural-sound illustration is based in the same sets of movements seen in representations in films like Frozen (Buck et al., 2013), or perhaps shamanist rituals as shown in documentaries where the sound accompanies the gestures evoking natural phenomena or animals. In Frozen, the main character Elsa’s magic comes out of her hands and is accompanied by sound in soundtrack and as foley. While the use of technology can expand our expressive capabilities, there may be no general framework to decipher, and solutions may be artist-by-artist centric. In the meantime, there remains a mystic quality for an audience based on audience feedback I have received.

5 Evaluating the interface for purpose

In some ways the system is very user friendly and in others it can be an extra chore to stop you from composing. The price of the system makes it inaccessible for most practitioners and inhibited me from experimenting vigorously for fear of damaging the interface. The gloves must be connected to a digital audio workstation, such as Ableton Live, through the proprietary software, Glover, which generates MIDI (Musical Instrument Digital Interface) continuous controls (CCs) that can be assigned in another applications. It can also send data as OSC (Open Sound Control), text-to-speech, and send feedback to the gloves. The glove data needs to be calibrated in each session to reset the zero positions and configure the flex sensor finger data.

Surprisingly, familiarity with another similar system (self-built) did not meaningfully make the MiMu system easier to master. Mary Mainsbridge (2016) noted that gestural interface design favours technical and functional accomplishments over the engagement with the activity and development of kinaesthetic sensibility which are important in composition and performance. This demands that the artists have highly refined proprioceptive skills and body awareness, like dancers.

There is some drift in the IMU (Inertial Measurement Unit) data making it unpredictable. This can be remedied through a zeroing gesture to reconfigure the mid-point of the gloves, but that requires spare gesture slots. Gestures sometimes get mis-recognised with other gestures (for example, “puppet hand” and fist) and multiple gestures can trigger when transitioning between gestures. There is no accelerometer/gyroscope machine learning built into the software, thus requiring additional software and processing which can be CPU intensive. The proxy for three-dimensional gesture capture in Glover (for example, a wave) requires you to visually snapshot positions within the whole of that movement. It can make it harder to write precisely due to these factors and requires careful planning in routing the parameters and boundaries. There is also a small delay in gesture recognition which makes it hard to take shorter, rhythmic vocal loops and have precise control over effects. Encountering technical errors mid-performance was not uncommon and would drag me out of any embodied state during practice.

Gestures cannot be triggered without using the interface and it is a time-consuming process. In this piece, there was much planning prior to gesture assignment and during practices technical challenges would arise and generate barriers to embodied states. I used a relatively simple set of gestures to balance between functionality of performance and memory capabilities so that I could remain engaged in the sound control. The laboriousness of routing data does not contribute to the engagement with the activity and makes it more difficult to get to the composing stage. It creates many niggling problems to overcome before one can control the sound. Assuming that one is looking to facilitate embodiment and interaction with the activity, this presents a problem.

However, once this process is complete, the interface does enable embodiment. The physical act of grasping sound and moving it through the speaker arrangement is tactile and it makes the invisible seem malleable. Through this process, play and performer agency are introduced and/or heightened. Additionally, it is a novel and playful way of interacting with one’s voice. A singer is not ordinarily able to point in a direction to change the panning or raise a fist to add distortion at will. After the initial loop recordings, there is room for improvisation which leads the performer back into a space of play and agency. This is space to react and interact with the sound in the allotted time. The yaw, pitch, and roll movement whilst holding a closed fist hand position manipulates audio effects and spatialization of each loop. This grants a tactile and visual representation of grabbing a sound to twist and shape it. The manipulation and effects routing leave room for improvisation within a set piece, enabling flow between choreographed and free sections, to allow the performer freedom to immerse themselves in the sound.

The role in expressions of trauma is emerging. So many actions of psychological trauma are full body postures (slumping, crumpling up, becoming small). Physical trauma differs by incident site. This interface would require Max (Cycling ’74, 2018), MuBu (IRCAM, 2014), and additional custom sensors to accurately capture more psychologically instigated gestures, and/or further exploration of the bounds of the MiMu system.

Trauma is deeply personal, but also shared and understandable. It is a multiplicity. Furthermore, songs may have different meanings per audient. The artist statement and title provide the context for the audience through the power of suggestion – which they can still choose to reject. An alternate reading of the piece might be mental turmoil (an alternate interpretation offered to me by a listener). The piece might have a similar impact, but the psychological context of that art would change. As a personal exploration of trauma, how I engage with the sound is informed by my narrative and what I know of that traumatic event. The confusion of electrical signals in the body, uncertainty and unpredictability through moving sound – being the personification of the trauma as a way of externalising it. But this is not a type of trauma that needs to be reintegrated into myself to exist in the past – it lives with me and will die with me. More work will be done here in the larger thesis project.

6 Preliminary conclusions

These conclusions were drawn through analysis and review of audio-visual recordings taken during composition and practice sessions, written notes about alterations and perceptions during performance, progression of the compositions form from inception and transference to the Glover-MiMu system (from my self-built system, as part of the technical analysis), and changes made to the piece and/or the data routing during practice and composition sessions. They were analysed for evidence of embodiment (where it was facilitated or hindered) and how they reflected the autoethnographic narrative (physical trauma and its aftereffects).

The Glover-MiMu system has been effective and useable. It is a convenient way of managing unwieldy sets of gestural data, though it lacks some creature comforts for ease of routing. The system is costly reducing its’ accessibility to a range of artists. A reduced user base also reduces feedback and refinement on the system. One might be better served by building their own WIFI-enabled OSC gloves to work within Glover’s infrastructure. The cost in the alternate solution is shifted to time.

Does this type of interface and routing encourage or enable an embodied experience? Yes, but it can be fleeting because of IMU drift, rigorous preparation demands, and reworking the gesture sets repeatedly while balancing artistic desire and real-life functionality. Technical issues interrupt the embodied experience. However, when working as expected, the ability to paint and malleate a sound environment without physically touching a computer is powerful and an absorbing activity. Grabbing, twisting, and twirling (sound) objects operates at a childlike level of enjoyment. It gives the performer agency when the controls are easy to decipher, code, and track in performance. In this piece, the most challenging part is orienting and remaining cognizant of what sound is which loop and position it is out of the 14 possible loops.

Challenges arise in capturing the full body experience of a traumatic event because the system uses only gloves. Psychological trauma manifests through the whole body and may include a full movement (like crumpling to the floor), not just the hands. Though the title of this piece could have an alternate meaning, the effect of a soundscape that exhibits elements of a “wall of noise”, electricity, human sounds manipulated to become inhuman or superhuman can translate effectively under the subject of frayed nerves as a mental or physical phenomenon. Gestural language, though, is tricky and elusive as it rubs up against an impossible goal of universality. Shared trauma, individual trauma, what is human, and the practicalities of capturing these difficult human experiences; these are the conundrums informing future works in the larger thesis project.

7 Future Research

Future works will address more confronting topics than the mild maiming of my left hand. These topics include sexual assault and harassment, gender discrimination, and the breakdown of body and mind when enduring long-term traumatic experiences. These pieces will incorporate trauma-informed therapeutic concepts as I externalize my voice and mold it with my hands. This could potentially thread into therapeutic uses such as music therapy and/or Somatic Experiencing (Payne et al., 2015). More research needs to be conducted here.

Dramatizing a physical injury (although disturbing to describe) is a “safe” and non-confrontational way to begin engaging with trauma related material. Most people have had a papercut; there is a relatability through widely experienced events. Any person that has had dental trouble might understand how awful nerve pain and pain referral can be. People engage with art to engage with difficult ideas and for catharsis (Bicknell, 2014). If you then make them aware that this event could jeopardize their livelihood, they could feel empathy. This could potentially build empathy in audiences over a series of works or build familiarity with one’s inner sensations (as in Somatic Experiencing) over time leading to greater empathy between individuals, societies, and self. Artists who suffer from unseen chronic illnesses (depression, chronic fatigue, fibromyalgia, et cetera) or other disabilities could use gestural control and their own experience to articulate their daily life, experiences, struggles, and triumphs to elucidate their own experience. So, intelligent, aware, and hopeful artists might look to these areas to make meaningful change in society on a larger scale.

8 References

Ableton. (2020). Surround Panner (Ableton 11) [MacOS]. Ableton. https://www.ableton.com/en/packs/surround-panner/

Ableton. (2021). Ableton Live 11 (Version 11) [MacOS]. Ableton.

Bicknell, J. (2014, September 29). 4 Reasons We Listen to Sad Music, When We’re Sad. Psychology Today, Why Music Moves Us(Online). http://www.psychologytoday.com/blog/why-music-moves-us/201409/4-reasons-we-listen-sad-music-when-were-sad

Buck, C., Lee, J., Bell, K., Menzel, I., & Groff, J. (2013, December 6). Frozen [Animation, Adventure, Comedy]. Walt Disney Animation Studios, Walt Disney Pictures.

Candy, L. (2006). Practice Based Research: A Guide (CCS No. 2006-V1.0 November; Creativity & Cognition Studios, p. 19). University of Technology.

Cycling ’74. (2018). Max MSP (Version 8) [MacOS]. Cycling ’74.

Davidson, J. W., & Salgado Correia, J. (2001). Meaningful musical performance: A bodily experience. Research Studies in Music Education, 17(1), 70–83. https://doi.org/10.1177/1321103X010170011301

Dewey, J. (1934). Art as Experience. Capricorn Books. https://sites.evergreen.edu/danceasart/wp-content/uploads/sites/124/2015/09/Art-as-Experience-ch.1.pdf

Höök, K. (2018). Designing with the Body: Somaesthetic Interaction Design. MIT Press. https://doi.org/10.7551/mitpress/11481.001.0001

IRCAM. (2014). MuBu. IRCAM. http://ismm.ircam.fr/mubu/

Kolk, B. van der. (2014). The Body Keeps the Score: Mind, Brain and Body in the Transformation of Trauma. Penguin UK.

Levine, P. A. (2015). Trauma and Memory: Brain and Body in a Search for the Living Past: A Practical Guide for Understanding and Working with Traumatic Memory. North Atlantic Books.

Mainsbridge, M. (2016). Body as instrument: An exploration of gestural interface design [Thesis]. https://opus.lib.uts.edu.au/handle/10453/43474

Mainsbridge, M. (2018). Gestural systems for the voice: Performance approaches and repertoire. Digital Creativity, 29(4), 315–331. https://doi.org/10.1080/14626268.2018.1541181

Mäkelä, M. (2007). Knowing Through Making: The Role of the Artefact in Practice-led Research. Knowledge, Technology & Policy, 20(3), 157–163. https://doi.org/10.1007/s12130-007-9028-2

MI.MU Gloves Ltd. (2010). MiMu [Control Interface]. https://mimugloves.com/

MI.MU Gloves Ltd. (2022). Glover (1.1.1) [MacOS]. MI.MU Gloves Ltd.

Payne, P., Levine, P. A., & Crane-Godreau, M. A. (2015). Somatic experiencing: Using interoception and proprioception as core elements of trauma therapy. Frontiers in Psychology, 6. https://doi.org/10.3389/fpsyg.2015.00093

Rose, S. (2022). Ode to My Frayed Nerves (installation, score, and rehearsal footage). Zenodo. https://doi.org/10.5281/zenodo.7856220

Schön, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action. Routledge. https://doi.org/10.4324/9781315237473

Soddell, T. (2019). A dense mass of indecipherable fear: The experiential (non)narration of trauma and madness through Acousmatic sound. https://researchbank.rmit.edu.au/view/rmit:162786

Spry, T. (2011). Body, Paper, Stage: Writing and Performing Autoethnography. Routledge. http://ebookcentral.proquest.com/lib/unimelb/detail.action?docID=843865

Stojanović, D. (2015). The inscription of the feminine body in the field of sound: Vocal expression as a platform of feminine writing (écriture féminine). Muzikologija, 18, 115–129. https://doi.org/10.2298/MUZ1518115S

Sullivan, G. L. (2009). Making space: The purpose and place of practice-led research. Practice-Led Research, Research-Led Practice in the Creative Arts, 41–65.