Living lens: Negotiating relationships between the performing body, image and sound as real time dance and audiovisual creation

Introduction

In using responsive technologies in live performance, it is essential that the artists involved in the production negotiate an understanding of the relationship between the performing bodies and the audiovisual media—if not the actual inner workings of how performer’s movements are mapped to sound and/or visual representations, or how sounds are mapped to visual elements and/or vice versa—then at least a discovery of a plausible or meaningful connection. The question thus arises for artists and others developing these kinds of environments, as to whether concerns such as complexity, subtlety, transparency or coherence, are of central interest in a work.

A further concern of this paper is the relationship between the organic and the mediated, an important focus in the development of the authors’ work, Living Lens (2005 – ongoing). In opening up these issues, this paper presents the views of several artists working in the field, as well as first-hand perspectives of creative participants involved in the Living Lens project.

The issue of perceptual coherence

In performances incorporating responsive relationships between body movement and virtual components in realtime interactive systems, there is a need for the artists involved to negotiate the nature and meaning of these relationships. Of particular relevance here is Palindrome Inter-media Performance Group’s notion of ‘gestural coherence’, described as "the perceptual coherence between sound and the movement that generates it" (Rovan et al., 2001). Although gestural and sonic coherence is here being specified, the notion equally applies to coherence between body movement and visual media 1 . This issue of perceptibility is thus a pertinent one, particularly in the case of Living Lens where visual images are not virtual bodies in the sense of anthropomorphic or human-like entities, but rather form an abstract tapestry-like environment where any interactive relationship may be more difficult to discern.

Within a broader context of performer-media relationships, it may be of interest to extend the notion of perceptual coherence beyond performances using real time responsive systems, to works that employ digital technologies such as motion capture 2 in the pre-production phase of the work. A notable example is the 1999 piece entitled Biped, a collaboration between dancer-choreographer Merce Cunningham and digital artists Shelley Eshkar and Paul Kaiser.

In this piece, movements of the dancers were first motion-captured; this information was then translated as hand-drawn humanoid and abstract animations and later projected onto a large transparent scrim across the proscenium in front of the dancers. The scrim created a shifting realm of live bodies and their doubled, virtual selves, the dancers seemingly materialising from booths located at the rear of the stage. In an essay on Biped, Kaiser (2000) states it was important that the connection between the dancers and their virtual counterparts remain credible, hence they

took care never to lose the underlying perception of real and plausible human movement. A case in point: when our stick figure leaped, its various lines were flung upward in the air, then gathered back together on landing. While no human body could do this, you could still feel the human motion underlying the abstraction. (Kaiser, 2000)

Although at times the animations transformed into abstract dots and lines, there was a perceived need to maintain anthropomorphic or dancer-like aspects of these virtual bodies in order for the connection to be understood. The transparent scrim played a crucial role in the illusion of dancers and visual projections co-inhabiting three-dimensional space yet the piece was still framed in a conventional proscenium arch environment.

The contrast and challenge in Living Lens is that it is located as an installation in an all-round viewing environment, where the visual imagery is more diffuse and organic, rather than a proscenium stage with seated auditorium that sites performers against a more clearly identifiable backdrop. Moreover, the concern is not with generating a virtual mirror of the performers’ bodies, but with representing the textural quality of their movement.

Responsive media: instrumental relationship between organic and electronic

Troika Ranch, a dance theatre company directed by Dawn Stoppiello and Mark Coniglio, is an example of a group using real time interactive technologies within proscenium stage and installation contexts to forge intimate relationships between performers and the visual and sonic media. Ongoing research into human gesture and the potential for interactive manipulation and control has resulted in the development of original sensory devices, including MidiDancer and the graphic programming environment, Isadora 3 . On their website, Troika Ranch state their aim is for media elements to possess the vitality and live-ness of the actual dancers themselves, yet claiming that the "linkage of body and technology highlights the uneasy relationship between the organic and the electronic" (Troika Ranch, n.d.), a central theme in their work. In an interview on the subject of mapping body gesture into electronic media (in Saggini, 2002), Coniglio uses a biological metaphor to explain this relationship:

In the past I have referred to our technology as ‘parasitic’, in the sense that it feeds off of the live performer, and cannot survive without the host. But this parasitic relationship goes both directions: do we modify the movement of a dancer who is being monitored to satisfy the needs of the technology that is reacting to her movement? The answer is yes, but, and I feel this is important, not unconsciously. We try to be hyper-aware of the ways in which we are accommodating the technology that we choose to make use of, so that these limitations inform the piece itself.

While these comments certainly convey a sense of unease in the relationship between performers and electronic media, there is also an impression of the dancer learning to ‘play’ the interactive components as an extension of her/his own body movements. In this way, the relationship takes on instrumental characteristics, with dancers making improvisational choices and on-the-spot decisions rather than following a preset choreographic composition.

A specific concern for Troika Ranch is the extent audience members are aware of the instrumental nature of this practice (Broadhurst, 2007, p. 120). In a web-based video clip describing the interactive technology used in the work, 16[R]evolutions (2006), Troika Ranch note that, "Our research has focused on the quality of a gesture, meaning we wanted to quantify parameters that relate closely to the viewer’s experience of a gesture" 4 http://www.youtube.com/watch?v=Rbv7n0ZgA98]]. Using MidiDancer or Eyesweb 5 http://www.infomus.org/]] movement pathways from multiple points of a performer’s limbs and trunk are analysed to measure parameters such as velocity and acceleration, straightness and curvature and convert it into computer data, which is then used to generate visual images and control aspects of the sonic score.

This complex process enables performers’ gestures to influence 3D visual imagery, such as encircling breathing ribcages and ribbon-like strands of DNA, to convey the idea of evolutionary change. Website commentary on 16[R]evolutions claims the objective is for the virtual elements to be ‘more “animal” than the characters onstage’ (Troika Ranch, n.d.). For Troika Ranch, therefore, it is in the projected imagery taking on greater attributes of the ‘animal’ or organic, indeed becoming larger than the life-sized figures of the actual dancers, that connections to the gestural actions of the dancers are made more explicit.

Although the specific focus of Troika Ranch is on the so-called uneasy linkage of bodies and technology, what piques our interest is the idea of the visual media assuming greater characteristics of the organic. In Living Lens, organic metaphors and textures and have been at the forefront of the creative process. Troika Ranch’s approach, therefore, resonates with our own exploration into more seamless connections between organic and mediated contexts.

Yet the focus in Living Lens is not as complex as Troika Ranch’s systems in terms of data mapping, for only occasionally is data measured from a single point on a performer’s body, or the tracking of a sonic speaker source is related in a more ambient way to a performer’s spatial positioning.  By comparison, in work such as Biped and in various works of Troika Ranch, emphasis is placed on generating virtual, skeletal-type structures created through multiple tracking and complex movement analysis to more intimately resemble human form or human motion. In Living Lens, conversely, no complex motion analysis is being undertaken. Rather, a more minimal mapping process is used to generate imagery that favours a sense of organic life processes, articulated as images of water, plants and abstract textures.

Image surface as virtual ‘skin’

As mentioned, it has been our approach in Living Lens that the visual and/or sonic media only intermittently become responsive to performers’ movement or positioning, either via motion sensors worn on a specific part of the body such as the wrist or upper back, or by tracking directional movement in the space to reflect ultrasonic beams from a performer’s body (further commentary on these areas is detailed later in this paper). The work has involved an iterative development process with intensive month-long creative developments and performance outcomes in 2005 and 2006, with further investigation currently taking place during 2008 – 09.

From its inception in 2005, Living Lens has developed around metaphors of organic life processes through a focus on microscopic forms and sensations, the forces of growth and decay, and in the transformation of one movement, visual image or sonic phase to the next. The process has also involved the creation of an original visual media system called the XV series 6 in response to specific aims, including the layering of multiple textures, the capacity for continuous projection across multiple surfaces and the rendering of three-dimensional graphic objects on the fly. Visual images were programmed to move continuously across a double-sided S-shaped projection surface, and in this way, the screen was conceived as a kind of a virtual skin or body. Through floating, fluid oscillations between surface textures and illusory spatial depths, the projected imagery appeared to breathe on the surface. By multiple overlaying of images in real time, layers were added and removed like skins, concealing or revealing innermost depths. The performers, as actual living bodies, further endowed the installation with kinaesthetic, sculptural qualities.

Figure 1: Living Lens—performers located against the organic imagery of the double-sided screen surface. Photo: Fiona Cullen

Extended touch: nerves and particle flows

Through the interplay of virtual and physical bodies and skins, a tactile sensibility emerges. Rodaway, in his book Sensuous Geographies (1994) refers to a spatial and temporal orientation of sense experience that he describes as a "multisensual and multidimensional situatedness in space and in relationship to places" (p. 4). Through what he refers to as "haptic geographies" (p. 41), he specifies four touch ranges (p. 53): "global touch" as the sense of the body in contact with an environment, for example, a surface, texture, pressure, or temperature; "reach touch" as the body stretching and exploring space; "imagined touch" as tactile experience embedded in past experiences or expectations; and "extended touch" as touch mediated by a tool.

Motion sensors, as tools or devices, are thus an example of extended touch, amplifying a performer’s movement as graphic representations on an illusory tactile surface. Our interest, however, is not only in the image surface being responsive via motion sensors, but that shifts in visual textures may also reflect a more poetic sense of a performer’s "reach touch" or "imagined touch". Accordingly, an understanding of tactility is here expanded to include invisible connections between the body and the surrounding space.

To activate this tactile sensibility in more tangible terms, the expression "poetic felt space" was coined as an intuitive frame of reference to describe the haptic dimensions of the installation space on the whole. 7 To develop awareness of the tactility of movement sensations, methods adapted from training in the Japanese dance-theatre form butoh include the tracing of nerves throughout the body and into the surrounding environment. 8

With the body envisaged as "a gigantic kingdom of nerves" (Waguri, 1998), performers imagine extending nerves, for example, from the fingertips to trace grains of wood in a distant door, then cracking tiny whips at the very ends of these nerves. This tactic provides a means for imagining invisible lines, with the highly tactile fingertips perceived as antennae extending from the body, an evocative mode for sensing a connection to the projected imagery. In this way, too, performers imagine invisible, resonating connections not only to the projection surface, but also to other bodies and points in space.

The idea of nerves extending and retracting through the body was used in connection with the ‘graining’ method of Alwin Nikolais (Nikolais & Louis, 2005). In this approach, the idea is to direct imaginary particles of varying density, within the body towards a particular focus point. This can either be from one part of the body to another; for example, from the hip to the hand, or stream outwards from the body to points beyond.

Within this process, one can also alter the textures of particles; for example, from the looseness of sand to the more viscous consistency of honey. Thus these two notions—extension of nerves and particle flows—were a means for performers to focus on tactile experiences and directional movement flows as the receiving and projection of both kinaesthetic and imaginary textural sensations to the external environment.

Perceptual coherence: participant views

Directional flow: ripples and brushstrokes

In terms of the performers interactively influencing the visual projections, directional movement flow became the focus of one Living Lens performer, Ko-Pei Lin, who through an optional-use motion sensor worn on her wrist, could activate two layers of the visual imagery: one layer of watery ripples and another layer comprising a brushstroke effect. Whilst the connection between body gesture and ripples may be perceived from the initial hand movement, the relationship becomes increasingly ambiguous, for the fluid nature of water, even in its virtual computer-generated form, makes it difficult to ascertain whether the performer’s movements are the source of successive ripples. For the brushstroke layer, on the other hand, when Ko-Pei Lin twirled her wrist in expansive curves, an amorphous white ribbon effect became the literal visualisation of the graining method.

From the perspective of body movement, Ko-Pei Lin (interview, March 10, 2006) found that although she could evolve her movement possibilities, it was difficult to delineate a pathway through the installation because it entailed negotiating several layers: not only the visual layers activated via the motion sensor, but her role or character in the piece on the whole. She found that one or other of these layers would somehow block the progression of her pathway. In working with the motion sensor however, she felt that an actual link between her body and visual images created a challenging sense of "being both in control and controlled". Specifically, she felt that the brushstroke effect created a tangible extension of her body movement, yet, as the location of the brushstroke on the screen often seemed unpredictable, she experienced a dimension to the relationship beyond her control. This, however, she found to be a point of interest rather than a problem.

Figure 2: Wearing a motion sensor on her wrist, performer Ko-Pei Lin generates realtime ripples and brushstroke effects in the projected imagery. Photo: Ian Hutson

‘Body resonance’: Nerves as pulses and oscillations

The focus on the body as a resonating system of nerves was the approach adopted for Elise May, fellow collaborator and performer in Living Lens. This was directly inspired by the method of butoh dancer, Yumiko Yoshioka, who describes her work as ‘body resonance’, an interaction between the interior of the body and the external environment, where the

world, including our body and soul, consists of vibrational waves that create constant resonances like echoes. When we tune our body to a certain frequency, we consequently get a resonance, and according to the frequency, we get different resonances (Fraleigh & Nakamura, 2006, p. 120)

To focus on vibrating sensations of the proximal part of the body, Elise May wore the motion sensor on her upper back—in contrast to Ko-Pei Lin who wore the motion sensor on her wrist or distal part of her body. In Elise May’s case, spasmodic movements of the spinal area between the scapulae were mapped to three visual projection layers. Two of these layers represented the idea of neural activity. One layer consisted of a white pulse representing a synaptic signal, the image flickering according to speed of movement.

The second layer was a cube, textured with branching ‘nerve’ fibres that expanded or contracted with horizontal and vertical movements of the performer. A third layer, representing the idea of seismogram, consisted of an oscillating waveform that fluctuated in response to horizontal movements. Due to the complexity of these multiple layers and effects, we found that the relation between body movement and visual images was not easily perceptible. Our main intention, however, had been for an overall disturbed atmosphere where jerky spasms of the body create almost subliminal alterations in the different textural layers of the visual projections, which was further accentuated by rhythmic and pulsating sound effects. In hindsight, May describes the effect of this approach (email correspondence, 10 March 2008):

As a performer the sheer effect of standing in a room with the large-scaled visual projected imagery for me evoked a kinaesthetic and bodily response, effected greatly by the diversity of images and textures that were used thoughout the development of the Living Lens.

Notwithstanding these reflective comments on the overall experience, May (interview, March 8, 2006) felt the performer’s role in using the motion sensors was unclear. As she claimed, she was not sure of whether to improvise freely with visual effects responding to her movements in an ad hoc manner, or to develop a specific set of choreographed movements to generate predetermined visual representations. The following comments (email correspondence, 10 March, 2008) convey the issues that the work raised for her as a performer:

With the addition of image, sound and interactivity between the performer and image or sound, there are many more considerations for the performer. As I began to explore the characteristics (and rules of operation) of the interactive device I began to understand that ‘success’ in producing a desired effect with the visual media did not necessarily equal ‘success’ in terms of movement intention. This raised a series of questions.... What is the purpose of the interactivity? Does the audience need to recognise the interactivity to fully appreciate its effect? Are the interactive elements then for the benefit of the audience, the performer or both? Does the success of the interactivity between perfomer/operator determine movement creation or vice versa? Is there a heirarchy of elements?

May’s comments are thought provoking, particularly in light of Mark Coniglio’s claim cited earlier in this paper that Troika Ranch, for their part, do modify the movement of a dancer to suit to the needs of their technology, whilst remaining highly conscious of this fact in their work. In relation to our work, however, visual media artist and co-author, Tetsutoshi Tabata (interview, April 2008), finds that the motive for using motion sensors is not so much that audience members need to understand connections to dancers’ movements, or, that these movements should fit the needs of the technology per se. Rather, the basis for their use is to expand the range and subtlety of the visual response.

In our earlier works, the media artists generated real time visual effects at the control booth via computer mouse or tracking pad, with a subsequent limited gauge of effects. Conversely, visual effects generated by movements of a dancer via motion sensors, comprise a wider scope of detail and nuance. For example, Ko-Pei Lin’s training in classical Chinese dance meant that movements of her wrist were delicate and refined, something that could not be replicated by the operators at the booth. For Tetsutoshi Tabata, therefore, it is essential that the dancer be aware of the instrumental nature of responsive media and hence, a perception of a more synergistic relationship: if the dancer does not view it this way, then s/he may feel a hierarchical ordering in terms of one element being subordinated to or privileged over another.

Sonic spatialisation: directional and immersive frequencies

A consideration of interdependent, shared relationships between performing bodies and audiovisual media within the context of an installation performance space, needs to include a focus on the sonic dimension, in terms of the way performing bodies and the surrounding acoustic environment affect one another. In the case of Living Lens, development occurred through dialogue between the participants, as well as by observing and listening to what was emerging collectively in the space. As Living Lens sound designer, Luke Lickfold, states in relation to sound (e-mail correspondence, October 20, 2006), his particular method was to

…find a goal, an emotional or conceptual destination inspired by the dance or visuals in any particular scene, and then use whatever sounds, processing and spatialisation to achieve this goal… most of the time there was an evolving, organic background layer to the sound, with elements forming and dissolving in the foreground as punctuation, often synchronised with significant moments in the movement and visuals.

The sonic layering involved real time manipulation through ascending and descending waves of sound, rather than "a piling up of bricks of sound to erect edifices of varying styles" (Takemitsu et al., 1995, p. 17). The approach thus involved an adaptable, evolving palette of sound textures to abstractly suggest spaces and realms, for example, a deep forest, the vault-like interior of a cave, or the inner realms of the body evoked through pulses or ‘firings’ in the brain.

To generate contrast and spatialised sound effects, sounds of different frequencies were used, for example, the sub-rumblings of low frequency sounds to create wrap-around or more immersive effects were used in combination with higher frequency, directional sound sources. This was achieved by the separation—and therefore the localised spatialisation—of sound through the use of a quadraphonic speaker system.

However, in order to achieve a more pinpointed localisation of sound, fellow collaborator and co-author, Junji Watanabe, introduced his research-in-progress, the Moving Ultrasonic Speaker system (MUS). 9 These speakers emit high-frequency sound in a focused beam, creating a kind of ventriloquist effect: when the beam reflects off a particular surface, it sounds as though it has originated from that position.

The specific innovations developed through initial work were firstly, to install the speakers into computerised moving light housings to enable the control of beam direction, and secondly, to integrate MUS to the imaging system and then align beam direction to a performer’s movement via camera-tracking. In regards to the responsive properties of MUS, Junji Watanabe (e-mail correspondence, October 15, 2007) found that the moving sound beams added an overall enlivening effect to the soundscape, speculating that viewer/listener attention may have been more attracted to the dynamic presence of the sound beams rather than to any connection between the sound beams and performer movement or visual images per se. Luke Lickfold, for his part as the sound designer, found the moving speakers created a unique ability to ‘throw’ sound, a capacity he thought warranted further experimentation to explore spatialisation possibilities in conjunction with surround sound systems.

Current and future directions: virtual skin, speaking body and polyphonic sound

In workshops conducted throughout 2008 in both Australia and Japan 10 we have been further developing Living Lens as a performance-installation, specifically the use of ultrasonic beams to enhance the idea of image surface as virtual skin, and the performer’s reach domain as extended touch, aspects of which are mentioned earlier in this paper. In relation to the image surface, we have found that angling an ultrasonic beam directly at a projected image adds a peculiar perceptual quality to the screen surface, with the visual image seeming to emit sound. In this way, it is possible to give a resonant dimension to the motion of the image texture.

A further test, still in its exploratory stage, involves the integration of a performer’s wrist movements to the MUS via motion sensor. One result of this test has been little curlicues of sonic gestures in space, transmitted from circular wrist movements to the mechanical motion of MUS. This is an additional aspect to the extended touch notion, yet is also one step towards our aim of developing a different kind of "speaking body".

Here, an additional area for testing is whether a performer’s movement can also affect changes in the ultrasonic sound. To combine the directional capacity of ultrasonic sound together with more wrap-around effects of regular speakers and subwoofer systems, we have also been processing live vocal input through all of the speakers to create a polyphony, or chorus of multiple, layered voices in the space. Here, we further speculate that directing ultrasonic sound beams to deflect off the performer’s body, could also pose interesting possibilities on the idea of the speaking, resonant body within a polyphonic structure.

Summary

A question for artists using responsive technologies in performances is the perceptual nature of the interaction. For some artists working in this way, it is important that human motion or gesture is somehow apparent in the graphic and sonic representations. In our work, we are indeed aware of this issue, yet are seeking relationships between the dancer and audiovisual media that suggest also an organic and textural, rather than specifically gestural, connection.

To this end, our approach applies responsive methods and devices to realise a tactile and resonant sensibility within the installation space. As such, the legible connections between imagery and motion preferred by artists such as Palindrome and Troika Ranch may not be evident, yet a relationship between the delicate quality of the performer’s movements and the textures of the virtual images and sounds may well be possible.

References

• Broadhurst, S. (2007). Digital Practices: Aesthetic and Neuroesthetic Approaches to Performance and Technology. Basingstoke and New York: Palgrave Macmillan.
• Coniglio, M. (1989). MidiDancer. [Wireless motion sensing system]
• Cunningham, M., Ashkar, S., & Kaiser, P. (1999). Biped. [Dance and technology]
• Fraleigh, S., & Nakamura, T. (2006). Hijikata Tatsumi and Ohno Kazuo. New York: Routledge.
• Kaiser, P. (2000). BIPED/1999: Illustrated Essay. Retrieved November 1, 2007, from http://www.openendedgroup.com/index.php/artworks/biped/essay/
• Nikolais, A., & Louis, M. (2005). The Nikolais/Louis Technique: A Philosophy and Method of Modern Dance Including the Unique Gesture. New York: Routledge.
• Rodaway, P. (1994). Sensuous Geographies: Body, Sense, and Place. London: Routledge.
• Rovan, J. B., Wechsler, R., & Weiss, F. (2001). Seine hohle Form: Artistic Collaboration in an Interactive Dance and Music Performance Environment. Crossings: eJournal of Art and Technology,1(2).
• Saggini, V. (2002). Mapping Human Gesture into Electronic Media: An Interview with Mark Coniglio of Troika Ranch. Theremin Vox: Art, Technology and Gesture. Retrieved November 10, 2007, from http://www.thereminvox.com/article/articleprint/24/-1/3/
• Takemitsu, T., Kakudo, Y., & Glasow, G. (1995). Confronting Silence: Selected Writings, Berkeley: Fallen Leaf Press.
• Troika Ranch. (2006). 16 [R]evolutions. [Dance and technology].
• Troika Ranch. (n.p.). Troika Ranch: Performance, Installation and Film. Retrieved October 5, 2007, from troikaranch.org
• Waguri, Y. (1998). Butoh Kaden. Kohzensha / Justsystem. [CD-ROM].