The science of art: how kinesiology, computation, and Kinect may reveal the ‘code’ that transforms movement into dance

In This Article

Southern Methodist University, United States

Abstract

The evolution of dance education over the last 100 years can be clearly contextualised by examining the developing technological lineage from Gray’s Anatomy to Dance Forms 2.0, highlighting the transformation of how we record and represent the human body and the physical act of dance. Column symbols and two-dimensional line drawings have metamorphosed into interactive anatomy software, and tele-immersion has created an entirely new way of being ‘present’. This paper summarises the capability and subsequent benefits of a new tool for recording human movement, the Physical Data Capture Lab. Movement is captured via infra-red depth mapping and gravitational pressure sensing, providing the physical data necessary for the creation of a personalised musculo-skeletal avatar. This personalisation is accomplished by digitally embedding the avatar with the measured physical properties of the subject dancer. Movement recorded in this manner may then be studied in detail, allowing for a more comprehensive examination of the internal geometry, architecture and physics that coalesce to become the external art of dance.

Keywords: kinesiology, computation, Kinect, data, avatar

Introduction

Each time we view dance we are presented with two worlds: the external world of context; i.e. gender, culture and intent, and the internal world of biomechanics; i.e. physics, geometry and architecture. As viewers, we often do not consider the relevance of the internal world, and to a certain degree overlook the reality that the physical act of dance can only exist as an external summation of internalised forms and forces. The recording of dance movement therefore has traditionally emphasised the external over the internal, requiring the latter to be extrapolated from the more readily perceived exterior.

Over time we have gradually honed our ability to record movement, from the cave and wall paintings of prehistory, to the high-definition video capture globally available today. Given the vast amount of physical information embedded in human movement, it would appear that we are not that far removed from even our relatively recent Neolithic predecessors when it comes to creating an accurate and complete record of this movement. Archaic painting practices and contemporary video capture can both replicate gender, culture and context, and yet neither can accurately record the internal torques, leverages and vertices that unite to erect the human form in space, and consequently give rise to motion.

As a professor of dance possessing an equal interest in both kinesiology and choreography, I am perpetually intrigued by the relationship between the science of the internal and the subsequent art of the external. As an avid videographer, I have benefited immensely from recent advances in video technology that now permit me to record movement in such definitive high-resolution. The external game of movement capture has ostensibly been won. The internal game, it would seem, has just begun.

The past

In her book Choreo-graphics, Anne Hutchinson Guest (1989) compares dance notation systems dating back to the fifteenth century. From the rudimentary notation system found in the Burgundian Manuscripts (ca. 1490) to the intricate and complex Labanotation of present day, dance record keepers have consistently demonstrated their creativity and innovation in the use of letters, words, abstract symbols and stick figures to represent the myriad movements of the human form. The majority of these systems are strictly confined to the aesthetic and style of the dance form they represent, from the bird’s eye view offered by the Beauchamp-Feuillet notation for Baroque dance (Feuillet, 1700), to the stick figure and musical note system designed specifically for ballet by choreographer Arthur Saint-Léon, who first published his system in 1852, in his book La Sténochoréographie, ou Art d’écrire promptement la danse. In 1928, Rudolf Von Laban presented his Kinetographie Laban, which introduced abstract symbols to denote concepts of space, effort and weight shift. As prior systems attempted to create a more natural frontal view record of human-like figures moving throughout evident floor patterns, Laban took a more phenomenological approach by using geometric symbols to represent the anatomy, expressivity and spatial patterning of the subject dancers.

Three decades after the inception of Labanotation, dance theorist Noa Eshkol partnered with architecture professor Avraham Wachman to establish an innovative system entitled Movement Notation, distinguishing between movement and dance. In the introduction to their book they state:

The difference between these two conceptions is this: while the term ‘movement’ includes in its meaning all the possibilities of movement of the human body in their various manifestations, the term ‘dance’ indicates, in every period, a certain range of movements expressing the choice of a composer and dancer, and fulfilling the demands of a particular society in a certain epoch (1958, p. vii).

The Eshkol-Wachman Movement Notation (EWMN) system focused on the evident biomechanics of the human form, more so, than the documentation and perpetuation of a specific style or technique of dance. This departure from a notation system based in dance terminology opened the door to a more scientific approach to movement analysis and capture. Eshkol and Wachman placed the focus on the mechanical movement potential of the joints of the human body and their notational language incorporated math and geometry to describe action and position in terms of horizontal and vertical coordinates. The circumductive nature of the joint articulations of the human body became clearly evident in the various conical and circular patterns represented in Eshkol-Wachman’s notation diagrams.

The emergence of dance kinesiology in the mid to late 20th century introduced the idea of defining movement from both an internal and an external perspective. Kinesiological analysis integrated anatomy, physics, geometry and math, thereby establishing a comprehensive assessment tool with the capacity to discern, and describe, any and all human movement with great accuracy. Although Laban identified and quantified the embedded efforts that power the motions of the human body, he did not incorporate internal anatomical nomenclature in his notational language.

The advent of computer animation in the late 20th century provided the initial platform for the successful marriage of external-based movement notation to the internal-based analytical science of dance kinesiology. In 1981, computer programmer Eddie Dombrower created the first computerised notation system entitled DOM (Dance on Microcomputer) (Forrest, 1986).This system allowed choreographers to use a simple set of codes to enter their work. These keyboard directives were then performed by an animated figure on screen. Nine years after the advent of the DOM dance notation system, a team of code writers from Simon Fraser University created LifeForms, a computer program that enabled choreographers to create entire dances in virtual space. Merce Cunningham made history in 1991 when he premiered Trackers at New York’s City Center Theater, one of the first dances ever to be completely choreographed in digital form and later set on dancers.

Concurrent with the rise of computer-based notation and choreography programs in the late 20th century was the development of motion capture systems designed to record movement in three dimensions. Unlike film and video, which display a two-dimensional semblance of three-dimensional space, motion capture provides the capacity to accurately record human subjects, and later animate them in three-dimensional virtual space. Although contemporary motion capture systems provide a relatively accurate tool for the capture and archiving of dance movement, the complexity and costs of such systems take them far out of the purvey of most dancers and choreographers.

The present

In my current position as a professor of dance in undergraduate education, I have a vested academic interest in capturing, analysing and archiving dance movement from both a scientific and aesthetic standpoint. My multiple research and teaching interests in choreography and dance kinesiology necessitate a multi-tasking approach in the recording of dance movement.

In my capacity as both a practitioner and educator in the art of choreography, I commonly use high-definition video as a functional replacement for dance notation. I am able to capture completed works on video and then digitally archive them on computer hard drives. I can then easily reference these videos, when the need arises, in order to restage the work on other groups of dancers. One chief drawback in this process is that I am archiving an interpretation of my work, not the work itself. Video does not provide an unbiased score of the work, as would be the case if it were recorded and archived using Labanotation.

As a professor of dance kinesiology it is my responsibility to analyse movement, whether filmed or live, for its biomechanical properties and then facilitate the discernment and usage of these properties amongst my students. I continually make informed estimations of internal biomechanics based on my knowledge of anatomy, physiology and physics. The process of kinesiological analysis is quite time consuming and not wholly accurate. It is a probabilistic science in that the assessment of hidden intermuscular actions is at best an educated guess based on observed phenomena. It would be extremely helpful to have an investigative tool that could accurately analyse and record both the internal joint and muscle actions as well as the summative aesthetic results.

My success as a professor in both dance science and dance art relies heavily on my critical understanding of not only traditional and contemporary tools for investigation and instruction, but also on my diligent search for progressive and emerging technologies that may aid in my dance research and pedagogy. Recent technological advancements and market affordability in motion sensing, optical imaging and computer processing have inspired me to envision a progressive new suite of tools for instruction in dance kinesiology and choreography.

In the past four years, I have methodically theorised and developed a new form of movement capture entitled the Physical Data Capture Lab, while at the same time envisioning initial design elements for an immersive and highly interactive dance kinesiology software application. When used together these two progressive tools will provide an engaging environment wherein dance students may call upon a virtual kinesiologist to guide them in accurate and investigative experiments in movement mechanics.

In its present ‘proof of concept’ form the Physical Data Capture Lab (PDCL) is comprised of four primary components: two Microsoft Kinect devices (infra-red depth sensors working in tandem), one gravitational pressure-sensing mat with over 4,600 sensors, one projection visual display wall and one high-speed ‘gaming’ laptop. The premise of the PDCL is to capture and record the geometric volume of space occupied by a subject dancer and accurately register this volume with the subject dancer’s gravitational contact points on the pressure-sensing mat. The placement and arrangement of the myriad joint articulations of the test subject are computationally extrapolated from the measured volume using the process of forward kinematics (Calvert et al, Nov. 1982). By registering the where and when of skeletal placement during contact, one can computationally identify the specific muscles and degree of engagement that produce the recorded movement.

This process of capturing the physical information embedded in all human movement is somewhat like a game of computational ‘cause and effect.’ Forward kinematics works on the anatomical premise that the joints of the human body must obey a mechanical rule set predicated by their articulated design and interconnectivity. For instance, if one knows the placement and position of the pelvis, then the femurs and the lumbar spine must be adjacently connected. Taken as a whole, the various contours of the measured geometric volume of the body provide the information that allows for the computational estimation of the placement and actions of all the bones of the subject dancer. By using gravity as a constant, forward kinematics as a guiding rule set, and accurately capturing the placement and degree of body pressure contact on the mat, one can calculate the probability of what muscles are engaged in the process of moving the skeletal structure through space.

As a dance kinesiologist, I am constantly aware of the line of gravity in movement analysis. The simple act of standing reveals a process wherein muscles must contract across multiple skeletal fulcrums (joint articulations) in order to counteract the force of gravity and thereby erect the body in space. If one were to represent the bones of the body as geometric equations, record the sequence and timing of the movement of these bony geometries, record the surface pressure of where and when these volumetric geometries contact the mat, and then factor in the constant force of gravity, one can procedurally compute the representative muscle equations that produce any and all human movement. The physics and geometry embedded in human motion are with us at all times and yet rarely do we articulate, process and learn from this embodied wealth of physical information. The key to a successful and effective use of this information, however, is the ability to present this inner world of physics and math in a language that is easily accessible to the student of dance.

The future

Dancers are visual learners. They observe and replicate. Verbal language, whether mechanically descriptive or metaphorically suggestive, is provided with a physical demonstration to help modify and contextualise the presented movement vocabulary of the choreographer or instructor. Taken as one, the combined physical and verbal information of dance acts as a type of movement meta-language. I envision a companion tool to the PDCL entitled the 3D Dance Kinesiology Interactive Software application (3DKIS) which will provide a visually descriptive dance meta-language that integrates movement, art, anatomy and biomechanics into an inviting and user-friendly human-computer interface.

In this interactive application the dance student is presented with two side-by-side visual panes. The one pane presents a musculo-skeletal-avatar (MSA) in animated three-dimensional space, and the adjacent pane displays kinesiological analysis and relevant contextual information. The student reads the anatomical text and then has the option to highlight specific portions of the text, at which point the MSA demonstrates examples of the chosen information in the virtual 3D environment. The student may observe specifically chosen animations of individual muscles, groups, or layers of muscles, or choose to view actions of the skeletal joints with no muscles present. Rudimentary variations on this approach are utilised in current computer-based anatomy programs; however, I am interested in creating a greater degree of interactivity between the student, the MSA and the descriptive text. I would like to provide the student with an option to ‘reverse-engineer’ the flow of information delivery.

I envision a student using a keyboard or modified game controller device that will allow the student to efficiently and effectively manipulate the MSA in virtual space. Any action by the MSA will simultaneously trigger an immediate anatomical and kinesiological description in the adjacent textual pane. This real-time interactive environment will allow the student to pose biomechanical questions to a virtual kinesiologist, programmed with the capacity to provide anatomical and analytical information on any movement the student may imagine.

Students participating in this interactive avatar-text environment will gradually become accustomed to the simultaneous ingestion of ‘physio-visual’ and textual information, thereby defining the parameters and effectiveness of this new form of dance meta-language. This interactive and dynamic learning environment can then be more streamlined and even personalised with the addition of the PDCL. Instead of using the keyboard/modified game controller to manipulate the MSA, students will have the option to perform their physical queries in the PDCL and then subsequently import this captured movement information into the 3DKIS application. Prior to entering the PDCL capture area the student is measured for his/her overall mobility, joint restrictions, and muscular strength. This morphological information is then imported into the kinesiology application, which generates the personalised avatar. The resulting musculo-skeletal-avatar will not only move exactly as the student moves, but it will also have to obey the real-world physical rule set of that student. If the student manipulates his/her personalised MSA beyond the avatar’s set limits while experimenting in virtual space, there is a high probability that the dancer will become injured if he/she were to attempt the same movement in reality.

The eventual combination of the PDCL and the 3DKIS application will provide an engaging and immersive meta-language learning environment that will allow the student to experiment with and process both kinesiological and aesthetic information. The primary goal is for the student to better understand the mechanics behind his/her own personal aesthetics, thereby encouraging the student to develop new ways to gain greater control of these mechanics with the omnipresent goal of personal artistic growth in the discipline of dance.

Summation

At the present time, the 3DKIS application only exists as a theorised possibility. Recent advancements in anatomical software animation for the web have provided such applications as iMuscle and Interactive Functional Anatomy. When combined with current dance kinesiology textbooks, these applications suffice to provide the student of dance with fixed and animated images of the inner workings of our biomechanical system. From a pedagogical perspective, we are able to visualise, with the use of current technology, an internal anatomical world that has not been accessible prior to this period in history.

The next progressive step, however, is to provide this access under the umbrella of a dance-based meta-language that inspires the student to become more personally invested, and therefore more deeply immersed, in the learning process. It is one order of difficulty to learn the muscles, ligaments and bones of the human body. It is a secondary degree of difficulty to develop an understanding of the physics and geometry that are at play in the biomechanical process. It is an even higher order of difficulty to integrate this information into a comprehensive theoretical understanding of human movement capabilities. And ultimately, if you are a student of dance in the early years of this 21st century, it is your goal to translate this wealth of theoretical knowledge into an effective and holistic practice within the art form of dance.

The PDCL in its current prototypical form provides the initial cornerstone for such a progressive and holistic approach towards dance education. Students are able to observe and record their plantar weight distribution in relation to structural alignment and centre of gravity. With the additional use of live streaming from a high-definition video camera, students are able to view themselves performing difficult balance movements while simultaneously viewing their planter distribution, thereby creating a biofeedback information loop. Certain stylistic values may also be addressed during this process. Students may experiment with the increased muscular engagement necessary to portray the tension of a Martha Graham contraction and subsequently compare this feeling state to the muscularly disengaged weightlessness often associated with contemporary release technique. As the students adjust to accomplish their chosen modality (captured on video), and additionally attempt to maintain a structurally sound plantar base (displayed via the pressure sensing mat), they are provided with highly relevant physical data to reinforce their physio-psychological awareness of stylistic clarity.

This physical experiment is also captured by two Kinect depth-mapping devices that record the constantly changing geometric volume of space occupied by the subject dancer. Using a current software application entitled Brekel Kinect it is possible to create a wire-frame representation of the dancer. An avatar is then generated and rendered, via the process of computer code-writing that can then be viewed and studied in three-dimensional virtual space on a computer display. The primary challenge is the accurate calibration of the measured distance from the Kinect devices to the pressure-sensing area. The accurate extrapolation of the subject dancer’s internal biomechanics from this prototype of the PDCL is not currently 100% reliable; however, there are still many benefits to be gained from this process.

In analysing one’s own stylistically chosen movement experiments, one can begin to explore the primary muscle groups and skeletal alignments that are most suited for a safe and efficient performance of the chosen actions. Through the process of visual and tactile bio-feedback, students may gradually expand their personally experienced kinesthetic library of biomechanically sound movement practices. From a choreographic and movement invention standpoint, students may view their currently generated avatar from multiple perspectives and distances in virtual space, thereby expanding their understanding of the many ways their work may be interpreted depending on audience orientation to the performance.

The future of these progressive technological tools is bright. I envision students learning about the physical potential of their bodies at a much younger age, in a dynamically holistic environment that fully engenders an expanded awareness of how the body may be presented and subsequently perceived. I see a future that does not supplant our time-honored tradition of oral history but one that encourages a more kinesthetically-aware relationship between mentor and mentee, a little less trial and error, and a little more informed interpretation and accelerated retention. By having an increasingly cognizant understanding of the internal we may begin to more readily interpret the external.

The inevitable advancements in high-performance computing and complex algorithmic code writing clearly represent a rich technological resource that will fuel the increasingly effective future iterations of the Physical Data Capture Lab and the 3D Dance Kinesiology Computer Software application. Whether or not these become ubiquitous tools for archiving dance and the pedagogical virtual textbooks of the new age, only time will tell. However as we progress into the unseen future of dance on the global stage and in the academy, we will most certainly continue to be the prime beneficiaries of the ever-solidifying marriage of science and art.

References

  • Calvert, T. W., Chapman, J. & Patla, A. (Nov. 1982). Aspects of the kinematic simulation of human movement. IEEE Computer Graphics and Applications, 2 (9), 41–50.
  • Eshkol, N. Wachmann, A. (1958). Movement notation. London, England: Weidenfeld and Nicolson.
  • Feuillet, R. A. (1700). Choregraphie ou l’art de de’crire la dance. New York, NY : Broude Brothers.
  • Forrest, J. (Fall 1986). Field report: a choreographer’s assessment of the DOM notation system. Dance Notation Journal, 4 (2), 47–48.
  • Hutchinson-Guest, A. (1954). Labanotation: alphabet of basic symbols. New York, NY: New Directions.
  • Hutchinson-Guest, A. (1984). Dance notation: the process of recording movement on paper. New York, NY: Dance Horizons.
  • Hutchinson-Guest, A. (1989). Choreo-graphics: a comparison of dance notation systems from the fifteenth century to the present. New York, NY: Gordon and Breach.
  • Laban, Rudolf von. (1995). Kinetographie Laban. Noetzel: Heinrichshofen-Bücher.
  • Saint-Léon, Arthur. (1852). La Sténochorégraphie, edited by Flavia Pappacena (Chorégraphie, n. s., a. 4, n. 4, 2004), Lucca, LIM.

This article has not been published, submitted, or accepted for publication elsewhere.

© 2015, Christopher Dolder