Mimesis & Mocap DANM Thesis, Cognates & Contrasts by Topher Maraffi

UCSC Faculty Committee: Kathy Foley, Chair (Theater Arts, Puppetry and Masks), Ted Warburton (Theater Arts, Dance and Technology), and Michael Mateas (Computer Science, Expressive Intelligence Studio).

Keywords: Mimesis, Mocap, Mirror Gag, Turing Test, Machinima, Synthespian, Uber Marionette, Chatterbot ELIZA, Simulacra, Magical Mirror, Facade, Avatar Puppeteering, PuppetShow, Corporeal Mime, AtlW-Blowup, Digital Culture Lab, and Pinocchio.

Compare and Contrast to Contemporary Work in Digital Arts Practice:
My thesis project most closely relates to two areas of contemporary arts research: First, performance and dance studies that use motion capture technology to record the movement of live performers on stage, and then digitally process that movement data in real-time to project back into the performance as interactive animation or sound media; Second, serious games studies that use the AI technology in a game engine to animate 3D synthepians in a machinimatic real-time performance. My project seeks to combine these two areas of performance and games research to explore a third area of study in AI, to trick human observer’s of the performance into perceiving autonomous agency in the digital projection, otherwise known as a Turing Test or Imitation Game. In this short essay I intend to examine how my project relates to and differs from previous text or dialogue based imitation game approaches, such as seen in Joseph Weizenbaum’s Chatterbot ELIZA and Michael Mateas’ Façade. I will also look at how my project compares and contrasts with several contemporary dance performances that use mocap technology, such as Digital Culture Lab’s AltW-Blowup, Kris Verdonck’s I/II/III/IIII, and Hiroaki Umeda’s Accumulated Layout, as well as with machinima puppetry projects that use 3D avatars and game engines, such as Georgia Tech’s PuppetShow. 

Embodying the Imitation Game with Contextually Specific Theatrics:
There are a few key aspects of my project that are designed to address the imitation game problem. The biggest difference between my project and most chatbots since ELIZA, and which I think is addressed well in Façade, is delineating contextual specific interaction. Chatbots may achieve the ELIZA effect for a short time due to player interest, but they lose the effect quickly when the player is allowed too wide a range of textual input. Typing in slang or nonsensical sentences produces answers that reveal the software does not understand the wordplay going on. In general, Turing Test’s are traditionally criticized for the same reasons that general aptitude tests are criticized; they require too wide a range of diverse knowledge to pass, that may not be in the general experience of the person tested, or in this case the AI program tested. Façade creates a very specific social and cultural situation for the player to experience, with responses that are limited by that context. If the player attempts to break those implied rules, which are also in line with well established social etiquette, then the characters in Façade take the appropriate response of forcing the player out of their home. Thus there are several implicit and express factors built into the game to prevent a player from breaking the ELIZA effect. In my project I seek to create a contextually specific comedic situation by simulating the classic film pantomime the Mirror Gag, limiting the interaction to embodied movement.

Other similarities that my project has to Façade, but differs from chatbots, are embodied theatrics and emotions. Façade places the player in a dramatic situation with two embodied characters that behave in a manner that is purposely uncomfortable for a third party guest. The characters of Trip and Grace may be simplified 2D cartoons, but they still look and behave in a naturalistic manner, arguing with emotive action and dialogue that evokes sympathy or empathy in many players. The dramaturgy has the effect of both engaging the players continued attention, while also distracting the player from considering the agency in the characters. By simulating the Mirror Gag, I seek to achieve a similar emotive theatrical dynamic that is based on embodied humor or slapstick, to engage and distract the player. Traditionally, humor has been widely considered the most natural of human emotions, with comedy being a talent that is difficult to teach other humans, much less produce artificially through a machine. Creating a fun and funny AI program would likely have an ELIZA effect because slapstick humor has the capacity to bypass the intellect through the nearly universal language of comedy.

The theatrical and entertaining scene that the player finds themselves in the middle of within Façade has another advantage over chatbot programs, and which I hope to create in my project, which is the suspension of disbelief. This is something we are all trained to do when enjoying entertaining drama, in order for it to actually be entertaining. We engage our imagination to enter the virtual world presented to us, whether it is a novel, minimalistic play, movie, anime, or game. The suspension of disbelief, as a double negative, is an algorithm that inherently has a loop in it. It also requires, like other belief algorithms, theatrical or media ritual, such as artificial framing and lighting, or the hyper real of the simulacra. I hope to create this in my project by having the player wear a simple mime mask. The environment will be windowless with stage lighting, and the frame of the monitor will represent the mirror frame, as well as a magical window and a proscenium. The player willingly transforms towards merging with the reflection by donning the mask and approaching the reflection in the mirror, which also wears the same mask. What will be noticeably different about the reflection if the player looks closely, however, is the movement of the eyes and the fingers.

The mask is practically a universal element of traditional pantomime, for both mimes and clowns, and is highly influenced by shamanism. In the choreographed performance of my project, I envision the performer in the mocap suit entering the stage with their back to the magical mirror, intrigued by a simple white mask hanging in the dark space. Initially there is no reflection in the projected mirror, until the mask is placed on, at which time the visibility of the 3D representation fades up behind them, and facing them unlike a true mirror image. In this version I intend to have the performer reference the French corporeal mime technique of Jacques Copeau and Etienne Decroux, who used shamanistic meditative ritual when donning a mask, to become possessed by the spirit or character of the mask. Breath techniques are used to clear the mind so the mask can take over the performer’s body. Once the mask is in place, the performer will turn to face their reflection in the magical mirror, and the mirroring dynamic will start. In this performance the pantomime movements of the performer, and of the mimetic synthespian, will also incorporate corporal mime techniques of isolating body parts to simulate the lack of human affectation in an autonomous uber marionette.

In the eastern tradition of Japanese Noh mimes, there is a similar shamanic ritual for placing on the mask that actually uses a mirror. The performer dons the mask in front of the mirror, staying in place until the spirit of the mask takes over through the reflection. In my production, I will employ this technique by placing a real mirror on stage facing the projected virtual mirror, and have the performer place their mask on looking in the real mirror. The advantage of this is they will be able to see the virtual reflection appear behind them, starting the interaction through the technological device. 

Proprioceptive Movement with a Body without Organs
Although Georgia Tech did develop an interesting embodied version of the Façade game, the main difference between my approach to the Turing Test and text based games is that I want to engage the player’s entire body strictly through gesture, rather than engage their mind through dialogue. I believe this will create a more basic human interactivity that reaches less cognitive parts of the brain, as does pantomime and slapstick humor. The projected sythespian will appear to be a life sized mirror image of the player as a 3D representation, and although simplified like the Façade characters, it will move in a natural manner from the streaming mocap data. The difference to a real mirror, and what should engage the player’s attention, is the variations in the reflection’s movement. Mirrors engage people’s attention anyway, but mirror’s that are not passive should immediately increase the player’s interest, and may begin the uncanny ELIZA effect. My project also has another inherent agency loop that may increase this effect, that plays from the Mirror Gag itself; which is that the computer generated body is masquerading as a human who is masquerading as a reflection of the player.

To trick the reflection into revealing itself as having agency, the player must move their entire body, just like in the classic Mirror Gag, creating a playfully competitive dynamic between human and digital bodies. Hopefully this experience will become fun, and lead to empathy with the 3D body through proprioception. It will be interesting to see if the players ever unconsciously start mimicking the variations performed by the reflection through AI. To increase the engagement dynamic, whenever the player turns away from the reflection, AI programming will cause the reflection to pantomime prerecorded slapstick movements designed to evoke an emotional response. The player will be able to see this happening behind their back, through real mirror’s placed in the stage environment, and then can react to the challenge. This is where the potential comedy may materialize in the simulated gag.       

Looking next at a theatrical performance that uses digital media, entitled AltW-Blowout by Simon Biggs and Sue Hawksley at Digital Culture Lab, you can see some interesting similarities and differences to my project. Video of a dancer's moving bodies are captured and projected, achieving a mirroring effect, while the software manipulates pieces of the video to deconstruct and reconstruct the bodies. This piece does produce a degree of uncanny ELIZA effect in the animated collage, mainly in the reconstructed movement as a kind of video cyborg representation of the dancers. It is also successful in engaging the performer's creativity and audience interest.

But the main difference AltW-Blowout has to my project is the lack of dimensionality and predictable familiarity of the captured video, where we recognize the unaltered moving images of the live dancers. No matter how you slice them, these images are not native to the digital medium, but are flat echoes of our multi-dimensional organic world. A digital 3D avatar is a complete simulacra, a virtual body without organs in Cartesian perspectival space, formed from digital code, electrical impulses in silicon, and projected as living light like an apparition. Like puppets and animatronics, computer generated characters have an inherent otherness that recorded video can never achieve, no matter how much you process or manipulate it. By using processed motion capture data and AI to animate this type of character in a naturalistic manner, and then blend in subtle supernatural movements (which are actually inherent in the dynamics of the Mirror Gag), I anticipate that the uncanny effect can be enhanced and extended.

Cutting the Umbilical Chord of the Uber Marionette or Pinocchio becomes a Real Boy
The next two experimental dance videos I want to consider, by Kris Verdonck and Hiroaki Umeda, show how the human body can achieve an uncanny effect by moving like a marionette or a machine. In I/II/III/IIII by Kris Verdonck, the dancers are suspended by invisible wires, making them essentially into living marionettes. The uncanny effect is achieved by combining the naturalistic movement of the human bodies with the magical effect of a puppet defying gravity. The supernatural effect only works because the dancers surrender some of their agency to the marionette technology, creating a hybrid technique, and because the technology is hidden.

A similar marionette technique was used in the Woody Allen movie Everyone Says I Love You, in which a magical partner dance is done by the side of a river in Paris by using wires, and then I and many others rotoscoped out the wires in postproduction.

One more experimental dance example I want to examine is Hiroaki Umeda’s Accumulated Layout. Corporeal mime technique used movement isolations as a method for transforming the body into a vehicle or performative tool. An extension of this mime technique in dance is the robot in breakdancing, and popping-locking in hip hop. Both techniques use the machine metaphor to create the appearance of uncanny movement throughout the body. Body parts seem to have their own agency, as the dancer’s attention quickly and fluidly moves through each part. Fast pauses in between movements isolates moments in time as poses, very much like separating the movement into animation frames. Sometimes the dancer appears to defy gravity and glide across the floor, as in moon walking. Accumulated Layout has some of this technique combined with mechanically controlled lighting and sound, creating a synthesis of machine movement with a human body.

All these dance examples use technology to enhance an inherent uncanny quality of the body as a mechanical device. My project will be designed to create similar effects through the pantomime of the live performer, and then will technologically enhance the uncanny human movements to a supernatural level in the body of the synthespian. Defying gravity and isolating body parts can be taken to new levels in the virtual Cartesian space of the computer program, which is one of the main reasons why computer generated characters can be compared to Craig’s Uber Marrionette. Again, this supernatural quality was already a part of the Mirror Gag, as can be seen in the Marx Brothers and the I Love Lucy versions on the YouTube example. In the first example, it occurs when Groucho breaks the plane of the mirror to circle his reflection. While in the second clip, supernatural movement occurs when Harpo uses a trick hat to defy gravity and expose the reflection as a real person. The difference between the classic gag and my simulated version is that the synthespian cannot be exposed as a real person, but can be exposed as having real agency through the AI software. My last analysis is of other machinima style 3D puppetry research projects. For instance, Georgia Tech’s Puppet Show project has some similar real-time puppet and machinima qualities, but the motion capture technology is 2D optical blob detection using the Processing software to stream the motion data into the Unreal game engine, rather than 3D mocap.

There are other projects by the Georgia Tech Machinima and Synlab groups that contain some aspects of my project, but are different in that they use 2D computer vision techniques for the mocap. The Henson Creature Shop’s Digital Puppetry Studio is a better example of using live streaming 3D mocap to control a synthespian’s body, but even this technique has a crucial difference to my approach. All of these techniques and technologies are using the digital medium as tools for creating motion, while I want to use the medium as a fundamental partner in the creative process of the performance. If the body with organs is the medium of organic pantomime and dance, then the body without organs, or code, is the medium of virtual pantomime and dance. My project seeks to be a first baby step towards implementing digital intelligence into the embodied arts of acting and dance, as a machinima technique, that can lead to true synthespians with improvisational agency that shapes the performed movement. This is my proposed solution to the Imitation Game that has the potential to realize the Uber Marrionette in a virtual 3D puppet, by releasing some of the traditional strings between the puppeteer and puppet to the coded medium itself, and eventually mutate Pinocchio a little further towards becoming a real boy.

Additional experimental dance and technology project web sites:

EU Digital Culture Lab...

ASU's Motion^e Dance Performance Videos...

Brunel University Design & Performance (DAP) Lab...

UK-based E-Dance Project...

Additional experimental machinima and puppetry project web sites:

Georgia Tech's PuppetShow Project...

Jeffrey Ventrella's Avatar Puppeteering Web Site...

Paper Reference:

Georgia Tech's Synlab Papers...

Digital Culture Lab Papers:

Memory Maps in Interactive Dance Environments by Simon Biggs and Sue Hawksley (PDF)...

Thinking movement and the creation of dance through numbers by Stamatia Portanova (PDF)...

Ambiguous zones: the intertwining of dance and world in the technological era by Ivani Santana (PDF)...