| Topher DANM WIKI | Digital Double Project / DANM MFA Thesis: Mimesis & Mocap v1.0 - by Chris(Topher) Maraffi, Summer-Fall '09, Updated 12/10/09. |
|||
UCSC DANM Thesis Faculty Committee: Kathy Foley, Chair (Theater Arts, Puppetry and Masks), Ted Warburton (Theater Arts, Dance and Technology), and Michael Mateas (Computer Science, Expressive Intelligence Studio). Keywords: Mimesis, Mocap, Marx Brothers Mirror Gag, Pantomime, Turing Test, Imitation Game, Memory, Machinima, Synthespian, Avatar, Uber Marionette, Eliza Effect, Gesture, Game Engine, Simulacra, Double, and Magic Mirror. >Update: Click here for the approved Thesis Proposal (PDF)... >Update: Click here for Cognates & Contrasts (Web & media comparisons)... |
||||
|
|
Introduction Through streaming motion capture (mocap) data into game engine AI software I intend to research a hybrid production technique that blends live pantomime with digital autonomous processes to enable the medium of code to become a creative partner in a movement dialogue. To minimize the technological risk of delivering the thesis performance in April, and to produce a clear roadmap for the desired interaction, I intend to take an iterative approach to this research by first producing a fully choreographed version, or a simulation of the simulation. This approach will allow me to build and test the conceptual framework that will ultimately support the technological approach, by first exploring representation, gesture, and memory between a living and synthetic body in a theatrical performance. Throughout this process, I will also be researching real interactivity using AI software, and will present the results of my research for use in future projects along the same line of enquiry.
|
|||
| First Draft of Choreographed Performance Script A corner rectangular space in the DANM DARC Theater is delineated as a stage set by theatrical curtains. Viewers stand at the fourth wall looking in on a theatrical dressing room in the style of old Broadway or classic Hollywood. Some glamour photographs of movie stars and painted posters of films from the early 20th century are interspersed with furniture, costumes, and stage props such as hats and masks on racks. The central focus of the room, however, is two opposing vanities with large rectangular mirrors framed by rows of white exposed bulbs. A variety of used makeup bottles and a vase of flowers crowd the top of each black table in front of the mirrors, but only one vanity is obviously still in use. Around this vanity are additional mirrors, such as a small round tabletop mirror with a flexible arm and a tall standing mirror to one side, and both are angled towards the audience to reflect the sitter. Also sitting on the table is a simple white mime mask on a mannequin head, and a pair of white mime gloves, as well as a laptop facing away from the audience. Through the reflections in the mirrors, the audience can see a YouTube clip of the Marx Brothers mirror gag from Duck Soup playing repeatedly on the monitor. The other vanity is obviously not in use, as it is almost alter-like in the careful arrangement of older makeup bottles, faded flowers, and just a couple old black and white photos in silver antique frames. One of the photos is a publicity still of the Marx Brothers, only with my own face composited on top of Zeppo’s. There is also a vintage print by Gilbert named Vanity, which contains a famous optical illusion displaying a woman sitting at a mirror, that can also be viewed as a giant skull. Vintage jazz music is playing on some unseen radio or record player, although occasionally it skips in a manner reminiscent of a contemporary DJ remix. I enter from the back of the stage space, away from the audience, stepping through the curtains. I am wearing a colorful oriental silk robe, and appear to be an actor relaxing between costume changes. My hair is slicked back in the manner of the 1940’s with a hair net, and my beard is neatly trimmed. I sit at the well-used vanity, look at myself in the mirror for a moment, and light a cigar. Just then an upbeat tune begins to play “It’s a great big beautiful tomorrow…”, prompting me to pull out my cell phone from my robe pocket as it continues to play the Carousel of Progress theme song. I answer it, stopping the song. “Hey. Yeah I’m getting my head into the game. Fifteen minutes? OK. See ya.” Then after taking one more puff, I walk over to the other vanity and place the cigar in an ashtray on the table next to the Marx Brothers photo. I look into the vanities mirror, but no reflection looks back. That mirror (which is not a mirror at all, but a rear-projected 3D rendering that represents a mirror reflection of the empty stage space) refuses to recognize me, will not retain my reflection in it’s memory. After contemplating my absence for a moment, I walk back over to my vanity, and take off the robe to reveal a black unitard that has many small white fluorescent spheres attached to it. Sitting down again, I place on the white gloves, flexing my hands as they come alive gesturing. Then I pick up the mask and carefully place it on, continuing to stare into my vanity’s mirror, clearing my mind in the tradition of western corporeal mime and eastern Noh pantomime, allowing the ghost of the mask to possess my body like a mechanical puppet. Abruptly I yell into a microphone on the table “Intent!”. At that moment, I dim the vanity lights by manipulating a switch I have on the table, and start a mirror ball rotating above the set with a pin spot to create theatrical light effects in the space. The audience can see me manipulating the technology that is setting the mood of the space. Then a movement becomes apparent in the other vanities mirror. I stand up, still looking into my mirror at the figure materializing in the other virtual reflection behind me. The figure in the other mirror behind me is not facing away like a normal reflection should although otherwise it is a 3D rendered mirror representation of myself in a black mocap suit, white gloves, and mime mask. The virtual space in the other mirror mimics the environmental lighting in the room. My digital reflection is staring and gesturing at my back. When I turn by body to confront it, the figure moves inhumanly fast (almost a cinematic cut) to mimic my position, and begins following my every movement. After some tentative pantomime, my digital reflection and I engage in the mirror gag, as I appear to be playfully trying to trick the reflection into exposing itself as having independent agency. I pantomime and dance about, occasionally seeing glimpses of improvised gestures going on when my back is turned, through the mirrors on the other side of the room. Our movements are revisions of the original mirror gag that is still playing on the laptop monitor at the other vanity. As the pantomime progresses, jarring mechanical repetitions of movement occur in both me and my reflection, with more extreme cinematic quick cuts in the synthespian, much like skips in a DJ remix. This slipping should have a trancelike possessed quality to the movement. There are also other supernatural variations that are already built into the tradition of the mirror gag, such as in the hat trick sequence. After we place on our hats, I lift mine as if to tip it, as does my reflection. However, my reflections head stays with his hat, stretching his neck several inches in an uncanny manner. I quickly drop the hat back on my head, feeling my own neck. When tentatively I try it a second time, it comes off normally. At that point, I gesture several times to drop my hat, as does my reflection, but then when we do drop them his hat bounces back up into his hand (like Harpo’s famous trick), while mine falls to the floor. Just then my phone rings again, “It’s a great big beautiful tomorrow…”, and I walk over to my vanity to get it. My reflection continues to improvise while my back is turned. “Yeah? OK, I’ll be right up.” I look up into my vanities mirror at the figure in the other mirror gesturing at me to come back and play. Turning, I quickly walk over to the other mirror as my refection starts to mimic me again. Moving my masked face very close to the surface, I quickly tap the glass where the edge of the figure’s mask is placed, knocking it off, as my mask continues to stay in place. There is no face behind the reflection’s mask, only the ghosted impression of a skull and eyeballs. The figure throws up its arms in a slapstick manner, and runs off back into the mirror, fading as it goes. I take off my mask looking after it into the empty reflection again. I grab the cigar off the table, pick up my hat and tip it towards the empty mirror, walk back over to my vanity, turn up the lights again, put on my robe, and start walking towards the curtains as the Carousel of Progress theme song begins to play again for the final time. “It’s a great big beautiful tomorrow…” This performance is meant to create a hybrid performance space that juxtaposes and layers opposites to create the perception of uncanny theatrical magic clearly using technology: the light comedic slapstick of vaudeville in the competitive game-like movements of the mirror gag is remixed with the serious ritualistic movements of mask theatre and corporeal mime; the nostalgic past media of film, whose linear memory passively repeats the ghostly movements of long dead performers like the Marx Brothers, is remixed with nonlinear reflexive new media processes to animate agents or spooks in digital memory; the organic embodied movement of the live performer in real space is remixed with the nonliving body-without-organs movement of the synthespian in virtual space. These metaphors are performatively explored by the actor through old and new techniques and technologies, from the mask and stage lighting to the virtual mirror and simulacra. It is conceptually important that the performance occur in a theatre and that the performance is recorded into digital memory, and broadcasted on the Internet as a YouTube video. Pre-production Designs and Visualizations:
References:
Thesis Project Production Schedule
Ongoing: Real-time embodied interactivity research using mocap and software. Thesis Production Stage, Technology, and Props & Costume Budget Stage:
Technology:
Props & Costume:
Conceptual Reference: Mimesis as a Reflective and Reflexive “Magic Mirror” I am equating this subjunctive translation of mimesis to Turner’s “magic mirror” concept, since representation is not just a passive reflection, but instead also encompasses a reflexive process that is actively re-modeling our physical environment. Because of the reflective quality of this “mirror”, the nature of the mind can be deduced as technological or machinic. But since the mind is also undeniably an organ, on the flip side of the reflection, the nature of technology can be deduced as organic. Also, because of the reflexive or “magic” quality of the mirror, the process driving technological “progress” may be viewed as a natural evolutionary mechanism. These reflective and reflexive cultural processes can be compared to replication and mutation processes in genetic and memetic evolution theory. Although I will touch on these broader issues of mimesis in my projects, I will mainly focus on technologically simulating a simplified version in pantomime and dance. “Mirror Gags” as a Simple Example of Mimesis in Theatre Marx Brothers YouTube examples of the "mirror gag": My thesis application of this type of mimesis will be to have a 3D character reflectively follow a live performer most of the time, although with some micro variations coded into the repeated movement, except when the performer turns away from the 3D character which prompts improvisation. Simulating this type of mimesis with technology adds another layer of complexity, as the digital character is pretending to be a human that is pretending to be a mechanical reflection of the human performer. This reversal may actually make the 3D character appear more "real" to the performer, by confusing the Turing Test aspect of the interaction. “Call and Response” as a Simple Example of Mimesis in Social Dance Breakdance YouTube example of a "battle": This technique originally evolved from African American spirituals, and is characterized by a delay in the repeated form, with improvisational variations added into the repetitions. In African American social dance, as personified in the jazz and swing dancing at the NYC Savoy ballroom from the 1930’s to the 1950’s, it was a cultural rule that a dancer could copy another dancer’s moves, but must vary them in some creative way, thus putting a unique and personal stamp of ownership on the movement. Over time this led to a variety of personal styles within the overall style of African American social dancing, with movements being customized for an individual dancer’s abilities and interests. The cultural directive to vary copied dance moves can be viewed as a mechanism for change that not only increases the diversity and complexity of the individual dance, but prompts differences in styles that can lead to the formation of new dance forms. The swing dances East Coast Swing, West Coast Swing, Carolina Shag, and Balboa all likely evolved from New York Lindy Hop, but by different individuals in separated geographic and cultural environments, where the representation of the movements were accidently and purposely altered. When viewed as a cultural evolutionary process, mistakes in mimicry produce micro-mutations in the copied information or new moves, which may then be selected or discarded as part of the original style. On the other hand, intentional cultural appropriations or fusions of foreign movements to a given dance, such as adding acrobatic movements from vaudeville or gymnastics, may lead to macro-mutations in the form, especially when the musical environment changes. Some jazz techniques, like "snake hips" may have evolved in this way to "popping and locking" movements in breakdance. The techniques involved in a street dance competition reflect the improvisational structure of jazz-blues influenced African American music. The classic Blues structure (A>A*>B) shows the call and response dynamic: an initial musical query is posed (A), then repeated with variation (A*), and then resolved with an answer (B). My goal is to incorporate this algorithm into the code controlling a digital character. Applying Park’s “Rep & Rev” Concept to Technology I am looking to explore a similar relationship with the “ghost” in the machine by using software to technological Rep & Rev the movement of a live performer, so that it affects the flow of a dance performance. My goal is to create an improvised dance dialogue between the performer and the digital representation, so that the borders between the real and the virtual become blurred through perception of shared agency in the creation of the movement. I will base my coded application of this mental technique on my experience as a social dancer, in how my mind appears to run pattern recognition and random algorithms when improvisationally leading a follower through social dance patterns. Embodying Kleist's "Puppet God" or Craig's “Uber Marionette” in a Digital Puppet A similar theatre concept I am exploring is Craig's “Uber Marionette”, which he envisions will "not be the flesh and blood but rather the body in a trance - it will aim to clothe itself with a death-like beauty while exhaling a living spirit". Craig develops Kleist's mystical ideas of a "degenerate form of a god" further while evolving them into a more secular direction for making the art of theatre more reflexive: "If you can find in Nature a new material, one which has never yet been used by man to form his thoughts, than you can say that you are on the high road towards creating a new art form." His description of "mechanical perfection" brings to mind both gollums and cyborgs: "We should see him as a statue in which the weakness and the tremors of the flesh are no longer perceptible". I am exploring both Kleist and Craig's views as becoming embodied in digital 3D characters that exist in the virtual Cartesian space of the computer. This character is definitely a virtual marionette, as the mathematical "rig" controlling the character’s movements represents physical puppet rigs, complete with simulated strings and handles. However, a digital character is “uber” because it can perfectly mimic human movement through motion capture technology, and go beyond anything a human or physical marionette could possibly accomplish. This alchemy of human movement with digital technology qualifies the 3D character to also be a species of cyborg that is both familiar and uncanny in its movements. The technology creating the 3D puppet has been evolving in culture from at least the Renaissance period, when time was mechanized and perspectival space was conceptually modeled, and then replicated and revised through the visual mediums of drawing, painting, theatre, film, television, and ultimately in the modern computer. Human anatomy, kinetics, and dynamic forces are all mathematically represented in the 3D character and the digital environment it inhabits. In addition, the model can be copied and manipulated until it is a perfect mimesis of the director’s thoughts. The likely next step in the evolutionary process would be to have this digital mimesis be the mind of a physical robot, situatng a virtual body in Cartesian space to control a real body, in the same way our minds model reality (but that is another project entirely). As much as Craig, and many other director’s such as George Lucas, would probably like this much control over an actor, the main issue is that the performance may become the product of a single vision, which limits the spontaneity and creativity that can come out of collaboration. Free play or improvisation is usually encouraged in live theatre rehearsals and repetitive movie takes, which allows for “happy” accidents and performing “magic” to happen. As noted previously, such “magic” can be seen as a cultural mechanism for evolutionary change, and usually emerges from interaction between the humans involved in the environment of the scene. Instructions related by the director may be mutated by a performer through unconscious mistakes in the representation of the director’s “vision”, or through purposeful creative acts that reflexively re-route the direction of the action. Even at an individual level, human minds have multiple thoughts or memes competing and cooperating to produce unpredictable behavior. To make a digital character more realistic, this type of freedom or agency must be bestowed through coding unpredictable choices into it's behavior. My projects conceptually explore programming such spontaneity into the 3D character’s performance, using coded algorithms that contain prioritized conditional and random functions. The code would blend a previously processed clip with the current movement according to software parameters, so that the clip relates to the action but is not the same every time, producing the illusion of improvisation. However, if the human performer and audience gets the feeling of improvisation and reacts to it, even if it is an illusion, the dynamic in some sense becomes real. At the very least, developing this realtime mimesis technique for a 3D character has the potential to fundamentally change the way director's employ digital actors in a theatre or film production, by integrating a degree of freedom for the character to "method" act. From Puppetry to Film Animation YouTube example of Disney 1964 Carousel of Progress Behind the Scenes Mocap: Another director that has been instrumental in making the digital actor evolve in film is Peter Jackson. His choice to make a major character like Gollum in the Lord of the Rings movies a mocap-driven 3D character showed that a human-technology hybrid, or cyborg, could create a memorable and believable performance. His decision to employ Weta digital to populate his movies with computer simulated extras using the software Massive opened the door to procedural animation in films. Also, Robert Zemeckis has consistently embraced performance capture technology, and even taken it a step further than other film makers. From animatronics and rotoscoping in movies like Who Framed Roger Rabbit, Back to the Future, and Forest Gump; to creating a production company, Matchmovers Digital, that uses mocap for the majority of movement in the all-CGI films Polar Express, Monster House, and Beowulf. Lastly, the Henson Creature Shop has started combining real-time puppetry techniques with mocap to produce 3D animated TV shows like Sid the Science Kid. YouTube example of the new Jim Henson Creature Shop real-time 3D virtual puppetry: What has not been fully explored yet with a 3D character, however, is the improvisational freedom given to real actors in multiple "live" takes. I believe this is the next step required to fulfill the promise of digital actors, by allowing the director to do a series of realtime simulations with both digital and human performers interacting together in realtime. My projects hope to begin exploring this dynamic in a simple, limited manner with mocap and programming, which can then be expanded in the future with more sophisticated AI techniques, and possibly robotics. From Film to Game Engines and Machinima Of course, computer games are another major source of innovation for developing real-time 3D characters using mocap, and games technology may be the best way to realize this type of project. There is new game development software being developed, called Ecstacy Motion by BrokeAss Games, that leverages the Torque game engine to allow interactivity, including rag doll physics simulations, with real-time streaming mocap data on 3D characters. In addition, this technique explores a new type of media production called machinima that is based on the use of a game engine to produce dramatic content by running real-time simulations with 3D characters in a virtual environment. Machinima, a hybrid combination of machine with cinema and anime, is arguably the next evolution of cinematography or movie making by extending traditional independent film and animation techniques into the digital medium of computer processes and coding. Film makers such as Steven Speilberg in the movie AI are already experimenting with machinima techniques in pre-production as a creative tool to block out scene action as a 3D animatic. My goal in this project is to explore the classic theatre technique of mimesis in this new digital medium by using machinima techniques to re-imagine and re-create the Mirror Gag. In addition to the novelty of digitally mutating this classic film representation of mimesis, I will also be exploring new applications of machinima techniques by using them on a live stage set with a live performer in a mocap suit. This has the potential to add another dimension of interactivity, by allowing a live embodied performer to communicate with the synthespians through mocap data, to further enhance th hybrid art form. YouTube video of Ecstacy Motion software streaming mocap data live into the Torque 3D game engine at SIGGRAPH 2009... Exploring the Mimesis Dynamic Process When looking at creativity or improvisation as an evolution dynamic, for instance, creative and purposeful innovations in technique can be seen as a type of cultural symbiogenesis or transposition which may ultimately spawn new forms, or a speciation of technique. The “magic mirror” effect of mimesis as presented here also creates game-like competition. In the mirror gags the main performer tries to “trick” the reflection into revealing its nature, while in a dance “battle” both performers must match their opponent’s movement while also surpassing it with original content. The reflexive desire to win in both situations drives the loop, like facing mirrors that produce infinitely fractal reflections, which in turn escalates and changes the reflection each round. Additional Theatre Influences to Explore For instance, both projects can be viewed as comedies of the mechanical (with a hint of the uncanny), while at the same time are tragedies as we recognize that the human performer is ultimately going to lose in the game against the “uber marionette” or evolving cyborg, because we simply cannot evolve at the same rate as technology, either genetically or memetically. I also intend to explore the opposite dynamic, or the mechanical aspect of human movement, which can also be uncanny, and relates to Bergson. Uncanny YouTube example of a human "robot" dancing:
Another aspect of the uncanny in theatre is the use of masks. To further blur the line between live performer and digital puppet, bringing out the cyborg quality, I will experiment with the live performer wearing a full-face mask, similar to a Venetian Carnival or Japanese Noh style mask, and model the same mask in 3D for the digital character as a face. By covering the human performer's face with a mask, it emphasizes the mirror effect, while also allowing for lighting and movement to create the facial mood, rather than having to animate facial expressions. An additional possibility would be to have changes in the digital lighting control real theatrical lights on the set through a serial connection. Here are some examples of masks that show some uncanny qualities:
Click to read a study on how lighting can change the mood of a Noh mask...
Interactivity Design Reference: Interactivity Design: A library of thematically related theatre or dance movements would first be motion captured into computer memory and categorized. These clips would represent learned movements in a human performer’s memory, which would then be algorithmically blended into the live motion captured movement of the represented performer. For instance, in the case of the “magic mirror” mimesis, a mirrored version of the captured data will be applied to the 3D character with no delay, in a Cartesian space that represents the real space inhabited by the performer across a 2D plane that represents the mirror surface. Minor variations will be created in the repeated movement by randomly blending small values into the captured data, so that it is barely noticeable to the performer. This is meant to catch the performer’s attention, and can be done in Motion Builder using custom Python scripts. In addition, major variations in the movement will be implemented when the performer turns away from the mirror a specified amount, causing the 3D character to move unexpectedly. This will also be done in Motion Builder through Python scripts by querying the head rotation data, and running conditional statements that retrieve a random pre-recorded motion clip. If the performer turns back towards the mirror, the character will resume following with only minor variations. The performer will only be able to see the major variations out of the periphery of their vision, so it will hopefully create a perception of anthropomorphism in the performer to give agency to the character. A successful result would be for the performer to engage in a sense of play to “catch” the character moving behind their back. This project conceptually addresses relations of movement, memory and agency between humans and technology. In creating the illusion of time and space in a 3D animation, computer processes and memory can reflect human processes and memory, so that mimesis is achieved, and the controlling agent is blurred in the relationship. A virtual camera will be set up in Motion Builder to frame the top half of the character, and coincide with the mirror surface. This camera view will be back-projected on a proportional screen that has a sheet of clear plexiglass mounted between the performer and screen, so the surface of the “mirror” can be touched by the performer. If using an optical motion capture system, cameras will be placed above the plexiglass, and around the performance space. For this project, only the upper half of the body, head, and hands need be captured, not requiring a full motion capture suit. Since this is an art installation in a gallery, and the performer is a gallery visitor, it is preferable not to use a spandex suit, which is hard to wear. A more practical solution is to use a custom fabricated jacket, hat, and gloves that are easy to wear.
|
||||
|
