PRESENCE Through The Looking Glass
,ezuag ekil tfos lla tog sah ssalg eht dneterp s’teL“ ”.hguorht teg nac ew taht os
when alice climbed up the chimney and went through the mirror, the silver looking glass, she transitioned from her ordinary world into the world beyond the mirror. She crossed through using her whole body. What was transported was her whole presence, from the real world into the world beyond the looking glass. Presence and tactility are linked. The word presence comes from the latin root praesentia, which translates as ‘being at hand’. Both represent human qualities. As designers when we want to design an interactive system relatable to people, we can use these two attributes. When you are working with a system that is perceived through a material to be manipulated with our hands, the appeal for tactility seems like an obvious choice; for certain design situations it would not be suitable to use your whole body in order to interact. But when we want to create an enchanting experience, an interactive full scale installation that uses our bodies to trigger the system, presence seems like a choice with more potential. In this situation, as Victor (2011) says: “with all the body at your command, do you seriously think the Future of Interaction should be a single finger?”
163 presence
BEING AT HAND
“Experientially, there never will be a simulated alternative to actually ‘being there’.” –John Thackara
166 presence
BODIES IN MOTION/BODY MOVEMENT AS DESIGN MATERIAL
Interactive experiences through body movement rely heavily on presence which is best represented in the world by our bodies. In order to create an interactive experience that uses body movement, we need to explore movement as design material. These explorations can be traced back to the work of Étienne-Jules Marey and Eadweard James Muybridge in the late 19th century. They both captured movement with the technology of their time: camera and film. Muybridge studied animal locomotion, horses in
Marey’s ‘The Horse in Motion’ is an example of studying movement in time. Thi study of motionhelped debunk the idea that horses’ legs never left the ground while riding.
Following the idea of ‘when you have to have it, you build your own’, Muybridge created his own chronophotographic gun. He devised his own tool to pursue his interest in capturing and analyzing body movement.
Using this black suit and white stripes, the ‘skeleton’ in body movement allowed to see how joints moved and twisted. A conceptual precursor of OpenNI, the application I used for my final piece that detects and tracks the body as a skeleton.
167 presence
paticular. Marey’s Le Mouvement explored the way bodies moved through time, by putting on a dark suit with white joints and skeleton, and using multiple photo shots per second to see how they moved. This is the beginning of motion capture, used now with more precise technology to track bodies and super impose, for instance, computer generated animations. Throughout their research, both Marey and Muybridge explored the construction of body movement and how it changed over time.
presence 168
presence
169
170 presence
BODIES IN MOTION/VIDEOPLACE
Body movement and presence in interactive systems were explored in the 1970s by researcher and computer artist Myron Krueger. He created Videoplace: an interactive system that responded to people’s movements and actions in realtime. It used projectors, video cameras, screens and special purpose hardware (it was created before the advent of the personal computer). Krueger was playing with the mirror metaphor: Videoplace captured in real-time body movement that was projected to a screen allowing participants to experience themselves in a new way. In Videoplace people could see how their bodily presence afected an interactive system, and perhaps Krueger’s biggest discovery was that the responsiveness of the system based on real-time interaction between the participant and machine is a medium in its own right (Salter 2010). Videoplace pioneered the use of presence and movement as design material through people’s participation in order to interact with the system.
Videoplace allowed participants to experience themselves in a new way, revealing them as creative actors.
Some of the graphics on Videoplace were made up from colorful shadows. This visual language has been repeated vastly in contemporary interaction design pieces.
171 presence
Videoplace allowed people to create shapes together: the system allowed them to innovate and experience a unique response every time.
172 presence
“The only aesthetic concern should be the quality of the interaction, which may be judged by general criteria: the ability to interest, involve and move people, to alter perception and to define a new category of beauty.” –Myron Krueger
presence
173
174 presence
A MIRACULOUS DEVICE/MICROSOFT’S KINECT
One of the reasons Videoplace was so innovative was the real-time response from the interactive system. The technology needed for this type of body detection nowadays relies mostly on infrared monochrome cameras: they film the body in black and white, and then this information can be used as binary data in the computer. Infrared cameras were first developed and kept as academic research. They were not afordable and thus not very accessible as design or artistic tools. This changed quite recently as a motion sensing device used for gaming came into the market that could function as an infrared camera. In 2010 Microsoft released its gaming console Xbox 360, and with it the Kinect: a device that has an RGB camera, a depth sensor and a multi-array microphone. The Kinect detects and tracks the player’s body in order to use body movement to play the game, rather than using the gaming control.
presence
175
176 presence
By hacking the Kinect and using it with a computer instead of the gaming console, artists and designers could incorporate it to be used in interactive experiments. The way the Kinect is built, paired with the appropriate software, allowed to erase decades of problems with computer vision in interactive works. Currently with a cost of under a hundred dollars, (around 800 NOK), the Kinect became an afordable tool for artists and designers. Even though the technology was not new, the fact that it was accessible made it anew: shiTing it from academic research to means for artistic expression. This device also comes with certain limitations. It is not accurate at detecting hips, or body movements below the waist. It was created for a gaming situation: to be on top of a TV screen in a living room while the gamers sit and play video games. This is also why Kinect’s range is of maximum 3 meters at an angle. As designers when using this tool to create interactive experiences we need to take into account these constraints.
3 m.
Microsoft’s Kinect in the environment it was designed for. It’s range is up to 3m. at an angle.
Clouds is an interactive documentary film that uses the kinect to generate this type of cinematic language, in real-time. Their creators, James George and Jonathan Minard give workshops showing people how to use this technique: RGBD Toolkit, an open source tool for exploring new ways to see.
177 presence
I started coding and exploring the use of Kinect with Processing using Daniel Shiffman’s library. The Kinect is in a fixed position, yet it changes point of view in real-time.
presence 178
179 presence
Sync As part of my fieldwork I met Lise Hansen who is doing a PhD at AHO (Arkitekturhøgskolen i Oslo), researching full body movement as design material for interaction. I’ve had discussions with her and observed her research process. I also have been exploring the tool she has co-created and is researching, called Sync. Sync “allows for variety of visualisations of both x,y,z points in space that may represent body parts or joints, or links between these. A simultaneous video feed enables comparison of visualizations…(it) is a tool to show what data we “have” (or rather for us to see what the computer registers) thus allowing designers and artists alike to work with movement and build interactions upon the possible data readings.” (Hansen, 2012: 2).
presence 180
The source code for Sync is available at Github: https:// github.com/HellicarAndLewis/Sync
By exploring Sync and looking into the design process behind it, I have learned to take into account the presence that body movement conveys. I have observed that there are design processes that run simultaneously when creating a tool at the same time as developing the idea. Some of these revolve around considering and teasing out your material (movement), think of the context of use and experiment with movement yourself to get insights into what it could do. To design for movement it doesn’t sufice to imagine it: you need to move yourself. “If one truly likes to design for movement-based interaction, one has to be or become an expert in movement, not just theoretically, by imagination or on paper, but by doing and experiencing while designing.” (Hummels 2006). It is like juggling: it doesn’t sufice to know in theory how it is done, you have to practice it yourself: “if you have ever juggled yourself, or tried to do so, you know that imagining is not the same as experiencing... we believe that the same holds for design.” (Hummels 2006: 677). I have experienced this type of insight while testing Sync myself. I have observed and learned from a project that deals with interaction design and body movement. For my research and practice this has resulted as a great source for learning how to design for body-movement based interactions and as a reference for my final piece, as I used presence and body movement as material.
181 presence
Hansen has studied what characteristics of movement (like frequency, for instance) can be attained as data and visualized as ribbons, lines, or circles. As a person moves, the designer can see the data generated by movement as well as the visualization it creates. This allows for designers to detect patters in movements through forms rather than just numbers, and then create design situations or solve problems that deal with this information. The relevance of this research relies on the fact that there is no tool like this that takes into account the nuances of body movement and translates it into data and then translates it back again as visualizations that designers can work with in real-time. This research is creating new ways to see, rather than new things to look at.
presence 182
183 presence
Sync responds in real-time to the participant’s movement creating visualizations of the joints of the participant’s skeleton, and how they move in time. This visualization is using the horizontal line mode.
presence 184
presence
185
186 presence
Two different visualizations, circles and horizontal line, of the same movement.
presence
187
Earlier version of Sync. Ribbon visualization. Bottom right: video feed and skeleton calibration.
192 presence
RAIN ROOM /CASE STUDY An example where presence afects an interactive system is Rain Room, an installation by Random International. When you enter the space, you expect to get wet by a downpour. Instead, you move around without getting wet. This shiT in expectations creates an immediate sense of surprise and enchantment. As we move in the space it becomes impossible for us to get wet or grasp the rain. We can fully understand the efect of our presence in the system. This project takes something from the real world (rain) into the realm of the digital (a complex interactive system), but it doesn’t stop there. We don’t see the rain on a screen or in a digital environment. It goes back to the real world in the form of this ungraspable rain. This shiTfrom the analog to the digital and then back to the analog creates depth in an interactive experience.
presence
193
presence 194
presence
195
presence 196
presence
197
198
RAIN ROOM IN CONTEXT
presence
I plotted this project and other case studies in a depth/ width chart, in order for me to analyze the attributes of projects where participants experience a stronger link between the piece and their presence. Projects get more interesting when they make a leap from the digital back into the real world. Here is where I placed my pilot project AR Hockey. Yet projects create real awe and surprise when on top of bringing a situation from the digitial into the real world (surfaces, space, tactility), they also have a message or a critique. We could say that on the width spectrum projects tend to be gimmicky, whereas closer to the depth spectrum, projects tend to be enchanting. There are things to be gained and learned from width, like taking something we know (an ordinary experience) and use new tools to play with their representation in the digital realm. Width can be an indicator of things that resonate with people or that are popular. I asked myself, how to go from width to depth? It could be honing into a certain aesthetic that diferenciates a project from all the others. It could also be having something to say, either about the medium itself or a message that with people’s participation generates meaning.
DEPTH/WIDTH FOLDOUT
200 presence
BODIES IN CODE
In the case studies mentioned before, we find that the body and the presence it holds is used as design material through a system. This is what Mark B. N. Hansen calls bodies in code: “By this I do not mean a purely informational body or a digital disembodiment of the everyday body. I mean a body submitted to and constituted by an unavoidable and empowering technical deterritorialization–a body whose embodiment is realized, and can only be realized, in conjunction with technics.” (Hansen 2011: 20). When using our bodies in an interactive system we experience how they rely on each other to make the design or piece function (like our bodies on screen changing in real-time as we move or the absence of rain as we move through the downpour in Rain Room). Hansen marks the uniqueness of the situation where bodies in code are in place: “Three important consequences flow from this distinction: first, the body is always in excess over itself; second, this excess involves the body’s coupling to an external environment; and third, because such coupling is increasingly accomplished through technical means, this excess (which has certainly always been potentially technical) can increasingly be actualized only with the explicit aid of technics.”(Hansen 2011: 39). We find then, how as participants, our presence linked to the interactive system makes us experience our bodies and each other in a novel way.
When interacting with a system through a screen we encounter a play between our body schema (our physical body, or as Hansen writes: “the embodied organism”) and our body image (the image of our body reflected in a mirror). We gained agency when we looked at the mirror and realize it was us being reflected. This is what Lacan referred to as going over the mirror stage: realizing that what you do in your physical body afects the body in the mirror: the reflection is yourself. When using a screen we rely on the body image, the mirror reflection of what our body is doing in the present moment to see the efect through the system. We know that the movements are made in the body schema, but we don’t look at our real bodies to see the efect, we look at the screen. When we don’t have screens at all and rely on our mere presence to afect the system, like in the case of Rain Room, we rely on our body schema, the physical body, both for cause and effect.
EMBODIMENT (BODIES IN CODE) BODY SCHEMA
BODY IMAGE
Situation
projection mapping on bodies or ungraspable rain
graphics on video feed (mirror/window metaphor)
Reaction
look at own body
look at screen
201 presence
BREAKING THROUGH TO THE OTHER SIDE
presence 202
This observation based on the case studies in this chapter made me realize that the type of experience I want to create is an experience that doesn’t happen on the screen or mirror alone, in the body image, but that is experienced in our body schema, our real physical body as well. In this way, presence could be really a factor in creating an enchanting experience. It is in this area that I situate my final piece. n
203 presence
When we interact with a system through a computer screen we are looking through the transparent window: we become disembodied: our body replaced by an arrow cursor, while the individualized sale is replaced by a computer software program or a set of rules (Bolter & Gromala 2003). Here our experience is mediated through a screen. Our experience is highly dictated, then, by what we see there (the content) and by the attributes the screen possesses: its size, weight, and what it allows us to see: whether relying on transparency, so we see through, or relying on reflections, seeing ourselves (body image). Presence is best perceived to afect the system when we inhabit the space in between the window and mirror. When we actually are situated in a place where the screens are past us; where there are no screens at all.