Augmented reality

Cheshire Smile and The Pool of Tears

“Well! I’ve often seen a cat without a grin,” thought Alice; “but a grin without a cat! It’s the most curious thing I ever saw in all my life!”

“You know very well you’re not real.”

“I am real!” said Alice, and began to cry.

“You won’t make yourself a bit realler by crying,” Tweedledee remarked: “there’s nothing to cry about.”

“If I wasn’t real,” Alice said –half-laughing through her tears, it all seemed so ridiculous– “I shouldn’t be able to cry.”

“I hope you don’t suppose those are real tears.”

augmented reality 140

when alice went down the rabbit hole, she found out that the world around her was somehow diferent. What she could perceive with her eyes, like the Cheshire cat, became visible or invisible at will. What she could feel with her senses changed in size and proportion: she was almost drowned by her own tears. Reality, the world in front of Alice’s eyes, was turned into something else. Into something more.

Reality is built on our perception of it. When we design interactive systems, we have to deal with our senses and human perception. Trying to perceive many things at once can mean to have many diferent layers of information at once, which is the main quality of augmented reality (AR). This term as used today is used to describe a combination of technologies that enable real-time mixing of computer-generated content with live video displays. It is distinguished from virtual reality (VR) in that VR involves creating immersive environments, while AR creates an annotated, or “augmented” composite based on the real world. (Mullen, 2012) Augmented reality allows us to see both what’s there and what’s not there. Augmented reality blends the real of our environment, with the virtual of what “magically” appears through our panels, smartphones or computer screens.

This type of experience can serve as a window allowing us to look inside the screen into the augmented graphics and data, or a mirror, showing us (usually through a webcam) the environment where the graphics are superimposed (where we aim our device at). The attempt to have more of what we can perceive stems from the desire to extend our human capabilities: see or feel more at the same time. We want to extend ourselves.

141 augmented reality

THE DESERT OF THE REAL

“Reality is augmented when it feels different, not when it appears different.”

–Kevin Slavin

144 augmented reality

ENHANCEMENT

As mentioned in previous chapters, tools allow us to extend our capabilities. With the use of tools and technological developments (like those behind media) we have extended what humans can do. As McLuhan mentions: “All media are extensions of some human faculty–psychic or physical. The wheel is an extension of the foot. The book is an extension of the eye. Clothing an extension of the skin. Electric circuity, an extension of the central nervous system.” (McLuhan 1969: 40). As part of my research I noticed that in spite of the current buzz of augmented reality, the underlying principles, to enhance and extend our capabilities in the form of multi-layering of information, have existed for a long time, for instance, in the use of optics (the microscope or telescope).

We have to consider that nowadays we are not only creating ways of enhancing our body, but we are looking at ways to extend our mind: “While the industrial revolution produced tools to augment the body, such as the steam engine and the automobile, the information revolution is producing tools to extend the intellect.” (Reas 2012: 17).

When we have more layers of information, like in the case of augmented reality, we can make more informed decisions. However, we can also fall in the trap of thinking that the symbols used to communicate that extra information is reality itself. In the contemporary context of the extensions that augmented reality ofer we need to be aware of the following bias: “mistaking the symbol for the thing symbolized while endowing the symbol with properties it does not possess.” (Hall 1977: 29). We need to remember that the map App is not the territory.

HISTORIC TRAIL FOLDOUT

146 augmented reality

TRANSPARENCY

When we try to see more of what is perceived through our senses with the aid of augmented reality we usually do it through a screen. These function as windows into where the information is. In this case, interaction designers have aimed to make it as simple and transparent as possible in order to concentrate on the information, so much so that the “window” disappears.

The desire to see through in visual representations of reality can be traced back to the discovery of linear perspective. Fifteenth-century architect Brunelleschi began to draw with perspective. This way of representing what he saw carried on well into the nineteenth century. This is relevant when attempting to augment our reality because: “Paintings, like digital applications, offer an experience, and perspective painting offered the same experience as the one now promised by virtual reality–the experience of “being there”.”(Bolter & Gromala 2003: 36). The desire for transparency has been carried culturally; from painting frescos to other forms of representation, like photography, computer graphics and interface design.

Aiming for an interface design that just disappears has certain implications. There are situations where perfect transparency would be a dangerous mistake. Certain contexts (like dealing with extreme and dangerous materials through visual representations on a screen) need for us to be aware of the real environment or else we can end up thinking that the graphical representations are the real thing: “The myth of transparency fosters the user illusion that the interface needs in order to be compelling. It’s not untrue, but it does simplify and exaggerate.” (Bolter & Gromala 2003: 49). As interaction designers we need to know the context we are designing for and be aware of the bias of the medium we use in order to make the best design choices.

STAGE OF TRANSPARENCY Goal Metaphor Response by user

information delivery interface as window look through interface

STAGE OF REFLECTIVITY compelling experience interface as mirror look at interface

In augmented reality we see the transparency of the window that started with Brunelleschi’s discovery of linear perspective, and also the reflection of our own world in a mirror-like fashion. It seems novel because usually these two modes (reflective and transparent) were not usually seen together. In creating augmented reality from this point of view, designers “try to do with pixels what Brunelleschi tried to achieve almost six hundred years ago with paint.

The pursuit of transparency is endless, because transparency is redefined with each new technology.” (Bolter & Gromala 2003: 52).

As part of my research I looked into applications of augmented reality. I discovered that some appealed more to the window metaphor than others, resulting in different reactions. I realized that in order to design with augmented reality through a screen it is important to consider both the reflective and the transparent qualities of it.

147

WINDOW AND MIRROR STAGES

augmented reality

Obtained from Windows and Mirrors: Interaction design, digital art, and the myth of transparency.

Bolter & Gromala (2003), p. 67.

GILDE AR /CASE STUDY The company Gilde launched an advertising campaign in the fall of 2011, using AR. It aims to augment the viewer’s reality, instructing what you have to do: the poster with a marker for your smart phone directs you to an online store where you can download the application. After doing this, it shows an animation of a boxing glove smacking you with “the flavor” of bacon.

augmented reality

149

150 augmented reality

In this case, our reality is augmented through the use of a smartphone screen. This screen functions more like a window than a mirror: allowing us to look into the flavor of ham or bacon. When we use technology because it’s the latest trend, we tend to forget meaning, and the real feedback loop that can generate surprise, ending up with a gimmick: we are using emergent tools with a linear way of thinking. A mindset that is based on a one-way input, that’s rigid to use and doesn’t take context into consideration.

augmented reality

151

152 augmented reality

Step by step reconstructions of Alberti’s perspective method.

Obtained from The Mirror, the Window, and the Telescope by Samuel Y. Edgerton (2009), p. 121.

Leon Battista Alberti was an italian architect and painter, and he build up from Brunelleschi’s work. He wrote a step-by-step method to achieve linear perspective in the form of his “window”: a frame gridded with string that allowed to replicate the real world in the paintings, proportionally. This allowed to create reality as he saw it.

It is interesting to see how the term window is used today as well on our computer screens. The naming of it reminds us what the designers intended us to focus on: “The word frame reminds us of the interface, while the word window helps us to forget the interface and concentrate on the text or data inside…Interface designers were depending on Alberti’s powerful cultural metaphor.” (Bolter & Gromala 2003: 42).

As part of my research I looked at examples in interaction design, as well as projects that focus on the future of interaction design using augmented reality.

153 augmented reality

FROM ALBERTI TO VICTOR

154 augmented reality

Screen shots of Microsoft’s Office Productivity Future Vision film (2011).

The pursuit for transparency and the window metaphor alone, as Victor says, can stop designers to address the interaction not only between us and the machine or system (our finger with the glass) but with each other.

155 augmented reality

We can see how Alberti’s metaphor is strongly rooted in these examples of interaction design: the people pictured here see through the device as they would through a window, sometimes forgetting the context they are in, limiting the interaction to a finger swap. Bret Victor writes about this vision of the future of interaction design, that appeals strongly to the myth of transparency and the window metaphor. As former user interface designer for Apple he could experience first hand, as a designer, what this vision holds: “This mindset can also be used to reflect and question the approach we give to interactivity: more of the flashy gimmicks, or actually addressing the core (the story, the idea, the presence) of what the interaction is all about?

I would rather do the latter.” (Victor, 2011).

156 augmented reality

Through participation I discovered that the real world as our canvas can connect people in a stronger way. Investigating the type of interactions that emerge from having people connect not only through a screen or window, but also with each other led me to realize that sometimes the interface needs to be reflective as well, and not only act as a transparent window into the information, but also as a mirror, where we can see ourselves in a new way.

This reflective quality has been explored as early as the Reinassance. The painter and architect Brunelleschi made a hole in the center of a painting of a building. Then, he placed the painting facing outwards, so that it could be perceived through a mirror. He did this standing in infront of the actual building that he painted on that canvas, and in this way he “permitted viewers to believe that they had penetrated the very “enigma” of the mirror, to see both the virtual reflection and actual Baptistery “face to face” behind the reflection.” (Edgerton 2009: 49) Perhaps by appealing to Brunelleschi’s mirror we could go beyond what interaction design for the future announces to be?

Drawing of Brunelleschi viewing his perspective picture in a mirror.

Obtained from The Mirror, the Window, and the Telescope by Samuel Y. Edgerton (2009), p. 53.

157

When I started my pursuit of creating an enchanting experience I wanted to find out what interaction meant in the context of interactive installations. The following definition gave me an insight into what we mean when we refer to interaction in design: “Interaction is the exchange of information between two or more active participants” and it involves “designing a system of some sort that a person can interact with in a way that is meaningful to them.” (Noble 2009: 3). Interaction in this sense means being an active participant, and creating feedback that in turn will affect the system to a degree that it allows it to give the participant information back. The way we think of interaction will affect the way we work for interaction and design for it. By becoming aware of what it entitles, we can make design decisions that go beyond assumptions. As Bolter and Gromala mention: “as designers, we want the interface to disappear for the user for part of the time, but not completely and not irrevocably.” (Bolter & Gromala 2003: 53). With this in mind we can design an interaction knowing the benefits and bias of transparency and making the best choice for the situation at hand, not necessarily following the latest technological trend: “The current state of the art does not necessarily represent the best possible state.” (Huhtamo & Parrika 2009: 52).

Either from the window perspective of interface design or from the blending of the real with the digital (reflective and transparent) that augmented reality can give, there is a richer experience to be created when we take in consideration our context. The challenge is then, to focus on the specific situation we are addressing, and seeing how a reflective (mirror) strategy, or a transparent one, will affect the design and the interaction. As Bolter and Gromala point out: “Because computer designers so often assume that the interface should be a window, digital art insists that the interface can also be a mirror. And in the process, it demonstrates the other great strategy of digital design.

Those who understand and master both strategies will be more effective designers.” (Bolter & Gromala 2003: 28).

augmented reality

INTERACTION/AR PARADIGM

158 augmented reality

AR Hockey needs the mirror metaphor to work (see my hand as paddle) and the window metaphor to look through to the ball and score.

Creating shapes together through the kaleidoscopic mode of the application Somantics.

My design process for AR Hockey explored play and attempted to blend the physical with the virtual. In what way did it pursue transparency? When was it reflective? I realized that when I developed this project at the stage of super imposing Pong with video from the webcam, I was creating an augmented reality situation: I had the transparent window that looked into Pong, yet also the mirror metaphor was present, as we saw ourselves (and our hands as paddles) on the screen.

The idea for the completed AR Hockey, however, did not involve any screen: it was digital projections on the physical world. This situation is more of a mixed reality rather than the current notion of an augmented reality game. This mixed reality situation, where you use your body and the physical spatial context of your surroundings lends itself to move away from transparency and its bias. It is almost impossible to concentrate on the digital graphics alone, when they are on real stairs, on the wall, on the doors. I realized, then, how much of our experience with interactive systems is mediated through screens, and how much these affect the ideas behind certain interfaces, and thus, the focus on certain content.

I discovered that the interaction not only with the systems in a machine, but with each other, through the systems in a physical space, humanized the technology, or the systems themselves. It had to do with something more than the physical touch (we do touch the interfaces in our smartphones and don’t feel that human-to-human connection).

I discovered that by engaging in an experience in the real, physical world, there was something that is not quite there on traditional virtual or augmented reality applications through only a screen: human presence. n

159 augmented reality

AR HOCKEY IN THE CONTEXT OF TRANSPARENCY