AR[t] 5 by AR Lab

may 2014

Augmented Reality, art and technology

How will we do it? A look into the future of 3D animation and AR

Wim van Eck

Towards Hybrid Disciplines in a Postdigital World Isjah Koppejan

The Great Pig in the Sky Interview with Theo Botschuijver

AR[t] Magazine about Augmented Reality, art and technology

May 2014 2

Interview with Theo Botschuijver, Page 12 ď&#x20AC;°

Documentary of Jeroen Visser: Theo Botschuyver: art, science, recreation (1986), photo by Roel Bazen

Colophon

Contributors

ISSN 2213-2481

AR Lab & Partners

Contact

Wim van Eck, Edwin van der Heide, Yolande Kolstee,

The Augmented Reality Lab (AR Lab)

Maarten H. Lamers, Stephan Lukosch, Robert Prevel,

Royal Academy of Art, The Hague

Hanna Schraffenberger, Jouke Verlinden

(Koninklijke Academie van Beeldende Kunsten) Guest Contributors Prinsessegracht 4

Rosen Bogdanov, Florian Heller, Isjah Koppejan,

2514 AN The Hague

Heide Lukosch, Marcello Gómez Maureira,

The Netherlands

Diego Montoya, Melo Montoya, Pawel Pokutycki,

+31 (0)70 3154795

Peter van der Putten, Dirk Schart, Carolien Teunisse,

www.arlab.nl

Tamiko Thiel, Alexander Verbraeck, Dorien Zandbergen

info@arlab.nl

Graphic design Editor-in-chief

Esmé Vahrmeijer

Hanna Schraffenberger

printing Editorial board

Media Krachtcentrale, Rotterdam

Mariana Kniveton, Laetitia Kolstee, Yolande Kolstee, Jouke Verlinden,

CoVer

Reba Wesdorp

Boooooom Photography: David Joosten Designer: Manouk Hasebos Make-up & Hair: Joëlle Romita Model: Wouter @ Republic Models

www.arlab.nl

Table of contents welcome to AR[t] Virtual IllusionS What Psychedelics, the Counterculture and Virtual Technologies have in common Dorien Zandbergen The great pig in the sky Interview with Theo Botschuijver Hanna Schraffenberger & Jouke Verlinden Illuminating Shadows Intuitive Interaction with Projected Augmentations Marcello G贸mez Maureira & Carolien Teunisse How will we do it? A look into the future and the past of 3D animation and augmented reality Wim van Eck Augmented Self Some thoughts on augmenting ourselves Yolande Kolstee The AR Curse Maarten H. Lamers Hello Plant! Deepening Human Connections to Plants by Sonic Augmentation Rosen Bogdanov & Peter van der Putten

06 08

12 20 30

36 40 42

Towards hybrid disciplines in a postdigital world Reflection on the future of art, science and tech in arts education Isjah Koppejan

Site Venice Site Biennale: The Manifest.AR Augmented Reality Intervention into the 2011 Venice Biennial Tamiko Thiel

Hitting imaginary walls, pulling virtual strings What augmented reality can learn from urban dance Hanna Schraffenberger

There is more than meets the eye in Augmented Reality Game Environments Stephan Lukosch, Heide Lukosch & Alexander Verbraeck

Corona: Audio AR for historic sites Florian Heller

Developing augmented reality applications: A future perspective Robert Prevel

Hyper-Mondrian Dirk Schart, Diego Montoya & Melo Montoya

The so-called Augmented Reality Pawel Pokutycki Radioscape, Wormhole Dordrecht and Beyond On the Sensation of Presence Edwin van der Heide

92 96

WelcomE... to the fifth edition of AR[t], the magazine about Augmented Reality, art and technology! Two and a half years ago the AR Lab, a collabo-

do it’ focusing on the future and past of 3D ani-

ration between The Royal Academy of Art, Delft

mation and Augmented Reality. He looks at how

University of Technology and Leiden University,

software has developed throughout the years and

started the AR[t] magazine series with the ambi-

uses this expansion to see where we are heading.

tion to compile and design an inspiring magazine

Hanna Schraffenberger continuous with her inter-

for the emerging AR community inside and out-

view articles. For this issue, she has interviewed,

side the Netherlands. Our experience and influ-

together with Jouke Verlinden, designer Theo

ence in the field grew over time, and our various

Botschuijver. Two former US presidents are also

contributors have written thought provoking and

featured in this issue: John F. Kennedy in Maarten

sometimes outspoken articles about AR. In all our

Lamers’ article ‘The AR Curse’ and Benjamin

issues we have shared our interest in Augmented

Franklin in my article ‘The Augmented Self’.

Reality, discussed its applications in fine arts and

The other articles cover, amongst others, critical

provided insight into the underlying technology.

views on Augmented Reality, artistic approaches,

We are proud of what we have established with

natural phenomena that are combined with the

the magazine in such a short space of time and the

latest technology and we learn what Urban Dance

international audience we have reached.

can teach us about Augmented Reality.

In this fifth issue of AR[t], we look at AR from vari-

Our thanks goes out to all researchers, artists

ous perspectives, however, the articles do have a

and lecturers at the AR Lab (whether based at

common denominator: the future of Augmented

the Royal Academy of Arts, The Hague or Delft

Reality. In some of the articles the future of AR is

University of Technology or Leiden University)

explicitly outlined, in others it is more implicit.

and all other authors from all around the world

We hope that this collection of articles guides

who have contributed in this and previous issues.

you in prospective projects. Stop predicting,

Moreover, a special thanks to you, the reader of

start producing: the future of augmented reality

AR[t] magazine.

is bright! I hope we keep sharing our experiments and ideas In random order, I would like to give you a quick

about AR and other intriguing, new techniques.

impression of some of the articles you will find in

We look forward to meeting you somewhere in

this issue.

the near future, in one reality or another!

Wim van Eck continues his series ‘How did we

Yolande Kolstee, Head of AR Lab

do it’, but it is now turned into a ‘How will we

Virtual Illusions What Psychedelics, the Counterculture and Virtual Technologies have in common by Dorien Zandbergen

In Spike Jonze’s latest movie Her [1], the lonely Theodore develops a relationship with his Operating System Samantha. After an initial period of romance, the two alienate from each other. Yet, different from human-to-human relations, the cause for this alienation is that Samantha really is – in her very being - alien to the human Theodore. While for Theodore, Samantha is the One and Only, Samantha has thousands of simultaneous connections with virtual other entities. And where Theodore relies on verbal communication to express himself to Samantha, Samantha is at a loss of words when she wants to explain to Theodore what happens to her while she is rapidly evolving into something else. Samantha’s capacity for connecting, learning, developing and evolving is so big, that she can’t rely on the lim-

people’s living rooms, voice-controlled personal-

ited repertoire humans have available for com-

ized digital assistants that seem able to read peo-

munication. In this she finds companionship with

ple’s minds, and the normalized and seamless use

the uploaded spirit of a certain Alan Watts, who

of small cameras, earphones and microphones as

will be further discussed later on.

habitual extensions of the human senses.

Jonze’s movie is set in a near future city modeled

Yet, the brief appearance of Alan Watts showed

on a combination of Los Angeles and Shanghai.

me something else. Namely, that our ideas about

The story could be interpreted as most people

the potential of digital technology – such as the

understand the science fiction genre in general:

idea that it can manifest higher forms of aware-

as giving people a taste of the future and particu-

ness - are inspired by cultural domains not com-

larly of the role of advanced technology in this

monly associated with technology. Domains such

future. In this understanding, a movie like Her

as spirituality and political orientation, for in-

has predictive value; it helps people anticipate

stance.

the strange realities of a world more and more characterized by the non-intrusive, intuitive, in-

Alan Watts (1915-1973) was a British-born, self-

terfaceless presence of forms of artificial intel-

educated theologian, philosopher and lecturer. In

ligence with emotional and rational intelligence

the 1960s and 70s he was a mystic teacher and a

that exceeds that of people. Jonze shows a world

spiritual leader for the so-called countercultural

of all-immersive games that become one with

movements predominantly in California. The term

Joaquin Phoenix in Her (2013), image courtesy of Paradiso Filmed Entertainment

“counterculture” generally refers to the post-war

and exploitation, and rebuild a new, holistic sense

international movements of students, poets, writ-

of awareness. Zen meditation, brought to Califor-

ers, academics, bohemians and others who felt

nia by Indian guru’s and returning Indian travelers,

united in a desire to reinvent western culture –

as well as psychedelics, yoga, absurdist theatre

away from corporate greed, war-related violence

and encounter groups were among this repertoire

and environmental destruction. Within this cultur-

of de- and re-conditioning techniques.

al milieu, Watts represented the so-called mystic strand: for him, cultural change had to come from

It is less well-known that electrical, electronic

a change in personal perception. Watts regarded

and later digital technologies played similar de-

“society’s official version of reality” as “silly and

and reconditioning roles. In his The Electric Kool-

inadequate.” For him, another type of reality can

Aid Acid Test (1968), American author Tom Wolfe

be experienced that reveals the “grandeur of the

[3] describes how this was the case for a group

cosmos.” (Watts in [2, p. 54]).

of hippies, The Merry Pranksters. The pranksters lived communally in a cottage in a forested area

Those within the counterculture who were drawn

south of Palo Alto, and made cross-country trips

to this type of explanation regarding the causes

in a 1939 International Harvester school bus that

of social alienation, warfare and environmental

they bought from a man with 11 children. They

destruction, embraced a variety of techniques.

had wired their house, the forest around it and the

These would help them “decondition” from social

bus with speakers, microphones and stroboscopes

narratives that preach competition, dominance

so as to create disorienting environments of sound

ics, conventionally known as a theory of digital information systems, formed one of the (semi) scientific backbones of this cultural environment where people tried to imagine the world and the connection between its parts (nature, humans, machines, etc.) in holistic terms. It is in this cultural context that the idea of the personal computer, the World Wide Web, Virtual Reality, and today the Internet of Things, Quantified Self and the Singularity have found very positive and hopeful reception. As Richard Barbrook and Andy Cameron (1995) put it [4], this Californian Ideology has become the dominant framework for interpreting digital technologies, outside of California as well. Whether phrased in explicitly spiritual terms or not, this ideology perpetuates the idea that humanity’s merging with ubiquitous intuitive technologies helps us empathise more with different kinds of reality, connect with a global humanity, helps us expand our knowledge of the world and make us more Biofeedback practitioner. From D.B. Payne and C.T.

creative and efficient.

Reitano, BioMeditation: The Scientific Way to Use the Energy of the Mind (Brookline, Mass.: BFI, Inc., 1977), ii

Yet, where it was the intention of these 1960s and 70s hippies to turn against, what they regarded as, rigid and blinding thought systems – like con-

and light. The culmination of this cacophony was

ventional religion, cultural convictions and norms

the three-night Acid Test held in San Francisco in

– today, this digital optimism has become dog-

1966. Like many other such events at the time, the

matic in its own right. As such, it blinds people to

venue of the Acid Test housed projectors, oscillo-

the very real possibility that the ever-increasing

scopes, music and strobe light. The idea was that

presence of tracing and tracking sensor-based

in this ‘cacophonous’ multi-media environment

technologies creates an orwellian nightmare. And

there was no room for rational contemplation and

to the fact that exploitation, warfare and corpo-

attachment to conventional thought systems.

rate greed simply continue to exist in our digital worlds, yet in ways that are perhaps even more

Ideas that came from this cultural background

difficult to discern. It also stops people from

would later play a role in the Californian enthu-

carefully contemplating how sensor technologies

siasm for Virtual Reality and Virtual Worlds. Also

may alienate humans from their own human sens-

the first non-military experimentations with sen-

es and direct day-to-day environments.

sor-based technologies for motion-tracking or for biofeedback purposes were informed in part by

It is my opinion that we need new types of de-

a similar motive – to decondition from engrained

conditioning techniques to challenge our collec-

patterns of human thought and to get in touch

tive blind faith in the digital. By beautifully show-

with a higher form of reality. Similarly, cybernet-

ing the painful yet insurmountable gap between

Theodore and his beloved operating

Dorien Zandbergen

System, Spike Jonze’s movie could be embraced as one such technique.

References 1. Jonze, S. (Writer, director, producer). (2013) Her [Motion picture]. United States: Warner Bros Pictures. 2. Anderson, W. T. (2004). The upstart spring: Esalen and the human potential movement: the first twenty years.

In 1996, Dorien Zandbergen worked for the helpdesk

iUniverse.

of the Dutch internet provider XS4ALL. Fascinated by the engaged culture she saw emerge around this

3. Wolfe, T. (1968). The Electric

new thing called the internet, she became a student

Kool-Aid Acid Test. London: Black

of digital culture as an anthropologist. Her MA thesis

Swan.

discussed gender dynamics and the political structure of Open Source hackers in a squat in Amsterdam. For

4. Barbrook, R., Cameron, A. (1995).

her PhD thesis, she studied the countercultural con-

The Californian Ideology. Alamut:

text in which information technologies developed

Bastion of Peace and Informa-

and popularized in California since the 1960s. Assisted

tion, Retrieved May 8, 2014, from

by a Fulbright Scholarship, she conducted this research

http://www.alamut.com/subj/

while spending a year in Silicon Valley. Currently, as

ideologies/pessimism/califIdeo_I.

an independent researcher, she is studying the way in

html

which European cities turn themselves into so-called Smart Cities. Following processes of digitization in dif-

Further Reading

ferent spaces in the city, she wants to understand how this works out in practice. This research will be part of a documentary and will give way to the founding of

Zandbergen, D. (2011). New Edge.

a critical cross-disciplinary platform on Smart Cities.

Technology and Spirituality in the San

The aim of this platform is to understand better, in a

Francisco Bay Area. Leiden University.

hands-on, ethnographic way, what the conditions are

Non-published dissertation, to be

in which information technologies can be empowering,

found at: http://dorienzandbergen.

and when they can be disempowering. In all this work,

nl/links/

Dorien likes to maintain a hands-on understanding of techniques and technologies varying from woodworking tools to software.  dorien@xs4all.nl  @DorienZ  www.dorienzandbergen.nl

The great pig in the sky

Interview with Theo Botschuijver by Hanna Schraffenberger & Jouke Verlinden

In my adolescent years, I stumbled upon a documentary on Dutch TV that had a huge impact on my dreams of the future. It featured a funky designer-computer scientist who was enthusiastically creating a pre-Kinect depth camera. He did this by hacking a Polaroid ultrasonic sensor and x-y frames, controlled by an Apple II with a custom-built graphics card. The documentary made my head spin: computers were cool, technology can “augment” the process of creating art … Years later, while I was graduating on VR technology in the US, I found out about this funky designer-computer scientist’s longtime collaboration with VR artist Jeffrey Shaw. Uncanny! They certainly explored the future in the past… If anyone should be consulted about the future of Augmented Reality, it should be him: Theo Botschuijver.

To be honest, I hadn’t even heard of Theo Botschuijver before my colleague Jouke suggested that we’d interview him. After checking Theo’s website [1], it turned out that, although I didn’t know him, I knew many of the works that Theo had realized together with media artist Jeffrey Shaw. More importantly, his oeuvre featured many Augmented Reality pieces – some of them were realized long before I was born. Of course, I was intrigued…. What was someone who had been working with AR in the 1970s doing now and what were his plans for the future?

When we arrive, the door to Theo’s studio is open.

plied practice immediately appealed to Theo. “We

We call out a shy “Hello?” and after some seconds

learned about woodworking, metals; we had theo-

of uncertainty, we enter the studio. Theo’s work-

retical lectures: they were a fantastic five years.

place is spacious; it is filled with his old works, big

We could spend entire mornings talking about

sawing machines, lasers and materials in all shapes

forms and shapes and structures.” As much as

and sizes. The huge studio is also rather cold. We

Theo enjoyed studying at the university, it was the

find Theo and Hannie - Theo’s partner and busi-

practical aspect of it that he enjoyed the most.

ness associate - in a small and comfortable little

“One year of the course was a practical year. Here

side-room. Hannie is filling the stove with wood

it all came together. I worked at a furniture fac-

and Theo’s self-made inflatable windows do a good

tory and at the glass division of Philips and at the

job at keeping the cold outside, illustrating Theo’s

precision moulding department where they built

expertise when comes to inflatables. We take off

the most accurate and fascinating little parts.”

our jackets, get acquainted with Theo’s dog and

Despite the great time he had at Philips, he soon

before we know it, we are entangled in a conver-

stopped with his path towards industrial design. “I

sation about the beginning of Theo’s career.

ended up in England, where I got in touch with a band called Pink Floyd. They were still unknown

Early career

back then, and I enjoyed their first performances. They did not restrict themselves to the stage.

“I would probably have pursued a degree in fine

They wanted to play with the whole space and in-

arts, hadn’t it been for my art teacher in high

volve the audience as well. That appealed to me.

school who pointed me towards a new school in

Those guys were more approachable back then….”

Eindhoven, which had just started and which of-

That was around 1967, the same time when he also

fered a degree in industrial design.“ The combina-

started his long-standing collaboration with Jef-

tion of technical classes, difficult theory and ap-

frey Shaw. “Jeffrey and I met around ‘64/65. He

in Europe. There was a farm close by Eindhoven

Augmented Inflatables – Inflatable Reality

where all kinds of people could work and experi-

Theo’s augmented inflatables are designed to be

ment. In 1967, we started working together, up

presented in combination with a virtual layer that

until 1983. We called ourselves the Event Struc-

is projected onto them. Theo figured out how he

ture Research Group.”

could combine a physical, inflatable object and a

had studied in Italy and wanted to do some things

virtual layer that is projected onto this object in

Walking on water and a smoking octopus

his Talking Head series. “I have several Talking Heads. I project footage of

The Event Structure Research Group is behind

a talking face onto an inflated head. This setup is

many of the early inflatables, such as the ‘Air-

connected to a microphone so that the projected

ground’ (1968), a large inflatable playground and

face moves its mouth when one speaks into the

the ‘Waterwalk’ (1969), an inflatable that allows

microphone. That way, the inflatable head ap-

a person to enter it and walk over water by walk-

pears to speak.” We can imagine that this can

ing in it. “Wait, I can show you!” Theo takes out

be quite entertaining at a party and liven up an

a book in which he keeps track of all his projects.

otherwise straightforward speech. However, we

As the pages turn, we get more and more famil-

are more fascinated by his more conceptual, ar-

iar with his work. There’s commercial work, such

tistic, and sometimes almost romantic, works. For

as amusement park-like experiences designed for

example, Cloud (1970, Event Structure Research

big car companies to promote their latest model.

Group, Theo Botschuijver, Jeffrey Shaw), which

There is political work; an inflatable world-bomb

was exhibited in front of the Stedelijk Museum in

that has circled over the heads of demonstra-

Amsterdam. Like the inflatable heads, the work

tors at the peace demonstration at the Museum

uses projection mapping, and combines virtual and

Square in Amsterdam in 1981. There are sketches

physical elements in the real environment. It con-

for shoes that use piezoelectric materials to shine

sists of a physical, inflatable cloud onto which – at

when you walk. There is an inflatable Octopus.

night – a daytime cloud is projected, accompanied

Most works come with an accompanying story. “It’s

by mist effects and weather sounds.

with knifes. But with time, you learn to anticipate things that could go wrong. We hooked the octopus up to a smoke machine. So that after the attack he emitted smoke from all his wounds. Until someone run off with the smoke machine….” As diverse as his work is, inflatables - most of which Theo made himself - in all sizes, forms and figures form a recurring theme throughout all of his work. If there would be a professorship in inflatable production, his name would be the first to come up. His collection includes figurative ones (the Octupus), useful ones (an inflatable drumset), architectural pieces (domes and buildings), beautiful objects (clouds), some questionable ones (huge breasts that can serve as party decoration) and finally, a collection of augmented inflatables.

 Kim Eshuis, Svp.sfeerbeheer

of the octopus, people were starting to attack it

Talking Head 2 (2002), inflatable human head as a projection screen.

always different how the public reacts. In the case

the city hall while a live band was playing. Then the plastic structure was inflated with smoke, while an array of film and water dye projectors constantly casted movies onto the structure. The audience was very enthusiastic and wanted to participate; some of them even undressed and jumped onto the inflatable. One of our favorite works is “Viewpoint” (Eventstructure Research Group: Theo Botschuijver, Jeffrey Shaw), an installation from 1975 that was shown at the 9th Biennale de Paris, in the MuCloud (1970), Eventstructure Research Group: Theo

see d’Art Moderne in France [2]. The installation

Botschuijver, Jeffrey Shaw.  Pieter Boersma

showed images of several staged events that had taken place in the museum’s space earlier. The

As we explore Theo’s oeuvre further, it becomes

animated slides of these events were projected

more and more apparent that those inflatables

onto a retro-reflective screen. Because of the

are not his only works that have explored key

retro-reflective properties of the projection sur-

concepts of Augmented Reality, such as combin-

face, the events were only visible through a view-

ing the physical and the virtual in real space,

ing console and the screen remained grey from

we mentioned earlier on. In fact, Theo has real-

all other perspectives. The projected images

ized a variety of AR works long before the term

were aligned perfectly with the real space. Con-

‘Augmented Reality’ was devised. An early per-

sequently, a look through the console revealed

formance in 1967 that experimented with archi-

a seamless collage of projected images and real

tectural structures, live action and projections

surroundings. In passing, unsuspecting visitors

was “MovieMovie”. The Event Structure Research

became part of the viewed scenery, resulting in

Group, which at that time consisted of Theo

a combination of real and virtual events taking

Botschuijver, Jeffrey Shaw and Sean Wellesly

place at the same time. “We had staged twelve

Miller, prepared a pitch to bring “something spe-

events. For example, one event showed a visitor

cial” to the Knokke arts festival that year. And

who went to sleep on a museum bench. Another

their proposal was accepted! Theo made a huge

image sequence showed how we built the instal-

inflatable projection dome. Dressed in white lab

lation, the process of setting up the retro-reflec-

coats, Theo, Jeffrey and Sean set up the dome in

tive screen in the museum space.”

MovieMovie (1967), Eventstructure Research Group: Theo Botschuijver, Jeffrey Shaw, Sean Wellesly Miller. An extension of the Corpocinema idea produced for the Experimental Filmfestival in Knokke, Belgium.  Pieter Boersma (left) and Suzy Embo (right)

A few pages of the photobook later, we enter AR

the pig, but they wanted it to look like one of

territory again. At least, the image Theo shows

those Walt Disney pigs or like the friendly pig

us looks an awful lot like an AR headset. As it

that stands in front of your butcher. Pink Floyd,

turns out, that’s exactly what the headset on the

however, wanted an aggressive and realistic pig.

picture is supposed to illustrate. It is a dummy

I made the model pigs for them and a company

prototype of Theo’s idea for a 3D desktop work-

produced three big scale versions.” Unfortunate-

space, which is situated in the real environment.

ly, the team responsible for the production didn’t

In 1986, he extracted the small CRTs from cam-

take the necessary precautions when it came to

corders, and used half-silvered mirrors to project

the photo shoot with the pig. “We let the pig fly

the Apple II’s crude 3D graphics.

in the sky. But even before we started with the real shoot, there was a sudden ‘pling!’ - the ring

To illustrate that the headset is in contact with

that connected the pig to its anchor broke... and

the computer, Theo has connected the headset

there was no safety rip-panel. The pig drifted

to the computer with a cable. “It turns your own

off! It wasn’t our fault, that was the job of the

living room into a memory palace. With the envi-

firm we hired.” Now he can laugh about it, but

sioned device, you can put your documents into

back then it definitely wasn’t funny. “Everyone

your real surroundings, and store them in real

thought it was a publicity stunt but it wasn’t.

space. It will be much easier to find them back...

The pig was floating into the worst possible di-

If I wear the glasses, and I look to the real book

rection. As soon as I saw the pig drifting into the

shelve, I can store files there…”

direction of the airport, I called Heathrow and warned them. They couldn’t believe their ears.

Looking back, looking ahead

“Could you spell that sir?’ – P. I. G., a flying PIG!” – ‘Oh, I see”. They sent a helicopter with a sniper

Theo’s most well-known work is probably the

for the pig. At some point the pig flew so high

huge inflatable flying pig that is featured on the

that it was not considered a hazard anymore.”

cover of Pink Floyd’s album “Animals” from 1977.

Ironically, the pig landed - and got itself injured

“This wasn’t easy. We hired a company to make

- at a pigsty. Theo and the team involved stayed

Early see-though HMD model. Documentary of Jeroen Visser: Theo Botschuyver: art, science, recreation (1986),  Roel Bazen

with the concept of a helium filled balloon with a minimized weight.” We can see sketches of the wing on Theo’s walls. We wonder, whether Theo is still planning on building an actual airwing. “This is a dream, yes, this is a dream. Unfortunately, we would need a lot of money to realize it, or we need to collaborate with a big company. The airwing has to be filled with helium and that’s exup all night, first picking up the pig from the sty

pensive. In the end, I envision it about 200 meters

and then repairing its holes… and finally shot the

in size. But even to realize a small one, approxi-

images for the album cover the next day.

mately 18 by 20 meters would be fantastic. That idea is very attractive to me. The airwing would

One more gig in the sky So far, the flying pig has been one of Theo’s most

be great as a means for transportation. But also as a place to live... I can envision whole villages floating in the sky.”

well-known projects. However, when it comes to the future he dreams about one more great gig in the sky: the airwing - a combination of airship and

References:

airplane. “We used to have two prevalent ways of flying, the airship and airplane. I always found

1. Botschuijver, T., & Van den Dop, H. (n.d.).

it weird, that those were so at odds with each

Spatial Effects, http://spatialeffects.nl/, ac-

other… And after the Hindenburg disaster in May

cessed on 6 May, 2014

1937, the airship era ended. But there are also catastrophes with airplanes and we still have those.

2. Shaw, J. (n.d.). Viewpoint. Retrieved from

Anyway, I have always been fascinated by the idea

http://www.jeffrey-shaw.net/html_main/

of a synthesis between the airplane and the air-

show_work.php?record_id=46, accessed

ship…. to combine the aerodynamics of a wing

on 6 May, 2014

Airwing – a synthesis between airplane and the airship (computer model by Wick van Rij)

Flying pig made for Pink Floyd (1976). Battersea powerstation, London, England. ď&#x20AC;° Phil Taylor, in: Mason, N. (2005). Inside out: a personal history of Pink Floyd. Chronicle Books.

Illuminating Shadows Intuitive Interaction with Projected Augmentations

by Marcello G贸mez Maureira & Carolien Teunisse 21

Light is an important aspect for all types of visual augmented reality (AR) installations. This is especially true for projection-based AR, or ‘spatial AR’, where video projectors are used to create a virtual overlay on a physical canvas. When audience members approach spatial AR installations and enter the path between the projector and the canvas, this usually results in undesired shadows that disturb the illusion. This is particularly problematic in the context of interactive installations. How can the audience physically interact with the display without causing disturbing shadows? One approach of dealing with such negative interference is to facilitate the interaction through separate computer input devices. With Illuminating Shadows we follow another approach: rather than seeing shadows as an undesired artifact, we embrace interference and explore the possibilities that it can offer. Illuminating Shadows is an ongoing research proj-

A working prototype – involving two video projec-

ect that uses shadows as an intuitive means to

tors, a Kinect motion sensor, and a large zebraf-

facilitate interaction in a spatial AR scenario.

ish model - is the test bench for our experiments

Our setup allows the audience to step in front of

in this project. We wanted to design a display

the projector and to interact with the display by

that could find its place in a museum exhibition

creating shadows on the canvas. Their shadows

next to traditional physical exhibits. Without ac-

are used to reveal additional layers of informa-

cess to a fitting model for our project we decided

tion. This is achieved with a secondary projector,

to build our own: the aforementioned zebrafish

which fills in the shadows with extra content. As

model. Now the zebrafish is probably not the

a result, the audience can use their shadows to,

first specimen that comes to mind when looking

look inside an object and reveal additional infor-

for a prototype subject. We learned about ongo-

mation by moving their shadow over the surface.

ing zebrafish research done by the Imaging and

Bioinformatics research group at Leiden Univer-

With this setup, people can interact with the pro-

sity. With the help of Fons Verbeek who leads

jection content by using the shadows they cre-

the research group and also supervised our own

ate when occluding the primary projection. One

research project, we were able to get access to

of the first features we implemented was using

internal images of the zebrafish, making it the

shadows to reveal the inside of the zebrafish in

star of our AR installation.

form of X-ray visualization. Creating a shadow on the zebrafish canvas would display the skeleton

Design and Functionality

of the fish wherever shadows were formed. Subsequent experiments added further visualization

The zebrafish model consists of a wood and wire

layers that could be displayed by moving closer

base mesh that has been wrapped in paper-

to or further away from the fish model. To do so,

maché and painted white, serving as projection

we used the depth information reported by the

canvas. One of the two video projectors – the

Kinect sensor to determine which visualization to

primary projector – faces the fish model from

display on the shadows. In its current form, our

the side and is used to project the outer ap-

interactive display can be used by several people

pearance of the zebrafish onto it. The second

at the same time, leaving the choice of which vi-

projector is then positioned at an elevated posi-

sualization is shown to the person closest to the

tion in such a way that it is unlikely to cross its

model. Since the depth of each person is tracked

projection path.

separately, it is also possible to give everyone their own shadow display. This is an intriguing

In its regular state – that is without user interac-

possibility that we intend to explore in the near

tion taking place – the second projector is simply

future.

inactive. When, however, a user enters the space between the zebrafish model and the first projec-

When using AR technology systems such as AR

tor, a shadow area is formed on the fish canvas.

glasses or smartphone and tablet apps with AR

This area is recognized by the Kinect sensor, which

elements, we are constantly aware of the equip-

passes the information on to a program that con-

ment that enables us to experience the augment-

trols the output of the second projector. Finally,

ed reality. We feel the weight of the phone we

the second projector displays an image on the very

hold in our hands when looking through our screen

same area that is covered by the shadow of the

at the world and we cannot look away from the de-

first projector, creating an ‘illuminated shadow’.

vice mounted on our heads or resting on our ears.

Block diagram showing the interaction between hardware, software, and human user.

Schematic illustration of the augmented zebrafish installation.

Early 3D sketch showing the projection of zebrafish skin on the left, and an X-ray projection on a userâ&#x20AC;&#x2122;s shadow on the right. 24

Usually when using projection-based AR, we would

are naturally curious about the possibilities and

like the equipment to disappear from sight to give

the questions that often start with “Wouldn’t it

the illusion that the projections are part of our

be great if ...”. In other words, we want to experi-

reality. In our project, however, we welcome the

ment and play with the technology to see what it

awareness of the projection technology, at least

can do. Essentially, this is the kind of curiosity we

partly. While the second projector remains an ele-

hope to inspire in the audience.

ment that is ideally hidden from the audience, the primary projector is deliberately on full display. Its

In order to do so, we invite them into the space

position marks the source of both light and shad-

between the projector and the physical canvas,

ow, and is therefore an important element for the

causing them to become part of the augmenta-

interactive experience.

tion and ultimately enabling them to experiment with it. By positioning users in this way, shadows

Seeing Shadows in a New Light

become an unavoidable component of the interaction. It is highly fitting then, that the creation of shadows can already in itself encourage play-

In the first issue of AR[t] magazine, Jouke Ver-

ful interactions. This is especially the case when

linden discusses spatial projection-based displays

bringing several people together to create shad-

in his article “Pixels want to be freed!” [1], and

ows on a shared canvas. While the personal space

mentions occlusion and shadows as known prob-

around a user is often reserved for close friends,

lems in this field. This is certainly a problematic

the perceived closeness between each individual

aspect for many installations. Working with shad-

shadow can encourage a playful atmosphere that

ows as intentional elements does not necessar-

wouldn’t normally occur between strangers.

ily solve the issues. And yet, we cannot help but feel that considering shadows a part of spatial AR

Another way in which we implemented the ele-

could be an interesting opportunity, both as form

ment of ‘play’ is through exploration. While users

of interaction, and as a visual aspect that inher-

are technically able to create shadows on the dis-

ently connects the virtual with the real world.

play that are large enough to influence the complete canvas, most of the time, the shadows only

The Element of Play

take up a small area of it. This causes the shadows to act like dynamic windows reaching into the ex-

The aspect of ‘play’ has been of great impor-

hibition model. Such windows focus the attention

tance in the concept of our installation. Creating

to the places they originated. A user might, for

space for playful interactions holds the promise

example, hold his or her hand in such a way that a

of captivating audiences in a fundamentally per-

shadow is created on the fin of the zebrafish mod-

sonal and emotional way. This makes the interac-

el. Consequently, the fin becomes the focus point

tion, as well as the elements that are part of it,

of the interaction. Seeing all the possible content

all the more memorable for the audience.

requires active exploration of the different areas by changing the shadows on the canvas. Here we

When talking about interaction within the con-

should remember that the distance between user

text of augmented reality, there is a certain

and the zebrafish has an impact on which shadow

playfulness present in many of the most exciting

layer is going to be displayed, which further in-

works in AR and especially in spatial AR. That is

creases the amount of exploration space.

not to say that spatial AR cannot be used in more utilitarian environments. And yet, the use of

There are, generally speaking, certainly more ef-

spatial AR still seems be enjoying its honeymoon

ficient ways to let users access information. Explo-

phase of technological novelty. In this phase we

ration of content, after all, already indicates that a

certain amount of effort is required. Furthermore, it hints at the possibility that some aspects might be left undiscovered. And yet, we believe this to be an absolutely acceptable element of our installation. In our project, playful interaction means that there should not be a right or wrong way in how the model is explored. The user experience is whatâ&#x20AC;&#x2122;s most important here. If users are captivated and enjoy interacting with the model, there is a high chance of them seeing everything there is to see. More importantly, there is a high chance that they will remember it.

Mind Your Step Throughout our project we went through several iterations in which we modified our augmented zebrafish display. Especially when creating a prototype, many aspects, whether technical or Zebrafish (Danio rerio). Image by Okinawa Institute of Science and Technology (OIST)

conceptual, were changed along the way. While

focusing mainly on the installation itself, it can be easy to lose sight of the surroundings, referring to both the environment in which the installation is presented and to the way in which an audience is invited to interact. In the case of our display we had to encourage people to step into the space between the projector and the physical model. The difficulty here is that many people actively avoid occluding a projection path, as this is usually discouraged in projection displays. A mindful audience might therefore entirely miss the possibility of interacting with our display â&#x20AC;&#x201C; a situation that we wanted to remedy as best as possible. We introduced two ways to help users understand that they are welcome to interrupt the primary projection. First, zebra stripes were added that indicate which visualization of the zebrafish corresponds to a certain distance. Here the symbolic nature of zebra stripes is meant to convey the message that passing through the projection area is not only possible but also required for interaction.

Second, a green animation was projected on the

Carolien Teunisse

pedestal of the model when the installation was not in use, showing a figure holding its arm up in

Carolien Teunisse is a media artist

front of a projector.

and student of the Media Technology program at Leiden University. In her

While we have yet to receive feedback on wheth-

early video and animation work, she

er the animation proved helpful, we do know that

looked for interesting dialogues be-

the addition of zebra stripes has actually caused

tween different kinds of media and

some confusion. Here it should be said that our

in her first encounter with a projec-

initial implementation of the zebra crossing was

tor she created animations that aug-

done with paper. In retrospect it makes sense,

mented the environment. Nowadays,

but as it turns out, people do not like to walk on

she is particularly interested in cre-

paper. The result of this implementation was that

ating compelling experiences that re-

users actively avoided the area towards which

flect on interactions between people,

we wanted to guide them. Even after the setup

media and our physical reality.

was improved by marking the area with white tape, we made the mishap of cross-hatching an

During her Digital Video Design

area that should have indicated the existence

studies at the HKU University of the

of an interactive area. While cross-hatching pat-

Arts in Utrecht, she met like-mind-

terns can absolutely be understood as defining an

ed artists who together now form

area, they are also often used to mark areas that

DEFRAME collective. They explore

should not be used.

the possibilities of current media technology by creating immersive

What this small example shows, is that there

experiences in the form of installa-

were not many issues with the zebrafish display

tions and live visual performances

itself. It rather shows how simple modifications

often using augmented projection

surrounding the installation can change how us-

technologies. Teunisse is also co-

ers perceive and interact with it.

founder of Fiber, an audio-visual network platform based in Amster-

Potential Implementations So far, we’ve discussed the conceptual and physi-

dam organizing events and a festival concentrating on the integration of media art and electronic music.

cal design as it was realized in our project. However, in many ways the zebrafish installation is very much a prototype, created for the purpose of experimenting with the possibilities of shadowbased interaction with spatial AR setups. As such, it represents only one of many possible forms of implementation. It is easy to imagine the use of other display content. Starting from the visualization layers, many other types of information could have been used



in the zebrafish display. An interesting example

 DEFRAME.nl

would be to visualize the development stages

 FIBER-Space.nl

Pim Graus

of the zebrafish, starting from the larvae to its

In this respect, it is also interesting to think of

adult appearance. While the physical shape of

different implementation in terms of scale. In

the canvas would then no longer match the dis-

their paper ‘Shadow Reaching: A New Perspective

play content, it could instead serve as a consis-

on Interaction for Large Displays’, Shoemaker et

tent reminder of the scale between the different

al. [2] describe the interactive possibilities of us-

development stages.

ing shadows to interact with very large screens. Since the reach of a shadow can easily be ex-

It is just as easy to think of completely different

tended by moving closer to the light source from

content. Augmenting a model of the human body

which the shadows originate, users can interact

could let users dive into the different anatomi-

with elements that they would not be able to

cal layers or even animate processes by moving

reach otherwise. On the other end of the spec-

closer to the model. Leaving the realm of organic

trum is the possibility to create small-scale ver-

matter, we can also imagine to augment architec-

sions of our installation for the use in classrooms

tural structures and implement shadow interac-

or in situations where multiple exhibition models

tion in such a way that it illustrates the changes

are presented in close proximity to each other.

made through history.

Marcello Gómez Maureira Marcello Gómez Maureira is an interactive media enthusiast who is on an ongoing journey through different professions and fields of research. Originally trained as mechanical engineer, video game artist, and video game designer, he is currently pursuing the Master’s Media Technology at Leiden University as part of his research into designing interactive spaces for audiences. Outside the academic environment, he develops video games and works as freelance game and graphic designer through his company ‘Dandy Unicorns’.  MARO.in

These examples show that there are many ways in which the future developments of shadow interaction in spatial AR might unfold. It is hard to not get excited about the possibilities and we certainly hope that our zebrafish installation was not the last of its kind.

Links  http://www.illuminating-shadows.com A video of Illuminating Shadows can be found at:  www.vimeo.com/94137178

Acknowledgements Over the course of this project we have received the display system as well as in constructing the prototype itself. We would like to thank Fons Verbeek and Hanna Schraffenberger, who supervised the development of this project from the start. We also want to thank Georgios Lampropoulos, Bram Snijders, Livia Teernstra, Meggy Pepelanova and Isabelle Kniestedt for the support they gave us whenever we were in need of a helping hand.

References 1. Verlinden, J. (2012, April). Pixels Want to Be Freed! Introducing Augmented Reality Enabling Hardware Technologies. AR[t], issue 1, 42-59. 2. Shoemaker, G., Tang, A., & Booth, K. S. (2007, October). Shadow reaching: a new perspective on interaction for large displays. In Proceedings of the 20th annual ACM symposium on User interface software and technology (pp. 53-56). ACM.

Zebrafish (Danio rerio). Image by Okinawa Institute of Science and Technology (OIST)

a lot of help, both in the conceptual design of

Image 1. 3-Sweep, image courtsey of Tao Chen

How will we do it?

A look into the future and the past of 3D animation and augmented reality by Wim van Eck In previous AR[t] magazines, I have used the ‘How

3D modelling and animation

did we do it?’ column to give an insight on how the AR Lab creates its projects and which software is

The majority of 3D artists use commercially avail-

used there. Since this issue’s main theme is ‘The

able software such as Maya, 3dsMax, Cinema 4d

Future’, I will look into my perfectly ray-traced

and Blender. In 2004, I was using Maxon Cinema

glass sphere to predict how we will develop our

4D R9 to create my 3D models, while currently

projects in the near future… Or on second thought,

I’m using Maxon Cinema 4D R15. When we look

maybe it is a better idea to have a look at how

at both interfaces we see nothing has radically

software has developed throughout the years to

changed, the biggest changes are cosmetic. Obvi-

see where we are heading, and at the same time,

ously, many improvements were made through-

find out who drives innovation in this field.

out the years; more realistic and faster render-

Image 2. Skammekrogen, photo by Åsmund Sollihøgda with the courtesy of Makropol

ings, better previews, more advanced physics,

fects with existing software. The visual effects

easier character animation tools etc. But these

companies hired to realise these special effects

are all small evolutions; the general workflow re-

have enormous budgets which enable them to

mained pretty much the same. The same goes for

develop add-ons for existing software, or even

most other commercial 3D animation packages.

develop new software to achieve a certain goal.

This actually makes sense though; most profes-

Sometimes, these programs are developed fur-

sional 3D animators don’t have time to constantly

ther and released as commercial software or a

get familiar with significant changes to their soft-

plugin afterwards, a great way to earn back the

ware. Having to learn big changes within your

invested money. The Lord of the Rings trilogy, for

software doesn’t go well with meeting everyday

example, gave birth to two new software pack-

deadlines. Many 3D professionals even work with

ages. Mudbox was developed to digitally sculpt

older versions of their software so they are sure

the fantasy characters, and crowd simulation

all bugs have been removed and their current

software Massive made it possible to realise the

workflow with other programs remains intact. We

spectacular battle scenes featuring thousands of

can be sure that the coming updates of commer-

characters on screen at the same time.

cial 3D animation packages will offer great new options and possibilities, but we should not ex-

The same goes for the computer game industry.

pect the biggest innovations here.

This rapidly growing industry has been largely responsible for the enormous increase of graphical

The movie industry, however, is an incubator for

power of personal computers. To comply to the

many fresh and innovative technologies. Big Hol-

high expectations of the players, the visuals of

lywood blockbusters have immense budgets and

new games have increased incredibly. The latest

their way to lure moviegoers is often to show

game-engines such as Unreal Engine 4, CryEngine

special effects that have never been shown be-

and Unity 5 are able to produce almost photoreal-

fore on the silver screen. This forces them to in-

istic visuals in real-time. Since it is almost impos-

novate, since they cannot achieve these new ef-

sible to create the accompanying highly detailed

Image 3. Project Tango, image courtesy of Google

environments manually, many game studios make

3D models by sketching simple 2D shapes, with

usage of procedural content generation, enabling

the software adding the third dimension auto-

them to algorithmically generate variations of

matically. A recent effort is 3-Sweep [2] (image

game assets, instead of creating them manual-

1), a research project by Tao Chen et al. 3-Sweep

ly. SpeedTree, for example, can generate whole

can derive a textured 3D model from a single

forests of unique trees, and Allegorithmicâ&#x20AC;&#x2122;s Sub-

photograph, only needing some simple input

â&#x20AC;&#x153;While in the seventies games were developed by small teams or even individuals, a current blockbuster game such as Grand Theft Auto 5 is estimated to have a development team of over a thousand artistsâ&#x20AC;?

from the user. I recommend watching the impressive demo on their site since it is quite spectacular to see their software in action. These applications are not often used by professional 3D artists directly, but the underlying technologies might find their way into commercial 3D animation packages at some point. So what will the near future bring? I am certain procedural content generation will play an even bigger role

stance Designer can generate variations of tex-

in the creation of pre-rendered and real-time 3D.

tures. Such techniques do not only speed up the

It simply will have to, since the consumer con-

creation process, but also significantly reduce

tinuously expects larger and more detailed digi-

memory requirements.

tal worlds and characters in movies and games. While in the seventies games were developed by

A different type of innovation comes from aca-

small teams or even individuals, a current block-

demic research at universities. These are places

buster game such as Grand Theft Auto 5 is es-

which allow focus on new technologies and in-

timated to have a development team of over a

teractions, without having to worry if the results

thousand artists. Instead of increasing the size of

are production ready and free of bugs. This can

these teams even further, it would be more fea-

results in innovative approaches to create 3D con-

sible to give the artists tools which allows them

tent. Teddy [1] for example allows users to create

to automatize even more parts of the process.

I also foresee exciting times for real-time render-

recently impressed by Metaio’s new 3D objects

ing. While it is currently mostly used for games and

& environments tracking. Besides better tracking

simulations, movies could also benefit. Since com-

performance, we also started using software that

puters, tablets and phones are getting increasingly

was easier to use. ARToolkit still needed a good

powerful processors, animated movies could be

knowledge of C/C++, while nowadays we can use

rendered in real-time instead of having to play

easy to use game engines such as Unity to devel-

back a huge pre-rendered video file. Current real-

op our projects. Applications such as Aurasma/

time render engines already offer sufficient im-

Junaio/Layar do not even require any program-

age quality, and since the upcoming 4K resolution

ming skills at all.

(Ultra HD) format will result in such huge video files it could make sense to render it out in real-

Augmented reality software developed quite

time. This was actually already quite common in

quickly since its introduction, and these develop-

the eighties, when programmers in the demoscene

ments will surely not slow down for the coming

programmed impressive real-time visuals only oc-

years. Google Glass gave a preview of things to

cupying small sizes such as 4096 bytes.

come, with hardware becoming truly wearable in-

Rendering movies in real-time would also offer

stead of being bulky and uncomfortable. Instead

the spectator the possibility to virtually walk

of mostly relying on cameras for tracking, we

around within the movie, choosing a unique point

see a trend of combining more sensors for more

of view to see the story unfold. Virtual reality

stable and markerless tracking. A good example is

goggles such as the Oculus Rift or Sony’s Project

Google’s Project Tango (image 3), which maps your

Morpheus are ideal to realize such possibilities.

surroundings (and creates a 3D scan) by combining

The Danish short film Skammekrogen by director

data from a motion tracking camera, depth sensors

Johan Knattrup Jensen (image 2) is an exciting

and a regular camera. Being able to sense your en-

attempt at such an approach, where you wear an

vironment’s depth gives (among other things) the

Oculus Rift to see a family dinner drama unfold

advantage that virtual objects can be occluded by

from the successive point of views of the five din-

real objects, creating a much more advanced inte-

nerguests. It is unlikely that traditional cinema

gration of the virtual within the real.

will be replaced soon, but these new technologies invite us to have a fresh new look at how

One of the biggest problems with augmented re-

stories can be experienced, possibly blurring the

ality is that there is still no ‘standard’ application

line between movies and computer games.

to see all the different augmented reality projects. There are in fact some standards, but there are too many of them. The current need to install

Augmented reality

and use many different viewers and applications ruins the whole experience. I don’t have high

Augmented reality software went through quite

hopes this will change in the near future, but a

a development since its introduction. I started

change is certainly needed to elevate augmented

working with augmented reality in 2006, a time

reality to a next level.

when popular augmented reality software such as ARToolkit was using fiducial markers (black and white patterns). While the tracking was already

3D scanning

quite stable, it was still sensitive to low-light conditions and partially occluded markers. The possi-

Professional 3D scanners have become more af-

bility to track high contrast images gave exciting

fordable throughout the years, and free software

new possibilities and also added better low-light

such as Autodesk 123D Catch already gives excel-

performance and more forgiving occlusion. I was

lent results. It is much easier to make a scan of

Image 4. Get Even, The Farm 51

someone’s face instead of modelling it by hand.

in movies such as The Curious Case of Benjamin

But did 3D scanning already start to replace 3D

Button, where a digital representation of actor

modelling? While 3D scanning offers exciting pos-

Brad Pitt was created at different ages. Another

sibilities it also has some big drawbacks. Firstly,

example are the photorealistic environments of

the object already has to exist (you cannot scan

the computer game Get Even (image 4, expected

something fictional), it should be accessible to

in 2015), which were created by 3D scanning real

be scanned and ideally shouldn’t be transparent,

world locations. This innovative approach makes

reflective or dark, as this gives scanning errors

it possible to seamlessly switch between live ac-

with most scanning methods. And once you have

tion recordings and the virtual game.

a 3D scan, it still needs quite some manual work to clean the model so it has a good topology (dis-

The previously mentioned Project Tango could

tribution and flow of polygons) and a reasonable

play an important factor by possibly putting a

amount of polygons. This doesn’t mean that 3D

3D scanner in everybody’s hands, similar to what

scanning is not widely used with often spectacu-

happened when mobile phones became equipped

lar results. 3D scanning played an important role

with cameras. The quality of the scans will in-

crease and open up exciting new possibilities.

times for developers and consumers. Letâ&#x20AC;&#x2122;s stop

Imagine 3D scanning a beautiful location with

predicting and start developing!

your phone while you are on vacation, so your friends at home can experience it virtually wear-

References

ing a device such as the Oculus Rift. This would surely be an great alternative for postcards or

1. Igarashi, T., Matsuoka, S., & Tanaka, H.

digital photographsâ&#x20AC;Ś Or why not 3D scan your ho-

(2007, August). Teddy: a sketching interface

tel room and place it online as part of a review so

for 3D freeform design. In Acm siggraph 2007

people can virtually preview it at home? Sony al-

courses (p. 21). ACM.

ready proposed such functionality during the GDC 2014 (Game Developers Conference) unveiling of its Project Morpheus virtual reality headset.

2. Chen, T., Zhu, Z., Shamir, A., Hu, S. M., & Cohen-Or, D. (2013). 3-Sweep: extracting editable objects from a single photo. ACM

This article has undoubtedly only shown a glimpse

Transactions on Graphics (TOG), 32(6), 195.

of things to come, and these are truly interesting

Augmented Self Some thoughts on augmenting ourselves by Yolande Kolstee

NOW Integrating certain aspects of the virtual world

network of sensor information and data trans-

into our day-to-day activities is taking place at

port working at top speed.

a slow but unstoppable pace. People are getting used to on the spot information that is brought to

To hear an exclamation like the one above is

them digitally. Location based information, like

quite common, but at a closer look, it is also a

density of traffic, weather circumstances or is-

bit weird: a person is looking for dynamic infor-

sues in public transport, is a logical addition. We

mation about the location while the person is

are getting used to the idea of meta-data about

actually at that place himself. Static analogue

our environment and we expect this to be avail-

information carriers, such as books, can pro-

able all day, every day. We might consider it a

vide us with information regarding our location

kind of natural right, or even feel entitled to this

and make us more aware of the special char-

extra information. This kind of awareness of the

acteristics it has; whether in historical, ethical,

existence of meta-data can be seen as a special

physical or cultural sense. The availability of

type of augmented reality. We usually consider

analogue information about a certain location is

augmented reality the addition of real time vir-

also available prior to the visit and will be so af-

tual information. However, certain other types

terwards. Location based information produced

of data addition can also be seen as augmented

by sensors, may be locally available or trans-

reality.

ported to the spot via our global positioning system. Location based information also offers

LOCATION BASED DATA

time dependent, dynamic information – something analogue information lacks.

A location without dynamic digital information available on the spot is nearly unimaginable.

People checking their smartphone for actual

“Hey, I’m here at the Vanderbilt Square and I

information, are not interested in the situation

don’t have any info on the actual weather here!

from 5 or 10 minutes ago. They could access this

What’s going on?”. Having immediate access

information since it is stored in log files, but those

to data about our physical environment seems

files are most often ignored. An app that reports

natural, and not everybody realizes that this is,

traffic jams that have already been solved, is ir-

in fact, the result of an extremely complicated

ritating and will soon be on the to-replace list.

LOGGING BODY RELATED DATA

Benjamin Franklin listed his thirteen virtues as:

If you are familiar with Bridget Jones, the protagonist of Helen Fielding’s Bridget Jones’ Diary [1],

1. Temperance (Tem.)

you should be familiar with lists like this: Mon-

day: 6 units of alcohol, 12 cigarettes. Weight:

2. Silence (Sil.)

75 kilos. The penetration of digital systems that

Speak not but what may benefit others or

measure our own data is increasing every day.

yourself; avoid trifling conversation.

Apps on smartphones give us insight in data con-

3. Order (Ord.)

cerning our body; for example, apps that mea-

Eat not to dullness; drink not to elevation.

Let all your things have their places; let each

sures our heartbeats per minute, our breath fre-

part of your business have its time.

quency, our burned calories based on our activity

4. Resolution (Res.)

(measured in footsteps for example), our sleep

Resolve to perform what you ought; perform

and wake patterns. We can input our weight and

without fail what you resolve.

calculate our BMIs (body mass index), which can

5. Frugality (Fru.)

be split up in fat, non-fat, bones and fluids. We

Make no expense but to do good to others or

can even record the sounds we produce by voice

yourself; i.e., waste nothing.

or other body parts while we sleep. By monitoring

6. Industry (Ind.)

our body, we are in a way augmenting our sense

Lose no time; be always employ'd in some

of ourselves. The quantified-self movement (see

thing useful; cut off all unnecessary actions.

http://quantifiedself.com/) expands on this by

7. Sincerity (Sinc.)

also logging external data like the composition of

Use no hurtful deceit; think innocently and

the air we breathe.

justly, and, if you speak, speak accordingly.

8. Justice (Jus.) Mental and/or emotional performance in combina-

Wrong none by doing injuries,

tion with information about temperature or brain

or omitting the benefits that are your duty.

activity via wearable equipment can be added to

9. Moderation (Mod.)

the image based on our bodies’ data. Although

Avoid extremes; forbear resenting injuries so

people have been tracking their activities and

much as you think they deserve.

feelings in diaries for centuries, life logging is dif-

10. Cleanliness (Clea.)

ferent. It gives an interesting twist to augmented

Tolerate no uncleanliness in body, cloaths,

reality. Let us consider a very nice example by

or habitation.

Benjamin Franklin of what we would nowadays

11. Tranquillity (Tran.)

call mental performance logging as on the website

Be not disturbed at trifles, or at accidents

http://thirteenvirtues.com/ [2]. In 1726, at the

common or unavoidable.

age of 20, Benjamin Franklin created a system to

12. Chastity (Chas.)

develop his character. In his autobiography, Frank-

Rarely use venery but for health or

lin states: “I propos’d to myself, for the sake of

offspring, never to dullness, weakness, or

clearness, to use rather more names, with fewer

the injury of your own or another's

ideas annex’d to each, than a few names with

peace or reputation.

more ideas; and I included under thirteen names

13. Humility (Hum.)

of virtues all that at that time occurr’d to me as

necessary or desirable, and annexed to each a short precept, which fully express’d the extent I gave to its meaning.” [3]

Imitate Jesus and Socrates.

ThirteenVirtues.com brings Franklin's simple an-

In theory, the picture in picture effect continues

alogue system to the information age. You can

deeper into the picture ad infinitum. This self-

track your progress against Franklin's virtues with

monitoring trend, and the fact that we have more

your favourite web browser. Or maybe you don't

images of ourselves than ever before, makes us

agree with all of Franklin's original 13 virtues - no

aware of ourselves in an unprecedented way. The

problem. Just add, remove, or change them so

psychological, social and health effects of this

that you only track what you are interested in.

new type of self-awareness are open for research in the coming years.

AUGMENTING OURSELVES THE DROSTE EFFECT

References

It is not unlikely that in 25 years all kinds of gad-

1. Fielding, H. (1996). Bridget Jones¹ Diary, Pengiun books.

get-like monitoring equipment, like smartphones or smart-watches, are replaced with microchip

2. Franklin, B. The 13 Virtues, http://thirteenvirtues.com, accessed 8 May 2014.

implants and possibly augmented reality lenses. As a result, that which has to be monitored and

3. Franklin, B. (2004). The Autobiography of Benjamin Franklin. Touchstone. In [2].

that which is reading the monitored data is one entity: our self. This creates a strange version of

4. Droste effect wikipedia, http://en.wikipedia.

what the Dutch call the Droste effect [4]: an end-

org/wiki/Droste_effect, accessed 8 May

lessness iteration of watching a picture in a pic-

2014.

ture. We are monitoring ourselves while we are monitoring ourselves…

Sun.

Mon

Tue

Wed

Thu

Fri

Sat

Tem. Sil.

•

Ord.

•

• •

Res.

•

Fru.

•

Ind.

• •

•

• • •

Sinc. Jus. Mod. Clea.

••

Tran. Chas. Hum.

•• •

•

Benjamin Franklin tracked his progress on self-made charts. The days were listed on the top row and the virtue were listed on the left column. If he violated a virtue on a given day, he placed a dot in the corresponding cell.

The AR Curse by Maarten H. Lamers I am not an “AR optimist”. In my opinion, augmented reality offers interesting possibilities, but few exciting realities. The ultimate killer app still needs to augment my reading glasses, fingertips or hearing aid. Perhaps Google Glass will inspire developers to come up with exciting new AR concepts, although the current outlook of this happening is bleak, from what has been announced. Could it be that AR is suffering from the VR-curse? In 1997, I visited CeBIT, Europe’s largest computer expo in Hanover, Germany with a friend. It was the age of virtual reality, and there we bought the Forte VFX1, a then state-of-the-art virtual reality system. It in no way matched our expectations. In the nineties, virtual reality was hyped through Hollywood movies and popular science. As a result, our expectations grew far beyond what it could really offer us. It promised us virtual worlds that were interesting and fun to move through. In reality it gave us Commodore-64 quality graphics, sloppy positioning, jurassic interaction latencies

VR pioneer Nicole Strenger,  CC BY-SA 3.0

and quite severe headaches. I think that this illustrates the VR-curse. Technology and interaction design have advanced greatly since 1997, but could

want to interact with a mix of virtual and real con-

it be that we are also expecting too much of AR?

tent, as promised during my youth in KIJK magazine1, where an AR gun could shoot the referee in

We want instant virtual content that seamlessly

your televised football match. But none of that has

overlays onto the world around us, whether in the

even appeared on the AR horizon.

car or when watching football. But so far, the virtual ad-signs overlaid onto Eredivisie football games

I may be completely wrong and just impatient.

work only for one fixed camera position, not for

The recent Oculus-Facebook deal may indicate

all other cameras recording the same match. We want maximum content availability with minimal

distraction. But for now, Facebook profiles are the

ticle. It was published in an issue of KIJK in early

pinnacle of detailed content on Google Glass. We

1980’s. Let me know if you find it!

Unfortunately, I cannot find the original ar-

John F. Kennedy

that the VR curse is finally ending. Equipment will

apps that match its specifications; instead, design

become more readily available, which must also

new ideas and ask the geeks to make equipment

be good for AR. But I don’t want to wait anymore.

to match it. To paraphrase from JFK’s inaugural

I want good AR, and I want it now! Fortunately for

speech,

me, I will either be happy when that happens, or proven correct in my pessimism (“I told you so in 2014!”). It’s an easy win-win for the pessimistic

”Ask not what AR can do for you; ask what you can do for AR!”

me. But in fact, I would like AR to succeed. Naturally, this is a utopian view of how technoAnd for that to happen, we need to engage non-

logical ideas may develop, but what the heck...

technicians in developing ideas for augmented re-

From what I’ve heard this is the last edition of

ality. Don’t look at available equipment and make

AR[t] Magazine, and my last chance to rant.

Hello Plant! by Rosen Bogdanov and Peter van der Putten

Nature and technology are often portrayed as en-

But does that mean that studying the bonds be-

emies. But can technology help us get closer to

tween humans and plants is an off limits, esoteric

nature? As a first step towards a ‘sonic garden’,

research topic? We don’t think so. Almost every-

we created a device that augments the experi-

one has houseplants at home. This is so pervasive

ence of interacting with an ordinary houseplant

that one may forget to ask why. We no longer

with audio feedback. It is not meant as some ar-

need houseplants to feed us or keep us dry, but

tistic statement, but its purpose is to run a sci-

keep plants to create a pleasant living environ-

entific experiment to determine whether audio

ment, or even to keep us company. If we ignore

feedback can deepen our bond with plants.

these bonds, it actually becomes quite difficult to explain why houseplants are so common. This

Over the course of evolution we evolved a need

human experience is a real world phenomenon

to bond with plants, as plants give us food, shel-

that can be studied.

ter and other resources important to our survival. This urge to affiliate with other forms of life was

Plants aren’t just basic, simple organisms either.

defined as the Biophilia Hypothesis by E.O. Wil-

One doesn’t need to ascribe humanlike high level

son [1]. This partially explains why many cultures

intellectual capabilities to plants to recognize

ascribed magical and human capabilities to vari-

that these are actually complex, learning, and

ous aspects of nature including the plant world,

networked interactive systems, with sophisti-

and introduced rituals suggesting communication

cated sensing, signaling and action behaviors [3].

with plants. And this was not restricted to our an-

Take for example the way many plants use their

cient forefathers. ‘The Secret Life of Plants’, the

roots to detect, at any point, numerous chemical

1973 best seller, claims for example that plants

and physical parameters [4]. Plants are able to

are high level sentient and intelligent beings and

distinguish between different positive and nega-

describes a range of prior experiments that lead

tive experiences [5], while also registering some

to far reaching claims about plant abilities to per-

sort of memory [6]. Studying these mechanisms,

ceive and understand human behavior [2]. Not

without making wild claims about humanlike in-

surprisingly most of the experiments couldn’t be

telligence, are an established practice in scien-

reproduced and it was generally seen as a prime

tific plant research.

example of pseudoscience [3].

Various artists have created installations cen-

justed to foster interaction and better measure

tered around the concept of plant sensing and

a subject's levels of empathy and affections to-

interaction. An early example is the ‘Interactive

wards plants (see [14] for full details).

Plants Growing’ installation, whereby interaction between people and physical plants impacts the appearance and growth of digital plants in a virtual environment [7]. An installation that takes the human out of the loop is ‘Sensobotanics’ by Thomas Hawranke. In this piece, a plant operates a first person shooter game, emphasizing the difference in terms of the time dimension between both realities [8]. An example that is more of a mix between the artistic and the practical is Botanicus Interacticus by Disney Research, in which plants are used as a control device [9]. A range of academic research projects also augment plants with technology to create interesting hybrids. Professor Stephano Mancuso from the University of Florence is an example of an accepted academic plant scientist making a case for seeing plants as more intelligent creatures, and also collaborates with others to combine plants and technology. For example the EU Pleased project [10] focuses on using plants as a network of sophisticated sensing devices [11, 12]. He also works with Barbara Mazzolai from the Italian Institute of Technology in Genua on the design and prototyping of hybrid, robotic plants

Augmented plant, equipped with electrodes.

called plantoids, for example for soil analysis in future space missions [3, 13].

The sound design is based on the idea that nature soundscapes induce fascination with nature and feelings of relaxation. Carefully filtered noise

Human Plant Bonding Through Sonic Augmentation

that reminds of rainfall drops with randomized intensity and pitch were designed in PureData. Low-frequency modulated envelopes are ramped every 5 seconds to set the rate at which each

So let us now discuss our own project in more de-

droplet is generated. Two different envelopes are

tail. The device used in this study provides sonic

assigned depending on data coming from the sen-

feedback from actual interaction (e.g. touch)

sors – one leads to a less saturated droplet gener-

with a regular houseplant, and the goal of the ex-

ation, while the other leads to a more saturated

periment was to measure whether this strength-

one, e.g. resembling a small waterfall.1

ens the bond between the human and the plant. We took in mind the background research and

pilot tests, whereby tasks were created and ad-

plantinteraction.mp3 as a sample example

Listen to http://www.liacs.nl/~putten/library/

same level of feedback. Plant processes typically change slowly, are hard to measure reliably and there is a lot debate on what response mechanisms truly exist. We rather focused on measuring the human experience or perception, which is a real world phenomenon.

Putting It To The Test Our prototype was tested for three days in the Hortus Botanicus in Leiden (see http://www. botanicus-interacticus.com/),

the

oldest

bo-

tanic garden in the Netherlands, with an overall sample group of 30 subjects and a single plant – Ornithogalum Thyrsoides, or Star of Bethlehem. First the participants filled in a survey to measure an individual's level of connectedness to nature, based on a scale used in psychological research [15]. Then the participants had to carry out a Ornithogalum, Star-of-Bethlehem

number of interaction tasks such as touching the leaves, inspecting the plant carefully, cut a leaf

At the sensor we measure electrical conductiv-

and spray water, to enable the participants to

ity in the plant using a Wheatstone bridge de-

‘bond’ with the plant. For half of the participants

vice and two copper taped electrodes. It detects

the interaction during this phase was augmented

two main events; touching of leaves and spraying

with the soundscape sounds through the device,

water on leaves. When touching a leaf, the less

the other half acted as a control group. They

saturated envelopes mentioned above are acti-

then had to carry out a number of tasks aimed

vated. However, big changes in capacitance can

at measuring whether one could claim there was

increase the amount of droplets: up to 10 drop-

an increased bond, for example setting a price on

let synthesizers each generating droplets every 5

the plant, attaching their name to the plant with

seconds. If the data begins to peak – in this case,

a choice between push pin or blue tag, add soil

if water gets on the electrodes, then all droplets

from which the participants could remove some

are generated with more saturated envelopes, in

cigarette ends, and to push the boundary, read a

order to get a different feedback without losing

poem to for the plant, either silent or aloud. Fi-

the effect of random generation. As opposed to

nally participants themselves would report their

a straight sonification, this randomness creates

level of connectedness to plants, using a visual

a sort of perceptual detachment from the direct

scale.

visual interaction between the human and the plant, without losing the feeling of actual feed-

The results show that people with sound feed-

back from touching and spraying water on plant.

back spent more time on the interaction tasks than the control group, though one can argue

A key principle is that we don’t even assume to

whether that was because of the sound itself or

measure any real or smart plant responses. In-

the actual interaction. The results on the mea-

animate objects could actually have provided the

surement tasks demonstrate an increase in inter-

Test setup at Hortus Botanicus, Leiden

personal and perspective-taking relations in the

of the experiment. However, this is not neces-

individuals that went through sound-augmented

sarily a negative point. In the future, a larger

interaction with the plant, though some of these

design focusing more on tightly connecting the

differences may have been due to random bias

sensory experience with the particular context

between test (those who received audio feed-

of human-plant interaction can give even more

back) and control (no audio feedback), which we

interesting results. One can think of placing the

measured through the survey at the start. In the

experimental design more inside the actual gar-

self report results the test group scored higher

den, instead of on a table in the entrance. More

than the control.

plants means more flexibility in the design of objective measures, and more flexibility in terms of

Towards a Sonic Garden More experiments would be needed to confirm

measuring and sensing interaction. Such design should also allow for more implicit measurements of bonding by tracking interaction, which should

some of these results, but in itself the experi-

lead to more reliable results. Other ideas include

ment serves as an early example of how such

doing controlled experiments around interaction

research can be performed and extended. The

with a fake plant â&#x20AC;&#x201C; i.e. looking for differences in

context itself, as seen from this study, is impor-

researching connectedness between actual life-

tant â&#x20AC;&#x201C; a botanical garden does shape the nature

forms and artificial objects.

References

The more long term idea of a ‘sonic garden’ fits well with an augmented end vision of

1. Wilson, Edward o. (1984). Biophilia, Cam-

extending man by interfacing it with both nature and technology. Marshall McLuhan

bridge: Harvard University Press.

stated: “In the electric age, we wear all mankind as our skin” [16]. What McLuhan

2. Tomkins, P. and Bird, C. (1973). The Secret life of Plants, Harper and Row.

meant with the quote is that by placing physical objects in the realms of our consciousness and our nervous systems, we

3. Pollan, M. (2013). The Intelligent Plant. The

translate everything (artificial or natural)

New Yorker, pp 92-105, December 23, 2013.

to a potential information system. The tools of such augmented physical realities

4. Brenner, E. D., Stahlberg, R., Mancuso, S.,

are, of course, often designed as direct

Viv- anco, J., Baluska, F., and Van, V. E.

extensions to our own sensory inputs/out-

(January 01, 2006). Plant neurobiology: an

puts - sensor interfaces and other electrical

integrated view of plant signaling, Trends in

media (screens, projectors, sound and radio

Plant Science, 11, 8, 413-9.

waves, etc.). In the context of our project and augmented reality his quote is perhaps

5. Goh, C. H., Nam, H. G. and Park, Y. S.

better re-phrased to “...we wear all nature's

(2003). Stress memory in plants: A nega-

skin”, given that we are not just extending

tive regulation of stomatal response and

our own senses, but are interfacing to other

transient induction of rd22 gene to light in

living creatures as well.

abscisic acidentrained Arabidopsis plants, The Plant Journal 36 (2): 240–255.

Peter van der Putten Peter van der Putten is a part time researcher at the Leiden Institute of Advanced Computer Science, Leiden University, the Netherlands, active in the Media Technology program as well as the Data Mining group. His background is in artificial intelligence and he is particularly interested how intelligence and complex systems in general can evolve through learning and interaction, in organisms as well as machines. Peter has a MSc in Cognitive Artificial Intelligence from Utrecht University and a PhD in data mining from Leiden University, and combines academic research with applying these technologies in business.

6. Volkov, A. G., Carrell, H., Baldwin, A. and Markin, V. S. (2009). Electrical memory in Venus flytrap, Bioelectrochemistry 75 (2): 142–147. 7. Sommerer C. and Mignonneau L. (1993). "Interactive Plant Growing," In Visual Proceedings of the Siggraph ’93 Conference, ACM Siggraph, 1993, pp. 164-165. 8. Hawranke, T. (2010). Sensobotanics. In Trogemann, G. (editor), Code und Material

Rosen Bogdanov

Exkursionen ins Undingliche. Springer Verlag,

Rosen is a MSc Media Technology gradu-

pp 88-93.

ate from Leiden University, Netherlands. He is interested in building interactive

9. Disney Research Botanicus Interacticus,

systems that augment biological processes

Retrieved from http://www.botanicus-inter-

in order to investigate the implications

acticus.com. Accessed February 22, 2013.

those can have on humans’ perception of their environment. For this he enjoys be-

10. EU Pleased project website, http://pleasedfp7.eu, Accessed February 22, 2013.

ing on the crossroads of science, art and technology. Recently, Rosen researched whether sound-augmented human-plant

11. Manzella, V., Gaz, C., Vitaletti, A, Masi, E.,

interaction can have an effect on people’s

Santopolo, L, Mancuso, S., Salazar, D and de

relationships to plants. Apart from that,

las Heras. J.J. (2013). Plants as sensing de-

he often engages with different com-

vices: the PLEASED experience. In Proceed-

munities and projects where co-working,

ings of the 11th ACM Conference on Embed-

co-learning and Agile-like processes are

ded Networked Sensor Systems (SenSys '13).

involved.

ACM, New York, NY, USA. 12. Flinley, K. The Internet of Vegetables: How

14. Bogdanov, R. (2013) Can sonic feedback from

Cyborg Plants Can Monitor our World. Wired

human-plant interaction increase connected-

Enterprise. Retrieved from http://www.

ness to plants? MSc Thesis, Media Technology,

wired.com/wiredenterprise/2014/01/inter-

Leiden University.

net-plants/. Accessed February 22, 2013. 15. Nisbet, E., Zelenski, J., & Murphy, S. (2009), 13. The Roots of plant intelligence, TED talk at http://www.ted.com/talks/

The Nature Relatedness Scale, Environment and Behavior, 41, 5, 715-740.

stefano_mancuso_the_roots_of_plant_ intelligence.html. Accessed February 22, 2013.

16. McLuhan, M. (1964) Understanding media: The extensions of man, New York: McGraw-Hill.

Towards hybrid disciplines in a postdigital world Reflection on the future of art, science and tech in arts education by Isjah Koppejan

The Golden Orb Weaver is a tropical spider which weaves its web with a unique material – stronger than steel, more elastic than nylon and a better conductor than copper. Sparked off by a scientific article and her own imagination, artist Jalila Essaïdi discovered new applications for this startling spider silk. Together with the Forensic Genomics Consortium Netherlands she developed a piece of bulletproof skin. “Innovation takes place when you look at something from different angels. I want Bulletproof Skin to show that more is possible than you realise and to provoke discussion of how far we are prepared to go.” She calls herself a hybrid artist: “I’m used to thinking beyond my own field”. [1] Essaidi is founder of BioArt Laboratories where artists can experiment with bio-based materials. This initiative, driven by the dramatically increased progress in biotechnology, aims to make this progress and its implications ‘the collective responsibility of society’ [2]. In the 21st century, a new form of creativity seems to be surfacing. Driven not only by the ongoing developments in material technology, but also by the rise of digital fabrication. In our postdigital world, artists and designers use these advanced means to create artistic expressions never before thought possible. Recently, the Museum of Arts and Design in New York has been examining trends in contemporary digital design and fabrication with the exhibition and accompanying book Out of Hand: Materializing the Postdigital. Although the book and exhibition have a highly sculptural approach it shows that digital technologies enable art and design to infiltrate the boundaries of different disciplines and is changing its relationship to materiality and craftsmanship. This takes place at the start of the exhibition demonstrated by a self-portrait by the artist Richard Dupont: a nude male figure with melty, deformed contours, a sculpture that looks more like a refection in a distorting mirror. Dupont has had his entire body scanned and ran the data through digital modelling programs. It’s made with computer assistance at almost every stage, from the design right through to the fabrication process, like digital milling and rapid prototyping. Another more radical and disruptive example is the work Ob-

MyceliumChair, Studio Eric Klarenbeek

ject Breast Cancer, which has inspired a research project at Weill Cornell Medical College. This work uses 3D software to transform MRIs of tumours into small sculptures and pieces of jewellery, which hint at the profound and often unnerving new ways that 3D technologies can explore the body. For a long time, art and design work aided by computers was usually shown ‘on screen’ (computer, projections, etc). Nowadays, the rise of digital

Richard Dupont, Untitled (#5), 2008, pigmented cast polyurethane. Courtesy Richard Dupont and Artists Rights Society (ARS), New York.

fabrication has created the possibility to materi-

alise the digital into physical, tangible forms. The influences of these developments on Augmented Reality are stronger than we might think. For example, scientists have recently been working on mediated touch, where you can touch things you cannot grab with an interactive glove (see, e.g., [3,4]. The non-invasiveness of these new developments is interesting for different applications. In the European MeSch project [5] for example, The Allard Pierson Museum has tested the use of AR in its recent exhibition Eternal Egypt Experience. The introduction and enhancement of the materiality of the experience of museum objects is a challenging one. How can you experience the physicality of an artefact which is behind glass and can usually not be touched? The MeSCH project approach is grounded on principles of co-design: the participation of designers, developers, scientists and stake-holders into the process of creation and evaluation as equal partners, and on a Do-ItYourself philosophy of making and experimenting.

Critical making The term postdigital in artistic discourse is often described as the period after the beginning of the age of digital design. [6] For me, the term postdigital goes a step further and introduces a way of thinking that deals with the consequences of the digital and the internet age, being more concerned with being human than being digital. It’s a sign of our changing relationships with digital technologies and enforces us to look past the optimist or even utopian views of the digital en-

hancements. What’s the impact of Facebook, or

to the postdigital. Digital fabrication tools or new

even of the future use of 3D bio printing to cre-

materials are not just the new paintbrush, the

ate new organs, or the influence of nanotech? The

new musical instrument or the new way of mak-

postdigital generation of artists and designers are

ing prototypes. This is why Essaidi has the urge

investigating, making use of and criticising the

to call herself a hybrid artist. She is redefining

implications of these ‘enhancements’.

the notions of creative production and creative processes, which becomes more like a blend of

Frustrated by internet restrictions, IDPW1 de-

different disciplines. Due to the nature of digital

signer/programmer Shunya Hagiwara made the

fabrication, the use of open source technology

Whatever Button. The Whatever Button is an ex-

and the culture of sharing, the artistic process

tended free online feature for the Google Chrome

is opening up. From ideation, concepting, mak-

web browser, which enables you to click all the

ing and distribution to a process of co-creation,

Facebook Like buttons on a page, from top to bot-

coproduction, crowdsourcing and collaboration.

tom. With one click you can automatically click

This changing practice requires new qualities in

every “Like” button that appears on the screen,

an artist, as the next example demonstrates.

regardless of the content: the good, the bad and the ugly. This work can in part be seen as a criticism of online conditions. While following hacking etiquette and defying social and technological conventions, the work is imbued with a richness that transcends any criticism. In 2012 The Whatever Button designed was awarded the ‘New Face Award’ at the Japan Media Arts Festival. Shunya Hagiwara: “I know we don’t really need this, but we depend so much on the architecture or structure arbitrarily set up by companies like Facebook. I wanted to do something witty to make fun of it. My goal is to make people know about it, rather than actually use it. If people know there is someone out there who is thinking enough to make a button like this, I think they won’t be tired liking things.” The rise of these ‘critical makers’ have a practice that requires a split identity; one foot planted in the craftwork of design and the other foot planted in the reflexive work of critique. It

Vivian Hartung, A new laced wheel

requires the understanding of underlying systems,

Fashion designer Iris van Herpen is one of the

technologies, methods and concepts.

early pioneers in 3D prototyping in fashion. She is well known for her 3D printed shoes for United

Co-creation melts crafts and technology into one

Nude and for a 3D printed dress that TIME Magazine names one of the 50 Best Inventions of the year 2011. Van Herpen collaborates with artist from different disciplines in almost all of her

‘Out of hand’ approaches the postdigital more or

projects. In an interview with SecondSight, Iris

less as an instrumental view, as a new way of mak-

van Herpen states: “Before I started this project

ing, which creates a new kind of aesthetic. But I

[Bored Dress] I didn’t like computers and tech-

would like to emphasize a more holistic approach

nology. Now I see the possibilities for the future

Bulletproof skin being pierced, '2.6g 329m/s' by Jalila Essaïdi.  JalilaEssaidi.com

are enormous. So much is going on right now,

also driven by the rising notion in recent studies

also in the production process. High-end fashion

that fine arts graduates contribute to innovation

doesn’t resolve around the big Parisian houses

throughout their working lives. The study “The art

anymore. It’s about new ways of working, coop-

of innovation” [8] shows that these artists have

erating, helping and strengthening each other”.

attitudes and skills that are conducive to innova-

These new processes and methodologies require

tion, that they work in a way that is organised

different attitudes and skills from the artists and

around portfolio and project work and continually

designers. As Iris van Herpen puts it: “I am not

have ‘crossover’ and cross-fertilisation of people

technical myself and the project involves quite a

and ideas across the arts, and within the arts and

lot of software aptitude which really requires the

outside of the arts. Artists and designers have a

aid of a specialist. That took some getting used

flexible way of thinking and acting, try to keep it

to: you speak different languages and you have to

simple, follow their passion, do more with less and

develop a common one”.[7]

find possibilities in difficult circumstances. These qualities are valued in innovative areas. Think for

Innovation by art

instance of the development of the 3D printed chair of mycelium by Studio Eric Klarenbeek. In one of our conversations, Eric Klarenbeek declared

For companies, science, artists and designers alike,

passionately: “This material mycelium, the thread-

new materials and technologies offer possibilities

like vegetative part of fungus, is amazing, yet we

for fascinating innovations. Recently, artist and

know so little about it! Even the scientists are just

designers have been playing a more significant

touching the surface.” In cooperation with the

role in research and development projects. This is

University of Wageningen and CNC Exotic Mush-

rooms they developed a concept chair that is made

fashion and other aspects of everyday life and

by mixing water, powdered straw and mycelium.

finding ways to navigate through them and imple-

That mixture is then coated with a thin layer of

ment these findings. This is not an easy task. In

bioplastic, and as the mixture dries and the myce-

this period of economic decline, the arts, higher

lium grows, it creates what’s described as a solid

education and research institutions’ legitimacy is

but light material. “This material has such amazing

constantly under fire and these institutions are

qualities. The chair is merely a metaphor for what

facing serious budget cuts. In the light of two

can be achieved with new materials and produc-

examples (the ArTechLab and the Open design

tion methods. It could be a table, a whole interior,

minor), I would like to show that arts academies

or even a house”, according to Klarenbeek.

realise that this interdisciplinary open practice is redefining notions of creative processes and pro-

This example of a new type of collaboration be-

duction and that new approaches are needed.

tween arts and science demonstrates the added value to combine the tacit with the explicit di-

Examples from a recent exhibition full of experi-

mension of knowledge. Artists and designers are

ment and exploration by the student-researchers

collaborating in new ways that drive innovation

of the ArTechLab at AKI ArtEZ include the tweak-

and are earning their place in the innovation pro-

ing and hacking 3D printers, photo cameras, 3D

cess. One of the projects I’ve worked on, called

scanners or opening up possibilities for new fabrics

CRISP Smart Textile Services, deals with the de-

to create a woven carbon bicycle wheel or to re-

velopment of applications for new ‘smart’ tex-

place plaster. In 2011, ArtEZ started the ArTechLab

tiles. ‘Smart’ stands for textiles that can conduct

as vrije minor (elective) for fine arts and design

heat, lights or currents and as a result become

students, as an innovative interdisciplinary spe-

an interactive product. In this project, an inspi-

cialisation for researching new materials and tech-

rational testbed of workshops is created where

nologies. This research is becoming much more of

designers and engineers come together to ex-

a participatory process, as the technologies are

periment and play with the materials and look

becoming more complex and co-operation with

beyond traditional approaches [9]. The increase

specialists from the field are needed and provided

of the involvement of artists and designers in

during the process. A team of advisors on which

research and development projects, indicates

students can rely on for their professional network

that a new field is emerging where design, art

and technological, engineering or artistic advice,

and science will persistently make use ofeach

coach the student researchers. Another specializa-

other’s interfaces. This new area requires artists

tion at another arts academy, the WdKA in Rotter-

and designers to rethink their role and the value

dam, under the name of Open Design Minor deals

of their creativity and imagination. This role re-

with similar implications but specifically focuses on

sembles that of a composer taking the creative

the use of open source technologies and methods.

lead in the project while speaking different lan-

It deals with questions like: How can you express

guages and conducting everyone’s imagination.

authorship, your identity and position as a designer in a participatory and open process? What are the aesthetics and poetics of Open Design?

New approaches in art education

I emphasize these two examples, as their approach

These new movements find their way into arts

cal and social developments as a new art discipline

and design academies. Art Academies are taking

or separate field. They try to make use of this in-

a closer look at the global impact that new tech-

terdisciplinary trend that renders the convergence

nologies have had on art, science, architecture,

of disciplines from within and outside the arts and

is similar; they do not approach these technologi-

design world. Rather, these specialisations stress the need of a new way of working and therefore emphasize skills like hands-on artistic research,

Notes 1. IDPW is a self-proclaimed ‘100-year-old

collaboration, co-creation, craftsmanship and

secret internet society’, which organizes

critical reflection. Students experience these ap-

events that combine internet with real

proaches as refreshing and different from the rest

places to do something new and experimen-

of the curriculum, which poses new challenges.

tal. See www.idpw.org

How do you connect these new approaches with the current curriculum? What kind of expertise do you ask of teachers? How do you support students in their search for their own authenticity while working with others? How do you facilitate the

References 1. Chris Gruijters, Chris, and Koert van

use of future technologies? How can you create a

Mensvoort. Crossover works #2. Amsterdam:

support structure for students’ cooperation with

Federation of Dutch Creative Industries,

companies and scientists? The road to the answers

2014. Print.

of these questions will help us to figure out how to

2. http://bioartlab.com

strengthen the combination of art, technology and

3. http://hmi.ewi.utwente.nl/MeTA/

research in future art education practice.

4. Huisman, Gijs, and Aduén Darriba Frederiks. “Towards tactile expressions of emo-

ArTechLab, cross discipline research lab, based at the ArtEZ Enschede

tion through mediated touch.” CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 2013. 5. http://mesch-project.eu 6. Alexenberg, M. (2011). The Future of Art in a Postdigital Age: From Hellenistic to Hebraic Consciousness. Bristol and Chicago: Intellect Books/University of Chicago Press 7. Wiegman, Andrea. “Bored dress.” SecondSight 1 July 2011: 110-111. Print. 8. Oakley, Kate, Brooke Sperry, Andy C. Pratt, and Hasan Bakhshi. The art of innovation how fine arts graduates contribute to innovation. London: NESTA, 2008. Print. 9. Koppejan, Isjah. “Making opportunities tangible.” CRISP magazine 1 Apr. 2013: 20-23. Print.

ArTechLab

Isjah Koppejan

ArTechLab is a cross discipline research lab

Isjah Koppejan is the programme man-

based at the Artez Enschede. The lab initi-

ager and lecturer for the Minor course

ates cross discipline projects, in collabora-

ArTechLab at the AKI ArtEZ Academy

tion with other institutes, universities and

for Art and Design in Enschede. She is

companies. As we have a broad team of

responsible for student supervision and

researchers, we’re able to make proposals

for developing the ArTechLab from a

and do research on complex artistic, social

minor course to a master’s programme.

and design issues. Every year we start with a new team of student researchers. They

Before joining the ArTechLab, Isjah

have a variety of backgrounds. Most of

worked in research and innovation

them are studying at different departments

at the Waag Society and at the HKU

of ArtEz, or at other institutions as Saxion

University of The Arts Utrecht. Isjah

or University of Twente. Our student re-

initiated and realised many innovative

searchers are creative thinkers, are pas-

concepts and projects at the inter-

sionate about what they do and have fresh,

section of culture, ICT, science and

innovative ideas. The researchers present

economics. She specialises in translat-

their research ideas and results at www.

ing social developments and trends into

artez.nl/artechlab. Our team of advisors

strategies, products, services and useful

coaches the researchers. This team con-

ideas.

sists of Eric Klarenbeek, Filip Jonker, Paul Jansen Klomp, Isjah Koppejan, David Menting, Mark Bakema en Thomas Choduba.  www.artez.nl/artechlab 55

Site Venice Site Biennale: The Manifest.AR Augmented Reality Intervention into the 2011 Venice Biennial by Tamiko Thiel

Introduction In 2011, using geolocative augmented reality (AR), the author was the primary organizer of the Manifest.AR cyberartist group intervention into the Venice Art Biennale, together with fellow artists Sander Veenhof and Mark Skwarek (Manifest.AR Venice Biennale Manifesto 2013). Using GPS coordinates we placed virtual artworks – visible in smartphone displays as overlays on the live camera view of the surroundings – inside the curatorially closed spaces of the Biennale. Unlike physical interventions, the artworks cannot be removed or blocked by authorities. The artworks exploit the site as their canvas while simultaneously questioning the value of location, and the power of the curator as gatekeeper, to canonize works of art. The Venice Biennale, founded in 1895, is the world’s oldest art biennial and the city’s main claim to relevance as a contemporary art destination. In the intervention we wished to question the biennial system, and the art world’s use of that system to define artistic value, but also address the site as artists: the reality of Venice’s contemporary concerns and of life in the city today.

Challenging and Exploiting the Primacy of Site

Artistically, our works often stand in dialogue

Manifest.AR [1] originally formed around an AR

presence of our artworks at the site increasing

intervention into the United States’ most iconic

the potency of their argument.2 In a time when

contemporary art space: the Museum of Modern

many question the relevance of galleries, muse-

Art in New York. In 2010 Sander Veenhof and Mark

ums and biennials – the gated communities of the

Skwarek realized that the institutional walls of

art world – we bring a new form of dialogue into

the white cube were no longer solid, and orga-

their institutions. [5]

with the “official” artworks at a venue, and with the curator’s theme and concept – with the visual

nized a guerilla exhibit of augmented reality artworks inside the walls of MoMA.1 Since time immemorial location has been used to consecrate objects and people. In the art world too, access to a location – a gallery, a

Manifest.AR Venice Biennale Intervention: Themes and Concerns

museum or other curatorially closed space – is tightly controlled to confer value and thus, via

At the 2011 Venice Biennale we wished to reflect

this exclusivity, to canonize the works shown

not on Venice’s past glory, but on its current con-

there as “high art.” What does it mean however

dition: not only wrestling with climate change

to control physical space when in geolocated

and overrun by tourists, but also fighting for rel-

virtual space anyone can place whatever they

evance in the art world. The national pavilions

want? [2]

that dominate the Venice Biennale reflect its origins at the end of the 19th century and the rise

Technically, it is a trivial difference in GPS coor-

of the nation-state with a presumed monolithic

dinates that moves a virtual object from a pub-

ethnic or cultural identity. They stand now in di-

lic space such as Central Park to the curatorially

rect contrast to the globalized, itinerant world of

closed space inside the sacred walls of MoMA.

contemporary artists and their multiple systems

The epiphany of AR however is that although the

of cultural reference. [6]

artworks are virtual, their presence at the site is

real, “actually existing as a thing or occurring in

Curator Bice Curiger’s opening statement ques-

fact; not imagined or supposed” [3], reproducible

tioned this structure as well: “By adopting the

by anyone who views the artwork at that loca-

title ILLUMInations the 54th International Art Ex-

tion. In this “consensual hallucination,” that was

hibition of the Venice Biennale also aspires liter-

the dream of the early cyberpunk authors and

ally to shed light on the institution itself, drawing

virtual reality evangelists [4], augmented reality

attention to dormant and unrecognized opportu-

redefines the barriers between “the real” and

nities, as well as to conventions that need to be

“the virtual.”

challenged... Far removed from culturally conservative constructs of ‘nation,’ art offers the poten

The artworks engage viewers with the site physi-

tial to explore new forms of ‘community’ and ne-

cally as well. Like bird watchers with binoculars,

gotiate differences and affinities that might serve

AR viewers scan their surroundings with their

as models for the future.” [7] Curiger also posed

smartphones, dodging real world obstacles in

five questions on identity to each of the artists

search of the artwork, situating themselves and

officially included in the Biennale: “Where do you

the act of viewing in their physical experience

feel at home? Does the future speak English or

of that site.

another language? Is the artistic community a na-

tion? How many nations do you feel inside your-

54th Biennial of Venice could not justify its reputa-

self? If art was a nation what would be written in

tion without an uninvited Manifest.AR Augmented

its constitution?”

Reality intervention. In order to “challenge the

conventions through which contemporary art is As an international artist collective that co-

viewed” we have constructed virtual AR pavilions

alesced around challenging conventions of in-

directly amongst the 30-odd buildings of the lucky

clusion and participation, we saw this as a per-

few within the Giardini. In accordance with the

sonal invitation to participate. Sander hijacked

“ILLUMInations” theme and Bice Curiger’s 5 ques-

Curiger’s curatorial statement and the Venice

tions our uninvited participation will not be bound

Biennale website to create our Venice Manifesto,

by nation-state borders, by physical boundaries

in which we proclaimed:

or by conventional art world structures. The AR pavilions at the 54th Biennial reflect on a rapidly

As “one of the world’s most important forums for

expanding and developing new realm of Augment-

the dissemination and ‘illumination’ about the

ed Reality Art that radically crosses dimensional,

current developments in international art” the

physical and hierarchical boundaries. [8]

Figure 1. Manifest.AR Venice Biennial Intervention website.

Questions about control of space are not confined

controlled curatorial space of the Venice Giardini,

to art venues; “public” art is always dependent

but also in the public space of Piazza San Mar-

on permissions from authorities, and many a

co, which has itself seen censorship of officially

“public” space is actually closely controlled. We

planned artworks. [9]

therefore placed our artworks not only in the

Manifest.AR Artworks in the Venice Biennale Intervention Tamiko Thiel’s Shades of Absence is a series of three “virtual pavilions” in the Giardini, in Piazza San Marco and inside the German National Pavilion. Anonymized golden silhouettes of artists whose works have been censored are enclosed by terms of censorship. In reply to Bice Curiger’s questions: “Is the artistic community a nation? If art was a nation what would be written in its constitution?” they posit a transnational community of censored artists. Touching the artworks in the

Figure 2. Shades of Absence: Public Voids, Tamiko Thiel, 2011. Augmented Reality, Piazza San Marco, Venice. A memorial for artists whose works in public spaces have been censored.

display of a smartphone calls up a website with cases of censorship. [10] Sander Veenhof’s Battling Pavilions directly addresses the role of the curator, the exclusivity of the Giardini and the limited number of national pavilions allowed within its Sacred Grove. Users outside the Giardini can subvert Curiger’s authority and create new virtual pavilions for nations of their choice inside the Giardini. Users inside the Giardini, in contrast, can help Curiger defend the Giardini against intruding pavilions by deleting them. In a classic twist, Sander’s interven-

Figure 3. Battling Pavilions, Sander Veenhof, 2011. Augmented

tion also became an official part of the Biennale:

Reality Game. Scoreboard on dropstuff.nl screen during the

dropstuff.nl invited him to show his Battling Pa-

Venice Biennale, displaying scoreboard of unauthorized virtual

vilions on their large screens in three locations

pavilions in the Giardini.

around Venice. [11] Mark Skwarek’s Island of Hope addresses the perpetual threat of Venice sinking into the lagoon. Skwarek posits new forces of continental uplift erupting as fully formed baroque gardens into the Giardini and in Piazza San Marco. The islands are full of objects of hope, and tweets with the hash tag #hope, in order to bring hope back to Venice. [12] John Craig Freeman’s Water wARs: Squatters Pavilion is a virtual squatter’s camp for refugees

Figure 4. The Island of Hope, Mark Skwarek, 2011.

of water wars, one inside the protecting walls of

Augmented Reality. Seen in the Venice Giardini.

the Giardini, and another “public” camp in Piazza San Marco. In Venice, a city founded by refugees

the rest of the world, as worldwide ecological disasters drive people in desperation to violate the boundaries of the nation-states in pursuit of sheer survival. [13] In John Cleater’s Sky Pavilions ships from outer space take over Venice: The mothership hovers over Piazza San Marco emitting a mixture of nonsense and guidance to confuse and help tourists, natives, and art seekers. In the Giardini alien “Floaties” lie in wait, begging to be touched, and when activated Figure 5. Water wARs, Giardini, John Craig Freeman,

by obliging visitors spin upwards, carrying secret

2011. Augmented Reality. Pavilion for undocumented

messages to the mother ship. Sky Pavilions goes

artists/squatters and water war refugees in front of

beyond the concept of the nation-state, beyond

the Giardini Central Pavilion.

the concerns of mere earthbound humanoids and reminds us that the last word in the control of space may not be ours to decide. [14] Lily and Honglei’s The Crystal Coffin: Virtual China Pavilion is inspired by China’s petrified symbol of eternal Party rule, Mao Zedong’s crystal coffin. In the Giardini it questions the traditional hierarchy of privilege among national pavilions in the Biennale and thematizes the rise of China as an important center of contemporary art. Another pavilion in Piazza San Marco dominates the heart of Venice, whose native son Marco Polo “discovered” China for the West, with this symbol of Chinese Party power. [15] Will Pappenheimer/Virta-Flaneurazine’s Colony Illuminati appropriated both the Biennale title “ILLUMInations” and the actual visual imagery of many artworks in the Biennale. A secret colony

Figure 6. Sky Pavilions, John Cleater, 2011. Augmented Reality and audio. Alien Mothership Sky Pavilion floats over Piazza San Marco.

of virtual bufo toads draws sustenance from high art; as a form of camouflage their skins appropriate imagery from artworks around them in the Giardini and spread out into the city, seeking the outlying venues of the Venice Biennale. When touched on the smartphone screen, the toads release psychotropic drugs that trigger halluci-

and threatened by constant flooding, Water wARs

nations in the viewer: a swirl of Internet infor-

calls attention to the escalating global struggle

mation on the Biennale and waves of Tintoret-

for this basic human need. It questions the ability

toesque ecstasy that Curiger proclaimed to be

of sovereign nations to isolate themselves from

the true essence of ILLUMInations. [16]

Naoko Tosa’s Historia addresses Curiger’s question “Does the future speak English or another language?” and her view that “art offers the potential to explore new forms of ‘community’ and negotiate differences and affinities that might serve as models for the future.” Historia appropriates iconic images from all nations and world cultures, modern and ancient, and allows visitors to arrange them in sequences, assigning them new meanings. It thus playfully examines the process by which artists appropriate and redefine existing cultural symbols to create their

Figure 7. The Crystal Coffin, Piazza San Marco, Lily & Honglei,

own individual languages. [17]

2011. Augmented Reality. Artwork inspired by the crystal coffin in the Mausoleum of Mao Zedong in Tienanmen Square, seen here in Piazza San Marco.

To end with a quotation from Bice Curiger’s curatorial text for the Venice Biennale: “ILLUMInations presents contemporary art characterized by gestures that explore notions of the collective, yet also speak of fragmentary identity, of temporary alliances, and objects inscribed with transience. If the communicative aspect is crucial to the ideas underlying ILLUMInations, it is demonstrated in art that often declares and seeks closeness to the vibrancy of life. This is more important now than ever before, in

Figure 8. Colony Illuminati, Will Pappenheimer/Virta‐Flaneurazine, 2011. Augmented Reality. Colony group on Giardini main concourse.

an age when our sense of reality is profoundly challenged by virtual and simulated worlds. This Biennale is also about believing in art and its potential.” [7] I could not agree more. Perhaps in ways that Bice Curiger did not anticipate. At the latest since the Salon des Refusés in 1863, questions about the validity of the art canon and the institutions that define this canon have been an important part of the evolution of modern art. Augmented reality interventions are the continua-

Figure 9. Historia, Naoko Tosa, 2011. Augmented Reality. Users compose messages by appropriating historic

tion of this modernist dialogue with 21st century

icons floating in the space and assigning a new meaning

means.

to their message. Seen in front of the Giardini Central Pavilion. 61

Coda: The Future of AR Interventions But how will the law react to increasing trans-

Can institutions use these existing laws to assert

gressions in virtual space? By 2013 technologies

“virtual air rights” to “their” GPS coordinates,

such as Google’s Street View and Glass were pro-

thus blocking AR interventions? Intellectual prop-

voking wide public discussion of the confluence of

erty lawyer Brian Wassom thinks not: “Property

locative, mobile, recording and display technolo-

law is about the right to exclude others from

gies, and what negative effects could come of the

physical space. But an infinite number of people

blurring of boundaries between real and virtual

can each create their own AR layer superimposing

space. Most public unease comes however not

digital data over the same physical space with-

from AR display technology, but from recording

out impeding anyone else’s ability to do so, and

(“surveillance”) technology. As Yolande Kolstee

without invading the rights of the real property

points out, the real debate here is not techno-

owner.” [19]

logical but social, and can probably be negotiated using existing legislation [18].

Notes 1. “We AR in MoMA” [20] was part of the Conflux Festival of Psychogeography [21]. Cyberpunk author Bruce Sterling blogged the intervention on WIRED [22], MoMA tweeted a somewhat nonplussed acknowledgment [23], and in a New York Times interview MoMA’s director of digital media welcomed our engagement [5]. 2. The author’s contribution to “We AR in MoMA” was a matrix of screaming faces titled “ARt Critic Face Matrix,” a self-referential artwork that critiqued its own validity as an artwork, reflecting on the role of MoMA NY to define what did or did not constituted art. [24] 3. Although Curiger refers frequently to the “five questions,” they are not to be found on the official Venice Biennale website. See however Flash Art [25]

References 1. Manifest.AR AR Art Manifesto (2011).

8. Manifest.AR Venice Biennale Manifesto

Manifest.AR artist group official website.

(2011) Venice Biennial 2011 AR Intervention

http://www.manifestar.info, accessed 12

by Cyberartist Group Manifest.AR,

March 2013.

Reflection on the official curatorial context. http://www.manifestar.info/venicebien-

2. Aceti, L. (2008) The Virtual Places We own:

nial2011/

When Communities and Artists occupy

To view the actual artworks in Venice go to

Your Place without Your Consent. Internet

the launch page – only accessible on mobile

Research 9.0: Rethinking Community,

devices: http://manifestar.info/vb11/

Rethinking Place: 15–18.

(both accessed 14 March 2013).

3. Oxford English Dictionary. Definition of

9. Magill, R. J. Jr. (2007, 16 April) For Gregor

“real.” http://oxforddictionaries.com/us/

Schneider’s cube, a long pilgrimage.

definition/american_english/real, accessed

New York Times.

12 March 2013.

http://www.nytimes.com/2007/04/16/ arts/16iht-cube.1.5303319.html,

4. Gibson, W. (1984) Neuromancer.

accessed 14 March 2013.

Ace Books, New York. 10. Thiel, T. (2011, May) Shades of Absence. Mani5. Fidel, A. (2010) Art Gets Un- masked in

fest.AR Venice Biennale 2011 Intervention.

the Palm of Your Hand. New York Times

http://manifestarblog.wordpress.com/thiel_

http://www.nytimes.com/2010/12/02/

venice-2011/, accessed 12 March 2013.

arts/02iht-rartsmart.html, accessed 30 April 2012.

11. Veenhof, S. (2011) Battling Pavilions. Manifest. AR Venice Biennale 2011 Intervention.

6. Madra, Y. (2006) From Imperialism to Trans-

http://manifestarblog.wordpress.com/

national Capitalism: The Venice Biennial

sander-veenhof-venice-biennial-2011/,

as a ‘Transitional Conjuncture.’ Rethinking

see also:

Marxism 18(4).

http://www.sndrv.nl/battle/, accessed 14

http://www.academia.edu/2072820/

March 2013.

From_Imperialism_to_Transnational_Capitalism_The_Venice_Biennial_as_a_Transitional_ Conjuncture_, accessed 14 March 2013.

12. Skwarek, M. (2011) Parade to Hope. Manifest. AR Venice Biennale 2011 Intervention. http://manifestarblog.wordpress.com/

7. Curiger, B. (2011) Introduction by Bice Curiger. ILLUMInazioni – ILLUMInations.

skwarek-venice-2011/, accessed 12 March 2013.

Venice Biennale website. http://www.labiennale.org/en/art/

13. Freeman, J.C. (2011) Water wARs: Squatters

archive/54th-exhibition/curiger/,

Pavilion. Manifest.AR Venice Biennale 2011

accessed 12 March 2013

Intervention website. http://manifestarblog. wordpress.com/freeman-venice-2011/, accessed 14 March 2013

14. Cleater, J. (2011) Sky Pavilions. Manifest.AR

21. Conflux Festival (2010) We AR in MoMA.

Venice Biennale 2011 Intervention website.

https://web.archive.org/

http://manifestarblog.wordpress.com/

web/20100928175908/http://www.

cleater-venice-2011/, accessed 14 March

confluxfestival.org/projects/conflux-

2013.

festival-2010/we-ar-in-moma/, accessed 12 May 2014.

15. Lily & Honglei (2011) The Crystal Coffin: Virtual China Pavilion. Manifest.AR Venice

22. Sterling, B. (2010, 6 october) Augmented

Biennale 2011 Intervention website.

Real- ity: AR uninvited at MoMA NYC.

http://manifestarblog.wordpress.com/lily-

Beyond the Beyond. WIRED.

honglei-venice-2011/, accessed 14 March

http://www.wired.com/beyond_the_

2013.

beyond/2010/10/augmented-reality-aruninvited-at-moma-nyc,

16. Pappenheimer W., Virta-Flaneurazine (2011)

accessed 30 April 2012.

Colony Illuminati. Manifest.AR Venice Biennale 2011 Intervention. http://manifestarblog.wordpress.com/

23. MoMa (Museum of Modern Art) NY Twitter site (2010, 8 october). Nice, looks like we’re

pappenheimer-venice-2011/, accessed 12

havin an “uninvited” AR exhibition tomor-

March 2013.

row! Part of @confluxfestival. http://twitter.com/MuseumModernArt/

17. Tosa, N. (2011) Historia. Manifest.AR Venice Biennale 2011 Intervention.

statuses/26786135774, accessed 12 March 2013.

http://manifestarblog.wordpress.com/naokotosa-venice-biennial-2011/, accessed 30 April 2012.

24. Thiel, T. (2010) We AR in MoMA exhibit. Tamiko Thiel website. http://www.tamikothiel.com/We-AR-in-

18. Kolstee, Y. (2013, May). Who owns the space

MoMA/, accessed 10 May 2014.

2, AR[t], issue 3, 40-43. 25. Flash Art (2011) Bice Curiger speaks about 19. Wassom B (2014) Augmented Reality as

the Venice Biennale.

Free Speech. Augmented Legality blog.

http://www.flashartonline.

http://www.wassom.com/from-the-archives-

com/interno.php?pagina=news_

augmented-reality-as-free-speech.html,

det&id=953&det=ok&title=Bice-Curiger-

accessed 2 April 2014.

speaks-about-the-Venice-Biennale, accessed 12 March 2013.

20. Veenhof, S. (2010, 9 october) DIY day MoMA oct 9th 2010 AUGMENTED REALITY art invasion! Sander Veenhof website. http://www.sndrv.nl/moma/, accessed 12 March 2013.

Tamiko Thiel Tamiko Thiel is an internationally known

IBM Innovation Award for Art and Technol-

visual artist exploring the interplay of place,

ogy, FACT Liverpool and Zero1 Biennial. She

space, the body and cultural memory. She is a

is also augmented reality artistic advisor for

founding member of Manifest.AR, participat-

the Caribbean Cultural Center and African

ing in 2010 in the pathbreaking augmented

Diaspora Institute’s augmented reality project

reality intervention at MoMA NY, and being

'Mi Querido Barrio' in Spanish Harlem, NY, for

the main curator and organizer of their 2011

which she helped bring in a Rockefeller Foun-

AR intervention at the Venice Biennale.

dation Cultural Innovation Award.

Her works are featured in the reference

Her guest professorships include Carnegie-

books Digital Art (Whitney curator Christiane

Mellon University, UC/San Diego, Bauhaus-

Paul - Thames and Hudson World of Art), The

University Weimar, the Berlin University of

World of Digital Art (DAM director Wolf Lieser)

the Arts and in 2014 at Nanyang Technological

and “Not Here Not There” AR special issue of

University School of Art, Design and Media

Leonardo Electronic Almanac.

(ADM), Singapore.

Her grants and fellowships include the Mac-

Tamiko Thiel

Dowell Colony, WIRED Magazine, Japan Foundation, MIT, Berlin Capital City Cultural Fund

 tamiko@alum.mit.edu

(Hauptstadtkulturfonds), Goethe-Institut,

 www.tamikothiel.com

Hitting imaginary walls, pulling virtual strings What augmented reality can learn from urban dance

by Hanna Schraffenberger A few weeks ago my colleagues convinced me

your arms should not hit the walls.” To be hon-

to join their weekly Hip Hop fitness exercise at

est, this tip didn’t help me at first. Rather,

the University Sports center. Moving my limbs

I was distracted – those invisible walls remind-

in the rhythm of well-known radio hits turned

ed me of my research into augmented reality

out to be more difficult than I had anticipated.

(AR) and the presence of virtual objects in real

After all, I had been running to similar music on

space. These walls we had to avoid were solely

a regular basis [1]! A particularly difficult move

a product of our imagination. Nonetheless, our

required us to turn 360 degrees while at the

movements acknowledged their presence. The

same time imitating a windmill with our arms.

walls were, in a most basic and fundamental

In order to help us get the movement right, our

way, becoming part of and augmenting our sur-

instructor gave us a simple but effective hint:

roundings... Could we call this a form of imagi-

“Imagine two walls, one in front of you and one

nation-based AR? Could it be that dance and AR

behind you. You can only move between them,

had more in common than I thought? 67

Only minutes later this suspicion got confirmed.

same time mesmerized the audience with move-

By now, our hands were connected to our feet

ments that made us doubt whether his hands

with imaginary strings. In order to move our

were constrained by the same kind of bones we had. Among the vid-

“I wanted to know how much illusion-based dance styles and Augmented Reality had in common and I definitely had to master some of those movements myself.”

eos that were shown, one dancer had left a lasting impression: Albert Hwang, a master in making three dimensional boxes appear in real space – solely by running his hands through thin air. A quick look at his YouTube channel [3] decided the matter; I had to find out how dancers created the illusion that imaginary objects existed in space, I wanted to know how much illusion-based

feet, we had to pull the strings. To my surprise,

dance styles and augmented reality had in com-

when our teacher illustrated the movement, it

mon and I definitely had to master some of those

appeared as if those strings did, indeed exist.

movements myself.

Although I knew that they were merely imaginary, and even though I could not see the strings, some part of me was fooled into believing that

Dance AR?

they were actually there. Given the teacher’s movement, her hands and feet simply had to be

Compared to learning the basics of liquid danc-

connected by a thin, invisible rope! There was

ing, my theoretical considerations were rather

no digital technology required, I was not wear-

simple. AR and illusion-based dance styles have

ing a headset, nor was I staring at a screen: a

one central aspect in common: both create the

relatively simple movement was sufficient in

impression that virtual objects actually exist in

order to convey the presence of virtual objects

our real, physical environment. If we understand

(or, to be precise, virtual strings) in real space.

augmented reality as a concept of combining and

It might not have looked like it, but watching

relating the virtual and the real [4] rather than

these invisible ropes certainly felt a lot like AR!

a collection of technologies, it is not far fetched to think of these dance-illusions as a time and

Over the next days, aching muscles reminded

movement based form of augmented reality.

me to investigate this phenomenon further.

What is more, the traditional, technology-fo-

Luckily, I already knew where to start. In 2013,

cused field of AR can learn quite a few things

I attended a presentation about illusion-based

from urban dance!

dance by Diego Maranan at the Creativity and Cognition conference in Sydney [2]. During his

So how does urban dance approach the virtual

talk, Maranan not only illustrated technologi-

and how do their methods inform the general

cal metaphors used in the urban dance styles

field of AR?

‘liquid’, ‘digitz’ and ‘finger tutting’, but at the

No technology required!

these ‘dance-objects’ could not differ more from real objects. First of all, dance-objects do

First of all, dance teaches us that there are

not adhere to our physical laws; they commonly

alternative means to display virtual objects in

float in space, right before the dancer. At the

space besides AR technology. AR most common-

same time, the way a dancer moves them about

ly uses smartphone screens, heavy headsets or

in space implies that they do, however, have a

other kinds of visual displays that overlay the

certain mass – it just does not cause them to

real world with virtual elements. In illusion-

fall down. And of course, unlike real objects,

based dance, imaginary objects are revealed

these imaginary objects are essentially invisible

to the audience through a dancer’s body move-

and certainly do not occlude what’s placed be-

ment. The dancer can, for instance, run his or

hind them. More than that, they often appear

her hands over the shape of an imaginary object

out of nothing just to disappear in thin air a few

in order to make it appear as if the object is ac-

seconds later. Fascinatingly, it does not bother

tually present [5]. Illusion-based dance reminds

us that these imaginary objects are not really

us that AR is not restricted to digital mediums

present, don’t look like real objects and do not

and that we do not have to resort to computer

behave like anything we know from the physical

technology in order to make virtual objects ap-

world – the objects are believable and convinc-

pear in real space. Maarten H. Lamers discussed

ing nonetheless (cf. [11])!

the Pepper’s Ghost as an instance of pre-digital AR [6] in the third issue of AR[t]. In this regard, dance-illusions can serve as yet another compelling example of AR that remains in the physical

What you see isn’t what you get

domain. I expect multimodal AR to become one of the more

Realism, really?

interesting topics in the future. However, I do not think that a multimodal or richer sensory experience is always better. In their paper on illusion-

AR should be more like reality and virtual objects

based dance styles, Diego et al. [2] make an in-

should both look and behave like real, physical

teresting observation: when dancers let imaginary

objects! At least, this is the impression I get

boxes appear in space through their movement,

from existing AR research. Scientists and devel-

the viewer can interpret this in two different ways.

opers strive for photorealism, they struggle with

Either there is no box in space and the dancer is

occlusion and investigate how virtual objects

moving in a very complicated way or there is a box

can cause reflections and cast shadows just like

in space that guides the movement of the dancer’s

real objects do (see, e.g., [7, 8, 9]). Likewise,

hand. While watching, our eyes tell us that there

virtual objects should behave and interact with

is no box but our body (or our embodied cognition)

the world like real objects [10]. A virtual ball is

tells us that there is. Diego et al. propose that it

supposed to drop and bounce on the floor, just

is “this moment of embodied/cognitive dissonance

like a real ball would. There is certainly noth-

[that] makes the movement compelling” [2, p. 173].

ing wrong with that. However, illusion-based

I believe that AR can benefit from a similar dis-

dance shows us that another approach is pos-

sonance: looking at a breakfast cereal box through

sible. Dance shines when it comes to expressing

our phone’s screen, we see the virtual dinosaur

simple geometrical shapes and structures, such

eating our cereal, but we cannot touch it. Our eyes

as rectangular boxes or walls. In some respect,

tell us “it is there” while our body tells us that it

isn’t. I do not claim that all AR benefits from such

Likewise, I am sure you cannot build any virtual

a dissonance. But I am convinced that getting con-

AR walls without imagining them beforehand.

tradicting information from our different senses can actually add to, rather than subtract from, the

In the future, AR will surely overcome many

overall experience.

technical challenges. However, the future of augmented reality is not only about what is or

The power of movement

will be possible technically. It is also about what we can imagine and how our imagination works. One of AR’s unique powers is that it can be dif-

Ultimately, AR can learn from illusion-based

ferent from our real, unaugmented reality. But

dance that movement is a powerful means to

how can virtual objects differ from real objects

express the presence and properties of virtual

without losing their believability? How can aug-

content. By moving virtual objects through

mented reality differ from reality? Studying re-

space, AR can communicate that which it could

lated arts such as dance, mime or magic helps us

hardly convey otherwise. If a virtual leaf moves

find answers and think outside of our imaginary,

through space in a certain way, its movement

invisible and virtual boxes.

shows us that there is wind. If a virtual ball rolls over a real floor, it tells us something about its weight and resistance. Furthermore, using

References

movement, we are able to create the impression of yet other – invisible – objects being pres-

1. Schraffenberger, H. (2012, November).

ent in space. How would you display an invis-

Chasing virtual spooks, losing real weight.

ible wall with AR technology? Dance gives the

Augmented Running and a side trip into the

answer: by having something bump against it, by

history of Audio Augmented Reality. AR[t] 2,

movement! And there are more possibilities: if

48-51.

a virtual object looks heavy but moves through space weightlessly, we might be able to discern

2. Maranan, D. S., Schiphorst, T., Bartram,

a change in gravity. By rewinding their move-

L., & Hwang, A. (2013, June). Expressing

ments, good dancers are almost able to fool me

technological metaphors in dance using

into believing that time goes backwards. Maybe

structural illusion from embodied motion. In

AR technology can evoke a feeling of time mov-

Proceedings of the 9th ACM Conference on

ing differently by rewinding the movement of

Creativity & Cognition (pp. 165-174). ACM.

objects or by varying their speed. I hope future AR will explore what can be expressed by simply moving virtual objects through real space.

Future AR is not reality, it is our imagination

3. Hwang, A., YouTube channel: http://www. youtube.com/user/phedhex 4. Schraffenberger, H., & van der Heide, E. (2013). Towards Novel Relationships between the Virtual and the Real in Augmented Reality. In Proceedings ArtsIT 2013 – Lecture Notes of

Let us return to the imaginary walls that were

the Institute of Computer Sciences, Social In-

occupying the university’s dance studio some

formatics and Telecommunications Engineering

weeks ago. I am not sure whether these walls

(pp. 73-80). Springer Berlin Heidelberg.

can be called AR. But I am sure that a dancer will not be able to create the illusion of a virtual wall in space without imagining the wall first.

5. Hwang, A. How to Dance Liq-

9. Kanbara, M., & Yokoya, N. (2004, August).

uid: Rails [video], http://youtu.be/

Real-time Estimation of Light Source Environ-

Wwxwc_2vDKQ?t=7m50s

ment for Photorealistic Augmented Reality. In ICPR (2) (pp. 911-914).

6. Lamers, Maarten H. (2013, May). Pre-digital Augmented Reality. AR[t] 3, 24-25.

10. Kim, S., Kim, Y., & Lee, S. H. (2011, July). On Visual Artifacts of Physics Simulation in

7. Agusanto, K., Li, L., Chuangui, Z., & Sing, N.

Augmented Reality Environment. In Inter-

W. (2003, October). Photorealistic render-

national Symposium on Ubiquitous Virtual

ing for augmented reality using environment

Reality (ISUVR), 2011 (pp. 25-28). IEEE.

illumination. In Proceedings of the 2nd IEEE/ ACM International Symposium on Mixed and Augmented Reality (pp. 208-216). IEEE.

11. Schraffenberger, H., & Van der Heide, E. (2013). From coexistence to interaction: influences between the virtual and the real

8. Gibson, S., Chalmers, A., Simon, G., Viguer-

in augmented reality. In Proceedings of the

as-Gomez, J. F., Berger, M. O., Stricker, D.,

19th International Symposium of Electronic

& Kresse, W. (2003). Photorealistic augment-

Art, ISEA2013, Sydney.

ed reality. In Second IEEE and ACM International Symposium on Mixed and Augmented Reality-ISMAR'03.

There is more than meets the eye in Augmented Reality Game Environments by Stephan Lukosch, Heide Lukosch & Alexander Verbraeck

1. The Vision: Actors within augmented reality games Orson Scott Card impressively shows the effect of high fidelity in simulation games in his novel Ender’s Game [1]. The main actor Ender and his team believe they are playing a training simulation game for fighting a war on an alien race. In the end, it becomes clear that Ender was in fact commanding the real fleet through the game, attacking and finally extinguishing the alien race. In his saga Otherland [2–5], Tad Williams describes a

which almost every object is networked and the

future world with a widespread availability of full-

use of augmented reality is normal. Humans in-

immersion virtual reality [6] installations. These

teract within augmented reality [8, 9] by wearing

installations allow people to access an online

smart clothes and contact lenses that can overlay

world, called simply ‘the Net’. Within the Net a

the physical environment with computer graph-

group of people aims to achieve immortality. In

ics. In Rainbows End [7], augmented reality is

his novel Rainbows End [7] Vernor Vinge describes

used for various purposes, e.g., large-scale com-

how the main character Robert Gu is slowly re-

mercial gaming areas, supporting maintenance

covering from Alzheimer’s disease due to medical

workers with blueprints of machines or buildings,

advances in the future. While recovering, former

communication with virtual avatars and diagnos-

technophobe Robert adapts to a changed world in

tic purposes in medical settings.

Realistic agents and environment of the CharliePapa game

Science Fiction authors Orson Scott Card, Tad

They will mature into holistic embodied experi-

Williams and Vernor Vinge forecast our vision on

ences, which are seen as a prerequisite for social

simulation games and augmented reality in the

cognition, with interaction as a crucial element

future. Several years from now, there will be

[10]. Similar to Enderâ&#x20AC;&#x2122;s game, augmented reality

more than meets the eye in augmented reality

environments will provide a degree of fidelity, or

game environments. Instead of only overlaying

realistic representation, which will make it diffi-

the physical environment with computer graphics

cult to distinguish the real world from the virtual

and thereby focusing on human vision, augment-

computer-generated world. The use and the in-

ed reality game environments will address all hu-

teraction within augmented reality environments

man senses, i.e. sound, smell, taste and touch, as

will become as natural as in Vernor Vingeâ&#x20AC;&#x2122;s novel

envisioned in the saga Otherland by Tad Williams.

Rainbows End.

First steps toward the combined visions and ideas

complex problems from different perspectives [16]

of Orson Scott Card, Tad Williams and Vernor

and provide changing scenarios. That makes them

Vinge have already been taken. There has been

powerful tools for players to gain new insights in

quite some research on introducing smell into

a given situation, and to build up or adapt their

movie theaters and television [11] and even more

mental models. The variety of uses for simulation

research on haptic feedback [12]. One of the

games related to complex problems is enormous,

most difficult aspects to reproduce, however, is

varying from teaching about complexity to un-

a high-fidelity interaction with other (real or vir-

derstanding system behavior or testing relations

tual) humans. In 2000, Olson and Olson [13] ana-

and rules between system elements [16, 17]. The

lysed the extent to which groupware technology

military, for example, has a long tradition of us-

allows geographically distributed teams to work

ing simulations for strategy and combat training,

together as if they were co-located. They came

because of the opportunity to clearly illustrate

to the conclusion that distance matters and that

consequences of actions in a safe environment,

the analysed technology is not mature enough to

without risk of injury or other damage [18, 19].

enable virtual co-location yet. Olson and Olson

Ender and his team experience this exact feeling

state that even future technology will struggle to

of safety, meaning to act without causing serious

enable virtual co-location, as providing aware-

consequences when “playing” their game.

ness among co-workers and enabling co-reference as well as spatial referencing will remain a

In Ender’s game [1], the simulation game has a

challenge [13]. Gaver [14], on the other hand, un-

high level of realism. Chalmers and Debattista

derlines the importance of supporting awareness

[20] show that for knowledge transfer, simula-

information to help actors shifting from working

tion games should be designed very much like the

alone to working together.

‘real’ world. However, abstraction and simplification can also lead to excellent training outcomes

Considering current groupware technology, this

[21]. Which level of realism or fidelity is necessary

forecast is still mainly correct. Complex problem

to make a simulation game effective has not yet

solving often requires a team of experts to physi-

been answered [22, 23]. So far, mainly the physical

cally meet and interact with each other, since

fidelity of simulation games, such as visual, spa-

identifying the problem and creating a shared

tial, auditorial and kinesthetic design, has been

understanding is a prerequisite for efficiently

considered [24]. For a simulation game in an aug-

solving a problem [15]. Typical scenarios are e.g.:

mented reality environment, as played in Ender’s

solving complex construction problems, training

game, more dimensions have to be considered.

the usage of complex machinery, analysing com-

These can be the psychological fidelity aspects

plex situations in emergency services or diagnos-

such as stress or joy [22], functional fidelity such

ing complex medical situations. Unfortunately, it

as learning goals or representation of tasks, and

is not always possible to bring a team together

social fidelity with regard to the ability of players

to handle a complex situation. This is due to ex-

to interact with each other or virtual avatars.

perts’ availability, critical timing issues or accessibility of a location. While in the novel Rainbows

technology is not yet there.

2. The present: approaches to augmented reality and simulation games

When considering ‘the Net’ in the saga Otherland

Recent research has shown, however, that vir-

[2-5], simulation games have the potential to pro-

tual co-location is in fact possible. Within the

vide a rich environment and the ability to approach

project CSI The Hague (http://www.csithehague.

End [7], such situations are supported with high fidelity augmented reality technology, current

com), the Netherlands Forensic Institute (NFI) in-

et al. [26] show that such virtual avatars are pos-

vestigated the use of new technologies to train

sible, but are currently fixed to one location due

forensic investigators and also help forensic in-

to the necessary technology such as 3D beamers

vestigators in the field. One part of the project

or depth cameras.

explored the potential of augmented reality for crime scene investigation. The project particu-

A project targeting the work of reconnaissance

larly focused on the real-time alignment of physi-

teams called CharliePapa, a joined effort of Delft

cal and virtual environments and interaction

University of Technology, the Dutch Police, a

with the environment using augmented reality

private security firm, a training institution and

to support collaborative spatial analysis on loca-

a developer of simulation game scenarios, ex-

tion [25]. By means of augmented reality, remote

plored the effect of highly realistic virtual sce-

experts were able to connect to investigators on

narios on team awareness and interaction [27].

the crime scene and guide and help in the in-

Three different scenarios of a highly realistic ur-

vestigation by providing their expertise, but also by adding virtual objects to the perceived reality of the investigator on the scene (see Figure 1), establishing virtual co-location. An evaluation of this approach has shown that the visual feedback in augmented reality provides mutual understanding of the analysis state. One expert stated: “There is not much arguing about something you can both see”. The information exchange was very visually oriented, e.g.: “behind the table”, “don’t cross the ribbon

Figure 1. A staged crime scene augmented with restricted area ribbons

on the far right”, “if you look slightly to the left”. There was ambivalent feedback on the recording and remote observation. Some were afraid of being monitored in their work while others welcomed the possibility to capture best practices. The evaluation further showed that the presence of the remote expert as well as the awareness of the remote expert’s activities at the crime scene needs to be improved, e.g. by using virtual avatars as described by Vernor Vinge in Rainbows End [7]. Beck

Figure 2. A scene from the CharliePapa simulation game

ban environment (see Figure 2) were developed

taken. However, this highly realistic set-up was

and tested with the target group, reconnaissance

not sufficient in really developing an immersive

teams. Tasks and stress level as well as time pres-

activity like those experienced by the heroes of

sure were represented as realistically as possible.

the science fiction novels mentioned earlier. Re-

The reconnaissance teams focused on protecting

alistic interaction with the virtual environment is

a VIP and detecting deviant behaviour of other

of utmost importance, but recent technology of

subjects at the site. The assignment of the most

â&#x20AC;&#x153;pureâ&#x20AC;? simulation game environments and agents

mature scenario was to walk through the environ-

does not have the ability to create a high-fidelity

ment with three players in a row, each of them

experience with a high level of physical fidelity.

confronted with objects, persons and actions that

Human ability to focus on an object is present in

were partly the same and partly different from

reality but is hard to resemble in a game, as one

those of the other players, and to pass over in-

only has a wide-angle view. Furthermore, the in-

formation gathered from the virtual environment

teraction with people and objects in the game is

in order to make a decision on the actions to be

still too slow and cumbersome, so functional and

Free hands user interface in augmented reality

social fidelity was too low as well. By add-

ficult to design a holistic embodied experience,

ing a time limit and competition element,

based on a high level of physical, functional, psy-

the psychological fidelity level was the only

chological and social fidelity with current com-

factor that was high enough, as reported by

mon simulation game technologies.

the test group, to simulate a realistic experience. From these experiments we can conclude

3. The future: augmented reality simulation games

that Olson and Olson’s observation that it remains very difficult for current technol-

For Ender, it is a shock to learn that the simulation

ogy to enable virtual co-location [28], also

he commanded was not a game, but that it was

holds for a high-fidelity virtual environment

reality, and that he killed a whole race with his

within a simulation game. This result can be

actions. Augmented reality game environments

enhanced with the finding that it is very dif-

for training purposes and virtual co-location in professional working environments should not be as shocking as his insight, and will not be as massive as the “game” Ender was made to play. On the other hand, combining augmented reality with simulation games and making use of the advantages of both technologies can provide environments of high fidelity and immersion, leading to high effectiveness of professional teams. A feeling of “being there” can be achieved, supported by a very realistic training experience. Augmented reality is already more than a visual experience; it enables graphical real-time simulation, which can support fast and intuitive understanding of a situation [8, 9]. Still, there are a lot of current and future issues in simulation games and augmented reality environments that need to be addressed in order to make the combined vision of Orson Scott Card, Tad Williams and Vernor Vinge possible. With advances in hardware and software capabilities, camera-equipped mobile devices continually gain more interest in augmented reality games [29, 30]. Recent studies show that such mobile devices (MD) can foster virtual co-location of distributed players [28, 29] and enlarge the game experience by blending virtual and physical game spaces [31]. Players are no longer bound to an experience provided only by a computer interface or a physical environment, but can enter an enriched physical world, with virtual objects and agents adding real-time information. The en-

tertainment gaming industry is already pav-

References

ing the way. Ingress [32] is an augmented reality massively multiplayer online game

1. Card, O.S. (1985). Ender’s Game. Tor Books.

created by Niantic Labs, in which players

2. Williams, T. (1996). Otherland - City of

belong to one of two factions (Enlightened or Resistance) for which they have to conquer territory. Although the physical fidelity of the game with regard to augmented reality is rather low, several hundred thousand players are engaged within the game. The future will bring even more immersive augmented reality games that make use of

Golden Shadow. Legend Books. 3. Williams, T. (1999). Otherland - Mountain of Black Glass. Legend Books. 4. Williams, T. (1998). Otherland - River of Blue Fire. Legend Books. 5. Williams, T. (2001). Otherland - Sea of Silver Light. Legend Books. 6. Milgram, P. and Kishino, F. (1994). A tax-

novel head-mounted devices (HMDs), e.g.

onomy of mixed reality visual displays. IEICE

Google Glass [33], Oculus Rift [34] or Meta

Transactions on Information Systems. E77-D,

Spaceglasses [35], and combine the visions of Orson Scott Card, Tad Williams and Vernor Vinge.

12 (1994). 7. Vinge, Vernor (2006). Rainbows End. Tor Books. 8. Azuma, R. et al. (2001). Recent advances in

In order to realize this vision for both entertainment and more serious purposes such as problem solving and training, augmented reality games first have to become more than meets the eye and address all human senses, i.e. sound, smell, taste and touch,

augmented reality. Computer Graphics and Applications, IEEE. 21, 6 (Dec. 2001), 34 –47. 9. Azuma, R.T. (1997). A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6 (1997), 355–385. 10. Hanne De Jaegher and Ezequiel Di Paolo

as envisioned by Tad Williams. Secondly, us-

(2007). Participatory Sense-Making An Enac-

ers will have to be aware of each other’s

tive Approach to Social Cognition. Phenom-

activities by using an augmented reality

enology and the Cognitive Sciences. 6, 4

environment for spatial remote collaboration, as described by Vernor Vinge. The fi-

(2007), 485–507. 11. Kim, H. et al. (2011). An X-Y Addressable

nal and most difficult challenge seems to

Matrix Odor-Releasing System Using an On-

be to create a realistic interaction between

Off Switchable Device. Angewandte Chemie.

real objects and virtual objects [36]. This will enhance the fidelity of the game play

123, 30 (2011), 6903–6907. 12. Samur, E. (2012). State of the Art. Per-

similar to Ender’s experiences as described

formance Metrics for Haptic Interfaces.

by Orson Scott Card. When these three chal-

Springer London. 9–26.

lenges are addressed, a realistic multi-mod-

13. Olson, G.M. and Olson, J.S. (2000). Distance

al, multi-user game experience can be cre-

matters. Human-Computer Interaction. 15, 2

ated that is in many ways indistinguishable from reality in most senses. This will enable

(2000), 139–178. 14. Gaver, William W. (1991). Sound Support for

a whole new generation of applications that

Collaboration. Proceedings of the Second

will benefit from distributed people inter-

Conference on European Conference on

acting naturally with each other and with

Computer-Supported Cooperative Work

their synthetic environment.

(1991), 293–308. 15. Piirainen, K. et al. (2012). The Joint Struggle of Complex Engineering: A Study of the Challenges of Collaborative Design. International

Journal of Information Technology & Decision Making (IJITDM). 11, 6 (2012), 1–39. 16. Geertje Bekebrede (2010). Experiencing complexity: a gaming approach for understanding infrastructure systems. Delft University of Technology. 17. Klabbers, J.H.G. (2006). The magic circle :

26. Beck, S. et al. (2013). Immersive Group-toGroup Telepresence. IEEE Transactions on Visualization and Computer Graphics. 19, 4 (2013), 616–625. 27. Lukosch, Heide et al. (2014). Building a Virtual World for Team Work Improvement. Frontiers in Gaming Simulation. Springer,

principles of gaming & simulation. Sense Pub-

London/Dordrecht/Heidelberg/New York.

lishers, Rotterdam, The Netherlands. 141–166.

60–68.

18. Bonk, C.J. and Dennen, V.P. (2005). Mas-

28. Broll, W. et al. (2006). Meeting Technology

sive Multiplayer Online Gaming: A Research

Challenges of Pervasive Augmented Reality

Framework for Military Training and Educa-

Games. Proceedings of 5th ACM SIGCOMM

tion. Technical Report #TECH-RPT-2005-1.

Workshop on Network and system support

Indiana University, Bloomington. 19. M. Macedonia (2002). Games Soldiers Play. IEEE Spectrum. 39, 3 (2002), 32–37.

for games, Netgames’06 (2006), Article 28. 29. Flintham, M. et al. (2003). Where on-line meets on the streets: experiences with

20. Chalmers, A. and Debattista, K. (2009).

mobile mixed reality games. Proceedings of

Level of Realism for Serious Games.

the SIGCHI conference on Human factors in

Proceedings of the Conference in Games and Virtual Worlds for Serious Applications (2009), 225–232. 21. Toups, Z.O. et al. (2011). The team coordination game: Zero-fidelity simulation abstracted from fire emergency response practice.

computing systems (2003), 569–576. 30. Rohs, M. (2007). Marker-based embodied interaction for handheld augmented reality games. Journal of Virtual Reality and Broadcasting. 4, 5 (2007), 1860–2037. 31. Magerkurth, C. et al. (2005). Pervasive

ACM Transactions on Computer-Human

games: bringing computer entertainment

Interaction. 18, 4 (Dec. 2011), 23:1–23:37.

back to the real world. Computers in Enter-

22. Alexander, A.L. et al. (2005). From Gaming to Training: A Review of Studies on Fidelity, Immersion, Presence, and Buy-in and Their Effects on Transfer in PC-Based Simulations and Games. Aptima, Inc., Woburn, MA. 23. Harteveld, C. (2011). Triadic Game Design: Balancing Reality, Meaning and Play. Springer, Berlin. 24. Feinstein, A.H. and Cannon, H.M. (2001). Fidelity, Verifiability, and Validity of Simula-

tainment (CIE). 3, 3 (2005), Article 4A. 32. https://www.ingress.com/, last accessed 07/03/14 33. http://www.google.com/glass, last accessed 07/03/14 34. http://www.oculusvr.com/, last accessed 07/03/14 35. https://www.spaceglasses.com/, last accessed 07/03/14 36. Schraffenberger, H., & Van der Heide, E.

tion: Constructs for Evaluation. Technical

(2013). From coexistence to interaction:

Report #2001-006. Wayne State University

influences between the virtual and the real

Marketing.

in augmented reality. In Proceedings of the

25. Poelman, R. et al. (2012). As if Being There: Mediated Reality for Crime Scene Investiga-

19th International Symposium of Electronic Art, ISEA2013, Sydney.

tion. CSCW ’12: Proceedings of the 2012 ACM conference on Computer Supported Cooperative Work (2012), 1267–1276.

Corona: Audio AR for historic sites by Florian Heller

The Coronation Hall in the historic town hall of Aachen, Germany

Many historic sites, as important they may have

Hall were the coronation feasts of the 15th and

been, often lack visual cues to events that took

16th century. However, the only visual remainder

place there. Sites become ruins or are changed

is a set of coats of arms engraved in the

intentionally through structural modification.

pavement. The Route Charlemagne1 (www.route-

The city hall of Aachen, Germany is an example

charlemagne.eu) project was initiated to make

of such a site. Built in the 9th century, it was

several historic buildings in Aachen accessible to

the seat of Charlemagneâ&#x20AC;&#x2122;s and several other

the public. The main focus was on the city hall,

emperorsâ&#x20AC;&#x2122; governments, and it is still in use

since it was only possible to be visited as part of

as city hall today. Among the most important

a guided tour. To allow a self-paced exploration,

ceremonies that took place in its Coronation

we created a series of interactive exhibits and

the Aixplorer audio guide, which automatically

gorging on meat and drinking from the fountain

detects the room you are in. For the Coronation

of wine. The nobility had to follow protocol. The

Hall, we wanted to create a landmark exhibit

ceremony of the coronation of Charles V from

that reminds visitors of the important ceremonies

the 16th century is very well documented, which

that happened there. As the hall is regularly used

gave us the opportunity to bring some of the at-

for public activities and is under monumental

tendees back to life.

protection, fixed installations were not an option. Since the visual impression of the room had to

In the Corona audio space, virtual characters are

stay untouched, we decided to create an audio-

placed at their historically handed down positions.

only exhibit.

In groups of two they discuss different aspects of the ceremony. For example, the newly crowned

The Corona audio space

King discusses matters of the Black Death with the Archbishop, servants describe the order of the dishes, and two persons standing by the window

Corona is an audio augmented reality experience

observe the festivities on the market place.

that overlays the physical space with a virtual audio space, generating an impression of being right in the middle of a coronation feast. You would probably imagine such a banquet as a lavish celeÂ

The technology behind the scenes

bration, but that was only true for the populace To create the illusion that the audio sources are located at fixed positions in the physical space, we need information about the userâ&#x20AC;&#x2122;s position and head orientation. The Coronation Hall is 45 by 20 meters large with four stone pillars in the middle of the room, which makes location tracking difficult if it is supposed to be invisible. Furthermore, optical tracking using video cameras has to meet strict privacy regulations. We opted for a Ubisense wireless radio location tracking system that performs with an accuracy of 1050 cm and delivers updates 4 times per second. The orientation is measured by a digital compass chip, with an update rate of 10 Hz, mounted to the headphones. This information is then fed into the OpenAL spatial audio rendering engine available on iOS. Since the results of this engine do not provide a good separation of sources that are directly in front or behind you (a common problem with spatial audio rendering called frontback confusion), we added a low-pass filter to the signal coming from a source behind the listener. To avoid an auditory overload resulting in poor localization, we pause sources that are more than 11 meters away. Corona use concept 82

The overall result of the rendering is not perfectly realistic, but plausible and results in an entertaining experience. In fact, a realistic simulation of the room’s acoustics would lead to a poor understanding of the dialogs as the hall’s natural reverb makes it impossible to communicate over a distance larger than eight meters.

Individual audio spaces As with traditional audio guides, the audio space is individual to the listener. Usually, this is used to provide the content in different languages or for different audiences, e.g., a simplified version for children. With our continuous audio space, however, we can create a more playful approach. Moving sound sources could lure people to areas that they have not yet explored or create an audio scavenger hunt. The continuous audio space also has the advantage that features which need manual intervention or additional implementation effort with traditional audio guides, such as synchronized

Spatial layout of auditory sources in the Coronation Hall

playback, can easily be achieved. Depending on the path that a user takes, the different sources

only actively listening to one of these. This means

start and stop playing at different points in time.

that key information has to be presented repeat-

Thus, to experience the audio space as a group,

edly, but using short audio fragments bears the

you just have to walk side by side, without having

risk that the visitor notices the loop. In our con-

to make sure that you press play at the same time.

tent, key information is repeated several times in different words, allowing a late drop-in.

A visitor explained his impression as follows: “Corona is an emotional experience, like a film. This is much more interesting and thus memorable than a normal audio guide”. The act

Simplifying the implementation

of discovering hidden information was clearly favored over learning plain facts as with tradi-

The human brain is quite good at making sense out

tional audio guides.

of sensory information that does not fit exactly. For example, if a virtual sound should emerge from

Challenging the authors

a specific physical artifact, but the perceived location of the virtual sound source and the location of the artifact slightly differ, the virtual source seems

The challenge in writing the dialogs of the audio

to snap to that physical location (the so called ven-

space was mainly the fact that we did not know

triloquist effect [1]). A visitor experienced a simi-

when the visitor would actively be listening to

lar effect when the network connection used to

the source. In contrast to classic audio guides,

transmit the location data broke down. Since the

several sources might be playing but the visitor is

rendering engine could only use the head orienta-

tion, the distance to the sources was not updated

device orientation does not dramatically affect

anymore. She interpreted this a bit different and

the perceived presence in the virtual environ-

told us: “That was amazing! After some time, the

ment. The localization accuracy does not neces-

voices started walking with me!”

sarily need to be as high as in our implementation. A series of outdoor installations, a sound

Based on these effects, and the observation that

garden in a municipal park [3] for example, have

many users of our system do not turn their head

used GPS to get location information and this was

to orient themselves in the audio space, but in-

not perceived as a problem.

stead turn their entire body, we investigated whether the implementation of a system like Corona could be simplified. The hardware require-

Outlook

ments for this exhibit make it a complex installation; every headphone needs to be equipped

Audio augmented reality applications are engag-

with a compass, which has to be interfaced with

ing experiences that go beyond the plain presen-

the smartphone. Similarly, the location informa-

tation of historic facts. In the context of a mu-

tion has to be communicated to the audio guides

seum it can be used to create an atmosphere the

as well. Since current smartphones are equipped

user dives into and that stimulates the visitor’s

with location and orientation sensors, the re-

fantasy. Current smartphones provide enough

quired hardware is basically already available.

processing power to handle the spatial audio ren-

Using these built-in sensors does not allow the

dering and if realism is not the top priority, their

same degree of realism, since they only measure

built-in sensors provide all required information.

the orientation of the device - not the head, but

So there aren’t any excuses not to have an audio

our current experiments [2] indicate that using

AR app for your museum.

Acknowledgements This work was financed by the German B-IT Foundation and the state of Northrine West phalia through its EU-ERDF program ``Ziel 2’’.

References 1. Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current biology, 14(3), 257-262. 2. Heller, F., Krämer, A., & Borchers, J. (2014, April). Simplifying orientation measurement for mobile audio augmented reality applications. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (pp. 615-624). ACM. 3. Vazquez-Alvarez, Y., Oakley, I., & Brewster, S. A. (2012). Auditory display design for exploration in mobile audio-augmented reality. Personal and Ubiquitous Computing, 16(8), 987-999.

Corona in use

Florian Heller Florian Heller is a PhD Student in the Media Computing Group at RWTH Aachen University in Germany. From 2009 to 2012 he was a fellow of the B-IT Research School. He worked on the implementation of Corona as his Diploma Thesis and continued his research in the field of physical interaction with audio. The physical component can either be movement like in Corona, or interaction with tangible interfaces such as an augmented DJ turntable or with wearable interfaces.  http://hci.rwth-aachen.de/heller  http://hci.rwth-aachen.de/corona

Developing augmented reality applications: a future perspective by Robert Prevel

Augmented reality (AR) applications, once

the public domain is to be encouraged, then

seen as a future technology, have in recent

our future hardware and software solutions

years become far more prevalent in the

must be affordable and readily available.

public domain. This is in large part due to

The quality of an AR experience depends on

the availability of affordable hardware that

both the hardware and software used to de-

can be used to facilitate many basic AR ap-

liver that experience. For example, a heads

plications, the most common example being

up display provides a far more immersive AR

the smartphone. These lightweight compact

experience than a hand held display. Unfor-

devices typically consist of a camera, on

tunately, unlike smartphones, AR headsets

board processing, a screen, and other use-

are not readily available. In addition, even

ful functions such as wireless capability. As

the cheapest self-assembled headsets are

smartphones and in particular smartphone

more costly than an average smartphone.

applications have become more widespread

As the hardware used in AR applications

in society, there has been a notable increase

develops, so too will the software allowing

in the number of AR applications and devel-

the best experience from the available hard-

opment companies.

ware. AR application development is a cycle where new applications require improved

This growth of augmented reality in the pub-

hardware, which allows for new software

lic domain is very positive. However, most

ideas that in turn require new hardware. We

existing commercial AR applications are con-

predict a number of ways in which both hard-

fined to small working environments, and

ware and software for AR applications will

provide a very basic AR experience; they

develop in order to improve on our current

allow you to play virtual ‘desktop’ games,

AR experiences.

or overlay a simple virtual object or image on top of a real world object. Many of these

In the future, hardware for AR will be far

‘desktop’ applications have very little knowl-

lighter and more comfortable, allowing for

edge of the working environment, and use

extended use. Devices capable of providing

what we would already consider ‘old technol-

a heads up display have already started to

ogy’ for tracking; namely a fiducial marker or

infiltrate the public domain, the most publi-

some equivalent. Future AR applications will

cised being Google Glass (GG). At present GG

most certainly require more detailed knowl-

is rather expensive and not a suitable display

edge of much larger working environments

for most existing AR applications (unless you’re

and more robust tracking solutions. If the

happy playing your desktop games in a small

continued expansion of AR applications into

box in the top right of your vision). Despite

this, GG is very promising in terms of hard-

our environment must also expand. We will

ware development for AR. Most existing head

need to be able to enter an entirely unknown

mounted displays (HMDs) capable of delivering

environment and learn the boundaries and

an AR experience are far too heavy and bulky

the kinds of objects therein. Software that is

for extended use. In this regard, GG gives us a

capable of mapping the environment in real-

glimpse of what we can expect from a light-

time is already being developed, an example

weight, compact HMD in the future.

of this is Google’s ‘Project Tango’. Knowl-

In addition to issues of comfort, the HMDs

edge of our working environment is key to

of the future will need to solve a number of

providing a believable interactive AR experi-

existing problems. Firstly, whilst the technol-

ence. Lastly, we will want multiple users to

ogy behind current HMDs varies, when used

share experiences in the same environment,

for AR applications they all provide a dimin-

so that the actions of one user are observed

ished view of the environment; a reduced

by others. This will require an advanced level

field of view or resolution for example. Sec-

of data sharing and communication between

ondly, there are safety concerns with display-

devices.

ing too much or ill-timed information directly in front of the user, particularly if that user

We have given many demonstrations of our

is performing an attention demanding task at

work at the AR lab, and people’s experiences

the time, such as driving (though it could be

are usually similar. Some come to us with no

argued that this is less of a safety issue and

prior experience of AR, others with some ex-

more a demonstration of the user’s lack of

periences through a smartphone or tablet.

common sense). Lastly, the public accept-

There is something special about putting on

ability of using HMDs in public places must

a HMD; whether it’s the highly immersive 3D

improve. We predict that once these issues

AR experience, or the novelty of wearing fu-

have been addressed, it will be the norm to

turistic looking headgear. We expect that this

wear lightweight compact HMDs in public. In-

novelty will wear off as people become more

deed, in the future our children will laugh at

used to wearing HMDs in everyday life. Our

the fact that we used to use our hands for

augmented reality applications will be pro-

such simple things as holding devices.

viding on the spot instructions for cooking, furniture assembly, and car maintenance, as

In order to leave the desktop behind and ex-

well as providing us with directions to the

plore the wider augmented world, a number

restaurant. In the future, AR HMDs will have

of software challenges must be met. The

become just as common as smartphones are

key to successful tracking in AR is to identify

now. We expect that the development of ad-

stationary landmarks, be they the corners

vanced, interactive AR applications will be

of a fiducial marker or the natural features

driven by the gaming industry initially. But

present in the environment; this becomes far

the potential for AR is limitless, not forget-

more difficult when working in dynamic en-

ting that our experience is not limited to vi-

vironments. If we are truly to explore larger

sion and audio alone. Certainly the most ex-

environments, we will undoubtedly observe

citing applications for AR are in the medical

moving objects. Calculating which landmarks

domain. Imagine your surgeon standing over

belong to stationary objects in a dynamic en-

you, with their AR HMD displaying where to

vironment is a must. Changes in environmen-

make the incision for your neural interface.

tal lighting will also become more frequent,

‘Yes sir, and if you put on your bedside HMD

requiring more robust tracking algorithms. In

you’ll be able to see what your bionic arm

addition to tracking, the process of mapping

will look like’.

by Dirk Schart, Diego Montoya & Melo Montoya

State of the Art – A future perspective The new advances in technology enable exploration of all kinds of possibilities. Works found in galleries don’t have to be static. Paintings and sculptures can move and come to life through these advances. The digital artistic works are no longer contemplative, but rather participative: actions by the observers nearby can be digitally measured and the work can react accordingly. The younger generations – especially the digital Piet Mondrian, Composition with Large Red

natives – like to discover through interaction, and

Plane, Yellow, Black, Gray and Blue (1921)

they know their way around technology. On the

Piet Mondrian contributed greatly to the De Stijl

to interact with the digital world in a more natural

movement, also known as neoplasticism, and

way. Also, the capabilities of algorithms running in

found beauty in the abstraction of simple, har-

powerful micro-processors allow for real-time art

monious forms. But what if he would have been

production on devices that fit in our pockets.

other hand, the older generations have found ways

free from the constraints of traditional two-dimensional painting, being able to use the form

With all this in mind, Diego Montoya, an Augment-

in all directions, including time? Would he have

ed Reality developer, and his brother and multi-

rather stayed in the flatness of the canvas, or

media artist Melo Montoya intervened in some of

would he have pursued his search for beauty in

Piet Mondrian's works. One of their goals was to

all dimensions? What if he could have given the

reach a wider range of target groups and fasci-

spectator a role, giving him a part to play in the

nate them for art. They also wanted to establish

harmony of his works?

a bridge between the latest digital developments,

often completely unrelated to art, and the work of the influential plastic artist Piet Mondrian, to present the digital technology not as an object, but as a medium for art.

The idea behind HyperMondrian

Piet Mondrian, Composition with Lines (1917)

The Montoya brothers chose a game engine in order to animate and present three-dimensional data and image tracking as a way to connect the

Compositions in 3D

physical works and the digital imagery. Using this technique, they developed an app in col-

The first one, Composition with Large Red Plane,

laboration with the Augmented Reality company

Yellow, Black, Gray and Blue, serves to extend

RE'FLEKT, that shows new features on top of the

Mondrian's theory. The elements found in the

existing paintings when seen through the lens of

original become three dimensional and mobile,

an Android or iOS device.

with time intervals and range of movements depending on mathematical relations. To respect

HMondrian stands for Hyper-Mondrian in refer-

the original harmony of all the elements, the

ence to the hyper-plasticism, a proposed exten-

augmented version of the painting reconstructs

sion of the neoplasticism. Through the use of the

the original work when seen from the correct

new technologies such as 3D tracking, gesture rec-

perspective, even if the pieces are in permanent

ognition and generative algorithms,neoplasticism

movement.

acquires new properties; exploring new dimensions and allowing the spectator to participate,

The second painting is Victory Boogie Woogie. Re-

and thus acquiring the prefix "hyper".

grettably, the painter died before finishing it. Because of that, the augmented version of this work

The first step was deciding what the main ele-

can't hold itself together, so it crumbles down in

ments of the app should be and what it would

pieces and reforms as the user moves closer to or

be ableto do, taking advantage of the capabili-

further away from the augmented painting in a

ties of the 3D visualization and image tracking.

juxtaposition of time and space.

Mondrian was a prominent person, and the idea was to explore his art, his life and his creative

For Mondrian, the working environment influ-

environment. After experimenting with various

enced the creative process greatly and therefore

paintings and reviewing the story behind them,

he arranged it accordingly. When seen through

three of his most symbolic works were chosen to

the mobile device, the augmented version of the

do this: Composition with Large Red Plane (1921),

third work Composition with Lines reveals a stu-

Victory Boogie Woogie (1942-1944) and Composi-

dio inspired by those were Mondrian used to work,

tion with Lines (1917).

showing decorations that recreate his paintings.

The presentation This intervention of Mondrian’s work was presented in the Monitor Digital (MOD) - one of Latin America’s most important digital art festivals in Guadalajara Mexico in 2013. There were some iPads available to the public to let them use the app and visitors were able to scan a QR code to download the app on site. HMondrian was well received by the public. The people visiting the festival were very diverse and they had all kinds of reactions, ranging from young children exploring the 3D work as if it was a real object, to the old lady who screamed “Oh! Piet Mondrian, Victory Boogie-Woogie (1942–44)

I broke the painting!” and who then dropped the iPad after seeing the virtual Victory Boogie Woogie crumble down.

The implementation

Armed with the experiences they gathered while developing and presenting this work, the au-

Presenting 3D imagery in a mobile device re-

thors are currently working on other hyperplastic

quires choosing the right tools to do it, and in

works that involve the tracking of sculptures and

the case of HMondrian, the 3D scenes were devel-

using the spectator as a living canvas.

oped using Unity3D, a game engine primarily used to develop mobile games. In order to locate and position the elements added to the paintings an

How does it work

image tracking framework, Vuforia, was chosen from a handful of options because of the charac-

Scan the QR code and download the app “HMon-

teristics of the project. This framework is espe-

drian” from the App Store or Google Play Store.

cially good at tracking simple geometric figures,

Once you open it there are two modes available

like those found in Mondrian’s work. The model-

– the Augmented Reality and the Virtual Reality

ing of the 3D objects as well as the animations

mode. Choose the Augmented Reality mode to

were done in Maya.

enjoy the extended versions of Mondrian’s works. Point the camera of your device towards one of

For this work, it was important that the tracking

the paintings featured in this article. You can also

was stable and fast to swiftly grab attention and

download them from the app itself. Just tap on

involve the user. Illumination of the room where

the info button and scroll down

the paintings were displayed had to be carefully

to find the download button. The

set up. Another factor that could affect the im-

Virtual Reality mode just serves

age tracking was the size of the images, but the

to show the 3D-functionality

artists verified that the sizes of the original works

in case you are not around the

were adequate and gave good results.

paintings or a copy of them.

Diego Montoya Having obtained degrees in electronics and information systems, he works on developing augmented reality and natural user interface applications and lives in Munich. He is an enthusiast of interactivity, generative design and the theories of the universe and an admirer of the modern and post-modern art movements. Diego enjoys creating digital art works using state-of-the-art technologies to blur the line between the art and the digital, the observer and the object, the real and the virtual.

Melo Montoya Melo studied Multimedia at CAAV. He is a programmer, open format DJ and loves the sun.. He has presented works along with collaborators in the main Digital and Contemporary Art museums in Guadalajara Mexico such as MOD, MAZ and MURA The internet loves him and is the source of his collection of quirky content. He now lives in “La perla tapatía” but he wishes to return to the Caribbean after doing a Master’s in Europe.  www.melomontoya.com

Dirk Schart Dirk Schart is a communication and PR expert, blogger and author with a strong passion for digital communication and interactive media. He holds a Master degree in communication science and was in 2013 awarded for his communication concept "connect & climb". Dirk is Corporate Communications Manager and Spokesperson at Munich-based Augmented Reality company RE'FLEKT.

The so-called Augmented Reality by Pawel Pokutycki

Augmented Reality is probably one of the most

ery technological innovation is about some kind

pretentious, exaggerated and yet philosophi-

of augmentation of our reality, no matter if it’s

cally challenging terms in the field of computer

an automobile, a vacuum cleaner or a light bulb.

technology. Next to concepts of Virtual Real-

Technology enriches our life (arguably of course,

ity, popularized by a spiritually inclined music

as it’s not always for better). Take, for instance,

composer and computer scientist Jaron Lanier,

religion. Isn’t it all about the augmentation of life

or Mediated Reality, formulated by a wearable

(and death)? Or science in general? Medicine? Lit-

computing inventor and “the world’s first cy-

erature? These disciplines offer a better under-

borg” Steve Mann, Augmented Reality stands for

standing of the world we live in and a potentially

a peculiar understanding of ‘reality’ defined not

better, augmented life as a result of understand-

by the acknowledged writings of Aristotle, Plato

ing it and doing something with that knowledge.

or Wittgenstein, but by the pioneers of digital

For example, we would never think of a pacemak-

revolution, or rather, geeks of our times. Just

er as an Augmented Reality application, yet it un-

google AR to see it’s not about Leibniz’s “pos-

doubtedly changes our reality for the better if we

sible worlds” or Kant’s “transcendental ideal-

suffer from an inadequate heart rate. So does the

ism”, but “computer-generated sensory input”

Bible or the Quran, if we are religious. Augmenta-

and “pattern recognition”. Computer-oriented

tion is a predominant desire in human life, either

terminology is dominating the discourse on Aug-

in terms of spiritual enlightenment, or purely

mented Reality and therefore is limiting its ideo-

pragmatic and rational development enhancing

logical significance, going otherwise far beyond

the quality of survival. The so-called Augmented

technology itself. But shouldn’t we - in times

Reality, as defined by its “pioneers”, computer

of Google Glass – not only discuss what kind of

scientists and hardware and software develop-

image processing software or head-mounted

ers today, is therefore only another step in the

displays are used in AR technology, but also, if

evolution of augmentations of all kind, claiming

not primarily, what does it mean to society and

rights to a phenomenon, which is much broader

culture to be in Augmented Reality, to live and

and universal, known already long time before

think in it permanently in the future? And if so,

the digital revolution.

how to reflect on it from the perspective of humanities?

Koert van Mensvoort, in his essay “Real Nature is not Green” [1], draws a parallel between nature

‘Augmented Reality technology’ might be con-

and culture, discussing processes in which nature

sidered a pleonasm. In other words, almost ev-

becomes culture, and culture becomes nature.

 (p. 93-94) Maarten Broekhuizen

“I believe the way we draw the boundary between nature and culture will change. The domain of origin, of ‘birth’ previously belonged to nature, while culture encompassed the domain of the ‘made’. Thanks to developments of science and technology, this distinction is blurring. (...) Culture is that which we control. Nature is all those things that have an autonomous quality and fall outside the scope of human power. In this new classification, greenhouse tomatoes belong to the cultural category, whereas computer viruses and the traffic-jams on our roads can be considered as natural phenomena.” Koert van Mensvoort 94

In the case of Augmented Reality, we deal with a

to Augmented Reality”, in order to finally upgrade

similar problem of classification and interpreta-

the meaning of this term in the public debate.

tion. This leads to questions such as: what kind

Otherwise, we may end up doing things irrelevant

of reality are we augmenting? What do we mean

to what really matters in the long term; only in-

by reality, how do we define it? Is augmentation

venting AR gadgets that become culturally obso-

making reality more real or less real? Is it culture

lete just a day later and therefore falling in the

or nature?

trap of “future thinking”, which is neither about future, nor about thinking.

These are, of course, difficult questions to answer. I claim, however, that in the discourse on Augmented Reality there is a tendency to over-

References

look the broader picture and to ignore the historical, ideological and philosophical foundations of

1. Van Mensvoort K. (2006). Real nature is not

this trending topic. It is perhaps time for some-

green. Article on NextNature.com, Retrieved

body like Slavoj Žižek (or another popular con-

from http://www.nextnature.net/2006/11/

temporary thinker) to make “The Pervert’s Guide

real-nature-isnt-green/

Pawel Pokutycki Pawel Pokutycki is an interaction designer, thinker and lecturer at the Royal Academy of Art (KABK) in The Hague, where in 2005 he initiated early activities of the RFID Lab, in collaboration with Yolande Kolstee and Wim van Eck later becoming the AR+RFID Lab. His recent artistic and educational projects focus on exploring relationships between new media theory and political, social or cultural studies by concept development and prototyping for interactive media. He believes in a methodology of design based on his own, peculiar interpretation of the Black Box Theory, presented at a TEDx event in 2012.  http://tedxtalks.ted.com/video/ Exploring-the-Black-Box-Pawel-P  Simone Engelen

Radioscape, Wormhole Dordrecht and Beyond On the Sensation of Presence by Edwin van der Heide In the AR[t] 2 magazine I wrote about my work

through the environment you develop a perceptual

Radioscape and proposed that, although it was

understanding (and sensation of presence) of the

not meant to be an augmented reality artwork, it

signals in the space around your body. Radioscape

makes a lot of sense to approach it from an aug-

is special in that the transmitted compositions mix

mented reality perspective [1]. Radioscape cre-

in space. Each different position (of the receiver)

ates a parallel reality intersecting the streets of

in space corresponds with a different unique mix

an urban environment. It uses 15 radio transmit-

of the signals.

ters distributed over the chosen city area, each transmitting its own specific composition. Togeth-

In 2008, Joost Rekveld and I were asked to orga-

er these compositions form the meta composition

nize an exhibition with alumni of the ArtScience

of the work. The audience gets a custom handheld

Interfaculty in the Hague for the CBK (Center for

receiver to explore and navigate through the work

Visual Art) in Dordrecht. For a number of years,

(and the city). By moving the receiver and walking

I had been considering a software version of Ra-

The locations and ranges of the sounds of one of the Wormhole Dordrecht sound worlds. Image courtesy Studio Edwin van der Heide. Map data: Google, Aerodata International Surveys, Tele Atlas.

dioscape: a version using a mobile handheld de-

perception, or the perceptual creation, of space.

vice containing a small computer that could play

Radioscape has a true immersive quality that I had

and mix audio files and would be equipped with

also expected as a result of the Wormhole Dor-

a GPS sensor to make it possible to specify and

drecht software. Any movement of the Radioscape

control which sounds would play (and how loud) in

receiver, small or big, leads to a very direct change

relation to the position of the device. In that year

in the received signals. Different movements and

Apple opened up iPhone software development

positions of the receiver result in equally different

to third party developers and I realized that the

changes in the balance between, and the spatial

iPhone could be used as the mobile device I was

representation of the received signals. Further-

looking for. I proposed the above idea as a gen-

more, the repetition of a movement leads to the

eral format and 'platform' for the upcoming exhibi-

repetition of that sonic result. There are many

tion at the CBK Dordrecht. For the exhibition, ten

sonic combinations and changes we can explore

alumni were asked to each make a location based

without walking and by just moving the receiver

sound world for the city center of Dordrecht. The

in space. By doing so we navigate through a very

exhibition received the title Wormhole Dordrecht

detailed and diverse space around us; a space that

because of the possibility to switch (and 'travel')

we experience as truly present. Although we can't

between the 10 different parallel sound words.

fully understand it, it appears to be a coherent and consistent space. It forms an intangible space that

While the exhibition led to ten interesting and very

gets a tangible character because of the proprio-

different artworks, it brought me unexpected new

ceptive and visual perception resulting from mov-

insights; insights related to Radioscape and the

ing the receiver.

Another quality present in Radioscape is the in-

interesting. For example, the quality of the works

teraction between the radio waves and the physi-

was a result from the musical or narrative quali-

cal environment. Because of the relatively long

ties used within the work.

wavelength (170 meters) buildings become possible resonators for the transmitted signals. Hold-

We are currently experimenting with additional

ing the receiver close to the facade of a build-

ways of creating the sensation of sounds being

ing often leads to unexpected amplification of

present in the space around us and relating or

certain signals, resulting in a direct interaction

interacting with the physical environment around

between the physical environment and the elec-

us. Mobile devices have developed greatly and

tromagnetic waves. The normally unperceivable

often incorporate gyro sensors besides the ac-

electromagnetic waves interact directly with the

celerometers, compass sensor and improved GPS

perceivable space around us. Lastly, Radioscape

sensing. We have experimented with head track-

uses what I call a stereo panoramic receiver. It is

ing and positioning sounds in the space around

a receiver with a stereo antenna similar to the

us with real-time binaural convolution. The bin-

principle of a stereo microphone. The position

aural convolution allows us to simulate the spa-

of the transmitter in relation to the receiver an-

tial position of a sound in any direction including

tenna determines the position of the transmitted

height. While striving towards perfection in the

sound within the stereo image on the receiver's

simulation, it turned out that that the simulation

headphone.

can, and never will be, perfect simply because the earth's magnetic field is often distorted and

Until Wormhole Dordrecht was realized, I was

influenced by factors within the local environ-

only half aware that Radioscape had such a self-

ment. It became clear that creating a convinc-

explaining spatial quality, such a sensation of

ing spatial environment is not about the (perfect)

presence and why. It was because of Wormhole

simulation of reality but about creating interest-

Dordrecht that I realized that the direct changes

ing relationships between the sounds, the space

within the 'empty' space around our body and the

around our body and our physical environment.

interaction with our surrounding physical space

This is exactly what programmed relationships in

were fundamental for the near tangible sensation

software enable us to do. We could have already

of the electro magnetic space around us. The use

learned this from Radioscape. Radioscape has a

of GPS positioning in Wormhole Dordrecht leads

natural complexity that has its own logic; a logic

to changes that are taking place while walking.

that is so convincing that we easily mistake it for

Standing still and moving the iPhone around

being present or real although we could just as

the body has no effect on which sound files are

well argue that it's all virtual. Interrelating the

played or on the balance between the sound

virtual and the real makes the virtual only more

files. The changes that occur while walking can

real (present) without having to copy or simulate

be technically perfect but don't necessarily lead

reality.

to a perception of presence in space. At the time the GPS sensing had a precision of approximately 10 meters, but even if the calculated GPS coor-

References

dinates would have had a precision of about a meter, there still would not be any sense of the

van der Heide, E. (2012, November).

sounds being present in space. It is important to

Radioscape â&#x20AC;&#x201C; in the context of Augmented

note, however, that the absence of the sensation

Reality. AR[t],Â issue 2, 18-23.

of spatial presence did not make the works less

Radioscape as part of the 'Urban Explorers Festival'. Image courtesy Studio Edwin van der Heide.