AR[t] 2

02

nov 2012

auGmeNted realIty, art aNd teChNoloGy

marty the New affordable ar headSet yolande kolstee & Pieter jonker

auGmeNted PrototyPING auGmeNted realIty to SuPPort the deSIGN ProCeSS

jouke Verlinden

radIoSCaPe edwin van der heide

Image courtesy STUDIO Edwin van der Heide

2

AR[t] Magazine about Augmented Reality, art and technology

november 2012

3

Colophon ISSN Number 2213-2481

Contact The Augmented Reality Lab (AR Lab) Royal Academy of Art, The Hague (Koninklijke Academie van Beeldende Kunsten)

Prinsessegracht 4 2514 AN The Hague The Netherlands +31 (0)70 3154795 www.arlab.nl info@arlab.nl

Editorial team Yolande Kolstee, Hanna Schraffenberger, Esmé Vahrmeijer (graphic design) and Jouke Verlinden.

Contributors AR Lab & Partners: Wim van Eck, Edwin van der Heide, Pieter Jonker, Maarten Lamers, ­Ferenc Molnár (photography), Maaike Roozenburg and Dirk Vis. Guest Contributors: Esther de Graaff, Vincent Hui, Barbara Nordhjem and Martin Sjardijn.

CoVer ‘Marty’, the new Augmented Reality headset by the AR Lab, worn by Mariana Kniveton.

4

www.arlab.nl

Table of contents

18

48 welcome to AR[t]

Penetrating deeper into the temporal lobe

06 08

Barbara Nordhjem

Marty — The new affordable AR headset!

14

Yolande Kolstee and Pieter Jonker

Radioscape — in the context of augmented reality

[AR]chitectural Education at Play Vincent Hui

Digital technologies and fine art — a complex relationship

Edwin van der Heide

42

Martin Sjardijn

Who owns the space?

18

36

Yolande Kolstee

Chasing virtual spooks, losing real weight

46 48

Hanna Schraffenberger

Esther de Graaff

24

Augmented Reality: A Story Dirk Vis

52

Smart Replicas — bringing heritage back to life

28

Unspecialize! The more you know the less you see

54

Finding what I didn’t seek

Maaike Roozenburg

How it was made: a tangible replica with mobile AR

32

Jouke Verlinden, Maaike Roozenburg & Wolf Song

Augmented Belief in Reality Maarten H. Lamers

34

A portrait of Daniel Disselkoen Hanna Schraffenberger

Augmented Prototyping: Augmented Reality to support the design process

62

Jouke Verlinden

The revival of Virtual Reality?

72

Wim van Eck 5

6

WelcomE...

to the second issue of AR[t], the magazine about Augmented Reality, art and technology!

Much has happened in the six months since AR[t]

London. For the second time in a row we have

magazine’s first issue. The AR Lab has developed

presented our research at the ISMAR conference,

a new Augmented Reality headset in close co-op-

which took place from 5 till 8 November 2012 in

eration with the Bio-robotics Lab at Delft Univer-

Atlanta and the AR Lab's presentation of the Van

sity of Technology. Our article “The Augmented

Gogh project was rewarded as ‘Best Demo 2012’.

Painting: Playful Interaction with Multi-Spectral Images” has been accepted at ISMAR 2012, the

The AR[t] magazine has developed as well. Six

International Symposium on Mixed and Augment-

months ago we started off as an aspiring maga-

ed Reality. The project described in this article

zine series for the emerging AR community in and

shows in an interactive way discoveries made by

outside the Netherlands. We received enthusias-

material specialists and art historians in Vincent

tic feedback on the first issue of the magazine.

van Gogh’s paintings. The AR Lab has grown: in

More than that, we were approached with vari-

September 2012 Dirk Vis joined our team for one

ous contributions from interested parties. The

day per week to promote AR and other new visu-

editors have selected several of these contri-

alization techniques like 3D printing, in various

butions to be included in this second issue. As

disciplines at the Royal Academy of Art by orga-

a consequence this issue is even more diverse

nizing a Pop-Up Art Gallery. We have furthermore

than its predecessor. Besides contributions from

welcomed Mariana Kniveton as an intern at the

researchers, artists and lecturers of the AR Lab

Lab. She researches the use of innovative visual-

(based at the Royal Academy of Arts, The Hague),

ization techniques in museums and has supported

Delft University of Technology (TU Delft) and

the editors of this magazine in various tasks. We

Leiden University, this second issue also features

have also invited DPI Animation House, a com-

texts by international researchers as well as local

pany based in Schevingen, to join the AR Lab. We

young writers. Contributions include technical

are experimenting with artistic projection map-

articles, entertaining essays, in depth discussions

ping, while Jouke Verlinden works in this field for

of AR art works as well as short columns.

industrial goals like rapid prototyping.

While the magazine undoubtedly has grown, its The hard work of the last few months has paid

core remains the same. In AR[t], we share our

off and the AR Lab won the 3rd prize at the

interest in Augmented Reality (AR), discuss its

RAAK-AWARD 2012 for the high quality of the re-

applications in the arts and provide insight into

search work done in the framework of a Raak-

the underlying technology. If you haven’t done

Pro Research Programme. Furthermore, the AR

so yet, we invite you to check out the first issue

Lab’s Van Gogh iPad project has been shortlisted

and our the website www.arlab.nl to learn more

for the ‘Most Innovative use of AR 2012’ Award

about Augmented Reality in the arts and the work

as part of the AR Summit held in June 2012 in

of the AR Lab.

7

PeNetratING deePer INto the temPoral lobe BarBara nOrDHJEM

CrEw Crew is an interdisciplinary collective of artists and scientists mainly based in belgium. the composition of the group changes depending on the project, but the performances are always at the intersection between art and science. Crew has been combining theatre and technology since 1998 with eric joris as artistic leader. their first immersive performance, “Crash�, premiered in 2004. Since then, the group has been experimenting with hybrid performances and installations where story-telling, live elements, and human-machine interfaces come together. there is no traditional separation between the actors on a stage and the audience. Instead the visitor is guided through different settings where active participation is required. a lot of Crew’s activities evolve around issues that are also currently being investigated by neuroscientists and philosophers. themes such as the human mind, senses and our experience of reality are explored with "multimedia as a prosthesis". apart from the performance group, there is also the Crew_lab which focuses on research related to immersive media and development of new technology.

8

I am always looking for art that messes with your mind and tricks your senses. The performance Terra nova by CrEw starts out as a perverted experiment with the audience as guinea pigs. In other words, I feel like I have come to the right place.

CrEw is an interdisciplinary group of artists and scientists based in Belgium. Their performances incorporate new technological environments with live performance. Terra nova can be experienced by 55 participants at a time. Before entering the performance space, we are all given headphones and divided into five smaller groups. During the first part of the performance, I am placed in a steel chair and can observe how other people are guided through an alternate reality. a line of people enters, wearing brown jackets and heavy backpacks. Their footsteps are slow and insecure, the outside world is sealed off by headphones and video goggles. Each person is accompanied by an actor. The movements of the actors appear to be orchestrated by a central person who gives directions by using gestures, as if they carry out a medical procedure or experimental protocol with care and precision. It looks like a scene taking place in a dystopian near-future society. “The fear of death is in the temporal lobe, we are going deeper�, a monotone voice announces.

The participants are then tied to wooden boards which are flipped backwards into horizontal position. On the sideline, we can also watch a video projection, images of narrow hallways, is there light at the end?

Crew, terra NoVa at deaf2012. Image by jaN SPrIj 2012

9

At some point, I am instructed to proceed into

turn my head. I can explore the environment as I

another room by a distant voice. I line up behind

look and walk around (or rather, I wildly bounce

other group members and we enter a room with

my head in all directions to see if the equipment

a bare-chested man. He tells a story about delir-

can keep up). This direct relationship between

ium and fatigue during a rough expedition to the

head movement and vision really gives the feel-

south-pole. The central theme is our perception

ing of being placed in the video.

of reality and how certain situations can create a distorted view of our surroundings. This way, the polar quest becomes a metaphor for explor-

Levels of reality

ing the brain and our senses. After a dramatic mind trip involving penguins, snowstorms, and

CREW collaborates with institutes for media tech-

the frontal lobe, I finally find myself in the role

nology to develop customized tools for immersive

of ‘immersant’: a traveller in a virtual surround-

environments. The group works with ‘omni-direc-

ing. I am eager to try out the video goggles and

tional video’ (OVD), which allows you to see sur-

see if I indeed will feel like I am somewhere in a

round video on a head mounted display (HMD).

different time and place. Perhaps a few people

The places you are exploring as an immersant are

from CREW noticed one person continuously turn-

pre-recorded while actors around you appear via

ing her head around after being ‘plugged in’. The

real-time video filmed with a small head mounted

projection on the video goggles gives a partial

camera. By using an orientation tracker placed

view of the scene according to the direction I

on the immersant’s head, it is possible to move

10

Crew, terra NoVa at deaf2012. Image by jaN SPrIj 2012

around in a multi-layered video environment.

mind travels in a completely computer-generated

what appears to be one coherent view of the

world detached from the physical body. CrEw

environment is created by combining 360 degree

takes a turn towards embodied technology and

pre-recorded video, real-time video from the cur-

physical presence by combining video, sound,

rent environment, and metadata from the orien-

motion tracking, and tactile stimulation given by

tation tracker. Different recordings are merged

the actors.

into one unified scene where the immersant can move around and explore.

Unlike augmented reality (ar) and virtual real-

Tapping into the senses

ity (vr) there are no computer-generated virtual elements, but different layers of video recorded

CrEw takes a lot of experimental findings from

reality. vr takes place completely in a virtual

cognitive neuroscience and puts them into prac-

3D world, whereas ar allows you to see the real

tice in the performance. In the beginning of an

world with added virtual elements. what CrEw

immersion, you find yourself in a state of sensory

presents is a form of mixed reality where video

deprivation. visual references to the surrounding

recorded elements from a different time and

space are concealed during an initial period of

place are combined with elements recorded

complete darkness. To add to the loss of orienta-

in real-time. In mixed reality there has been a

tion, you are then strapped onto a standing bed

strong focus on the visual aspect of the experi-

which is slowly flipped backwards. This creates a

ence, especially in earlier vr scenarios where the

very effective feeling of disorientation by mov-

11

ing the body around in space without any visual

your own hand being touched the same way. The

references. By shielding off visual input and relo-

combination of visual and tactile stimulation can

cating the body in space, the reference to reality

create a powerful illusion: you start experiencing

becomes weaker.

the virtual hand as your own. In other ways, your

It is already known that the brain is more likely to

sense of bodily self is moved into the virtual world,

fill in information when there is no sensory input.

you feel like it is really you on the screen. This

Early studies in the 1960s and 1970s showed that

effect works with a principle called the rubber

people start to hallucinate when they are put into

hand illusion (rHI), which has been used in several

tanks where they are floating in complete darkness. More recently, there has also been an experiment showing

‘your sense of bodily self is moved into the virtual world’

some hallucinations after just 15 minutes of senso-

scientific experiments. The participant’s real hand

ry deprivation. It seems like you begin to perceive

is hidden and a rubber hand (or someone else’s

your own reality in the absence of external input.

hand) is visible instead. Synchronous stroking and

In the case of CrEw, the initial period of complete

tapping of both the participant’s hidden hand and

disorientation seems to create a heightened sense

the rubber hand creates the illusion that the rub-

of presence by preparing you for accepting the

ber hand actually belongs to the participant. If

new mediated reality. If you don’t know where

stimulation is not synchronous, the illusion does

you are, you might as well be located in the video

not take place. You can try the illusion at home

projection appearing in front of you.

by putting a glove on the table and keep your own hand hidden under the table. Let someone stroke

virtual body

both the glove and the real hand in the same way at the same time. The rHI is an example of visual capture, where the visual sense dominates over

another tactic used to enhance presence — the

other senses. You feel tactile stimulation on your

feeling of being there — is to create a link between

body, but see it happening somewhere else. The

what is shown on the HMD and what you feel your

brain has to integrate conflicting information with

body. During the immersion, you will see a virtual

the result that the fake hand feels like a part of

hand being touched via the video goggles and feel

your own body.

Image by reNé PaSSet

12

negotiating reality?

further reading

CrEw tells a story about perception, the brain,

■

Botvinick, M., & Cohen, J. (1998).

and what we experience as real. It is a story

rubber hands ‘feel’ touch that eyes

about the mind and a poetic interpretation of

see. Nature, 391, 756.

scientific studies. a lot of fundamental questions are posed about who we are and what we believe

■

Lenggenhager, B., Tadi, T., Metzinger,

is real. Can we trust our senses at all if our ex-

T., & Blanke, O. (2007). video ergo sum:

perience of where we are and what we percieve

manipulating bodily self-consciousness.

can be manipulated so easily? CrEw states that

Science, 317, 1096-1099.

neuro-philosopher Thomas Metzinger inspired the performance Terra nova. The idea of our bodily

■

Mason, O. J. & Brady, F. The psychoto-

self is something we take for granted and which

mimetic effects of short-term sensory

we see as a fundamental part of being a person.

deprivation. the Journal of Nervous

Metzinger argues that how we experience our-

and Mental Disease, 197, 783–785

selves is an ongoing process rather than some-

(2009).

thing fixed and stable. The experience of being a person with a body and a mind is the result of

■

Metzinger, T. (2007). Empirical perspec-

neural processes and sensory information. when

tives from the self-model theory of

your brain begins to receive evidence that your

subjectivity:

body is located somewhere in a virtual world,

a brief summary with examples. Prog-

that becomes the experienced reality.

ress in Brain Research, 168, 215-278.

It may be a disturbing idea that our perception of

■

wynants, n., vanhoutte, K., & Bekaert,

reality and of having a self is not stable but can

P. (2008). Being Inside the Image.

be manipulated. at the same time, it also means

Heightening the Sense of Presence in a

that we are able to see and feel much more than

video Captured Environment through

what we may think. after the performance, I walk

artistic Means: The Case of CrEw.

outside with my unstable self and my deceivable

Proceedings of the 11th annual Interna-

senses. I look down at my hands a second time to

tional Workshop on Presence, 157-162.

check that they are really mine.

BarBara nOrDHJEM I graduated in 2011 with a master’s degree in Cognitive Neuroscience from the university of leiden. Now, I am in the Visual Neuroscience Group at the university medical Center in Groningen. I am generally interested in visual perception and how we are able to extract the most useful information from the environment in different situations.

I have a rather mixed background. before I started my Phd research about different ways of seeing our surroundings and the neural mechanisms involved, I initiated a festival for live visuals and electronic music in denmark and have worked at the media art institute V2_ in rotterdam.

13

the New affordable ar headSet! YOLanDE KOLSTEE anD PIETEr JOnKEr The ar Lab has developed a new augmented reality headset in close co-operation with our partner the Delft Biorobotics Lab of the Delft University of Technology. It is a successor to the well-known headset George, which became famous because of the work done on co-operation through ar by our partner, the Systems Engineering Department of the Faculty of Technology, Policy & Management at the Delft University of Technology for the CSI-The Hague project (www.csithehague.com).

14

MarTY

15

Our new headset deserves an appropriate name:

sitioned. In contrast to markers, natural feature

‘Marty’ refers to the scene in “Back to the Fu-

tracking is non-intrusive as it uses key points from

ture (2)” in which VR headsets were used. In con-

the world around you.

trast with George, the new headset is a videosee through system. Niels Mulder designed both

Open Source and low-cost

George and the new Marty.

This project is open Source. The AR Lab will make

Optics

the 3d model and print associated with this this low-cost AR headset available to the public. This

The Sony HMZ-T1 VR optics was used as a starting

allows anyone to make an AR headset at 720p for

point for building Marty. The basic components of

only € 1.500, paving the way for high quality Aug-

the Sony head mounted display have been re-used.

mented Reality for the general public.

Two Logitech C905 cameras were added to the design and the glasses have been made more robust.

With this new headset and software, developed

In line with the design of the George headset, sim-

by the Delft University of Technology and the AR

plicity and robustness have been integrated with

Lab, it will be possible to walk through any space

state of the art design. Niels drew inspiration from

and experience Augmented Reality as never be-

the famous View-master; a simple, low cost setup

fore. The design files of Marty will be made avail-

which has been in use for decades and allows you

able a.s.a.p. on Sourceforge. The tracking, map-

to see real 3D images in a simple stereo setup.

ping and user interface software will be made

Tracking

available (on Sourceforge) after Oytun Akman defends his PhD thesis on December 3, 2012.

Oytun Akman and Robert Prevel, PhD research-

The AR Lab will open a blog for help and com-

ers at the Bio-Robotics Lab at Delft University of

ments on the design of Marty and the tracking,

Technology, developed the software for this

mapping and user interface software.

headset. The software allows: ■ Head-pose tracking and on-line building of a coarse 3D point-cloud map at 30fps; ■ Reconstruction of the 3D scene and on-line building of a fine grain 3D point-cloud map

The PhD thesis of Oytun Akman (see Figure 1) can be downloaded from: http://repository.tudelft. nl/v iew/ir/uuid:3adeccef-19db-4 a0 6-ab268636ac03f5c0/

at 2fps; ■ Hand pose tracking for human-machine interfacing to control the headset.

The software works by tracking 3D natural features on multiple scales, using only a stereo video-feed from a pair of webcams.

It is still very difficult to position virtual information in three-dimensional space for AR headsets, so we are very glad that mayor breakthroughs in software development by the TU-Delft have made 3D natural feature tracking possible. With natural feature tracking, no markers are needed; instead, salient points in the 3D space — on walls, books, and paintings for example — are tracked and used to provide invisible anchor points by which the virtual objects or scenes can be po-

16

Figure 1: Oytun Akman’s PhD thesis

Advantages of AR head-sets over smartphone or tablet PC based AR

say ‘Restaurants around’, and then wait for the

Using a smartphone or tablet PC to see AR informa-

cases, you also have to lay the smartphone flat

tion has a lot of advantages because smart phones

to prevent compass errors. In addition, distortion

and tablets are so widely used. Many users expe-

of the virtual layer in relation to the real world

rience an extra layer of information by scanning

makes smartphones insufficient for certain appli-

AR markers or by using GPS and compass informa-

cations.

tion on their location. This can link them to cul-

Head-set AR displays, on the other hand, permit

tural, historical or commercial information.

the user to see the real world in 3D, which en-

However, there are some disadvantages too.

ables the user to move around safely and to see

The screen size is very limited, even when using

and experience virtual reality at the same time.

an iPad or Android tablet. Moreover, smartphone-

The experience of a virtual layer when using AR

based AR requires that the user hold his phone

eye-wear is immersive, as we know from our ear-

at eye-level to position the phone between his

lier experiments with AR headsets in an exhibi-

eyes and the real world he is augmenting. This

tion in Milan, where we showed virtual furniture.

puts the user in a tiring pose that cannot be sus-

By designing the headset-frame properly, we can

tained and interferes with other activities such

keep occlusions to a minimal and maximize the

as walking around in a street finding historical or

user’s field of view. This allows the AR glasses

architectural information to insert into the AR

to be used hands free and lets the user do tasks

environment.

requiring both hands, as many people have seen

Another annoyance is the process of setting up

in the BMW commercial where a mechanical en-

an AR application via, for example, Layar. You

gineer is repairing a motor.

phone to get the correct information. In most

have to start up Layar, activate the application,

The CYBER-I BI-Ocular: the new professional AR headset The ARLab ordered the manufacturing and obtained the very first optical-see through headset from our partner CybermindNL.com, the CYBER-I BI-Ocular. This is an XSGA OpticalSee-Through AR headset with two USB 2 uEye LE cameras. It is about a factor 10 more expensive than Marty, but unlike Marty apt for the professional market. We are negotiating with business partners to bring a professional version of the software of Oytun Akman on the market for this headset.

Contact info@arlab.nl if you have a business proposal.

The CUBER-I BI-Ocular headset

17

Radioscape in the context of augmented reality Edwin van der Heide When I initially got the idea for Radioscape, I was thinking about creating a parallel reality that traverses the existing physical space and reveals itself by navigating through that physical space. It was later, when I encountered the term augmented reality, that I started reflecting about artworks in which the augmentation of reality plays a central role. During this process I beRadioscape can be seen as an artwork in which reality gets augmented. It’s important to mention that I consider all modalities (not only the visual) when thinking about augmentation and I believe that it’s a requirement for an augmented experience that the perceived result is more than the combination of the real and the virtual. The real and the virtual should not just co-exist but relate to, and influence each other, leading to an experience that is more complex than the addition of the two. I like to see it as a perceived ‘multiplication’ of the world around us with a virtual constructed reality.

Image courtesy STUDIO Edwin van der Heide

came more and more convinced of the idea that

In this text I’ll first give a more technical explanation of Radioscape. I believe it’s important to understand the parallels between the acoustic

traverses and reflects in the three dimensional

space and the electromagnetic space that form

space around us, it is also spatial because our

the foundations of the work. After that I’ll go into

listening results in the perception of sound as a

more detail on the nature of the augmentation

spatial phenomenon. Furthermore, any spatial

that occurs in the work.

experience, such as the perception of sound, has

It was a call for artworks for an area in the

a bodily component to it because we relate the

country side of Japan that brought me to the fol-

perception of (things in) space to the perception

lowing question: We perceive sounds as individual

of our body in space. We can act in, and move

sources that merge and interact with the acoustic

ourselves through the acoustic space.

space around us; would it be possible to achieve

Radio is often used to transmit individual sig-

a similar experience with electromagnetic (ra-

nals from one point to one or more other points.

dio) waves? Sound is not only spatial because it

The spatial nature of the electromagnetic waves

18

and the multiplicity of sources are ignored in or-

shift it back down. This is a form of transmitting

der to realize individual communication channels

without using any form of modulation and just

(examples of exceptions are certain radio ama-

using a certain frequency range within the elec-

teur contests, fox hunting and gps). To establish

tromagnetic spectrum. what I especially wanted

these channels, different forms of modulation

to know is what would happen if we would trans-

(like aM, FM, GSM, etc.) are being used. Individu-

mit two independent sounds by shifting them up

al transmitters broadcast on their own frequency

individually, transmitting them at two different

and with a receiver we tune in to a specific fre-

locations, receive them at another location, shift

quency while filtering away the others.

them back down and listen to the received signal.

I wondered what would happen if we would

I was expecting that the signals would ‘mix’ in the

simply shift sound up in frequency, transmit it at

electromagnetic space just like acoustic sounds

one location, receive it at another location and

‘mix’ and co-exist in the acoustic space. It is an

19

approach to radio where we think of it as a field

we increase the distance to the source because

of transmitted sources with the receiver moving

the radiated energy gets spread over a bigger sur-

through this field, instead of a receiver that is

face in space and besides that, the air itself ab-

tuning in to a single transmitter and establishing

sorbs a bit of the energy as well. Electromagnetic

a (one directional) non-spatial, point-to-point

waves have that same behavior but when using

channel.

regular radio transmission techniques, this effect

The above setup of transmitter(s) and re-

is avoided or compensated for. The reason for this

ceiver can simply be compared to a setup of

is that both FM and AM use a carrier signal that is

speaker(s) and a microphone, but addressing the

modulated by the audio signal.2

electromagnetic space instead of the acoustic

The proposed system of shifting the audio sig-

space. The transmitters can be seen as amplifiers

nal up (and down) in frequency is a form of trans-

with an antenna instead of a speaker connected

mission without using a carrier frequency. Any sig-

to the output. And the receivers use an antenna

nal the receiver receives within the chosen fre-

instead of a microphone. The used frequency

quency range becomes audible. It is the individu-

range is determined by the amount of frequency

al strength of the received signal(s) that directly

shift used by the transmitter and receiver.1 The

corresponds to the loudness of the signal(s). The

loudness of acoustic sound sources decays when

closer the receiver gets to the transmitter, the

20

louder the signal becomes and vice versa.

approximately 2.5 seconds for a radio signal to

while sound travels with a speed of about 340

travel to the moon and come back. The moon has

meters per second, radio waves travel with the

an average distance of about 384,400 kilometers.

speed of light: 300,000 kilometers per second.

The wavelength of a wave depends on its fre-

an acoustic sound reflecting in space is some-

quency and traveling speed. Frequencies in the

thing that we perceive as both a timbral and a

FM band range have a wavelength of about three

temporal phenomenon. we can hear an acoustic

meters. when these waves reflect between build-

sound reflecting in space, resulting in reverb or

ings in a street this leads to standing wave pat-

echoes. It is temporal not because it just happens

terns with cancellations points that occur - for

in time but temporal because we perceive it as

example - every 1.5 meters. we can avoid these

something that happens in time. we speak for ex-

standing wave patterns by using a relatively long

ample about a reverb with a duration of 1.5 sec-

wavelength of 175 meters (=1.7 mHz). at this

onds or an echo that arrives half a second later.

wavelength buildings are not ‘just’ reflectors,

radio waves can also reflect in space but since

they start to become conductors and resonators

they are that much faster than sound we would

for the transmitted signals. This means that the

need enormously big spaces in order to be able to

physical environment is excited by, and responds

perceive these temporally. For example, it takes

to, the transmitted radio waves. while developing the work I learned about different antenna principles. a vertical antenna has an omni-directional sensitivity pattern and relates to the electric component of the electromagnetic field. a coil or loop antenna is only sensitive from the sides and relates to the magnetic component of the electromagnetic field.

I realized that these two directivity patterns were equal to the patterns of the omni directional microphone and the figure eight microphone.

There is a stereo recording technique called mid-side (m-s) that uses exactly these two microphones and I started to wonder whether it would be possible to realize a stereo receiver with such an antenna setup. It would not be a receiver that receives a signal that is broadcasted in stereo but a receiver that creates a stereo image with the positions of the individual (mono) transmitters.

Transmitters that are to the left of the antenna will be heard more on the left and transmitters to the right side of the antenna will be heard more on the right. rotating and moving the receiver changes the stereo image directly.

Each transmitter is transmitting its own layer of the meta-composition. The layers are slowly changing and eventually repeating after 4-10 minutes. The changes within a layer are the slow-

Image courtesy StudIo edwIN VaN der heIde

est changes that you can experience in this environment. It’s a result of not walking and not mov-

21

ing the receiver and just listening to the changes

transmitter that is otherwise inaudible because of

within the received layers themselves. The next

its large distance or that one of the transmitted

level of change is the interaction that occurs

signals becomes dominant over the others. The

when you don’t walk but just move the receiver.

resonating buildings are an interesting example

By doing so, you re-orient yourself in the field of

of a situation where the real world interacts with

received signals and find new perspectives to the

the added layer. The radio signals are not just a

environment. The last level of change is the re-

parallel reality that leads to associated relations

sult of simply walking and thereby getting closer

between the physical space and the electromag-

to certain transmitters while moving away from

netic space. They directly influence each other.

others. Certain transmitted signals will decrease

Besides the intentionally transmitted signals,

or disappear while other signals will fade-in or

the receiver picks up other signals that are pres-

become louder. while listening, you alternate

ent in the same frequency range. neon lights and

your focus and the way of interacting.

computers that control street lights can emit

The radioscape receiver is a handheld receiv-

strong fields around them that don’t carry far but

er. By moving the receiver you explore the space

can take over in loudness on a local scale. already

around you. Every movement of the receiver has

present signals that are otherwise unperceivable

a direct effect on the received signals. The scale

become perceivable and part of the received en-

and the speed of the changes match well with the

vironment. These moments add to experience

space that we describe with the movements of

and make it even more real.

our hands and arms. Therefore the space around

radioscape is a work in which you can share

you becomes an almost tangible space in which

your experiences. By navigating the city you gen-

you explore and remember positions and transi-

erate your own sonic order, combinations and

By navigating the city you generate your own sonic order, combinations and timing of the composition.

timing of the composition. when

walking

together

with other people you share the same augmented space. The global experience will be more or less identical while local signals really differ from each other. I’ve seen par-

tions. It’s intuitive to navigate in the sense that

ticipants, talk about what they experience, imi-

it reveals itself easily but is complex enough to

tate each other and notify each other of particu-

keep on exploring.

larities they discover. They are in an augmented

radioscape takes place in the public space.

world that is based on the intersection of the

My preferred locations are areas within a city

physical world with a composed parallel world of

with a lot of diversity, that are easy to walk and

transmitted signals. Furthermore, there is a true

have streets that are close to each other so that

interaction between these two worlds.

the participants frequently have to choose where to go. It is the chosen city area that dictates the

Footnotes

possibilities of where you can go. as mentioned earlier the buildings in the

■ 1 Examples of frequency bands are the Lw, Mw,

environment become conductors and resonators

Sw and FM band. radioscape is however not

for the radio waves. There is often a clear effect

using the corresponding modulation tech-

when you move the receiver close to a building.

nique and therefore incompatible with con-

It happens for example that you start hearing a

sumer receivers.

22

â– 2 The carrier signal is generated by an oscil-

transmitter signal (carrier) is multiplied with

lator that corresponds to the transmitter

the audio signal. The received signal decays

frequency. when using frequency modula-

with increasing distance between the trans-

tion (FM) the transmitted audio signal modu-

mitter and receiver, but this is compensated

lates the transmitter frequency (carrier) and

for by an automatic gain control system build

therefore the content is independent from

into the receiver and resulting in a constant

the received amplitude. when using ampli-

amplitude of the carrier. It does not influ-

tude modulation (aM) the amplitude of the

ence the dynamics within the audio signal.

Image courtesy StudIo edwIN VaN der heIde

23

“I’ve never been here”

Finding what I didn’t seek Esther de GraafF “I’ve never been here”, says one of the participants who has lived in Rotterdam for eleven years. At the beginning of the “Serendipitor Walk” ten minutes earlier, he told us he knows every part of Rotterdam like the back of his hand. Within a few minutes, my initial scepticism about this navigation-app has disappeared. Mark Shepard, a charming and pleasant American artist, architect, and researcher, smiles. In 2010, he developed the “Serendipitor” app together with the V2 _ Institute for the Unstable Media in Amsterdam.

It is part of his longstanding project “Sentient City Survival Kit”; a collection of objects for surviving futuristic urban life. ‘Surviving’ may sound rather severe, but the term indicates that Shepard hasn’t just made a nice toy.

His work is an investigation into the relationship between the ubiquitous computer and the city, in an attempt to provoke thought about our controlled society. By following various instructions, you are forced to relate to the city in a novel way. For example: follow a man with an umbrella, or buy a rose and give it to the first passerby. Each walk is determined by four types of instructions: social (meeting people), geographic (go left), abstract (track a square) or architectural (enter the tallest building). You decide how long and complicated your walk will be and how many assignments you wish to do, depending on how much time you have to play, explains Shepard. It is not surprising that his instructions draw on artists like Vito Acconci and Yoko Ono of the Fluxus movement, an international avant-garde art movement from the sixties which focused on art based on mundane things.

24

During the DEaF festival in rotterdam (May 17 -

him/her to draw a map of his/her childhood. at

20, 2012), Shepard was present every day to walk

Café Brasserie Dudok, we meet a woman from

the “Serendipitor walk” with a group; an amazing

Turkey who is waiting for a friend. while enjoying

opportunity. On a beautiful Friday afternoon, we

a drink she tells and draws us her life story in just

accompany the artist from the DEaF main site at

ten minutes. She has established an impressive

the Coolsingel into the city. Our destination has

career as a doctor in many European cities. De-

been entered, but the exact route depends on

spite how short our encounter was, I am moved

the instructions we are given. For our first as-

by the beautiful story. next, we placed her draw-

signment, we must walk down the Coolsingel to

ing over the map of rotterdam, so that the lines

the Meent, find a quiet spot, and stay there for a

literally determine our route. This dictates that

while. OK. So we arrive at a very busy intersec-

we head west.

tion, not the easiest place for quietness. we spot a wide staircase at the side of a building, which

Current navigational products, such as TomTom,

leads us to a higher, quieter place. Suddenly we

make it very easy to get from a to B. Previously,

are faced with a door, behind which we find an

computer technology could only be found in-

oasis of calm and beauty. we’ve come across a

doors. nowadays, the Internet is everywhere in

gallery unknown to me: the wTC art Gallery.

the city. we are surrounded by these invisible

after walking around here for a while, we press

digital information systems, which constitute a

‘next’ for our next assignment. we must go east

kind of virtual layer over our reality. This cre-

and find a bar, offer someone a drink and ask

ates a capacity to gather and process information

25

that is increasingly interwoven with the physi-

places in rotterdam that I would otherwise have

cal structure of our urban space. For instance,

walked right by. we found places unexpectedly

various applications for smartphones and tablets

when we were searching for something else. and

provide digital information about your immediate

I’ll be damned. He inspired me and got me think-

surroundings. Simply hold your phone in front of

ing. I often have no idea what route I traveled,

a street to see where you can find the nearest

simply because I look at the screen instead of

aTM, or point it at an old building to read its his-

my surroundings. The app stimulates you to re-

tory. This is incredibly convenient, but the rise

ally look around and to make contact with peo-

of computers also makes it easier for futuristic

ple. The app is available for download and use

cities to keep an eye on us. You could say that the

in every city in the world. want to skip a task?

city is getting smarter. Shepard wants to make us

anything is possible, simply press ‘next’. Each as-

think about the consequences this could have on

signment gives the opportunity to find something

our culture and politics, in regards to privacy for

you were never really looking for. Something we

example. His app presents an alternative way to

should do much more often.

navigate through a city where everything is regulated, public and controlled. Shepard’s point is to

the app is free to download from appStore:

become aware of your surroundings and the route

itunes.apple.com/us/app/serendipitor/

you travel. However the instructions themselves

id382597390?mt=8

are fairly concrete and thus exert some form of

Or from:

control. Shepard responds with a smile: “Free-

www. serendipitor.net

dom is not possible without control”.

Small detail: unfortunately not suitable for android users.

after having walked for an hour and a half, we collectively decide to end the ‘artist walk’. The

more information:

assignments kept guiding us away from our origi-

www.andinc.org

nal route. we did not reach our destination, but that is not important. During this walk, I found

26

© June 2012, Esther de Graaff

ESTHEr DE GraaFF www.estherdegraaff.nl

Project manager, curator, writer, coordinator — call me any of these things; I do them all. My love for organizing events is central. It’s all about the end result. I support artists, curators and institutions in organising various projects, both in content and in organisation.

Social themes are essential to my work. I am not interested in art for the sake of art; it’s all about sharing experiences.

Culture is an encounter. The projects, texts and exhibitions created by Esther de Graaff are characterized by creative solutions and efficiency. Inspiring people with beautiful images and deepening their view of the world is the whole point.

27

Smart Replicas bringing heritage back to life Maaike Roozenburg Treasuries of objects

at this time? And what is their significance in a

Our museological heritage comprises collections

rapidly digitizing society?

of objects — from paintings to pottery — which

Museums and cultural heritage sites are the

have been categorized, grown and preserved

“institutions” who manage this heritage; store-

over the course of centuries. Objects we value

houses of objects, stories, images and historical

for their historical, cultural and social context;

knowledge. The conservative way these objects

objects that play a key role in the way we view

are made accessible — you must to go to a muse-

our shared history and cultural identity. Our his-

um or exhibition — means you can admire, but not

tory is examined, reconstructed and visualized

touch, let alone use them. This strips the objects

through these objects, but how can we relate to

of their main purpose and function — their origi-

all these stored items? What is their relevance

nal use — and isolates them from our daily lives.

28

Image by SaM rEnTMEESTEr

‘enriching’ objects The knowledge and information which museums

tural heritage into new design? How can you link

and cultural heritage sites hold, gather and de-

these objects to the historical information which

velop about the items they own and manage is

is stored in a museum? How could visitors interact

barely accessible to the public. It is fixed within

with these “enriched” objects? These questions

the confines of the museum and is often reduced

will be examined in the project Smart replicas.

to a short text on a museum plaque or an item in a catalogue.

In the project Smart replicas, we investigate how

Smart replicas

3D prototyping — such as 3D scanning, printing and

Smart replica is a term we have given to repli-

reproductions which make digital objects “real”

cas of historical objects made useable again by

and vice-versa — and augmented reality (ar) can

means of 3D scanning and printing techniques.

contribute to the accessibility of museum objects

Smart refers to ‘intelligent’; these objects are

while providing the means with which we can also

not just copies, but replicas enriched by innova-

increase our knowledge of the objects. How can

tive technologies such as ar to bring informa-

you “reverse engineer” a museum piece? will

tion across, so that they can serve their original

this allow you to take the museum piece out of

intent and provide information outside the mu-

the museum and put it to use? Can you turn cul-

seum at the same time. That way, heritage can

29

provide new designs; objects shaped and refined through the centuries can be used in the present. Smart Replicas is a project by Studio Maaike Roozenburg in cooperation with four partners: Delft University of Technology, Faculty of Industrial Design and Faculty of Civil Engineering; Delft Heritage; Museum Boijmans Van Beuningen Department of Archaeology, the AR Lab and students from the Graphic Design department of the Royal Academy of Art. A team of very different parties with their own background and expertise contribute to this project: academics, students, curators and technical experts.

Smart Replicas is a project in which the worlds of museological heritage, design, art history, 3D prototyping and AR come together.

'Reverse engineering' historic items Seven tea cups from the Boijmans Van Beuningen collection form the “base material” to explore how reverse engineering and 3D prototyping can be used to produce useful replicas of museum pieces. Previous tests have shown that a (medical) CT scanner can scan very vulnerable and “untouchable” historical objects. This does not damage the objects and meets the museum’s guidelines for handling these objects. The resulting data can be converted to a digital reconstruction of the object which can be printed out on a 3D printer. The aim is to combine 3D prototyping techniques with the historical porcelain techniques used to create the original cups. This includes experimenting with milling porcelain casting moulds straight from 3D models as well as printing onto porcelain directly. It is important that the replica is a complete designer product that invites everyday use and can lead a new life outside the museum’s walls.

'Smart' objects AR brings the physical and digital worlds of an object together, thus transforming it into an information medium. Or rather, AR can show the information, history, context and stories associated with the object. Each porcelain cup is Images by Maaike Roozenburg 30

a historical source, with a wealth of historical

Maaike Roozenburg Maaike Roozenburg is a designer. She develops concepts, projects and products on the border of heritage, design and visual communication.

After graduating at the Gerrit Rietveld Academy in Amsterdam she founded Studio Maaike Roozenburg in witch she combines a historical fascination with high tech materials and techniques and ‘traditional’ crafts. ‘Research by design’ plays an important role in these projects.

The studio aims to research and communicate the value of historical (museum) heritage for us now. This approach can result in a design collection of ceramics pieces, a work for the facade of a building, an exhibition for a museum, or a mobile application that unlocks a historical collection. Besides her work at the studio Roozenburg teaches at the Post Graduate Course of Industrial Design at the Royal Academy of Art.

Image by Julia Blaukopf

production processes, use and rituals. AR allows

What is the real significance of our heritage?

us to visualise this information and actually link

With the project “Smart replicas”, we examine

it to the object; connecting knowledge to the

how 3D prototyping can be used to put heritage

object itself, outside the context of a museum.

back to use and how AR can turn objects into

This allows the knowledge, carried by the rep-

information carriers outside the museum. The

lica, to leave the museum and enter the wide

underlying questions we want to ask are: What

world.

is the meaning of all these stored historical ob-

information; stories about travelling, locations,

A group of students from the Graphic Design de-

jects? How can we relate to them? Are they rel-

partment at the Royal Academy of Art is working

evant to us now? What happens when you copy

on structuring and shaping digitized historical

them? What does this change in our perception

information concerning the cups. They can de-

and appreciation of the real object? What is the

velop their own storyline or narrative and design

relevance of these objects in our increasingly dig-

it for a corresponding medium. Routes can be

ital society? Augmented Reality and 3D prototyp-

accessed with Google maps; moving images can

ing offer opportunities to pose these questions

be displayed through animation and text can be

(indirectly), and to identify and investigate them.

used by aid of visual or audio stimuli. These designs will be linked to the replicas using AR.

The result is a trial installation in Museum Boi-

The research also includes the markers that cre-

jmans Van Beuningen which will run in spring

ate that link. Here we examine how to combine

2013,where the smart replica prototypes can be

the historical techniques and decorations of the

‘touched’, used and tested by the public.

original museum pieces with technology aided

The project can be followed at http://smartrep-

by markers and object recognition.

licas.blogspot.nl/

31

How it was made: a tangible replica with mobile AR Jouke Verlinden, Maaike Roozenburg & Wolf Song

Replicated cup and lid in porcelain, augmented by animated 3D graphics on a smartphone.

As part of the “Smart Replica” theme initiated by

3D scanning

Maaike Roozenburg, we made a proof of concept in

The original Loosdrecht’s porcelain objects are

the minor on Advanced Prototyping at the Faculty

part of the collection of museum Boijmans Van

of Industrial Design, Delft University of Technology.

Beuningen. They measure approximately 8 cm

The top figure presents an impression of the final

and due to their fragility non-contact scanning

result: an exact replica in porcelain of an 18th

had to be selected. A medical Computer Tomog-

century sugar cup + lid, the decorations function

raphy scanner was employed, in which a radia-

as markers for a handheld AR overlay. On the cup,

tion source and the detectors rotate around the

the augmentation is a 3D animation floating on top

sample and measure the attenuation of the x-rays

of the physical object, while the physical lid is cov-

of the sample from different angles. In the slice

ered exactly with a virtual golden decoration that

the pixel resolution is approximately 0.2 mm,

matches the original lid.

yielding a result which is still sufficient to inspect

Most of the technologies that we used require

object thickness and geometry. Furthermore, the

only a basic skill level with some experience in 3D

scans gave a surprising effect: the gold decora-

modeling (CAD or visualization). A close account of

tions (“goudluster”) caused distortions in the

this developmental process can be found on the

scanned geometry.

weblog

http://porcelain2011.weblog.tudelft.nl

Below, the most essential steps are discussed.

32

3D Reconstruction

Acknowledgements

A substantial part of the reverse engineering is

This project was done with students Anne Lien

the digital reconstruction of the scanned point

van der Linden, Kenneth van Kogelenberg, Mariet

clouds into a valid, 3D CAD model that is print-

Sauerwein and Elizabeth Berghuijs. Furthermore,

able. In this case, much effort was spent to con-

we would like to thank Wim Verwaal and Wim van

vert the collection of jpeg pictures of 2D slices

Eck for their technical advice and guidance, as

into a valid 3D mesh. However, this was not pos-

well as Alexandra Gaba-van Dongen of museum

sible given the time constraints and we ended up

Boijmans Van Beuningen for trusting us with such

extracting a vertical section view to generate a

a unique piece of cultural heritage.

working revolve in Rhinoceros - a regular type of CAD package.

In the cup, an AR tag was embossed, to be used as an optical marker. A 3D print was made with a polyjet technique in maximum resolution (an Objet Eden, accuracy 16 micron).

Manufacturing and decoration Based on the 3D prints, molds were made in plaster to pour the porcelain clay into. After drying the mass, the molds were carefully removed, baked and painted, after which a final transparent glaze was applied.

Augmentation For mobile AR, we chose Junaio – a straightforward smartphone application that offers location-based augmentation as well as image recognition. The so-called GLUE functionality allowed us to overlay 3D models and animations on top of predetermined images (pattern) with a webbased interface to adapt the scale, rotation, and position of the objects. For modeling we used 3D Studio Max, which could export in the proprietary .md2 format with a special plugin. The augmentation of the lid was a gold version of the same object, which was simply a modified version of the same reconstructed 3D file of the lid. The

Original Loosdrecht sugar cup and resulting mesh

cup was extended with an animation of several

model after the CT scan.

portraits that seemed to float in the air.

33

Augmented Belief in Reality Maarten H. Lamers

At the Ars Electronica Festival 2008 I encountered artist Julius von Bismarck, who refreshingly resembled more an Alaskan bearded mountain-man than the typical sleek newmedia artist. Anyway, Julius was there to present his work, the “Image Fulgurator�, a low-tech device that secretly adds elements to photographs taken nearby. Truly augmented reality, to my opinion.

Image courtesy the artist and alexander levy, Berlin

34

let me explain.

If you were to take photographs with your own

our hands. we know that the bunny isn’t re-

camera of, let’s say, barack obama, and the Im-

ally in front of us, but we happily ignore this

age fulgurator was nearby, then a peace dove

knowledge so we can enjoy it longer. In gen-

or a tomato could appear in all your photos. but

eral, to appreciate most augmented reality,

you wouldn’t see the dove or tomato with your

we must willingly ignore what we know is real.

own eyes, only in the photos that you made. It works by projecting the visual element into the

and that is what sets the Image fulgurator

scene, for example onto barack, exactly during

apart from most ar that I encounter: it does

the flash of your own camera. your eyes won’t

not rely on our willing suspension of disbe-

register it, but your camera will.

lief! when tricked by the Image fulgurator, we are truly confused by what is augmented

‘but how is this augmented reality?’ you may

and what is real. just go online and watch the

be asking now. well, the Image fulgurator

faces of the photographers that julius tricked

definitely augments something, namely your

and filmed, while they stare puzzled onto the

photos. and since we accept photos as real-

lCd of their fancy digital cameras. their faces

ity, particularly the ones we make ourselves,

do not show understanding and acceptance of

it augments what we believe to be reality. ac-

what happened. Quite the opposite! they are

tually, that is what all see-through ar does; it

completely surprised by what appears to be

adds elements to our mediated vision of the

reality.

real world. and with most ar systems we willingly suspend

It is like the pink bunny just jumped from the

our disbelief that, for example, a pink bunny

book and kicked them in the nuts: ultimate

really appears from the pages of the book in

augmented reality.

35

[AR]chitectural Education at Play Vincent Hui

36

Image courtesy of SaM GHanTOUS

as a child, video games seemed to be an escape

augmented reality (ar) app, entitled arch-app, at

from the rigors of school yet learning continued

the nation’s largest architecture program at ryer-

in the gaming environment — whether it was

son University in Toronto, Canada.

keeping track of scores and experience points,

The widespread ubiquity of ar technologies has

checking my health meter, or even simply reading

extended into video gaming platforms such as the

in-game dialogue, my entertainment was fueling

Playstation vita and xbox Kinect pioneered by

my education. The superposition of gaming data

its rapid integration in mobile computing devices

atop the fantastic onscreen activity ensured that

and smart phones. In these gaming worlds, ubiq-

I could better understand the world I was play-

uitous data is persistently on display, informing

ing in. It comes as no surprise then that as an ar-

players about everything from mapping informa-

chitecture professor, I bring these same edutain-

tion to enemy data thereby enhancing the gam-

ment and gaming perspectives into my pedagogy

ing experience. where current generation gaming

through the development and deployment of an

technologies have pushed the boundaries of rep-

37

vInCEnT HUI

resentation and simulation of the real world with

Vincent hui (mraIC, assoc. aIa, leed aP) received

incredible fidelity, ar access has empowered

his bachelor of environmental Studies (architec-

people with the ability to bring this persistent,

ture), master of architecture, and teaching Cer-

ubiquitous data to the real world. with a quick

tification from the university of waterloo as well

glance through a smartphone, entirely new lay-

as a master of business administration (specializa-

ers of information are made visible. recent rev-

tions in marketing and Strategy) from the Schulich

elations that the rapid adoption of smartphones

School of business at york university. after

is accelerating (approximately one third of the

gaining international and domestic work experi-

global cell phone market) serve to highlight the

ence with architecture firms around the world, he

inevitable ubiquity of an ar-enabled population.

became a partner at atelier anaesthetic in 2003.

It is within this context of interface familiarity

he has been awarded several teaching citations

and hardware access that I have worked with my

while at the university of waterloo since 2001

ryerson University colleagues, Graham McCarthy

within both the Schools of Planning and architec-

and Steven Marsden, in assembling the infrastruc-

ture. he currently teaches a variety of courses

ture of an ar app that effectively provides a mo-

within the department of architectural Science

bile database of building information to anyone

at ryerson university in toronto, Canada, ranging

with a smartphone.

from design studios to advanced architectural

One of the biggest challenges faced by educators

computing, and digital fabrication. Vincent’s

in architectural science programs is providing

works with physical computing and digital fabrica-

meaningful and self-directed pedagogical tools

tion have been exhibited and published inter-

that are accessible, current, and engaging. To

nationally. his recent work with architectural

many students, architectural engineering materi-

appropriation of ubiquitous computing, augment-

al is often discussed in the confines of classrooms

ed reality, and data-scapes has culminated the

rather than contextualized in real world projects.

development of tools that allow users to access

It is incumbent upon educators to seamlessly con-

data on any landmark in the built environment.

nect academia to real world application. augmented reality proved to be the most suitable technology to ameliorate this condition. In the case of ryerson University, the vast majority of new students are commuters who take public transit from their suburban homes to what amounts to a relatively unfamiliar environment in the downtown core. Despite the incredible architectural renaissance Toronto has experienced in the past three decades, many incoming students arrive unfamiliar with the acclaimed work literally across the street from their classrooms.

Developed within four months, the early prototype of the arch-app was a simple database of notable architectural projects within the

38

The arch-app in use in the summer allowing a user to see a building in the winter.

Image courtesy of SaM GHanTOUS

downtown Toronto campus that highlighted basic

It has since grown into an ar interface that allows

project information such as the completion date,

users to visualize everything from historic imag-

the architects, and some basic text describing the

ery and structural drawings, investigate the his-

project’s architectural relevance alongside sup-

tory and theory behind a building’s design, lever-

porting imagery. Originally developed in HTML 5,

age global positioning to offer tours and guidance

the project was a website that catered to a stu-

to particular sites, and even watch interviews

dent audience that would explore the city while

with the architects behind the projects. Though

mobilized with access to a database of architec-

a proprietary standalone application is currently

tural landmarks. The prototypical infrastructure

underway, the arch-app content from the robust

for the app, known as rULa Maps (ryerson Univer-

database has been integrated into widely acces-

sity Library & archives Maps) became a tricorder

sible ar viewers such as Layar and Junaio.

of sorts where any user with a web-enabled phone could bring up individual layers of pedagogically

Instead of listening to lectures about the built en-

valuable information. Like the venerable tricorder

vironment within the classroom, the arch-app has

of science fiction, the rULa Maps project trans-

become a tool to introduce students to their sur-

formed a simple phone into a device able to un-

rounding buildings and serves to contextualize the

cover pinpoint data on nearby landmarks.

academic discussion via real world application. For

39

example, rather than assign readings and show

in seeing what they have learned in class applied

slides of notable steel buildings in a Structures

in the real world.

course, students could follow a preordained tour

Taking advantage of the web 2.0 paradigm of in-

to visit relevant buildings in the city in person

formation sharing and user-generated content,

using the Google maps interface and global posi-

the arch-app also serves as a platform for stu-

tioning system integrated in the arch-app. Once

dents to add and update content as part of their

at a key landmark, students could also leverage

academic responsibilities in various courses. Stu-

ar interface identifying nearby architectural landmarks

Image courtesy of Sam GhaNtouS

the online database through the app to not only

dents are not only part of the end-user audience

simultaneously see historic imagery atop what

community, but also content creators and editors

currently exists in the real world, but also bet-

who vigilantly ensure app content is current, rel-

ter understand the building through detail and

evant, and robust. For one course students are

interior imagery as well as future design propos-

asked to create 3D structural models that can be

als by other architects. an indispensable feature

pulled up in the app while in another they would

to students within the arch-app is the ability to

be asked to take photos of a construction site

arbitrarily use it to discover information about

to maintain an archive of progress imagery once

buildings they find interesting as they explore the

the building is complete. The aggregation of this

city. Beyond showcasing orthographic, rendered,

process results in an archive that is both current

and historic imagery, the arch-app also provides

and diverse while also leveraging students’ learn-

users a glimpse into a building at different times

ing and sense of accomplishment in having their

of day and through the seasons. The app effec-

content uploaded on the app. This has proven to

tively became an augmented reality architecture

be quite successful as anecdotally students have

professor in students’ pockets, uncovering facets

not only drawn material from courses to the built

of the buildings inaccessible or not necessarily

world, but have also leveraged the arch-app in

visible in person. as though immersed in a video

integrating material from multiple courses. For

game environment, students engage the built

example, using the ar component of the app,

world with a sense of discovery and confidence

students are able to view structural models in

40

tandem with existing conditions and historic im-

tegrative thinking with other courses, connecting

agery to understand why certain architectural

students to a community beyond the university,

decisions were made. This level of integration

and spurring greater curricular engagement within

among courses and connection to existing build-

the department. Its success since its deployment

ings in real-time is a unique opportunity that is

a year ago has merited great interest from both

only made possible via the ar opportunities pro-

within the academic community and a variety of

vided by the arch-app.

external stakeholders including the general pub-

The pedagogical model that the arch-app has es-

lic. Future directions for greater integration with

tablished has since partially become a victim of

other courses within the architectural science

its own success. as students and even members

program and even with other departments such

of the general public continued to upload content

as Interior Design are excellent opportunities in

freely, the limited capacity of the testing servers

continuing the positive academic trajectory of

indicated a need to find better models of maintain-

the arch-app while interest from professional ar-

ing content. The current model in use relies upon

chitecture organizations has generated requests

providing students enrolled in specific classes us-

for collaborations to provide content from its

ing the arch-app with accounts whereupon they

membership to serve as a platform documenting

may upload content on their own. Unfortunately as

great local projects. another dimension currently

with any wiki-based system, there remains to be

in discussion is how to make the arch-app more

a stronger mechanism for curating content which

engaging by integrating social media and possibly

currently is in the hands of a few professors and

adopting game mechanics within the ar interface.

research assistants.

ar-gaming hybridization under the arch-app may

The arch-app has served to address several edu-

prove to be an excellent opportunity to maintain

cational issues in architectural engineering includ-

learning in an engaging, entertaining framework.

ing increasing accessibility, maintaining currency,

ar-gaming interfaces fostering learning — appar-

and drawing in real world application while also

ently some things never change!

netting secondary benefits such as encouraging in-

41

“There’s only one mind, the one we all share”

Image by Martin Sjardijn

John Cage

Digital technologies and fine art a complex relationship Martin Sjardijn 42

3d print weightless Sculpture 10x10x10 cm (3d printer kabk)

The launch of the first Sputnik in 1957, also called

at the same time, we see the appearance of art

“the start of the space age” had major implica-

that has transitioned through cyberspace as a

tions for the ensuing Internet. The fact that man-

‘wormhole’, and thereby took a super-, supra- digi-

kind wanted to leave the earth seemed to run

or megamodern form1. an interactive digital (vi-

counter to the retrospective mentality of the art-

sual) language is developing globally. works of art

ists, who turned to mythology rather than tech-

are created in real-time, sometimes through social

nology. with the advent of digital media, integra-

media, sometimes through virtual and augmented

tion is gradually becoming noticeable in society,

reality, sometimes converting physical material to

which, instead of opposing modernism, gives the

virtual forms and sometimes the reverse. with the

modern perspective a higher plan through digital

advent of 3D printing, sculptures can be made with

possibilities.

modern software that never could have been real-

43

Image by Martin Sjardijn Gerwin de Haan tests Elements installation with virtual sculpture Blue Cyber, TU Delft

ized before. The results of digital media also have

techniques. Only social media seem to attract the

clear effects on image development, which can be

attention of fine arts students as a contemporary

seen, for instance, in architecture and literature2.

means of communication. In ‘Not Just for the Bou-

Developments in the field of virtual and augment-

tique: Art schools and digital everyday culture’,

ed reality are still young and provide challenging

Florian Cramer reports on the use of digital equip-

possibilities for aesthetic and visual experiments,

ment and software programs, among 350 3th year

especially from a traditional fine arts perspective.

Art Students in The Netherlands and observes:

These new forms of expression deserve attention

"Conversely, art schools are now mostly chosen by

from artists with a free mindset, because they al-

students who love manual craft such as drawing

low free investigations, backed by a rich tradition.

and hand making of tangible products. Often, for

However, besides technological advancement,

example in zine- and print making, this is moti-

also significant cultural developments have taken

vated by cultural opposition to electronic media"4.

place during the digital revolution in the past fifty years. At art schools this has led to a critical at-

In line with this, the exhibition called “Simply

titude towards the modern desire for innovation.

painting” which was shown at the Gemeentemu-

Around 1978, ‘the exhausted modern artist could

seum The Hague from 10th march till 17th June

no longer keep pace with new technologies, and

2012, seemed to be a protest against the ever-

was left behind in myth and irony’ as was con-

present digital media and its profound cultural con-

cluded by Achille Bonita Oliva in his book Trans-

sequences. Not that there is anything wrong with

Avant-garde International3, a review of art in the

handmade art — on the contrary, it is an essential

late 1970s and early 1980s. The consequences can

cog in the fine arts machine, and the closest one

still be seen in art academies and especially in the

to mankind — but computers have penetrated so

liberal arts departments. The new tools that ar-

deeply into our society, partly due to rapid min-

rived with the advent of computers, such as the

iaturization, that research and experimentation

Internet and digitally controlled equipment, are

from the aesthetically oriented field of fine arts is

available at art academies, but their application

of paramount importance. This research is current-

is still geared towards industrial design — the fine

ly being conducted by academics with technical

arts departments still focus mainly on traditional

backgrounds and at academies of art, mostly in the

44

photography, film and applied art departments.

References

During my research into zero-gravity in sculpture at the University of Technology Delft, I was much

1 Introduction to digimodernism

more impressed by their interactive technical vi-

http://www.alanfkirby.com/Introduction.pdf

sualizations, than by those I found at academies of art.

2 Anabela Sarmento, Issues of Human

Opportunities at art academies abound in the

Computer Interaction, 2005, IRM Press,

form of knowledge, software and available equip-

London.

ment. In addition to traditional subjects, a wider knowledge base should be offered; digital literacy

3 Achille Bonito Oliva, Trans-Avantgarde Inter-

courses should be offered as part of the liberal

national, Review of art in the late 1970s and

arts, to allow students to understand the artistic

early 1980’s, Published 1982 by Giancarlo

possibilities these techniques present. Cooperation

Politi Editore

with Universities of Technology provides highly interesting results, as seen from the AR Lab of the

4

Not Just for the Boutique: Art schools and

Royal Academy. The artist, born from romanticism,

digital everyday culture. Florian Cramer

is now both an artist and an explorer and is able to

2012.

poeticize widely introduced and socially accepted technology in dialogue with technically oriented scientists.Â

Martin Sjardijn Martin Sjardijn was born in The Hague, The Netherlands.

He Studied Fine Arts at the Royal Academy of Art and for some years Cultural Sciences and Philosophy. After painting for many years, he started the Weightless Sculpture Project in 1985. In 1990 he began to work with virtual reality, using, for example, head-mounted displays and Datagloves with tactile feedback.

Since 1998 he has been developing an Art and Educational Project using interactive 3D technology in collaboration with the Groninger Museum in the Netherlands.

His latest concept is called ArtSpaceLab, which contains a virtual exhibition, a database and a proposal for an artproject inside the International Space Station. The virtual Coop Himmelb(l)au Pavilion gives access to a database, containing a big collection of free software and (video)tutorials as used by Sjardijn.

At the Royal Academy of Art he teaches virtual modeling

Drawing by

for autonomous art.

Cora Beijersbergen van Henegouwen

www.sjardijn.com

45

Who owns the space?

A legal issue on the use of markerless Augmented Reality Yolande Kolstee Using the right AR app, we can see virtual commercial and cultural information and even virtual art placed all around us. But what if we dislike the content made visible by the app? Who owns the space that surrounds us? Can one object to any of this information placed in AR space? What if the information is false?

To be aware of a virtual layer that is placed on top of physical space is one thing, but to be able to erase or correct the virtual layer is another.

God commanded Moses to inform all the Israelites to mark their doorposts with lamb’s blood by which the Lord will pass over them sparing all the Israelite first-borns: they will not “suffer the destroyer to come into your houses and smite you”.

Although AR markers are still in use, in the last years they haven’t been required for every AR application. Nowadays, AR often uses spatial information derived from the worldwide satellite-based global positioning system. Using GPS (and compass) it is possible to post additional location-based information all around us — even inside our own house — without us even noticing.

With smartphones and apps such as Layar, we are able to see location-based information around us. For example, we can see who tweets around us. And we can even get directions to reach the tweeter via an app like “Maps”.

AR Artist Sander Veenhof’s works serve as a good example for AR art that uses locationbased information. Without the board of MoMA

When using visible markers, such as AR or QR

in New York knowing, he placed virtual art in

codes, there is at least a chance to notice that

the museum and even added a virtual 7th floor.

virtual information might be available. A mark-

He did not only augment the MoMA, but also

er might even work as a warning.

the Pentagon and the White House. Through

The use of visible markers (by an invisible en-

this art invasion, he showed clearly that there

tity) can even be found in one of the most ter-

are no physical borders anymore.

rible stories in the bible, Exodus 12:7–13, where

(see http://www.sndrv.nl/moma/ and http://

46

www.youtube.com/watch?v=wyEy2DLu7Wk )

“CDD”) filed a Complaint and Request for In-

Other, less playful uses of AR stand in stark

vestigation with the Federal Trade Commission

contrast to Sander’s artistic use of AR. In a col-

(FTC) against PepsiCo and its subsidiary, Frito-

umn by Lester Madden, he discusses informa-

Lay. The ultimate point of the Complaint is to

tion that might be useful to burglars.

argue that Frito-Lay’s campaign deceives teens

There are more critical comments on the ap-

into eating too many unhealthy snacks, thus

plication of AR as we can see in the blog of

contributing to the childhood obesity prob-

Brian Wassom, a commercial litigator, who dis-

lem.”

cusses legal issues related to AR. He describes his first legal issue in http://www.wassom.com/

In AR space, we can add information on politi-

doritos01.html.

cal preferences for example, or information on sexual habits. This is a really serious topic, be-

“It’s on. For real, this time. A newly filed legal

cause we don’t have a system yet to track own-

complaint raises non-imaginary (although cer-

ership of this ‘space’ around us, we don’t have

tainly still-untested) legal theories concerning

to give our personal approval, and many of us

an actual, commercial use of augmented real-

are unaware of the virtual information, which

ity. AR litigation is now a cold, hard reality.

is placed in the air around us. That is why we

And the result of this initial salvo could have a

will pay close attention to this topic in the next

huge impact on AR campaigns across the board.

issue of AR[t].

On October 19, 2011, four consumer advocacy groups (the Center for Digital Democracy, Con-

Burglary-AR:

sumer Action, Consumer Watchdog, and The

http://www.augmentedplanet.com/2010/01/

Praxis Project–who I’ll refer to collectively as

the-case-against-augmented-reality/.

AR can be used to display information that might be useful to burglars.

47

ChaSING VIrtual SPookS, loSING real weIGht auGmeNted ruNNING aNd a SIde trIP INto the hIStory of audIo auGmeNted realIty haNNa SChraffeNberGer

a strange voice tells me to run. My heartbeat rises

happens visually. By now, probably all of us have

as I follow the instructions without giving them a

seen some three dimensional objects popping up

second thought. The voice’s manner of speaking

upon designated markers, virtual pink bunnies

reminds me of my parents’ TomTom. The only dif-

above augmented cereal boxes or walking direc-

ference: instead of telling me to take a turn, I am

tions superimposed on real streets. However, ar

instructed to accelerate, slow down, to run or — if

does not have to be visual. Sound, in particular,

I am lucky — to walk. I am running with my new

has already brought forth some fascinating ar

mobile app and virtual trainer. The app tracks ev-

applications and artworks such as Edwin van der

ery move, knows when my heartbeat rises, and is

Heide’s Radioscape1 and Theo watson’s audio

supposed to help me gain speed and lose weight.

Space2.

Today, I run to clear my head after a mentally exhausting but physically unchallenging day. Howev-

Entering the latter, visitors can hear the sounds

er, trying to catch my breath, my thoughts return

left by previous visitors, spatialized, as if they

to work. More precisely, I pore over my research

were actually still there. at the same time, they

topic, non-visual augmented reality.

can leave their own audio messages at any point within a room. It is not just the fact that the

In augmented reality (ar), virtual content is

physical space is augmented with the ghost-like

added to our real environment. Most often, this

presence of previous visitors that makes me term

48

it ar. what convinces me is that visitors can re-

I will have to try this app while running. I can al-

late their own sounds and messages to those left

ready hear the sound of my steps on the asphalt

earlier by others; thereby establishing connec-

evolving, blending into a rhythmical soundscape,

tions between the virtual and the real. I imagine

slowly displaced by the wind of heavy breathing,

walkers, cyclists and other runners leaving their

interrupted by pitched variations of my sudden

sound-trails behind on the road, leaving it up to

greetings whenever I meet another runner.

me to add my own sounds and follow their steps, which are spread across time and space.

while RjDj and successor apps like Inception and Dimensions3 are a rather recent phenomenon,

My favorite mobile app, RjDj, can also be consid-

the idea of remixing the sonic environment is not

ered ar sound art. The app remixes the sounds

new. The artist akitsugu Maebayashi has worked

of the surroundings and provides you with a

with similar concepts for a long time. His portable

soundtrack to your life that blends in, makes use

Sonic Interface4 was built in 1999 — years before

of and accompanies your environment. although

mobile phones gained comparable sound-process-

it is certainly no typical ar application, the rela-

ing abilities. The custom built device consists of

tion between the sounds of the real environment

a laptop, headphones and microphones and uses

and those produced by the app is so strong that

delays, overlapping repetitions and distor-

often, they seem to melt into a single soundscape.

tions in order to recompose ambient sounds

49

in urban space. The resulting soundscapes break

everyday sounds. Yet, the effect is similar; they

the usual synchronicity between what one hears

create “a new reality within existing realms, a

and what one sees. Unsurprisingly, Maebayashi is

form of ‘augmented reality’.” 8 Clearly, the de-

not the only one who has been exploring sound-

velopments in non-visual AR were in no way infe-

based augmentations of the environment early

rior to the development of its visual counterpart.

on. In fact, audio augmentations of our environ-

Taking slow steps, I imagine being on such a walk

ment have quite a history of their own. Unfortu-

right now… Listening to instructions on which

nately, they are less known in the context of AR

route to take, where to look, superimposed foot-

and are often not even considered to be part of

steps here, sounds recorded there, on this path

AR history.

earlier, maybe altered with special effects. I imagine those sounds mixing in with the naturally

“Walk!”, my virtual trainer gives in to my exhaus-

present sounds of the river, bikes, and the oc-

tion and I slow down. However, my thoughts keep

casional mopeds passing by. “Run!”, my trainer,

racing. Quickly, they approach the early 1990s:

whom I decide to call Tom, puts an abrupt end to

Tom Caudell is believed to have coined the term

this walk. The fact that AR sound art like Cardiff’s

Augmented Reality. It describes a head-worn dis-

and Erens’ walks are not usually mentioned in the

play that superimposes visual information onto

context of AR leaves me wondering what else we

real objects5. In Caudell’s case, the new AR sys-

miss.

tem helps workers assemble cables into an aircraft at Boeing. What usually goes unnoticed is

After Tom’s instruction, my music fades back in.

that around the same time, Janet Cardiff started

The song is intended to get me to run even faster.

After my footsteps have

“Stop!”, apparently my position adapted to the new it hits me: these has changed enough. My run is over. rhythm instructions about how recording her so-called audio walks. Those walks

fast to run, the information about my heart rate,

are designed for a certain walking route and

distance covered and calories burned and options

confront the listener with instructions such as

such as racing against a virtual running partner in

“Go towards the brownish green garbage can.

real physical space — this is just like AR.

Then there’s a trail off to your right. Take the

In fact, my virtual running trainer shares most

trail, it’s overgrown a bit. There’s an eaten-out

of the characteristics commonly found in AR

dead tree. Looks like ants.”6. While the listen-

applications. It adds another layer of content

er navigates the space, he gets to listen to ed-

to my running. It is interactive and operates in

ited mixes of pre-recorded sounds, which blend

real-time. Just like many other GPS based AR ap-

in with the present sounds of the environment.

plications, it reacts to my position in the world.

Cardiff’s virtual recorded soundscapes mimic the

Most importantly, Tom fulfills my own, personal

real physical one “in order to create a new world

requirements for an AR experience: something is

as a seamless combination of the two.”7 By su-

added (the instructions), something is augmented

perimposing an additional virtual world onto our

(the running), and most importantly, there is a

existing one, and thereby creating a new, mixed

relationship between the two.

reality, Cardiff’s sound art explores one of the key concepts of AR. And Cardiff is not alone with

When another runner passes me slowly, my heart

this idea; as early as 1987, Cilia Erens introduced

rate drops. I wonder whether it might be his heart

sound walks, soundscapes and sound panoramas

rate that is mistakenly reported back to me. I am

in the Netherlands. In contrast to Cardiff, she

astonished, that without the sensor’s help, I can-

forgoes spoken content and uses largely unmixed

not even accurately perceive such basic and vital

50

References facts as my very own heart rate. Maybe this is farfetched, but with respect to that, the running app

■ 1 An article about Radioscape by Edwin

relates to the kind of AR applications which allow

van der Heide is featured in this maga-

us to perceive things about the world that we nor-

zine on pp. 18-23

mally cannot perceive, such as seeing heat, feeling magnetic fields or hearing ultra-high frequencies.9

■ 2 Theo Watson, Audio Space (2005),

So why are virtual Tom and his colleagues not

http://www.theowatson.com/site_

considered to be AR? Perhaps because there are

docs/work.php?id=15

also numerous differences between running apps and common AR applications. To begin with, this running app does not augment the environment.

■ 3 For RjDj, Inception and Dimensions see http://rjdj.me/

Rather, it augments an activity — my running. And to be honest, despite the fact that Tom follows

■ 4 Akitsugu Maebayashi, Sonic Interface

my every move — chasing a virtual competitor

(1999), http://www2.gol.com/users/

or running with a virtual trainer — it still feels

m8/installation.html and http://www.

like they are running on my phone while I have

v2.nl/archive/works/sonic-interface

to tackle the real road. What is more, locationbased AR applications usually display content re-

■ 5 T. P. Caudell, and D. W. Mizell, “Aug-

lated to the user’s absolute position in the world.

mented Reality: An Application of

Tom, on the other hand, is only interested in the

Heads-Up Display Technology to Manual

change of my position over time.

Manufacturing Processes”, Proceedings of 1992 IEEE Hawaii International Con-

“Stop!”, apparently my position has changed

ference on Systems Sciences, 1992, pp

enough. My run is over. The result: 583 kcal

659-669.

burned, 5 miles run and the revelation that the combination of the virtual and the real encom-

■ 6 Janet Cardiff, Forest walk (1991),

passes much more than just adding virtual vi-

http://www.cardiffmiller.com/art-

sual objects to the real physical environment.

works/walks/forest.html

There is a whole field of augmented activities as well! I cannot wait to jam with virtual bands, to

■ 7 Janet Cardiff, Introduction to the

try augmented eating or to take an augmented

Audio Walks, http://www.cardiffmiller.

nap. As if to approve, my heart rate makes a

com/artworks/walks/audio_walk.html

last excited jump. Who knows, in the future, Tom might learn from existing AR. He might then

■ 8 Cilia, Erens, The Audible Space,

have a look at my environment and direct my

http://www.cilia-erens.nl/cilia-erens-

turns so that I discover new routes, point out

2/?lang=en

sights or, when needed, help me find a shortcut home. Considering current developments

■ 9 The course ‘Perceptualization’ which

in lightweight AR glasses, I guess it cannot be

is taught by Edwin van der Heide and

long until we can also see our virtual competitor

Maarten Lamers as part of the MSc

passing by, are asked to design avatars repre-

Media Technology at Leiden University

senting our personal best time in races against

discusses such translations of infor-

other runners and are challenged to chase vi-

mation to our human modalities. See

sual virtual spooks. I would not mind that. And I

http://mediatechnology.leiden.edu/

bet that that is when augmented running will be

programme/curriculum/perceptualiza-

truly considered to be AR.

tion/ and http://www.maartenlamers. com/PZ/ 51

auGmeNted realIty: a Story DIrK vIS augmented reality (ar) is used for many literary purposes: secret pages can contain additional information that is only visible on smartphones; promotional texts can be linked to gabel stones and bus stops, a book can be scattered all over the world virtually, etcetera etcetera. however, ar can also be used as a medium to tell a story. an ar-story consists of separate parts, that each tell a story of their own, which are integrated into a wider story-arc and which interact with the physical reality of the reader. the text below is an exploration of the narrative possibilities of ar. all of your decisions are stored: all of your purchases, your travel destinations and your doubts are recorded. not only far away on Google’s server farms, but also nearby, where they can be seen best: in your face.

Mike writes at night. He is working on the code and the interface for an application. He writes non-stop. He is making an application that calculates how your face will look when you get older. He has forgotten whether it was his idea or ada’s. They invented it as a joke, just as they came up with everything else. But this night, in a single night, he will write the code that will earn him money.

The afternoon before, ada and he were stan-

clay or in pixels, and reads. Imagine if the text had eyes and scanned the reader back as he read.

ding by his door. She was on the sidewalk; he on

The text records the movements of the reader

the doorstep. In her hands she held her sketch-

and responds. The reader finds his way through

book. It was the last thing she still had to collect,

the text with movements of the head. The story

the last thing left of her in his house.

is different — and the effect of the story is dif-

- “People are scared of the idea”, he said, “— you

ferent — because the position, the facial features

are too — that everything is recorded by Google,

and the expressions of the readers differ.

by cameras, but they forget— you forget, that

Mike tests his work on himself that night. The

every misstep is already stored in the place

program scans his face. The color and texture of

where it hurts the most: in your face. I’m going to

his skin are linked to his personal data from the

build that application: The future face one.”

cloud. His medical records, credit card informa-

- “You’re crazy”, she said. “There’s a reason

tion and favourite music are combined, supple-

nobody can see how we will look when we are

mented and extrapolated with the facial scan.

older. True youth cannot know what it means to be old.” reading is a complex task, but the reality of it is simple: the reader sees a text on paper, in

52

He writes his code with fervour. He is constructing the interface, which the user will read later. Sometimes he is afraid of what he does; that things might arise that he’d rather not see,

much like how Ada sees things that she would

be printed that you wanted to be alone, that you

rather not. Illusions, nocturnal animals and fears

could not make the decision yourself, and that

that remain invisible to others, she sees. She has

you cheated on me. You think you have solved it

started to draw them.

and got rid of it, but everyone will read it in your

From the porch door he touched Ada’s cheek.

Probably for the last time, he thought.

eyes, your skin, your nose: ‘do not trust her’.”

Ada and Mike stared silently at each other.

- “Your skin registers the amount of wine you

He touched the tip of her nose with his forefin-

drink, cigarettes you smoke, men you use. Ciga-

ger. That was his favourite place. She had asked

rettes make your radiance fade. Literally. You

him once to draw lines of altitude on her face,

may think, my light will dim as I get old anyway,

like they are on a map. Her nose was the highest

but old women can shine. You say that your genes

mountain. The lines gave a cross-sectional view

determine how you look, but you decide whether

that made her face look like a layer cake.

to smoke and so you too decide your future face.

- “I know you hate my programming. You think I

Whatever you do, even if it’s nothing at all, it is

make gadgets that contribute nothing to society,

engraved in your face.”

but imagine if you could see your future face, the

The city is quiet. Only the taxi drivers and

effects of a night out, if you could be your own

bats are still awake. Mike holds his face in front

Tamagochi, shape your own true face. Plastic

of the test version of his application. The camera

surgery will shrivel by comparison.”

scans the bags under his eyes as he writes: “The

Mike lets the program scan a height map of

system compares colors, patterns and textures

his face, in order that a virtual 3D model can be

with data from others to determine your lifestyle

made. He continues his work inflamed. The light

and future face.”

is returning outside. Based on his personal data

Ada was still standing right there on the side-

from the Internet, the program calculates how his

walk. Mike held his face close to hers.

face will look like in five, ten or twenty years. His

- “You think you can get away with this.”

nose is finished first. Noses never stop growing, as

He saw the skin around her eyes, of which he

if they record all the smells of a lifetime and all

knew every square millimeter.

the faces that came close. The program creates a

- “...but your wrinkles will tell your tale. It will

virtual model and as Mike looks into the camera,

be written plainly on your face for all the world

it superimposes the computer-generated future

to see, no matter how small it is. In Saudi Arabia

face over his own. It’s as if he’s talking to a 60

adultery carries the death penalty; some tribes

year old Mike.

paint a purple spot on the face of the adulteress

- “Finish your application,” one Mike says to

which remains for months. I’m going to make that

the other. He says it himself, but it looks as if

program that we thought of once, that shows

his elder counterpart is speaking. “Finish your

your future face. Somewhere on your face it will

program, show Ada and then go outside.”

Dirk Vis Dirk Vis (1981) studied Beeld & Taal (Image & Language) at the Rietveld Academy and Design at the Sandberg Institute. He published Bestseller (2009) in limited edition, made electronic poems ​​with K. Michel, is editor of De Gids and teaches Interactive Media at the Royal Academy of Fine Arts. www.dirkvis.net

53

Unspecialize!

The more you know the less you see Image by Mareike Bode

A portrait of Daniel Disselkoen Hanna Schraffenberger 54

Image by DanIEL DISSELKOEn

I’m looking out of the window of a tram. Daniel Disselkoen, the graduate from the royal academy of art I’m about to meet, lives in the Statenkwartier ― a part of The Hague I don’t know yet. It’s a nice district and although there is nothing spectacular about this neighborhood, I enjoy the view of

During my studies I took the same tram to art academy on an almost daily basis. on this ride there simply was nothing new to see and hence no motivation to look outside. I realized how easily it becomes boring when you live in a place you know so well.

streets I haven’t seen before. a few minutes later I meet Daniel in a small café

Leaving the familiar behind, Daniel spends several

and he tells me more about this area. He grew up

months abroad. First in america, at the Minne-

in this neighborhood. after studying law and philo-

apolis College of art and Design, then in Japan,

sophy in Groningen for four years, he moved back

doing an internship at the advertisement agency

to the very same street he was born in. That was

wieden+Kennedy.

four years ago and marked the beginning of his graphic design studies at the art academy. Just recently he graduated. It is his knowledge of the city that has motivated his search for new perspectives and inspired several of his works.

When you don’t know a place you look around and spot all those new and interesting things.

But the better you get to know the area the less you look around. While commuting in Japan I 55

paid a lot of attention to my surroundings. I realized that there are three main groups of passengers: those staring at their phone, those who read, and those who sleep or look down.

It’s a general phenomenon that when people know the area they don’t look outside.

Upon returning home, Daniel decides to take matters into his own hands and sets out to make his regular route to the academy interesting again.

The online world calls the game the real-world version of Pac-Man. However, the arcade game was not what inspired Daniel.

I used to play this game with dots on the window. I think there are quite a lot of people who played a game like this when they were kids. Those people enjoy rediscovering it now.

Others like it because it is entirely new to them.

The result is Man-eater, the simplest augmented reality game I’ve ever seen. Without the use of

His story is convincing. Indeed, Man-eater makes

phones, headsets or computers, merely relying on

me relive my own rides to school, pretending the

two stickers, the game adds an additional layer

bird poop on the window was Super Mario who

on top of our view, changes our focus and allows

had to jump from passing car to car. I like the

us to experience our well-known surrounding in

beautiful and unexpected update to the child-

a new way.

hood game and appreciate that Daniel turned it

The first sticker shows a little monster, the so-

into a visually appealing version. However, reli-

called Man-eater. It is placed on the window of

ving childhood memories wasn’t Daniel’s origi-

the tram. The second sticker is a manual, placed

nal intention with the game. His goal is to make

on the headrest right in front of the potential

people notice the outside world again.

player. The manual is hard to overlook and states the four simple rules:

1. Close one eye and look to the right 2. With the Man-eater, eat as many heads of pedestrians as possible 3. Time to play: between two stops of the tram 4. If you haven’t eaten enough heads, start again at the next stop The first level challenges you to eat at least 3 heads, the third level asks for 12. Daniel is aware that this won’t always be possible.

Sometimes the circumstances outside make the game completely unplayable with those rules. For example, if there is simply no one outside then you can’t play, or if there are far too many people it will be too easy. But I think the simplicity is also the beauty of the game.

56

Traveling by tram, I just think this is a great moment for looking around and experiencing your environment in a new way. With the game, I want to provide a fresh perspective, give them a new experience of the city they know so well.

Contrary to what one might think, Daniel doesn’t think that people always have to look around.

I don’t necessarily think it is bad that people don’t look outside. I enjoy those rides where you do nothing at all. The last thing I want is to force commuters into another thing they have to do while they travel. I think in the tram or in a train your mind can be satisfied with the fact that you are traveling. So you are free to do nothing at all, your mind can reverberate, you can just let your thoughts travel as well.

But for those that don’t look around anymore because they don’t expect something interesting to see, I made the Man-eater to let them explore the familiar in a new way.

Image by DanIEL DISSELKOEn

Besides the fact that he doesn’t want to push people to ‘do something’ in the tram, Daniel has more reservations about whether or not to place the stickers.

Very often I don’t like street art like graffiti and stickers. too many times it’s just somebody tagging his name. I think when you push your work in public places, you have to be mindful of your audience.

passengers were not playing the game. Before playing, people looked around cautiously to see whether someone is watching them. only when they didn’t feel observed, they felt comfortable playing it. Consequently, I couldn’t look at them the whole time. So I just sat there and filmed them with my phone.

The dilemma whether and how to observe people with or without their knowledge is also well known in science. Once you tell somebody that you are

Daniel’s unobtrusive solution: in case the stickers

observing him, he might behave differently. If

are not appreciated or even considered vandalism,

you don’t tell, there are ethical considerations to

they can easily be removed due to his use of re-

take. Daniel’s approach paid off but also gave rise

movable glue.

to doubts about his method.

However, not all aspects of the project’s realization are so commendable. In order to test how his project is perceived and get the natural reaction of those playing the game, he filmed the tram passengers sitting next to the sticker without them knowing about it.

I wanted to know how people react by shooting first, and ask for permission to use the footage afterwards. at first, it seemed like the

From my short glances, it seemed like they were not playing the game. only when I checked out the video footage, I noticed the little movement of their heads ― they were playing it after all. However, a few people noticed that I was filming them and apparently felt really bad about it. they stopped playing; sat somewhere else or even stepped out of the tram. I felt terrible about it.

57

The underlying question I asked myself is “Can we change the way we look at our environment and how we experience it?” With Man-eater, I have shown it is possible. With the new app, I take the concept of providing a new experience of the surroundings even further. This game can be played everywhere. I would really like it if people started using it in their room, continued outside, looked around, and then were driven to also discover different places.

I am curious how he intends to alter my view of the world this time.

Image by Daniel Disselkoen

His close observations of commuters, his interest in their behavior and reactions to the game not only lead to some valuable insights regarding the success of Man-eater, but also inspired his newest project.

Just like I had noticed before in Japan, we also have those different types of commuters: those reading, those looking at their phone or tablet, and those sleeping or looking down. During my observations, I noticed it’s the people staring at their phone or tablet who did not play Maneater at all. They just sat in the tram, looking at their phone, not even noticing a sticker right in front of them. That observation made me want to create something that turns the tablet or phone into a device that makes you look around and take part in your surrounding, rather than isolate yourself from it.

Indeed, his newest project does just that. Or to be precise: it will do just that, as soon as it is published in the App Store. Right now, Daniel is in con-

I noticed it is not only knowing an area well that makes you perceive less about it. Knowledge in general makes you see your environment in a certain way. I’m a designer and the more I’ve learned about design, the more I started to perceive the world in terms of design. Of course, I am exaggerating when I say that, looking at the world, I see typefaces, patterns, grids, packages, posters and advertisements. At the same time, I miss other things about the world. For example, I might only see the package and not its content. To generalize and exaggerate a bit more: a biologist might spot plants, an architect might focus on buildings and a psychologist might perceive more about people. What you perceive about the world is influenced by what you are specialized in. My app is intended to free people from their specialized view of the world and will provide them with another way to look at it.

“When you don’t know a place you look around and spot all those new and interesting things.”

tact with programmers, who will turn his concept into an actual app. Until then, the details remain

To achieve this, the app will basically use every-

a secret. However, from what I could deduce, the

thing a phone or tablet has to offer: it will make

upcoming app will be focused on a similar idea.

use of the modern phone’s computational capabi-

58

lities and mobility, it will use the camera as well as information about the user’s position, make use of the Internet connection, use the fact that people always have it with them and incorporate social aspects. Hearing about the technical elements such as spatial awareness and the use of the camera, I expect an ar application. However, this much is for sure: Daniel did not start out with the intent to make an ar app.

I don’t have a strong opinion about what aR is.

I don’t have a fixed definition of it. For me, aR is a bit like labels or branding in advertisement. It’s something you put on top, and suddenly, people experience a product in a different way. I would never start out thinking ‘I should make something with augmented Reality’, and then come up with a project. But in retrospect, I can say that the Man-eater can be seen as an augmented Reality project. I don’t know whether the new project will fit your definition of aR, but it will definitely let people perceive reality differently.

know everything from one subject, it becomes harder and harder to see the big picture and to come up with a new idea.

In contrast to Man-eater, which manages to aug-

Ordering our second round of coffees, I have a

ment the outside with as little as two stickers, this

pretty clear idea that with his work, Daniel wants

app sounds rather technical. I wonder to what de-

to create something new and intends to confront

gree technology served as an inspiration.

us with another perspective on the world ― an

Image by daNIel dISSelkoeN

ambition he shares with many artists. However,

of course, you do need to know something about the material you are working with. But it’s only because I realized that people don’t look around anymore when they are busy with their phones and tablets, that I thought it was an interesting subject and medium to use. I wanted to know more about it, so I got a tablet. Knowing its functionality and possibilities allowed me to come up with the final idea. But the inspiration was not the technology, but my observation of people being completely immersed in it. For me technology is not a target, but a tool.

I think when you know too much about something technical, you don’t come up with ideas that create something new. You then might come up with a technical improvement or technical innovation or something like that. If you

given all our talking about apps, games, travelling and trams, I am not sure yet whether he considers his games works of art.

I would call it ‘applied arts’. I like it if what I do also has a specific function which is not abstract. of course the work and shape can be abstract. But I like it when I can see if it works.

If people look outside, play the Man-eater and enjoy it, it works. this concrete functionality is an important aspect of my work.

Finishing our drinks, we talk about his current work: right now he is designing a crazy bridge, which presumably will never be built. we talk about his other works, including documentaries about Mall-walkers and 47-second long interviews

59

held in the middle of Japan’s busiest intersection.

We talk about the commercial potential of his app, and calculate the hours needed to realize it.

Quickly, the time has come to pay our coffees. Usually, at this point in an interview, I ask my dialog partners a last question about their plans for the future. I already know that Daniel will be working hard on getting his app done. I ask about his plans anyway.

I think I might want to work in advertising. It is one of the little areas where you can come up with an idea and it doesn’t matter which medium you use. You can adjust the medium to your idea. Ideally, I want to first come up with an idea, then choose the suitable medium and then show it to a client. A good agency where you can work like that ― that’s a place where I want to work.

I thank Daniel for the interview. On my way back home, there’s no Man-eater to keep me company. The effect of seeing something new is wearing off and my interest in looking outside is fading quickly. So I take a look around inside the tram. Daniel’s right: there are the ones with the phones, the readers and the sleepers. But there are also quite a few people gazing out the window. Just when I get suspicious, I notice the way they pronounce Scheveningen. There’s no doubt about it: they are German tourists.

I’m not sure whether locals really lost their interest in the outside world. However, I have to yield a point to Daniel’s observations. What we perceive is shaped by what we know. Knowing German makes me spot Germans on the tram. What I have learned about Daniel is most probably similarly shaped by my knowledge and specialization.

Given my own background in creative science, I see the scientist in Daniel: a young observer, driven by the question how we can change what we perceive periments. And if his app will let me perceive things differently, I can’t wait to take part and try it out myself.

Daniel’s website: www.danieldisselkoen.nl

60

Image by Daniel Disselkoen

about the world. I see his games as a series of ex-

61

auGmeNted PrototyPING: aUGMEnTED rEaLITY TO SUPPOrT THE DESIGn PrOCESS JOUKE vErLInDEn In the last decade, new augmented reality tech-

this combination of the physical and digital realms

niques emerged to visualize and interact with physi-

offers both a highly dynamic model that can cover

cal shapes and related artifact knowledge. This en-

engineering and aesthetic aspects simultaneously.

ables so-called augmented Prototyping approaches

The physical interaction allows a tactile and haptic

that combine physical models with principles of

dialogue between participants and artefact model.

projection or video mixing to establish a dynamic

This article draws upon research presented in my

prototype at a relatively low cost. For example,

doctoral research, where I apply ar techniques

nam and Lee documented the prototyping setup for

to the field of industrial design. It illustrates the

an interactive tour guide: a physical mockup was

benefits and systems by summarizing a number of

equipped with microswitches to control a computer

installations found in academia. It will then elabo-

simulation while the output is merged either by a

rate on the investigation of what the true impact

see-through Head Mounted Display (see Figure 1,

of such techniques could be in practice.

left) or by projector (Figure 1, right). In both cases, physical cues such as proportions, key layout and grasping control can be combined with alternate screen graphics and workflow.

IaP as design support in literature Since the inception of ar, speculative design support scenarios for IaP have been devised. For example, Bimber et al. (2001) predicted five application scenarios without much detail: augmented design review, hybrid assembling, hybrid modelling/ sketching, visual inspection of moulded parts and

Figure 1. Two augmented Prototyping scenarios (nam and Lee, 2003).

hybrid ergonomic design. we surveyed the existing IaP applications that cover a mixture of domains and employ various display types (head-mounted

Interactive augmented Prototyping (IaP) requires

displays, video mixing or see-through and so forth).

imaging technologies (output) and sensor tech-

a detailed characterisation of these IaP systems can

nologies (input) to establish an interactive spa-

be found in Table 3, including domain, objective,

tial experience. Furthermore, aP technology em-

and interactivity. In terms of design domains, the

braces existing physical prototyping methods and

applications cover information appliances, automo-

can include virtual prototyping/ simulation tools.

tive, architecture and factory planning, while some

In comparison to traditional physical prototyping,

systems propose a general-purpose IaP system.

62

objective

Interactivity

ar display type

Presentation

Presentation of concept cars

virtual object on turntable, user moves around

HMD video mixing

Cheok et al., (2002)

Geometric modeling

Generating curves and surfaces

Index finger is tracked, creation of control points in air

HMD video mixing

Fiorentino et al.(2002) ”Spacedesign”

Geometric modeling

Surface modeling

Free-form surface modeling, inclusion of physical models.

HMD video mixing

Bimber et al., (2001) “augmented Engineering”

Interactive Painting

Several scenarios

Supporting sketching on mirror, grasp physical objects

Semitransparent screen

Interactive Painting

Painting on physical objects

Moving object and paintbrush, selecting color from virtual palette

Fixed multiple projectors (projectorbased ar)

verlinden (2003a)

Interactive Painting

Exploring component features on CnC or clay models

Moving object on turntable, Change texture/paint by menu.

Fixed projector

verlinden (2003b)

Interactive Painting

3D sketching and rP

Change texture/paint

Fixed projector

Factory planning: layout check, collaborative reviews

Moving objects in 2D plane

Tabletop projection

Interactive simulation for Urban architecture reflections/ shadows/ wind

Moving objects in 2D plane

Tabletop projection

Publication Klinker et al.

(2002) ”Fata Morgana”

Bandyopadhyay et al. (2002) “Dynamic shader lamps”

rauterberg et al.

(1998) “Built-it” Underkoffer and Ishii, (1999) “UrP”

System function

Layout design

Layout design

Fründ et al. (2003)

Layout design

Support automotive modeling

Moving components in 3D space (rotation, translation, scaling)

video mixing HMD

verlinden (2004a)

Layout design

nightclub layout with pedestrian flow simulation

Small-scale physical objects

Fixed projector

verlinden (2004a)

Layout design

Kitchen layout, full scale

Full scale cardboard cabinets and voice control

Multiple fixed projectors

nam and Lee (2003)

Simulate Information appliances

Usability assessment of a digital tour guide

Operating the switches while grasping the object.

video mixing HMD and fixed projector

nam (2005)

Simulate Information appliances

Dialogue definition and evaluation

Sketching screens and screen transitions, operating the switches while grasping the object

Embedded screen and Fixed projector

Kanai (2005)

Simulate Information appliances

Usability assessment of a remote control

Operating switches

Fixed projector

verlinden (2004b)

Simulate Information appliances

Handheld voice recorder

Handheld Mockup with projection

Fixed projector

table 1 : topical overview of augmented Prototyping research.

63

Five different scenarios were identified by inspecting the design activities: presentation, geometric modeling, interactive painting, layout design and simulating information appliances.

These are discussed in the following subsections.

Presentation The main takeaway of most ar systems is the fact

Figure 3. Full-scale car mockup with projection (htas, 2009).

Interactive painting

that product visualizations can be shared with other stakeholders in the design process (e.g.

Interactive painting systems such as dynamic

higher management or potential users). Such pre-

shader lamps indicate the advantages of digital

sentation systems have been specifically devised

drawing on physical objects (Bandyopadhyay et

in automotive design. For example, Klinker et al.

al., 2001). Based on raskar’s shader lamps tech-

(2002) investigated the presentation of (virtual)

nique, a white object is illuminated by a collec-

concept cars in a typical showroom by observing

tion of video projectors from different angles.

the behavior of designers and by presenting some

a tracked wand acts as a drawing tool. when in

proof-of-concept examples. The resulting system

contact with the object’s surface, strokes are

is shown in Figure 2.

captured and rendered in an airbrush effect. as it copies natural drawing on objects, this establishes an easy-to-use interface that has been positively evaluated by kids and graphic artists. a restriction of such interactive painting systems is that the shape of the physical object cannot differ from that of the virtual object — the haptic and visual display of the virtual object will then be misaligned with the physical object.

Figure 2. Example of a presentation system (Klinker, 2002).

Instead of using head-mounted displays, projector-based ar have also been documented that employ mockups with and projectors, for example the system shown in Figure 3. This setup is on demonstration at the High-Tech automotive Cam-

Figure 4. Dynamic shader lamps system in use (Bandyopadhyay et al., 2001).

pus in Helmond, the netherlands and comprises

More intricate progress in software techniques al-

of a full-scale model of a racing car and 3 video

lows sketching on arbitrary surfaces from various

projections that are carefully aligned. In both

distances (Cao et al., 2006). Specific applications

cases described above, the as such, the interac-

have been developed for customizing ceramic

tion is passive, merely to inspect designs that are

plates, cf Figure 5.

modeled in separate applications.

64

on the components themselves, cf . Figure 6. The Built-it system supports the layout of assembly lines in a similar way; simple data are projected on top of the blocks while a large view on the wall shows the resulting manufacturing plant in a 3D perspective.

Both systems exhibit the potential of using physiFigure 5. blueBrush (van den Berg, 2006).

cal design components as user interfaces – the parts are managed by direct manipulation and a design can be reconfigured by multiple hands/

Geometric modelling

users simultaneously. Furthermore, the light reflection and other simulation modules’ support in combination with this tangible interface show

Some geometric modelling tools that originated

the combination of physical spatial reasoning and

from vr were adapted to ar. For example, Cheok

computational simulation. In contrast, an aug-

et al. (2002) presented a see-through ar system

mented modelling system developed at a German

that tracks the index finger by arToolkit (Kato and

car manufacturer deployed virtual components

Billinghurst, 1999). The user can generate curves

on a physical global shape (FrĂźnd et al., 2003).

and surfaces that float in the air. as opposed to

Modelling operations were limited to component

regular vr, this enables awareness of phenome-

placement (translation, orientation, scaling),

nological space, which is relevant for most prod-

while a Pinch Glove supported the dialogue.

uct design activities. However, the system provides no tactile feedback as no physical objects are included. Furthermore, interaction is difficult to scale up to multiple users at a single location, as the movement envelopes of such tracking systems are small. a similar system was presented

by Fiorentino et al. (2002), who adapted their free-form vr modelling application to work with see-through display technologies and infrared 3D tracking. again, interaction takes place in mid-air and although physical objects can be included, these are only used to project texture maps.

Layout design

Figure 6. UrP system displaying an interactive wind simulation (Underkoffer & Isshi, 1999).

Simulating information appliances Examples of this type of design support are pri-

Layout design systems like UrP (Underkoffler and

marily proposed for the design of information

Ishii, 1999) and Built-it (rauterberg et al., 1998)

appliances, referring to consumer products that

offer a number of fixed physical components that

include electronics, e.g., mobile phones, MP3

can be reconfigured on a planar surface. In UrP,

players, etc. In this case, augmentation is used to

the components represent buildings, the augmen-

overlay graphics or other types of visual feedback

tation focuses on the simulation of light reflec-

to a physical mode and simulate navigational

tion, shadows and wind and simulation results

behaviour. Much emphasis is put on measuring

such as flow fields are directly projected in 2D

button interaction to assess the usability of a de-

65

sign by capturing and time-stamping click events. as presented in the introduction of this article, nam and Lee documented the design evaluation of a hand-held information appliance. They used arToolkit to track the position and orientation of the object; the optical marker is attached to the back of the object. In this case, a video-based ar HMD and a projector-based ar are compared and

Figure 7. thermostat mockup with rFID based interaction (Kanai et al., 2007).

evaluated; the projector-based ar display reportedly performs more accurately in the interaction

Kanai et al. (2007) developed a usability assess-

tests. nam (2005) expanded this simulation to the

ment tool to rapidly test interaction with mock-

interaction modelling of both screen and buttons

ups by rFID technology and video projection. The

for a hand-held tour guide by the use of state tran-

level of interactivity with the prototype is even

sition graphs. although the aspect of button inter-

higher when the user can model and interact with

action is well elaborated, the modelling of shape

a product simulation: simple tags can be glued on

and its features is limited (e.g., the location of

foam mockups and the user dons a glove with a

the buttons). Support to track layout modifications

rF antenna.

is lacking and this has to be performed manually.

Characterizing the IaP support scenarios The aforementioned design support scenarios can be characterized by three activities: browsing, interaction with the artefact’s behaviour and alteration of the model. In the table below, a brief summary is shown.

Interaction with behavior

type

browsing

Presentation

view model from all sides – either handheld or fixed to the environment

-

Interactive painting

view model from all sides – either handheld or fixed to the environment

-

adaptation of textures/ materials/ annotations

Geometric modeling

view model from all sides – either handheld or fixed to the environment

-

Creation of new geometry (curves/surfaces)

Layout design

view layout of multiple components

real-time rendering of environment (simulations)

Change assembly (2D or 3D)

Simulating information appliances

Manual handling (physical mockups)

Digital product interaction

Button/screen layout

Table 2. Characterization of IaP design support scenarios from literature.

66

alteration of model

-

assessing the impact of IaP in industry To assess the possible impact of such systems, we

IaP. The responses are summarized in Table 4,

interviewed 13 top design and engineering agen-

ordered by the number of participants that ex-

cies in the netherlands (verlinden et al. 2010).

pressed these. The most prominent strengths are

we targeted senior project managers in various

in line with the envisioned benefits: allowing to

domains, ranging from well-known interior and

explore multiple design variants and to commu-

furniture designers to studios in automotive and

nicate with the client and other stakeholders. On

product design. with each participant, we showed

the other hand, primary weaknesses signify the

a short demonstration of IaP and a 90-minute

effort needed to realize such prototypes and the

semi-structured interview. we asked their opin-

resulting quality.

ion regarding the strengths and weaknesses of

Strength

#

weakness

#

Strong idea, Fills a gap in current design process, different than present methods.

5

Might take too much effort to realize (labor intensive, complex) .

4

Exploring/Presenting many variations rapidly without making it costly.

4

Quality of the projection vs regular finished models (e.g. wood finish).

3

Flexible with fabrics and lay-out, styling-tool.

3

Physical model is required, also needs updated when applied in a new situation.

3

Bring ideas rapidly to the customer .

3

Only 2,5 D (just a “skin”).

2

Might be spectacular — esp. with a handheld version.

1

requires a lot of knowledge/might be difficult.

2

Good for acquisition/marketing.

1

Tactility not the same as end version (interior).

1

Easy to use.

1

User can occlude the projection.

1

Cannot support 1:1 scale for large products/ setups.

Possibly wrong perception of the product by client.

1 1

Table 3: Expressed IaP strengths and weaknesses and their occurrences.

Then we asked about the envisioned benefits of

i) does variability in a design matches the physi-

the IaP to their own design process. The results

cal – virtual division, ii) what is the trade-off be-

are depicted in Figure 4. The participants men-

tween speed versus quality, iii) how does this af-

tion as most important benefits: Communication

fect the emancipation of other stakeholders. The

with external parties, Facilitate user testing proc-

first is related to product domain and activity – the

ess, facilitate concept development, improve in-

fit with some (like website or kitchen design) is

sights, and reduce errors/mistakes. Concerning

less obvious. The second concern touches the need

the method, three main issues need addressing:

for speed versus the need for credible prototypes,

67

which differ based on customer and studio tradi-

turers, engineers and other experts. It remains

tion. The last demarcates the role of the designer

difficult to bridge the differences in knowledge,

in the design process and the power that clients

skills and attitudes among the stakeholders.

and other stakeholders have in the process to ex-

when considering decision making during design

plore alternatives that might be undesirable.

process, physical models can also be regarded as “boundary objects” - as interfaces between the stakeholders in transition between design phases (Smulders’ et al., 2008). These boundary objects encompass both product specifications and argumentation, providing a platform to create shared insight and to freeze the status of a product design for later use. The authors argue that miscommunication and misinterpretation are often related to the characteristics of the used boundary objects, which should bridge the interfaces between design phases and the related discourse domains. There is some evidence that the inclusion of prototypes enhances the performance of collaborative engineering teams (Yang, 2004)(McGarry, 2005). virtual prototypes such as 3D renderings and virtual reality models do provide a good insight; yet have challenges in providing a proper perception of context, scale and proportions (Kuutti et al., 2001). Because the end result is typically a physical object, a materialization of the idea is better approachable than technical drawings or specifications – and often more eco-

Figure 8: Envisioned benefits of the I/O Pad (n=13).

nomical to fabricate.

Based on the functions that other IaP systems

The results show that almost all design and en-

portray, we hypothesized that a key element

gineering firms extensively use models during

of augmentation is not just to enrich a physical

design reviews, and they consider IaP to have a

mockup with additional product information but

potential to avoid miscommunication. However,

to add support for design reviews. Design reviews

they perceive the inclusion of such new technol-

represent formal discussions between the stake-

ogy as time consuming which might not be worth

holders of a design process, and are key in deci-

the risk of adopting such systems.

sion making during the design process (Huet et al., 2007). Our initial concept of the IaP Design

augmenting the design discourse

review system was been devised to support synchronous, co-located meetings that typically do not employ advanced recording techniques. The interaction concept is shown in Figure 9: allow-

Product design is never a solitary process – often

ing to host presentations and discussions on de-

fellow designers are involved in projects, while

sign alternatives while using handheld and large

the act of design requires collaboration with

projector-based ar systems to add information to

client, prospective users, marketers, manufac-

the models as described in the previous sections.

68

During the presentation, audio and video feeds of all IaP systems are captured, while interaction with the pens and navigation through the presentation are captured as notes; input events are stored as well in the segment index, similar to the where were we System (Minneman and Harrison,1993).

The augmented prototyping technologies allow a fusion of combine audio, video, model changes, and user annotations. This results in a large, data warehouse of multimedia indexed by semantic tags (decisions, tool usage, camera switching and the like). These sessions can be inspected later

Figure 9. Interaction concept of IaP as a design review system.

to refine and reflect on the decisions and planned design activities. To our knowledge, this endeavour is the first to capture experiences by recording all channels of augmented reality sessions.

In our current implementation, we have made a handheld system, based on a picoprojector, a webcam and a small UMPC tablet (Figure 10), running a customized version of arToolkit. The ar cube – the desktop version- is packed in a flightcase equipped with a ultra-short throw projector and a specialized Ir tracking system (Personal

Figure 10. working prototype of our handheld IaP system

Space Technologies) and a large TabletPC running vrmeer – based on OpenSceneGraph. It can be placed on a table and be up and running in less than 10 minutes (Figure 11).

Conclusions although some commercial ar solutions have emerged, most inspiration can be drawn from the IaP installations created in academia. as design support tools, these systems showcase the power

Figure 11. ar Cube in use with several 3D printed objects.

of tangible computing as natural and embodied interaction. By inspecting the collection three

as the missing link, while the proper configura-

different interaction characteristics to support

tion of the overall solution concept is difficult to

design emerged: browsing, interaction with the

grasp at this moment. Based on this feedback and

artefact’s behaviour, and alteration of the model.

other empirical studies, a design review concept

In assessing the impact of IaP, Dutch design stu-

was developed, incorporating two different pro-

dios were surveyed. Their reactions are positive;

jector-based ar systems: handheld and a larger

Most of the proposed hardware solutions are al-

desktop model. Pilot studies and additional dem-

ready available. However, software is regarded

onstrations show promising results.

69

rEFErEnCES ■

Bandyopadhyay, D., raskar, r. and Fuchs, H.

environment for testing and assessing the

(2001) “Dynamic shader lamps: painting on

usability of information appliances using di-

movable objects “, proceedings International

gital and physical mock-ups “, Lecture notes

Symposium on augmented reality (ISMar),

in Computer Science, vol. 4563, pp.478–487.

pp.207–216.

■ ■

tracking and HMD calibration for a video-

“Projection-based augmented engineering

based augmented reality conferencing

“, Proceedings of International Conference

system “, Proceedings of International

on Human-Computer Interaction (HCI’2001),

workshop on augmented reality (Iwar 99),

vol. 1, pp.787–791.

pp.85–94.

Bochenek, G.M., ragusa, J.M. and Malone,

■

Kato, H. and Billinghurst, M. (1999) “Marker

Bimber, O., Stork, a. and Branco, P. (2001)

■

Klinker, G., Dutoit, a.H., Bauer, M., Bayer,

L.C. (2001) “Integrating virtual 3-D display

J., novak, v. and Matzke, D. (2002) “Fata

systems into product design reviews: some

Morgana – a presentation system for product

insights from empirical testing “, Int. J.

design “, Proceedings of ISMar ‘02, pp.76–

Technology Management, vol. 21, nos. 3–4,

85.

pp.340–352.

■ ■

Kuutti, K., Battarbee, K., Säde, S., Mat-

Cao, x., Balakrishnan, r. (2006) “Interacting

telmäki, T., Keinonen, T., Teirikko, T. and

with dynamically defined information spaces

Tornberg, a. (2001) “virtual prototypes in

using a handheld projector and a pen”, aCM

usability testing “, Proceedings of the 34th

UIST Symposium on User Interface Software

annual Hawaii International Conference on

and Technology, p.225-234.

System Sciences (Hicss-34), 3–6 January, vol.

5.

Cheok, a.D., Edmund, n.w.C. and Eng,

■

a.w. (2002) “Inexpensive non-sensor based

■

McGarry, B. (2005). Things to think with:

augmented reality modeling of curves and

understanding interactions with artefacts in

surfaces in physical space “, proceedings

engineering design. PhD Thesis. University of

ISMar’02, pp.273–274.

Queensland, School of Information Technology and Electrical Engineering.

Fiorentino, M., de amicis, r., Monno, G.

■

and Stork, a. (2002) “Spacedesign: a mixed

■

Minneman, S.L., Harrison, S.r. : where were

reality workplace for aesthetic industrial de-

we:Making and Using near-Synchronous, Pre-

sign“, Proceedings ISMar’02, pp.86–96.

narrative video, proc. aCM Multimedia ‘93, pp. 207-214 (1993)

Fründ, J., Gausemeier, J., Matysczok, C.

■

and radovski, r. (2003) “Cooperative design

typing platform for hardware-software integrated interactive products “, Proceedings of

Proceedings of CSCw in Design, pp.492–497.

CHI’05, pp.1689–1692.

ama, a. and Kikuta, Y. (2007) “an integrated

70

nam, T-J. (2005) “Sketch-based rapid proto-

ment using augmented reality technology “,

Kanai, S., Horiuchi, S., Shiroma, Y., Yokoy-

■

■

support within automobile advance develop-

■

nam, T-J. and Lee, w. (2003) “Integrating hardware and software: augmented reality

based prototyping method for digital pro-

rison of virtual and augmented prototyping

ducts “, Proceedings of CHI’03, pp.956–957.

of handheld products “, Proceedings of Design 2004, pp.533–538.

■

rauterberg, M., Fjeld, M., Krueger, H., Bichsel, M., Leonhardt, U. and Meier, M. (1998)

■

verlinden, J.C., Horvath, I., nam, T-J. (2009)

“BUILD-IT: a planning tool for construction

“recording augmented reality experiences to

and design “, video Program of CHI’98,

capture design reviews”, International Jour-

pp.177–178.

nal on Interactive Design and Manufacturing, volume 3, number 3 / august, 2009, pp. 189-

■

200.

Smulders, F.E., Lousberg, L., Dorst, K. (2008), “Towards different communication in collaborative design“International Journal of

■

■

Yang, M.Y. (2004) “an examination of pro-

Managing Projects in Business, vol. 1(3), pp.

totyping and design outcome” proceedings

352-367.

DETC ’04, Paper no. DETC2004-57552.

Underkoffler, J. and Ishii, H. (1999) “Urp: a luminous-tangible workbench for urban planning and design “, Proceedings of CHI’99,

links to youtube videos of these systems

pp.386–393.

■

van den Berg (2006) “Project Light Blue”,

■

website http://studiolab.ide.tudelft.nl/van-

Underkoffer’s UrP and other Luminous room Demos http://vimeo.com/2235474

denberg/lightblue.html ■ ■

Fleld et al. Built-it video vor CHI’98 at

verlinden, J.C., de Smit, a., Horváth, I.,

http://www.ibiblio.org/openvideo/video/

Epema, E. and de Jong, M. (2003a) “Time

chi/chi98_03_m1.mpg

compression characteristics of the augmented prototyping pipeline “, Proceedings of

■

Euro-urapid’03, p.a/1.

■

verlinden, J.C., de Smit, a., Peeters, a.w.J.

Bluebrush prototype van den Berg http:// www.youtube.com/watch?v=Dvvl9-C2rU0

■

Tek-Jin nam “Sketch-based rapid pro-

and van Gelderen, M.H. (2003b) “Develop-

totyping” http://www.youtube.com/

ment of a flexible augmented prototyping

watch?v=wK4h9Goa7qc

system “, Journal of wSCG, vol. 11, no. 3, pp.496–503.

■

verlinden aP videos http://youtu.be/F-3BrBfigEw

■

verlinden, J., de Smit, a. and Horváth, I. (2004a) “Case-based exploration of the

■

augmented prototyping dialogue to support

Fiorentino’s Spacedesign http://www.youtube.com/watch?v=GOMx_sytCmU

design “, Proceedings of TMCE 2004, pp.245– 254.

■

Bandyopadhyay et al. Dynamic Shader lights http://www.youtube.com/

■

verlinden, J., van den Esker, w., wind, L.

watch?v=qfwdMZIo4Cg

and Horváth, I. (2004b) “Qualitative compa-

71

Image by Oculus VR

The revival of Virtual Reality?

Wim van Eck In the early 90’s, virtual reality (VR) was the ‘next

seems to have been widely accepted, we can safe-

big thing’. Popularised by movies such as “The

ly say goodbye to VR. Or can we…?

Lawnmower Man”, it was expected that soon ev-

In August this year, a VR enthusiast named Palmer

erybody would be emerged in virtual worlds us-

Luckey started a Kickstarter project which prom-

ing head-mounted displays (HMDs) and CAVEs. The

ises a truly immersive virtual reality headset, the

industry invested enormous amounts of money,

Oculus Rift. While currently the best consumer

but never managed to deliver affordable hardware

headsets only have a 45 degree horizontal and a

which would live up to the people’s high expec-

52 degree diagonal field of view, Palmer’s design

tations. The technology simply wasn’t ready yet.

succeeds in offering a stunning 90 degrees hori-

VR soon became a niche product only used by in-

zontal and 110 degrees diagonal field of view. This

dustries which could afford it, such as the army

means that instead of having the equivalent of a

and research institutes. Since Augmented Reality

large screen at a couple of meters distance, you

(which is even more technologically impressive)

can actually hardly see the edges of the screen

72

anymore. Combined with an ultra-low latency head

such as Unity and the Unreal Engine have prom-

tracker with 6 degrees of freedom, you should fi-

ised support for the Rift, and the first compatible

nally be able to experience VR as it was originally

games are currently being announced. Many are

meant to be, at an affordable price of $275.

already saying the Rift will be the next big inno-

Palmer’s Kickstarter project was quickly backed

vation in gaming.

by some of the most influential game developers such as John Carmack (id Software), Gabe Newell (Valve) and Cliff Bleszinski (Epic Games), who

At the AR Lab, we are also very curious about

were completely sold after experiencing an early

this headset. Not only for VR purposes, but also

prototype of the Rift. After a month the project

to see if we can turn it into an Augmented Reality

was backed by almost 10.000 people and raised

headset, just like how we turned the Sony HMZ-

$2,437,430 — ten times more than the $250,000

T1 into ‘Marty’ (see page 14-17 of this magazine).

goal. The first batch of headsets is mostly meant

We backed the project and we will receive two

for developers so they are able to already de-

prototypes of the headset in December. We will

velop projects for it, while a further improved

keep you posted on our experiences!

consumer version of the Rift is expected to be on sale at the end of 2013. Popular game engines

Website: http://oculusvr.com

73

Contributors Wim van Eck

Maarten Lamers

Royal Academy of Art (KABK) w.vaneck@kabk.nl

Leiden University lamers@liacs.nl

Wim van Eck is the 3D animation specialist of the AR Lab. His main tasks are developing Augmented Reality projects, supporting and supervising students and creating 3d content. His interests are, among others, real-time 3d animation, game design and creative research.

Edwin van der Heide Leiden University evdheide@liacs.nl

Edwin van der Heide is an artist and researcher in the field of sound, space and interaction.

Beside’s running his own studio he’s part-time assistant professor at Leiden University (LIACS / Media Technology MSc programme) and heading the Spatial Interaction Lab at the ArtScience Interfaculty of the Royal Conservatoire and Arts Academy in The Hague.

Pieter Jonker Delft University of Technology P.P.Jonker@tudelft.nl

Pieter Jonker is Professor at Delft University of Technology, Faculty Mechanical, Maritime and Materials Engineering (3ME). His main interests and fields of research are: real-time embedded­ image processing, parallel image processing architectures, robot vision, robot learning and Augmented Reality.

Yolande Kolstee Royal Academy of Art (KABK) Y.Kolstee@kabk.nl

Yolande Kolstee is head of the AR Lab since 2006. She holds the post of Lector (Dutch for ­researcher in professional universities) in the field of ‘Innovative Visualisation Techniques in higher Art Education’ for the Royal Academy of Art, The Hague.

74

Maarten Lamers is assistant professor at the Leiden Institute of Advanced Computer Science (LIACS) and board member of the Media Technology MSc program. Specializations include social robotics, bio-hybrid computer games, scientific creativity, and models for perceptualization.

Ferenc Molnár Photographer info@baseground.nl

Ferenc Molnár is a multimedia artist based in The Hague since 1991. In 2006 he has returned to the KABK to study photography and that’s where he started to experiment with AR. His focus is on the possibilities and on the impact of this new technology as a communication platform in our visual culture.

Maaike Roozenburg Royal Academy of Art (KABK) m.roozenburg@kabk.nl

Maaike Roozenburg is a designer. She develops concepts, projects and products on the border of heritage, design and visual communication.

She founded Studio Maaike Roozenburg in witch she combines a historical fascination with high tech materials and techniques and ‘traditional’ crafts. Besides her work at the studio, Roozenburg teaches at the Post Graduate Course of Industrial Design at the Royal Academy of Art.

Hanna Schraffenberger Leiden University hkschraf@liacs.nl Hanna Schraffenberger works as a researcher and PhD student at the Leiden Institute of Advanced Computer Science (LIACS) and at the AR Lab in The Hague. Her research interests include interaction in interactive art and (non-visual) Augmented Reality.

Esmé Vahrmeijer

Vincent Hui

Royal Academy of Art (KABK) e.vahrmeijer@kabk.nl

Ryerson Universitt Department of Architectural Science

Esmé Vahrmeijer is the graphic designer and webmaster of the AR Lab. Besides her work at the AR Lab, she is a part time student at the Royal Academy of Art (KABK) and runs her own graphic design studio Ooxo. Her interests are in graphic design, typography, web design, photography and education.

Jouke Verlinden

Vincent Hui teaches a variety of courses within the Department of Architectural Science at Ryerson University in Toronto, Canada, ranging from design studios to advanced architectural computing, and digital fabrication.

BArbara Nordhjem http://nordhjem.net

Delft University of Technology j.c.verlinden@tudelft.nl

Jouke Verlinden is assistant professor at the section of computer aided design engineering at the Faculty of Industrial Design Engineering. With a background in virtual reality and interaction design, he leads the “Augmented Matter in Context” lab that focuses on blend between bits and atoms for design and creativity.

Barbara Nordhjem is a PhD student at the Visual Neuroscience Group at the University Medical Center in Groningen. She is interested in visual perception and how humans are able to extract the most useful information from the environment in different situations.

Martin Sjardijn http://www.sjardijn.com/

Dirk Vis Royal Academy of Art (KABK) dirkvis@arlab.nl

Dirk Vis (1981) studied Beeld & Taal (Image & Language) at the Rietveld Academy and Design at the Sandberg Institute. He published Bestseller (2009) in limited edition, made electronic poems ​​with K. Michel, is editor of De Gids and teaches Interactive Media at the Royal Academy of Fine Arts.

Guest ContributoRS Esther de Graaff www.estherdegraaff.nl

Esther de Graaff studied art history with a focus on contemporary art. Now she works as a project manager, curator, writer and coordinator.

She supports artists, curators and institutions in organizing various projects, both in content and in organization. It is her mission to bring culture at the hart of our live.

Martin Sjardijn is a painter, sculptor, digital and conceptual artist. He was born in The Hague, The Netherlands where he also studied Fine Arts at the Royal Academy of Art and for some years Cultural Sciences and Philosophy. At the Royal Academy of Art he teaches virtual modeling for autonomous art.

Special thanks We would like to thank Mariana Kniveton, Reba Wesdorp, Tama McGlinn and last but not least the Stichting Innovatie Alliantie (SIA) and the RAAK (Regionale Aandacht en Actie voor Kenniscirculatie) initiative of the Dutch Ministry of Education, Culture and Science.

Next Issue The next issue of AR[t] will be out in the second quarter of 2013.

75