Sounds Like Play looks Like Play

SLPLLP Sounds

like

Play

Looks

58 90

“

... and suddenly Ben Schouten came up with a group of students who were crazy enough to work with Philips

Like

P L AY

See what we can do with a

KINECT

the ins and outs of the

FESTIVAL

108 I N T E R V I E W

WITH FRENS FRIJNS and FLORIS PROVOoST

“

Floris Provoost

“

We created a dynamic, interactive scenery of colours which could be influenced by the movement, speed and position of passersby. The visitors are (subconsciously) making a dynamic ‘painting’, and in the process creating a performance for the lounging people in the surrounding area of the playground. The ‘painting’ is created through the collaboration of the movements of visitors, either consciously or unconsciously involved in the game.

All individual parts of the concept already existed, Luminous Textile Panels (LTP) by Philips, Kinect depth-sensor by Microsoft, Processing

by Ben Fry and Casey Reas, Kinect Library by Thomas Diewalt.

The combination of these parts created a whole new platform for giving shape to an unlimited number of interactive activities and visualizations. This also means there was no (real) final design, it was evolving every day it was exhibited at the STRP-festival (and still is in our minds). Since Philips already has a number of corporations using LTP in their buildings, our concept might accelerate the process of interactive environments being incorporated in everyday-life.

“

http://youtu.be/wRSV0vU4VQc

Written by Marieke de Rooy

Table of Contents 01. Introduction - Students

6 8

04. Interaction Design

46

06. Visual Programming

48

- Testing with Kinects

- Interaction designs

52

- Streaming content to Philips Luminous Textile Panels 72 74 - Streaming at STRP

- Construction of the report

10

- Approach

12

- Project

12

05. Technical Programming

- Stakeholders

14

- Choosing a sensor

58

- Kinect library

60

- Blob detection

60

07. Production of the festival

02. Midterm Exhibition

20

66

- Interaction requirements

56

70

- Designing the interaction

76

- Programming interaction during STRP

78

90

- First year students

22

- Multiple Kinects

61

- Floor plan and technical drawings

- Tijmen & Marieke

24

- How to connect multiple Kinects

61

- Vivian

26

- Multiple Kinects connected to one PC

61

- Production of the concept & stakeholders agreements 93 94 - Making a model of the festival area

- Pitfalls

61

- Mounting the hardware

96

- Electricity and network

96

03. Explorations

28

09. Interviews

110

- Floris Provoost

112

10. Conclusion and Recommendations

- Using a database

62

30

- OSC

62

- Communication and approval of STRP

32

- Setup with multiple Kinects using OSC

62

08. STRP Festival

98

- Video capturing

34

- ArrayList

64

- STRP festival

100

- Mock-up panel

40

- From received data to visual

64

- Research and Development

101

- Mock-up panel 11

42

- MinDistBlob

64

- Observations

102

- Brainstorm interaction

44

116

- Conclusions

118

- Recommendations

120

92

- Design for festival environment

108

- Frens Frijns

11. Acknowledgments

122

12. References

126

13. Appendix

130

Seven Faces of

Sounds like Play looks like Play

8 | Introduction

9

The Researcher and producer

The Creative Director

The Technical Programmer

The Visual Programmer

The Allrounder

The Artistic Director

The Communication Designer

Vivian Welten, Master Student Industrial Design Focus: The understanding of the interction in search for behavioral patterns.

Marieke De Rooy, Master Student Industrial Design Focus: Trigger creativity and curiosity through playful experiences.

Tijmen Van Gurp, Bachelor 3.2 Student Industrial Design Focus: Creating physicycal dynamic interactions with surprizing factors

Anika van der Sanden, Bachelor 1.1 Student Industrial Design Focus: creating a co-crea-

Tom van ‘ T Westeinde Bachelor 1.1 Student Industrial Design Focus: Creating a performance interaction.

Max Verhoef, Bachelor 1.1 Student Industrial Design Focus: Creating an interaction which makes the player an artist.

Alex Schepers, Bachelor 1.2 Student Industrial Design Focus: Creating an interaction which triggers communication.

My special interest is in

I aimed this project for

In this project my focus was

Creating a co-creation inter-

With this project I wanted to

At the early stage of the

This project my focus was

designing for music concerts

designing co-experiences;

on technical parts of pro-

action that could elite

make an interaction with a

project my focus was on an

on creating a social trig-

and festivals. I think the

visitors create meaning and

gramming. In the ideation

communication. Because

unclear starting moment. I

interaction implemented

ger. During one of the first

design should go beyond

emotion together through

phase of this project my fo-

our interactive system was

wanted an interaction that

with a carrier, which makes it

brainstorm I had come up

the limits of the stage area

a social and playful activity.

cus was on interactions with

going to be placed at a

provokes a performance for

clear whether you are play-

with a way to make people

and involve the visitors in

Next to this I wanted visitors

physical cariers that would

festival I found it interesting

visitors in a lounge and

ing or not. Later on I found

communicate and share

the performance. For that

of the festival to be able

suply for a direct interaction.

to see if it could elite com-

create a dynamic atmos-

it more important to create

information.

I strive to create surprising

to express themselves in a

I believe that interactive con-

munication between festival

phere. I wanted to par-

the possibility for the player

Furthermore making sure

and memorable experiences

non-auditive way; through

cepts should be intuative

visitors.

ticipate in every part of the

to create his own things, so

the creative process is

through interactive media

gestures and colors.

and fun. One way to do that

process to get an idea of

everyone can be artistic in a

depicted as it is supposed

design.

is by giving people a sense

every aspect.

fun way.

to be. Create something to

Let the spectator become

of control, for example the

show what has been cre-

participant.

control of a physical object.

ated.

tion interaction that could elite communication

10 | Introduction

10

Report

11

Construction of the

S

ince we have been working as a group for most of the time, we agreed upon creating a group-report describing the process of us working in a team. The reporting will start from the moment we started to work together. How did we combine our individual ideas into one concept? After the festival we split up again, all doing our own analysis of the project presented at STRP.

I

n the group-report, we will discuss our group-exploration, technical and visual realization, production of the festival and the exhibition at the STRP-festival.

All parts that were done individually are described in a separate report. The conclusions of these individual activities are combined to formulate one overall conclusion. This conclusion is presented at the end of the group-report.

12

Approach We started out individually, all with our own approach and goals for this project. During the midterm exhibition we showed our own concepts. After this exhibition we teamed up and started to formulate an overall concept, in which every one of us would be visible as a designer. All strengths were bundled to create, organize and build a playful, interactive lighting installation for the STRP festival. In four weeks we succeeded to create an interactive lighting installation that enhanced the atmosphere of the lounge area at the STRP-festival.

The Project This project was set up to create an experience of expressiveness such as artists perceive expressiveness. This experience would take place during the STRP festival in the fall of 2011 in Eindhoven.

The project aimed at creating a fun, surprising, unconstrained interactive activity that made for a feeling of being creative in its users.

The design of the (playful) interaction was the main challenge of the project, since the target-group, output device and location were fixed at the outset. (1)

Introduction | 13

Stakeholder STRP festival “The STRP-festival is a yearly festival in Eindhoven and is one of the largest art & technology festivals in Europe, that focuses on music, art and technology. The multidisciplinary program is a mix of a 360-degree experience with adventures which appeal to a wide audience. At STRP there are projects of young game designers next to major works from the international art circuit and experimental live cinema next to successful pop artists and DJs. At STRP you find interactive art, light art, robotics, concerts, DJs, theatrical and dance performances, experimental music, interviews discussion, live cinema, films, lectures, video art, animation and workshops. This year it is mainly about an overview of the last 50 years of Dutch media art and technology. The STRP festival took place from November 18 to November 27 at the ‘Klokgebouw’ in Eindhoven.” (2) The client STRP is looking for interaction between unrelated visitors. The STRP-festival has a wide variety of visitors. The visitors of the STRP festival are looking for experiences, distraction, a good time with friends and inspiration, and they go there out of curiosity. They like to create meaning and emotion together by attending different events during this festival.

STRP’s requirements Interactive lighting in the lounge/café area of the festival Playful experience Connecting visitors Creating a good atmosphere

15 | Introduction

14

17 | Introduction

16

Stakeholder Philips The client Philips Large Luminous Surfaces is looking for the possibilities in interaction with their product.

They are interested in the reaction of people and the communication between people and the Luminous Textile Panels (LTP). This medium uses LED’s under textile to create an atmosphere through blurry visuals.

The panels as they are now are static objects which change appearance in a passive way. Philips provides eight LTPs with black fabric to be used at the festival.

Their sizes are 250 x 120 cm. (3)

Philips’ requirements Interactivity with and through the LTPs Exhibiting new product, expanding brand with interactive lighting Exploring the technical limits of the Panels Beta-testing for the software

Stakeholder TU/e Industrial Design The goal of the faculty of Industrial Design is: “Educating unique opportunity creators for societal transformation through intelligent systems. (Systems are networked technology, products, services and users within a societal context, and the interaction between them).”(4) The focus of the theme Playful Interactions is on changing people’s lives by designing playful systems that seduce people to activities that contribute to their health and well-being. As Huizinga wrote in his book “Homo Ludens” (1938) people are inherently playful beings.

How can we design products that allow for playfulness in daily activities?

The playfull interaction group examines the range between designing for play, such as play environments and designing for a playful approach to all sorts of other activities, for example supporting playful communication between older adults. (5)

TU/e ID’s Requirements Playful interaction Intelligent systems Societal transformation

18 | Introduction

Introduction | 19

Midterm Exhibition Results For the midterm exhibition every student came up with their own idea for an interaction. We worked out our own ideas to present different possibilities at the exhibition.

Anika, Tom, Alex, Max

The first year students want to stimulate communication between visitors. Every visitor is related to a light spot on the panels which tries to get their attention.

These lights want to urge you to follow them and lead you to another person so you can start a conversation.

Because of the interaction between the lights above you, you might be triggered to interact with the person standing in front of you.

23 | Midterm Exhibition

22

25 | Midterm Exhibition

24

Tijmen Tijmen created a concept where people could manipulate content by movement of a physical object. Tijmen made use of the accelerometer in his phone to test his interaction. By changing the angle of the phone the content on the panels would move accordingly. By putting the phone in a chair people were able to manipulate the content by changing the angle of the chair. More about this concept can be read in Tijmen’s individual part in the latter part of this report.

Marieke Marieke created a concept where people could paint on

By working together with other visitors you can create and

the LTPs: the system would use the colour of your shirt, your

change the colours on the panels and thus the lighting in

skin, your hair or the expression of your face to relate to

space. Your swatch would change according to your speed

you on the panels. By looking at the panels you will leave a

and direction. By gathering all people underneath one

trace of the colour of your shirt. If someone is just walking

panel the game play will change.

through the field without looking at the panels he will only mess up the colours or leave a rainbow of colours.

M1.2 Vivian came with the idea of a visual world seen from underneath.

27 | Midterm Exhibition

26

Vivian Motivation

What changes are being recognised in the architectural context?

I find it interesting to investigate how interactive multimedia design stimulates

By the use of multimedia systems architectural spaces evolve in dynamic, respon-

user experiences within the context of public events and entertainment. Within

sive or interactive environments. One of our biggest concerns is that the visitors will

these event settings interactive media is recently introduced for commercial or

not recognize the ceiling to be interactive. With subtle changes in the environment

entertainment purposes. I am not directly referring to the social media. I would like

we try to catch the attention of the ones who pass by.

shows and performances to evolve in interactive play and let spectators become participants by designing such interactive physical environments. Many people are

To what extend are the visitors aware of their influence on the output?

not used to this type of interaction yet, often the scope of interaction is limited to

The interaction between human and environment is difficult to define and will not

human computer interaction. That is why I wonder what kind of input is interesting

immediately be recognized. More changes in the architectural context might ques-

for creating an interaction with the environment.

tion people what their influence could be on this.

Research question

What influence has the number of visitors on the behaviour of the participants?

What behaviour is relevant for understanding the interaction between human and

When more people interact with the same medium it might become unclear who

environment?

is responsible for the effects. A lot of individuals creating input for the system can result in the same large variety of output. The individual input and its effect on the

Aim and objectives

installation is than more difficult to relate. This could be one of the causes for visi-

Leads the possibility of interaction between visitor and installation in an architec-

tors not being aware of the interaction.

tural context to playful behavior among festival visitors? I would like to find out whether interactive media in the architectural setting is being recognized and how

What were custom behaviours of the visitors interacting with the panels?

it adds value to the experience of visitors in a temporary event setting.

This might be important in making a difference between functional and playful behavior. By observations we could identify the behavior which is performed to discover new outputs and exceeds the custom behavior in that setting.

Design proposal for content creation on Luminous Textile panels A direct mapping of the position at the ceiling as if seen from underneath

Screen shot of Surface Film, part of thesis project MFA Design+ Technology ‘09

Design

for the festival environment

The space

The panels were going to be placed in the lounge of the light café of the STRP-festival. The light café is an area for the festival visitors where they have the possibility to eat and drink something and escape from all the impulses from the other areas. In the light café there is a continuous passage of visitors.

Positioning of the Panels

30 | Explorations

For this project we had 8 panels that we could use for the interaction. Before we came up with an interaction we decided to reach a clear agreement on how the panels were going to be placed. We made a template in Google SketchUp of the space where our panels would going to be placed at the festival. In this template we all presented different ideas of the positions of the panels. The only requirements we got from STRP and Philips was that the panels had to be fixed on a minimal height of 3 meters. This was due to the varying public that would visit the festival and we could not afford any damage to the products of Philips.

31

32 | Explorations

1.

2.

Communication and approval of STRP

After discussing every idea for the positions of the panels we chose four ideas and sent these proposals to STRP so they could give feedback.

Of the four proposals we preferred option one because of the fact that this option covers a large space. This option also offers many opportunities for interaction since it is effective regardless from which side it is approached. In option 3 the panels are positioned close to each other, which lights up one area which is like a spotlight.

The feedback we received on proposal 1 was that with a half open curtain it would give too much light pollution on the podium and proposal 3 was too squared. After this feedback we made one new proposal.

3.

4.

5.

The feedback we received on our new proposal was that it would interfere with other light sources and that large parts of the space (two blocks) we had available were not covered. STRP came up with a change in our new proposal. This proposal seems to them the best option because it is a bit asymmetrical and it covered the area we had completely.

They also agreed with our idea to position the middle four panels horizontally and the other four a little tilted.

After we discussed the proposal of STRP we agreed that this would be a good option for the positions of the panels.

6.

35 | Explorations

34

Video capturing After the first client meeting with Philips we were much

For the next appointment with Philips we made some new

more familiar with the panels. For the first explorations we

footage. This time we created footage based on the brain-

would experiment with different kinds of content for the

storm sessions which resulted in the use of natural elements

panels. We started with filming all sorts of footage in the

and upside down worlds. Max made a Flash animation of

photo studio. We all brought different materials, such as

walking footprints and Vivian used an enlargement of an ex-

crepe, a disco ball, a flashlight, different colored liquids. We

isting video of YouTube to create the same idea as Max. We

used different light settings to play with shadows and light

also shot new shadow material using cloth and spotlights.

effects. We switched between a black and white background

While we were testing this footage we noticed that the foot-

to see what was the best effect was but we soon agreed on

prints had just the same effect as the shadows we tested;

the black background because this gave the best contrast-

they really had to be big enough to be noticeable.

ing effects with the black textile on the panels. We also filmed drinking glasses that were slowly filled with a liquid and reversed by drinking through a straw. Finally we shoot moving objects like rolling and bouncing balls.

With the footage we went to the High Tech Campus to test

http://www.youtube.com/watch?v=X26kt53N35s&list= PL825402CE33CC8B00&index=9&feature=plpp_video

this together with material from YouTube. We noticed that using a variety of colors has a nice effect on the panels.

Movement of visual images only works when the movement is very recognizable, for example a silhouette of a shadow play of hands was easily recognizable, but the footage of quick moving balls did not work out on the panels.

Shadows did have a nice effect but the silhouettes need to be of such a size so that they are clearly visible and recognizable. http://www.youtube.com/watch?v=W17Ce9TrD7M&list =PL825402CE33CC8B00&index=8&feature=plpp_video

36 | Conclusions and recomandations

37

37 | Conclusions and recomandations

36

36 | Conclusions and recomandations

Exploring movement

Conclusions and recomandations | 37

38

Philips Content Manager In the last exploration phase with footage we were able to test at the university since we were equipped with software from Philips to use on our own laptops. The software would show a preview of how uploaded videos would look when uploaded to the panels. We used natural material as visuals to test with that program. We filmed moving trees, falling leaves and flowing water. We also got a new idea about the upside down world. We placed a camera with a fish eye lens on the floor and walked around it to get the idea of a transparent floor. When importing these files in the Philips preview-software it seemed that the leaves and the water were hardly visible because of the blurry effect of the panels.

The trees worked out pretty well because of the color contrast and the adapted speed of the video file.

Mockup Panel Motivation/reason:

Examples

At this point in the project we wanted to go from paper

At first we started to beam YouTube content on the

towards reality. Because we did not have a panel avail-

panel. This was just to see what kind of effect this had on

able to test content we decided to make a test-panel so

the people who walked by. The videos we played were

that we could get a better feeling of the situation at the

mainly very abstract and colorful. Bit by bit the concept

festival.

got shape, and we started to find out which visuals we wanted to have on the panels.

We built a spruce frame, and a surface of white sheets.

We chose the dimension of the panels we would get in

With our requirements in mind Tom made content in

the final setup. We used a beamer with a slightly unfo-

Particle Illusion to simulate this. There were two particles

cussed output to simulate content on the blurry panels.

on the panel which moved around each other and even-

This panel was installed in the hallway for a couple of

tually met. With the sitting bags underneath the panel

weeks. The purpose of this was the possibility to observe

we could experience the effect of video animation from

the passing students in a very early stage in the project,

above just like it would be in the lounge at STRP festival.

and see how they reacted to different footage. What we

But it still was not interactive yet. Tijmen made a Process-

concluded from these observations was that people only

ing sketch which was interactive. An iPhone was used

noticed the visuals when there was an obvious variation

as accelerometer. Simply by moving the iPhone around

in color and movement.

different kinds of balls danced around on the panel. As soon as it got interactive, the effect of the combination with interactivity and a large panel above you could be experienced.

Conclusion The decision to make a real physical panel brought us one step closer to the real thing and helped us to test our ideas in context. The further developed context also narrowed down the ideas we had towards more concrete ideas for the setting at the STRP festival. After the panel was finished we all went our own way in developing our ideas further. Because we tested our ideas on the panel during this development we were able to give each other feedback on the results.

40 | Explorations

Explorations | 41

Mockup Panel II Motivation/reason: When we decided to use multiple Kinects we needed to simulate a test situation with multiple Kinects that was as

43

close as possible to the real situation at the STRP festival.

Because we were not able to use the LTPs until the festival we needed a test situation to find new opportunities and possible pitfalls.

Construction/situation Until now we had a panel hanging at a height of 2 meters, at the festival it would hang at 3.5 meters. Therefore we chose a spot at the Technical university where we could make a construction from wood, some duct-tape and wires to simulate the situation at the STRP festival. In this setup we mounted 2 Kinects to the wood with a space of approximately 2.5 meters in between.

Conclusion With this setup we were able see how large the new playground was. We found out that we would need high quality USB extension cables to prevent data loss of the Kinects. Because of this setup we could also decide the direction of the Kinects, and the amount of space needed between them.

42 | Explorations

http://www.youtube.com/watch?v=ljrGmuCznKs

45 | Explorations

44

Brainstorm interaction After the midterm exhibition we started thinking about the

that we thought had the best potential. Using these ideas

we did a fourth brainstorm about ceilings because of the

Eventually there were seven brainstorms on the

concept for the interaction again. The best way to do this

as elements for another brainstorm we continued to

way we are fixing the panels and this resulted in another

table. The themes we thought were most fascinating

as a group was through several brainstorms.

expand the options of these elements. This brainstorm

brainstorm about an upside down world. An important

were the things you see from below and the ele-

With the idea that colors really work well with the devices

resulted in 5 new criteria. With the idea in mind that we had

part of the content was the way we would draw people’s at-

ments of nature. These were the two concepts that

of Philips we wrote down everything that came to our

to make a vague interaction because of the low resolution

tention to a spot in the room that you normally hardly look

we started to work with, so we started to make foot-

minds concerning movement and different colors. Of all

of the panels in comparison to “normal panels� we did a

at. For that we closed the brainstorm sessions with three

age of this to experience the effect on the panels.

these ideas we chose the ones

third brainstorm about vagueness. eparate from the last

brainstorms about seduction and deception.

brainstorm

In the 6th week there was a group discussion about the concept requirements.

Everybody mentioned his or her personal requirements, interests and ambitions. an ve ia r h va ld ou ome s of h l s ve ds pt ce e an nt le e in l in n c co n ris efu s en ere a me the surp diff peri e us j i T o h lls, f x b g ks s e o l n n n i i i a t s a sk Th en mak e ha ch c . He m es eatt i s y e H l h p . b e y w nce en y cr on lexit ng, ds o n i c ati db p a m the m ve m m e r o i a c r ing is fo ach og e pr elop p h v lo uld b ut. e e v d -o de is co lay t to h r dt po an a re ing

Anika act to une system to re Would like th ogramns. She likes pr intended actio in PHP e experience ming, has som nguages velop other la but likes to de her visuts to develop an w e Sh l. el as w ter. s next semes alization skill

Alex Think s that the co prese ncept nt in a should s ubtle need be way. I a clea t doe r purp s enter n ot ose, b tainm ut ser ent; g ves as oppo iv e the vis rtunit y to c itors t exper reate. he ience H e h in a s p s o rogra would me mmin like to g and develo would p this like to . He a impro tion s lso v e kills, b h is v is u u t a not n lizathe sc ecess ope o arily w f this it proje hin ct.

racf inte o m for end rect eke t the re di a o d l m Mari ou ious ea m sh onsc ld lik e c t u s o d y W an the s ectded and conn inten n n tion o a is o She rise t ther. ocus f e e e v g i h o g t d n. T s, an ating actio isual d cre v n inter e a h t e g the eopl rk on akin t o , w n ing p o ti ld e to terac wou ld lik and he in wou t , he t n i u rig t to t abo reac t of o e n l i think o p o p w pe ut as outp ut ho o d o fin like t es. imag

He ts. c to bje g h o cts in nin g g g u i e j s tin ro th ry ob y de eres as n tio dina ary b n int He h c s . in x or era is a sed o thi on Ma s int turn raord sity u s , cti e t o be ing du ts Lik ts to g ex Curi c to o m r n t n i in on. ist ram ce in rime wa eth ti i ter rog c e m c n a p so er ara in p as a e ex ept. int r ch e e a c nc v n its a en er oftw e co i s m r hu xpe th es uld e co g th ts of no ject n en rvi o se lopm pr b o ve by de d an

Viv ian Lik es the un d tar y to evelo tio pm w ns ar ent con . To im ds co nsc from ple cep inv iou m t alre ols in ady we sh ent th t e ou i r the pre s a in t cld t sen int he ake era t as set t c h tio up ea n. T point te ctio ana his of o ns lysi st cas p r r i e s g o wo , st s an in f jec ron uld o t d she r wo g like ing rk o wil to d visual ski l n lls. lan the eve gu lop age vid and eo ani ma t-

Tom Wan ts th e in ous tera . He ctio thin defi n to k s ned ther be n m e ot t part ome sho oo o uld in th nt w bvin ot b here e in som tera eac peo ethi c l t earl ple ng u ion. exp dec y nex The erie ide pec s nce y s t o ta the tem ted. i n ke soft pro sho gram Like M war uld He i ax h e de d m o s int i e n velo eres pme g and w has no the ted con n ill o t i n from cep bse the and t, th rve ove the out e r side all d rela put line tion eve and . lopm bet how wee ent peo n o the f ple inpu will reac t t.

After discussing everybody’s concept requirements we combined them into the following requirements:

Considering an interactive system consists out of three elements - input, system processing and output - we separated the team in terms of these categories. We would first explore the possible solutions, then combine the system and improve the quality.

INPUT Tom and Anika

SOFTWARE Tijmen (towards input) and Alex (towards output)

OUTPUT Marieke, Max and Vivian

Variable complexity Surprising element by random factor Turn ordinary into extraordinary Subtle in its presence No clear moment of participation Trigger intended interactions at the end

Stimulate co-creation

Sketches

Results

Since the previous brainstorms were more about content

good impression of what a certain interaction will look like.

The most important thing that came out of this brainstorm

player. He or she would be able to control, paint or influence

for the panels instead of interaction scenarios, we decided

The actions that had to be performed for the interaction still

was the principle of having at least two hot spots or play-

the panel by just walking underneath it.

to do another brainstorm, more focused on interaction. This

ranged from drinking out of a glass to waving the arms. Also

grounds. Using these playgrounds the user would be aware

time it was based on sketches. We thought about different

there were some options on how to start the interaction.

of whether he/she was joining the interaction or not. The

scenarios and immediately drew these on paper. This gave a

second thing we wanted was to make a creator out of the

Conclusion By thinking of these different scenarios we set a major step towards our definitive concept. The hot spots and painting principle would be essential for the end result.

Technical Programming When reviewing the scenarios we found most interesting,

of this sensor. Its basic properties satisfies the exact require-

means that beside the regular software of Microsoft© we

of the head during the measurement. This resulted in a

we could clearly see that most scenarios would use the

ments of what we want to do with it. The combinations of an

could find a lot of other software that could help us to

small, non-profitable rectangle in which hardly any inter-

position and movement of people to create an interaction.

infrared camera, a RGB camera and the infrared laser make

realize our ideas. (6)

action was possible.

Using a Kinect as sensorial input sounded very promising,

it possible to capture 3D images. The depth sensor consists

We wanted to detect people from above, since this

it was the solution to track the movement and position of

of an infrared laser projector combined with a monochrome

would enable us to precisely locate the position of

The second time we defined a much bigger rectangle

people in space. Regarding the other options we had, this

CMOS (image)sensor. The Kinect is capable of detecting the

multiple people in space. We had to find out what the

because we changed the range to detect people’s feet

would be the least fragile and most reliable sensor to use.

skeleton from one up to 6 people. Because of the motorized

Kinect exactly detects when it is fixed parallel with the

which gave more options for an interaction. We even

tilt the sensor can vertically detect in an angle of 47 degrees

ground. For that reason we made a small setup in which

tried to enlarge this playfield using a fish eye lens espe-

and can be tilted up to 27 degrees up or down. Horizontally

we placed the sensor on a height of 3 meters and with

cially made for the Kinect. It appeared that this had no

When we choose the Kinect as the sensor for our interac-

the Kinect “sees” an angle of 57 degrees which provides a

the software that the technical team had provided we

value for the detection because this lens also changed

tion the input team started to look at the specifications

playfield of 6 square meters. Next to this, we found out that

defined the playground for the interaction. Because of

the accuracy of the sensor in a negative way.

and possibilities of this sensor with all its multiple options.

this sensor is a popular hardware product that has already

the adjustable depth sensor we had to go through these

While searching the internet we discovered the versatility

been hacked by hundreds of creative programmers. This

steps twice because the first time we detected the top

Technical specifications of the Kinect

59 | Technical Programming

Choosing a sensor

Kinect Library

Blob Detection

After a lot of research about which drivers and libraries

Blob detection is a method that is used to filter objects and

to use (see Appendix ) we decided to go for the library

people out of a map of pixels detected by a camera using a

of Thomas Diewald. The main reason for this was that his

certain threshold. Because the Kinect senses depth the use

sketches from his OpenCV(source) kit worked without

of Blob detection is much more precise than when using a

calibration of hands or skeleton. Another advantage of this

normal camera. This is because in a normal camera needs

library was that Tijmen could use Thomas Diewald as an ex-

an contrasting background if you want to detect blobs.

pert during the process as he responded quickly to pro-

A depth camera makes a high contrast image even in a

cessing-related questions via e-mail. The processing sketch

chaotic background. Because of this advantage of a depth

that we eventually used made use of the depth vision of the

camera blob detection is faster with a depth camera than

Kinect and blob detection to detect persons in the scene. (7)

with a normal camera.

Lumalive Panel

Kinect Playground

Multiple Kinects

How to connect multiple Kinects

When we first tested one Kinect searching for blobs from

When we had decided that we wanted to use multiple

above (from the ceiling to the floor) we measured the

Kinects we had to find a way to combine the data into one

amount of space in which it could detect blobs. This was

visual output. To do this Tijmen thought of several options

175 x 135 cm for detecting the head when the Kinect was

to get this done:

at a height of 3 m and 245 x 185 cm when the Kinect was at 3,5 m.

Multiple Kinects connected to one PC

The total space that the panels would cover at the STRP

One option would be to use one computer for connecting

festival was 1000 x 420 cm. Therefore it was logical that we

the Kinects. The advantages would be that we would only

needed more than one Kinect to extend our playground.

need one pc to receive the data and make a visualization.

Because there was no budget it was not possible to buy

Together with Marieke and Anika, Tijmen tried a sketch of

more than 3 Kinects (3*120 euro) from our own money, this

Thomas Diewald made especially for multiple Kinects.

meant that we could work with a total surface of : 735 x 185 cm.

Pitfalls We would need an extremely fast computer to run 2 Kinects at the same time, we tried this on the newest computers with a Core i7, on this computer it worked with 2 Kinects but there was another problem. Adding 3 Kinects to one motherboard was not possible as the USB slots of a motherboard can only handle 1 Kinect per USB Slot. On these fast computers there were only two USB slots available. So this would not work with three Kinects.

60 | Technical Programming

Technical Programming | 61

Using a database When was decided that we needed to connect all the data

Setup with multiple Kinects using OSC

of the Kinects in a different way Tijmen thought of an option to use a MySQL database or to update XML files. We didn’t choose any of these options because it was not practical for

When we had decided to use OSC we had the following

the amount of data we had to transfer.

setup:

OSC

This setup can be divided into 2 different parts, a sending part and a receiving part. For each part a processing sketch

Another solution was to use OSC (Open Sound Control

was written. Without going too much into detail the follow-

which uses the UDP (User Datagram Protocol) protocol to

ing text will explain what these 2 parts did.

send data over a Local area network). Tijmen already had some experience with this from the midterm exhibition where an iPhone was connected with WIFI to a PC. During the midterm exhibition this was done using an iPhone App

Sending messages

Touch OSC, and a library for processing called oscp5 (8).

In the Kinect sketch we incorporated OSC to send data

OSC is also able to transfer data between multiple PC’s and

about each blob. We sent the data of each blob to the visual

therefore extremely useful when we would connect multiple

pc. Each computer would send its data with a unique tag so

PC’s with each its own Kinect to one PC. (9)

that in the receiving part on the visual pc the data could be

Advantages

ordered and mapped in the right way.

In the end of the process OSC was the right choice for us.

Example adding message:

We had no loss of data at all using the UDP protocol. With

blobMessage.add(bb.xSize());

OSC we were able to send the data of the Kinects towards multiple IP addresses. Because of this it was possible to test

Example sending message:

new visualizations during the STRP festival without having

oscP5.send(blobMessage, myRemoteLocation);

to stop the visualization on the visual PC. OSC showed us just a glimpse of what was possible, Resolume a VJ software

Example receiving message:

also has an option to receive OSC messages and thereby

int blob_Xsize =(theOscMessage.get(i+1).intValue());

many more like VVVV, MAX MSP and touch OSC. This tool is highly recommended when you want software to communi-

Receiving messages

cate with each other.

The visual PC received all the messages of the 3 computers.

The incoming data were in pure random order. For example sometimes we first received 5 messages of one pc until another pc came through. Therefore we had to filter out all the messages and combine them into one set of data that could be used for the visualization.

62 | Technical Programming

63

ArrayLists

The Kinect sent the data of each blob it sees in the order it detects these blobs,

Before we go into the visual programming a few things need to explained first.

During our process of programming we bumped into a lot of, for us new things. One of these things was ArrayLists. with an ArrayList data items can be easily added or removed from the ArrayList.

Another reason we decided to choose to work with ArrayLists instead of normal arrays was that a ArrayList is a resizable-array implementation of the Java List interface.

Because of this a ArrayList has many functions we could use to control or search for content in the ArrayList. Some of the functions we used in our processing code were size(),

a book. This could mean that in the first case person one had position one in the ArrayList. When a new person walks in the playground this new person would get position 1 and the one who had first position 1 would get position 2.

This meant that the visual object that belonged to the first blob would receive new x and y values from the second blob. This made it look as if this first blob had moved a lot.

add(), remove(), get() and shuffle(). For instance the function size() returns the length of the ArrayList and with the get() function you can request the variable number/object from a chosen position. We used the function shuffle() to create a

With one Kinect this would not be such a big problem but when you have a

random Mondrian. (10)

lot of persons and 3 Kinects you would

From received data to visuals

change position to such an extent that it

Once the data received were filtered and ready to be sent to the visualization there were several difficulties to overcome.

What will happen when someone new walks into the vision of the Kinect?

Per blob a new visual object had to be created. Each of these

only get chaos and all the visuals would would appear as random.

Therefore we had to find a way for the position of each visual object to change according to what was the nearest blob (person).

objects was assigned their own color and stored the X and Y values of the blob in an arraylist.

MinDistBlob:

To do this we wrote a function that would calculate every distance between all blobs and objects and engaged the blobs to an object which had the small-

One big problem during programming was to assign one specific visual to one person. For the amount of blobs there were on all the 3 Kinects there were x and y values stored in an ArrayList. To prevent this problem the order of the blobs in the ArrayList had to be the same as the order of the objects in the ArrayList.

est distance. The x and y of this blob would be used to update the visual object. In this way the order of the ArrayList of blobs could change without having to change the order in which the visual objects where stored in their ArrayList.

65 | Technical Programming

One reason why we chose to work with ArrayLists was that

and this in the same way that you read

Visual Programming Testing with mouse While Tijmen worked on the technical part of making the Kinect communicate with Processing some first tests were done with a computer-mouse as input. Marieke adjusted an already existing code to fit the project better, and tested this on the Philips Luminous Panels. This first code existed of “fireflies” circling around the position of the computer mouse. The speed of the mouse influenced the way the fireflies circled around the mouse. Since the Panels blurred the output considerably, the details in the original code would be faded out. A simpler version of this code was created to save processor-capacity. This code had the same effect on the panels.

Marieke also tested her midterm idea, “Painting” in a simplified “mouse”-version on the LTPs. When pointing to a color on one panel one could choose the color of their swatch. By moving your hand around you could “paint” on the panels. Immediately it became clear that the chosen colors were clashing, and that the stroke was not wide enough. These factors would have to be taken into account when continuing with this concept.

Anika started with writing a collide function for the blobs (objects perceived by the Kinect as people). In this function different things could happen when blobs would collide, for example: objects would change color when they collided. What happens in this collide function is calculating the distance between all objects; when this distance is under a certain value collision has occurred. When objects collide this function triggers another function, in which is written an action on this event.

68 | Visual Programming

Visual Programming | 69

71 | Visual Programming

70

Testing with Kinects These relatively simple codes were rewritten to react to the

switch names, and as a consequence their colors will also be

output of one Kinect. The codes reacted to one variable X

switched.

and Y value (of the mouse), but now had to react to multiple

Since your object was now always assigned to your blob,

X and Y values of different blobs. (11)Classes were needed

Marieke saw the opportunity to put her mid-term idea into

to keep the code from getting too extensive. Since Tijmen,

practice; with a small adaption to the code painting became

Anika en Marieke were not that familiar with this kind of

possible. A stroke of your color would follow you along the

programming, they asked Wouter van der Heide (2nd years

playground.

bachelor student Mathematics) for some help. Wouter helped them setting up a processing sketch in which differ-

Meanwhile Anika focused on creating random objects when

ent classes worked together in a structurized way.

there are no people within the Kinect’s sight. She added

Wouter wrote an AssignObjects class which had the func-

a random blob class to the processing file with some help

tion of adding or removing objects according to the amount

from Bram van der Sanden (3th year bachelor student Infor-

of blobs. Next to this class there was an Object class, which

matics). These random blobs are just like normal blobs but

contained the characteristics of the objects generated in the

are always present. These random blobs could collide with

AssignObjects class. Bit by bit they learned to use Object

each other and/or with the normal, “human” blobs (detected

Oriented Programming, and how they might be able to put

by Kinect). Some things that can happen when a random

the envisioned interaction scenarios into practice

blob collided with a normal blob is that the normal blob would take over the color of the random blob or the normal

A problem occurred when they found out the Kinect

blob killed the random blob.

changes names of blobs when they have a different position. More about this problem can be found in the technical

When connecting all three Kinects to the visual PC a new

programming part of this report under mindistBlob. This

challenge arrived: combining these values into one visual

had as result that when 2 blobs switch position they will also

output. Marieke altered the incoming X and Y values in such way that every Kinect had his own part of the output. For

example, blobs of Kinect 1 were drawn in the first one-third of the panel, and blobs of Kinect3 were drawn in the last one-third of the panel.

After all the LTPs were installed, Marieke could start on finishing the mapping in the processing file to a final version.

It appeared that the blobs could have higher X and Y values than expected. This meant that the resolution of the Kinect output was not as expected (was not 640 x 480).

Blobs directly underneath the Kinect were not visible on the Panels. These blobs had to be replaced to a visible area.

Streaming

content to Philips Luminous Textile Panels Philips developed software for send-

72 | Visual Programming

ing content wireless from your pc to the textile panels over a local network. Philips content manager, which was the version available at the beginning of this project, was able to stream pre-created video material to multiple panels. The setup of the panels could be adjusted as long as they are positioned in an angle of ninety degrees in relation to each other.

This was actually not enough functionality for the concept to be presented at STRP festival. This time wireless streaming of live content to multiple panels was required. During a meeting at Philips Lighting a programmer from Philips wrote this custom piece of software which we only had to adjust to our panel setup. The software made it possible to stream live content, selected from our laptop screens, to the individual textile panels.

GraphEdit connects filters to a video capture source, like a webcam or in this case the laptop screen output. The filter which sets this source to the laptop screen output is the one we developed at Philips lighting and was called ucapture.grf. This filter has to be opened in GraphEdit. To startup the streaming connection ucapture.avs needs to run in a media player.

For this we used Media Player Classic. In ucapture.avs the addresses of the panels and the total pixel mapping are defined.

Several parts of the software were updated during the design process to meet our requirements.

73

Streaming at STRP 75

When the LTPs were installed at the festival-area, together with

which does all these calcula-

Philips Marieke started to adept the previously created file for

tions.

the streaming of the panels to the mapping of the panels as it was now. A video file had to be created of the whole screen.

In the first picture on the

This was done using uCapture. From this file the pixels which

right you can see the output

were meant to be sent to the Panels had to be downscaled. Also

screen of processing, and

the blank areas between the panels had to be calculated and

in the second picture you

taken into account.

can see the outcome of the avisith file in uCapture. This would then be the file which would be send to the panels.

At the end of the avisynth file all the data of the pixels was

Since the panels draw their visual input in a

send towards a panel with its respectively ip address.

vertical way, horizontal installed panels needed to get the information of the pixels also in a vertical way. The pixels meant for these panels needed to be rotated 90degrees before they could be send to the panels.

All these calculations to

Example code rotation:

prepare the processing output for sending towards the LTPs where done in AviSynth. Marieke worked together with Floris Provoost (Philips) to write a file

clip1 = TurnRight(rclip1) Video file: 1600x1024 px Processing output: 1600x720 px

Example code sending:

Total resolution of all Panels together:

160x72 px

clip1 = SimpleSample(clip1,

Resolution of one Panel:

20x40 px

40,20,2,�192.168.0.186�, 6038, 2,1)

Designing the Interaction After the streaming was finished, Marieke could start on finishing the mapping in the processing file to a final version. It appeared that the blobs could have higher X and Y values than expected. This meant that the resolution of the Kinect output was not as expected (was not 640 x 480).

Blobs directly underneath the Kinect were not visible on the Panels. These blobs had to be replaced to a visible area.

When Tijmen was moving around in the lounge area he found out that he wanted the system to react on him spreading his arms. This could be done easily by measuring the width of the blob. Tijmen and Marieke added different functionalities to this variable. When pointing your arms to other Kinects you would create a white flash when in the playground of Kinect1, create a random Mondriaan-painting when being in playground 2 or emit lightning when in area 3. If you spreaded your arms in a different direction you would create a huge blob above you. With this big blob you could push other blobs off the panels.

Tijmen tested his Mid-term exhibition prototype too.

Using his mobile phone with accelerometer, he could change the gravity of the balls on top of the panels.

These balls changed colors on the sound of the beat.

76 | Visual Programming

78

Programming interaction during STRP During the festival Tijmen, Anika and Marieke kept programming additions to this basic interaction.

Visual Programming | 79

http://www.youtube.com/watch?v=f9kOuoorjds&list= PL825402CE33CC8B00&index=4&feature=plpp_video

80

Programming interaction during STRP Painting The ‘painting’ code was tested and adapted several times. Marieke made the brush size variable to the amount of people in the playground. More people meant a smaller brush-size. Some experiments were done with the colors. First random light colors were assigned to the blobs, but these pastel-colors did not show much contrast with each other. This had as result that your collaboration in the painting was not directly noticeable. More bright colors were tested, but some colors really clashed with each other. Some ‘ugly’ colors were filtered out, and in the end the combination of colors felt satisfactory.

Visual Programming | 81

http://www.youtube.com/watch?v=LqdCKmohGOY&list =PL825402CE33CC8B00&index=2&feature=plpp_video

Programming interaction during STRP Micheal Jackson Dorien van den Hurk (STRP) gave us the feedback that visitors of the concerts during the night needed clearer interaction. Marieke created a code in which panels would light up when someone was standing underneath that panel. When walking around panels would light up above you, always in the same color. Since this interaction method did not work that well when there are many people walking around in the area, an idea was created to split the panels into parts; when four people are underneath one panel the panel would split into four sections. A start for this code was made (a panel splitting in two sections when more than one person was underneath it). It had not been worked out more because of the complexity of the idea.

82 | Visual Programming

http://www.youtube.com/watch?v=y5EhWBHdNf0&list= PL825402CE33CC8B00&index=3&feature=plpp_video

85 | Visual Programming

84

Programming interaction during STRP Extra Layer Hands During STRP festival we saw that children always used their hands to interact with the panels. In our interactive installation nothing happened when they moved their hands. That was the reason why we decided to add an extra layer (extra blob detection area) to the interaction.

When you put your hand in the air there would appear a pulse above you on the panel; a growing ellipse with no fill and a white stroke. When there was more than one blob detected in the second blob detection area, processing would draw a line (connection) between the positions of those blobs. It would connect the position of blob 0 with blob 1, blob 1 with blob 2, etc. .

When testing this at STRP festival we noticed that this wasn’t visible enough so decided to not develop this further. (12)

http://www.youtube.com/watch?v=x8kG6yG7pUM&list =PL825402CE33CC8B00&index=7&feature=plpp_video

87

Programming interaction during STRP Adjusted Mapping for walking

86 | Visual Programming

We also noticed that because the interaction was on the

wanted to add that something would happen not directly

panels above you people didn’t notice that they actually

above you but a few meters in front of you. Tijmen made a

where causing a change in visuals. They were more perform-

program that calculated the speed in a certain direction and

ing for the people who were sitting in the lounge that they

moved the visualization accordingly to the direction of the

actually noticed that they did something. Therefore we

walking route.

Programming interaction during STRP 89

Sound for Saturdaynight We all noticed that during the concert on Saturday night the atmosphere was completely different than during the day. There were at least 4 times as many people as normal and they were there not only for all the art but especially for all the music. This music was most of the time with a very loud bass also in the area where we had our setup. Tijmen had the idea to make a visualization on the panels that would react on the sound. The program automatically detected the right sound level and create content that was created dynamically on the sound. More sound meant brighter colors and thicker lines, but it was namely the difference in sound that measured.

http://www.youtube.com/watch?v=e0cWmtvPBOs&list= 88 | Visual Programming

PL825402CE33CC8B00&index=1&feature=plpp_video

Floor plan and Technical drawings

Production of the concept and Stakeholder agreements

Max and Tom did not get full approval from STRP for our

with all the supplies we needed for the realization of the in-

When you are working with tree major parties, a lot of ar-

Because the installation would hang on a height of 3,5 me-

panel-mapping that we used to make our physical model.

stallation. This list contained all the extension cords, Ethenet

ranging is needed to let things run smoothly. When the festi-

ters we had to work with a servicing platform. We checked

So we adapted a new version into the old model. The final

cables, power cables, routers, switches, computers and

val approached there was still some missing equipment and

who was liable if something went wrong. This seemed a bit

position of the panels was set up as a unit which had a clear

sensors that made the installation into a whole. With this

due to some miscommunication there were still some am-

more complicated than we thought. Since STRP said that

division with three hot spots. We used this division to create

information we could edit the aforementioned map. First we

biguities about the division of tasks between the different

they could not be responsible for that we informed the

three separate playfields for the interaction. In this way the

did this by sketching over the original map and after that we

parties. As soon as the panels from Philips were transported

University. They referred us to the insurance agent of the

playfields were divided over the surface so we could sepa-

digitalized after on to make it easily editable and enable it to

from Berlin to Eindhoven they had to be delivered to STRP

University. Finally, after a couple of e-mails it seemed that

rate different interactions. Because we received an updated

be sent to the other parties. For this technical drawing Max

as soon as possible since there was no space to store them.

event insurance would cost us a lot of money and still gave

floor plan from STRP we were able to implement this new

used a program called Microsoft Visio 2010.

This concerns four boxes with approximately the size of two

us a high risk. We decided that the costs of the insurance

refrigerators and a weight of 200 kilos. Nor the University

had not enough benefit over the amount of risk we would

nor STRP could store these boxes in such a way that they

take with the servicing platforms.

mapping into the 2D map of the location. After we reached an agreement about the positioning we had to further elaborate this mapping with technical data. We made a list

would be safe. Philips had to store them until the first day of assembly. For the assembly neither parties had time or staff

According to the agreement we would arrange four laptops

to spare to coordinate the construction. That is why Alex and

that could stay at the festival for ten days. Because there was

Tom got instructions and material from Philips to guide this

no budget available within the project we had to arrange

process. There was still some confusion about the exact sup-

these computers within our own network. From Jeanette

plies. Vivian set up a list with what each stakeholder needed

Schoumacher, who manages promotional material of the

to provide.

faculty of Industrial Design, we could borrow two laptops, which were capable enough for what we wanted to do with

List of materials and services

it. Another two laptops were provided by the UCE-research

Philips

group of our faculty.

Eight Luminous Textile Panels (120x250)

It was essential for the remote control of the computers that

Material for hanging up the panels

we had an iPad. We could borrow two iPads from ID educa-

Power- and Ethernet cables

tion to easily turn on, access and apply changes into the

Steel cables

computers while they were on a height of 4 meters.

Router and two switches

We made a schedule for the festival so that there was always

Workshop for mounting to two students

at least one person present that could reboot the system

STRP

Pliers and clamps

Available room for a computer on the ground

Power supply

Different color carpet tiles

Possibility for testing the installation

when something went wrong.

TU/e

4 computers

Cables

2 students for the assembly

Production of the Festival | 93

95 | Production of the Festival

94

Making a model of the festival area

tion of normal Kinects and Kinects with an extra zoom lens.

Once we found out the first specifications of the sensor

Besides

Tom started to place it in different ways into the available space. He tried to cover the complete field underneath

the fact that the overlapping part would become a dif-

the map of the panels so that every spot under the panels

ficult programming problem, there was no money to buy

could be used for the interaction. It appeared that the size

at least five Kinects. With this information Tom started to

of the playfield could not fit within the mapping of the

build a physical model of the situation at STRP. In an earlier

panels without overlapping. Even not without a combina-

stadium of the process we voted for the panel mapping and he used this to create a miniature of the conditions. http://www.youtube.com/watch?v=2UJfpmQqYcw&list =PL825402CE33CC8B00&index=6&feature=plpp_video

97

Mounting the hardware

The eight panels were separated again at two sockets. Because the panels could be interconnected, meaning that the power cable from one could continue to the next, we were able to make two groups of four without a need for long cables. The Ethernet cables between the

The tools we were given were clamps, cable and all the electronics.

panels however could not be linked and had to be fed to every panel on

Gloves were needed to ensure that we could handle the panels safely.

its own.

Putting the panels up had some complication because of the size and

The second group was a mixture of electronics:

the technique with which they were hung. More can be read in de indi-

• Three Laptops that were used to collect data from the Kinect and send

vidual part of Alex Schepers

it to the visuals laptop

Electricity and network

Two electrical groups were made to ensure that if all were turned on at once, no overloading would take place: 1. The panels 2. The data network The first group consisted only and entirely out of the LTPs.

• A LAN-router • Two LAN-switches Apart from all parts in this group needing power, ethernet cables were also needed to establish a network.

Every laptop was connected through a power source which we could individually power on or off. This meant that if a laptop would for some reason fail, the rest of the system could still continue without a problem.

Should one of the groups have failed, the other would simply wait for the other group to come back online.

96 | Production of the Festival

100

STRP festival

Research and development Throughout the festival we changed and

During the STRP festival our concept is exhibited at the Light

updated the program for the interaction and

CafĂŠ. This is the perfect opportunity to collect the opinions

visualization. The presence and behavior of

and reactions of the visitors and participators on the interac-

visitors gave inspiration for improvements and

tive play.

adjustments.

The festival will be visited by art lovers, creators, designers,

For research purposes we conducted some in-

students, creative professionals, music fanatics, and many

terviews and did observations on the behavior

others who are interested in art and technology.

of the visitors at our installation. All interviews

The setting

and observations were acted out during the finalized version of the program during the

Our ceiling concept was positioned above a lounge interior.

last weekend of the festival. The functionality is

The play field was in the centre of the lounge. The rectangu-

explained at page‌ of this report. During the

lar shaped area was circumcised by half room dividers, which

research we sat at the lounge on the side of

created lounge booths at the sides of the play field. The play

the play field and took notes on our IPad.

area was crossed by all those who were looking for a place to sit down, met friends or were on their way to the next exhibit.

Diversity The program of STRP festival is very divers. During day time the exhibition is the main activity, together with workshops, readings and lectures. Throughout the week also several schools will visit the festival, so in the mornings the overall audience will exist of ten year olds. During the weekends the festival is also opened during the evenings. The planning states performances of great music artists and DJs and these will attract a lot of festival visitors. Because the Light CafĂŠ is opened during all activities, we can be assured of a large variety in visitors for our installation.

Production of the Festival | 101

Data gathering

To gather information for concept improvements, research analysis and reflections we documented our observations during our stay at the festival. Below are some results we

Observations

got from interviews and live observations. The observations were conducted according to the categories proposed in the outdoor play observation scheme for evaluating headup play (ref ).

Physical activity There appears to be a relation between the level of physical activity and the understanding of the interaction. The more movements are tried, the more functionality is discovered.

Most physical activity is shown in trying out the functionality of the system. Repetition of the movements appears to create awareness on how the system works, because the movements then show the same effect. The behavior of most visitors is actually limited to the standard behaviors which are appropriate in the setting.

Intensive physical activity What stands out is the fact that only children are showing intense physical activity. When the supervisors of the school groups have given an explanation about the panels and interaction, the children were given the opportunity to explore the interaction for themselves. There are always a couple of children waving and jumping below the panels.

This was one of the reasons that we added a second layer which could detect this activity.

102 | Production of the Festival

Non-intensive physical activity This applies for almost all the visitors. They are walking in the lounge unsuspectingly that they were causing an interaction. This interaction, moving lines and changing colors, was noticeable for the audience that enjoyed the sitting area of the lounge.

But once they saw someone causing this interaction they were more likely to copy the behavior. In the behavior of the visitors it was remarkable that they started to wave above their heads once they recognized that the system was interactive. They expected that the panels would react on this movement, which did not happen in the beginning of the festival. Later on the programmers adjusted the code so the hands could also be used in the interaction.

No physical activity If someone is standing still below the panels, a colored circle appears above his or her head but mostly this person doesn’t notice.(13)

Production of the Festival | 103

Observations Focus There seems to be a big difference in focus between the visitors who are walking underneath the installation and the ones who are sitting down. Most people who are walking under the panels only look up once, continue walking and leave the play field. While sitting visitors look at the panels as well as at other players and take more time examining the surroundings. This might have to do with the fact that the walking people have something else on their mind at that moment.

Looking at other players Both adults and children are looking at the movements made by other visitors. Movements of others that have a new interaction as result were copied by bystanders. When there are quite a lot of people this creates a certain continuity between watching and performing. People who are watching want to try the interaction. With that they are the new performers and provoke a new performers for other people in the lounge. People who understand the interaction can explain it to somebody else but they do not make it more clear. We think that the most people were too embarrassed to take the initiative to make “shameful movements�, but when we as experts of the interaction showed was possible by some extremer moves people were more likely to step out of their comfort zone. In this way they see that they can create something together.

104 | Production of the Festival

Looking at game objects Under the panels there are hardly any objects. Only some Fat Boy poufs pushed into the playfield. These poufs do not have any influence on the interaction because the Kinect only detects blobs above the shoulder. We think that this is something that the visitors understand because they did not try to get an interaction with any of the above named objects.

Looking at something out of sight, possibly part of the game When the visitors walk below the panel and like the thing that they create, they often look at other visitors for confirmation.

105

Observations Social Interaction As expected there was a lot of social interaction within the setting of a lounge. We focused on the social interaction which seemed to be related to the installation.

Notable was that standing people in the play field got instructions from other visitors sitting on the side line, which seems to origin from the fact that the sitting ones have a better overview of the situation. And the level of social interaction seems to decrease when the number of visitors increases. It might be that people feel embarrassed to share their findings or incomprehension when they are surrounded by a lot of people they don’t know.

Functional, with another player The special movements leading to erasing or coloring

Non-functional positive/neutral, with another player Players often proudly shout to other visitors what they have discovered. They also repeated their discovery.

Unintended physical contact Because people are looking up very often they sometimes bump into another person, or accidently hit someone when spreading their arms.

The differences in physical activity, focus and social interaction clearly defined two different roles in our play field; the one of active player, and the one of spectator.

require more explanation and work better individually than with more people in the playing area. These movements are not movements you make in daily life which is quite funny in itself. People make these movements often unconsciously by looking and pointing at something. Also when people are toasting, shaking hands or greeting by giving three kisses something happens. The same effect is reached when somebody with a large waste is passing underneath the sensor.

106 | Production of the Festival

Production of the Festival | 107

To find out more about the overall impression of the stakeholders about the project, we interviewed Frens Frijns (STRP Festival) and Floris Provoost (Philips) .

What did you think of the final interaction at STRP festival with respect to your expectations?

“The interaction was quite underdeveloped in the first weekend, but I saw a distinct improvement towards the second weekend. De interaction

What did you like about the STRP festival?

definitely increased, you could see a lot of people

111 | Interviews

110

starting to interact with it more. The continuous variation was at least as important as how it

“I was very happy with the number of visitors and

looked. I thought it fun to see how people didn’t

the feedback of the artist world. I was also very

always notice what was happening above them.

glad to see there was an increase in the number of

That is much more fun than when it is all crystal

in art-viewers, symposium-visitors and all the other

clear. That is the exact reason why I was of the

visitors which came during the day. In whatever

opinion that your project was far more interest-

way you look at it, the thing that makes STRP spe-

ing than the other Philips project within the same

cial are still the art pieces. And with 31000 visitors

room. At this kind of festival it is very important

we surely can’t grumble.”

that the image is much more powerful and that had happened a lot more during the second week.”

How did this project come about?

Because I thought the link with the TU/e was very

What was your general impression of the collaboration between all the parties (Philips, STRP, TUE) and everyone’s share?

interesting, especially within the playful interac-

“I can’t really evaluate that. My cooperation with

tions part, I was the entrepreneur as a manner of

Philips was limited to delivering the items neces-

speaking.”

sary. I was able to make some nice agreements

“I came in touch with Floris Provoost for the panels and I was already talking to Ben Schouten at the time. Floris preferred artists to use his panels, mostly regarding an interaction with the panels.

Did we measure up to your expectations/requirements?

and that’s it. The miscommunications, as you have stated, were not very clear on our side. You could notice that you were not familiar with this kind

“Your determination during the festival was super.

of situations and certainly not on such a location.

It was really interesting to see the cooperation

Overall it was very nice to work with you, every-

between you and Philips and how that showed

thing was agreeable. “

itself at the STRP festival. The way you were at work ensured me that it would all end well. To be honest there was some chaos and some things that were unclear at the beginning, but it turned out great at the finish. “

“In whatever way you look at it, the thing that makes STRP special are still the art pieces. And with 31000 visitors we surely can’t grumble.”

FRENS FRIJNS

FLORIS PROVOOST “I really liked the festival this year. Probably because I’ve been more involved in the construction of one of the artworks. It was very versatile in the way that they used video, a lot of good music and there were a lot of good media artists.

I thought that the line-up was of high quality and was also very inspiring. Normally you have 4 or 5 stars and the rest are pretty much in the middle but at this festival I found

Of course I had some expectations with this project. I wanted to have a report with a piece of research I could link to experiments, play and interaction. In general I find this project very successful especially through the insights we got in interaction. It was our first step into unfamiliar territory and now I want to make some next steps with that. I definitely see opportunities for further research to extend to personal

everything of quite an appealing level of quality. “

development and perceptions of a space combined with

How did this project come about?

products on which, together with the university, research

“Ad van der Meulen, someone here at Philips, wanted a collaboration with STRP. He wanted to combine certain research to exhibit something at the festival. Ad linked me to Frens Frijns who sent me to the artist/designer Edwin van der Heijden. Suddenly Ben Schouten came up with a group of students who were crazy enough to work with Philips.

Finally Ben Schouten presented you to me as my experiment team. The idea already ran for a while. At one point we supported the idea of Edwin van der Heiden and then your project came in and I found those two projects so interesting that I wanted to support them both. In your project I had a little more say. The thing I liked about your project was the interactive part. For me, playful interactions has always been something I’ve been involved with through previous collaboration with Ben Schouten and Rombout Frieling so I had enough experience with it. Your group was well suited for an experiment to expand our knowledge around a product like your group used. This also gave you a nice opportunity that you normally don’t get. On the other hand I will receive feedback that I also normally would not get.

animation and play. There are endless possibilities of such can be done. “

What did you learn from working together with different parties?

“I know that you spilled a part of your valuable time on figuring out how to deal with some technical issues, but this certainly led to solutions. Finally, you made a one year during project worthless in just two weeks. “

What did you think of the final interaction at STRP festival with respect to your expectations?

“To my mind the interaction was limited. People reacted to the system but just not enough. It did not matter for the interaction if you noticed your presence or not. Besides it was too chaotic to be clearly recognizable. People understood that the system was reacting to their presence but to me they did not feel in control of the system. I would have used less content. But it was certainly beautiful, decorative, fun and it was what I wanted, also qualitatively. The interaction was just too scantily developed. How I see it, there is too little documentation about interaction. “

113 | Interviews

What did you like about the STRP festival?

What was your general impression of the collaboration between all the parties (Philips, STRP, TUE) and everyone’s share?

“In the first place I thought it was a little bit chaotic. There were seven students around the table, each with a different objective. You are people with all different perI noticed that team division was not evaluating equally.

The way I saw it I thought that not everybody was equally involved. A reason for this could be that I did not see everybody at the HTC when there was another meeting.

Secondly, I found that in the brainstorm phase there was too little communication about the ideas that were created, partly due to the lack of time. Some ideas may now have slipped through the net. Generally I am as excited about what eventually came out as the other people involved, and this is definitely a reason to keep doing things like this. “

What would you do differently next time in relation to the collaboration? (pro’s/con’s)?

“I thought that there was too little time to properly work out and test the concepts. You could partly blame that on the panels that came available in a too late stadium of the developments. Otherwise we could do more brainstorms to gather more valuable information. If we had more time we could have used a better, more systematical approach.

Therefore we had quite a slow start because it took a while to create some clarity about the requirements. We knew what making an interaction on the ceiling was difficult but we could have paid more attention to the different options. As to the technical programming we should have taken a better look at what already exists, instead of diving into those problems. The communication was also mainly about technical problems while more attention might have been spent on the evaluation of the idea. “

115 115 | Interviews

sonalities and goals within the project. For the long term

Conclusions In this project we have developed a playful interaction with

The provided interactive installation was not a game, but it

Philips Luminous Textile Panels. This light-installation was

was more a medium to let people freely explore interactiv-

exhibited at the STRP-festival 2011.

ity. There was not more to that than exploring, so people did not make up any games. They would only explain to others

We have established a feeling of curiosity for interacting

their knowledge about the interaction. No one fully under-

with our installation through vague visuals reacting inexpli-

stood the interaction, but this gave the whole installation a

cably on the presence and movement of visitors. We gave

magical feeling, and gave the visitors the curiosity to discov-

meaning to a static light medium by enhancing the feeling

er. We were able to trigger others’ interest in our installation

of connectedness between the people and the medium in

by performing ourselves, because we noticed people would

their environment.

copy the behavior and movements of others.

Against our expectations, this relatively simple interactive installation was still interesting enough for the festival-visitors. We did not foresee the big role of the spectator. Since the spectators were more aware of the interactivity and the change the performers made to the content, which gave them an important role in initiating the explorative interactions.

Despite the limitations of the medium, being blurry, 2-dimensional, hanging on the ceiling, we came to a suitable interaction.

Conclusion and Recommendations | 119

Recommendations Future perspective

Recommendations on multiple clients

Recommendations on working in a team which consist of students from all years

The impact of the performance depended on the number of

Since we were not involved in the foundation of this project,

Working together in a group of students from all years was

useful for them to validate the knowledge they gained over

people within the playground. The more people, the more

it was not clear to us which arrangements were made and

very interesting, and very challenging at the same time.

the years. By explaining the process and argumentation for

dynamic the visuals would get. This would also influence the

more importantly if there was something lacking in the

different decisions the senior students realized how much

clearness of who was responsible for the change. The num-

agreement. During a first meeting with STRP we got their in-

The first years did not have a clue how to run a project and

ber of people on the playground for the optimal effect was

formation on the project and during a meeting with Philips

which steps are required in a design process. Working in

model it is difficult to point out what you learned since there

rather low; between 3 and 7 persons. In future perspectives,

we heard their part. But we never had a group meeting, with

a team with senior students worked out quite well. You

are no exams or books required.

the system should be more flexible in relating the interactiv-

all parties involved. This resulted for us in unclearness about

have to make sure though that the senior students do not

ity to the amount of people.

regulations in responsibilities and arrangements. For future

become the clients of the first years. Not all steps should be

For Tijmen, a B3.2 student, it was very useful to see what stu-

projects it would be preferred to have all regulations down

executed as a group, since the first years have to find their

dents do in the masters. Because you get a better overview

in black and white.

way in the education and acquire a proactive attitude. While

of the education through working together with master

keep an overview of the situation on the floor and the ceil-

the senior students should be able to go through their own,

students than through going to information sessions. We

ing at the same time. The playground was situated exactly

Our planning and internal task divisions were not communi-

individual, process as well.

think it will attract more students to our masters if they all

in-between the panels, which resulted in no direct action

cated to the different parties. Floris (PHILIPS) mentioned in

For the more experienced students working in a group is

get a glimpse like Tijmen of what is expected and done in

above you (since there was no panel directly above you).

his interview that he would have liked to be more involved

especially helpful when in the need of motivation. It was

this final part of the study.

People would try to influence the visuals on the panels by

in our process, since he was especially interested in our

standing directly underneath them, but there was no play-

ideas for the interaction with the panels. Now he had only

ground directly underneath the panels. It was interesting to

seen our end-result, and some test-results. We were not

notice the system directed the movement of people in such

aware of his expectations of the project. A next time we

a way. Although for creating awareness in the interaction

would have to ask our clients to what extent they would like

the mapping should match the direct location.

to be involved in the process.

The playground was not clear enough, since it is difficult to

they actually learned over the years. In this educational

Also we communicated with the clients through one contact person, Vivian for STRP and Tijmen for Philips. Especially Floris from Philips mentioned in his last interview that this gave him the impression of some people not being involved as much in the process as others. He would have liked to see more from everyone.

Conclusion and Recommendations | 121

Acknowledgements 124 | Conclusions and recomandations

125

“

We are grateful that this project ended in such a success, this would not have been possible without the help of all the people involved in this project. Therefore our word of thanks goes to the STRP festival, Frens Frijns, Dorien van den Hurk, Jan-Bart van der Tuuk and Marnix de Nijs, who made it possible for us to showcase our work at this fantastic festival; to the Philips Large Luminous Surfaces team who lend us their fantastic panels; to Floris Provoost for providing the panels and his help with technical implications; to Ben Schouten for making this project possible and the guidance he found time to give us; to Frans Parthesius who guided us through the process and the process of communication between all the different parties; and Bram van der Sanden, Wouter van der Heide, and Thomas Diewald for their help in programming. All these people were indispensible for this project to be a success!

“

124 | Conclusions and recomandations

Conclusions and recomandations | 125

125 | Conclusions and recomandations

124

References 1. Playful Interactions. Technische Universiteit Eindhoven: Intranet. [Online] 2011. http://w3.id.tue.nl/fileadmin/id/Educa-

2. About STRP. About STPR, context, STRP festival. [Online] 2011. http://strp.nl/en/context/about-strp/ .

3. Bring spaces alive. Bring spaces alive. [Online] 2011. http://www.lighting.philips.com/pwc_li/main/application_areas/assets/luminous-textile/product_leaflet.pdf.

4. Foundation ID. Technische Universiteit Eindhoven: Foundation. [Online] http://w3.id.tue.nl/nl/education/foundation/ .

5. Current Themes and their Spaces. Technische Universiteit Eindhoven: Current Themes & their Spaces. [Online] http:// w3.id.tue.nl/nl/education/foundation/themes/current_themes_their_spaces/.

6. Xbox 360 manuals and specifications. Xbox 360 manuals | Xbox 360 Spec. [Online] http://support.xbox.com/en-US/xbox360/manuals-specs/manual-specs.

7. Processing Library – Computer Vision – diewald_CV_kit. http://thomasdiewald.com/blog/?p=1107. [Online] September 6, 2011. http://thomasdiewald.com/blog/?p=1107.

8. UDP (User Datagram Protocol). What is UDP (User Datagram Protocol)? [Online] October 2000. http://searchsoa.techtarget.com/definition/UDP .

9. A implementation of the OSC protocol for processing. oscP5. [Online] December 19, 2011 . http://www.sojamo.de/libraries/oscP5/index.html.

10. ArrayList. ArrayList. [Online] August 30, 2010. http://www.processing.org/reference/ArrayList.html.

11. What Is a Class? What Is a Class? [Online] http://docs.oracle.com/javase/tutorial/java/concepts/class.html .

12. Processing Layers. Nootropic design. [Online] April 22, 2011 . http://nootropicdesign.com/processing-layers/.

13. Saskia Bakker, Panos Markopoulos, Yvonne de Kort. OPOS: An Observation Scheme for Evaluating. NordiCHI 2008. October 20-22, 2008, p. 3.

14. Blob Detection . Blob Detection - The Lab Book. [Online] http://www.labbookpages.co.uk/software/imgProc/blobDetection.html.

129 | References

tion_Documentation/Project_descriptions/1112-S1-Playful_Interactions-Longs.pdf.

http://dl.dropbox.com/u/4067465/DATA_

http://dl.dropbox.com/u/4067465/DATA_

PERMANENT/Receving_data_visuals.zip

PERMANENT/Sending_data.zip

http://youtu.be/wRSV0vU4VQc

Receving Data Processing

Sending Data Processing

Final SLPLLP Movie

http://dl.dropbox.com/u/4067465/DATA_

http://dl.dropbox.com/u/4067465/DATA_

PERMANENT/Uitgaven.project.xls

PERMANENT/Sketchup.zip

http://dl.dropbox.com/u/4067465/DATA_ PERMANENT/scenario%27s.html

Spendings Project

Sketchup

Scenario’s