Research Report (Es)Sense

REPORT (ES)SENSE ITERATION ROUNDJOB #1 HUBERTS

(ES)SENSE

ABSTRACT The (Es)Sense project explores the influence of perceptive qualities in objects around us. In several projects over the years, research has been done on the way people perceive their selves and the world around them. Perceptive qualities in products can make the interaction more personalized and therefor more pleasant.

SEMRE & MESRO is a research set up designed to explore the perceptual interplay between a spatial event and the intervention of a person in this.

SEMRE is an environment that creates sound based on the behavior of the visitor, MESRO tries to explore the space. The behavior of the two artifacts reflects human like behavior in order to create an interaction on human level. The research outcomes can be translated to a real world context in order to create design notions.

Abbreviations and concepts used in this report are explained in the chapter “Concepts list� at the end of the report.

3

INTRODUCTION

INDEX (Es)Sense _ Abstract Introduction _ Index _ Project description _ Vision on project _ Process Pressure Cooker _ Goal _ Pressure cooker _ Conclusion & Reflection Iteration round #1 _ Connection to Literature _ Context _ Research Question _ Two artifacts _ Form Design _ Goal first experiments _ Acting out experiments _ Test results _ Conclusion Iteration round #2 _ Abstracted behavior _ Design of artifacts _ Technology

5 _ Experiments _ Reflection Iteration round #3 _ New set up _ Perceptual crossing _ Context, Question & hypothesis _ Design of SEMRE _ Design of MESRO _ Programming _ Programming schemes _ Final set up _ Set up test _ Experiments _ Outcomes / Results _ Conclusions _ Improving parameters Conclusion _ Design opportunities _ Connection to literature _ Future steps _ Reflection _ Concepts list _ References Appendix

6

PROJECT DESCRIPTION The (Es)Sense project explores the idea of perceptive qualities in objects. It is questioned how perceptive behavior can be designed in an artifact and engage people in a meaningful reciprocal interplay with this artifact.

The project is situated in the Wearable Senses topic ‘Space’ in which we explore textile applications in our environment. Exploration and integration of textile properties with technology, through making, is highly valued in the project.

VISION ON PROJECT This research project was used as a tool to support statements that I make in my vision on design and society.

The project provided the matrix to do my research in phenomenology and in particular the way how people perceive perceptive behavior. I, as a designer and world beholder, see the world we live in as a nice place full of unexpected events. In my opinion, people should appreciate it more to look around and experience their surroundings. In this research project I want to investigate how people can be triggered to explore and what the role of the human body is in this.

The main method I use is Research through Design (RtD). For me there is great value in creating a physical hypothesis for a question and testing this hands on. I think this is important because people will be experiencing the hypothesis with their body, providing more sincere outcomes.

This means that there will be several moments during the project where people are used as reference and input for next iterations.

7 Starting with the principle of perceptual crossing (perceiving while being perceived) provides an opportunity to investigate how people will react on personified interactions. Do they think it is more appealing? And can it trigger them to start exploring?

8

PROCESS

ound #1 R n atio Research r e tI Question

Decisioning sure Cooke Pres r

Thinking

Reection

uc Introd tion

Two Artifacts Form Design

Building Bird

Making Exploring Context

Exploring

Project description

Exploring Phenomenology

Pressure Cooker Literature Goal

Vision on project

Envisioning

Conclusion

Goal First Experiments Conclusion

9

Iteration Round #3 New Setup

Test Results

n Round #2 o i t ra Ite Interaction

Design of SEMRE

Acting Out Experiments

Final Setup

Design of MESRO

Setup Test

Technology Programming

Conclusion Experiments

Experiments

Abstracted Behavior

Reflection

Outcomes/ Results Improving Parameters

Context

Diagram

Design of Artifacts

Question & Hypothesis

Design Opportunities

Connection to Literature

Perceptual Crossing Conclusions

Reflection Future Steps

PRESSURE COOKER

GOAL To get acquainted to the subject of perceptual crossing, a one-week pressure cooker was held. With a group of master students a brainstorm was done on the possible directions within the project. The emphasis was put on triggering the senses. How can we stimulate the use of certain senses by showing specific behavior? Four categories where defined; sound, light, (fleeing) movement and touch.

Each category was to be elaborated on by two students. The final goal was to reflect on the outcomes and make a list of interesting research subjects in the field of perceptual crossing.

11

12

PRESSURE COOKER The fleeing movement was based on the behavior of birds flying around.

When approached by a person, they will get scared and fly away, but eventually they might return out of curiosity. Often, this behavior is shown in flocks, transforming the movement from individual to collectivistic. There is not one mindset that decides the behavior, the behavior is a result of the whole. For this pressure cooker, the movement of one bird was taken as starting point.

A bird like object was created and mounted to the ceiling. It had the ability to go up and down. It was programmed to slowly fly around (go up and down), while scanning the environment with his sensor. When approached, the bird would fly up to get away from the approaching person. The bird would perceive whether the person was staying or leaving and adapted the behavior on this human behavior. The focus was put on the senses of the eyes, the bird had one moving “eye� to perceive the world. The involved person was also

dependent on his eyes, canceling out senses as hearing and touch. In the case the eye of the robot bird would be taken away his connection to the world would disappear and the perceptive behavior would vanish, resulting in an abortion of the interaction.

13 The bird, created during the pressure cooker, hanging from the ceiling

PICTURE

14 Model of perceptual crossing between subject and artifact with perceptive qualities action

perceiving perceptive activity of subject

sensory feedback

action

Subject

perceiving perceptive activity of object

sensory feedback

Object with perceptive qualities

CONCLUSION & REFLECTION This installation consisted of one perceptive body. To get to know more about the behavior of flocks, more bodies would have to be added in order to create a bigger perceiving body consisting of multiple birds.

In an evaluative session with the involved students and Eva Deckers it came forth that the form of the object did not match its behavior. A bird, or plane, flies horizontal, not vertical. The fact that the object did look like a bird turned this observation into a problem, the form and behavior did not match.

In order to solve this it would be good to create more ambiguity, leaving room for own interpretation. Only then, there would be space to focus on triggering the senses of a person instead of triggering his mind.

The fact that the sensor was part of the actuating body made the interaction stronger, creating a standalone object that was able to perceive and to react on it. This strong connection between sensors and actuators gives the impression that the object is perceiving and

directly reacting on his environment autonomously.

Looking at the next step to be taken, defining a research question, the behavior and movement in space appealed. Having explored this during the pressure cooker it appeared that it provided opportunities to investigate the initial research goal; investigating the role of the human body in exploring space.

15

ITERATION ROUND #1

CONNECTION TO LITERATURE To acquire more scientific background on the subject of perceptual crossing, papers were consulted. In the department of Industrial Design already quite a few projects have be done in the field of phenomenology and perceptual crossing. The (Es)Sense project is based on the PhD project of Eva Deckers. The first step in this project is to understand why one would design for perceptual crossing.

“Design for perceptive behavior, relevant to enhance the quality in interaction in the design of intelligent artifacts” (Deckers et al. 2012). Creating perceptive behavior in an artifact can aid in the personalization of the interaction one has with the artifact.

If one feels emotionally connected to an artifact, the interaction threshold will be lower.

Perception happens all the time, when one perceives another being or object. Perceptual crossing, however happens only when the perceiving one perceives that the other being or object is perceiving back.

“Perception is inherently interactive and participatory: it is a reciprocal interplay between the perceiver and the perceived” (Abram, 1996). The fact that we perceive the world is caused by the fact that our body is a thing that is able to experience. Our body can touch, smell, see, hear and taste.

17

18 For the direction indicated in chapter “Vision on project” the environment has to be taken into account. How do people perceive an environment? Applying the words of Merleau-Ponty “The body is the centre point of perception”(Merleau-Ponty, 1958), it appears that the environment, or space, is experienced through the body. Our body is our tool to perceive the world. To trigger people to start exploring, an intelligent triggering object has to be created. To give people the perception of sharing a space with an intelligent object, a consideration has to be made between two kinds of perception over time: “perceiving the other as part of environment, versus perceiving the activity of other perceiving me” (Lenay et al. 2007). This means that while exploring a space, the person will base his thoughts about the artifact on the movement of the artifact in space and the way the artifact reacts.

CONTEXT Perceptual crossing occurs at many different occasions in daily life.

It can be experienced while being in a conversation with other people, but also while interacting with an intelligent product or service. To delimit this wide term, a context was set to avoid that the research would get undirected. As stated before, the emphasis of the research would be put on exploring space.

Previously in the (Es)Sense project, Thomas van Lankveld did a project (Deckers et al. 2012). which consisted of two autonomous artificial bodies communicating with each other. The event created by the communicating bodies shaped the environment and the way people perceived this space. Building on this project, the interest arose to focus on this artificial conversation in a dynamic space. How do people react on a conversation like this and what feeling does it trigger?

An artificial conversation can trigger different reactions and changes in the behavior of people.

19 The idea described above is quite abstract. Example situations were gathered to make a reference to the real world:

_ Two people are in an intense conversation, the door opens, how does this change the conversation? _ One is studying, a person enters, how does the book react? _ Interaction with vending machine, other machine starts attracting attention. _ A dog is playing with a toy, a cat approaches, what do the dogs do? _ Two people are in an argument, third person enters the space, how does the conversation change?

20

RESEARCH QUESTION Based on the inspiration got from the pressure cooker, a first research question was formed. The research subject involves the behavior of people in a space. Second to this, perceptual crossing between three artifacts will be used as a tool to trigger the motion through space. The first version of the research question was composed as follows:

“How do people react on being the third player in an ongoing interaction between two non-human artifacts?”

The reaction of people will be influenced by the mood and behavior of the two artifacts. Based on this behavior, the person could either flee, embrace or get angry. The corresponding hypothesis for the question is:

“Dependent on the shown behavior, people will adapt their reaction and become part of the conversation”.

This hypothesis would be translated into a physical hypothesis; the test installation. With this physical hypothesis an answer to the question could be searched for.

21 Model of perceptual crossing in case of two perceptive artifacts action perceiving perceptive activity of #1 sensory feedback

action

Object #1 with perceptive qualities

Object #2 with perceptive qualities

perceiving perceptive activity of #2

of #1 ck

ep

tiv e

io n

ac t

ck

ba ed fe y

or ns se

Subject

pe

ba

ed

rc

fe

ei

y

vi

or

ng

ns

pe

se

rc

k

se

ns

or y

fe

ac tiv ity

ed ba ck

ac

n

c ba ed fe y n or tio ac

ns

se

tio

tio

ac

n

sensory feedback

22 Model of perceptual crossing in case of one perceptive space and one perceptive artifact action perceiving subject sensory feedback

action perceiving object

Subject

sensory feedback

action

perceiving object

sensory feedback action

perceiving space

sensory feedback

action

perceiving subject

sensory feedback action

perceiving space

sensory feedback

Space with perceptive qualities

Object with perceptive qualities

23 Models of perceptual crossing in different situations, involving multiple subjects and/or perceptive artifacts

24

TWO ARTIFACTS Like in the project of Thomas van Lankveld, two artifacts would form the core of the research installation.

The two artifacts would use different senses to observe each other. Also their communication means would differ, providing a ground for the objects to use different senses. One object would be limited to hearing and sending light, the other object would be limited to seeing and sending sound. In this way the objects can establish a unique conversation that allows them to perceive each other. The restrictions of the artifacts can be translated to the human body, they embody the qualities of a blind person versus the qualities of a deaf person.

For research purposes, the behavior of the artifacts has to be manageable to a certain degree. If certain parameters can be controlled, it is easier to acquire the desired research outcomes. The focus can be put on the important aspects of the research, canceling out noise data.

The main parameter for this artifact would be behavior, with a focus on mood. When changing parameters in the coding of the artifact, its behavior will change, making it able to express different moods. Comparing these moods can give insights in how people are affected by intelligent perceiving objects.

To give people the impression they are dealing with an intelligent product, certain human behavior characteristics have to be taken in mind. These notions (Deckers et al.

2012), describe certain characteristics in behavior that express human qualities. For the two artifacts, a number of these notions apply. First, “Focus the Senses�, senses and acting have to be embodied in order to look natural. Perceiving is an active activity, it is based on actions made and the feedback received. Senses will be focused in the direction of the stimuli.

This note provides the opportunity to create an object that is looking around and exploring. Next notion

25 that should be taken into account is the “Reaction to an External Event� of the artifact. When an external event (e.g. a person entering) is added the opportunity arises to create a common history with the artifact.

Knowing this, the interaction will become more personalized. It also provides the opportunity to perceive through another artifact or person.

When the artifact reacts to an external event, a third artifact or person can notice this.

26

FORM DESIGN The form of an object closely relates to its function. It shows the physical affordances of the object which can be translated in its options and limitations. For the two artifacts, the form would be of importance to show which object has which role in the conversation. The hearing object has to communicate that it can not see or feel, but only sense sound. For the other object goes the opposite, it has to communicate that it can only perceive by seeing.

Since both objects have an explorative nature eventually, this also has to be communicated by the form and behavior. By letting them ‘look’ around to explore this quality can be emphasized. However, this way of communicating behavior has to be grounded in tests.

GOAL FIRST EXPERIMENTS To prove the first statements made, experiments with real people would be needed. The most important subject to find evidence for is the fact how people perceive an artificial conversation. Do people regard two artificial objects able to have a conversation? The output of this experiment provides information for the following iteration round.

Besides the perception of an artificial conversation, information can be gained about the behavior of people in a test environment. The way they move through the room and how they use their body can provide inspiration for the further specification of the process. When people stay static or motionless, the focus should be put on triggering people to start moving around. Or, it could be used as inspiration to make the objects more dynamic and start exploring instead of the person. People that move without paying attention to the objects can indicate that the space itself is too revulsive.

27

28

FIRST EXPERIMENTS The emphasis of this (Es)Sense research project is put on the quality of bodily skills. It is questioned how bodily skills can help people in exploring space. The body is the tool to explore the world. Based on this notion, the set up of the first experiment session was slightly changed. Instead of building two artifacts embodying human qualities, the two artifacts would be acted out by two human beings. This would provide an interesting view on the bodily characteristics of the future artificial objects. The way people behave as an artifact with limitations provides insights in how to create human like behavior in the artificial objects.

The experiment set up would approach a lab setup. Taking place in a neutral space, the focus would be pulled to the conversations going on.

During the experiment, one person would be blindfolded, making him blind and only able to hear. The person would be given a flashlight to use as a communication mean. While the other person would be made deaf, by

putting noise on his ears. This person is provided with a synthesizer to send sound. The interaction possibilities that arise give room for a specified interaction. Both test persons would be allowed to turn around their axes, but not to walk, giving them the affordance of an object that is fixed to its place but with the ability to ‘look’ around.

In the first stage of the experiment the two persons have to try to establish a conversation, using their options and limitations. In the second stage, a third person would enter the test room.

This person gets the assignment to interfere in the conversation that is going on between the two human artifacts. This assignment is given to trigger people to explore what is going on. After this stage, a questionnaire is used to interview the three persons as a group. The two human artifacts are asked how they did establish a conversation and what feeling this evoked. Following this, the third participant is asked to give his opinion on the conversation and the degree to which he felt involved.

29 Setup of acting out sessions, with one deaf person, one blind person and a subject initial conversation

of #1 ac tiv ity

pe ck

fe or y ns

se

rc

ba ed

fe

ei

vi

y or

ng

ns pe

ck

rc

ep

tiv e

or ns se

se

Subject

ed ba

n

tio

ac ck ba ed fe y n or tio ac

ns

se

y fe ed ac tio ba n ck

Blind artifact ac tio n

Deaf artifact

30 Acting out session with three students to investigate the behavior of artifacts and subject

PICTURE

ACTING OUT EXPERIMENTS For this first round of experiments, a total of seven groups of three people were interviewed (Appendix A & B). People were selected based on their knowledge of the subject of perceptual crossing. People knowing more about the subject were placed in the position of human artifact. People fresh to the subject were placed either in the role of human artifact or of human being.

In the behavior of the human artifacts a clear division between the deaf and blind was seen. The blind person showed a more wait-and-see approach. Without external input, the blind person did hardly ever start to explore or to communicate. The deaf person took more initiative in general, producing sound and trying to get the attention of the blind person.

Most blind participants reacted on this by flashing their light. When both persons got used to their capabilities they started experimenting by creating patterns or triggering each other to look around.

The third person was asked to enter the room silently so the conversation would not roughly be distorted. After entering most participants tried to distort the communication line by standing in between the artifacts.

If this did not work they started observing what was going on. What could be seen was that people started moving around, spreading their attention between the two artifacts.

To interfere they tried to seduce one of the objects by imitating the communication line from the other object. This led to confusion for both objects. Eventually this could lead to a situation where the blind one perceived the third person through the deaf one. This happens when the deaf started to follow the third person and the blind person started following the change in sound production.

31

32

TEST RESULTS In the experiment, a division can be made between the people representing an artifact and the people that symbolize a real human being. Those who represented an artifact indicated that they were able to establish a conversation in which they could communicate with the other human artifact. The intensity of this conversation varied. One couple was able to create patterns based on the light and sound they both created.

They indicated that they perceived each other and had the feeling that they were communicating. A next couple indicated that they felt the possibility to communicate but struggled with the way this was done.

Five of seven couples indicated that the light (the blind object) followed the sound (the deaf object). The other two couples stated that the sound was following the light. This can be attributed to the fact that the flashlight gave more affordance to give direction with the light beam.

The second group, the people that were assigned to interfere, experienced diverse situations. When entering the

room two out of seven participants had difficulties in recognizing a form of conversation. The other five persons indicated that they perceived a conversation between the human artifacts. The way to interrupt was difficult to determine for participants.

Three of seven participants interfered physically, pushing and trying to turn the subjects. This led to a disturbance in the conversation between the artifacts and did not benefit a possible conversation between the person and an artifact. Three of the test persons started talking to the blind object in order to start a conversation and direct the movements of the light.

In all three cases the test person did succeed in this. Only in one case the deaf person started following the new movement of the light. The seventh person communicated with the deaf person by hand gestures, trying to let him direct the sound in another direction. The blind person reacted on this change of direction by starting shining around with the flashlight in an attempt to find the source of the sound.

CONCLUSION People feel uncomfortable when entering a room where they don’t know exactly what is going on. This feeling of uncertainty withholds them of exploring the scene at first. When people get to know what is happening they become more free in their attempts to influence and explore the event. The moment people get the feeling that they can apply influence to the conversation marques the moment people become part of the conversation. This moment should take place early in the process.

Providing people the ability to act as an artifact made it possible to investigate how a non-human artifact could move in this situation. This information provides a starting point for the design of the non-human artifacts. People exhibit explorative behavior in order to establish a conversation.

33

ITERATION ROUND #2

ABSTRACTED BEHAVIOR Based on the shown behavior of the people taking the role of human artifact the behavior of the artifact was designed. For each artifact a default behavior derived from the acting out experiments. For this, the Embodynamics framework was used (Klooster, 2003).

The blind object embodied an exploring but reactive nature. It reacts on the surroundings, purely based on the sounds it hears. It has a wait-andsee attitude. The way it reacts asks for reaction of an opponent. The turning movements it makes are unsure and staccato. The uncontrolled motions show a way of uncertainty.

The deaf object exudes confidence in its behavior. It is the initiator of the conversation. The deaf one tries to attract the attention of the blind one and directs the conversation initially.

Its movements and sound express intensity. In the course of time his patters become more uniform but fragmented.

35

36

DESIGN OF ARTIFACTS In order to create a clear connection between the object and the behavior, the form has to be well thought through. The form has to fit the function and behavior of the artifact. This makes the interaction people can have more directed and gives people an indication of the possibilities of the artifact.

Both objects are based on a high standing form like a rounded cylinder or extracted cube. These basic starting forms give the opportunity to knead the form in a way it fits the function.

In the base of the form a turning point is placed, giving the artifacts the possibility to turn around and ‘look’ around. The turning point is placed in the base so that the whole body can turn, including the sensors and actuators, allowing the artifact to focus the senses (Deckers et al. 2012).

The blind object is based on a rounded cylinder, this to emphasize on the smooth motions and shy character of the artifact. It has to have a clear light emitting point and a part

that is responsible for the hearing. The first sketches show a fluent form with a big hole for the light and a smaller on top for the microphone. For quick building purposes, this form was translated to a laser cut ready form.

The round forms were preserved to keep the expression of the artifact.

Since this object emits light and receives sound, it is called SELR. The deaf object is based on a tapered extracted cube, this to emphasize the dominant role of the object. Like the blind object, is has to have a part to emit sound and a part where light is received. In the sketching stage, this was translated to a high straight form with a speaker in the upper part and a light sensor in the lower part. The MDF laser cut version replicates this and maintains the form.

Since this object emits sound and receives light, it is called LESR.

37 Model of interaction between SELR and LESR

sends light signals perceiving perceptive activity of SELR receives sound signals receives light signals perceiving perceptive activity of LESR

SELR

sends sound signals

LESR

38 SELR and LESR, the two artifacts designed to communicate by sound and light

39 Model of interaction between SELR, LESR and subject action perceiving perceptive activity SELR sensory feedback action perceiving perceptive activity of LESR

SELR

LESR

ck

of #1 ac tiv ity tiv e fe

ed ba

ck

ep or y

ns se

rc

Subject

pe

ba ed

fe

ei

vi

y or

ng

ns

pe

se

rc

se

ns

or

y fe ed ac tio ba n ck

n

ck ba ed fe y n or tio ac

ns

se

tio

ac

ac tio n

sensory feedback

40

TECHNOLOGY Both artifacts have an own Arduino board to make them operate independent. They communicate analogue, based on the light and sound they emit. To get the behavior clear, flow charts were made, showing the different steps in the behavior.

Based on these charts the two artifacts were programmed and different actions were assigned to different events (Appendix C & D). To give SELR and LESR extra capabilities to perceive the world, an ultra sensor was added.

This made it possible for them to notice if someone was approaching.

The ultra sonic sensor on LESR was not used in the end, since this object is supposed to be blind and reactive on SELR.

During the programming of the artifacts a problem occurred. The signals that both artifacts did send were not strong enough to trigger the sensor on the other artifact.

Particularly the synthesized sounds were not strong enough. Due to this, the two artifacts had to stand at a short distance of each other in a completely empty and silent room. Even then, the signals were too soft and low.

In order to still be able to do tests, the artifacts were given more independent behavior. In this way it looks to unprepared participants that something is going on. The objects were still able to sense with their given sensors, but the experiment was not dependent on this anymore.

PROGRAMMING SCHEMES initiate sound

41

reaction with sound

start conversation

exploring movement

move around

no feedback external movement subject

bleep

SELR

start conversation

initiate light

reaction with light

start conversation

exploring movement

move around

no feedback external movement subject

ash

LESR

start conversation

42

EXPERIMENTS During the theme day of Wearable Senses, a short experiment was done.

The set up was similar to the setup of the first experiments, only this time, the human artifacts were replaced

by SELR and LESR. People were asked to observe the situation and try to interfere with this after a while. The fact that the two artifacts did not really react on each other distorted the experience. People were perceiving this after a while and this made it hard for them to observe an ongoing conversation. After three sessions the experiments were aborted. The conclusion was that to create a perceiving environment, the artifacts had to be able to perceive each other.

Artificial independent behavior shows no perceptive qualities.

43 Setup of second round of experiments, involving a conversation between SELR and LESR and a subject initial conversation

LESR 1 of #

ck

ity

ba e

n

tiv

tio

ac

ck

ck ba ed fe

y or ns se

Subject

pe

ba

ed

fe

rc ei

y

vi n

or

g

ns

pe

se

rc ep

k

se

ns

or

y

fe

ac tiv

ed

n

c ba ed fe y n or tio ac

ns

se

tio

ac

ac tio n

SELR

44

REFLECTION Creating SELR and LESR gave insights in how perceptive qualities can be translated to a artificial body.

An artificial body has no direct connection to another artificial body.

In order to create this connection, the objects have to relate closely, in shape and behavior. If this is established, the conversation becomes more meaningful and people can try to interfere. To establish perceptive behavior, and finally perceptual crossing, the Design Notions deriving from the PeP+ research project (Deckers, 2012) have to be used as guiding line.

For the next iteration round, the role of the two artifacts has to be evaluated. The relation they had up till now is one on one. However, the behavior of SELR provides opportunities to create a more hierarchic interaction where LESR follows the directions of SELR.

45 Performing the acting out experiment during theme day as introduction to the SELR and LESR experiment

ITERATION ROUND #3

NEW SET UP Based on the outcomes of the second iteration the direction of the project was evaluated. When talking about exploring space, space itself is the most important parameter.

The space influences how people behave and what they will perceive.

“The ‘sense of space’ is constituted by multimode sensory stimulation changing according to the organism’s movement relative to its environment” (Schonhammer, 2004). To investigate this value of space, artifact SELR was transformed to a space. This space would perceive the entering person and the artifact. In reaction on this, sound would be created, creating an ever present soundscape. Within the space a reincarnation of LESR would be placed. This artifact explores space based on the soundscapes, trying to persuade the person to explore the space using his bodily skills. The space will be called SEMRE, since it emits sound and receives movement. The artifact will be called MESRO from now on, this due to the fact that it emits movement (and light) and receives sounds.

47

48

PERCEPTUAL CROSSING This new set up means that the interaction structures changes somewhat.

SEMRE creates in cooperation with MESRO an event that is ongoing. The soundscape is created based on movement and is kept going due to the fact that MESRO reacts on the sound with movement.

SEMRE seduces MESRO to move and explore. Since MESRO is still blind, the artifact can only identify the entering person based on the sound it ‘hears’ created by SEMRE.

49 Model showing perceptive event action

EVENT

perceiving perceptive activity of SEMRE sensory feedback action perceiving perceptive activity of MESRO

MESRO 1 of #

ck e

n

tiv

tio

ck

ck ed fe

y or ns se

Subject

pe

ba ed

fe

rc ei

vi n

y or

g

ns

pe

se

rc ep

ac

se

ns

or y

fe

ac tiv

ity

ed ba

n

ck ba ed fe y n or tio ac

ns

se

tio

ac

ac tio n

sensory feedback

ba

SEMRE

50 model of communication between SEMRE, MESRO and subject

Subject

MESRO

SEMRE

CONTEXT, QUESTION & HYPOTHESIS The real-world context this research set up can be seen in is slightly more specified then during the acting out experiments. To make people discover new details in their environment, a for them not completely known environment is needed. Preferably this is a public space that is often visited by people.

This can be a railway station (hall versus ticket machine), public square (square versus lamp post) or the main hall of a public building (hall versus stairs).

SEMRE represents the public space that is asking to be explored. MESRO represents an artifact that aids SEMRE in this and tries to seduce visitors to a personalized interaction.

The implementation gives opportunities for for example signing in public space, the position of ticket machines or the location of bus stops.

The results of this research can be used to create opportunities in productspace interaction design.

The set up of space versus exploring artifact brings a new research question forth:

“Can an adaptive object trigger people to explore a reactive environment?”

The physical hypothesis for this is worked out in SEMRE and MESRO, those two artifacts form the research installation that tries to answer this question. The textual hypothesis is:

“Create a guide that adapts to the behavior of people and the environment and directs people through a space.”

51

52

DESIGN OF SEMRE SEMRE is the evolved version of SELR, putting it in the position of conversation trigger. SELR was the object that triggered a conversation, asking from LESR to react and follow.

SEMRE embodies a larger entity. In fact SEMRE inhabits the whole space with a body of sound. It has the ability to virtually move through the entire space. It can be seen as a sounds entity floating in the air and flying around.

This sound body gets attracted by the visitor and starts following him, creating a sound based ghost of the visitor. SEMRE will adapt his behavior based on the activity of the visitor. If the visitor stands still for a longer time, SEMRE will eventually lose interest and start conversing with MESRO. The physical embodiment of SEMRE consists of four speakers in this research setup. In every corner a speaker is placed, creating a surround setup. By panning the sound between the four speakers, the virtual location of the sound changes. The four speakers on itself create a space, the space between the four speakers. This does

mean that there is no real need for a physical space consisting of four walls.

It can also be a part of a public square or a part of railway platform.

The appearance of the speakers is based on the forms of SELR. The straight form represent the dominant character of SEMRE. On the other hand is the form created in a way that it can easily merge with a public space where a lot of people pass. The face like form is designed to be put on head height to merge between peoples heads.

53 Speaker that provides the sound for the body of SEMRE, a total of four speakers is mounted in the corners

54

DESIGN OF MESRO MESRO, the explorative artifact is evolved from LESR. The emphasis is put on the movement that it uses as tool of expression. MESRO is the one following the directions of SEMRE, in order to explore the environment.

MESRO tries to attract the attention of the visitor by searching for it and asking to follow his movements. The artifact will adapt its behavior to the soundscapes that are produced by SEMRE based on the activity of the visitor. Since SEMRE perceives the visitor, MESRO perceives that SEMRE perceives the visitor, the visitor on his turn perceives MESRO perceiving him indirectly. This leads to an indirect way of perceptual crossing.

The way MESRO moves around is based on the Embodynamics Framework (Klooster, 2003). The movement while searching is stretching and intensive. In a matter of space this means that the movements goes through space, but not per se directed. Over time, the movement will accelerate. The movement stops when the visitor is perceived. To seduce

the visitor, the movement becomes more directed and strong, releasing force in the desired direction.

Physically, MESRO changed significantly pertaining to LESR. MESRO is a long arm that has the affordance to reach and look around.

It can indicate direction and has the freedom to look around. It is placed on a pedestal, raising it from the ground, and making it more of notable player in the interaction. At the end of the arm, a light is placed. This light is used

by MESRO to indicate direction and as an extra communication tool towards the visitor.

55 MESRO moving around while exploring the environment in the test lab

56

PROGRAMMING In order to create the behavior of the artifacts, they had to be equipped with a degree of perceptive behavior.

To solve the problem that showed up between LESR and SELR, a data wire was put between the two artifacts.

In this way SEMRE could directly communicate the position of the person to MESRO. The test person would not perceive this directly since he or she will expect MESRO to sense and act autonomous. This solution was discussed with Eva Deckers, she agreed on the fact that the influence of data wires on the test subject would be negligible (Appendix E & F).

SEMRE had to be able to emit sound based on the movement and position of the person. For this Max MSP was used. In Max MSP it is relatively easy to synthesize soundscapes based on input from a sensor connected with Arduino. To define the space, a base sound was created. This low pitch tone waved through the space like some kind of hearable fog. To embody the sound body, a higher pitch sound was added that moved towards the position of the person.

This sound was influenced by the distance people had with respect to a speaker.

57 MESRO had to be provided with explorative behavior. In idle mode, MESRO had to explore the space, while communicating with SEMRE. For this, different sets of random explorative behavior were programmed on the Arduino board.

In idle mode, SEMRE moves slowly in all directions or investigates one particular direction if SEMRE indicates something interesting. As soon as a person enters the room, the sound body will start following this person and send data to MESRO, telling that something entered. MESRO will adapt its behavior on this and will start exploring that part of the room. For this action, MESRO uses the same data that SEMRE used to determine the position of the person. The servos in MESRO’s body will turn it towards the right position with a bit of tolerance to make it look more natural. As long as SEMRE keeps providing data, MESRO will keep looking and moving. This influences the behavior of the person, eventually influencing SEMRE.

58

FINAL SET UP The final research setup for the experiments was built in a separate room. This was done to cancel out the distorting influences of external events. MESRO was placed in the middle of the room on a pedestal. In the four corners, the speakers of SEMRE were placed, creating a perceptive space as big as the room. At one side of the room a camera was placed to tape the experiments.

59 Setup of final experiments in test lab with SEMRE and MESRO and subject Speakers SEMRE

s

Sound body

SEMRE

a

re

MESRO

at

initial conversation

Subject

i w

D

60

SET UP TEST The experiment setup was based on the Lab, Field, Gallery approach by Koskinen (Koskinen et al. 2008). A lab setting was created as described before to be able to condition settings. For to the point research this is preferable above a field or gallery setup. Like in previous (Es)Sense experiments, the setup was kept clean and the used physical hypotheses was designed frugal (Deckers et al. 2012). This makes it easier to notice differences in the behavior of test persons.

For evaluating purposes, a questionnaire was created (Appendix G). The information that was supposed to be retrieved from this questionnaire was qualitative, people are asked to explain their own feelings. In order to get the essential information, the questionnaire was formed according to the laddering technique. This technique enables the establishment of a hierarchy of events (Corbridge et al. 1994). With help of open questions people were triggered to give the essences of their feelings during the experiment.

The questionnaire focused on two points in the experiment. First a list of questions was used to explore the feeling people had during the experiment while they where not told what was supposed to happen.

They were given the assignment to freely explore the situation and try to understand what was happening.

When they considered they were ready, the questions were asked. After this list, the test person was told what the purpose of SEMRE and MESRO was. People got the opportunity to experience the research installation for a second time after which a short evaluating question list was used. This second sessions was meant to test if people did understand the events in general and to compare if their behavior did change to the situation where they did not know what was going to happen.

61 For this experiment the following hypotheses were tested:

1. People will understand that MESRO follows the sound body created by their own behavior 2. During the second session people will understand the setup and feel more freedom to play around

62

EXPERIMENTS To test the setup of the experiments, a pilot test was conducted. Based on this test the questionnaire was specified. After that, the experiments were conducted with a total of seven participants.

Participants where selected on their knowledge of the subject of perceptual crossing. This resulted in test persons mainly from the Bachelor and from outside the department.

During the experiments, participants were observed and recorded. Their movements were tracked in order to see how they moved through space and if this changed over time. Most participants walked around all the time, taking breaks for observing the situation.

During the questionnaire the emphasis was put on getting to know if people did understand the situation and how they perceived it.

CHAPTER Pitchfork shoreditch leggings, american apparel Austin church-key terry richardson fashion axe synth mumblecore sriracha. Cray american apparel wolf raw denim, dreamcatcher blue bottle cosby sweater mixtape butcher ethnic. Wolf gastropub fap, tumblr american apparel +1 chambray art party. Flannel chillwave odd future, literally stumptown high life iphone brunch wolf banksy lo-fi cliche vice hashtag. Twee small batch ennui, thundercats irony skateboard vegan wes anderson. Meh disrupt portland, polaroid vice ugh brunch PBR trust fund. Tumblr iphone postironic organic deep v fixie lomo, synth 90’s ethnic irony fashion axe hella.

Church-key letterpress leggings, bespoke VHS fap you probably haven’t heard of them craft beer.

Vinyl flexitarian fap wolf ennui, banksy before they sold out hashtag viral mlkshk photo booth trust fund meh deep v. Tofu thundercats polaroid, tattooed jean shorts vegan sartorial wayfarers terry richardson cosby sweater mcsweeney’s art party

63 fingerstache.

Squid disrupt Experiment with SEMRE and brunch MESRO small batch,the iphone wayfarers try-hard to validate perceptive behavior of literally synth narwhal plaid truffaut. the installation You probably haven’t heard of them master cleanse PBR, skateboard organic whatever meh stumptown leggings pickled lomo banjo before they sold out street art. Raw denim actually pitchfork blog cred. Bespoke ethical master cleanse, thundercats tumblr food truck aesthetic small batch occupy.

Wes anderson chambray authentic, tumblr fashion axe cliche viral forage literally bushwick biodiesel.

Williamsburg tattooed 90’s plaid odd future. Selfies pour-over ethnic, direct trade readymade fap vice swag VHS terry richardson Austin forage echo park cardigan. Aesthetic hoodie yr swag trust fund. 3 wolf moon post-ironic four loko cliche tumblr letterpress. Chillwave mustache synth blue bottle, jean shorts forage pourover selfies. Aesthetic mcsweeney’s next level, meh mustache sustainable high life YOLO readymade fap salvia.

64

OUTCOMES / RESULTS People that did understand the situation had a more positive feeling about it (Appendix H & I).

People who did not understand the situation felt neglected and did not enjoy the situation. This indicates that a better perceptual connection provides a smoother interaction. Their experience in the second session was shorter than people who did not understand it the first time.

The participants that had difficulties with understanding the situation indicated that this was due to the unclear connection between SEMRE and MESRO. This can be assigned to the fact that MESRO did not always have a one-on-one reaction on the position of people.

If people moved fast through the space, MESRO got delayed and kept searching in just-left corners.

Those who discovered the meaning of the event on their own had a playful interaction with MESRO and used SEMRE to guide MESRO and to be guided. For them the event provided an opportunity to have a perceptive conversation with MESRO, while SEMRE being the intermediator.

Most participants got the feeling that both SEMRE and MESRO were perceiving them eventually. People interpreted the behavior of MESRO as recognizing, even when MESRO was not following the directions of SEMRE. This shows that the suggestion that an artifact is reactive can adapt in creating a relation between person and artifact.

65 The direction giving movements MESRO made when a person was perceived were not understood by all people. The participants that did understand it had the feeling that they were send back and forth between two corners. Notable, they did perceive the moving arm as an object that could point direction.

The walking patterns of the people indicated the degree to which they said they understood the situation.

Understanding people took longer rests to lure MESRO. Participants who did not understand the situation stayed longer in the room and walked around more.

66

CONCLUSIONS From the SEMRE and MESRO experiment a set of conclusions can be made. When looking at the corresponding hypotheses the following can be concluded.

At the beginning of this iteration round a hypothesis for the project was composed, based on the previous iteration rounds and the research question.

1. Participants did understand that MESRO reacts on the sound created my SEMRE. Not all participants realized this sound moved like an entity through the room. To MESRO characteristics were attributed like exploring and curious. SEMRE was indicated as dominant and loud.

“Create a guide that adapts to the behavior of people and the environment and directs people through a space.�

2. Participants indicated that they felt it was easier to work with SEMRE and MESRO when they did understand the principle. They perceived particularly the behavior of MESRO as more natural.

Concluding on this we can state that MESRO did adapt to the behavior of people, its movement was focused on the person. The longer a person stood in one corner, the more effort MESRO would undertake to lure this person to another position. The way MESRO tried to direct people and trigger their urge to explore was felt less by people. Some did understand that MESRO wanted something, but the overall meaning of this fact was missed by most people.

67 Person moving through the installation while being huided by SEMRE and MESRO

68 IMPROVING PARAMETERS SEMRE and MESRO can be seen as a start in creating a setup that contains a perceptive reactive environment and a perceptive body. Regarding this test set up, several parameters could be improved in order to create a clearer interaction that will be easier to understand for people. _ Meaning of movement.

The movement of MESRO is now based on the pointing movements people made in the first acting out experiments. The meaning of this can be improved by testing this specific part of the installation. Testing with different sets of movement can lead to a more concrete way of moving.

Parameters that can be changed include speed, direction, delay and smoothness. _ Meaning of sound. In the current setup the sound is kept minimalistic and produced by only 4 speakers. To increase the perception of a body of sound, the programming of SEMRE has to be done differently. Currently the location of sound is based on the

distance a person has to a speaker.

This could be improved by creating a virtual X/Y grid over the space. In this way a specific location can be assigned to the sound body and all speakers will aid in creating this sound at this place. The sound itself can also be changed from this relatively bleak high pitched sound to a more characteristic repleted sound. Eva Deckers commented during the final exhibition that this influencing sound can be interesting enough to investigate more by emphasizing on making it more extreme to have a bigger impact. _ Integration of MESRE in environment. Now, the speakers of MESRE are clearly visible on the wall.

In questionnaires people referred to these as heads of talking boxes. To eliminate this issue, the boxes have to be integrated in the environment.

This can be done by creating speakers that fit better in the environment (e.g. hid in the corners) or by changing the environment (e.g. put same shape as speakers everywhere).

69 _ Research lab. The room that was now use as research lab was not completely blank. One wall was open, another wall contained blackened windows. On the floor cables were put for power and data transmission.

If the room would be transformed in a completely white room (or in another way so that it fits the experiment like stated before), there would be less distraction from the truly essential objectives of the experiment.

To get a more reliable outcome of the next experiment, two things have to be changed. First a control group is needed. This group is told before hand what the idea is of the setup. The rest of the experiment will be the same as for the people that do not know what the idea of the setup is. In this way it can be compared if it is essentially to know how the installation works in order to understand it. Secondly, this time, not everything was plotted.

Movements and interviews with people were recorded. Experiment specific things like sensor data and random generated directions were

not recorded and therefor impossible to evaluate.Particularly the sensor data can be of significance when judging the behavior of people.

CONCLUSION

DESIGN OPPORTUNITIES Where the project was started as a research project, a practical translation was extracted as input to future design projects. The philosophy of phenomenology is a topic of several design research projects. The translation to the creation products seems to be missing. In my opinion, the principle of perceptual crossing can play a catalyzing role in the development of subject-artifactspace interaction. A product that perceives his user and adapts his service, appearance or function to this can provide great opportunities in (public)space design.

This project emphasized on the relation between a person and a perceptive space and the role a perceptive artifact has in this. The conclusions of this project can be translated in design notions for further projects.

_ Perceiving position. In a space, our position is essential to the way we perceive a space. Changing this position or changing our routes changes the perception of space (Huberts, 2012). To create a rich subject-artifact-space interaction, both the artifact and space have to be able to perceive the position of the subject. _ Event based behavior. An external event will enrich the common history a person can obtain with an artifact (Deckers, 2012). The interference of a subject works as an external event, creating unique shared history between subject, artifact and space. This shared history amplifies the sense of interaction.

71

72 _ Explorative suggestions. In this project I experienced that the artifact can function as a catalyst in the explorative behavior of the subject.

The connection between subject, artifact and space delivers takes the behavior of artifact and space away from two separate following entities.

One event is created that is subject to changes by external events. _ Inviting behavior. Based on the directions of space and external events, the artifact displays behavior that aims at inviting for interaction.

The extent to which the artifact applies this determines the feeling people get. This can vary from not appealing to embracing.

CONNECTION TO LITERATURE To explore a space, one needs his body. “It’s a truism that the best way to experience a space is by walking” (Schonhammer, 2004). Seducing people to start moving around will show them more of the space they are in. Adding a perceptive body in order to aid, creates an event in which subject and artifact share their intentionality; explore space. “In order to realize mechanisms of shared intentionality, we should design systems which are sensitive to perceptual crossing” (Marti, 2010). If the artifact in the shared space is able to perceive both subject and space a perceptual interplay can be established. This happens only when one is able to perceive the other’s ability to perceive (Deckers et al. 2010). If this connection is not possible, the establishment of a shared intentionality is distorted.

73

74

FUTURE STEPS As stated before, perceptual crossing provides interesting features to intensify the human-artifact-(space) relation. In my Final Master Project I want to focus on this thought. Main goal will be to create a marketable product or service based on the principles of phenomenology and perceptual crossing.

I hope to use the design notions I converged as conclusion on this research project as a starting point for my FMP. Elaborating on these, I want to create a set of tools that aids me in designing for perceptual crossing in products. This will be part of my FMP proposal.

The context for this is still very broad (e.g. public space, interactive art, domestic environment) but will be defined in the first weeks of the project. In the process the emphasis will be put on creating a product that embody a personalized interaction setup based on the perceptive qualities of the artifact and the bodily skills of the user.

REFLECTION I chose the (Es)Sense project since it is a fitting opportunity to ground statements I make in my vision on design and society. Besides that, it provides a framework to learn how to use the Research through Design methodology.

A clear theoretical framework, phenomenology, was provided at the start of the project. Starting to think from the human senses proved to be a good manner to forge a research question and hypothesis. People have to be able to connect to a product with their senses but also to escape these.

In my opinion, this means there has to be some room for ambiguity. When there was no room for ambiguity, people indicated the interaction was less interesting, ambiguity leaves room for own perception. To support this I learned that a form has to be inherent to its function, including its sensors and actuators. In this way an independent body is created that can interact on an equal level with a person.

75 My goal was to go through a highly iterative process according to the Research through Design method. I experienced that short quick iterations can mean a lot to develop certain parts of the concept. At the start of an iteration I configured a question and a corresponding hypothesis. A physical version of this hypothesis (model), is a good way to test the hypothesis and find answers on the question. This way of working taught me to start building and learn to explore while building the model.

Scientific research was something I hardly got in touch with before. In the (Es)Sense project I grasped the opportunity to use scientific papers as grounding and inspiration for decisions. I noticed this aids well to substantiate my design. It also forces you to check sources and references which can lead to new information for your project.

76 In my opinion, the human factor can not be ignored in research.

Therefor I wanted to involve people in my experiments, and avoid a paper based research approach. In an early stadium I applied acting out to investigate human behavior and movement. For me this proved to be a useful technique to gain inspiration and information about the bodily skills and how these can be translated to an artificial body. In discussion with Rene Ahn we agreed that this way of human like movement is important to reach the desired level of affection between subject and artifact.

I learned to converge what I want to get from a test to make a test continue smoothly. Looking for examples and answering “why?� questions helps in this. During tests it is wise to record everything so it can be analyzed afterwards. This aids in abstracting the answer to the research question. To get to the essence of what people experienced a good questionnaire is needed. Applying the laddering technique works well

in this since people are forced to give their most elementary thought on the subject.

While prototyping I tried to combine the learning of new skills with achieving result. I learned to use Max MSP in a short time by focussing on the skills I needed. Due to this I now know the basics of the program an can easily expand my knowledge in the future. Concerning physical prototyping I learned to combine sensors and actuators in one body.

This taught me to use electronics on a small scale and program in a compact way, to spare micro processors.

77 Overall, I can state that the (Es) Sense research project made me into a designer that is able to substantiate his decisions in a scientific way. By using provided theories and the right papers, inspiration and direction can be gained. In the future I will try to use this Research through Design methodology to establish a result based on hands on research. As mentioned I want to continue with the subject of phenomenology in my M2.1 project. My goal will be to integrate the principles of phenomenology and perceptual crossing in a marketready product.

78

CONCEPTS LIST LESR: Light Emitting Sound Receiving. Blind object created in second iteration round

SELR: Sound Emitting Light Receiving. Deaf object created in second iteration round

MESRO: Movement Emitting Sound Receiving Objects. Blind object created in third iteration round

SEMRE: Sound Emitting Movement Receiving Environment. Deaf space created in third iteration round

Phenomenology: Philosophy that describes the structure of the objects of awareness and of awareness itself in abstraction from any claims concerning existence Perceptual crossing: Others are recognized as intentional subjects, I distinguish their lived-body from their body as an image and that they recognize my presence as intentional subject (Deckers et al. 2012) Research through Design: Research methodology were a physical hypothesis is created to get answers on a research question.

Reflection on action.

REFERENCES 1. Abram, D., (1996) The spell of the Sensuous: Language and Perception in a More Than Human World. Vintage Books, New York, NY, USA. 2. Corbridge, C., Rugg, G., Major, N.P., Shadbolt, N.R., Burton, A.M., (1994) Laddering: a technique and tool use in knowledge acquisition. Department of Psychology, University of Nottingham, University Park, Nottingham NG7 2RD. 3. Deckers, E.J.L., Levy, P.D., Wensveen, S.A.G., Ahn, R.M.C., Overbeeke, C.J., (2012) Designing for perceptual crossing: applying and evaluating design notions. Eindhoven University of Technology, Industrial Design, Eindhoven, the Netherlands.

4. Deckers, E.J.L., Levy, P.D., (2012) Designing for Perceptive Qualities: 7 showcases. Eindhoven University of Technology, Industrial Design, Eindhoven, the Netherlands.

5. Klooster, S., (2003) Embodynamics Framework. Eindhoven University of Technology, Industrial Design, Eindhoven, the Netherlands.

6. Koskinen, I., Binder, T., Redstrom, J., (2008) Lab, Field, Gallery and Beyond.

Volume 2, Issue 1, pages 46-57.

7. Lenay, C., Thouvenin, I., Guenand, A., Gapenne, O., Stewart, J., Maillet, B., (2007) Designing the ground for pleasurable experience.

8. Lenay, C., (2010) “It’s So Touching”: Emotional Value in Distal Contact. UTC/ COSTECH, Compiegne, France.

79

80 9. Marti, P., (2010) Perceiving while being perceived. International Journey of Design, 4(2), 27-38.

10. Merleau-Ponty, M., (1958) Phenomenology of Perception (C. Smith) Routledge & Kegan Paul.

11. Schonhammer, R., (2004) Human ‘sense of space’, moving images and architecture. Psychology of Art and Design, Faculty of Design, Burg Giebichenstein Hochschule fur Kunst und Design Halle/Saale, International Symposium Aesthetics and Architectural Composition, Dresden, 2004.

81

APPENDIX

APPENDIX A: QUESTIONNAIRE ACTING OUT Questionnaire acting out What role did you play during the acting out session? deaf artifact / blind artifact / person If you played one of the artifacts: What did you try to communicate to the other artifact?

What was your feeling when the person entered the room?

Was there some kind of emotion you tried to express when the person entered the room?

If you played the person: How did you perceive the ongoing conversation when entering the room?

How did you try to participate in the conversation?

What feeling did you get while doing this?

83

84

APPENDIX B: RESULTS ACTING OUT Results Acting Out Experiments Session

Conversation established?

Way of communicating

Percept

Yes, although hard to

Sending sound and light in reaction on each other

Subject

communicate Yes, light reacting on sound

Start creating patterns after while in conversation

Perceive

Yes

convers Deaf focusses on environment, lights moves instead Feels lef

Yes, one way communication

of flashing Blind reacts on deaf, deaf guides

commu Hard to

Yes, conversation started

Reacting on each other, trying to create evolving

through Hard to

without problems Yes

patterns Sound reacts with different pitches on light

Tries to m

Round 4 7

No

Try to attract each others attention

Subject

8

Yes

Try to copy and morph each others patterns

succeed Artifacts

Round 1 1 (pilot) 2 (pilot) Round 2 3 4 Round 3 5 6

other

ation

85 Perception subject

Comments

Subject feels left out and sees artifacts in bubble

What are the characteristics of the artifacts?

Perceives conversation, though to interfere in

conversation instead Feels left out and disconnected, tries to distort

ving

Artifacts make use of turning, abilities of body

communication Hard to perceive conversation, influencing blind

Subjects succeeds in attracting blind through deaf

through deaf Hard to interrupt, tried to interrupt physically

Conversation between subject and blind was there

Tries to make contact by making sound

Conversation faltered when one participant failed

Subject tries to bring artifacts together, does not

Subjects functions as interpreter between artifacts

succeed Artifacts ignored subject

86

APPENDIX C: CODE SELR //A900acj7 //include libraries #include <Servo.h> #include “pitches.h” //int sensorPin = 0; int lightVal; int pos; //position of servo int turn; //degrees of turning servo int time1; //timer to count time in idle mode to start exploring int time2; //timer to count time in idle mode to initiate sound int sensorDelay = 29; //sensorDelay for HC-SR04 sensors int shortTime = 60; //time to start acting after short interval int longTime = 300; //time to start acting after long interval Servo myservo; //Set fixed numbers #define maxDistance 300 //max distance that HC-SR04 measures //Define pins sensor #define sensorPin 0 #define speakerPin 11 #define trigPin 8 #define echoPin 6 long distanceSensor; long durationSensor; int melody1[] = { //melody1 to communicate NOTE_D4, NOTE_D4,NOTE_A4, NOTE_A4, //NOTE_D4, NOTE_D4,NOTE_A4, NOTE_A4, NOTE_G4, NOTE_F4, NOTE_E4, NOTE_D4, NOTE_C4, NOTE_D4, NOTE_E4,NOTE_F4, NOTE_G4, NOTE_A4, NOTE_D4 NOTE_D4, NOTE_A4, NOTE_A4, NOTE_G4, NOTE_F4, NOTE_E4, NOTE_D4, NOTE_C4, NOTE_D4, NOTE_E4, NOTE_F4, NOTE_G4, NOTE_A4}; int melody2[] = { NOTE_C2, NOTE_C3}; int melody3[] = { NOTE_D2}; int noteDurations1[] = { //duration of notes 4,4,4,4}; //4, 4, 4, 4,4,4,4,4,4,4, 4, 4, 4,2,4,4,4,4,4, 4, 4, 4,4,4,4,4,4,4,2 };

87 int noteDurations2[] = { 4, 2}; int noteDurations3[] = { 1}; int singleNoteDuration = 500; void setup() { Serial.begin(9600); pinMode(sensorPin, INPUT); pinMode(trigPin, OUTPUT); pinMode(echoPin, INPUT); myservo.attach(9); } void loop() { sensors(); light intensity timer();

//void of HC-SR04 measuring distance and light sensor measuring //timer that counts time until action has to be taken

if (time1 >= shortTime){ //initiate sound for conversation after short time for (int thisNote = 0; thisNote < 29; thisNote++) { int noteDuration = 1000/noteDurations1[thisNote]; tone(speakerPin, melody1[thisNote],noteDuration); int pauseBetweenNotes = noteDuration * 1.30; delay(pauseBetweenNotes); noTone(speakerPin); time1 = 0; } } if (time2 >= longTime){ reaction has come

//initiate exploring for conversation after longer time when no

turn = random (0, 60); //looking around for(pos = turn; pos <=2*turn; pos+=1){ myservo.write(pos); delay(15); } for (int thisNote = 0; thisNote < 2; thisNote++) { //play melody to start conversation int noteDuration = 1000/noteDurations2[thisNote]; tone(speakerPin, melody2[thisNote],noteDuration); int pauseBetweenNotes = noteDuration * 1.30;

88 delay(pauseBetweenNotes); noTone(speakerPin); time2 = 0;

}

} for(pos = 2*turn; pos>=0; pos-=1){ myservo.write(pos); delay(15); } for (int thisNote = 0; thisNote < 2; thisNote++) { //play melody to start conversation int noteDuration = 1000/noteDurations2[thisNote]; tone(speakerPin, melody2[thisNote],noteDuration); int pauseBetweenNotes = noteDuration * 1.30; delay(pauseBetweenNotes); noTone(speakerPin); time2 = 0; } for(pos = 0; pos<turn; pos+=1){ myservo.write(pos); delay(15); }

if(distanceSensor < 20){ //subject approaching SELR for (int thisNote = 0; thisNote < 29; thisNote++) { //play tune when subject approaches int noteDuration = 1000/noteDurations1[thisNote]; tone(speakerPin, melody1[thisNote],noteDuration); int pauseBetweenNotes = noteDuration * 1.30; delay(pauseBetweenNotes); noTone(speakerPin); time1 = 0; } } if(lightVal < 550){ //receives light, reacts with sound tone(speakerPin, NOTE_E2, 100); } delay(100); }

89 int sensors(){ digitalWrite(trigPin, LOW); delayMicroseconds(2); digitalWrite(trigPin, HIGH); delayMicroseconds(10); digitalWrite(trigPin, LOW); durationSensor = pulseIn(echoPin, HIGH); distanceSensor = (durationSensor/2) / 29.1; //calculate distance Serial.println(“distance sensor “); Serial.print(distanceSensor); Serial.println(“cm”); lightVal = analogRead(sensorPin); //get sensordata from light sensor Serial.println(“Light intensity “); Serial.println(lightVal); delay(sensorDelay); }

return (distanceSensor);

int timer(){ if (lightVal > 550){ //count when no light is received to start exploring time1++; time2++; Serial.println(“timer 1”); Serial.println(time1); } }

90

APPENDIX D: CODE LESR //tty.usbserial-A900ac4W //include libraries #include <Servo.h> #include â&#x20AC;&#x153;pitches.hâ&#x20AC;? int int int int int int int int int int int

sensorPin = 0; micVal; pos; //position of servo turn; //degrees of turning servo time1; //timer to count time in idle mode to start exploring time2; //timer to count time in idle mode to initiate sound sensorDelay = 29; //sensorDelay for HC-SR04 shortTime = 100; longTime = 450; brightness = 0; fadeAmount = 5;

Servo myservo; //Set fixed numbers #define maxDistance 300 //max distance that HC-SR04 measures //Define pins sensor #define LEDPin 10 #define trigPin 8 #define echoPin 6 long distanceSensor; long durationSensor; void setup() { Serial.begin(9600); pinMode(sensorPin, INPUT); pinMode(trigPin, OUTPUT); pinMode(echoPin, INPUT); myservo.attach(9); } void loop() { sensors(); light intensity timer();

//void of HC-SR04 measuring distance and sound sensor measuring //timer that counts time until action has to be taken

91 if (time1 >= shortTime){ //initiate light for conversation after short time analogWrite(LEDPin, brightness); brightness = brightness + fadeAmount;

}

if (brightness == 0 || brightness == 255) { fadeAmount = -fadeAmount ; }

if (time2 >= longTime){

//initiate light for conversation after short time

turn = random (0, 60); for(pos = turn; pos <=2*turn; pos+=1){ myservo.write(pos); delay(40); } digitalWrite(LEDPin, HIGH); delay(200); digitalWrite(LEDPin, LOW); delay(200); digitalWrite(LEDPin, HIGH); delay(200); digitalWrite(LEDPin, LOW); delay(200);

}

for(pos = 2*turn; pos>=0; pos-=1){ myservo.write(pos); delay(40); } digitalWrite(LEDPin, HIGH); delay(200); digitalWrite(LEDPin, LOW); delay(200); digitalWrite(LEDPin, HIGH); delay(200); digitalWrite(LEDPin, LOW); delay(200); for(pos = 0; pos<turn; pos+=1){ myservo.write(pos); delay(40); }

92 if(distanceSensor < 20){ //subject approaching LESR for(pos = 0; pos<=60; pos+=1){ //turn towards subject myservo.write(pos); analogWrite(LEDPin, brightness); brightness = brightness + fadeAmount;

}

if (brightness == 0 || brightness == 255) { fadeAmount = -fadeAmount ; } delay(40);

}

}

if(micVal < 200){ digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); digitalWrite(LEDPin, delay(200); } delay(100);

//receives sound, reacts with light HIGH); LOW); HIGH); LOW); HIGH); LOW); HIGH); LOW);

93 int sensors(){ digitalWrite(trigPin, LOW); delayMicroseconds(2); digitalWrite(trigPin, HIGH); delayMicroseconds(10); digitalWrite(trigPin, LOW); durationSensor = pulseIn(echoPin, HIGH); distanceSensor = (durationSensor/2) / 29.1; //calculate distance Serial.println(“distance sensor “); Serial.print(distanceSensor); Serial.println(“cm”); lightVal = analogRead(sensorPin); //get sensordata from light sensor Serial.println(“Light intensity “); Serial.println(lightVal); delay(sensorDelay); }

return (distanceSensor);

int timer(){ if (lightVal > 550){ //count when no light is received to start exploring time1++; time2++; Serial.println(“timer 1”); Serial.println(time1); } }

94

APPENDIX E: MAX MSP PATCH SEMRE

APPENDIX F: CODE MESRO // usbserial-AH00ZY2U #include <Servo.h> Servo myservo1; // Big servo in foot Servo myservo2; // small vertical servo Servo myservo3; // small horizontal servo int pos1 = 0; // variable to store the position of Servos int pos2 = 0; int pos3 = 0; int PinA = A0; // input pins from distance sensors of SEMRE int PinB = A1; int PinC = A2; int PinD = A3; int lightPin = 3; //Pin that connects to powerLED in arm of MESRO int valA = 0; // input val from distance sensors int valB = 0; int valC = 0; int valD = 0; int valA1 = 0; // mapped input val from distance sensors int valB2 = 0; int valC3 = 0; int valD4 = 0; int corner = 1; //1=A 2=B 3=C 4=D int dir = 0; //variable indicating direction to explore in idle mode int ACont = 0; //variable that states if MESRO is looking in right direction int BCont = 0; int CCont = 0; int DCont = 0; void setup() { myservo1.attach(9); // attaches the servos to indicated pins on Arduino myservo2.attach(10); myservo3.attach(11); Serial.begin(9600); }

95

96 void loop() { valA = analogRead(PinA); //reads values from distance sensors valB = analogRead(PinB); valC = analogRead(PinC); valD = analogRead(PinD); valA1 = map(valA, 0, 1255, 0, 400); //maps values from distance sensors to eliminate differences between sensors valB2 = map(valB, 0, 1255, 0, 400); valC3 = map(valC, 0, 1255, 0, 400); valD4 = map(valD, 0, 1255, 0, 400); Serial.println(“valA”); //print lines to check states of distance sensors Serial.println(valA1); Serial.println(“valB”); Serial.println(valB2); Serial.println(“valC”); Serial.println(valC3); Serial.println(“valD”); Serial.println(valD4); if ((valA1 + valB2 + valC3 + valD4)/4 <= 100){ idleMode(); //idle mode is standard activated when there is no one in the space } else if(valA > valB && valA > valC && valA > valD){ //check to see which distance sensor measures highest input, this is the corner where the person stands corner = 1; } if(valB > valA && valB > valC && valB > valD){ corner = 2; } if(valC > valB && valC > valA && valC > valD){ corner = 3; } if(valD > valB && valD > valC && valD > valA){ corner = 4; } Serial.println(“Corner”); //print line stating the corner MESRO is going to explore Serial.println(corner); if (corner == 1){ movingA();

//dependent on the corner, the corresponding void is called

97

}

} if (corner == 2){ movingB(); } if (corner == 3){ movingC(); } if (corner == 4){ movingD(); }

98

APPENDIX G: QUESTIONNAIRE Questionnaire (Es)Sense SEMRE & MESRO Test subject # Recording # You will enter a room. In this room, a conversation is going on between the space (SEMRE) and the artifact (MESRO). The goal is to intervene in this conversation and either become part of it or distort it.

Evaluation: Can you describe the conversation you encountered?

Do you feel the space (SEMRE) is able to perceive?

What does SEMRE perceive?

Do you feel the artifact (MESRO) is able to perceive?

What does MESRO perceive?

How did you perceive the interaction between SEMRE and MESRO? What did you perceive?

Which of the three (artificial) players did you perceive as dominant?

Did you understand the behavior of MESRO (the artifact)?

Did you recognize the behavior of MESRO (the artifact)? What did you recognize?

99 Did you understand the behavior of SEMRE? How did you perceive the entity of SEMRE? Did you feel involved? Why did you get this feeling?

Did the behavior of either of MESRO or SEMRE trigger changes in your own behavior?

Validation session: With the extra information given, does this change your perception of SEMRE & MESRO? Thinking of a translation to product, in which situation can you see the principle of SEMRE & MESRO in practice?

100

APPENDIX H: WALKING PATTERNS = short rest

= long rest

= making sound

#1 time taken: 1:35

#2 time taken: 1:50

#3 time taken: 1:38

#4 time taken: 6:25

#5 time taken: 2:35

#3 time taken: 4:37

101 = short rest

#7 time taken: 3:11

= long rest

= making sound

#8 time taken: 3:20

#9 time taken: 1:42

102

APPENDIX I

Results Experiments Session Perception of SEMRE

Perception of MESRO

Perception of conversation SEMRE and

Dom

1 (pilot) SEMRE perceives me, produces If it is able to perceive me, it flees

MESRO There is some kind of interaction, although

MES

sound 2 (pilot) SEMRE measures my distance

hard to define, is collective or individual?

It is a status quo between 4 speakers and

SEM

absence of sender (SEMRE) MESRO, reacts on blocking of signal MESRO is not able to perceive me, only SEMRE dictates the conversation, MESRO

thei MES

MESRO learns to perceive, reacts on

3

to one of the speakers SEMRE perceives me based on

4

my position in the room SEMRE can do this First thought: SEMRE perceives MESRO does not perceive me directly

speaks in monologue, hard to interrupt this Sound calls MESRO, made me feel clumsy and SEM

5

me via MESRO, second: via walls but via absence of sound SEMRE notices my position and Does not react on me, reacts only on

uncomfortable Conversation between SEMRE and MESRO,

liste I ha

6

gives the sound a higher pitch the absence of sound Coupling is very direct, acts as a Does not react directly, but uses

distorted when I interfere Does not feel like conversation, I can make

SEM MES

7

mirror I think SEMRE tries to lure me

speakers as sensors myself visible through installation It seems to look around, I would expect Hard to notice, what is way of

8

SEMRE has the ability to

it would notice me at one point In the begin I thought that MESRO

communicating? 4 corners seem to lure object Action-reaction, they react on each others Me

9

perceive where I am SEMRE can perceive me, and

perceived me, changed later MESRO does not perceive me directly,

actions MESRO investigates corners where something Me

chances sound according to it

but reacts on sound

is happening

Non

103

d

Dominant artifact

Understanding of SEMRE

Understanding of MESRO

Feeling of involvement

Chan

ugh

MESRO

-

No understanding, perceives

I am welcome, although not

Yes, m

? d

O

SEMRE, they protest when Underwater world due to

movement as animal like invited It feels not complete, has body of It did react on me, was not

behav You m

their signal is blocked MESRO

bubbling sounds It has no own behavior, it is a

a tentacle No understanding, no link

happy, no acceptance I can influence, but not on a

try to Searc

space SEMRE does not want user, is in

between me and MESRO I am not sure, gets disturbed by

personal level not totally, I have influence,

Yes, tr

this sy and SEMRE, determines were

RO,

listening should happen I had the most control,

own conversation with 4 speakers subject, bit autistic but no involvement movin It is an entity with lots of stimuli Tries to explore, when obstructed Not involved, I had the role of Yes, w

ake

SEMRE was most present MESRO and I

tries to look around subject Can feel me, no presence of entity Behavior is too vague, looked

None

I have no idea what is happening

around with face Clearly in search of something,

obstructor Was involved, searching for

worki No, I w

answers Since I donâ&#x20AC;&#x2122;t know what

explo I start

object ers Me

not clear for what reason I did not understand it completely, Reacts on sound, moves

happens I didnâ&#x20AC;&#x2122;t feel involved enviro No I was the disturbance Starte

ething Me

why the bubble sound? according to sound, just like I do SEMRE calls when I am in the way MESRO searches for sound of 4

factor Not concerned, Although I

speakers

was part of interrupting

becam No ch

104

t

Changes in own behavior

Changed perception with

h not

Yes, more explorative

more information?

More humane, less

not

behavior You move more directed,

mysterious Yes, experience is clearer

on a

try to test own hypothesis Search for meaning

I miss direct influence on

Yes, tried to explore, faster

MESRO Confirms own expectations

ence,

moving between corners role of Yes, when I discovered

More clear, interaction

g for

working started to test No, I was already here to

t

explore what was going on I started to explore Makes clearer what is

volved environment ce Started to move more,

gh I

g

became curious No changes

becomes clearer Did match expectations

supposed to happen Confirms own expectations Did match expectations

105

JOB HUBERTS S070991 j.y.j.huberts@student.tue.nl M1.2 COACH: SIMONE DE WAART THEME: WEARABLE SENSES