GHOST by Pepijn Verburg

Effects of augmented sound in shape-changing interfaces

dated on January 2015

written by BSc. P. Verburg (m1.1) coached by prof.dr. P. Markopoulos

for the GHOST project

process report research project

table of contents

part 0: introduction

part 2: research question

0.1 abstract

2.1 deformable display

0.2 introduction

2.2 perceptual change

0.3 process overview

2.3 research direction

2.4 emotional states

part 1: exploration

2.5 physical properties

1.1 definition

2.6 ecological approach

1.2 auditory information

2.7 research question

1.3 soundless prototype

1.4 efficiency calculations

part 3: prototypes

3.1 paper fold

3.2 textile handheld

3.3 wooden rectangle

3.4 upgrading

part 4: experiment

4.1 study plan

4.2 execution

4.3 results

4.4 discussion

part 5: reflection & bibliography

5.1 reflection

5.2 bibliography

part 0: introduction 4

This part introduces the design & research process and structure of this report.

0.1 abstract The purpose of this research is to

explore

the effects of auditory information on the perception of physical properties in shapechanging interfaces. A paired t-test analysis has been conducted on the results of a questionnaire with four visual analog scales (VAS). Each scale represented a physical property. During the research the perception of brittleness has been the main focus. On the scale â&#x20AC;&#x2DC;fragile - strongâ&#x20AC;&#x2122; an effect can be observed. However, it is not significant. This

research

puts

forward

soundless

prototyping platform for shape-changing interfaces to freely explore the effects of auditory information or other purposes. A total of three prototypes were made along the research-through-design process.

0.2 introduction Within the field of shape-changing interfaces

the ways to influence the representations and

digital information is represented in a more

thereby accomplishing some kind of shape

physical, three-dimensional, shape. It would

change on a perceptual level. Research

be fair to ask what the effects of auditory

has shown that auditory information has a

information are on the perception of these

significant influence on the visual modality

representations. Research now attempts to

(Vroomen & Gelder, 2000) and perception

let the representations react on for example

of the physical properties (Gaver, 1993). Even

actuation, projection and changing material

performance and usability can be improved

properties as seen in BubbleWrap (Bau et

when using auditory information (Brown et al.,

al., 2009), PneUI (Yao et al., 2013), inFORM

1989; Brewster et al., 1994). This research aims

(Follmer et al., 2012) and Shutters (Coelho

to investigate this specific topic by questioning

& Zigelbaum, 2010). Still, the influence of

how the perception of physical properties

sound remains untouched. There is a great

can be influenced by augmented sound.

opportunity in addressing sound as one of

0.3 process overview 1.2 auditory information The observation is made that soundless interfaces are not well explored within the field of shapechanging interfaces. Auditory information during actuation is often even hidden in the presented material.

2.1 deformable display The magnetic actuation allows a soundless elastic, deformable display supporting both input and output, making it interesting for certain applications.

1.1 definition The following four properties define shape-changing interfaces. The change in shape is: 1) deformable 2) touchable 3) automated 4) reversible

1.3 soundless prototype An initial prototype is made based on BubbleWrap (Bau, Petrevski, & Mackay, 2009). A soundless actuation is created by using magnetic fields.

2.2 perceptual change The magnetic actuation is also a possible tool to research the influence of auditory information, as it is soundless and thus creates the freedom of adding any sound.

These properties are used to evaluate whether an object can be classified as a shape-changing interface.

1.4 efficiency calculations Mechanics expert Frank Delbressine advises on looking into the mechanics and efficiency of these magnetic actuations by performing some basic calculations.

2.3 focus shift Both research topics are of value for the GHOST project. Based on the application of the technology the topic related to perceptual change is chosen.

This page shows a condensed overview of the design & research process. In the future chapters all these topics are explained more into depth by providing background information, literature and visual material.

approximate time

2.4 emotional states The research starts with a focus on influences of augmented sound in shape-changing interfaces on emotion where most of the theories are based the field of Psychology. However, researching emotions appears to be too complex. 2.5 physical properties A more superficial subject is chosen: the perception of physical properties. This is one cognitive level lower, where emotion is an reaction to these properties.

2.6 ecological approach Sound expert Beggy Eggen advises on using an ecological approach (Gaver, 1993) when creating augmented sound for perceptual changes.

3.1 paper fold An initial prototype is crafted where the compression of folded paper results in augmented sound resembling cracking of the object.

3.2 textile handheld The same augmented sound is implemented in a textile handheld object, being more close to the body and having more degrees of freedom in its interaction.

3.3 wooden rectangle The paper fold and textile handheld prototype are combined in a wooden rectangle where an optimum is found in degrees of freedom, interaction and material.

3.4 upgrading The wooden rectangle prototype is upgraded to be capable of storing interaction data and switch to different modes.

4.1 study plan An extensive study plan is written to provide the experiment with the sufficient background information, literature and methods.

4.2 execution The experiment has been conducted with 14 subjects. All subjects are nondesign students.

4.3 results The results are analyzed using a paired t-test. Minor effects are noticed in the scatter plots. However, they are not significant.

4.4 discussion Future research should attempt to increase the significance or dive into other influences of augmented sound in shape-changing interfaces.

approximate time

part 1: exploration 10

This part will put forward a definition of shape-change and the process towards the focus of using soundless shapechanging interfaces in this research project.

1.1 definition No clear set of properties of a shape-changing interface can be found within the literature. When can we classify something as being a shape-changing interface? First of all, a shape-change is about deformation in its material. However, deformation is already found in many of the everyday technologies: LCD screens for example. On a very small scale liquids move around creating the desired visual composition. One wants to exclude these small scale cases. Therefore the deformation needs to be touchable. In other words: the change should be perceivable with our touch senses. Furthermore, a shape-changing interface often implies a relation between digital information and the physical world. The behaviour of this interface should be automated and reversible, responding to digital information at once. Visual 1.1.2 shows an overview of all the properties listed in the paragraphs above. Of course there could be more or less, but these properties have guided design & research process of this project.

visual 1.1.1 - variety of known shape-changing interfaces

deformable

touchable

shape-changing interfaces automated

reversible visual 1.1.2 - property overview

1.2 auditory information During the exploration as shown in visual 1.1.1 an important observation is made: sound is very present due to the actuation or is even hidden in the presented material (e.g. videos). The effect of this auditory information due to the actuation is not known. To be able to explore this a soundless interface is needed. This will gain the freedom of adding any sound enabling the researchers to control it accurately. Visual 1.2.1 shows such a soundless interface,. BubbleWrap (Bau et al., 2009) uses magnetic fields to change the firmness in its material. The next chapters will explore whether this technology could be engineered on a bigger scale.

visual 1.2.1 - BubbleWrap prototype

1.3 soundless prototype An initial prototype is build with a permanent magnet and a copper coil to increase the change in shape. The first tests are successful (see visual 1.3.1): the coil is able lift the cloth at 5,00 Volt and 1,00 Ampere. However, after a few minutes the coil gets very hot due to its resistance and power consumption. Mechanical Engineering expert Frank Delbressine advises on providing this principle with some indetail calculations about power usage and efficiency.

visual 1.3.1 - internal components

1.4 efficiency calculations This chapter illustrates the process of finding an optimum in the power consumption and heat production. Property formulas First off, some basic properties were calculated or measured of the power, the coil and the material. Copper density: 8930 - 8940 kg/m3 Length wire: 3,80m Power consumption: Pc = U * I = 5,00 * 1,00 = 5,0 W Coil resistance: Rc = 6,2 Ω Thermal conductivity copper (Granta, 2010): 160 - 390 W / cm / oC Specific heat capacity (Granta, 2010): 372 - 388 J / Kg / oC

Outer coil ring radius & wire radius rc = 0,020m, rw = 0,001m Heat & magnetic field formulas Secondly, formulas relating to heat and magnetic fields are listed below (Hugh et al., 2011). Heat and temperature change: Q = m * c * ΔT Heat current for conduction: H = k * A * (TH - TC) / L Heat current for radiation: H = A * e * σ * T4 Wire magnetic field: B = (μ0 * I * a2) / (2(x2 + a2)3/2) Coil magnetic field: B = (μ0 * N * I) / 2a Energy in magnetic field: uB = B2 / (2 * μ0)

Magnetic field First, let’s calculate how strong the magnetic field is at the center of the coil. To calculate this for the hand-winded coil (see visual 2.1) is far too difficult, because of its inconsistencies. To make this easier the calculation will be done with a perfectly winded coil consisting of a similar 60 windings distributed equally over a starting radius of 0,02m and an ending radius of 0,001m. To simulate this situation a Processing Sketch was build to generate the coil and to calculate the total magnetic field in Tesla (see appendix 1). The calculation was done as following for every winding: B = (u0 * I) / 2a For example, with the outer winding this would give: Bs = (u0 * I) / 2a Bs = (4π * 10 -7 * 1,00) / (2 * 0,02) Bs = 0.00003141592 T The same calculation is done for all the windings and this will result in: Btotal = 0.0060 T

Power consumption For the magnetic field in the coil it can be calculated how much energy is stored in this field: uB = B2 / (2 * μ0) uB = 0.00602 / (2 * 4π * 10 -7) uB = 14.32 J / m3 With the coil acting as its body the actual amount of energy stored in the magnetic field can be calculated. First the volume of the wire is required: Vw = π * rw2 * L Vw = π * 0,0012 * 3,80 Vw = 0.000012 m3 Now the total energy in the field can be determined: Em = UB * Vw Em = 14.32 * 0.000012 Em = 0.00017 J To make a conclusion, this energy used in the magnetic is very little in comparison with the total amount of energy put into the coil all the time: namely 5,00 J /s.

Heat generation When the levitated coil is in a stable state it can be assumed only the force of gravity is countered with energy in the magnetic field. The remaining amount of energy is lost as a temperature rise in the wire. For these calculations the amount of energy lost due to radiation and dissipation to air are neglected (Hugh et al., 2011). One can say that almost all the energy is lost to heat generation as the levitation force in comparison with the power consumption is very low. Below shows a calculation of how fast the wire will heat up 50 oC. First the mass of the wire needs to be calculated: mw = 0.000012 * 8935 mw = 0.11 kg An average for the heat capacity was taken for the wire: 370 J / Kg / oC

This implies that to heat up 0.11 kg of copper 50 degrees a certain amount of Joules is needed: Q = m * c * ΔT Q = 0.11 * 370 * 50 Q = 2035 J With the amount of Joules known the amount of time can be calculated: Δt = Q / P Δt = 2035 / 5 Δt = 407 s This means almost 7 minutes are needed to heat up 50 degrees, which is dangerously hot. In conclusion it can be said the coil will heat up quickly after intensive use and that this technology is not very energy efficient. Still, it has proven itself to be useful for shortterm usage and allows enough force to deform materials. Also, with power regulation the intensity of deformation can be controlled accurately and silently.

part 2: research question 18

This part shows the formulation of the research question for this project. The technology from part 1 is used as a starting point. In the end a focus emerged on perceptual change of physical properties in shape-changing interfaces.

2.1 deformable display The first possible research direction goes towards the field of mechanical engineering. The BubbleWrap prototype shown in part 1 provided a useful technology. However, what was not explored is ways to let this technology function as an input device and perhaps even as input and output. An application for such a technology could be an elastic, deformable display (see visual 2.1.1) developed at Copenhagen University. Currently, this prototype can sense deformations and reacts through a projection. Using coils and magnets could make it possible for the display to have the ability to actuate. The research direction would be focused on developing this technology and to create an elastic, deformable haptic screen that could change its firmness and/or shape. The ability to change firmness is already put forward in BubbleWrap (Bau et al., 2009).

visual 2.1.1 - elastic, deformable display

2.2 perceptual change visual modality + haptic modality auditory modality

? perception

Magnetism provides a special characteristic for actuating objects: silence. The second possible research direction aims to use this property to its advantage. Shape-changing interfaces aim to get closer to the physical world in representing digital information. The interpretation of these representations are crucial in shaping the user experience. Until today sound in particular is not taken into consideration when looking at the interpretations. The technological principle of BubbleWrap (Bau et al., 2009) creates the freedom of adding any sound to explore the effects of it on the perception of shape-changing interfaces. This research direction leans towards the field of psychology.

visual 2.2.1 - schematic of influence

2.3 research direction Both research directions put forward in chapter 2.1 and 2.2 are technology-driven. To truly find which research direction is the best three aspects are considered: 1) application of the technology 2) significance for the GHOST project 3) expertise of the researcher First of all the silence is the most important property of the technology. The focus on perceptual change uses this property more to its advantage; the elastic, deformable display doesnâ&#x20AC;&#x2122;t necessarily need this. Secondly, both directions are of significance within the GHOST project and will add to the research according to Panos Markopoulos (one of the proposers of the project). Finally, the researcher has a background within the field of psychology and has experience with researching the perception of safety with street lighting.

Adding up these three points one can conclude that the psychology focus is more suited for this research project (see visual 2.3.1 for an overview). Still, the focus on perceptual change needs several specifications. First of all, there are many ways to define the perception of a shapechanging interface. Secondly, the question remains how this perception is envisioned to be influenced. The next chapters of this part will dive into these problems with some theories based in cognitive sciences. display

perception

technology

GHOST project

researcher

visual 2.3.1 - decision overview

2.4 emotional states James’s theory stimulus

perception

(bear)

(danger)

arousal

emotion

(pounding

(fear)

heart)

One way to approach the perception of shapechanging interfaces is to look into the emotions people experience and how these can be influenced by auditory information. There are two well-known models describing the process of emotions within our body on a psychological level: one by William James (James, 1890) and one by Stanley Schachter (Schachter, 1962) (see visual 2.4.1). The question is where auditory information could interfere.

Schachter’s theory stimulus

perception

(bear)

(danger)

arousal (pounding heart)

emotion type

(fear)

intensity

visual 2.4.1 - process overview emotion (Gray, 2010)

However, when going more into depth on design and emotion one quickly comes to the conclusion emotion is a complex topic to research in relation to shape-changing interfaces. For example, Spillers describes it as: “Users generate emotion as a way to minimize errors, interpret functionality, or obtain relief from the complexity of a task.” (Spillers, 2004). Vroomen & Gelder give a new direction. They describe effects where sound merely enhances the visual perception of objects (Vroomen & Gelder, 2000) without relating to more complex cognitive processes like emotions.

2.5 physical properties For the scope of this research within an Industrial Design department it is sensible to stick with lower level cognitive processes. Therefore the direction is taken to look into the perception of physical properties (Vroomen & Gelder, 2000). Physical properties in comparison with emotion are on a lower level. Emotion can be seen as the interpretation and reaction to these properties as Spillers describes (Spillers, 2004). Physical properties can be defined as constructs related to the shape-changing interface without any personal feelings: it is merely a sensory experience (Gray, 2010). In visual 2.5.1 one can see an overview of how physical properties are in relation to other definitions according to this research. Please note that this schematic is merely to give an overview of the different definitions and where auditory information will be playing a role, the relations between them are still questionable.

physical properties auditory information

? perception interpretation behaviour

visual 2.5.1 - physical properties in relation to other definitions

2.6 ecological approach sound-producing event

sound modified by environment

sound near the ear

conscious experience

auditory system visual 2.6.1 - parameters ecological approach (Gaver, 1993)

Influencing physical properties through auditory information requires a method to describe and synthesize sound. Gaver puts forward a method of doing so through an ecological approach (Gaver, 1993). Gaver showed that people describe sounds through comparing them with everyday entities. They rarely describe the sound itself like ‘high pitched’ or ‘1000Hz followed by a longer 28000Hz tone’. There is a preference to describe sounds like ‘someone walking the stairs’ or ‘a slam followed by several tinkling sounds’. Visual 2.6.1 shows what environmental parameters determine the recognition of these entities. An ecological approach results in a description of auditory events, instead of an array of mathematical sound properties. For example breaking can be described as “an initial rupture burst dissolving into overlapping multiple damped quasi-periodic pulse trains, each train having a different crosssectional spectrum and damping characteristic” (Warren & Verbruge, 1984). This approach will be used as a guideline for synthesizing the augmented sound.

2.7 research question A proper research question can now be formulated when looking back at the previous chapters. In summary, this research aims to explore the possible effects of auditory information on the perception of shape-changing interfaces.

Research questions The research question can be written down as: how can a shape-changing interface be perceived as more brittle through augmented sound?

Specific physical property Still, there are many properties to influence. To contain the scope of this study one specific property is considered: the perceived brittleness of a shape changing interface. Brittleness is naturally and effortlessly noticed during our everyday tasks, when interacting with deformable objects.

Hypothesis H1: by imitating the cracking and/or breaking of a material one can create the illusion of brittleness in shape-changing interfaces.

The strength of shape-changing interfaces comes from the strong relation with physicality and being closer to digital information, because it is expressed in everyday physical forms human beings can relate to. This is preserved when choosing a property like brittleness.

H0: there will be no difference in the shapechanging interface due to the cracking and/or breaking sound. This will make the measurement of the perception of brittleness follow μsound = μsilent Goal The ultimate goal of this research is to provide the field of shape-changing interfaces with an initial direction on auditory information. If such an effect is found one is able to change the brittleness over time, creating some sort of variable material properties. The question still remains whether other properties can also be influenced the same way.

One could almost define this as one of the ‘degrees of freedom’ within shape-change, but only on a perceptual level. Study plan This chapter has only formulated the research question and the hypothesis. Part 4 of this report will go into depth on the research question through specifying methods, data collection, analysis, etc. Prototypes Before the research experiment is specified an exploration is done of creating a soundless shape-changing interface to test with: part 3, prototypes.

How can a shape-changing interface be perceived as more brittle through augmented sound?

part 3: prototypes 28

This part shows the prototypes created to iterate towards a shape-changing interface used in the research experiment. Two explorative prototypes were made without shape-changing capabilities. These guided the design process for the final prototype.

3.1 paper fold folded paper An initial prototype is build to explore how to the synthesized sound is experienced in a user to product interaction. Only augmented sound is investigated, this prototype will not have any shape-changing capabilities. Fundamentals The idea is to stack several layers of folded paper capable of being compressed when pushed on it (see visual 3.1.2). Under the stack will be a pressure sensor, which registers the push and initiates the synthesis of the augmented sound by the microcontroller. Ecological description To be able to synthesize a convincing sound the cracking and/or breaking sound events of several everyday objects are observed using the ecological approach of Gaver (see chapter 2.6). It appears there is a strong relation between the applied force during the interaction and three variables. A greater force (1) creates a higher pitch, (2) consists of shorter samples and (3) shows a smaller delay between these samples. Visual 3.1.1 shows relation 2 and 3. Please note that this

is purely observational. Research should point out whether such a description is generally valid. Sound synthesis The description of visual 3.1.1 is implemented on the Arduino platform. One can find the code in appendix 2. The synthesizer will use one base frequency that will slightly get higher in pitch, change length and delay when more force is applied. Measurements Applied force is measured through calculating a delta pressure. This delta pressure slowly decreases over time. A quick and hard push will then automatically result in a higher delta.

force

sound silence

visual 3.1.1 - relation force and events

pressure sensor top cover

speaker

microcontroller

casing

visual 3.1.2 - components

Materials The casing is made out of MDF. Raw velostat is used for the pressure sensor to be able to create a custom size sensor, covering the whole bottom area of the folded paper. Finally, a standard 8 Ohm speaker is integrated closely to the folded paper. Result Visual 3.1.3 shows the prototype. It was informally given to other Industrial Design students to let them talk about their experiences with it. Generally there are a few remarks: 1) The latency of the sound is very low, showing the relation between pushing and the sound well. 2) The sound is a bit like a Geiger counter. 3) The sound is perceived as somewhat unnatural, because paper is the material being deformed. 4) The interaction is a bit distant from the user, possibly also causing unnatural behaviour.

visual 3.1.3 - paper fold prototype

3.2 textile handheld The remarks of the paper fold prototype are used to go through a second iteration in building an other prototype.

Measurements The determination of the applied force remains the same.

Fundamentals This next iteration attempts to make the interaction more natural by using existing samples in the sound synthesis and bringing it more close to the body. To do this a cylinder shape has been chosen that can be held by the whole hand. Visual 3.2.1 shows the concept for this idea.

Materials A cloth has been crafted with flexible foam on the inside. The speaker and the sensor are placed within the cloth. The microcontroller is external due to the lack of space.

Ecological description The ecological description remained the same, because the latency is good and the changes in synthesis are convincing when varying in applied force. Sound synthesis Granular synthesis is used to generate the new sound. This means small parts (randomly chosen) are played of an existing (open-source) sample of an object cracking. Some other minor tweaks are done in order to make it work properly with the new components. The new code can be seen in appendix 3.

speaker pressure sensor

textile cylinder

Result Visual 3.2.2 shows the prototype. Again, some questions were asked informally towards Industrial Design students. General remarks: 1) The sound is perceived as natural 2) One can feel the vibration of the speaker a little bit, showing a close relation between the object and the sound. 3) The sound is not associated with a Geiger counter. 4) The object is very playful; people are keen on bending it.

wiring

external microcontroller visual 3.2.1 - components

visual 3.2.2 - textile handheld prototype

3.3 wooden rectangle The two prototypes from chapter 3.1 and 3.2 are made to explore the general effects of augmented sound on the perception of brittleness in inanimate objects. Through these prototypes several conclusions can be made required for the shape-changing prototype of the experiment. Relation with the body The paper fold prototype is rather distant in relation to its user. The textile handheld prototype felt more close and easier to interact with. A reason for this could be, because the handheld prototype covers a larger surface on the body and thereby also creates a more noticeable haptic feedback caused by the vibration of the sound. This haptic feedback gives the impression of a close spatial relationship between the object and the sound. Playfulness Furthermore, because of the flexibility in the objects the brittleness is perceived as something to play with. People donâ&#x20AC;&#x2122;t mind hearing the sound.This is where shape-change blends in. Changing the object its physical state could be

a crucial element in the perception of brittleness, because people are expecting a change in shape with a fragile object. Another possibility is that the playfulness is trigged by the high amount of degrees of freedom in the interaction. Fundamentals In summary, there are several requirements: 1) The amount of degrees of freedom is limited to avoid playful behaviour. An object that attempts to be breakable should not be playful. 2) Soundless shape-changing capabilities is integrated; preferably using magnetism, as demonstrated in part 1. 3) A larger surface than the paper fold prototype is used to increase bodily contact. Shape-change basics The paper fold prototype provided a simple interaction, suitable for the experiment. One degree of freedom will make the behaviour of the participant predictable and thereby easier to anticipate on with the sound synthesis. In visual 3.3.1 one can see the proposed shape change. visual 3.3.1 - shape-change basics

Actuation The actuation is based on spring mechanics: the magnetic force deforms the object to a new shape. Turning the force off again will transform the object back into its original shape due to its resilience. This is in line with the reversibility component of a shape-changing interface from chapter 1.1. However, with more resilient objects a greater force is needed to accomplish a deformation. Visual 3.3.2 shows the copper coil suitable for this. A large permanent magnet is lifted above this coil. Initial tests with the coil show it consumes around 12,00 Volt. For safety reasons a maximum of 5,00 Ampere will be used to power the coil, making the total power consumption 60 Watt.

closing force

z-axis

y-axis

z-axis Heat generation Chapter 1.4 showed possible problems with the heating of the coil. For this reason mounting of the coil is done through force closure to minimize the surface in contact with other materials. In visual 3.3.2 one can see how the x-, y- and z-axis are blocked. In such a configuration more heat can be dissipated to the air.

x-axis

visual 3.3.2 - actuation coil

Flexible wood A solution to make only one degree of freedom possible can be found with wood and lasercutting. With different patterns a diverse set of material flexibilities can be created. It is even possible to create wood flexible in two directions. Visual 3.3.3 shows an exploration with several known patterns (Porterfield, 2014). Each pattern is briefly tested for the desired shape-change according to the following parameters: 1) permeability, a lower permeability is preferred to prevent distractions by the uneven surface. 2) flexibility, a higher flexibility is preferred to have a bigger deformation with the same magnetic force. Pattern 5 and 7 are the best ones (see visual 3.3.4). However, after extensive bending pattern 5 appears to be more fragile and breaks faster, making pattern 7 superior.

visual 3.3.3 - final prototype visual 3.3.3 - flexible wood exploration

permeability flexibility

visual 3.3.4 - flexible wood conclusions

Components In visual 3.3.5 one can see all the components required to make the shape-change and the augmented sound possible. To make the placement of these components possible a non-flexible segment is added to the top and bottom plate (see visual 3.3.6).

top plate

top layer

Measurements Measuring a push on this prototype is a bit more difficult, because the object itself can also create an applied force due to the shape-changing capabilities. For this reason an absolute measuring sensor is used; a flex sensor between the top and bottom plate. Definition All these components make sure this prototype satisfies the four properties of a shape-changing interface as defined in chapter 1.1. The flexible plates create (1) deformability that is (2) touchable due to its big radius. This deformation can be (3) controlled by adjusting the power on the coil and itâ&#x20AC;&#x2122;s (4) reversible because of sturdiness of the wood.

speaker

magnet

flex sensor

coil

bottom layer

bottom plate

visual 3.3.5 - components

flexible segment for deformation (pattern #7)

segment for components

flexible segment for deformation (pattern #7)

visual 3.3.6 - material composition

Layer integration One of the final hurdles is the integration of components when the two layers meet. This happens when the prototype is in its smallest state. Visual 3.3.7 shows this situation with the top layer removed; when the shape increases in size the magnet and the speaker will be pushed upwards. The size of the magnet and the speaker is determined by the placement of the z-axis blockages needed for the coil (introduced in visual 3.3.2). The flex sensor appears to be most effective when placed in an s-shape (see visual 3.3.9) between the two layers. The s-shape also allows the two layers to be pushed away from each other. Result In visual 3.3.8 one can see the result of this iteration. This prototype is fully capable of (1) changing shape, (2) measuring its current shape and (3) creating sound. In part 4 detailed specifications are given how these three properties work together in the experiment. visual 3.3.7 - layer integration

visual 3.3.9 - flex sensor

visual 3.3.8 - wooden rectangle prototype

3.4 upgrading The wooden rectangle prototype is upgraded to increase the adjustability. This is done with the thought quick changes might be needed when for example pilot tests are executed. Power supply interface The Arduino is not capable of controlling high power consumptions such as the coil. To make it able to control the shape-change an interface needs to be build. The power supply can be controlled with the computer using serial communication (not compatible with Arduino). The Arduino sends command to the computer which it then parses to the power supply in the right format. Visual 3.4.1 shows the schematic of how the three components communicate with each other. The Java code for this interface on the computer can be found in appendix 8. Data recording Another feature added is the recording of the current amperes on the coil and the current bend of the flex sensor. These use the same interface as the power supply control, the only difference is that the data is written to a file instead of parsed to another device.

Arduino current flex sensor value + requested coil amperes

flex sensor value

prototype

pressure

user

amperes + sensor value + time

computer requested amperes

power

power supply visual 3.4.1 - data flow in prototype

file

Sound latency The latency in the synthesis of the augmented sound has been decreased significantly by using raw audio files converted to a byte array in the Arduino code. In appendix 9 one can see these large arrays of bytes. Modes Another feature added to the wooden rectangle prototype is the ability to switch and add modes. By rebooting the prototype goes to the next mode, making it easy for the researcher to cycle across all the modes during the research. In appendix 9 the full updated code for the wooden rectangle prototype can be found.

visual 3.4.2 - push interaction for sound synthesis varying per mode

part 4: experiment 44

The wooden rectangle prototype from part 4 will be used in the experiment. This part will go into-depth into the details of this experiment by presenting a study plan, report on the execution, the results and a discussion.

4.1 study plan A study plan is written to provide the experiment with sufficient background information. Many sources of this plan are already included in this report. Therefore the study plan can be read in appendix 4. This chapter will present the process of the experiment with the important specifications summarized. Please refer to the study plan when more information is wanted.

Definition The definition of the experiment consists of the previous parts of this report. A research question is formulated (part 1 and 2) and a prototype is made capable of being used for the research question (part 3).

Methodology A shape-changing interface is presented in three situations: (A) with a sound representing brittleness, (B) with a subtle flowing sound and (C) without augmented sound. A list of tasks and a questionnaire is given to the participants, consisting of several visual analog scales (VAS) assessing the prototype on certain properties. Data collection Data collection takes place in three ways: (1) the researcher takes notes during the execution of tasks of the participants to write down extra remarks, (2) the participant is questioned through a questionnaire and (3) all the data of the interactions is recorded.

Pilot test A pilot test is conducted to check whether the setup is clear for the participants. Also, the VAS for the questionnaire are determined with the pilot. Execution The execution follows 7 steps for each participant: 1) preparations 2) Consent form 3) Introduction 4) Briefing 5) Execution 6) Questionnaire 7) Debriefing Chapter 4.2 will provide the details about this process.

Results The marks on the VAS are converted to numeric values used in a paired t-test analysis. The data of the interactions is also analyzed if the time allows it, as no specific methodology has been specified for this. The questionnaire results presented in scatter plots.

are

Discussion Conclusions are discussed and future directions are presented in this final part.

4.2 execution

This chapter shows the process of the execution of the experiment. In visual 4.2.1 one can see an example for the setup: 1) wooden rectangle prototype 2) computer, for data processing and power supply control 3) suitcase, for storage of the power supply and other components

Preparations A random order is generated for the three situations of the prototype: (A) one without sound, (B) one with a cracking sound and (C) one with a flowing sound.

Consent form Participants are asked to read and sign the consent form (see appendix 5).

Introduction The researcher briefly shows the prototype as being a shapechanging interface and explains the participants they will see this prototype in three different situations. The researcher also mentions they needs to compare them afterwards.

14 x 48

Briefing Also, the researcher explains the task. They are asked to draw an object in front of them (put there by the researcher). At the same time the prototype will deform and participants need to reverse the deformation by pushing it. During this interaction the current situation (A, B or C) determines which sound is produced.

Execution The three situations are presented to the participants, each with a duration of 1 minute. The researcher announces when a new situation starts.

Questionnaire Participants are asked to fill in a questionnaire (see appendix 6).

Debriefing The goal of the project is briefly explained and they are thanked.

visual 4.2.2 - power supply

visual 4.2.1 - setup example

4.3 results Each scale presented in the questionnaire has its own scatter-plot on these pages (see visual 4.3.1). The raw data set can be found in appendix 7. Only the questionnaires are analyzed due to the lack of time. The recorded data of the interaction will have future purposes. Scattered The first observation to be made is that all colours are scattered everywhere most of the time. The expected output of such a plot according to the hypothesis (especially for the fragile - strong plot) is that one would find a clear order in ‘none’, ‘cracking’ and ‘flowing’. Fragile to strong However, when looking more into detail one sees that on the fragile - strong plot there is a slight tendency of the cracking and flowing data points to be lower. And especially the cracking data points. This shows a tendency of the hypothesis to be true.

Significance Unfortunately, nothing appears to be significant when conducting a paired t-test on all scales and for all possible situation combinations. For example when looking at the fragile - strong scale and the difference between none and cracking the paired t-test gives: t = -1.4725 p = 0.1647 Fcritical = 2.16 With a confidence level 5% (p < 0.05) one can state this relation is far from significant and therefore the null hypothesis (see chapter 2.7) cannot be rejected. Conclusion There is a slight tendency on the fragile - strong scale to be influenced by the augmented sound according the hypothesis.

visual 4.3.1 - scatter plots VAS

4.4 discussion In this chapter the results and the setup of the experiment are discussed. Directions are given to future research how to improve answering the research question. Significance There are two options in creating a higher significance. First of all, increase the amount of participants. For quantitative research this was really at its minimum. Secondly, many participants made the remark the differences are very subtle. More extreme differences could increase variance on the scales. Questionnaire The visual analog scales (VAS) have proven itself to be efficient. However, participants noted it is was sometimes hard to remember all the situations properly and assess them all at once at the end of the experiment. Future research could let people assess each situation right after they are done with one. The random orders decline a possible effect of people knowing what scales there are when going through the final 2 situations.

Non-design participants Also, the VAS were rather abstract, making it hard for the participants to fully understand it and leaving it open for their own interpretation. During the pilot tests this did not emerge, because the people had experience within the field of design. A more descriptive scale could solve this problem. Many participants for example asked: “what do you mean with soft and hard?“. Recorded interactions The researcher observed a slight difference in the behaviour of pushing. The analysis of the recording should investigate this more into depth. At the time of writing there was no time for such an analysis. An example is shown in visual 4.4.1 (the interaction of participant #4 in the ‘cracking’ situation). Comparing such data is difficult, because the interactions don’t take place at the same time across all situations; the moment when the shape change starts was randomly generated. This analysis focuses more on unconscious processes and doesn’t need a questionnaire asking people to make perceptual conclusions conscious.

visual 4.4.1 - interaction cracking situation

Surprised Several participants made the remark at the end of the test they noticed a difference in behaviour. They were surprised when the researcher stated only the sound changes. Smell One unexpected effect was the heating of the coil causing a smell of wood to emerge later on in the test. Two participants mentioned this and asked if it was part of the experiment. With the random orders this effect is mostly declined, but for future research about perception it is something to take into account.

Future research The remarks people made and the (not significant) data imply such an effect of augmented sound on the perception of brittleness or other properties in shape-changing interfaces could exist. However, a higher sample size, a more clear VAS and analysis of interaction data should point out whether it can really be confirmed. Perhaps a jump towards other physical properties or emotional influences (chapter 2.4) or personal constructs (Kwak et al., 2014) can be made when looking at this effect of auditory information.

Notifications An important note to make is that the sound can be experienced as simple notifications. A few times the sound synthesis was not that convincing, because of an unexpected interaction pattern by the participant. When the augmented sound doesnâ&#x20AC;&#x2122;t match with the object it is quickly seen as a notification also causing a change in perception. Future experiments with auditory information should make sure the augmented sound is correctly mapped to the object.

part 5: reflection & bibliography 54

5.1 reflection First of all I would like to say I enjoyed researching such a complex topic. In this chapter I will briefly and informally reflect on this design and research process. Complex research direction I would like to start off with the research direction. I noticed a tendency of making things too complex. For example I started off with a focus on influencing emotions through auditory information. Therefore, Iâ&#x20AC;&#x2122;m happy I made the step to the more simple concept of physical properties. Still, this research remained complex, because it is dealing with the perception of people and not necessarily with their conscious processes. Where I definitely missed out on is how to measure this perception. I had the assumption I could make it conscious by simply asking and let people think about it. I am still not sure if this was the right way to go. For future research I would like to most of the tests were executed I noticed people had a hard time filling in the visual analog scales. If I had done my pilot tests more extensive and also with non-design people this could be avoided.

Process From the beginning there was a technologydriven process: I could create a shape-change that is silent, what can we do with it? This was the first time for me I experienced this approach. Normally a societal context is central within our projects. I enjoyed it and I now come to realize this process developed itself to be a researchthrough-design process. The three prototypes that were quickly and more extensive evaluated show this. I think there is no good or bad process in general, it really depends on the project and for this project a hybrid form between technologydriven and research-through-design was most suitable.

Significance Secondly, I was surprised to see a small effect in one of the results. During the tests I was convinced all the data points would be scattered. In user tests of my previous semesters I sometimes noticed the expected pattern, but with this project I was completely oblivious. In the end Iâ&#x20AC;&#x2122;ve learned that this feeling of truly now knowing is the right attitude for a researcher. It motivated me to continue gathering data and analyzing the results. Unfortunately, the significance was below the required level, but for me that doesnâ&#x20AC;&#x2122;t mean the hypothesis is invalid.

Lastly, I would like to thank Panos Markopoulos and Janpaul Verburg for the vivid and motivating discussions about this research subject.

5.2 bibliography References

Bau, O., Petrevski, U., & Mackay, W. E. (2009). BubbleWrap: a textile-based electromagnetic haptic display. Brown, M. L., S. L. Newsome, and E. P. Glinert (1989). An experiment into the use of auditory cues to reduce visual workload. Brewster, S. A., P. C. Wright, and A. D. N. Edwards (1994). The design and evaluation of an auditory-enhanced scrollbar.

Hugh D. Young & Roger A. Freedman, Sears and Zemansky’s University Physics with Modern Physics Technology Update, Thirteenth Edition

James, W. (1890; reprinted 1950). The principles of psychology. New York: Dover.

Coelho, M. & Zigelbaum, J. (2010). Shape-changing interfaces

Kwak, M., Hornbæk, K., Markopoulos, P., & Alonso, M. B. (2014). The design space of shape-changing interfaces: a repertory grid study. In Proceedings of the 2014 Conference on Designing Interactive Systems. (pp. 181-190).

Follmer, S., Leithinger, D., Olwal, A., Hogge, A., Ishii, H. (2012). inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation.

Porterfield, A. (2014). Curved Laser Bent Wood. http://www.instructables. com/id/Curved-laser-bent-wood. Visited on 4th of December 2014

Gaver, W. W. (1989). The SonicFinder: an interface that uses auditory icons. Human-Computer Interaction.

Schachter, S., & Singer, J. (1962). Cognitive, Social, and Physiological Determinants of Emotional State. Psychological Review, 69, pp. 379–399.

Gaver, W. W. (1993). What in the World Do We Hear?: An Ecological Approach to Auditory Event Perception. Ecological Psychology.

Schachter, S. (1971). Emotion, obesity and crime. New York: Academic Press.

Granta (2010), CES Edupack. Gray, P (2010). Psychology, Sixth Edition.

Houben, M. (2002). The Sound of Rolling Objects, Perception of size and speed

Spillers, F. (2004). Emotion as a Cognitive Artifact and the Design Implications for Products That are Perceived As Pleasurable.

Vroomen, J., & Gelder, B. D. (2000). Sound enhances visual perception: Crossmodal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance. Warren, Jr., W. H. & Verbrugge, R. R. (1984). Auditory perception of breaking and bouncing events: a case study in ecological acoustics. Journal of Experimental Psychology: Human Perception and Performance. Yao, L., Niiyama, R., Ou, J., Follmer, S., Della Silva, C., & Hiroshi, I. (2013). PneUI: pneumatically actuated soft composite materials for shape changing interfaces.

Sources

Visual 1.1.1 - BubbleWrap (Bau et al., 2009), PneUI (Yao et al., 2013), inFORM (Follmer et al., 2012) & Shutters (Coelho & Zigelbaum, 2010). Visual 2.1.1 - http://vimeo.com/94511974, visited on 11th of January 2014 Visual 2.2.1 - http://www.olivierbau.com/bubblewrap.php, visited on 11th of January 2014 Visual 2.4.1 - adapted from Gray (2010), pp. 223. Visual 2.6.1 - adapted from Gaver (1993).

Thank you for reading.