augmented memory/ personal process journal

Yara Al Husaini Memory

6

14

30

3

The Atomised Society

01

Memory

38

Concepts

02

44

04

Prototype

Memory Scapes

03

50

05

Reflection

06

4

Preface

In an exploration on digital culture, I focused on exploring the interaction space of digital materialities, IoT, and memory.

Establishing that IoT allows for physical and digital, the material and immaterial to be increasingly interlinked, and the critique that some sociologists of memory have surrounding digital ways of collecting and commemorating memories put societies at the ‘technologically produced forgetting’ with increased flows of information that reduce our ability to synthesise the past and present (McDonald R. et al, 2014).

What we might be facing in the future is a decrease in physical things (as people dispose of things), and a continuous existence of the collection and commemoration of memories that will need to have an alternative physical materiality as opposed to its original one, through what Chris Speed calls Blank Objects in his essay The internet of things that do not exist (Speed, 2011).

To this end, I have embarked on a journey of exploration on what such a ‘blank object’ might be within the context of individual and collective memories. I started off with the question: what if IoT were employed in the preservation and augmentation of meaningful memories through better interactions.

5

6

7

In this section I present desk research that I undertook on digital cultures, spiking an interest in the atomised society, and what that entails. From there I was able to see strands and connect the concept of the atomised society

to

memory.

‘Digitally-mediated

memories’, ‘Persistence of memory despite time’, ‘Digital shadow’ and ‘Digital vs physical

01

preservation’ are interpretations much like that of the atomised society.

8

9

Intro Technology is fast paced, it is being produced at a faster rate than we can fully understand its implication on culture and the world around us. Digital cultures have led to the fluidity and hybridisation where mass mediated experience is controlled by the interlocking of virtual industries, such as film, photography, music, and the arts.

Thus, leading to a flattening of hierarchies to a certain extent.

The Networked distributed reality of digital cultures is based upon its participatory nature, where the line between the producer and the consumer is blurred, however it has also led to segregated cultural niches. Through this reality, our society is undergoing the risk of a selfatomised society, where self promotion and disassembled meaningless interaction are pervading.

10

11

12

Digital Cultures Today all aspects of culture are influenced and mediated by computerisation, where the cultural process is weaved through the practices, values, and expectations of people as they interact with our contemporary networked society.

digital culture is conceived; participation, remediation, and bricolage.

As participants, we are active agents in the process of meaning making. We engage in remediation through adopting while simultaneously modifying and manipulating ways of understanding Society has shifted to a computer- reality. We are bricoleurs by reflexively mediated way of producing, preserving, constructing our own versions of this and understanding new media, reality. (Deuze, 2006: pp65) however, new spaces brought about by communication technology are not necessarily new forms of culture.

Digital cultures today is an expression of a highly globalised culture, where producers and consumers use of media forms an emerging value system and expectations.(Deuze, 2006: pp63) In his essay Participation, Remediation, Bricolage: considering principal Components of a Digital Culture, Deuze argues that digital culture can be said to be a process by which media is constructed, deconstructed, manipulated and assembled for media users to make sense of the world around them, and how we intervene allows us to adjust our perceptions and understandings accordingly, thus shaping and renewing the properties of media.

Within this framework he identifies three principal components by which

13

14

15

In this section I present desk research that I have taken on in this project in order to define an interaction space. In the early explorations of what I might be interested in I looked at connected object, memory objects, how we remember and forget, how we add meaning to memories with the stories we tell ourselves and others, and memory

02

triggers. I looked at the relationship between time, place, trace, emotion, and memory.

16

Persistence of Memory The overwhelming flows of information undermine our ability to synthesize the past and the present, leading to the idea of technologically produced forgetting. People used to filter through their memories, and reinterpret them and add new meanings by ridding of and altering physical objects, however with the digitisation of memories, people are increasingly unable to filter through the abundance of information and the form a sense of personal and collective identity. The physicality of materials through their decay plays a significant role in the creation of a contiuum between the past and present.

In Salvador Dali’s Persistence of Memory (Dali, 1931), memory persists as an organic, biological, psychological , and spiritual reality. Dali brought science, psychology, technology, religion, philosophical and spiritual subjects into his painting. He often used objects in his painting to convey

17

meaning; for example he used the dial phone to symbolise communication in his work.

In this particular painting, Dali depicts his hometown’s - Lligat - sandy beaches, corrupted by age sailboats, and other imagery he was exposed to in his hometown.

Posing the question of whether time is relevant at all, as if it is an illusion. Time does not persist, memories do. It is also believed that the sensory memory is what influenced his painting Melted Clock. (Anon, 2017)

On Mediated Memories 18 Our perceptions of our memories are not constant, memories are ever changing as those who hold them reinterpret them and modify them whenever they are revisited.

of these objects thus allow for a mental continuity between the past and the present.

For example, playing a decaying video tape to revisit a home video manifests the continuity in the process of memory production. (Van A grandmother reassembling photos of her Dijk, 2007: pp38) grandchildren is an example of the case in point. The process of remembering through Van Dijk (2007) defines mediated memories an object is not a mere process of a trigger as material triggers for future recall, produced by objects, our minds, brains, technology through media technologies, ranging from and materiality compose an intertwined a pencil to a camera. However, he argues process, that allows us to revisit our past. that they are also instruments of production The decay of the materiality of the object and communication. We use these objects thus becomes part of a mutating memory to define our identities within the vicinity of that is being constantly altered, through our immediate and larger surroundings. The the reinterpretation of the memory, and case of bloggers today present an example new meanings being assigned to it by its of the construction of the self identity and its beholder. relation the external world.

In Eternal Sunshine, Joel’s and Clementine’s desire to dispose their memories manifests our human desire to edit our past. A scene in the film at Dr. Mierzwiak’s waiting room, where people are holding on to boxes of memory objects they want to dispose of represent the innate modifiability of objects, and the reinterpretation of the meanings attached to them. (Van Dijk, 2007: pp37) Objects of Memory serve as a representation of the self and the past, and their physical existence acts as a stable anchor for memory retrieval, almost like an index of one’s lived experiences. The imperfection

Memories constructed through media are in themselves cultural and social tools by which a memory is constructed, preserved, and communicated to the self and to others.

By looking at the case of family holiday photography for example, people take photographs not only to capture the event for themselves, and share it with others, but also to reinforce the notion of belonging to a family. (Van Dijk, 2007: pp39)

19

20

Memory Triggers Memory is defined as the learning that persisted over time; information that has been stored and in many cases, can be recalled. The three stages of memory identified by neuroscientists are mainly encoding, storage, and retrieval. External events are perceived through the sensory memory, followed by a process of encoding.

Encoding is done through mnemonics, which help us remember us remember those events later on. The processing can be shallow, done through the chunking of information, or encoded semantically, where we make relation with our own personal experiences and emotions, otherwise known as the best way to remember something. The encoding process stores information into our short term memory, and by maintenance rehearsal, we are able to store those memories in our long term memory.

The retrieval of information, aka remembering, is a reconstruction and a reproduction of past events; we are largely a product of the stories we tell ourselves. Our mnemonics are stored in webs of association aided by retrieval cues and influenced by context and mood. (How We Make Memories - Crash Course Psychology #13, 2016)

21

Context-Dependent

Context based recalling of memories is based on specific episodes or information when the context present at encoding and retrieval are the same. A typical example of this is when an item is lost, and we retrace our steps to recreate the same context for information retrieval. Another example of this is when you need to write and can not find a pencil, you leave the room to find one, when in the new room you forget why you moved, upon return to the original room, you remember -and feel half-witted.

State-Dependent

State dependent recalling of memories is also known as the most efficient way to remember to something, as we are able to recall memories when we are in the same state of consciousness as the one we were in when the event occurred.

It is when memory is state or emotion dependent, so for example if we are sad, we become more likely to remember past sad experiences, when we are angry with someone we are likely to remember past experiences where we felt the same way.

Order-Dependent

Order dependent memory is based upon our ability to recall the chronological order in which we received information, however this phenomenon does not necessarily mean that we remember all of the information. We are likely to remember the first and the last pieces of information we received than those in the middle. An example of this is how we tend to remember the first and last items on a grocery list more than the items listed in between, -and that’s why you forget yogurt.

22

Story Telling

23

Definitions of storytelling vary widely, but it definition lies between storytelling being an account that must conform to a skeleton of criteria in order to be considered a story, and any subjective viewpoint on “what happened”. One definition that describes this is that “narrative deals with the vicissitudes of human intentions.”

on them; and they rehearse them during the composition of the story. (McGregor and Holmes, 1999)

The discourse around storytelling is that rather than it being a logical argument, it is largely based on that it makes abstracted truth claims. Stories aim for compounded believability, based on the idea of “that could happen”.

Storytelling also causes the release of dopamine in the brain when an emotionally charged event occurs thus aiding in memory retrieval with greater accuracy. Another phenomenon that relates brain activity to storytelling is that neuro coupling allows people to re-experience the story by relating it to their own ideas and experiences.

(Lifehacker.com, 2017)

This is related to the encoding processes that happen during the experience of an event, rather than retrieval. Participants in storytelling are likely to “spin” evidence by picking on certain pieces, and telling them through their own perspectives, and thus generate biased meanings, they elaborate

When looking at the brain activity caused by storytelling, we see that the cortex activity engages several areas in the brain allowing people to remember.

24

Trace Traces are an interesting phenomenon in the notion of memory as they lie between the sensible and insensible. They can approached as process, either that produced by nature or man; a sign, a trail that tells a story that has been completed, and holds cues for the future. Traces are not something that necessarily remains, they might serve as a reference that is not present.

The difference between material and immaterial traces is difficult to pinpoint, they interlock and interconnect, and therefore must be addressed in tandem and correlatively; the material becomes meaningless without its immaterial companion, and vica versa. Simultaneously, traces serve as a tool for the restoration of memories. The trace of an event or act can be either material, such as footprints, people, signs of wear and tear etc, or intangible, such as experiences, thoughts, memories, time, perception, senses.

More specifically, we are unable to experience time without the material world. We are able to perceive the passage of time through its effect on people and places. Our memories are stored in places through the senses and the objects. The coexist to form an identity that not only defines our past and who we are individuals and societies, but also directs what we do now, and in the future. (michailidis, 2014) A rather artistic endeavor that I took on was to think of what trace representation might mean using watercolors, that is to help me articulate my thoughts, together with the engagement I created, I directed my interest towards creating a prototype that speculates upon how we might experience connected objects in the provocation of embodied meanings through memory.

25

26

& Place This has inspired me to take an artistic approach to thinking about memory, which has also provoked a phenomenological way of thinking about memory and experience and how this might be implemented in a prototype that will play on triggering the memory through meaningful triggers.

27

28

Memory Individual

through Media & Connected Objects

Storytelling

Collective

29

After this research I decided to focus on carrying out an exercise that would help me direct my research further, as well provide grounding to the desk research carried out up to this point. In this exercise I aimed to test the idea of memory triggers through the use of media and see where that carries my research.

30

31

In this section I present an exercise that I conducted on soundscapes as a trigger of memory. In the exercise I chose familar sounds, asked participants to illustrate their memories, and followed up the exercise with interviews. This helped me focus on a narrower direction to test in an interactive

03

prototype where I would utilise the findings from this exercise.

32

“The whole experience is etched in my memory, we walked all the way up to the mountain, and in the middle of mountain she stopped pulled my sweater’s zipper up, and kissed me for the first time”

33

“This a memory that brings me peace inside. It was the first day of the new year, my friend and I were sitting at the beach waiting for the first sunrise, and that’s all I’m going to say”

34

“My memories of water bodies are so intermingled, but all I could think of listening to the sound of hard waves, was a moment in time, when my boyfriend and I were sitting, listening to hard waves, and they were so loud and profound, we could not speak.�

Engagement tool The aim of this exercise was to carry out my research in a way that felt personal to the participants. A question based research where I asked about their favourite, happiest, saddest memory would have only resulted in, I do not know, nothing comes to mind right now, and so instead I aimed to design my engagement around two factors, one, is creating a trigger to conjure memories to mind, and the other is a manifestation of the memory as the sound played in the background. The set-up allowed for the participants to feel comfortable as I provided them with the space and time to be on their own, listen, remember and manifest. Given that the way I designed the engagement tool provided the participants the means to be exploratory, creative, and the ability to make their contribution tangible, they were left with a happy feelings about the exercise.

I chose to use sounds from our environments without trying to direct the experiment into a certain type of environments, However I chose familiar sounds. half of the sounds I chose represented the sounds of natural energies around us, the other half from daily life. The participants found the exercise to be fairly emotional, and we concluded together that this might be due to the intensity of the natural sounds experienced brought about the deepest emotional lived experiences.

35

aug men

In my interaction space I look at how objects, senses, and media used to augment a familiar environment could act as a memory trigger together with the participants behaviour and emotions, and how different this might be when done individually or with others. In testing I hoped to see how people would behave, the associated experiences that would spring to mind, and how they would express them

Memory Trigger

Objects, Media, & Senses

on m ent

Physical Behaviour & Emotional State

n vir

participant

a re

Individual

af

ili a

36

ti n g

m

ag ing

l l i ng ’

e ng

r yt e o t ‘s in

Collective

Interaction Space

37

three users air

sea

land

air

sea

land

one user

installation

activate

sensor

sound

38

39

In this section I present some of the concepts I worked on to get to the final one, these were not all conceived at the same time, as I came up with concepts as I was researching, the design process was heavily based on an internal thinking a direction process, whereby I used concepts to help formulate the final

04

interaction space that I wanted to prototype and test.

Sketches 40

I started off thinking of concepts that would help people tell a story and convey it to others, this is seen in the first three sketches from left to right. The concepts here consisted of an interaction between an individual and others, whereby a trace of story is left from someone to hear, see, etc and respond to.

An example of this is a puppet show that is initiated by participants, recorded, and allows others to see it, interact with it, and add their own stories. The aim of this was to see how storytelling -to oneself and others- affects memory. Another example is using soundscapes to tell the story of a place or an event.

The second series of concepts focused on how I could create an interaction that limited the information received by our environment and augmenting our senses by zooming in one, thus aiding in creating a specific memory trigger.

The Third series of concepts was about creating an interactive installation whereby personal memory objects become connected to that of others, thus seeing how these objects connect to each other beyond their actual function, and focusing on how they connect in the way that people interpret them to be . For example a shoe and a car could serve as reminders of the same person from two individual’s perspective.

The last series of concepts was around creating a an interactive experience that would be in itself a trigger of past experiences or memories of the beach, a familiar place, that most people have a deep connection with, as it is one of the rare places where we become connected to the natural environment. This would be an installation in a public space. By utilising technology and design artefacts I hope to see how sounds and motion triggers affect the memories that come to mind.

41

42

After much thinking about the kind of interaction I want to create, I decided to use land, sea and wind as the triggers of memory through sounds and physical movement. The main idea of the concept is an interactive installation placed in a public space. The installation consists of 3 main frames which the user is invited to be “inside�, each frame represents an element of the seaside environment, land, sea, and air. When the user is in the frame, sound relating the specific element is played.

I chose land sea and air as the main elements to provoke memories is because I think that these connect us physically and emotionally to the world around us- something that we have not recreated through the use of technology just yet. I think that this is due to the fact that technology is not used to create truly immersive experiences, but rather distracting ones.

The interaction is meant to provoke a conversation around how emotion, motion, sound, space and time interplay to trigger memories that otherwise would not surface. Through this interaction I aim to speculate on how connected objects and the way we use technology could be utilised to create a meaningful interactions for people.

43

44

45

05

Making 46

I tried to bring some of the ideas from my research and test them through an interactive prototype.

In this prototype, I designed an installation where there are material triggers, which link closely with a familiar environment - the seaside-, and separated it into three levels to augment the experience of the environment. The participants physical behaviour and emotional state would act as the subjective trigger. Whether the memories are triggered or not remains an open question that the participant can think about, tell a story to themselves, or others that are interacting with them in this installation.

To realise my concept I started out with a sketch, designed the main interactions which would involve triggering of certain sounds related to the specific area in the installation. I decided to make it so that different sounds are triggered when more than one person is using the intallation to provoke them into starting a conversation about the experience that they are having. Thus giving participants to share stories of how they interpret this interaction, and how it relates to their own experiences of the seaside. Based on these decisions I decided to use arduino, distance sensors, and an mp3 shield to create the interactive part of the installation.

In building the prototype I had to use basic tools as it was relatively simple to make. In its first iteration however I found that people found it difficult to bend underneath the frame to get to the third zone, so I deconstructed it, and made it at a more reasonable height. I also faced much difficulty with getting the code to work, which was very time and energy consuming, thus I seeked the help of a friend. This made me realise that I should have been fideling with arduino at a much earlier stage to give myself time to learn it and be able to spend more time on improving the interaction rather than the code or components.

47

48

Testing In order to test my prototype with participants I installed my prototype at our campus gallery space where students and faculty members were able to interact with the installation and provide me with feedback.

For purposes of both analysis and presentation I filmed participants with their consent as they interacted with the prototype. They often forgot that I was there and generally enjoyed using it.

Almost everyone found it particularly fun to play with, especially when more than one person where ‘playing’ together. I use the word play here because this is how participants expressed what they were doing. The element of fun in the prototype was not really something that I purposefully designed for, but emerged through testing.

In small conversations following the exercise taking place, I asked participants how the experience was for them, their sentences usually started with, ‘it reminded of when’ or ‘it reminded me of how’..

One response that I would like to highlight here is “it reminded me of a day when I carrying my bike on my shoulder on the beach, and I kept hearing different sounds that my bike was making as a result of both my movement and the wind. I think the way the sounds started and stopped because of my behaviour might have triggered that’ Following testing the prototype with participants I created a short video that explains the installation and shows how participants interacted with it.

I found it to be the best tool to communicate my findings.

I think the next iteration for this prototype would be to put the sensors in different places hidden from sight, and see how that might affect the behaviour and the memories that trigger.

49

50

Reflections

This project has taught me a lot, not only about the discipline of interaction design, but also about my preferences. This is the first research project I undertake on this course, and I believe it has helped me significantly in growing more independent as a researcher and a designer.

As students we were lucky to be given the freedom to choose a topic that interested us, without any constraints on how specific or broad the topic we choose is. This was very challenging, especially midway through the project when I was trying to find a ground to all the abstract ideas and research that I did. However I still did want to keep my project at an abstract level, and did not want to divert to a specific context, as this is what I have been trained to do so far. By specific context I mean choosing a very specific problem on a local level.

This project reminded that with my background as an industrial designer I enjoy creating artifacts and interactions that provoke people, and I think that this an area in my practice that I want to develop further in my next project.

Another learning from this project is to start prototyping at an earlier stage if possible because the second, the third, and fourth iterations of a prototype take the project a step further every time.

06

I look forwards to applying the learning outcomes, gaps and challenges faced in this project in my thesis project, and refining my practice further through continuous reflection, a practice that I managed to do on this project through writing.

51

Bibliography Anon, (2017). The Persistence of Memory by Salvador Dali - Meaning, Interpretation, Analysis. [online] Available at: http://www.authenticsociety.com/about/ ThePersistenceOfMemory_Dali [Accessed 6 Mar. 2017].

Cowling, M., Tanenbaum, J., Birt, J. and Tanenbaum, K. (2016). Augmenting reality for augmented reality. interactions, 24(1), pp.42-45.

Deuze, M. (2006) ‘Participation, remediation, Bricolage: Considering principal components of a digital culture’, The Information Society, 22(2), pp. 63–75. doi: 10.1080/01972240600567170.

Lifehacker.com. (2017). Cite a Website - Cite This For Me. [online] Available at: http://lifehacker.com/5965703/the-science-of-storytelling-why-telling-a-story-is-the-mostpowerful-way-to-activate-our-brains [Accessed 6 Mar. 2017].

How We Make Memories - Crash Course Psychology #13. (2016). [video].

MacDonald, R., Couldry, N. and Dickens, L. (2015). Digitization and Materiality: Researching Community Memory Practice Today. The Sociological Review, 63(1), pp.102-120.

McGregor, I. and Holmes, J. (1999). How storytelling shapes memory and impressions of relationship events over time. Journal of Personality and Social Psychology, 76(3), pp.403419.

Michailidis, K. (2014). Memory and Reminiscence: Traces in the city, pp.5 - 8.

Speed, C. (2011). An internet of things that do not exist. interactions, 18(3), p.18.

Van Dijck, J. (2007) Mediated memories in the digital age. United States: Stanford University Press.

Mdes Design Innovation

Glasgow School of Art 2017