Thesis-<Face to Face>-Non-linear narratives applying facial interaction by Ningli Zhu

FACE TO FACE

non-linear narratives applying facial interactions

Ningli Zhu <non-linear narratives applying facial interactions> Graduation Thesis, 17th, May, 2019 Thesis tutor: Annemarie Quispel Personal tutor: Joris Landman Thesis submitted to the Master Institute of Visual Cultures, St. Joost School of Art & Design, sâ&#x20AC;&#x2122; Hertogenbosch In partial fulfilment of the requirements for the degree of Master of Arts in Fine Art & Design

Abstract This article discusses how virtual interactions, especially facial responsive applications on the computer, provide a sense of compensation and fictional self-perception for users. Combined with social-psychological research on both human-human and human-machine interactions, I question and reflect on the humanness in relationship to machines in this AI technology-driven age. Besides, the included design project, <face to face>, driven from my social anxieties in communication, is an attempt to apply real-time facial sensors into a self-reflective narrative video. The video installation presents a mirroring effect on the audience with facial recognition technology. While, the whole project provides a personal perspective on the topic of relations between human, machine and digital technology.

Contents 4 Preface 5 Design statement Chapter 1. Introduction 6 1.1 Motivation 6 1.2 Aim and communication goal 7 1.3 Research question 7 1.4 Research method 7 1.5 Research process Chapter 2: Technological trend and its influence 8 2.1 Facial biometric technology in industries 10 2.2 Cyber Effect on mental health 11 2.3 Emotion technology and virtual products Chapter 3: Theory research 12 3.1 Facial interaction in social communication 13 3.2 Facial Action Coding System 14 3.3 Self-presentation in social and virtual world 15 3.4 Understand Cyberspaces 16 3.5 Relations between Human, Machine, and Technology 17 3.6 Feedback-loop model 18 3.7 Compensation in digital and virtual world 18 3.8 Reflection and conclusion Chapter 4: Case studies 20 4.1 Topic related artworks 24 4.2 Video Essay Chapter 5: Graduation project 26 5.1 Introduction 28 5.2 Concept for Storyline 30 5.3 Narrative Strategy Chapter 6: Design process 32 6.1 Scriptwriting and Autido effect 36 6.2 Test facial expression responsive interactions 40 6.3 Moodboard and visual sketches 46 6.4 Further developments 48 6.5 Testing and Feedback 50 6.6 Final design 56 Reflection and Conclusion

Preface In modern society, our social communication has become more and more a machine-mediated dialogue. People may feel more lonely and isolated, because of less physical (faceto-face) interactions. However, because of the developing Emotionally Intelligent AI technology, our empathy and emotional feedbacks with digital avatars or virtual human are becoming more real with the algorithm system. Like the film <her> (Fig. 1), in the future, we might build emotional connections with the virtual products to get mental supports or psychological compensation. Digital interactions in Cyberspace reflect our social desire or loneliness in real life. Meanwhile, as we are being fed by feedbacks from the machine every day, our behaviors and perceptions are also being reshaped by the virtual world. We are embracing both intimate and conflict relationship with digital technology. The blended identity between offline and online, reality and virtual, makes us kind of a cyborg.

Fig 1. A scene from Film <Her>.

Design statement As a visual designer, I'm interested in storytelling about cultural identity and social relationships. I often use my own experience to tell stories which contain strong emotions such as loneliness, inner struggles. Besides, I like to integrate theoretical research and content into a form of speculative narratives, such as an essay film. Nowadays, people spend more and more time on digital devices. â&#x20AC;&#x153;What shaped the intimacy between us and our device?â&#x20AC;? â&#x20AC;? To open up discussions, I develop my own story to communicate my social anxieties in real life and to reflect on the influence of digital technology. I work and compose with different narrative elements, including scripts, audio monologue, visual metaphors, shootings, to build the video installation. Also, I apply real-time facial responsive interactions on the screen to engage my audience. While experimenting with different digital techniques and narrative strategies, I offer a personal perspective on humanness in relationship to machines.

Chapter 1. Introduction 1.1 Motivation Sometimes, I have some social anxieties in face to face communication, and find it hard to express myself naturally. Oppositely, I feel more comfortable and active when talking with people through a device. There was a period that I played with many applications that do facial modifications. I find the digital interactions and the behind programming system are quite interesting. After that, I began to consider how those facial technologies were applied in industries? How we present and identify ourselves in the virtual world? How is our intimacy with the machine and technology in the digital age?

1.2 Aim and Communication goal I aim to present and communicate with my personal story in a poetic self-narrative video, showing how my self-perceptions are extended through the digital interactions in the computational algorithm system. For the production part, I plan to build up an installation screening my story, which applies facial detection technology in non-linear narrative videos. The audience could participate in the project by playing with some real-time interactions.

1.3 Research question 1. What is facial and emotion recognition technology? Where is that used in real life? 2. How to apply emotion sensing in digital interaction? Which technical tools can be used? 3. What are the possibilities of facial responsive interactions on screen with the coding language? 4. How to understand emotion and facial expressions in different (social and virtual) contexts? 5. How to apply facial interactions in the non-linear Narrative video? 6. How to use certain strategies and effects in the video to make a personal narrative more touching or poetic? 7. What are the related philosophical and social-psychological theories on topics of human-machine and cyberspace?

1.4 Research methods 1. Testing the effect and possibilities of facial interaction technology and apply them into the project. 2. Building interactive installations to increase the physical experience in the exhibition and testing with the audience to get feedback and more discussions. 3. Case studies and analyzing how other designers approach similar topics. 4. Theory study for the related human-machine relationship and the Cyberspace topics.

1.5 Research process I divided my process into four steps: Step 1. Emotion Technology & Tools: Test facial expression responsive interactions. Step 2. Theory & Research: Understand facial expressions used in different (social and virtual) context; Research about the human-machine relationship, etc. Step 3. Design & Research: Case study; Test and sketch visual elements and footages. Step 4. Structure Narratives & Prototype: Write scripts and make storyboard; Test installation and video.

Chapter 2. Technological trend and its influence 2.1 Facial biometric technology in industries Facial detection and emotional recognition are the facial biometric technologies which are used to track facial expressions and movements of humans by the machine and to analyze facial data in related algorithm systems. It is related to many other disciplines, such as machine learning, computer vision, neural network. So far, facial biometric technologies have been widely used in many different fields and applications, for example, public surveillance (Fig.2), online content marketing (Fig.3), smile detection on camera, autonomous vehicles (Fig.4), , digital entertainment products (Fig.5), etc. Apart from its large use in surveillance by governments and military, Facebook, Apple, and many other companies have already established research and development teams to apply the same technology in their products. A notable case with Face ID is the application â&#x20AC;&#x153;Animojiâ&#x20AC;? (Fig.6) on iPhone X. Users can get real-time animate 3D face icon with different characters on screen and share it. However, at the same time, since those face tracking technologies and resources are partly open online, many designers, artists, and programmers also apply interactive facial technology in their projects, where I will mention in the next chapter.

Fig 2. Facial Recognition Tech for public surveillance

Fig 3. Affectiva video emotion analytics dashboard

Fig 4. Affectiva Automotive AI for Driver Monitoring Systems

Fig 5. ObEN, an artificial intelligence (AI) company creates Personal AI (PAI), an intelligent avatar.

Fig 6. Animoji Applictaion on iPhone X

2.2

Cyber Effect on mental health

Internet and digital technology have changed our communicational behaviors. Since our face-to-face conversations reduced by device-mediated dialogues, people become more and more isolated from each other. More importantly, this kind of isolations leads to many different mental health problems. Stanford University researchers have found that heavy digital multitasking and long screen time are related to poor emotion and social health1, including less social confidence, not feeling well, etc. Besides, some scientific articles have stated that smartphone technology arises obsessive-compulsive behavior, addiction and depression2. Dr. Mary Aiken, whose research is about â&#x20AC;&#x153;cyberpsychologyâ&#x20AC;?, argues all those impacts of technology on changing human behavior, especially on the growing youths in her book <The Cyber Effect>3.

1. Gorlick, A. (2009, August 24). Media multitaskers pay mental price, Stanford study shows. Retrieved from https://news.stanford.edu/2009/08/24/multitask-research-study-082409/ 2. Nichols, H. (2018, January 18). Obsessive-compulsive disorder: Symptoms, causes, and treatment. Retrieved from https://www.medicalnewstoday.com/articles/178508.php 3. Aiken, M. (2016). The cyber effect: A pioneering cyber-psychologist explains how human behavior changes online. 4. Gonzalez, R. (2017, October 17). Virtual Therapists Help Veterans Open Up About PTSD. Retrieved from https://www.wired.com/story/virtual-therapists-help-veterans-openup-about-ptsd/

2.3 Emotion technology and virtual products Digital technology can benefit our lives by building emotional connections and virtual relationships. One case is the virtual games which offer mental compensations in certain ways. For example, <second life> (Fig.7), which provides an immersive social space for people including those with physical or mental disabilities, to find comfort and security interacting through anonymous avatars. Moreover, there is an increase in the use of all kind of media technologies (apps, VR) to diagnose and even treat mental health problem. Scientists are trying to help solve mental problems with developing AI and Emotion technology. They train a machine to interpret the emotional state of an individual and adapts its response accordingly in order to enhance the dialogue between the individual and the machine. For example, a virtual therapist (Fig.8) which is an artificially intelligent avatar rendered in 3D, is used to talk with people who have symptoms like depression and post-traumatic stress4.

Fig. 7. The comunity in the online virtual world <Second Life>.

Fig. 8. SimSensei - a virtual human platform specifically designed for healthcare support, developed by The USC Institute for Creative Technologies.

Chapter 3. Theory research 3.1 Facial interaction in social communication I read some psychological articles to understand more about how people use facial interactions in social life and communication. First of all, there is a “facial feedback hypothesis”5 developed after some studies by Charles Darwin. This theory states that facial movements can influence emotional experience. Also, the interactions between two people contain empathy and emotional feedbacks. That is why we establish social connections immediately with others in real life. I think this (facial) mirroring effect can be an angle for me to explain my story and discuss self-perception. It is also a good metaphor used to talk about reflection and identity. Secondly, as it is discussed in the article <Facial Displays as Tools for Social Influence>, “ faces do not always reveal our moods and emotions. Instead, they are about where we want a social interaction to go” 6. It is quite reasonable that we always express ourselves in certain ways to get a good impression from the audience. However, depended on personal cases, some introverted people might not feel as comfortable as others.

5. Andréasson, P. & Dimberg, U. J Nonverbal Behav (2008) 32: 215. https://doi.org/10.1007/ s10919-008-0052-z 6. University of California - Santa Barbara. (2018, March 27). Facial expressions as tools for social influence. ScienceDaily. Retrieved May 11, 2019 from www.sciencedaily.com/releases/2018/03/180327132019.htm 7. Facial Action Coding System. (n.d.). Retrieved from https://www.paulekman.com/facial-action-coding-system/

3.2

Facial Action Coding System

So far, most facial biometric technologies are based on a system called “Facial Action Coding System” (Fig.9). Described as, “It is a comprehensive, anatomically based system for describing all visually discernible facial movements. Based on this system, emotion recognition software can analyze our emotions by deconstructing our facial expressions into temporal segments that produce the expression, called Action Units (which is developed by prof. Paul Ekman) and deducing them into certain percentages of six basic emotion: happy, sad, angry, surprised, scared, and disgusted.7” I also find some non-commercial sources and techniques online to make facial expression interactions by myself, applications such as FaceOSC (Fig.10), Face Tracker. These applications can record and send inputs of facial data to software, such as processing, Max/MSP, openFrameworks. After many experiments with these techniques, I realize that the facial data could transform into any outputs by an algorithm written language, like music or mechanical movement.

Fig.9 A imgage of FACS Action Units

Fig.10 FaceOSC Application used with Processing software

3.3 Self-presentation in social and virtual world We, as individuals in the social community, always like to present a better or best self to others. This common "Self-presentational behavior“ shows not only in real social life but also in online environments. In the book <Presentation of self in everyday life>8, Erving Goffman discusses that our activity in front of the different audience is kind of a “performance” in a theater. For me, it is interesting that he uses the “mask” as a metaphor to talk about the performed self in face-to-face social interactions. While, In the virtual world, this behavior turns to use certain digital imagery to show self-identities, such as emojis, or online avatar. It can be explained by “The Proteus effect9” which states that our online behaviors and expectations are strongly associated with the characteristics of the avatar we use. I think, broadly speaking, the subjective visual outputs created by a designer or an artist in digital platforms could also be partly seen as the embodiments of the self.

8. Goffman, Erving. (1959). The presentation of self in everyday life. Garden City, N.Y. :Doubleday. 9. John R. Suler. (November 2015). Psychology of the Digital Age. Rider University, New Jersey. 10. Suler, J. R. (2016). Psychology of the digital age: Humans become electric. New York, NY: Cambridge University Press. 11. Foucault, Michel. "Of Other Spaces: Utopias and Heterotopias." Rethinking Architecture. Ed. Edmund Leach. London: Routledge, 1997. 12. Sherman Young. (1998) Of cyber spaces: the Internet & heterotopias. M/C: A Journal of Media and Culture 1(4). <http://www.uq.edu.au/mc/9811/hetero.php>

3.4

Understand Cyberspaces

Cyberspace, which to describe the online-digital environment, is quite a psychological space. As John Suler elaborates that “The speed at which information is processed enables a prolific capacity to alter, amplify, isolate, and combine in unique ways the Eight Dimensions of Cyberpsychology Architecture –the identity, social, interactive, text, sensory, temporal, reality, and physical dimensions.10” “The Cyberspace is therefore a very unique kind of psychological space, a space “out there” that is an extension and blending of human minds, a space where these minds can experiment with words, images, time, interpersonal connections, self-representation, and influences on the physical body and world never before possible in human history.” —— John Suler, <Digital Age: Humans Become Electric>

Apart from that, <Of Other Spaces (Heterotopia)>11 by Michel Foucault offers a more philosophical view to look at the Internet and Cyberspace in relation to psychology. In his theory, cyberspace is a “heterotopic” space that contains compensation and reflects the constitution of social human relationships. I agree that, as Foucault discussed, the Internet and digital platforms as "Heterotopias" (space) are not illusory but more real in a way, because people can more easily pursue an ideal identity which might be complementary to the real one. His words about virtual space recall me of a masterpiece, film < A space Odyssey> (Fig. 11), an example of fictional visual storytelling. Foucault also took a mirror as an example. He discusses that “A mirror, is at the same time a utopia and a heterotopia. On the one hand, a mirror is a place without a place, and on the other, it is a real place.12”

Fig 11. A "Slit-Scan" effect about Space Travel, in film < A space Odyssey >

3.5 Relations between Human, Machine, and Technology Donna Haraway’s book <A cyborg manifesto> expanded my view on the human-machine relationship. What inspires me most is the modern study of cyborg politics from her book. “Cyborg”, as human-technology mixtures in the abstraction, is a strong point that I want to discuss. As D. Haraway says, “How we interact with machines and technology in many ways defines who we are. The physical attachments humanity has with even the most basic technologies have already made us cyborgs. Anything that is an external prosthetic device creates one into a cyborg13”. “I have come to see cyborgs as junior siblings in the much bigger, queer family of companion species...14” We are all cyborgs nowadays, as we use technology to augment our cognitive processes, such as VR, digital games. Besides, Haraway mentions that “The cyborg politics is the struggle for language and the struggle against perfect communication, against the one code that translates all meaning perfectly…15” Refering to real life, as we spend a lot of time on digital conversations and social media, many young people feel a struggle for better communication and self-identity between online and offline. Here, the word “struggle” actually shows our relationships between “inner space” to “outer space”, “self” to “others”. Digital technology offers us a reflection, as the article <A dream of an Algorithm> describes, “The virtual has transcended my perception of myself and my life. Technology becomes a tool for my thinking and seeing things around me. The boundaries have become vague; it’s no longer clear where the human stops and the technology starts. There is a blurred area where me, myself, technology and the rest of the world merge together. 16” These provoking words drive me to think about how I can develop my own story in scripts.

Fig.12 <Simians, Cyborgs, and Women>, Donna Haraway

3.6

Feedback-loop model

“Feedback-loop model”17 is universally implicated in both our social communication (Fig.13) and machine interaction, because all kind of communication is a circle of “act-respond” or “callback-react”. In human activity, this is called biological feedback, our social behavior in both our body and brain. While, in the mechanical, algorithm system it is related to the transformation of data with inputs, which is a “C3 system– Command, Control & Communications” (Cybernetics). It implies “the science of control and communication, in the animal and the machine”18. For example, computer games offer players stimulation in a virtual world, which builds up a feedback-control loop as interactions.

Fig. 13. A diagram for Communication loop

13. Cyborg. Retrieved from Wikipedia wbsite: https://en.wikipedia.org/wiki/Cyborg 14. Haraway, Donna Jeanne. (2003). The companion species manifesto : dogs, people, and significant otherness. Chicago, Ill. : Bristol :Prickly Paradigm ; University Presses Marketing, p.11 15. Donna J. Haraway. (1991). "A Cyborg Manifesto: Science, Technology, and Socialist-Feminism in the Late Twentieth Century, " in Simians, Cyborgs and Women: The Reinvention of Nature (New York; Routledge), pp.149-181. 16. Agnieszka Zimolag. (2016).A dream of an algorithm. 17. A feedback loop is a system structure that causes the output from one node to eventually influence input to that same node. It was applied broadly on different complex systems, such as Digital software, Mechanical engineering. 18. W. Ross Ashby, An Introduction to Cybernetics, Chapman & Hall, London, 1956. Internet (1999): http://pcp.vub.ac.be/books/IntroCyb.pdf

3.7 Compensation in digital and virtual world “In psychology, ‘compensation’ is a strategy whereby one covers up, consciously or unconsciously, weaknesses, frustrations, desires, or feelings of inadequacy or incompetence in one life area through the gratification or (drive towards) excellence in another area. Compensation can cover up either real or imagined deficiencies and personal or physical inferiority. "19 As Foucault describes, cyberspace contains compensation. Social relationships and reality drive our online behaviors.

3.8 Reflection and conclusion After these theoretical research mentioned above, I realized that digital technology offers an extension of a mental-self, a virtual-self. So, In my project, I hope to primarily discuss "otherness", which shows the relationship between two parts, "myself" and "others", or "inner space" and "outside space". The "others" refers to interactions with people and the computer. They are both like a "mirror" which offers "reflection" of me.

19. Compensation. Retrieved from Wikipedia wbsite: https://en.wikipedia.org/wiki/Compensation_(psychology)

Chapter 4. Case studies 4.1 Topic related artwork # Zach Blas <Face Cages> <Facial Weaponization Suite> His idea of facial detection technologies as “a cage of information” offers me a political perspective to look at this technology. Also, these “facial cages” with sharp lines contains very powerful visual language and straight political message. His work <Facial Weaponization Suite> consists of “collective masks” that cannot be detected as human faces by biometric facial recognition software, which was designed to visualize the way computers mathematically understand and read biometric data. In that project, he criticizes the standardization and abstract violence of those technologies.

Fig. 14 <Face cages>

# Coralie Vogelaar <Interface> <Emotions from an Algorithmic Point of View> She developed a series of work about facial emotion and detection technology. In her project, she uses film, performance, infographics to present research. Also, she mainly uses the portrait of a performer to construct the whole visual narrative, which looks quite consistent in all parts of the project. <Interface> ( Fig. 16) "In this two channel video work a movement analysis of emotions is being done by deconstructing it into temporal segments that produced the expression. On one screen emotions are being expressed by actress. On the right screen the observed components of facial movements are re-enacted the day after. The result is a non verbal exchange between computerized movements and their corresponding emotions, in which the detection technique is triggered by the emotions but also the other way around."

Fig. 15 <Emotions from an Algorithmic Point of View>

Fig. 16 <Interface> -- A research on emotions by pattern recognition

Chapter 4. Case studies 4.1 Zach Blas < Face Cages > < Facial Weaponization Suite > In his project, he criticizes the standardization and abstract violence of those technologies and design with new computational biometric (facial) diagram. He transformed the digital diagrams to physical metal masks which can be worn by different minoritarian people. His idea of facial detection technologies as “a cage of information” offers me a political perspective to look at this Fig. 17 <Face Projection> technology. Also, I think these “facial cages” with sharp lines contains very strong visual language and straight message. His work <Facial Weaponization Suite> consists of “collective masks” that cannot be detected as human faces by biometric facial recognition software. The work was designed to visualize the way computers mathematically understand and read biometric data, which shows the problematics of digital standardization.

Fig. 18 <Face Instrument>

# Daito manabe <Face Projection> <Face Instrument> ''His practice is informed by careful observation to discover and elucidate the essential potentialities inherent to the human body, data, programming, computers, and other phenomena, thus probing the interrelationships and boundaries delineating the analog and digital, real and virtual.'' ------ biography, Daito MANABE I find his projects of face projection and other facial interactions with new media technology interesting and playful. These work show me lots of possibilities of interaction design in a performative way.

Fig.19 <Fragmented Self>

<Fragmented Self>, 2014 UWM Arts, by Wyatt Tinder It is an interactive installation that allows viewers to navigate the artist's portrait. In the exhibition, a webcam tracks movements of viewers on a grid below and motion-tracking software triggers corresponding videos to start and stop. It is an example for me to think about applying physical interactions into a video display, as well as the use of portrait image in my narrative.

Fig. 21 <How Not to Be Seen>

Fig. 22 <Factory of the Sun>

4.2 Video Essay Since I consider to develop a fictional and poetic visual storytelling in my video installation, so I learn from a narrative form called "video essay" ("Essay Film") which is the video with content that, much like a written essay, advances an argument or open a discussion. The work from Chris Marker is classical masterpieces (Fig.20) in this kind of storytelling. I also looked through artworks from artists like Hito Steyerl (Fig.21-22), Metahaven (Fig. 23). Their speculative Film Essays, or Postcinematic Essays, are cases which connect visual practice and theory in experimental animations. The narratives are unique, socially engaged, reflexive about technology and digital practice with a range of digital techniques, visual collages and computer-generated effect. At the same time, those visual languages and metaphors are quite strong and attractive.

Fig. 23 Film <POSSESSED (Trailer)>, Metahaven

Fig. 20 <Editing In Storytelling | CRISWELL | Cinema Cartography>Video, from YouTube, https://www.youtube.com/ watch?v=CvLQJReDhic

Chapter 5. Graduation project 5.1 Introduction Stemming from some communication anxieties in social life, I’ve subconsciously developed a fascination for facial responsive interactions on screens. Based on this experience, the design project <face to face> is an autobiographical story, which talks about my struggles on self-presentation and social communication, while at the same time how the digital interactions on a computer relieve me from the social anxieties and offer me a sense of intimacy and connection. The work provides a personal angle on discussing the human-machine relationship. The final work is a series of videos on two screens, which tells about social and virtual reality. The scenes in videos switch between the social and virtual world. One screen mainly shows scenes in real social life, for example, face to face conversations; Whereas the other screen shows digital environments and interactions on a computer, including codes, facial data, 3d model. In the story, the relationships between“myself” and “others”, “human” and “machine”, “inner space” and “outside space”, defines me as a blended character like a "cyborg". For the final exhibition, I apply the real-time facial interaction on one screen for the audience to play.

5.2 Concept for Storyline In the video, I want to talk about self-reflection in social relations, to show my longing and struggle for sincere communication, authentic connection, and a better self-representation. So, my idea is to compare facial expressions used in two contexts (Fig. 24), which are social interaction (social communication) and computer interaction (virtual environments). Correspondingly, my storyline is based on these two kinds of interactions in social and virtual space. I consider the face portrait and mirroring effect are the main visual elements in my video. In the beginning, I use my face image and a digital avatar as a sketch for two screens (Fig. 25). I choose the "mask" and a "facial diagram" as the visual metaphor to develop further.

Fig. 24 Concept for making storyline

/MASKED FACE/ social communication struggle

/EMOTION TRACKING/ control diagrams

Fig. 25 Visual sketches

5.3 Narrative Strategy Two-screen non-linear narratives The split-screen techniques were usually used in filmmaking. In my case, I used two screens to help me structure my storyline and build subtle connections between social and virtual interaction. It offers Human and Machine two perspectives (self-vision & machine-vision) in my video. Apply facial interaction in narrative In order to increase the users’ experience, I decided to apply real-time physical interactions in my video. It helps to build up a active and strong connection with the audience. Also, it is important to test interactions with viewers to know if they would understand the content easily in this way or how they feel through it. Monologue in narrative To show a personal and reflective story, I wrote scripts in a form or language like a diary to express thoughts. Poetic and Fiction I want to make my video in a more poetic style through montage effects, and link the related techniques to “essay film”, which is used in film criticism to describe a self-reflective and self-referential documentary cinema that blurs the lines between fiction and nonfiction. So, I apply imagery such as screen reflection, slit-scan transitions in my video.

Chapter 6. Design Process 6.1 Scriptwriting and Audio effect I first focused on developing the scripts which could outline a clearer structure. In the beginning, I wrote texts in two perspectives which are "human-human" interaction and "human-computer" interaction. After many trials, finally, I put the narrative into three chapters, and each section divided into sub-content as below: Chapter 1: Opening: Scenario introduction; * I prefer mediated communication * I am not good at social communication Chapter 2: Main part: self-narrative and facial interaction; * Emotion detection (data flow) * Facial expression interaction (Algorithm system) * Virtual space (3d modeling) Chapter 3: Conclusion and reflection: self-identity. * Mirroring effect/ what makes me human? * Cyborg in a struggle for identity In order to lead the audience into a personal story, I found it necessary to write the story in a first-person tone to support the diary likes style, rather than an academic or descriptive style. So the story starts with a scene that I am chatting with friends on the device, “The notifications from Wechat groups constantly flash on my phone, one after another without a pause. I slash open the screen again to check the message…” Audio effect To have a more articulate spoken voice in my video, I collaborated with a friend to test on different tones. I have tried many kinds of effects when editing audio materials and added a robotic effect on the speech to enhance the viewers' feeling about the story.

Script

Chapter 1: Opening: Scenario introduction The notifications from Wechat groups constantly flash on my phone, one after another without a pause. I slide open the screen again to check the message. Today, as usual, I was chatting with many friends. We roast about the little pieces in our life. But now, I can’t really recall any meaningful words. Maybe none of those screening bits really get into my heart. Even so, I kinda prefer this way more than a face to face conversation. I am not very social in real life. Sometimes, I get stressed easily when facing a group of people in conversations. I care about their responses and fear their judgments. I can’t help but censor every word and reply with cautions. Gradually, I lose the confidence to show any expressions on my face. My face becomes very stiff as if I am wearing a mask. I want to be confident; I want to be cheerful; I want to hide my fear and worry; Unconsciously, I start to capture the details from my environment, the subtle movements from my audience. From their eyebrows, the spirits in their eyes, to the corner of their mouth… I imitate the similar movements from their faces, to create a sense of connection and belonging. In those moments, my body seems to be no longer a device for expression, rather, an information processing machine. Do I become a robot? Or if I am still a human?

Chapter 2: Main part: self-narrative and facial interaction As a daily routine, I work on my computer for many hours. I’m familiar with its appearance and every slight touch on it. Every time, I feel an extended space between us, a blurred area where me, technology and the rest of the world merge together. Shapes, grids, colors, pixels, codes… these are my languages, to build up a sense of connection to this world. The camera captures me on the screen. In machine vision, I was an “object” projected on an XYZ Coordinate. My face is tracked as an encoded mask; All the lines in this diagram change with my subtle facial movements. It looks a bit weird, while also fun because I never look at myself in this way.

Within this algorithm system, my smile transforms into a circular shape; The waves of my eyebrows change the color of the screen, the melody of the music. The system is training my facial muscles, or more properly, I am sensing and exploring my own physical body, in a way I have never experienced before. This strong sense of connection with an embodiment of myself is a bit fascinating for me. Even though I can not tell if I am controlling the system or the computer is controlling my movements. Maybe we have already merged into each other as one hybrid being through these feedback-controlled loops.

Chapter 3: Conclusion and reflection: self-identity. The virtual has transcended my perception of myself into fiction. In this heterotopia, I can create a second self or replicate hundreds of me. I become a dynamic model, floating in this endless â&#x20AC;&#x153;outer spaceâ&#x20AC;?. I am exploring an infinite self in the pool of my own imagination. Perhaps my computer, is more than a tool for me, it builds my language, it shapes my mind, it reflects my desire, more so, it became my close partner, or even my extend prosthesis. I look at my slight shadow on the screen and start to observe it as if I am a stranger. This mirroring figure seems a bit unfamiliar at this moment. Subconsciously, I stretch a big smile to her and lost in my thoughts. I wonder if I have been a kind of Cyborg. A cyborg between human and machine, mind and body, reality and fiction, self and others. I define myself with the response from others; I check myself against my figure on the screen; I edit myself into an illusion in a computer; I feel the struggles, the struggle sfor deep communication, for authentic relation, for a better self. I realize that I have always been a modern Cyborg, a cyborg who is such human.

6.2 Test facial expression responsive interactions With the open application "FaceOCS" and "Affectiva Emotion SDK" to track real-time facial data, I have tested many facial responsive interactions (Fig. 26-27). For example, degrees of smile control a circular shape; face movements change the melody of the music. I worked in multiple applications and software, including P5.js, Processing, Maxmsp, to understand their running environments, to find possibilities to construct them together within a video, as well as controlling interactions across different platforms. After many technical try-outs, in the end, I decided to use MaxMsp as the main application to display videos and communicate with Processing software.

Fig. 26 Facial application with Processing software, which prints the facial movements with lines.

Use facial movements to make a music

A Smile interaction

A facial interaction with movements of the head

Interaction test with cube

Motion fragments of face, Interaction test with Maxmsp

Interaction test with Maxmsp

Test with Processing and P5.js

Test with FaceOSC and Processing

Fig. 27 Different facial interaction tests with sofware Processing, Maxmsp, P5.js.

6.3 Moodboard and visual sketches In the first rounds, there are several keywords that I considered when making moodboard for the video, including face mask, space transition, reflection. I did some visual sketches as below. They are scenes separately about social communication (Fig. 28), facial responsive interactions (Fig. 29), and virtual identity with 3D modeling (Fig. 30).

Fig. 28 Visual experiments and installation test for scenes about social communication.

Fig. 29 Sketches for facial interaction design part.

1. Mask is a common visual metaphor about identity or hiding emotion. I used it to perform an unconfident, shy state; 2. Grid space forms a spatial shuttle feel and transition; 3. Mirroring effect creates a metaphor of reflection. To create these visual representations in the video, I related them to visual references like film <Tron> (1982), and <A space of Odyssey> (Fig. 31).

Fig.31 Images from <Tron> and <A space of Odyssey>

These 3d visual sketches (Fig. 30) are based on my script, where talks about my virtual identity, "The virtual

Fig. 30 Sketches for virtual identity with 3D modeling.

has transcended my perception of myself into fiction. In this heterotopia, I can create a second self or replicate hundreds of me ..."

6.4 Further developments In the process of previous testing and sketching, I realized that the final interactive video is a collage that includes different kinds of elements. It combines the work of scriptwriting, video shooting, video editing, audio editing, responsive interaction design, and visual modeling effect. I decided to make use of the screen in vertical to create a sense of mirror or portrait frame. Also, I found the visual sketches in the first-round trials were too illustrative and cliche â&#x20AC;&#x201D; for example, the shots of wearing a white mask, or the shots of typing on the phone. So, I considered using more abstract video shots to evoke the emotion in the video. For example, I captured flashes to interpret the scene about constant dialogues on the phone (Fig.31); I recorded the blurred screenshots (Fig.32) to reflect on my digital practice as a designer. However, all these visual frames with different styles still distract the narrative in some ways. I started to reconstruct the content again and looking for a visual language to make consistency through all footages.

Fig. 31 Storyboard and footage for Chapter 1 (mediated dialogue on the phone) Fig. 32 Storyboard and footage for script (cybersapce) â&#x20AC;&#x153;A blurred area, where me, technology and the rest of the world merge together."

6.5 Testing and Feedback After testing different versions of interactions (Fig.33) with my classmates, I finally decided to translate facial data to 3d objects instead of 2d ones, because it was more engaging and connected to the idea of "algorithm as beings". Many people found the real-time applications are playful. It helps them to understand the video content much more when playing with these face-trackings.

Fig.33 Further developments (from 2d to 3d objects) on the interactions in Processing.

6.6 Final design In the final stage of visual design, I strengthened the concept, "algorithm (digital space) as a kind of being", by creating a cell-like floating sphere model (Fig.34-35); I designed a Cyborg character (Fig.36) which tributes to <Metropolis>, an early masterpiece of a sci-fiction movie. The whole video essay (Fig.37) was available on Vimeo webpage https://vimeo.com/338519313.

Fig. 34-35 Visual design for narratives about virtual space. Fig. 36 Visual design for "Cyborg" character in the video.

Fig. 37 Final design: scenes of the whole video on two screen

Chapter 1

Chapter 2

Chapter 3

Chapter 4

Reflection and Conclusion I use my own story to express shared feelings (social anxieties and struggles) in more isolated relationships among people in this digital age. My goal was to increase awareness of the influence of intelligent technology on communication or emotion, as well as considering the missingness that counts the human relation, rather than to seek a real solution for communication problems. To achieve it, I choose to make an interactive video eaasy with a combination of responsive interaction design and video composing. Although it is a new attempt for me on narratives with visuals and interactions together in an installation, I find the possibilities to apply responsive interactions into my design in the future. I also become more interested in technology-based design topics and the speculative film essay. Overall, looking back at the graduation year, I went through many difficulties in the process, for example, positioning myself when discussing this complex bio-scientific & technical related issue. However, I learned a lot from researching in other disciplines (ethics, social-psychological fields) and some interactive techniques for design. Without this challenge, I may not have such a chance to get a deeper understanding of the “human” & “technology” relation. At the same time, as a designer, it is important to consider users’ emotional needs and offer a kind of empathy in the digital product to enhance human social connection. It is also necessary to reflect on the technology (software or technique tools) that we used in everyday digital practice.

Master Thesis Ningli Zhu <Face to Face> Non-linear narratives applying facial interactions Tutors: Joris Landman Annemarie Quispel Marnix de Nijs Arthur Roeloffzen Yin Aiwen Master Institute of Visual Cultures St.Joost School of Art & Design 's-Hertogenbosch, May, 2019