FACIAL EXPRESSION AS A TOOL FOR COMMUNICATION YICHAO ZHANG
Abstract Expression have socially functional, emotion comes as an integrated activity of brain at various levels, including cortex, subcutaneous marginal structure, thalamus system, endocrine system, autonomic nervous system and musculoskeletal system. Emotion can exist either temporarily to affect and disturb psychological activities at the time or permanently to have a lasting influence on individual psychologies. Emotion can appear as a form of mental state; also it can come as a lifetime companion to individual personalities. Emotion comprises three inseparable components, namely state, experience and expression. Facial expressions are the reflection of emotions. As a communication tool for human beings, facial expressions convey information like signals.
Contents Abstract Introduction Way in & question Affect & Expression & Emotion Design process Face Dance Reflection Conclusion Rrference
Introduction This essay will introduce my research on facial expression and experience about design process. The first part of the essay a documenting research. Facial expression is the interpretation of human emotion or internal state, like nonhuman’s animal signal. People’s expression is an important way of adjusting social communication. but whether the facial expression could reflect the human’s emotion or not ? This part is the focus of the discussion on this question. The second part is my designing process, I will discuss how I chose this topic and why. How I conduct the concrete procedures, as well as what challenges I encountered in the designing process. The structure of this part is more like a blog, you will see my designing process and my reflection. The third part is the final project. The aim of this part is to enable the readers to understand the project in advance through reading the text before looking my project formally, even produce some imagination. Hope in this essay you could find something you are interested in.
Way in & question Traditional facial expression studies began with René Descartes. who continued the Grecian view of expression. The passions are, in effect, two things: a “passion,” with regard to the subject to which it happens; and an “action,” with regard to that which makes it happen. Passion is the opposite side of rationality. Descartes analyzed the relationships between passion and body through the dualistic method analysis. He thought that people’s passion is connected with the body through the Pineal in the brain. Soul and perception are linked together, and the perception influences the body. According to Descartes’ theory, '''there aren’t many simple and basic passions ... you’ll easily see that there are only six: wonder, love, hatred, desire, joy, sadness.''(Hoffman, 2009)' According to Descartes emotion theory French painter Charles Le Brun reflected the facial expression in the painting through the anatomy method. Besides, Charles Darwin mentioned that all the human being, even animals could express emotion through the very similar behavior, in his book “The expression of the emotions in man and animals”. Darwin presented that the human being who crosses all the cultures has special and unique facial expression towards the certain emotion. In the same way, Descartes also presented similar animal spirit theory, passion is an entity, which could disturb the inner animal spirit. Basic emotions theory (BET) was established on the Descartes and Darwin theoretical basis, for example, “fear” is
represented as “scared face”, “rage” is showed as “angry face” and so on. This is called basic emotions theory (BET), Internal essence or emotion is externalized through our different facial expressions. This is also recognized in the field of cognitive science, and this classified research on expression is also applied in many fields, such as the artificial intelligence and algorithm of social contact robot. Although it seems fair and reasonable, BET theory has been existing for a long time. For centuries, we always think the facial expression reflecting our inner emotion. However, recent research found it may be far from the fact. One radical thesis pointed out, facial expression is not the emotional express, but the tools with social influence. The author Carlos Crivelli, he challenged the traditional research on emotion, he found that the research participants did not attribute the emotion to the face, like the westerners. He showed a picture to Trobriand islander, that is a standard picture expressing “fear”, and asked them to make sure what did they see. The islanders did not see a fear or horrible face, however, what they saw is the sign of threat and invading. '''our faces are ‘social tools’ that, like many animal displays, are used as lead signs of contingent action in social negotiation, and how they function depends upon the context of the current social interaction, the interactants, and their interaction histories.'' (Carlos, 2018,p2)' So that, If our expression could not actually reflect our feeling, then it would produce the tremendous consequences. One of them is the AI (Artificial intelligence) area, especially the robot technology. Many people are using these classical “poster” faces to train their artificial intelligence and social contact robot, with the rigid interpretation of robot’s signal.
If frowning does not mean rage, but the forced smile or embarrassing emotion. Then AI may wrongly respond them. Meanwhile, in the daily social contact process, people’s wrong interpretation of facial signal would cause the misunderstanding between the two sides. It turns out that we might communicate better if we saw faces not as mirroring hidden emotions, but rather as actively trying to speak to us. Therefore, my question is the facial expression could not reflect our
feeling, then what could it do?
Affect & Expression & Emotion I questioned the traditional research on expression. In order to better understand why the facial expression could not reflect peopel feeling. I researched affect, expression and emotion. Recent years have seen increasing research interest in affect from the social science circle. So, what is affect? The book Affect Theory Reader gives an explanation '''Bodies can catch feelings as easily as catch fire: affect leaps from one body to another, evoking tenderness, inciting shame, igniting rage, exciting fear-in short, communicable affect can inflame nerves and muscles in a conflagration of every conceivable kind of passion.''(Gregg,2010)' The scientific community generally studies emotional state or its interference with or physical activity. For example, blushes, sobs, snarls, guffaws and levels. So how do people perceive the emotional state of others ? '''The hilarity, the fear, the rage, the relief, the agony, the desperation, the supplication are what are most visible about those we look at. At a glance we see that the cop or the office manager is incensed, even though we find we cannot say later what colour his eyes are and had not noticed that she had dyed her hair. Indeed, the mirth and the despondency, the irritability and
the enthusiasm, the rapture and the rage are very visibility of a body. ''(lingis,2000, p16-17)' It is a kind of transmission or reflection of emotional state. By watching expressions of others, we can have a certain idea of their emotional state. Not only can there be emotion without expression. However, Expression comes as reflection of emotions. Even the same expression conveys varying forms of emotion. For example. '''It can act as either a reflection of really enjoyable experience or no more than a polite response on social occasions. Smiles remain the same, yet the facial expressions conveyed by them vary from each other. Facial expressions express emotional states, particularly feelings. Dif- ferent emotions or feelings correspond to different facial expressions, at some level of categorization of "different emotions.''(Ekman, and Friesen, 1975)' so What does facial expression express? Different levels of facial expression symbolize differentiated emotional states. This is particularly true for feelings. Emotions or feelings have their differentiated facial expressions to go with. Facial expression serves as a barometer to various emotions. When we’d like to know what someone is thinking about, how he is feeling or what he is doing, the first thing we typically do is to watch his face. There is good reason for doing this. Face is able to produce lots of expressions related to emotional state. in addition; most
individual muscle actions are common to multiple expressions, For example, our eyebrows will go upright when we are terrified. Similarly, our eyebrows will also go upright when we feel surprised. Although the same muscle movement can be caused by different expressions, such expressions typically result from a certain emotional state. '''There is no single muscles whose contraction unambiguously signals fear, nor an ‘anger muscle’ to signal anger. Instead it is the overall pattern of activity across the face that is distinct for each emotion.'' (Ekman,1982).' As a social tool, emotion works as effectively at communications as lots of other human communication systems like telephone, satellite, etc. Information is transmitted by expressions between the sender and receiver. Facial expression, whether acquired congenitally or through learning, unexceptionally constitutes a component of dynamic information transmission and decoding systems. Transmission: the sender encodes the messages (I’m happy) into a certain form of information (e.g. facial expressions) and sends them to the receiver through communication channels (e.g. visual or auditory systems). To decode the messages received, the receiver will resort to their previous knowledge for information extraction, perception and interpretation. In this way, messages from the sender are accurately translated and the receiver will know “he is happy”.
Human face has large quantities of striated muscles, each of which can be combined or activated at varying levels over time. Consequently face is able to produce considerable amounts of complex and dynamic facial expressions. Besides, facial form (shape and structure) is able to make changes in spatial distance, including facial width and height, protruding level of superciliary ridge and cheekbone, jaw width, distance between eyes, mouth width, lip thickness, forehead protruding level and so on. For instance, wide-open eyes and nose symbolize terrified facial expressions and wrinkled nose and narrowed eyes typically aversion. By encoding such complex information, facial expressions are made functional in social communications.
Animals and humans share similar communication signals. During communications, birds tend to accompany their sounds on facial or head movements. Among vertebrate animals, monkeys feature distinct facial movement patterns which play an important role in communications about threat, harassment, yielding and excitement. Facial expression presentation of monkeys can be independent of body movements. Scientists are able to identify their emotions of excitement, sorrow or satisfaction. A tool for communications, human facial expressions are able to convey more different and diversifying information, independent of body gestures and movements. Emotional signals have survived the evolution from primates to humans, the process of natural selection and group survival struggles, the genetic heredity and
variation. Routinization and congenital trait are typical of emotions. Based on the groundbreaking biological and evolutionarily original theories of Darwin on emotional facial expressions, Ekman made breakthroughs and proposed a set of specific facial expressions. Included in each of the facial expression is a facial action coding system (FACS) based on facial movements. A facial movement design associated with specific emotions, FACS divides the face into forehead area, eye area and nasolabial area. It is able to measure such emotions as happiness, surprise, aversion, anger, fear and sorrow. FACS is suitable for measuring majorities of fine and small facial movements which can be observed and identified reliably by the system. FACS was developed by Ekman who conducted measurement on his own face. He stimulated each piece of muscular tissue to let them release electric charge and the facial changes at the time would be recorded by a camera; then he matched the movements of each piece of muscle with its corresponding facial changes from the whole set of photos, in an effort to identify which facial change was caused by what muscular movements. For example, the tightly closed lips correspond to mouth
circular muscle; protrusive lips correspond to changes in contracting muscles in mouth corners. FACS is a very useful observation system and opens up medically referential ideas to our project. In my project, however, I used 3D software instead of electric and nerve stimulation.
Design process Face Dance To better understand expressions, I used 3d software to observe my own facial muscles. The software faceshift, concect with Xbox Kinect sensor, could realize real-time transmission and convert data into 3d models through algorithms. With this software, I could analyze the range of muscular movement. The facial expression is a form of body language that is able to complete elaborate information to communication. With 42 facial muscles, people could present a wide range of expressions, thus delivering different states of mind and feelings accurately. Upon contraction of the muscles of facial expression, the skin of the face tightens, changing its strength and appearance, generating all kinds of expressions. By observing the movements of facial muscles, I categorized these shapes and drew the range of movement and muscle strength of different expressions in the software. Such shapes of the strength reminded me of the intensity and rhythm in music. As a result, I laid my thoughts on the connection between music and emotions. '''Emotional response components that likely reflect emotionally motivated demands include patterns of autonomic arousal and somatic muscle tension, aspects of facial expression that contribute directly to instrumental action, such as widening the eyes with increased attentional activity, and certain parameters of vocal expression, such as pitch, that covary with arousal or tension level.'' ( Scherer, 1986, p243).'
Then, I thinking about the relationship between music and expression. Music is emotional contagion, one’s perception of an emotion could sometimes trigger the same emotion therein. For instance, when people were seeing facial expressions of fear and sadness, they would frown voluntarily. When listening to music, people would feel the emotions contained therein—sad Cello playing might lead the audience into a sad emotional state. We could be moved by music; in turn, could expressions create music? To figure out the answer, I looked through books on music and performance including Labanotation or Kinetography Laban, a system used to record and analyze human beings’ body movement. The Laban system was an “alphabetic” system, of which the symbols stood for the movement components that were able to “spell out” each mode. In standard labanotation, the three vertical scales represented the executor. Employees were divided by the centerline into the left and the right row that stood for the main part of the body. These works were interpreted bottom up from the perspective of the performers. Each symbol stood for a body part; in the meantime, the symbols were correlated to the alphabet A stood for the head, capital B epual the left shoulder, capital D epual the right elbow. In addition, Laban used plane graphics to show the direction and angle of movements. The shape of the graphic showed the direction of movement; its shadow indicated the level; its length stood for the duration of movement (the shorter and faster, the longer the time). Based on the theories of laban, I designed a set of facial expression-related graphic system. Through survey of muscle movements for different expressions, I distributed these movements onto two coordinate lines, namely shape and strength; then, I arranged these graphics according to strength (low-to-high). I used simple graphics such as the circular arc standing for the angle of muscle movement, triangle for the direction of muscle movement and the variation between dotted line and full line for the movement of mouth and eyes. Upon permutation and combination of these graphics, I noticed that the permutations were quite similar to the music score—like the experimental music scores of Cornelius Cardew. Graphic notations presented the music using visual symbols outside the range of traditional music notations. This system left the notations, abstract forms, numbers and shapes to the performers to interpret freely. The repeated graphic elements indicated a structure; yet the interpretation to each part stood unparalleled for the performer had to set his or her own rules in front of the notations. I thin that this form of creation has provided new possibilities for me.
To realize my conception, I studied programming software, by which I could convert data about facial muscle movements into midi signals and use them to operate a virtual piano. Changes in facial muscle movements could change the piano melody; for example, the data about eyes was matched to chord C on the virtual piano. Every time the user blinked, the virtual piano would play one note from chord C; different blinking speed would produce notes of different frequencies. During programming, I used the open source software OSC and the voice interaction software Max for interaction. Then, I analyzed the relationship between the performance and emotions. Because expressions reflected people’s emotions. I hope to express my ideas by performance. Because of when performing, the performer tended to highlight character’s personality with exaggeration. There are two acting styles that inspire me a lot. first, In Peking Opera, Performers express their characters through exaggerated
colors, for example, the shape, color and pattern of facial makeups could directly tell the audience the personality of each character. Also, different colors of the makeups stood for different characteristics, foe example, red for loyalty and integrity; black for bravery and straightforwardness; white for fraudulence, viciousness and treacherousness. Such representations shared certain similarities with the western psychology of color, in which colors were endowed with specific connotations and were oftentimes related to people’s emotions. Colors could influence people’s mental or physical conditions. The second example is Kathakali or Malayalam. The Kathakali is very similar to feature movies, which included such body language as highly coded gestures and facial expressions corresponding to nine basic emotions in the Indian dramatic theory; The nine emotions are Sringara (amour/love), Hasyam (comic), Karunam (pathetic), Raudram (furious/anger), Viram (heroic), Bhayanakam (terrible), Bibhatsam (odious), Adbhutham (wonder) and Santam (tranquility/peace). performers had complete control over their facial muscles and could thereby deliver their emotions. kathakali dancing. They are said to follow the rules of "rasabhinava," which means "communication of emotional states by expressions of the face" (Bonneau-Le Breton, 1994, p. 175)' Facial expressions were of great importance in the Kathakai dance and Peking Opera LianPu. Both dances have exaggerated forms of performance, Lianpu's facial expressions are generated by internal emotions and are
based on the characters' emotions. Indian dance is the active control of expression. So that's the difference between these two forms. I think both of these dances are good phenotypic forms. Through analysis, I came up with the inspiration for my project performance. Through the study of Lianpu and Kathakali, I started to think how to combine facial expression and sound performance. On the basis of my facial expression observation method, my programming and the voice interaction software, I invited an expert performer to compose a facial performance with the sound controlled by facial expressions. In this project, I invited a professional stage actor who has more than twenty years of acting experience. Besides, she has a background in sociology and she is interested in my project. Before the show I interviewed her and in the interview, she talked about her views on facial expressions, among which I was deeply impressed by the emotional transmission and cultural differences. She mentioned that facial expressions are congenital, that they are learned to act when babies are born, that they act as mirrors in childhood, that they reflect cognition, so emoticons are used as learning tools. Similarly, I think facial expressions like animal signals it is a tool for communication. Through discussion with her, we together developed a facial expression dance. This dance includes eye, mouth, nose, and muscle changes. I filmed the action, which will be shown in my video. And we tried a combination of hyperkinetic emotional changes, from anger to sadness, from sadness to surprise. During the design of the program, I also noticed that the changes in Midi signals could be reflected by the sine wave. The sinusoid or sine wave was a curve from the trigonometric functions in mathematics and a representative of the analog signals corresponding to the square wave standing for digital signals. Since each Midi signal could correspond to a number, different sound intensity would create various combinations of sine wave. Considering the relationship between data, light and sound, I built the connection between music and mathematical graphics. This part was an attempt to interactive software, with which I aimed to build a sound space for the audience with graphics and sound. People used facial expressions to interact with the software and generate sound. In this way. the audience, became joint creators as a part of the project.
Reflection The project involved data, I considered ethical issuesIn, the past few years, with the development of the facial recognition technology, simple algorithms could recognize human beings facial information for computer analysis. Such simple algorithms, in an open source manner, could be easily downloaded by every one through Internet and then apply to website information collection, animation and special effects, exchange design and other fields. However, behind such convenience brought by the technology, there also lay negative effects. Along with the application of the technology, topics on data and privacy protection were widely discussed. Therefore, in the project design, data was for real-time interaction only, won't be stored or collected. In today’s society, people have become more and more concerned about the ethical issues produced by technology. In the prediction of the future world, would these questions grow in complexity? At the initial stage of the research, I was greatly inspired by a critical design project ''Bernd hopfengaertner belidf systems''. So at the end, I went back to the original point of view, which ran through the whole project. In the project author designed six scenarios, in one of which, a person tried to recognize the muscle groups on his or her face so as to learn to control them, thus hiding his or her true feelings; he tried to learn such an impersonal skill to protect himself or herself. In my opinion for some people, it was the ultimate dream of the user-centered mode. It was a preset of the future, which magnified the negative effects of today’s technology and introduced them into the future life. In the future world, the combined algorithms and camera systems would read people’s emotions out of their faces and behavior; advanced neurotechnology, though couldn’t interpret people’s thoughts accurately, could provide reasonable guesses on people’s mind… Speculative designs were oftentimes of profound or heavy topics, but not blindly pursuing the darkness therein. Darkness could be the technology against naivety, the detoxicant to utopianism; it drove actions and gave birth to the critical thinking of the people. Speculative designs were like the mirror, reflecting problems in today’s society for discussion in the parallel universe of the future. In this way, it exhibited multiple possibilities, helped people to think and facilitated their actions.
Conclusion During this project, I started to doubt the traditional study on expression and began my research on the basis of the question of whether facial expressions were able to reflect the truth of the heart. According to the laban movement recording theory, I designed a set of methods recording facial expressions, considered the connection between sound and programed. The sound changes according to the movement of facial muscles. The most difficult part in this project was the program software learning process for programming required movement programming language and the programming logic. However, after understanding the basic knowledge, it became very interesting. The most memorable experience is to work with performers. From the perspective of the project, the challenge against a problem and the survey around the raised question gave birth to my design project. For me, design was not to solve the question, but to raise the question and to explore possibilities
with design as a tool. In the end, the control of sound via facial expression was never an answer-there was not the mere solution or a better way as I believe, but only another way. I hope that this project could be an open ending, from which I would find other, more possibilities in future studies.
Rrference Book 1. Hoffman, P. (2009). Essays on Descartes. New York, NY: Oxford University Press 2. Gregg, M. and Seigworth, G. (2010). The affect theory reader. Durham, N.C.: Duke University Press. 3. Scherer, K. and Ekman, P. (1982). Handbook of methods in nonverbal behavior research. Cambridge.: Cambridge University Press. 4. Russell, J., Fernandez,D., Josa,M. and Mandler, G. (1998). The psychology of facial expression. Cambridge.: Cambridge University Press. 5. Newlove, J. and Dalby, J. (2003). Laban for all. London.: Nick Hern 6. Schafer, M. (2011). The soundscape : our sonic environment and the tuning of the world. Rochester, Vt..: Destiny Books. 7. Wetherell, M. (1954). Affect and emotion : a new social science understanding. London.: Sage. 8. LaBelle,B.(2018). Sonic Agency: Sound and Emergent Forms of Resistance. London.: Goldsmiths Press.
Article 1. Carlos, C. (2018). Facial Displays Are Tools for Social Influence. Trends in Cognitive Sciences, Volume 22 (5), pp. 388-399. Available at: www.cell.com/trends/cognitive-sciences/fulltext/S13646613(18)30029-9 [Accessed: 5 October 2018]. 2. Johnson, H. G., Ekman, P., & Friesen, W. V. (1975). Communicative Body Movements: American Emblems. Semiotica, 15(4), pp.335-353.
Images/visual mediums 1.Paul Ekman Group. (2013). The Development of the Facial Action Coding System Part 1. [YouTube video]Available at: www.youtube. com/watch?v=QaTg7G7_xs8 [Accessed 25 October 2018]. 2. Audiopedia. (2015). Facial Action Coding System. [YouTube video] Available at: www.youtube.com/watch?v=Y6mHaDnCj8c [Accessed 26 October 2018]. 3. Dancenotationbureau. (2013). Intensive Course in Elementary Labanotation. [YouTube video]Available at: www.youtube.com/ watch?v=bIhCI7WIBJg [Accessed 16 October 2018]. 4. Gbtimes.(2015). Peking opera An introduction. [YouTube video] Available at: https://www.youtube.com/watch?v=PnMRIzpO4nU [Accessed 6 October 2018]. 5. Indiavideodotorg.(2009). Kathakali Performance - Onam Video Greetings [YouTube video]Available at: www.youtube.com/ watch?v=Tl3UKV1z9lM [Accessed 7 October 2018].
FACIAL EXPRESSION AS A TOOL FOR COMMUNICATION YICHAO ZHANG