BCI and emotional states

BRAIN COMPUTER INTERFACE BASED ON NEURAL TRAINING AND EMOTIONAL STATES N. Bohora, G. Burdo, B. Niclassen, R. Sonne

Keyword list: Brain Computer Interface, Electroencephalography, Neural training, Emotion recognition.

Abstract This paper describes a foray into the field of Brain Computer Interfaces (BCI) and Electro Encephalography (EEG) signal recognition and comparison. The primary focus is on neural training when trying to help subjects stabilize their relative theta and alpha waves amplitudes. The secondary focus is to use music induced emotion and episodic memory as a means of control in a BCI. The test results indicated that this kind of neural training while it worked did not inspire much feeling of being in control in the test subjects. A simple analysis of the results from the episodic memory emotion test shows a promising difference in the signals created by the segment subjected to happy versus sad music. Also the episodic memory seemed to generate a strong enough signal for a recognition algorithm to use it for controlling a BCI.

Introduction Research in Brain Computer Interfaces has reached a stage where it can overcome otherwise impossible hurdles. One such is paralysis which can hinder people from expresing their emotions. How to train a person in the use of BCI and how BCI can be used to express sad and happy emotions(Ekman, et al., 1987).

Previous studies show how it is possible to recognize feelings with electroencephalography measured by electrodes on the scalp (Ko, Yang, & Sim, 2009).

The initial step was to explore the basic step in BCI, where neural training in which the test subject tries to achieve a relaxed or meditative state is considered one of the most basic experiments. Another subject is whether emotionally laden music can inspire certain emotions in people and whether they are able to sustain this emotion through different training exercises.

This paper proposes an emotional recognition method using Electroencephalography (EEG) signals. Each EEG signal was decomposed into five sub-bands: delta (0÷4Hz), theta (4÷8Hz), alpha (8÷13Hz), beta (13÷30Hz) and gamma (30÷50Hz). The delta band was removed to clear noise such as electrical currents created by neck and oculomotoric movement and ECG signals.

Initially the simplest form of evoking signals was tested; to see

whether relaxation can create stable alpha and theta waves to be used as a control output. When that succeeded the next hypothesis was that a similar setup could be done with the goal of sustaining emotional states. If true this concept could be used to control external devices. Dealing with the different shades of emotions is complex and required quite a bit of research to test. Apart from recognizing the test subject’s emotional state with eeg signals it can be assessed using self reporting methods (Isomursu, Tähti, Väinämö, & Kuutti, 2007) or from inferring emotional states from other physiological signals (Mandryk, Inkpen, & Calvert, 2005) (Chanel, Kierkels, Soleymani, & Pun, 2009).

Neurofeedback Neurofeedback is an ongoing operant procedure, where the subject learns some degree of control of his/her EEG activity.

Research in cognitive performance indicates that neurofeedback improves the cognitive performance in human subjects (Hanslmayr, Sauseng, Doppelmayr, Schabus, & Klimesch, 2005). Studies applying EEG measurements indicate that there is a correlation between the EEG alpha value and intelligence (Doppelmayr, Klimesch, Stadler, & Heine, 2002). The chosen approach to neurofeedback in this paper, takes a different view on EEG Alpha values. Stress reducing states can

1

be directly connected to the level of alpha value(Hammond, 2006). Hence the preliminary study will concern whether the chosen EEG equipment can be used in order to record a chosen frequency range, analyze the signal and give a feedback to the subject. Another goal is to examine whether it is possible to train the subject to sustain a calm (meditative) state over longer periods of time. If this proved possible, the next step was to examine if it is possible for subjects to sustain a chosen emotional state, induced by music.

Neural Training Test

Experimental Measurement Method The market for eeg apparatus was researched to identify a likely candidate for investment by Aalborg University (Sonne, 2009). The decision was taken to invest in two PET 2.0 devices from Brainclinics (Brainclinics, 2009).

Their five active electrodes enable them to measure 2 bipolar channels each and send the recorded signal wirelessly to a computer via bluetooth. BioExplorer receives the signal and passes it through flash and arduino to control the balloon pump.

The purpose of this test is to determine whether focusing on a simple physical setup can help test subjects stabilize their alpha and theta signals. The state of mind of the test subjects was assessed before and after the test using a Self Assessment Manikin as described by (Bradley & Lang, 1994). After the test the subjects were interviewed to investigate the perceived functionality of the installation feedback during the test.

The installation consists of a mechanically inflatable balloon which initially is deflated. The hypothesis is that focusing on inflating the balloon with the mind can help the test subjects achieve a stable state of alpha and theta waves.

Positive Negative Ground

Figure no 2. shows the electrode position according to the 10/20 international placement system. > 5000 msec

Bioexplorer

+ 0.1

- 0.1

The electrode placement was done according to the 10/20 international placement system. The best suited electrode positions for measuring alpha values is considered to be Pz (Hanslmayr, Sauseng, Doppelmayr, Schabus, & Klimesch, 2005). Three electrodes were placed in: one positive (Pz), one negative (A1) and ground (A2).

Figure no. 1. Illustrates the neurofeedback loop, starting with the BioExplorer reading EEG signal values from subject, into Adobe Flash for processing, if condition is met Adobe Flash will send a signal to Arduino that inflates the balloon and subject perceives feedback.

Results Ten people of age 20รท25 were tested of which 2 were female.

2

Emotion Recognition Test

10 x

0

30

60

90

From the end of nineteenth century studies concerning music and emotions have been conducted (Gabrielsson & Juslin, 2003). The majority of studies focused on how the listener perceives the emotions expressed in the music. Consequently theories about music and emotions were formulated, that focused on the representational features of music that enables the listener to perceive music (Juslin & Västfjäll, 2008). However the process of perception of emotions does not necessarily say anything about how the listener is feeling himself. Since perception of emotions may well occur without any emotional involvement (Juslin & Västfjäll, 2008).

120 sec

Emotions and their correlation to music, music inducing emotions

CONTROL

ACTIVATION VALENCE

Figure 3. shows the amount of balloon inflation, illustrated by a line, and timer duration, illustrated by the thickness of each line in the figure. Valence, activation and control are rated by circles, empty indicate before test and full circles indicate after test.

Discussion The results from the test indicate that the setup of the neural training had a positive influence on the valence of the test subjects. But did not have a calming effect. The results indicate present neural training setup does not increase the users feeling of control of the situation.

There can be three main reasons for this result. Firstly, the user did not understand the feedback loop. Secondly, the user did not understand SAM why the results do not correctly describe his/her emotional state. If the two first cases are proven incorrect, the neural training procedure has been proven to be insufficient to fulfill its purpose. The duration of inflation of the balloon, does not indicate a linear learning curve in controlling the thoughts, which can explain the users feeling lack of control in the neural training session.

In spite of it not being clear for the subjects of the first experiment that they were in control of the installation, it was in fact the case. Control is important if the technology is to be used as a substitute for otherwise lost communication skills by a paralyzed person. This instigated an interest in testing a different branch of BCI that might give the subjects a stronger sense of being in control. Research in emotion recognition gave birth to the idea that paralyzed persons could express their emotional state using BCI.

A definition of emotions can help understand how music can induce emotion in people. Although researches may not agree on a precise definition of emotion, they largely agree on characteristics and components of an emotional response: ”Emotions are typically described as relatively brief, though intense, affective reactions to potentially important events or changes in the external or internal environment that involve several subcomponents…” (Juslin & Västfjäll, 2008).

In addition to the definition of emotional response, several theories of emotions have been proposed. The most well known models are the basic emotions(Ekman, et al., 1987) and valence-arousal space (Russell, 1980). Basic emotions are defined as cross-cultural common emotions, considered biological universal to all humans. The valence-arousal space allows for an incessant representation of emotions in two axes; valence ranging from pleasure to displeasure, and degree of arousal ranging from calm to exited.

Juslin and Västfjäll have identified six different mechanisms, combined with cognitive appraisal (Folkman, S. Lazarus, Dunkel-Schetter, DeLongis, & Gruen, 1986), which can explain the situation when music induces an emotion in a person.

One of these six mechanisms is the episodic memory, and the hypothesis is that this could be used as a means of control in BCI. To identify the eeg signal created by the episodic memory a signal recognition algorithm is necessary.

Pattern Recognition In order to be able to use an emotion as a means of control in BCI a pattern recognition algorithm is necessary to identify the signal created by the emotion.

One approach would be to utilize a Hidden Markov Model (HMM), known from areas of temporal pattern recognition such as speech recognition. It is a statistical model in which the system being modeled is assumed to be a Markov process with unobserved state (D.Novak, 2007). An HMM can be considered as the simplest dynamic Bayesian network.

In regular Markov models the state is visible to the observer,

3

thus the state transition probabilities are the only parameters.

In a hidden Markov model the state is not directly visible, only the resulting output. Each state has a probability distribution over the possible output tokens. Therefore the sequence of tokens generated by a HMM gives some information about the sequence of states. Note that the adjective ‘hidden’ refers to the state sequence through which the model passes, not to the parameters of the model; even if the model parameters are known exactly, the model is still ‘hidden’.

Another way of recognizing the emotions from the recorded signals is to use a Radial Basis Function (RBF) network (Dezhong, 2002). As the name indicates it is an artificial neural network consisting of a linear combination of radial basis functions. It requires system learning to make effective, which means a lot of user tests to form the learning data.

Neither method was implemented for the comparison of the signals but it would be the next step in creating a useful and more adaptive BCI system.

TEST I

II

III

GROUND + MUSIC

The electrode placement was done according to the 10/20 international placement system. Five electrodes: two positive (C3, C4), two negative (Fz, Pz) and the other ground (middle of the forehead) were used. These placements were chosen based on the study of (Pfurtscheller & Neuper, 2001).

Positive Negative Ground

PROCESSING IV

results

GROUND

Electrode placement

figure no 5. Shows the electrode position according to the 10/20 international placement system. The electrodes are placed in Pz,C3,C4,Fz,Fpz.

EXPERIENCE EXPERIENCE EXPERIENCE + + GROUND GROUND GROUND + MUSIC MUSIC

Figure no. 4. Illustrates the outcome of the emotion test.

From the four different steps the final step is to calculate the experience emotion subtracting the ground value.

Selection of happy and sad music as Stimuli This test includes the recording of signals of the subject’s brain while hearing the sad or happy music and recalling the sad or happy memory without the music feedback. The purpose of this test is to compare the EEG signal of test persons in four states: 1) while relaxed and thinking of nothing, 2) listening to sad/happy music, 3) while listening to the music and re-living a memory that fits the music and 4) re-living the same memory without the music.

Two classical pieces of music were selected that qualified as happy and sad, namely The Little White Donkey

by James Galway and Adagio for Strings by Samuel Barber.

They were meant to induce a certain emotion in the subjects.

After being exposed to the music for a minute they were asked to think of a previous memory befitting the music. Lastly the music was turned off and the subjects asked to relive the memory from before as vividly as possible. The hypothesis is that the signal generated when the persons relive their memory without music should be recognizable by pattern recognition. Initially the signals from all phases were recorded for later comparison and analysis.

4

Results In the attachment it is possible to find all the samples for the recording sessions in all the steps: ground, music, music and memory, and just memory. For each step there is a collection of theta, alpha and beta frequencies.

Apart from all the similarities, it is clear that there is a difference due to the kind of music subjects were exposed to. The graph shows an average based on the ten users. The results show more activity in theta for the happy song and less in alpha and beta.

Noise artifacts is a factor to be taken into consideration when analyzing the EEG signals. Experience has shown EEG equipment is sensitive in recording any noise created by the surrounding environmental. Skin surfaces had to carefully cleaned and conductive gel applied in order to ensure a clear EEG signal possible.

Put aside the possible noise artifacts in EEG signal, subjective assessed results from the neural feedback indicate the subjects are controlling their alpha and theta values. The Self Assessment Manikin results indicate test subjects did not perceive a control of neural feedback.

Although a pattern recognition algorithm was not developed, a subjective comparison of the signals recorded from subjects, was conducted. Results indicate subjects listening to sad music had lower theta channel average frequency than the signals from the subjects listening to happy music. This result can be an indication of how sad and happy emotion signals can be distinguish. The authors acknowledge the need for a more comprehensive test study, if the results should be considered valid. Also implemented pattern recognition would give a more accurate picture of acquired experimental data. Even though the equipment had a limited number of electrodes, it performed reasonably well and served as a good starting point for research into BCI.

SAD

Θ

HAPPY

5,5582

4,9257

α

9,6507

9,8191

β

19,4688

20,3182

Figure no. 6. Illustrates the difference in theta, alpha and beta EEG signal values. The Table shows the difference between happy and sad emotion, in frequency bands theta, alpha and beta.

Acknowledgments This research was funded by Aalborg University Copenhagen.

We are grateful to Associate Professor Luis Emilio Bruni and Assistant Professor Sofia Dahl for their guidance throughout the process of this research. We would also like to express our gratitude to PhD student Smilen Dimitrov for helping us in the area of EEG equipment and Assistant Professor Daniel Grest for giving advice on signal recognition.

References Discussion In present study brain correlation to emotional states has been divided into two instances. The first part of the paper examined alpha/theta neural feedback, secondly music induced emotional state happy and sad was analyzed. With a small amount of electrode placed on the scalp, used in current study, it can be discussed if best possible electrode placement has been employed. Through extensive research it has become clear that the most optimal solution was chosen for neural feedback. Placement of electrodes will be changed for the next iteration of memory induced emotion detection testing to reflect the research of (Flores-Gutiérrez, et al., 2007).

Bradley, M. M., & Lang, P. J. (1994). Measuring Emotion: The Self Assessment Manikin and the Semantic Differential. Journal of Behav. Ther. & Exp. Psychiat. Vol 25 , 49-59.

Brainclinics. (2009, 12 15). Neurofeedback equipment - Wireless Brainquiry PET EEG and ActivEEG. Retrieved 15/12/ 2009, from Brainclinics: http://www.brainclinics-products.com/ Chanel, G., Kierkels, J. J., Soleymani, M., & Pun, T. (2009).

Short-term emotion assessment in a recall paradigm. International Journal of Human-Computer Studies (67), 607-627.

D.Novak, T.a. A. (2007). Electroencephalogram processing using Hidden Markov Models. Conference on Control Applications.

Proceedings of the IEEE, volume 89.

5

Dezhong, Y. (2002). High-resolution EEG mapping: a radial-basis function based approach to the scalp Laplacian estimate. Clinical Neurophysiology , 956-967.

Pfurtscheller, G., & Neuper, C. (2001). Motor Imagery and Direct Brain-Computer Communication. Proceedings of the IEEE, volume 89.

Doppelmayr, M., Klimesch, W., Stadler, W. ,., & Heine, C. (2002). EEG alpha power and intelligence. Intelligence, 30, 289–302.

Russell, J. A. (1980). A Circumplex Model of Affect. Journal of Personality and Social Psychology , 39, 1161-1178.

Ekman, P., Friesen, W. V., O’Sullivan, M., Diacoyanni‐Tarlatzis, I., Krause, R., Pitcairn, T., et al. (1987). Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion.

Journal of Personality and Social Psychology (53), 712-717.

Flores-Gutiérrez, E. O., Díaz, J.L., Barrios, F. A., Favila‐Humara, R., Guevara, M. Á., Río Portilla, Y. d., et al.

(2007). Metabolic and electric brain patterns during pleasant and unpleasant emotions induced by music masterpieces. International Journal of Psychophysiology , 65, 69-84.

Schoenemann, P. T. (1999). Syntax as an Emergent Characteristic of the Evolution of Semantic Complexity. Minds and Machines , 309–346.

Sonne, R. (2009, 12 16). EEG equipment. Retrieved 17 12, 2009, from Bsonne.dk: http://www.bsonne.dk/eeg/equipment/

Folkman, S., S. Lazarus, R., Dunkel-Schetter, C., DeLongis, A., & Gruen, R. J. (1986). Dynamics of a Stressful Encounter: Cognitive Appraisal, Coping, and Encounter Outcomes. Journal of Personality and Social Psychology, 50 (5), 992-100.

Gabrielsson, A., & Juslin, P. N. (2003). Emotional expression in music. In R. J. Davidson, K. R. Scherer, & H. H.Goldsmith, Handbook of affective sciences (p. 1161). New York: Oxford University press inc.

Hammond, D. C. (2006). What Is Neurofeedback? Journal of Neurotherapy , 10.

Hanslmayr, S., Sauseng, P., Doppelmayr, M., Schabus, M., & Klimesch, W. (2005). Increasing Individual Upper Alpha Power by Neurofeedback Improves Cognitive Performance in Human Subjects. Applied Psychophysiology and Biofeedback, 30 (1).

Isomursu, M., Tähti, M., Väinämö, S., & Kuutti, K. (2007).

Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. International Journal of HumanComputer Studies (65), 404-418.

Juslin, P. N., & Västfjäll, D. (2008). Emotional responses to music: The need to consider underlying mechanisms. BEHAVIORAL AND BRAIN SCIENCES (31), 559-621.

Ko, K.E., Yang, H.C., & Sim, K.B. (2009). Emotion Recognition using EEG Signals with Relative Power Values and Bayesian Network.

International Journal of Control, Automation, and Systems , 865-870.

Mandryk, R. L., Inkpen, K. M., & Calvert, T. W. (2005). Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology (in press).

6