11_ by ides editor

Short Paper Proc. of Int. Conf. on Advances in Computer Engineering 2012

Dynamical Switching between EEG and ECG for Emotion Recognition in Living Space Kanlaya Rattanyu1, and Makoto Mizukawa2 1

Shibaura Institute of Technology/Graduate School of Functional Control System Engineering, Tokyo, Japan Email: m709502@shibaura-it.ac.jp 2 Shibaura Institute of Technology/Department of Electrical Engineering, Tokyo, Japan Email: mizukawa@sic.shibaura-it.ac.jp

Abstract—This paper presents our approach for emotion recognition based on wireless and wearable multichannel of Electroencephalogram (EEG) and Electrocardiogram (ECG) sensors for mobility and convenience of users’ daily life. We took advantages of the combining that EEG gave more precise recognition rate and ECG was more stable with less noise. In the ECG module, we propose to use the ECG’s inter-beat features together with within-beat features. In order to reduce the feature space, post hoc tests in the Analysis of Variance (ANOVA) were employed to select the set of eleven most significant features. Our designed system applied EEG’s power density spectral and fractal dimension (FD) features in normal situation and using ECG features when EEG signal degrades. We conducted experiments on 18 subjects according to Mirror Neural System (MNS) theory to elicit emotion. For simultaneous classification of six emotional sates: anger, fear, disgust, sadness, neutral, and joy, the Correct Classification Ratio (CCR) was 74.1% in EEG module and 60.6% in ECG module.

increase of heart rate associated with fear (e.g., [8]) and anger (e.g., [7]), and increase of heart rate variability associated with stress (e.g., [5]). However some results were controversial: sadness has been found to sometimes lead to an increase of heart rate (e.g., [11]) and sometimes to a decrease (e.g., [8]). II. EQUIPMENTS There are many biological signals related with emotion. The sensors were selected based on three important criteria. The first criterion was that it’s signals had to be strongly related with the human emotion. The second criterion was that the sensor had to adhere to human skin without discomfort. The last criterion was that the sensor had to be wearable and convenient for use in notmal daily life. A. Wireless ECG Sensor (RF-ECG) RF-ECG was used to measure an elctrocardiogram (ECG) signal generated by electrical activity of the heart muscle. The sensor is a low weight (12 g) and small-sized sensor (40 mm × 35 mm × 7.2 mm). This sensor can record and wirelessly transmit ECG signals to the server with 204 Hz. The wireless RF transmitter had an open area range of up to 15 m.

Index Terms—emotion recognition, EEG, ECG, ANOVA

I. INTRODUCTION Although EEG and ECG are commonly used for emotion recognition, there are some remaining issues. For EEG, the quality of data depends on user activities and sensor set up. For example, artifacts will stem from muscle activities when user moves. Signal from sensor may be weak over some short periods when user changes their head position which make EEG sensor loosens. To overcome the above issue, this paper proposes dynamic switching between sensor/data sources when EEG signal degrades. The ECG signal has an advantage over the EEG signal in that the ECG’s amplitude is quite large compared with the EEG signal. The amplitude of ECG signals is measured in mV. For a typical adult human, the EEG signal is about 10-100 µV in amplitude when measured from the scalp. However the main limitation of emotion recognition by using only ECG signal was the number of emotional categories. Facial feature expression can categorize emotions into many categories, while most successful studies by using ECG signals classify only few categories such as positive/ negative feeling[1-3], feeling of being stressed/relaxed [4, 5], or fear/neutrality [6]. Some studies (e.g., [7–10]) overcome this limitation by combining ECG with other physiological signals that are related to organs that are affected by the Autonomic Nervous System(ANS). Among these studies, some correlations between emotion and ECG can be identified: © 2012 ACEEE DOI: 02.ACE.2012.03. 11

B. EEG Emotiv EPOC Headset The Emotiv EPOC headset was selected to measure elctroencephalogram (EEG) signals that present information regarding brain activity and global information about mental activites and emotional states. The neuroheadset consists of 14 electrodes following the American Electroencephalographic Society Standard. It also integrates two internal gyroscopes to provide user head position information. III. METHODOLOGY To achieve EEG signals without artifacts: (1) the user must avoid moving in the EEG signal acquisition; and (2) filters or some signal processing algorithms can be accomplished to remove artifacts from the EEG signals acquired. Although we have filters to remove the artifacts, removing artifacts entirely is impossible in the existing data acquisition processes. It is better to avoid them. Most successful researches [11-16], the participants were asked to keep less movement as possible while measuring EEG. As this reason our system was designed for dynamical switching between sensors when EEG signal degrade. We employed emotion recognition using ECG 85

Short Paper Proc. of Int. Conf. on Advances in Computer Engineering 2012 instead of EEG when bad contact quality of EEG signal or the user’s moving occurs as described in figure 1.

Figure 3. Detecting user’s movement

C. EEG Processing The Emotiv headset acquired EEG signal with 14 sensors placed on the user scalp.The signals were recorded at a sampling rate of 2048 Hz through a C-R high-pass hardware filter at 0.16Hz cutoff, pre amplified and low-pass filtered at 83 Hz cutoff and preprocessed using two notch filters at 50 Hz and 60 Hz. The signal is down-sampled further to 128 Hz.  Filters: A high-pass filter with cutoff frequency at 1 Hz was first applied in order to remove linear trends in the EEG raw signals and a low-pass filter with cutoff frequency at 41 Hz was employed for high-frequency noise removal.  Signal Enchantment: There was no timestamp data available by the Emotiv headset. The amount of data loss was calculated from time period and frequency. The signals were simple compensated by using scale interpolation.  Feature Extraction: EEG signals are complex signals seem like noise, so gathering information from the time domain was hard. In the time domain data seem irrelevant. These compensated signals were converted from time domain to the frequency domain. The signals were split by frequency ranges. In this work, we applied a combination of power spectral density (PSD) of different EEG frequency band and the fractal dimension (FD) of each electrode location as our set of EEG features. A fast Fourier transform was used to estimate PSD for each electrode in the δ (1-4 Hz), θ (4-8 Hz), α (814 Hz), β (14-26 Hz), and γ (26-41 Hz) frequency band. For each data point normalized PSD of sub band s at electrode location i was calculated as equation (1).

Figure 1. Dynamical switching between EEG and ECG for emotion recognition

A. Checking contact quality of EEG The headset provided contact quality flag variables associated with each EEG channels. The color of the sensor circle is a representation of the contact quality as shown in figure 2. In EEG emotion recogntion module, we accepted only excellent(green color) signal. To achieve the best possible contact quality, all of the sensors should show as green. Other sensor colors indicate: yellow fair signal, orange poor signal, red very poor signal and black no signal.

PSD si 



5 s 1

PSDsi

(1)

The fractal dimension (FD) is a quantity that conveys information about the space filling and self-similarity of an object. For calculating the FD we used a method from a recent study [20] reporting better performance that Higuchi algorithm (traditional method); it computed directly FD from the waveform. At the first step, the time series t with length N is normalized with respect to time and amplitude:

Figure 2. Checking contact quality of EEG

B. Checking contact quality of EEG We detected the user’s head movment via two internal gyroscopes in the headset device. To avoid to use noisy data, the system rejected to process the EEG signal when the user did not keep less movement.

PSDsi

nnew 

n N

(2)

Short Paper Proc. of Int. Conf. on Advances in Computer Engineering 2012

tnew 

t  tmin tmax  tmin

used to explore all possible pair-wise comparisons of means comprising an emotion factor  using the equivalent of multiple t-tests. We used the post hoc test to identify the significance of each feature. As described in the previous section, there are 42 features. We selected only 11 features (HR, SDNN, RMSSD, QT, SDQT, PR, SDPR, QRS, SDQRS, ST, and SDST) from 42 features that had a level of confidence more than 85%. We did not apply 25 non-significant features in the classification process.

(3)

Where tmin and tmax are the minimum and maximum of the signal amplitude, respectively. Now the FD of waveform can be approximated as follows: FD  1 

ln( L) ln(2( N  1))

(4)

C. Normalization

Where L is the length of the normalized curve,

Each parameter was normalized by subtracting each parameter from its mean in the neutral emotion.

L   (tnew (n)  tnew (n  1)) 2  (nnew  (n  1)new )2 (5) n2

D. Classification Linear Discriminant Analysis (LDA) was used to classify emotion into six categories (anger, fear, disgust, sadness, neutral, and joy). We ran a cross-validation 10 times using the 20 percentage holdout method to have a better estimation of the classifier performance.

This produced 5+1 features at each electrode location which made a total number of 84 features for each emotional data. D. ECG Processing The ECG signal was sampled with a sampling frequency of 204 Hz. The digital signal was then transmitted wirelessly to the server.  Annotation of the ECG: The Continuous Wavelet Transform (CWT) and Fast Wavelet Transform (FWT) were used for automatic annotation of the ECG cardio cycle [8]. The annotation method consisted of two phases: QRS detection followed by P, T wave location.  QRS detection: To amplify the QRS complex and separate low frequency (P and T waves) and high frequency (noise), the CWT transform was applied at 12 Hz with an inverse wavelet. The CWT spectrum obtained was further filtered with FWT using an interpolation filter to remove frequency content below 30 Hz and the rest of the spectrum was denoted with a hard threshold using a MINIMAX estimate. The reconstructed ECG signal after denoising contained only spikes with non-zero values at the location of the QRS complexes.  P,T waves detection: After QRS complexes were detected, the intervals between them were processed for detection of P and T waves  Features Extraction: After determining the location of each wave on the ECGs, several parameters indicating each part of heart’s activity were calculated. In this process, we calculated not only the inter-beat information of the ECG (RR-interval or HR), but also the within-beat information of ECG (PR, QRS, ST, QT intervals, PR, and ST segments).  Statistical Data: We calculated six types of statistical data (maximum, minimum, medium, mean, SD, and RMSSD) of each parameter during the 6 second period. In total, we have seven parameters(RR, PR, QRS, ST, QT intervals, PR, and ST segments) and six types of statistical data, so the corresponding Cartesian product has 42 (7×6) elements. This means that the number of possible features were 42 features. Post Hoc Tests in Analysis of Variance (ANOVA): Least Significant Difference (LSD) test was © 2012 ACEEE DOI: 02.ACE.2012.03. 11

IV. EXPERIMENT According to the Mirror Neuron System (MNS) theory[17], the subjects’ biological signal is supposed to reflect the same activity as when they are really overcome by the same emotion. Thus, the exploitation of MNS’s functionality during the emotion elicitation process would help gathering more representative biological signals. In order to elicit the basic emotions as defined previously Pictures of Facial Affect (POFA) [19], showing people expressing the six aforementioned emotions were subsequently demonstrated, separated by black and counting down (5 4 ... 1) frames before the projection of the new picture. A five-second period was demonstrated to accomplish a relaxation phase and emotion-reset. More specifically, black screen was projected for five seconds follow by five seconds counting down period then one second cross to focus user and finally a randomly picture is projected for six seconds. This seventeen seconds procedure is repeated for every one of the sixty pictures. Each trial began with a preparation step to familiarize the subjects with and understand the experiment. We started to measure the biological signals at the same time the first picture was presented, so we were able to separate the biological signal into 6 seconds of 60 emotions per subject. VI. RESULT AND DISCCUSION The experiment was conducted with 18 subjects (mean±SD age = 27.5± 5.1 years). Each subject was recorded both ECG and EEG with 60 emotional data. In EEG module, some samples that the contact quality of signal was very poor were rejected. The total of selected-sample was 912 samples. The previous studies extracted only inter-beat information of ECG such as RR-interval or Heart Rate (HR) time series. Some studies use statistical data to record this information 87

Short Paper Proc. of Int. Conf. on Advances in Computer Engineering 2012 (min, max, average, Standard Deviation of Normal-to-Normal of R-R intervals (SDNN), and Root Mean Square of Successive Different of RR intervals (RMSSD). In order to maximize efficiency of ECG. we propose to use ECG’s interbeat features together with within-beat features in our recognition system. Table 1 showed the final result of emotion recognition accuracy based on Mirror Neural System. The experimental result showed that our 11 ECG features approach performed better than convensional 3 ECG features approach 21.4% and EEG module gave more precise recognition rate than ECG module 13.5%, when the subjects were asked to keep less movement as possible while measuring the signals.

[8]

I. C. Christie, “Multivariate Discrimination of EmotionSpecific Autonomic Nervous System Activity,” Master’s thesis, Virginia Polytechnic Institute and State University, 2002. [9] G. Chanel, K. Ansari-Asl, and T. Pun, “Using Neural Network to Recognize Human Emotions from Heart Rate Variability and Skin Resistance,” In Int. Conf. of the IEEE-Engineering in Medicine and Biology Society (IEEE-EMBS), IEEE, 2005, pp. 5523-5525. TABLE I. EMOTION RECOGNITION ACCURACY

V. CONCLUSION AND FUTURE WORK This system, we focus on emotion recognition in the living space. The wearable EEG and ECG sensors with wireless connections were selected. The sensors provided the user mobility and convenience in normal daily life without the limitation of audiovisual problems. The experiment results showed that the primary module (EEG) gave more precise recognition rate than ECG secondary module. However there is a known issue that the EEG’s amplitude is very sensitive with artifact and the quality of data depends on user activities. Removing artifacts entirely is impossible in the existing data acquisition processes. As this reason, we proposed for dynamical switching between sensors when EEG signal degrade. To take advantage of the combining that EEG gives more precise recognition rate and ECG is more stable with noise.

[10] J. Kim and E. Andre, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.30, No.12, 2008, pp. 2067-2083. [11] Sun S and Zhang C. “Adaptive Feature Extraction for EEG Signal Classification,” IEEE Transactions on Medical and Biological Engineering and Computing, Vol 44 (10), 2006, pp. 931-935. [12] Takahashi K. “Remarks on Emotion Recognition from BioPotential Signals,” IEEE Transactions on Autonomous Robots and Agents, 2004, pp. 186-191. [13] Takahashi K and Tsukaguchi A. “Remarks on Emotion Recognition from Multi-Modal Bio-Potential Signals,” IEEE Transactions on Industrial Technology, Vol 3, 2003, pp. 16541659. [14] Iizuka T and Nakawa M. “Emotion Analysis with FractalDimension of EEG Signals,” IEIC Technical Report, Vol 102 (534), 2005, pp. 13-18. [15] Chanel G, Kronegg J, Granjean D., and Pun T. “Emotion Assessment: Arousal Evaluation using EEG’s and Peripheral Physiological Signals,” Lecturer Notes in Computer Science, Springer, 2006, pp. 530 – 537. [16] Savran A, Ciftci K, Chanel G, Javier Cruz Mota, Luong Hong Viet, Sankur B, Akarun L, Caplier A., and Rombaut M. “Emotion Detection in the Loop from Brain Signals and Facial Images,” Final Project Report, eNTERFACE’06, 2006. [17] G. Rizzolatti and L. Craighero, “The mirror-neuron system, “ Annual Review of Neuroscience, vol. 27, 2004, pp. 169-192. [18] E. Oztop, M . Kawato, and M . Arbib, “Mirror neurons and imitation: A computationally guided review,” Neural Networks, vol. 19, 2006, pp. 254-271. [19] P. Ekman and W. V. Friesen, “Pictures of facial affect,” Human Interaction Laboratory. San Francisco, CA: Univ. California Medical Center, 1976. [20] C. Sevcik, “On fractal dimension of waveforms,” Chaos, Solitons & Fractals, vol. 28, pp. 579-580, 2006.

REFERENCES [1] C.-H. Yang, J.-L. Wang, K.-L. Lin, Y.-H. Kuo, and K.-S. Cheng, “Negative Emotion Detection Using the Heart Rate Recovery and Time for Twelve-Beats Heart Rate Decay After Exercise Stress Test,” In Int. Joint Conf. on Neural Networks (IJCNN), IEEE, 2010, pp. 1-6. [2] J. Thayer and G. Siegle, “Neurovisceral integration in cardiac and emotional regulation,” Engineering in Medicine and Biology Magazine, IEEE, Vol.21, No.4, pp. 24-29, 2002. [3] W. Wu and J. Lee, “Improvement of HRV Methodology for Positive/Negative Emotion Assessment,” In Collaborative Computing: Networking, Application and Worksharing 2009, CollaborateCom 2009, 5th Int. Conf. on, IEEE, 2009, pp. 1-6. [4] C. Lee and S. Yoo, “ECG-based Biofeedback Chair for Selfemotion Management at Home,” In Int. Conf. on Consumer Electronics 2008 (ICCE 2008), Digest of Technical Papers, IEEE, 2008, pp. 1-2. [5] J. D. Rodriguez and L. Santos, “Comparative Analysis Using the 80-Lead Body Surface Map and 12-Lead ECGWith Exercise Stress Echocardiograms,” J. of Diagnostic Medical Sonography, Vol.22, No.5, 2006, pp. 308-316. [6] S. R. Vrana, B. N. Cuthbert, and P. J. Lang, “Fear Imagery and text processing,” The Int. J. of the Society for Psychophysiological Research (Psychophysiology), Vol.23, No.3, 1986, pp. 247-253. [7] B. L. Fredrickson, R. A. Mancuso, C. Branigan, and M. M. Tugade, “The Undoing Effect of Positive Emotion,” Motivation and Emotion,Vol.24, No.4, 2000, pp. 237-258.