Facial expression recognition using pca and gabor with jaffe database 11748

Scientific Journal of Impact Factor(SJIF): 3.134

e-ISSN(O): 2348-4470 p-ISSN(P): 2348-6406

International Journal of Advance Engineering and Research Development Volume 1,Issue 5, May-2015

Facial Expression Recognition using PCA and Gabor with JAFFE Database Jigarku mar A Patel Computer Department, R C Technical Institute Abstract — In this paper I discussed Facial Expression Recognition System in two different ways and with two different databases. Principal Component Analysis is used here for feature extraction. I used JAFFE (Japanese Female Facial Expression). I implemented system with JAFFE database, I got accuracy of the algorithm is about 70-71% which gives quite poor Efficiency of the system. Then I implemented facial expression recognition system with Gabor filter and PCA. Here Gabor filter selected because of its good feature extraction property. The output of the Gabor filter was used as an input for the PCA. PCA has a good feature of dimension reduction so it was choose for that purpose.

Keywords- Face Recognition, Eigen Vector, Eigen Value, Principal Component Analysis (PCA), Gabor Filter , Human Computer Interface (HCI), Facial Expression Recognition (FER)

I. INTRODUCTION Facial expression recognition means identify ing a correct expression of the person fro m the image or video sequence.

Face is the first focus of attention in social intercourse. It playing major role in conveying the identity and emotions. To make a good Hu man Co mputer Interface exp ression recognition is very essential. Facial Exp ressions provide important communicat ive cues, which constitute 55 percent of the effect of a co mmun icated message; because of that recognition of facial expressions became very important in HCI. Facial expression recognition can be useful in many areas, for research and application. Studying how hu mans recognize emotions and use them to co mmunicate informat ion is very important topic in anthropology. And the emotion automatically estimated by a co mputer is considered to be more objective than those labeled by people and it can be used in clinical psychology, psychiatry and neurology[1]. Facial exp ression recognition can be added into a face recognition system to improve its effect iveness. In a real-time face recognition system where a series of images or videos of an individual are captured, FER System picks the one wh ich is most similar to a neutral expression for recognition, because normally a face recognition system is trained using neutral expression images.

There are some other possible applications, including emotion surveillance for emp loyees in high work intensity industry, pain assessment, image and video database management and searching, lie detection and so on. With growing terrorist activities all over the world, detection of potential t roublemakers continues to be a major problem. Body language and facial exp ressions are the best ways to know the personality of a person and the response of a person in various situations. The facial expressions tell us about hided emotions which can be used to verify if the information provided verbally is true. These expressions represents the emotional state of a person can serve an important role in the field of terroris m control and fo rensics[2].

II.

Facial Expression Recognition Algorithm

The input of a facial Expression recognition system is always an image o r video stream. The output is an identification or verification of the subject or subjects that appear in the image or video. So me approaches define a facial Exp ression recognition as a mainly four step process - see Figure1.

Figure 1: FER Algorith m @IJAERD-2014, All rights Reserved

887

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406 2.1. Image Preprocessing The image is first processed to extract the features form the image or video sequence , which describe it s contents. The processing involves filtering, normalization, segmentation, and object identification. The output of this stage is a set of significant regions and objects. Image pre -processing often takes the form of signal conditioning , togethe r with segmentation, location, or tracking of the face or its parts. Expression representation can be sensitive to translation, scaling, and rotation of the head in an image. To co mbat the effect of these unwanted transformat ions, the facial image may be geo metrically standardized prior to classificat ion.

2.1.1 Face s pace and Face di mensionality Co mputer analysis of face images deals with a v isual signal that is registered by a dig ital sensor as an array of pixel values. The pixels may encode color or only intensity. After proper normalization and resizing to a fixed x-by-y size, the pixel array can be represented as a point in an xy-dimensional image space by simply writing its pixel values in a fixed order. A critical issue in the analysis of such mult i-d imensional data is the dimensionality, the number of coordinate necessary to specify a data point.

2.1.2 Image s pace Vs Face space In order to specify an image in the image space, we needs to specify every pixel value. “Thus nominal” dimensionality of the space, dictated by the pixel representation, is xy -a very high number even for images of modest size however, much of the surface of a face is smooth and has regular texture.

The value of a p ixel is typically h ighly correlated with the values of the surrounding pixels. Moreover, the appearance of faces is highly constrained, for examp le, any frontal view of a face is roughly symmet rical, has eyes on the sides, nose in the middle, etc. A vast proportion of the points in the image space do not represent physically possible faces. So the natural constraints dictate that the face images will in fact be confined to a subspace, which is referred to as the face space.

2.2 Feature Extraction Feature extract ion- involves obtaining relevant facial features fro m the image or video sequence. These features could be certain face regions, variations, angles or measures, which can be human relevant or not. This phase has applications like facial feature tracking or emotion recognition. Finally, the system does recognize the facial expression.

Feature ext raction involves several steps - dimensionality reduction, feature extraction and feature selection. These steps may overlap, and dimensionality reduction could be seen as a consequence of the fea ture ext raction and selection algorith ms. Both algorith ms could also be defined as cases of dimensionality reduction.

Dimensionality reduction is an essential task in any facial expression recognition system. The performance of a classifier depends on the amount of sample images, nu mber of features and classifier co mplexity. One could think that the false positive ratio of a classifier does not increase as the number of features increases. Ho wever, added features may degrade the performance of a classification algorith m. Th is may happen when the number of train ing samples is small relative to the number the features.There are Several A lgorith ms can be used for Feature Extraction of an Image.

I used Principal Co mponent Analysis and Gabor wavelet transform ( Gabor filter) fo r feature ext raction in my Facial Exp ression Recognition System. Firstly feature of the image extracted by Gabor filter and the result of that is inputted as Principal Co mponent Analysis for dimension reduction so classifier can easily clas sify the expression of an input image.

2.2.1 Principal Component Analysis Principal Co mponent Analysis is a standard technique used in statistical pattern recognition and signal processing for data reduction and Feature extraction. Principal Co mponent Analysis (PCA) is a d imensionality reduction technique based on ext racting the desired nu mber of principal co mponents of the mult i-dimensional data. The purpose of PCA is to reduce the large dimensionality of the data space to the smaller intrinsic d imensionality of feature space which are needed to describe the data economically. This is the case when there is a strong correlat ion between observed variables.

The first principal co mponent is the linear co mbination of the orig inal dimensions that has the maxim u m variance; the nth principal co mponent is the linear co mbination with the highest variance, subject to being orthogonal to the n -1 first principal co mponents. The main idea o f the princip le co mponent is to find the vectors that best account for the distribution of face images within the entire image space. These vectors define the subspace of face images, wh ich we call “face space”[3]. @IJAERD-2014, All rights Reserved

888

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406

PCA also known as Karhunen-Loeve (KL) transformation or eigenspace projection, a frequently used statistical technique for optimal lossy compression of data under least square sense, provides an orthogonal basis vector-space to represent original data. The first introduction of a low-dimensional characterization of faces was developed at Brown University by Kirby and Sirovich in 1987. Th is was later extended to eigenspace projection for face recognition by Pentland, Turk, Moghaddam and Starner at the Vision and Modeling Group of MIT in 1991. The eigenspace is a subspace of the image space spanned up by a set of eigenvectors of the covariance matrix of the trained images. These eigenvectors are also called eigenfaces because of their face-like appearance. The pro jection of an image into eigenspace will transform the image into a representation of a lower dimension which aims to hold the most important features of the face and make the co mparison of images easier. There are t wo main approaches of recognizing faces by using eigenfaces.

In the appearance model each face in the database is represented as a linear co mb ination of e igenfaces.

The recognition process is done by projecting a test image to be identified into the same eigenspace. The resulting vector will be a point in eigenspace and comparison with the training images is normally done by using a distance measure between the points in eigenspace[4]. The other recognition scheme is called the discriminative model. Two datasets are obtained by computing intrapersonal differences (matching t wo different images of the same individual in the dataset) and ext ra personal differences.Two datasets of eigenfaces are generated by performing PCA on each class and a similarity score between two images is derived by calculat ing a Bayesian probability measure.

These eigenvectors can be thought of as a set of features that together cha racterize the variation between face images.

Each image location contributes more or less of each Eigen vector; so that we can display the eigenvector as a sort of ghostly face which we call an eigenface.

Each individual face can be represented exactly in terms of a linear co mbination of the eigenfaces. Each face can also be approximated using only the "best" eigenfaces -those that have the largest eigenvalues and which therefore account for the most variance within the set of face images. The best M eigen faces span an M-Dimensional subspace- "face space" – of all possible images.

Results show that eigenfaces methods are robust over a wide range of parameters and produce good recognition rates on various databases. However outside this parameter range the algorith m can break down sharply[5]. Results show that eigenfaces are very robust to low resolution images as long as the preprocessing step can extract sufficient features for normalizat ion. They also handle high resolution images very efficiently. But significant variation in scale, orientation, translation and lightning will cause it to fail.

STEPS OF PCA  Acquire the initial set of face images (the training set). 

Calculate the eigenfaces fro m the train ing set, keeping only the M images that correspond to the highest eigenvalues.

These M images define the face space. As new faces are experienced; the eigenfaces can be up -dated or recalculated.

Calculate the corresponding distribution in M-dimensional weight space for each known individual, by projecting his or her face images onto the "face space".

MATHEMATICAL FORMULAS OF PCA Images of faces, being similar in overall configurat ion, will not be randomly d istributed in the huge space and thus can be distributed by a relatively low dimensional subspace. The main idea of principal co mponent analysis is to find the vectors that best account for the distribution of face images within the entire image space. These vectors define the subspace of face images, wh ich we call "face space". Each vector is of length N square, describes an N-by-N image, and is a linear combination of o rig inal face images, and because they are face-like in appearance, we refer them to as "eigenfaces".

1.

Let the training set of face images be X1, X2, X3…Xn , then the average set or mean of faces be defined as:

m= Notice that the symbol m to indicate the mean of set X. 2.

The average distance of each face fro m the mean of the data set is given by:

@IJAERD-2014, All rights Reserved

889

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406 Q1 = X1- m; Q2 = X2- m … Qn = Xn- m 3.

The covariance matrix is given by

C = A*A’ Where A= [Q1 Q2 Q3…………..Qn].

In order to reduce the dimensionality, co-variance can be calcu lated as C = A’*A. 4.

Now Eigen values and Eigen vectors are calculated for the covariance matrix. All the face images in the database are projected in to Eigen space and weight for each image is calculated. Then image vectors for each face image is obtained as Image Vector =

5.

Sort eigen values and corresponding eigenvectors in decreasing order.

6.

Select first d <= n Eigen vectors and generate data set in new representation.

7.

Projected test image is co mpared to every projected training image and result is the train ing image closest to t he test image.

Figure 2. FER using onl y PCA To summarize the eigenfaces approach to the facial expression recognition involves the following steps: 

Firstly, the train images are utilized to create a low dimensional face space. This is done by performing Principal Co mponent Analysis (PCA) in the t rain ing image set and taking the principal co mponents (i.e. Eigen vectors with greater Eigen values). In th is process, projected versions of all the train images are also created.



Secondly, the test images also projected on face space.



Thirdly, the Euclid ian distance of a projected test image fro m all the projected train images are calculated and the minimu m value is chosen in order to find out the train image which is most similar to the test image.



Fourthly, in order to determine the intensity of a particular expression, its Euclidian distance fro m the mean of the projected neutral images is calculated.

@IJAERD-2014, All rights Reserved

890

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406 Somet imes projections may suppress important features. This is one difficulty associated with this alg orith m since smallest variance features may not be unimportant. Another disadvantage is that PCA do not focuses on class discriminatory informat ion.

2.2.2 Gabor Wavelet transformati on (Gabor Filter) Dennis Gabor was a Hungarian physicist who is most notable for inventing holography, an achievement fo r which he later received the Nobel Prize in Physics in 1971. It was in 1946 that the first time frequency wavelets (Gabor wavelets) were introduced by Dennis Gabor , who at that time was researching into commun ication theory.

Gabor filters are often used in image processing and are based on physiological studies of the human visual cortex [6].

Gabor filter is one of the most successful approaches for processing images of the human face. Lyons et al. (1998) proposed a Gabor wavelet based facial exp ression coding system and show that their representation method has a high degree of correlation with the human semantic ratings. In Zhang etal. (1998), Gabor filter banks based facial expression coding for feature extraction and multilayer perceptron(M LP) based feature classification is reported to have performed better than geometric feature based facial expression recognition. In this paper, the feature ext raction method proposed in Lyons et al. (1998) is adopted. Principal co mponent analysis (PCA) is used for reducing the length of the feature vector.

Gabor filters are used extensively for facial feature ext raction. A set of Gabor filters with d ifferent frequencies and orientation are used to recover various frequencies and orientations of image textures [7]. The feature outputs from Gabor filters encode the spatial frequencies as well as the orientation activities of various textures in an image. In most cases, Gabor wavelets of five different scales, {

}, and eight orientations, {0,

}, are

used for face or facial expression recognition.

Fig 3 The real part of the Gabor filters with five frequencies and eight orientations for fmax =π/2, the ro w corresponds to different frequency fm, the column corresponds to different orientation θn Gabor filter works as a band pass filter for the local spatial frequency distribution, achiev ing an optimal resolution in both spatial and frequency domains. The 2D Gabor filter Gθ,f,σx,σy (x, y) can be represented as a complex s inusoidal signal modulated by a Gaussian kernel function as below:

where, xθ = xcosθ + ysinθ Yθ= -xsinθ + ycosθ σx σy are the standard deviations of the Gaussian envelope along the x- and y- dimensions, f is the central frequency of the sinusoidal plane wave, and θ k the orientation. The rotation of the xy plane by an angle k θ will result in a Gabor filter at the orientation θk .The angle is defined by:

@IJAERD-2014, All rights Reserved

891

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406

(k-1) for k=1,2,3……p , p ε N θk =

Here we have applied gabor filter on an images. In this we use values of frequency 12 , orientation pi/3 , respectively 2 and 4. The result of that image is shown here

Input image

σ x σy is

Output Image

Fig 4: Gabor filter example The use of Gabor filtered facial images has been shown to result in imp roved accuracy for facial expression recognition.

One approach to using thes e filters is to generate a bank of filters across mult iple spatial frequencies and orientations.

The filtered outputs are then concatenated and down-sampling using PCA. It is often used to reduce dimensionality.

Figure 5: FER using Gabor & PCA 2.3 Classification Exp ression categorizat ion is performed by a classifier, which often consists of models of pattern distribution, coupled to a decision procedure. A wide range of classifiers, covering parametric as well as non -parametric techniques, has been applied to the automatic expression recognition problem. The t wo main types of classes used in facial exp ression recognition are action units (Aus), and the prototypic facial expressions defined by Elman. The 6 prototypic expressions relate to the emotional states of happiness, sadness, surprise, anger, fear, and d isgust. Ho wever, it has been noted that the variation in co mplexity and meaning of expressions covers far mo re than these six exp ression categories.

Moreover, although many e xperimental expression recognition systems use prototypic expressions as output categories, such expressions occur infrequently, and fine changes in one or a few discrete face parts communicate emotions and intentions. @IJAERD-2014, All rights Reserved

892

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406 Eigen Faces are also used to classify facial expression. It has been assumed that, facial exp ression can be classified into some discreet classes (like anger, happiness, disgust or sadness) whereas: 1. Absence of any expression is the “Neutral” exp ression.

2. Intensity of a particular expression can be identified by the level of its “dissimilarity” fro m the Neutral exp ression.

Representing the facial expressions in this way has several advantages. Firstly several kinds of expressions can be represented using only two types of informat ion (1. Class that an expression belongs to and 2. Intensity of the expression). Secondly, it is possible to identify an expression as a mixtu re of two or more expressions (such as 60% anger, 20% disgust and 20% sad etc.).

IV. Experi mental Results In order to improve efficiency of Facial Exp ression Recognition we have to use more than one algorithm for feature extraction. We are using Gabor Filter and Principal Co mponent Analysis for feature extract ion.

By using Gabor Filter we can efficiently e xtract the feature of the image and then after we will reduce the dimensionality of the features from the image. principal component analysis will also help to get the direction of the principal features.

After that we will perform classification using Euclid ian distance between the training images and test image.

Firstly, For input we have taken total 31 pictures. There are 7 anger faces, 9 d isgust faces, 9 happy faces, 3 Neutral faces and 3 Sad faces in the testing set.

A confusion matrix is an excellent means to present the correctness of the facial expressions determined by humans or algorith ms. It states the confusion of facial expressions based on the ground truth, compared to the results of our survey or the results of an algorithm.

The recognition rate is determined by computing the ratio of the cases in which one facial expression was classified correctly and the total number of classificat ions for that facial expression.

Total 20 expressions are truly classified fro m the 31 expressions of the testing set. That means the accuracy of the algorith m is about 65%. Fro m above table we can say that true positive ratio o f the anger and disgust has highest and happy has lowest true positive ratio.

We used standard facial exp ression database JAFFE, and used only PCA fo r facial exp ression recognition.

For input we have taken total 66 pictures. There are 10 anger faces, 8 disgust faces, 10 fear faces, 9 happy faces, 10 Neutral faces and 9 Sad faces and 10 surprise faces in the testing set.

Table 2: Confusion Matrix of algorithm with PCA using JAFFE Database

Now we implemented Gabor filter and the output of that Gabor filter given to the PCA as an input for dimension reduction. For input we have taken total 66 pictures. There are 10 anger faces, 8 disgust faces, 10 fear faces, 9 happy faces, 10 Neutral faces and 9 Sad faces and 10 surprise faces in the testing set.

@IJAERD-2014, All rights Reserved

893

International Journal of Advance Engineer ing and Research Development (IJAERD) Volume 1, Issue 5, May 2014, e-ISSN: 2348 - 4470, print-ISSN: 2348-6406 Table 3: Confusion Matrix of Gabor filter and PCA with JAFFE Database

Comparison of Results: Here we got 70% efficiency of the Facial Exp ression Recognition System using only Principle Co mponent Analysis with JAFFE database. Then if we implement Facial Expression Recognition System using Gabor filter and Principle Co mponent Analysis with standard facial exp ression JAFFE database, we g ot 76 % efficiency.

IV. Concludi ng Remarks We have implemented Firstly Facial expression Recognition System using Principal Co mponent Analysis. But I got accuracy of this algorith m is around 65-70%. I felt that feature extraction Capability of PCA is no t as good as its dimension reduction capability. So I used another Feature Extraction Technique wh ich is Gabor Filter. By using Gabor Filter and Principal Co mponent Analysis in the Feature Ext raction phase, got accuracy of the algorithm is 76%. So now I can say that fro m the results, By using Gabor filter for feature ext raction of the face and after apply ing Gabor Filter result of that image we can g ive as an input of the PCA for dimension reduction purpose may give me imp roved results then only Principal Co mponent Analysis based Facial Exp ression Recognition System.

REFERENCES [1] M. Pantic, L.J.M . Rothkrantz, "Automatic Analysis of Facial Exp ressions: the State of the Art", IEEE Trans.

Pattern Analysis and Machine Intelligence, Vol. 22, No.12, pp. 1424-1445, 2000 [2] Mehrabian.A, 1968. "Co mmunicat ion without Words", Psychology Today, Vo1.2, No.4, pp 53 -56.

[3] Essa, A. P. Pentland, “Coding, Analysis, Interpretation, and Recognition of Facial Exp ressions”, IEEE Trans.

Pattern Analysis and Machine Intelligence, Vo l. 19, 1997, pp.757-763 [4] G. R. S. Murthy, R.S.Jadon, “Effectiveness of Eigen spaces for Facial Exp ressions Recognition ” International Journal of Co mputer Theory and Engineering, Vo l. 1, No. 5, December, 2009,1793-8201.

[5] Peter N. Belhu meur, P..Hespanha, and David J. Krieg man, “ Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Project ion” , IEEE Transactions on Pattern Analysis and machine Intelligence, VOL. 19, NO. 7, JULY 1997.

[6] Daug man, J.G., "Two-dimensional spectral analysis of cortical receptive field profiles", Vision Res., vol 20 (10), pp.

847–56 [7] Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S., 1998. “Co mparison between geometry-based and Gabor wavelets-based facial expression recognition using mult i-layer perceptron.” Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition., 1998

@IJAERD-2014, All rights Reserved

894