Toward Interactive Computer Systems Based on Eye-tracking Technology Modernizing Didactics of Visual by CyberEmpathy Journal of Visual Communication

BOLESŁAW JASKUŁA, KRZYSZTOF PANCERZ

Toward Interactive Computer Systems Based on Eye-tracking Technology Modernizing Didactics of Visual Art Perception Mode of access: Internet via World Wide Web.

Abstract In the paper, we outline the problem of application of eye-tracking technology for creation interactive computer systems enabling us to

BOLESŁAW JASKUŁA, KRZYSZTOF PANCERZ

modernize didactics of visual art perception. Different aspects constituting the theoretical rudiments of such systems are characterized on the basis of the literature.

1. Introduction Four eras of eye-tracking research can be distinguished by the emergence of interactive applications. In 1, Rayner gives the characteristics of first three eras. The first era (ca. 1879–1920) is defined

Researches on the changes of art perception done in Institute of Biomedical Informatics, University of Information Technology and Management Rzeszów, Poland

by the discovery of many basic eye movement facts, including saccadic

Scientific team:

suppression, saccade latency, and the size of the perceptual span. The

Prof. Andrzej Głowacki; Dr Bolesław Jaskuła; Dr Krzysztof Pancerz; Mgr Marika Wato;

second era (ca. 1930–1958) is characterized by a more applied research focus related to the behaviorist movement in experimental psychology. The third era (ca. 1970–1998) is determined by improvements in eye movement recording systems facilitating increasingly accurate and easily obtained measurements. According to 1

Putko, A.: Cognitive interpretation of works of art in the artist's and the layman's unsophisticated experience. Polish Psychological Bulletin, 20, 1989, pp. 33-41..

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception Duchowski 2, eye-tracking research is entering its fourth era. This era can be distinguished by the emergence of interactive applications. In general, varied eye-tracking applications, used contemporarily, can be divided into two main groups, diagnostic or interactive, according to the system analysis. An excellent survey of eye tracking applications in reading and other information processing tasks (scene perception, visual search, natural tasks, auditory language processing, etc.) is given in 1 and 3. In our research, we are interested especially in a scene perception task. No apparent strategies for scene viewing have been easily discerned 2. Contrary to reading, there appears to be no canonical scanpath for particular objects 4. There may be context differences at play. Kroll [19] states that while there may be similarities between reading and scene viewing tasks, the tasks are very different. Henderson and Hollingworth 5 discerned at least three important reasons to understand eye movements in scene viewing: • eye movements are critical for the efficient and timely acquisition of visual information during complex visual-cognitive tasks, • how we acquire, represent, and store information about the visual environment is a critical question in the study of perception and cognition, • eye movement data provide an unobtrusive, online measure of visual and cognitive information processing.

A part of scene perception studies, particularly interesting, is related to perception of art. Fixation positions are found to be highly regular and related to the information in the pictures 5. Data coming from such analyses provide the evidence that eye movement patterns during complex scene perception are related to the information in the scene, and hence, to perceptual and cognitive processing of the scene.

2 3 4 5

Buswell, G.T.: How People Look at Pictures. University of Chicago Press, 1935. Rayner, K. (ed.): Eye Movements and Visual Cognition: Scene Perception and Reading.Springer-Verlag, New York, 1992. Kahneman, D.: Attention and Effort. Engelwood Clifts, 1973. Henderson, J.M.: Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11), 2003, pp. 498–504.

www.CyberEmpathy.com

2. Eye tracking as the method examining visual art perception Our eyes can be used as a rich source of information about us. Eyes are reported to be “windows to the soul”. Surely, they are windows to the mind. The connection between eye movements and thoughts is one of the important parts of a number of major research areas. Some of them are listed in 6:

• In psychology and the cognitive sciences, eye-movement is studied to help analyzing the cognitive processes in a variety of task domains, among others, reading, arithmetic, and word problems. • In neurobiology and vision research, attention is focused on the underlying mechanisms of eye movements as well as solid foundations for understanding fundamental characteristics of human visual processing.

• In human-computer interaction, eye movement is studied to better understand interface use and to develop successful eye-driven user interfaces. Shortly speaking, the eye-movement analysis becomes very popular in many research fields. Eye-movements deliver important guides to human behavior, for example, which information is used in problem solving and when it is used, how much time people is needed to process given information, when previously encoded information is forgotten and reviewed 6. It is worth noting that, in many applications, eyemovement data are collected non-invasively. A data collection process does not disturb the task performance. Eye-movement data can be supplemented with other kinds of data, for example, verbal. The process of collecting eye-movement data is not a simple task. On the one hand, eye-movements are very flexible and informative, and hence, they are an excellent source of data for many studies and applications. On the other hand, gathering eye-movement data is very timeconsuming, and the data are difficult to analyze. Complex experiments can deliver enormous sets of eye-movement data. Moreover, they include a great deal of noise due to human and equipment variability. For these reasons, there is a need to use computer tools in the eyemovement analysis. Such tools use a new class of algorithms that 6

Saarinen, V-M., Laine-Hernandez, M., Saarelma, H.: The Influence of Image Content Levels and Looking Type on Eye Movements. Graphic Arts in Finland 35(2), 2006, pp. 1-12.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception analyze eye movements using a rigorous form of protocol analysis called tracing. Tracing is the process of mapping observed action protocols to the sequential predictions of a cognitive process model. Tracing enables us to analyze data in many contexts, for example, in the cognitive sciences, tracing supports prototyping and evaluating models of visual attention using large sets of eye movement protocols. Eye-tracking is a technique for measuring eye movements enabling us to know both where a person is looking at any given time and the sequence in which their eyes are shifting from one location to another. A short description of eye-tracking technology is presented, among others, in [26]. Here, we remind only the most important facts. Many different devices have been used to track eye movements, for example, electrodes mounted on the skin around the eye, large contact lenses covering the cornea and sclera. Such devices were quite invasive. Unlike mentioned devices, most modern eye-tracking systems use video images of the eye to determine where a person is looking, i.e., the so-called “point-ofregard”. Most commercial eye-tracking systems available today measure point-of-regard by the “corneal-reflection/pupil-centre” method 7. They usually consist of a standard desktop computer with an infrared camera mounted below or next to a display monitor, with image processing software to locate and identify the features of the eye used for tracking. The software identifies the centre of the pupil and the location of the corneal reflection. Next, the vector between them is measured, and, with further trigonometric calculations, point-of-regard is found. Videobased eye trackers need to be fine-tuned to the examined person (its eye-movement). It is done by the so-called calibration process. What a person is looking at is assumed to indicate the thought “on top of the stack” of cognitive processes 8. Eye-movement recordings can deliver a dynamic trace of where a person’s attention is being directed in relation to a visual display. Additionally, measuring moments when the eyes are relatively stationary gives information about the amount of processing being applied to objects at the point-of-regard. Eyemovement recordings enable the HCI researcher to define “areas of interest” over certain parts of interface under evaluation. In the eye-tracking analysis, two measures play a very important role. They are fixations and saccades. Fixations are determined by moments when the eyes are relatively stationary. Saccades are quick eye 7 8

Duchowski, A.T.: A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments, and Computers, 34, 2002, pp. 455-470. Jung, R.: Art and Visual Abstraction. In: R.L. Gregory (ed.), The Oxford Companion to the Mind, Oxford University Press, New York, 1987, p. 46.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception movements occurring between fixations. In some applications, pupil size and blink rate are also studied. Fixations and saccades are the basis for calculation of other metrics, for example, used in “gaze” and “scanpath” measurements. A scanpath describes a complete saccade-fixate-saccade sequence.

Fig. 1. Elements of eye-tracking analysis.

In 9, Zics taken heed of two still significant and relevant claims concerning the relationship between attention and eye movements. Both of them indicate an importance of the eye in processing information and give significance to the underlying voluntary processes of the mind as the control mechanism of attention or imagination.

The first claim comes from Hermann von Helmholtz. He stated that the eye’s optical characteristics and information gathering is rather poor. Therefore, vision is only possible with some form of unconscious inference that makes sense of the information based on prior experiences of the world 10. Studying the relationship between eye movements and visual attention, Von Helmholtz discovered the phenomena of “covert attention”, which explained that visual attention is not always where eye fixation is directed to. One can attend to stimuli without shifting visual focus. The second claim comes from William James. He provided an explanation of a different aspect of visual experience. He claims that attention and imagination are directly related. When attention does not regulate one’s sense organs it consists in imagining the things or actions that one is attending to, or looking 9

Zangemeister, W.H., Sherman, K., Stark, L.: Evidence for a global scanpath strategy in viewing abstract compared to realistic images. Neuropsychologia, 33, 1995, pp. 1009-1025.

Görgen, K.: Combining Eyetracking and EEG. Publications of the Institute of Cognitive Science (PICS), vol. 15-2010, Osnabrück, Germany, 2010.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception for 11. • In the analysis of eye-movements, the physiological capacities of the eye play an important role 9. The size of the visual field is limited. It can be divided into: • “parafoveal vision”, between 2 and 10 degrees off centre, responsible for low-resolution compressed information, • “peripheral field”, more than 10 degrees off centre. The visual field cannot be processed from one single fixation (lasting between 180 and 275 ms), as a result of the limited acuity of the retina. Therefore, rapid eye movements (saccades) are necessary to bring the retinal image of an object of interest to lie on the fovea (lasting between 10ms and 100ms). Attention initially assigns a target before saccade eye movements happen. During saccades, vision is dormant and new information is acquired only during fixation 12. Zics in 13, takes notice that although there is a long history of research on the relationship between eye movement and aesthetic experience, the methods applied often reduce the viewer’s act to a mechanistic agency and, as such, makes the aesthetic claims of such work questionable. She critically assessed this fact and quoted some examples showing that psychological accounts have subsequently been popular in the study of eye movement research, among others, Alfred L. Yarbus who used a scanpath (a graph of saccades and fixations) and the visual recording of eye movements (fixations and saccades) to study complex scenes to identify mainly task-dependent patterns of fixations

and Daniel

Berlyne who used a concept of “diversive-specific” behavioral patterns of the viewer 15. According to Zics, it is argued that real-time dynamic processes allow a meaningful exploitation of eye movement for particular aesthetic production. In 16, Josephson and Holmes taken heed of the significance of scanpath theory. Noton and Stark’s scanpath theory

predicts that a subject

Iwasaki, M., Inomara, H.: Relation Between Superficial Capillaries and Foveal Structures in the Human Retina. Investigative Ophthalmology & Visual Science, 27(12), 1986, p. 1698-1705.

Henderson, J.M., Hollingworth, A.: Eye Movements During Scene Viewing: An Overview. In G. Underwood (ed.), Eye Guidance in Reading and Scene Perception, Elsevier, Oxford, 1998, pp. 269-294. 13

Zangemeister, W.H., Sherman, K., Stark, L.: Evidence for a global scanpath strategy in viewing abstract compared to realistic images. Neuropsychologia, 33, 1995, pp. 1009-1025. 14 15

Vogt, S., Magnussen, S.: Expertise in pictorial perception: eye-movement patterns and visual memory in artists and laymen. Perception, 36, 2007, pp. 91-100.

Ahmad, G., Ohsawa, Y., Nishihara, Y.: Cognitive Impact of Eye Movements in Picture Viewing.International Journal of Intelligent Information Processing, 2(1), 2011, pp. 1-8. 16

James, W.: The Principles of Psychology, Vol. 1. Henry Holt, New York, 1890.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever BolesĹ&#x201A;aw JaskuĹ&#x201A;a, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception scans a new stimulus during the first exposure and stores the sequence of fixations in memory as a spatial model, so that a scanpath is established. When the subject is reexposed to the stimulus, the first few eye movements tend to follow the same scanpath established during the initial viewing of the stimulus, which facilitates stimulus recognition. Research also indicates that when a subject is presented with a blank screen and told to visualize a previously seen figure, the scanpath is similar to when he or she viewed the figure 18. Noton and Stark claimed that the internal representation of a pattern in memory is a network of features and attention shifts, with a habitually preferred path through the network, corresponding to the scanpath. During recognition, this network is matched with the pattern, directing the eye or internal attention from feature to feature of the pattern 19. In

20,

Saarinen, Laine-Hernandez and Saarelma taken heed of the

influence of image content levels and looking type on eye movements. Visual perception is based on the collaboration between the eyes and the brain. Eyes collect visual stimuli from a scene and the brain forms visual percepts and interprets eye movements. During observations of visual scenes, the gaze shifts from place to place with rapid voluntary eye movements (saccades) and vision collects information with static fixations, which occur between saccades. The most interesting areas of the scene are observed first and then the gaze goes through the same areas again. Hence, the least interesting areas are not observed at all and the most interesting areas get the attention over and over again 21. According to Kahneman

22,

eye movements can be classified

into three categories with respect to viewing situation: spontaneous looking (free viewing of a visual scene), task-relevant looking (viewing with a certain task in mind), and orientation of thought looking (a person is concentrated on something else than the visual scene). Visual content consists of syntactic and semantic levels. Syntactic levels define the visual elements and are in close relation to visual perception. Semantic levels are concentrated on visual concepts and they define the meanings of the visual elements and of their arrangements. Factors that influence observation at the syntactic level include contrast, size, shape, color, and movement. Interesting semantic level elements include different human 17 18 19 20 21 22

Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004. Salvucci, D.D.: Mapping Eye Movements to Cognitive Processes. Ph.D. Thesis, Carnegie Mellon University, Pittsburgh, USA, 1999. Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004. Rayner, K.: Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin, 124(3), 1998, pp. 372-422. Vogt, S., Magnussen, S.: Expertise in pictorial perception: eye-movement patterns and visual memory in artists and laymen. Perception, 36, 2007, pp. 91-100. Just, M. A., Carpenter, P. A.: Eye fixations and cognitive processes. Cognitive Psychology, 8, 1976, pp. 441-480.

www.CyberEmpathy.com

3. From a layman to a connoisseur of art The eyes can be trained to seek out aesthetic qualities of visual compositions and that this aspect of formal art training can directly influence a trained viewer's perceptual analysis and appreciation of visual compositions

23.

24,

Jung claimed that in general, the

significance and beauty of art are only apparent to those who can see and are trained in viewing. According to this claim, different questions on the relationship between training and the perception of art can be posed

25.For

example, how does formal art training influence

perception? Does such art training result in changes in how visual compositions are scanned and what type of information is processed during perceptual analysis? How does the scanned information contribute to the formation of aesthetic judgments? The use of eyemovement recording as a method of studying the role that the eyes play in analyzing and interpreting visual compositions is one of research topics. Buswell

and Brandt 27 indicated important differences in eye-

fixation patterns between art-trained and untrained viewers. A number of researches tried to quantify the nature of these differences, for example, Yarbus

28,

Molnar 29, Locher and Nodine

30, 31.

Fig. 2. Comparison on scanpaths: (a) expert – connoisseur, (b) expert – teacher of art, (c) novice - student

23 24

Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004.

Josephson, S., Holmes, M.E.: Visual Attention to Repeated Internet Images: Testing the Scanpath Theory on the World Wide Web. Proc. of the 2002 symposium on Eye Tracking Research & Applications (ETRA'2002), New Orleans, Louisiana, USA, 2002, pp. 43-49.

25 26 27 28 29

Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004. Brandt, H.F.: The Psychology of Seeing. Philosophical Library, New York, 1945.

Vogt, S., Magnussen, S.: Expertise in pictorial perception: eye-movement patterns and visual memory in artists and laymen. Perception, 36, 2007, pp. 91-100.

Locher, P.J., Nodine, C.F.: Symmetry Catches the Eye. In: J.K. O'Regan, A. Levy-Schoen (eds.), Eye Movements: From Physiology to Cognition, Elsevier, Amsterdam, 1987, pp. 353-361.

Kroll, J.F.: Making a Scene: The Debate about Context Effects for Scenes and Sentences. In: K. Rayner (ed.), Eye Movements and Visual Cognition: Scene Perception and Reading. Springer-Verlag, New York, 1992, pp. 284-292.

Locher, P.J., Nodine, C.F.: The Perceptual Value of Symmetry, Computers and Mathematics with Applications, 17, 1989, pp. 475-484.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception Picture perception in artists and laymen has been compared in several studies on the assumption that artists view the world differently from others, for example, Putko found that artists employed different cognitive interpretations of 16 well-known paintings

32,

Nodine, Locher, and Krupinski found that

artistically untrained viewers tended to spend more time viewing individual objects than relationships among elements in paintings

33.

Mentioned authors

used descriptive eye-movement data. Other authors have examined different eye-fixation patterns in artists and non-artists viewing pictures, for example, Antes and Kristjanson found that eye-movement parameters alone could differentiate artists from non-artists by fixation densities on less important aspects of paintings 34, Zangemeister, Sherman and Stark found that artists and sophisticated viewers used a more global scanning strategy than artistically untrained participants when viewing abstract pictures In

36,

35.

Nodine, Locher, and Krupinski concluded that compositional design

influences how trained viewers look at paintings by supplementing the viewer's perceptual strategy in guiding attention to structural relationships among pictorial elements that express narrative themes. An individual's aesthetic judgments are based on the perceived structure among pictorial elements (lines, shapes, colors, surfaces) and the interpreted narrative theme.

4. The characteristics of didactic computer systems based on eye-tracking technology The eye-tracking system constitutes one of the main parts of didactic interactive computer systems. A principle of operation is based on the optical measurement. The infrared rays come from the system to eyes for tracing the eye-movements. Next, the infrared rays are reflected to the system from eyes and the data of eyemovements are recorded. The eyes are controlled by human brain where cognitive processes are produced. The eye-movement data can be recorded using two most common formats such as Heat Map and Sequenced Gazing with circle of concentration. In case of Heat Map, the track of eye is recorded as illumination and intensity of infrared light rays. In case of Sequenced Gazing, the eye tracks are recorded as numbered circles with their areas indicating the time duration of eye’s gazing in those areas, respectively. The measurement of eye-movements is now common in

Poole, A., Ball, L.J.: Eye Tracking in Human-Computer Interaction and Usability Research. In: C. Ghaoui (ed.), Encyclopedia of Human Computer Interaction, IGI Global, 2006, pp. 211-219. 33 34

Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004.

Ahmad, G., Ohsawa, Y., Nishihara, Y.: Cognitive Impact of Eye Movements in Picture Viewing.International Journal of Intelligent Information Processing, 2(1), 2011, pp. 1-8. 2. Antes, J.R., Kristjanson, A.F.: Discriminating artists from nonartists by their eye movement patterns. Perceptual and Motor Skills, 73, 1991, pp. 893-894. 35 36

Yarbus, A.L.: Eye Movements and Vision. Plenum Press, New York, 1967. Niedermeyer, E., da Silva, F.L.: Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincot Williams & Wilkins, 2004.

www.CyberEmpathy.com

CYBEREmpathy ISSUE 1/2012. Cyber Fields Forever Bolesław Jaskuła, Krzysztof Pancerz, Toward Interactive Computer Systems Based on Eye-tracking Technology Medernizing Didactics of Visual Art Perception many diverse areas of cognitive psychology. In general, the measurements consider each eye movement event independently. However, the pattern of inspection can only be revealed by considering a sequence of successive fixations. These patterns are sometimes referred to as scanpaths or scan patterns 37. Functioning of the module of didactic correction is based on realtime supervision of deviations of scanpaths of a learning person from pattern scanpaths (artist-expert’s scanpaths of image analysis). Several different basic methods for comparing scanpaths are reviewed in

38.

The main criteria

for evaluating methods for comparing scanpaths take into consideration whether they appropriately capture the degree of similarity between different

scanpaths and the ease of testing this statistically.

Figure 3. A system structure

5. Conclusions Interactive computer systems based on eye-tracking technology are only the first step towards modernizing the means supporting didactics of visual art perception. Conducted research indicates functional and topographical differences between two groups, artists and nonartists, during the performances of visual perception and imagery of paintings were presented by means of EEG (Electroencephalography) phase synchrony analysis (cf. 39). EEG measures voltage fluctuations resulting from ionic current flows within the neurons of the

37 38

Helmholtz, H.V.: Treatise on physiological optics. Continuum, Rochester, 1911.

Stark, L.W.: Sequences of fixations and saccades in reading. In: J. Ygge, G. Lennerstrand (eds.), Eye movements in reading, Elsevier, Tarrytown, 1994, pp. 151-163. 39

Ahmad, G., Ohsawa, Y., Nishihara, Y.: Cognitive Impact of Eye Movements in Picture Viewing.International Journal of Intelligent Information Processing, 2(1), 2011, pp. 1-8.

www.CyberEmpathy.com

40.

The analysis of phase synchrony from EEG signals yields new

information about the dynamical co-operation between neuronal assemblies during the cognition of visual art. Therefore, it is necessary to equip interactive computer systems with additional correction loops, which functionality is based on analysis of the EEG biosignals. Combining eye-tracking and EEG is one of the most fascinating and challenging problems (cf. 41). Scientists are dealing with this problem to close the gap between the psychological mental processes leading to the sensations we experience in everyday life and their underlying physiological biochemical processes. They are trying to understand perception, the process by which our brain makes sense out of the signals coming from our senses. Eye-tracking alone is not sufficient because provides only psychophysical data. It provides valuable information about the gaze location, but does not provide any information about neuronal activity. On the other hand, EEG, measuring neuronal activity in humans, does not directly provide information about the gaze position. For this reason, projects combining EEG and eye-tracking into one common setup have been started by scientists.

Molnar, F.: About the Role of Visual Exploration in Aesthetics. In: H.I. Day (ed.), Advances in Intrinsic Motivation and Aesthetics, Plenum Press, New York, 1981, pp. 385-413. 41

Duchowski, A.T.: A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments, and Computers, 34, 2002, pp. 455-470.