Let’s think about music an approach to user eeg congruent dynamic music in video games final

Let’s Think about Music: An Approach to User EEG Congruent Dynamic Music in Video Games Paul R. Murdoch

Abstract From the dawn of computers to the current day the PC and its processing capabilities have taken massive leaps. Computers once as big as rooms can now be housed inside a mobile phone. However the way we interact with computers has remained essentially the same; a mouse and keyboard, for the most part, is all that is utilised.

The world of video games is perhaps the best place to look for advancements that address this problem. Game controllers; from Guitar Hero controllers to Xbox’s Kinects gesture tracking technology allow users to interact with technology in new ways. Utilising technology such as eyetracking and leading in the field of virtual reality, the multi-disciplinary nature of games makes them a fascinating area for research.

An area that has not been researched in depth in relation to games is BCI. Brain-computer interfacing utilises user brainwaves as a means of communicating with computers. The aim of this research is to create a system that will allow for the altering of dynamic music in video games using BCI technology. Developing a working system that will allow users to interact with game audio with nothing more than their thoughts, passively and actively controlling the nature of the sound within the game.

This study outlines the development of such a system. Created using consumer technology and software and contextualized within existing research, the two prototype systems discussed have shown promise in early usability testing.

Acknowledgements I would like to thank Steven Walters my project supervisor, Dr Don Knox and the other lecturers who have expanded my understanding and allowed me carry out this research. Also I would like to acknowledge the authors of all research used to inform my study. Finally to Christopher Mitchell-lay for his assistance with soldering and Craig Easton and Craig Stewart for their interest and input in the development of my work.

Contents 1. Introduction ........................................................................................................................................ 1 1.1

Context .................................................................................................................................... 1

1.2

Scope and Objectives .............................................................................................................. 1

1.3

Achievements.......................................................................................................................... 1

1.4

Overview ................................................................................................................................. 2

2. Background ......................................................................................................................................... 3 2.1 Music in Video Games................................................................................................................... 3 2.1.1 The History of Game Sound ................................................................................................... 3 2.1.2 Categorizing Game Sound ...................................................................................................... 4 2.1.3 Problems with Basing Game Sound Theory in Existing Film/Television Literature ............... 4 2.1.4 Implementing Dynamic Game Music ..................................................................................... 5 2.1.5 Procedural Music ................................................................................................................... 5 2.1.6 Limitations of Current Game Sound ...................................................................................... 6 2.2 Immersion ..................................................................................................................................... 6 2.2.1 Immersion’s Foundation in Existing Psychology .................................................................... 6 2.2.2 Understanding Immersion ..................................................................................................... 8 2.2.3 Measuring Immersion ............................................................................................................ 8 2.3 Music and Immersion, the Psychology of Music........................................................................... 9 2.3.1 Music and Emotion ................................................................................................................ 9 2.3.2 Music as a Tool ....................................................................................................................... 9 2.4 Biometric Data, Brain-Computer Interfacing and Music ............................................................ 10 2.4.1 EEG ....................................................................................................................................... 10 2.4.2 ERP ....................................................................................................................................... 10 2.4.3 Brain-Computer Interfacing ................................................................................................. 10 2.4.4 Brain-Computer Musical Interfaces ..................................................................................... 11 2.5 System Testing ............................................................................................................................ 11 2.5.1 Utilising the System Usability Scale (SUS) ............................................................................ 11 2.5.2 Interpreting SUS data ........................................................................................................... 12 2.5.3 SUS as More Than a One Dimension Scoring Method ......................................................... 12 3. Developing the Systems .................................................................................................................... 13 3.1 Overview ..................................................................................................................................... 13 3.1.1 The Neurosky Mindwave Mobile ......................................................................................... 13 3.1.2 Arduino Boards .................................................................................................................... 14 3.1.3 Max MSP .............................................................................................................................. 14

3.2 Configuring the Hardware........................................................................................................... 14 3.2.1 Configuring Arduino One ..................................................................................................... 14 3.2.2 Configuring the BlueSMiRF Silver......................................................................................... 15 3.2.3 Configuring Arduino Two ..................................................................................................... 16 3.3 Developing the Dynamic Music System in Max MSP .................................................................. 17 3.3.1 Overview and Approach....................................................................................................... 17 3.3.2 Arduino2Max ....................................................................................................................... 17 3.3.3 Developing the AGRO system .............................................................................................. 17 3.3.4 Creating Music for the AGRO system................................................................................... 18 3.3.5 AGRO’s Potential Uses in Games ......................................................................................... 18 3.3.6 Developing MESS ................................................................................................................. 19 3.3.7 Creating a Dynamic Soundscape .......................................................................................... 19 3.3.8 MESS’s Potential Use in Games ........................................................................................... 19 3.4.1 Implementing BCI Data in Game Engines ............................................................................ 19 3.4.2 Applications Outside of Video Games.................................................................................. 20 4. Testing the Systems .......................................................................................................................... 20 4.1 Overview ..................................................................................................................................... 20 4.2 Testing ......................................................................................................................................... 21 4.2.1 Setting .................................................................................................................................. 21 4.2.2 Participants .......................................................................................................................... 21 4.2.3 Ethics and Participant Consent ............................................................................................ 21 4.2.4 AGRO System Testing ........................................................................................................... 21 4.2.5 MESS Testing ........................................................................................................................ 22 4.2.6 Possible Issues with Testing ................................................................................................. 22 4.3 Results ......................................................................................................................................... 22 4.3.1 AGRO System Usability Score .............................................................................................. 22 4.3.2 MESS System Usability Score ............................................................................................... 23 4.3.3 A Look at Collected AGRO Qualitative Data ......................................................................... 23 4.3.4 A Look at Collected MESS Qualitative Data ......................................................................... 23 5. Discussion.......................................................................................................................................... 23 5.1 Considering the Results........................................................................................................... 23 5.2 Utilising Systems of this Nature in Games .............................................................................. 24 5.3 BCI and Immersion .................................................................................................................. 24 5.4 Uses Outside of Gaming .......................................................................................................... 24 6. Conclusion ......................................................................................................................................... 25 References: ....................................................................................................................................... 26 ii

Appendix 1 Collected Test Results .................................................................................................... 30 Appendix 2 Individual Digitized Test Results with Participants Statements ..................................... 32 Appendix 3 Participant Consent Form .............................................................................................. 71 Appendix 4 System Usability Testing Questionnaire ........................................................................ 72 Appendix 5 AGRO Max MSP Patch.................................................................................................... 73 Appendix 6 MESS Max MSP Patch .................................................................................................... 74 Appendix 7 Additional Resources ..................................................................................................... 75

iii

iv

1. Introduction In the study of games and game sound user experience is often the focus, developers and designers maintain this focus to better understand and ultimately enhance the experience of their games.

Their aim is to create a great experience for the user, they want the user to be engaged with the game. Immersion has been described as ‘the prosaic experience of engaging with a video game’ (Jennet,) it is the crux of user experience and the term given to the act of becoming engaged with the game world.

Different types of immersion exist within games such as imaginative immersion; having an emotional investment in the game or its characters and challenge-based immersion; the involvement created when the user is challenged by the game’s tasks. Audio elements in games relate mostly to the third type of immersion described by Ermi and Mayra (2005); sensory immersion. Sensory immersion relates to the audiovisual world created in games, the increasingly realistic and dramatic graphics and sounds of the virtual world.

Measurements of the user’s physiological state are increasingly being used to understand their experience of the game, outside of more conventional observational methods and questionnaires.

Biometric readings have been found to help with game testing with up to 63% more problems being identified than with observation alone (Mirza-babaei, 2011). However biometrics could lend more to games than just research, it has been suggested that future work could be carried out to use players physiological state to inform the game itself. In regards to physiological data’s potential relating to sensory immersion the literature proposes that: “The creation of affective on-the-fly sounds according to the player’s psychophysiology state can be used to enhance or counteract a state.” (Nacke and Grimshaw, 2010) This research hopes to explore the possibility of using EEG to influence game music.

1.1 Context The research used to inform the development of the systems described in this study draws from many different fields. Neurology, psychology, human-computer interaction, game development and particularly sound design and implementation of sound assets in games. This research was used to guide the development of technical aspects of the audio systems.

1.2 Scope and Objectives The fact that this work benefits from the research of several fields also means that it could successfully benefit several areas. The dynamic systems developed by no means would be exclusive to music, the same system could, in theory, inform other aspects of the game world. Furthermore BCI as a means of interacting with music and audio could easily have a variety of uses outside of game development.

The objectives of this study are to develop a working BCI congruent audio system. With a focus on usability, responsiveness and creating a system the user can feel to be in control of.

1.3 Achievements Two systems where developed utilising different readings from the EEG headset used. The ‘Attention Guided Re-Orchestrator’ (AGRO) which uses player’s current level of attention to alter the intensity of the music. Users have been successfully able to control this system by focusing or allowing their 1

minds to wander, responding with very little apparent latency. The ‘Meditation Enhancing Soundscape System’ tracks the user’s level of meditation, this is used to reduce intensity of the soundscape heard by the user. With high levels of meditation users can reduce the soundscape to white noise, which is said to aid meditation. Users again successfully interacted with this system showing the ability to successfully manipulate the system by becoming meditative.

1.4 Overview The following research analyses existing literature on dynamic game music, immersion in games and BCI. It also outlines a method for developing EEG congruent dynamic audio systems for implementation in games and analyses these systems for effectiveness.

2

2. Background 2.1 Music in Video Games 2.1.1 The History of Game Sound The role of sound and music in games has changed drastically as computer games and the way we interface with them has developed. This section looks to briefly study the progression and history of game music in order to better contextualise modern techniques and approaches to creating game music.

When establishing a timeline for game sound Collins (2008) refers to early computer science and arcade machines as the technological and functional ancestors of video games.

“If video games had parents, one would be the bespectacled academic world of computer science and the other would be the flamboyant and fun penny arcade.”

Collins sees the amalgamation of early computer games such as ‘Spacewar!’ which was developed in 1962 and like many early games included no sound and arcade gambling machines that used sounds to attract and intrigue users. In early slot machines we see the first examples of sound being used to communicate information about the game to the user, with noises indicating a win or other events.

When discussing the history of game music McAlpine et al note rise of popularity of video games in the late 1970s and early 80s. Here we see the convergence discussed by Collins in early arcade games such as Space Invaders and Pacman. The sound in these early games was used in a similar way to slot machines to attract attention and communicate the player’s progress and events within the game.

It is in these early instances that the important relationship between the development of games and technological restrictions revealed. Game sound up until and including today has very much been shaped by the limitations of technology. At the time of Pac Man analogue music such as vinyl and tape was popular, however these formats could not easily be integrated with arcade machines and so computer chips where used to generate game sound using synthesis. This led to the characteristic 8-bit music and synthetic sound effects of early video games.

As home consoles for gaming began to gather popularity in the 1980s and sound chips such as the Yamaha YM and Motorola 68000 allowed for more audio channels to be available to game designers along with the ability to play multi-channel sampled sounds. This resulted in dramatic improvements in the realism of sound and music replicated in games. As this technology was realized composers started to push the limits of game music, fantastic examples of game scores where created such as Super Mario Bros. and Zelda. McAlpine considers these examples, though limited by the technology of the time, to still “remain current” with the music from these early games still being popular.

With the introduction of CD-ROM in the 1990s marked the introduction of even greater quality audio in video games, with the introduction of CD to newer consoles synthesised sounds and generative methods lost popularity when creating music in games. Instead high quality recordings could be used allowing an influx in composers for games as creating game music became less computer science and more accessible to composers who had traditionally written for other mediums.

3

Today a series of techniques are used to create game music, the power of modern computers and consoles allows for music to react to in-game events and player actions. Middleware software such as FMOD and wWise allows sound designers to carefully create and arrange music to be coherent with game play.

2.1.2 Categorizing Game Sound Perhaps the most unique trait of game music is its non-linearity, unlike older media such as television or film where the score accompanies the visual aspect of the show or movie in a predetermined, start to end fashion. In games the user can participate for hours and controls the nature of the games events and progression, so the music therefor has to respond appropriately and faces the challenge of not becoming repetitive and boring.

Despite this much of game sound theory is still heavily based in film theory and literature. However some have tried to create new ways of categorizing sounds in games, for instance Stockburger (2003) categorizes in-game sounds in relation to the objects they are attached to within the game engine. Such as score or music sounds, zone sounds relating to a particular area in the game world or interface sounds created for example by interacting with in-game menus. Collins categorises game sounds in based on their relationship to the player or the game world. She describes ‘interactive audio’ as sound events that react to the player’s interaction with the game world such as the sound of a door opening or a weapon pick up. As well as adaptive audio or sound events in game that react dynamically to changes in the game world environment. These two elements she uses to define dynamic audio in games, where by the game creates its own sound that responds adaptively and interactively to create unique music for the user.

McAlpine et al categorizes game music into four main classes: title music, menu/hi-score music, cut scenes and in game music. Relating these categories to existing media comparisons title music is described as playing ‘a similar role to that of the theme tune of a television program’, establishing the mood of the game and musical themes. Menu or hi-score music is compared to elevator music, as the short theme related music played as the user navigates the games menus and options. Cut scene music is most similar to traditional film or television music in that it narrates and provides context to the cut scene in a linear fashion of which the user has no control. In-game music in comparison is the most interactive, allowing for the most scope for dynamic music as this is what is heard as the player interacts with the game.

2.1.3 Problems with Basing Game Sound Theory in Existing Film/Television Literature Sound and music in video games is by its very nature different from linear sound implemented in film and television. For this reason film sound terminology when applied to game sound can be “confusing and at best inaccurate” (Jorgensen). Whalen on the other hand argues that many tropes of cartoons and horror films are heavily relied upon by games today suggesting that apparels are obvious. However this problem still causes division in game sound literature.

An example of this discussed by Jorgensen in ‘Time for a New Terminology’ is that of the diegesis.

The diegesis is a term first coined by Plato in the Republic, Plato describes two forms of narration; diegesis and mimesis. Diegesis is when the speaker speaks as himself while recounting a tale and does not suggest that he is narrating as anyone else. Mimesis refers to imitating someone else and narrating a story from their perspective.

The term was introduced into film literature in the 1950s to describe the fictional world of movies.

Where the sounds within the fictional world are counted as diegetic while sounds seen to come from outside of this world, for example background music, are seen to be non-diegetic. Jorgensen points 4

out that this model applied to game music can be inaccurate, as game music could be seen traditionally to be non-diegetic. In games music can react to things happening within the fictional world of the game, music can alert the player to an approaching enemy for instance. Considering this game music is then in some way involved and informed by the fictional world of the game and so the traditional idea of the diegesis is challenged.

This outlines the difficulty of creating new ways of categorising game sound, where continued work is needed to continue to effectively define game audio with respects to its non-linear and often complex nature.

2.1.4 Implementing Dynamic Game Music So now having reviewed dynamic music in games, its history and definitions how is it implemented in games today? McAlpine et al describe three main methods of integrating adaptive music into games that have been established since the mid-1990s.

The first of these is event-driven music cues, music that is related and triggered by in game cues or events such as completing a level or fighting a boss. This is described as the ‘simplest and often the most effective system of adaptive music’ as it reacts simply to predetermined events in the game.

Horizontal resequencing is slightly more complicated than the previous method, segments or loops of music are transitioned between based on parameters from the game engine. Such as proximity or number of enemies or player health. Music may increase move to a more tense or dramatic loop when enemies approach, then return to the previous calmer loop when the enemies are gone. This approach like event-driven music cues can be programmed in the game engine, horizontal resequencing also can be achieved using middleware such as FMOD. Using middleware allows much more fluid transitions between music segments.

Vertical Reorchestration refers to the method of using game parameters to arrange a musical score in real time. More elements or tracks in the score can be introduced as the game gets more intense for example; building up tension. The music can then also get simpler as the games state of tension decreases. This depends heavily on creating good music capable of working with this kind of arrangement. This method can also be refined through the use of middleware software such as FMOD.

2.1.5 Procedural Music Procedural music in video games differs from standard sample based music discussed so far, it uses algorithms to generate random musical content in real time. On the simplest level random numbers can be generated and assigned to musical notes to be synthesised this is what is referred to by Wooller et al. as generative algorithms. The problem with these simple truly generative algorithms is that music can be erratic and not always strictly musical, a more common approach are ‘transformational algorithms’. Transformational algorithms lay out musical phrases or structures that can be filled with randomly generated notes, this more controlled method allows game engines to create randomized music that can be augmented by in game parameters.

This method of creating music ‘on the fly’ can be extremely advantageous for games as algorithms can replace music samples that can take up valuable memory space. It also allows for the creation of limitless music, lowering the chance the player will hear repetitive music. Though the benefits of this method are obvious, it is still fairly new and has not been implemented in a great deal of games.

An example of this is the music created by Leonard Paul for the game Sim Cell, using Pure Data (a visual programming language akin to Max MSP the software used for this study) procedural music 5

was synthesized in real time for the game. This shows the potential for utilising this different approach to creating in game music and its benefits on storage space.

2.1.6 Limitations of Current Game Sound It can be seen that historically technology has restricted game sound, however with increasingly powerful processing, advanced game engines, software and techniques game sound and music is reaching new levels of adaptability and complexity.

However storage space still remains a problem as game sound is only part of the overall game and only has allocated space, procedural methods may help address this problem in the future. Game engines could also become more aware of game states, at this time game music reacts to the current state of the game. Game music could theoretically be improved by allowing game engines to monitor not just current but also previous states of the game over time. This way music will not only react to the exact state of the game but will refer to previous states, this is useful because if the game state is fairly intense for a long time then increases a small amount typical game engines would not recognize the significance of this. However if states where monitored game music could be contextualised better, reacting more dramatically to changes in the game.

Another limitation of current game sound models is their reliance on in game states, the focus of this study is to take biometric data directly from the player to evaluate other methods of informing adaptive audio systems. Effectively cutting out the middle man of the game engine and getting data about the player directly, allowing the player a new level of passive interaction with the audio system. In the future these techniques could be combined with existing audio approaches discussed to expand the potential interactivity of adaptive sound generation.

2.2 Immersion 2.2.1 Immersion’s Foundation in Existing Psychology Video games have progressed massively from early arcade games like Pong, however the study of immersion in games is still in its infancy. Though immersion had very little written on it until recently, engagement and experience are areas that have been investigated. A psychological concept that is a corner stone for most writing on immersion is Csikszentmihalyi’s concept of flow.

Csikszentmihalyi (1990) described flow as: “A state of concentration so focused that it amounts to absolute absorption in an activity” His research showed that people enjoyed challenging activity that they had the skills to complete, becoming absorbed by those activities was a pleasant, focused and relaxed state. Where immersion relates to gameplay experience alone, flow is a much broader concept, instead relating to engagement with any task, from a game of tennis to writing an essay or playing the clarinet. People are often aware of the happiness they achieve with these tasks and also the challenge in them. Flow was described as having many elements, we can see how these elements relate to and can be encouraged by video games.

A challenging activity that requires skill To experience flow an individual must be suitably challenged yet able to complete the task. In video

6

games the challenge of the game is an integral component, it often adjust over time getting more and more difficult as the user progresses and gains a better understanding of game mechanics. So a game can be seen a continually challenging activity that requires concentration.

The merging of action and awareness This relates to the experience of engaging with an activity, when great effort either mental or physical is exerted to engage with that activity. This results in a deep involvement and has obvious parallels in video games though often with greater regard to mental focus.

Concentration on the task at hand During flow an individual experiences deep focus, often ignoring other tasks or aspects of their life.

Similar to finding serenity in playing an instrument, the focus exerted on games can also remove the user from other aspects of life.

The paradox of control In a challenging task a sense of ownership or control over that task occurs, exercising that control enhances the user’s experience. In games we are often able of a great degree of control, over virtual characters, events and environments.

The loss of self-consciousness This is the experience of becoming more involved in the task than you are with yourself. This kind of brief disconnect is often found to be enjoyable, in video games players often experience not only this disconnection from themselves but often a connection with in game characters they are controlling.

The transformation of time A common experience of flow is the loss of time or an altered sense of time, while carrying out the activity. The idea that ‘time flies while we’re having fun’ is a popular one and something that is experienced often when playing video games.

So flow is the joyous phenomena experienced when challenge is met with skill and focus.

Understanding flow we can equate immersion to the experience of flow in video games, this is a common viewpoint in schools of thought relating to immersion.

7

2.2.2 Understanding Immersion Using existing psychology we can start to understand and study immersion, many researchers are beginning to construct models to explain the experience. One such model is the SCI model of immersion proposed by Ermi and Mayra. Their model suggests that we engage with a game on multiple levels and that these must be understood to fully evaluate immersion. This model suggests that immersion goes just beyond the challenge based concept of flow to include other immersive factors. The first point of contact for immersion with video games is ‘sensory immersion’, this is collaboration of audio and visual aspects of the game to create an exciting and immersive virtual world. In regards to sound design in video games, the focus of this study, sensory immersion is key.

Not only does the game audio involve the user, it also is often the first means of getting a user to become more engaged with the game. Again challenge is acknowledged in immersion, the second type of immersion described the model is challenge-based immersion eluding to flow. Finally there is imaginative immersion, the player’s eventual emotional connection to the games plot and characters. In the world of sound design it could be argued that sensory immersion is the most important. Though audio could perhaps effect the emotional responses of players or increase the tension around a challenge, being able to look for a sound systems on sensory immersion as well as over immersion could lead to stronger findings.

Brown and Cairns (2004) also further develop the idea of immersion, suggesting that immersion is made up of three distinct stages that can be achieved. The first is ‘engagement’ this achieved when the player begins to invests effort into understanding the game and mastering its controls, the player must overcome the barrier of ‘gamer preference’ in other words they must have an interest in the game to initially become involved with it. Brown and Cairns describe the next level of immersion as ‘zen like’, the player becomes ‘engrossed’ in the game the player has invested time in the game and has overcome the challenges of its construction. The player is now at ease with the controls and the game itself, experiencing the kind of relaxed, controlled state of focus often referred to in literature relating to immersion. The final stage of total immersion sees the player feel empathy for the game’s story or plots and they experience a very deep engagement with the game, ignoring everything outside of the screen and focusing entirely on the game world.

Work like this highlights the similarities between immersion and the experience of flow, both in the kind of barriers that have to be overcome to achieve it and its consequences but also the nuances of immersion in games and its unique variation of flow. The involvement of the player with a challenging experience housed in a vivid virtual world combines engaging media and user involved challenge creating a catalyst for flow experiences. However upon beginning to establish a concept of immersion the next question is how to measure it. Effective ways to measure the experience must be created, founded in this existing knowledge, if we are to better understand immersion and potentially use player immersion to inform games.

2.2.3 Measuring Immersion Measuring a player’s user experience or UX is done in several ways in the game industry. Most games go through user testing to eliminate bugs and problems before release, so though not always observed with regards to immersion, user experience is often monitored. Standard methods include observation of players, think-aloud methods where the user talks through their experience of the game and commonly interviews and questionnaires. A well-established questionnaire in game testing is the Game Experience Questionnaire (GEQ), this will be utilised in the study to gather qualitative data. The GEQ is informed by existing knowledge about immersion and tests for concepts like those found in flow, the questionnaire has been tested by Jeanne and Brockmyer (2005) using classical and Rasch analytical methods, they described player engagement as a ‘quantifiable 8

construct’ using the GEQ, however they noted certain problems with the development of the GEQ. Including the demographics tested in its creation.

Increasingly physiological readings are being utilised to test user experience, biometric data can provide real-time information on a player’s physiological and immersed state. Several studies have been carried out to test the validity of physiological data for measuring immersion, various different methods have been tested including facial Electromyography (EMG) (Hazlett, 2008). Others have tested Galvanic Skin Response (GSR), heart rate, EEG and even eye-tracking (Nacke and Lindley, 2008). These measurements can give us interesting insight outside of subjective observation, however though the players state can be measured with biometrics their state could always been influenced by stimulus outside the game. A combination of both methods can give strong results measure player experience.

2.3 Music and Immersion, the Psychology of Music 2.3.1 Music and Emotion In Collins “Making Gamers Cry: Mirror Neurons and Embodied Interaction with Game Sound” she explores the ability of audio to further user interaction to enhance users embodiment in the game world. She uses examples such as ‘Guitar Hero’ where users ‘play’ the game music using a guitarshaped controller. In this way the player has a more intimate, physical relationship with the game sound. It could be possible that using BCI in a similar way when generating audio in games could also help to increase the players feeling of embodiment within the game world.

She also discusses the player’s emotional relationship with game music, suggesting that certain things can be done to increase the emotional content of game music and sound and its importance.

“Music is one of the key elements that drives emotion in media.”

She discusses how we associate the sound of how instruments are played with our mental schema of the performer. When we listen to music we imagine the performer and understand the emotion as they perform and convey it. Collins suggests that “particularly in scenes of high emotion, music in games could be more effective in communicating emotion if recorded with live performers.” Collins identifies the human feeling applied to music through gestures or physical performance allow the music to be more relatable and therefor convey greater emotion. This has interesting implications on adaptive and particularly generative music used in games. Any music however has a psychological impact on us, as it takes up our sense of hearing it elicits a “psychological interpretation by the individual” (Cunningham, 2011).

2.3.2 Music as a Tool Jorgensen (2006) describes the essential nature of music in games claiming “the fictional world seems to disappear and that the game is reduced to rules and game mechanics.”

Outside of conveying mood and emotion to the user, music can also work to give the player useful information. Musical cues can be used to indicate approaching enemies, low player health or the discovery of a new location. Music works here to subtly enhance the communication between the game world and the user.

9

2.4 Biometric Data, Brain-Computer Interfacing and Music 2.4.1 EEG EEG or electroencephalogram is a method of using electrodes placed on the scalp to measure electrical brain activity, pioneered by Hans Berger in 1924. When neurons in the brain communicate information to one another the small electrical signals can be detected, together some 10^6 neurons create a measureable reading of a few microvolts that can be monitored this way. Despite being filtered by the meninges, skull and scalp there is no latency in this method of monitoring (Woodman, 2010).

EEG is a particularly useful technique in that it is not intrusive, it ‘has become one of the most useful tools in the diagnosis of epilepsy and other neurological disorders’ (Miranda, 2005). Various methods of EEG analysis exist including Hjorth, power spectrum and spectral centroid analysis, these methods take the frequency of the EEG and extrapolate readings from them often using FFT or similar approaches.

EEG has some downfalls however; eye movement, eye blink and heartbeat can all cause artefacts in the EEG readings many modern methods of EEG monitoring use algorithms to account for this kind of bodily noise that is introduced.

2.4.2 ERP Event-Related Potentials (ERP) are a form of biometric measurement that uses non-intrusive means to measure the brains activity. Like EEG they monitor brainwaves but rather than presenting individual frequencies of brainwaves they average the signals and relate that signal to a brain function, such as concentration or relaxation. A commonly used ERP is the P300, this ERP component is thought to be related to decision making. Working with more tangible concepts such as concentration means we can take this physiological reading and apply it to immersion in a more direct way. We know that player focus is important to immersion so directly measuring it is far more useful than attempting to relate EEG readings to immersion.

2.4.3 Brain-Computer Interfacing From the dawn of computers to the current day computers and processing have taken massive leaps. Computers once as big as rooms of small buildings can now be housed inside a mobile phone.

However the way we as users interact with machines has changed very little, we still rely on peripheral devices such as the mouse and keyboard to control and interface with machines.

Brain Computer Interfacing mean’s to provide new ways for users to communicate with computers, by utilising predominantly ERP signals machines can be trained to react to the specific brain functions of the user. The possibilities of BCI where explored by Vidal in 1973 in his paper ‘Towards Direct Brain-Computer Communication’, seeing the potential of BCI in future technology Vidal asked if “observable electrical brain signals can be put to work as carriers of information in man-computer communication?”. Suggesting that BCI could be used in the future to control “prosthetic devices or spaceships”.

However these concepts are being realized today, in ‘A Novel BCI-Controlled Pneumatic Glove System for Home-Based Neurorehabilitation’ Coffey discusses development of prosthetic hand for rehabilitation, importance is given in the study to the fact that this can be achieved inexpensively and with consumer technology. This is just one example of many BCI systems being developed 10

today. The recent presence of EEG monitoring headsets available to consumers such as the Epoc Emotiv or the Neurosky Mindset used in this study has resulted in an increase in popularity of this kind of research. With brain signal monitoring no longer being restricted to neurology labs and hospitals, BCI is becoming a keen area of study across many disciplines including game development and music generation.

2.4.4 Brain-Computer Musical Interfaces BCMI (brain-computer musical interfaces) combine BCI technology with musical instruments or software to allow user brain signals to generate or augment music or a performance. Surprising examples of BCMI can be traced back to the 1960’s, with the first musical performance by EEG being by minimalist composer and pioneer Alvin Lucier. In his 1965 “Music for Solo Performer”, Lucier utilised amplified EEG signals to communicate electronically with loudspeakers. These loud speakers where mounted with different types of percussion allowing Lucier to play a percussive arrangement only by thinking.

Today research focuses on creating systems for creating more complex music with EEG and ERP data. In Miranda 2005 paper “Towards Direct Brain-Computer Musical Interfaces”, a nod to the early work of Vidal, methods of creating BCMIs is explored. A method of using certain brain signals to generate musical passages of a specific style was created and tested. One key problem faced when creating these BCMIs outlined by Miranda is that though neuroscience is progressing massively “we still lack a good understanding of their analytical semantics in relation to musical cognition”.

In this study a slightly different approach is taken, user control of a generative musical system is not as direct as in previous studies. Rather the player’s passive and natural cognitive states shall subtly augment the music, with more of a focus on flow experiences and immersed states than on direct ‘brain control’ of the musical system.

2.5 System Testing 2.5.1 Utilising the System Usability Scale (SUS) As this study is focusing on creating a simple system that can be used by anyone user testing was carried out (Testing will be detailed in section 4). The hope was to demonstrate that the system was truly interactive and that users felt they could comfortably and reliably use the systems.

For this reason the System Usability Scale was used. SUS was created by John Brooke in 1986 as a “quick and dirty” method of getting reliable data user data relating to usability, when defining usability Brooke (1986) described it as “the appropriateness to a purpose of any given artefact”. The scoring system was initially used for testing system usability with early computers but has since been used to test usability in a plethora of other areas, from website design to various computer applications such as printers and office equipment.

The SUS questionnaire uses 10 questions (using a Likert scale of 1 to 5) to access the user’s experience of the given system. SUS has gained popularity and is considered to be reliable even on small groups of participants and correlates well with other usability tests. For these research this scale was chosen as a simple and concise means of demonstrating the successful functioning of the developed systems.

11

The questions and Likert Scaling of SUS

2.5.2 Interpreting SUS data The answers from SUS questionnaires are interpreted as a score that is representative of overall system usability. This score is calculated from the submitted answers and not a percentage score.

The answers scaled from one to five are scored differently depending on if they are odd or even.

Even numbers are subtracted by the scale (5) to give a score, while odd numbers have one subtracted from them to give a score. These scores are then summed and multiplied by 2.5 to give a score out of one hundred, this again is not a percentage but a representative score.

In terms of scores 68 is considered average scores are also represented in bands A (80+), B (70+), C (60+), D (50+) and F (below 50).

2.5.3 SUS as More Than a One Dimension Scoring Method The SUS score often come’s under criticism as a one dimensional scoring method. Meaning that the score is only representative of one thing usability. This in a sense also makes the scale robust, but it has also been suggested that other measurements could be extrapolated from the data sets.

Questions 4 and 10 it has been suggested could be used to represent a ‘learnability’ score, allowing ease of learning to be measured. The other 8 questions can also be tallied as before to give a separate usability score. This approach is detailed in ‘The Factor Structure of the System Usability Scale’ (Lewis, 2009) where it is suggested that one can “can decompose the Overall SUS score into its

12

Usability and Learnability components, extracting additional information from their SUS data with very little additional effort.”

3. Developing the Systems 3.1 Overview The early systems created are intended to demonstrate the potential of this approach. They are fairly simplistic in nature. This approach to design was chosen to reduce the overall complexity of the systems and hopefully result in less difficulty creating efficient prototypes. The two main hardware components that operate within the system are Neurosky’s ‘Mindwave Mobile’ and two of Arduino’s Uno boards. Max MSP was utilised as the software used to create the audio system. The hardware communicates with the software via serial data, as detailed in the following sections.

3.1.1 The Neurosky Mindwave Mobile Neurosky produce a series of products, all focusing on their safe and unintrusive BCI technology.

Neurosky produce reliable headsets for measuring EEG brainwaves. Outside of novel consumer uses these products have been used by developers from a series of different fields as a reliable source of EEG data.

The Neurosky Mindwave Mobile is a wireless headset that uses one monopolar electrode on the forehead and an electrode clip that functions as the null electrode, attached to the earlobe. These electrodes use EEG to read and analyse user brain activity. The ‘ThinkGear’ technology used inside the headset uses algorithms to determine the user’s level of attention and meditation. These ‘eSense’ values (measured between 0 and 100) are then sent via Bluetooth to a PC or device.

The research into the validity of this technology has been researched by Katona (2014) among others. In Katona’s evaluation of the Mindwave it was concluded that: “On the basis of the results, the information obtained by the processed brain waves can be used in several research areas, for instance medical research, multimedia applications, games etc.”

The Neurosky Mindwave Mobile, source: Mindwave Mobile Quick Start Guide

13

3.1.2 Arduino Boards Arduino develop open-source easy to use microprocessors, the user has the ability to program these small computers with code. The board can then work with data and communicate electronically and digitally with other devices. Two Arduino Uno boards where used to process and communicate data in this system.

Arduino layout diagram taken from ‘Learning JavaScript and Arduino programming with Johnny-Five’

3.1.3 Max MSP Max MSP is a modular visual programming language used to create interactive media. It has been used by several sound designers, performers and developers in the past because of its flexibility and its user friendly UI. For these reasons it was chosen as the primary software for testing the capabilities of the BCI system.

3.2 Configuring the Hardware 3.2.1 Configuring Arduino One As two Arduinos where used, each with a different function within the system, the Arduinos will be referred to as Arduino One and Arduino Two. The function of Arduino One was to take the eSense values being sent out by Bluetooth, process that data and display it using LED’s. To communicate with the Bluetooth headset a Bluetooth dongle, the BlueSMiRF Silver, was used.

This method was outlined by the Neurosky Developers guide, the diagram below shows the Arduino configuration. It is worth noting that the Arduino’s TX pin should not be connected when using the UNO. 14

Arduino One layout, www.developer.neurosky.com The BlueSMiRF is powered by the Arduino and communicates with the Mindwave headset using Bluetooth. The Bluetooth data is sent to the Arduino from the BlueSMiRF via serial using its TX or transmit pin. The Arduino receives this serial data through its RX pin, from here the code uploaded to the Arduino process the eSense data. The Arduino displays the eSense value (the value used is determined by the code) using a series of ten LED lights. This scales from one LED lit; representing that the headset is properly connected. To ten lit LEDs showing the user has reached ‘100’ for the chosen eSense value.

3.2.2 Configuring the BlueSMiRF Silver The BlueSMiRF module can be loosely connected to the Arduino and powered up. Once this is done it can be connected to a PC as a Bluetooth device. Once connected terminal software is used to access the BlueSMiRF and issue it commands.

CoolTerm software used to access and program the BlueSMiRF.

15

Using a PC to identify the Mindwave’s MAC address the module can be programmed to search and connect to the Mindwave. The appropriate BaudRate and authentication settings can be configured to allow the devices to communicate with one another. It is worth noting that, though not included in the developer documentation, the authentication level or ‘sa’ should be set to 0. Making the BlueSMiRF only discoverable by the headset but not setting the authentication level this way results with an undiscoverable module that you cannot reprogram to accept the headset.

Once this is done the Bluetooth module and the headset can be paired, after this connection is established the readings are sent from the headset to the module and so into the system. The connections on the module where soldered to insure a good signal.

3.2.3 Configuring Arduino Two Having configured the first Arduino the system now has a means to read and display different eSense values. The signals from Arduino One’s digital pins light up the LED array on a bread board.

The signal is then taken from the breadboard into Arduino Two’s corresponding digital pins. When this is done the eSense value can be communicated to Arduino Two, the code uploaded to this second Arduino allows it to send serial data via USB to the systems software. Arduino Two requires this USB connection to communicate however Arduino One can run without USB from a power source. Having this configured the system now displays the chosen eSense value and communicates it via USB to the Max MSP patch.

Completed system hardware.

16

3.3 Developing the Dynamic Music System in Max MSP 3.3.1 Overview and Approach Max MSP is used as software to test the effectiveness of the system, once the data was in Max MSP the patches began being developed. The challenge was to recreate the studied approaches game sound designers use in Max. A familiar approach often used in middleware software such as FMOD, vertical re-orchestration was chosen as the solution.

A simple Max patch mimics the fading in and out of layered stems or tracks of a particular looped section of music. Creating a sense of swelling and falling music, however instead of utilising conventional in game events and parameters this system uses user EEG. The attention or meditation eSense levels act as the parameter that’s value dictates the re-orchestration of the dynamic game music.

3.3.2 Arduino2Max Arduino2Max is a Max patch available online as an open-source resource on Arduino’s ‘Playground’.

This patch developed by Daniel Jolliffe based on a patch by Thomas Ouellet Fredericks and was intended to allow art students easy integration of their projects with Max. This patch is one of many solutions for reading serial data into Max, but it was found to be the most effective. It allows the user to select a COM or USB port, once connected it represents the data on its UI. The UI displays the Arduino’s pins, displaying their current value. It is worth noting that one connection on the patch needs to be changed based on the operating system being used.

Arduino2Max’s easy to use UI.

3.3.3 Developing the AGRO system The Attention Guided Re-Orchestrator patch works by using the Mindwave’s Attention eSense value.

A series of stem tracks can be imported into the patch where they are looped and played continuously. These tracks pass through a gate which stops the audio being heard unless a certain eSense value is met. As the user’s attention increases more tracks are introduced. An envelope is 17

placed after the gate to create a gradual fade in transition for each stem track. The currently active tracks are then passed through a reverb module to create a more natural sound. This creates dynamic music that changes and sounds drastically different controlled directly by the user’s level of attention. The music has a generative quality as when the user’s eSense values naturally change the music alters accordingly. It is also possible to control the system; focusing on a particular object, doing mental arithmetic or listening intently to increase the music’s intensity. The music can be caused to decrease in intensity by looking around quickly or trying not to focus on anything in particular.

A version of the AGRO patch in Max MSP

3.3.4 Creating Music for the AGRO system Music was performed using a MIDI keyboard, recorded in Pro Tools 11 the four tracks used in the system where carefully thought out. The ‘bottom’ track or track heard first at the lowest attention value was created in Absynth 5, the static sound is intend to represent a ‘disconnect’ between the user and the device at low levels of attention. The second track introduced as attention increases is some vocal pads created using Outputs ‘Exhale’ vocal engine, this track was used to create depth and a sense of basic melody to the track. Thirdly dramatic percussion is introduced representing high levels of attention and focus, finally a quiet synth is introduced at the highest level of attention. This completes the track, as attention fluctuates between different values the music responds appropriately.

3.3.5 AGRO’s Potential Uses in Games This simple approach to user influenced dynamic music could theoretically be used effectively in games. Games are immersive and demand for high levels of focus and attention from the player, having music that reacts to a user’s intensity of focus could allow the player to have more of an effect on the virtual game world. Static could similarly represent the player break in immersion as they stop focusing on the game or are distracted. Overall this kind of system could interesting overall effects on player immersion and sense of presence in the game.

18

3.3.6 Developing MESS The Meditation Enhancing Soundscape System was created to explore the capabilities of the headsets meditation eSense value. It similarly uses four stem tracks simultaneously looped and reorchestrated. The challenge with MESS was purposely reengineering the system to reduce the number of tracks heard as mediation value increases. Instead of gates automated live.gain objects where used, automating the level of the music to varying degrees as the meditation level increases.

This was to test different various approaches to creating systems of this kind. The transitions in MESS are not as smooth as the fade in transitions of AGRO, though as gates are not used the MESS system is able to fade tracks out.

3.3.7 Creating a Dynamic Soundscape Similarly the music and soundscape of MESS where created in Pro Tools and performed live, in an effort to give the music a more human feel. At total meditation the user hears only white noise.

White noise is often used in meditation to create an audio blanket over other background noises, creating a more uniform and less distracting audio environment. It also works simply to let the user know they achieved a high level of meditation. As meditation lowers a loop of lapping waves is gradually introduced, again in a hope to add to a tranquil and meditative soundscape. A final two tracks; a pad and a peaceful piano track are introduced. Being fully heard when the user isn’t meditating.

3.3.8 MESS’s Potential Use in Games Using a system like MESS games aiming to create relaxed or pleasant virtual worlds could monitor a player’s level of meditation and use dynamic music to react to the user’s state. The implications with MESS do not seem as apparent as with AGRO, however extreme immersion is considered to be like a ‘state of Zen’. So perhaps levels of meditation, like attention, could give some insight into player immersion.

3.4 Possibilities for Further Development 3.4.1 Implementing BCI Data in Game Engines Though the systems developed for this study utilised Max MSP, which was in a sense mimicking the nature of dynamic audio systems in game engines. Further work was carried out in order to establish the possibilities of using the same systems and data inside of real game engines. Using the Unity game engine and a plugin for Unity called Uniduino. Uniduino allows Unity to communicate with the host PC’s serial ports, by doing this it can send and receive data from and to the Arduinos connected. Data is sent to Unity in the form of an integer for each pin on the connected Arduino, a 0 represent an off or LOW pin and a 1 representing an on or HIGH pin reading.

A simple test was carried out to demonstrate the system functioning within Unity, far from a real game it is instead just a cube that can be rotated when the player’s attention value is high enough. Uniduino does most of the work establishing a connection, all that is required is for it to be set up to communicate with the correct COM port. When the connection is established the values can be used in real time within the Unity. A short piece of simple

19

code in the form of a Unity C# script allows these values to interact with the game world, in this instance a floating rotating cube in 3D space.

Simple single page of code used in the floating cube demonstration This additional work shows that the systems created could be utilised inside game engines. This data could then be used to interact with almost every aspect of a game and would not be limited to audio and music. As demonstrated in the spinning block test the system can work to utilise other parts of game engines such as the physics which were applied to the 3D cube using simple code.

This data could also be used to allow players to interact with game mechanics or alter the visual content of the game. The implementation of these early systems could create near endless possibilities for users to interact with the virtual world of games.

3.4.2 Applications Outside of Video Games Outside of video games these systems or similar systems could be used for a variety of purposes.

Enhancing meditation, new means of performing music and new ways of interfacing with virtual instruments. Potential new means of viewing and experience different types of media, means to assists disabled musicians or composers and perhaps ways to inform generative music algorithms with computer learning. A user in early testing of the system likened it to Pandora’s Box, with the potential to incorporate this kind of BCI technology across many fields relating to human-computer interaction.

4. Testing the Systems 4.1 Overview Similar to approaches to other aspects of this study a focus was placed on simplicity. The aim of the tests was to establish the successful functioning and usability of the system. In essence: do the systems work as intended? By giving the subjects simple tasks to carry out while using each system it was hoped the subjects would be able to pay attention to how the audio feedback responded to their thoughts and cognitive activity. The participants were asked to consider how responsive and 20

easy to use the system was before filling out a SUS questionnaire and where encouraged to write down in thoughts they had on the system in the areas provided. This allowed a robust source of quantitative and qualitative data with which to access the systems.

4.2 Testing 4.2.1 Setting The tests were carried out in the author’s study/office room where the system was originally developed. The tests where all carried out between 12am and 6pm over the course of four days to allow for similar lighting conditions at the time of testing. The room was well lit and the area is fairly isolated from extraneous sounds which could cause distractions during testing. Participants tested the system one at a time and where observed during the experiment. They spent the duration of the test seated in a comfortable office chair at a computer desk. The audio playback from the computers M-Audio speakers was set a standardized volume to allow a similar acoustic environment for testing.

It was noted during testing that as I was more familiar with some of the participants than others; some participants therefore where likely to be more familiar with my study than others. This could perhaps cause different results as a newer environment may result in higher attention levels, where as a familiarity with the test environment may be conducive of a more relaxed state in subjects.

4.2.2 Participants Twenty participants where used in the study aged between 18 and 30, as could be expected of this demographic participants had varying, but generally good, experience and knowledge using technology. The participants where made up of seventeen males and three females, a more even distribution of men and women could have been more useful for this study. However it is felt that brain activity, especially at the kind of general levels utilised here are not effected by gender and are innately human allowing for the success of EEG in medical and psychological fields.

The participants also hard varying levels of experience with both tasks used in the experiment: playing Minesweeper and meditating. As the study was more interested in the user’s interaction with the system itself the tasks just acted as a means of a controlled condition throughout the tests.

4.2.3 Ethics and Participant Consent Each participant was issued a consent form (Appendix 3) with their questionnaires (Appendix 4), the consent form informed users of what they would be asked to do and what potential risks there where. As far as risks the headset can cause some discomfort to the earlobe and head if worn for long periods of time but is totally safe. It was felt to be important to acknowledge that though the headset monitored brain activity it in no way directly effects or interacts with participant brain activity.

The participants were told they could back out or end the experiment at any time, that they had the right to withdraw their results and that their results were going to be kept confidential. They were also told that they could request access to this study and any further research carried out using their results.

4.2.4 AGRO System Testing Firstly each participant was assisted with putting on the Neurosky headset and the system was initialised. To test AGRO, the attention based system, participants were asked to play Minesweeper for five minutes and pay attention to how the audio responded to their actions. Minesweeper was chosen as it is a very simple game that would work well in conjunction with audio generated by the 21

system. After the five minute period of time they were encouraged to take a further five minutes to just experiment with headset anyway they wished. Participants were encouraged to experiment with activities they felt influenced the system. However no activates outside of Minesweeper where suggested to those taking part in the study as it was felt this may affect the learnability and usability scores of the system.

After the participant was happy with the time spent using the headset they were given the first part of the questionnaire to fil out pertaining to ‘System A’ the AGRO system.

4.2.5 MESS Testing Again the system was initialized and the participant was seated. This time the participant was simply asked to meditate, being meditative was described as relaxing and trying to clear your thoughts, aided by breathing deeply and closing the eyes. Participants were asked to try and relax in order to bring the music in MESS down to the quiet white noise generated at high levels of meditation. Again this exercise was carried out for five minutes. After this the subjects were given a further five minutes to experiment with the system and get a feel for it.

After this they were asked to fill out the second and final part of the questionnaire pertaining to ‘System B’ or MESS.

4.2.6 Possible Issues with Testing Though measures were taken to control variables the test could still have been a more controlled environment. Though the study used is quiet perhaps an alternative setting would have been better.

Though the system is portable a laptop or computer would have to have the appropriate software installed and configured and in the face of time restraints testing where the system was being developed was more efficient.

The participant group used consisted mostly of employees from a local hotel, most of the participant’s new the author to varying degrees and this may have caused biased results. The participants were encouraged to be honest with their answers.

Literature around SUS has suggested that question 8 ‘I found the system very cumbersome to use’ could cause difficulty with participants. During the test many subjects were unsure of what was meant by ‘cumbersome’. It would perhaps have been better to amend cumbersome to awkward as is sometimes done to make the questionnaire more user friendly.

4.3 Results 4.3.1 AGRO System Usability Score An average of the System Usability Scores of the twenty participant was used to give an overall average SUS score. The attention based system scored an 81/100 which would be considered an A, this very high mark reflects the ease participants demonstrated when attempting to control the system. This score is representative of a system that is very easy to use, taking a further look at the ‘learnability’ score that can be taken from the data 61/100 leaving a ‘usability’ score of 66/100.

Unlike the overall SUS score which is truly a score rather than a percentage these additional two scores are shown as percentages, 61% and 66% respectively. Though extensive use of the SUS score is utilized at an industry standard when developing systems the additional percentages are not. They have been included as means of providing more useful data on the systems and allowing for comparative scoring between the two systems.

22

4.3.2 MESS System Usability Score Again an average score was taken from the twenty participant scores, this resulted in a SUS score of 85/100 for the meditation based system. Again this would be an A and is an extremely high score.

Suggesting an extremely accessible system. Comparing the additional learnability and usability data to the AGRO system MESS scored a 63/100 for learnability and a 69/100 for usability. Suggesting this system was even easier to use and become familiar with than the attention based system.

4.3.3 A Look at Collected AGRO Qualitative Data Participants were asked to comment on what they felt caused a reaction from the system. Any particular activities they could single out or felt that allowed them to manipulate the audio feedback. Generally the feedback was very positive, supporting the high SUS score achieved by the system. Accessing the feedback from the system tests a clear keyword is focus. Users felt that ‘when focusing on something the music became more intense’ and that focusing on an object or particular point allowed them to increase the intensity of the music. Participants also found that looking around quickly or not pay attention to anything in particular decreased the intensity of the music.

4.3.4 A Look at Collected MESS Qualitative Data Overall participants seemed to find MESS easier to use, which was not only reflected in the SUS score awarded to the system but also in participant statements. Participants felt that this system was “a lot better than previous test” and that generally this system was easier to influence than the attention based system. Participants seemed to find the experience relaxing and an audio representation of the user’s meditation level allowed using the system to be an informative and ‘reflective’ experience for users.

As user’s seemed to hone in on the idea of focus with the previous system, closing the eyes, relaxing, breathing deeply and clearing the mind of thoughts where central to user control of MESS.

5. Discussion 5.1 Considering the Results Mechanically the systems work, in terms of the successful communication of EEG data to a dynamic music system. However two areas that caused concern approaching user testing where latency and general usability. Though, as discussed earlier, there is no latency created by the nature of the human skull, it was possible that the signal processing between that point and the audio system could have led to latency. However analysing user feedback and observations made during the twenty tests made it apparent that the latency in the system was very low allowing the system to be extremely responsive. Users where able to quickly manipulate the systems, being able to move between the maximum and minimum values easily and with control.

To create an effective and robust system that utilizes BCI it has to be usable, focusing on usability is perhaps the best way to answer the question of ‘does it work?’ Testing has indicated that the system does work, in the sense that the participants that tested and used the system found it both relatively easy to use and responsive. With both systems achieving an A band SUS score and both systems receiving positive user feedback during testing and in the participant questionnaires.

One obstacle that was considered before testing was the abstract nature of what the subject was being asked to do, individuals spend countless hours focussing on objects and being relaxed but never actively consider how focused or relaxed they are. This creates a situation where when using the system the participants preconceived ideas of attention and meditation are challenged. Subjects 23

seemed to have a realization shortly into using the systems, where they recognized what they could actually do to control the system, in a sense what the system wanted from them. For the AGRO system paying attention or focusing on a particular on a particular object or area allowed users to increase the intensity of the music, while ‘zoning out’, looking around quickly and lack of focus lead to a decrease in the music’s intensity. With MESS users found closing their eyes and clearing their head of thoughts to be extremely useful, in regards to ‘realizations’ many subjects with no prior experience of meditating claimed to have a better idea of what meditation actually was. While subjects who had practiced meditation found that meditative techniques allowed them to control the system easily.

Considering the results it seems that the systems work effectively, allowing users to utilise brain computer interfacing and interact with audio with their thoughts alone. In this sense the objectives set out in this study have been met, though as a very basic version of a BCI system a great deal of further research and understanding remain.

5.2 Utilising Systems of this Nature in Games The ‘floating cube’ demonstration has shown that this system can be utilised to work within game engines. The EEG data from the headset can successfully be sent to Unity, this means that this data could be utilised by game developers to make ‘mind control’ possible in games. From an audio perspective the values could be fed into middleware software such as FMOD allowing for dynamic audio of a far more polished nature. However the data would not be restricted to audio, the users readings could be used to influence the environment passively, for instance colours of objects in the game world could change based on the users level of meditation or attention. The user could also actively use BCI to move objects in the game world or in other ways interact with it.

Essentially within the expanses of a game engine the possible use of this EEG data would be endless and only limited to creativity of the game developer. The tests and development of the system have shown that even though the system is very basic, in regards to utilising consumer technology and software, the system is extremely robust and capable of reliable use as a means of interacting with media.

Of course utilising this kind of system effectively within games would require an understanding of how games effect a player’s state and how immersion and game interactions affect the players attention and meditation levels.

5.3 BCI and Immersion It could be argued that there can be a correlation drawn between high attention and meditation and states of immersion. The act of honing focus in on the game and the ‘zen’ like state described by literature could be represented by player attention and meditation levels. This of course would require further, but it could be possible to not only use a system of this nature to inform game engines and effect the users experience but also monitor it. Attention and mediation levels could be monitored in real time during video game testing. The possibility of the system effecting the game world while simultaneously monitoring the user’s state of immersion and involvement adds an interesting possible duality to this kind of system. Studying the nature of such systems and their interaction with user experience is key to successfully utilising these systems in the future.

5.4 Uses Outside of Gaming There are obvious uses for BCI systems of this nature outside of video games, both in creating new means of interacting with audio but also exploring the potential for such systems to be used for other purposes. In terms of interacting with audio these systems could be used to create new ways 24

of interacting with sound and music. The same approach could be used to create new ways of performing music, creating music generatively and allowing individuals with disabilities to interact with audio systems with greater ease, The ‘reflective’ experience provided by these systems could also act to inform individual users of their own methods of mediation and the nature of attention. Allowing users to familiarize there selves with meditation and attention in a way that could be seen as informative. Many users in particular saw the meditation based system being useful as a meditation aid, even describing it as a potential ‘anxiety aid’ allowing an individual to focusing on relaxing. Research into utilising audio feedback to aid meditation is an area in particular that could be focussed on for future work.

6. Conclusion In conclusion it is felt that the systems created for this study worked successfully, allowing users to effortlessly interact with dynamic audio systems, similar to those in video games, with their thoughts alone. This study was in essence exploratory and was intended to be a first step towards successfully incorporating brain computer interfacing in game sound design. The development of these early systems outline the possibility and potential for using EEG monitoring as a reliable and implementable method when developing video games. Technology has advanced massively since Alvin Lucier’s first ‘brain music’ performance in 1965 and as technology such as virtual reality grows increasingly popular within the world of video games it is not beyond reason to believe that BCI will eventually grow in a similar way. It is hoped that this research will work as a stepping stone from which brain computer interfacing can be used bravely and creatively in the future.

25

References: Agarwal, R. and Karahanna, E., 2000. Time flies when you're having fun: Cognitive absorption and beliefs about information technology usage. MIS quarterly, pp.665-694.

ARTZ, J. 1996. Computers and the quality of life: Assessing flow in information systems. Computers and Society 26, 3 (1996), 7-12.

Bos, D.P.O., Reuderink, B., Van de Laar, B., Gurkok, H., Muhl, C., Poel, M., Heylen, D. and Nijholt, A., 2010, October. Human-computer interaction for BCI games: Usability and user experience.

In Cyberworlds (CW), 2010 International Conference on (pp. 277-281). IEEE. Brockmyer, Jeanne H., et al. "The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing." Journal of Experimental Social Psychology 45.4 (2009): 624634.

Brooke, J., 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), pp.4-7.

Brown, E. and Cairns, P., 2004, April. A grounded investigation of game immersion. In CHI'04 extended abstracts on Human factors in computing systems (pp. 1297-1300). ACM. CĂĄceres, M., 2011, February. Music from the Environment: Perception of Music Created from Sounds of a Video Game Environment. In Audio Engineering Society Conference: 41st International Conference: Audio for Games. Audio Engineering Society.

Christiansen, M., 2013, February. Behind the Mix—An In-Depth Look at the Audioengine in Hitman: Absolution. In Audio Engineering Society Conference: 49th International Conference: Audio for Games. Audio Engineering Society.

Coffey, A.L., Leamy, D.J. and Ward, T.E., 2014, August. A novel BCI-controlled pneumatic glove system for home-based neurorehabilitation. InEngineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE (pp. 3622-3625). IEEE. Collins, K., 2008. Compositional Approaches to Dynamic Game Music.Game Sound.

Collins, K., 2009. An introduction to procedural music in video games.Contemporary Music Review, 28(1), pp.5-15.

Collins, K., 2011, September. Making gamers cry: mirror neurons and embodied interaction with game sound. In Proceedings of the 6th Audio Mostly Conference: A Conference on Interaction with Sound (pp. 39-46). ACM. Collins, K., 2013. Playing with sound: A theory of interacting with sound and music in video games.

MIT Press.

Collins, K., Hodge, A., Dockwray, R. and Karpalos, B., 2015, February. Veemix: Integration of Musical User-Generated Content in Games. In Audio Engineering Society Conference: 56th International Conference: Audio for Games. Audio Engineering Society.

Csikszentmihalyi, Mihaly, and Mihaly Csikzentmihaly. Flow: The psychology of optimal experience.

Vol. 41. New York: HarperPerennial, 1991.

Cunningham, S., Grout, V. and Picking, R., 2010. Emotion, Content, and Context in Sound and Music. Game Sound Technology and Player Interaction: Concepts and Developments: Concepts and Developments, p.235.

D. C. Dennet, Brainstorms - Philosophical Essays on Mind and Psychology. Brighton, UK: Harvester Press, 1978.

26

Drossos, K., Floros, A. and Kanellopoulos, N.G., 2012, September. Affective acoustic ecology: towards emotionally enhanced sound events. InProceedings of the 7th Audio Mostly Conference: A Conference on Interaction with Sound (pp. 109-116). ACM. Durr, G., Peixoto, L., Souza, M., Tanoue, R. and Reiss, J.D., 2015, February. Implementation and Evaluation of Dynamic Level of Audio Detail. In Audio Engineering Society Conference: 56th International Conference: Audio for Games. Audio Engineering Society.

Ermi, Laura, and Frans Mäyrä. "Fundamental components of the gameplay experience: Analysing immersion." Worlds in play: International perspectives on digital games research 37 (2005).

Gargava, P., Sindwani, K. and Soman, S., 2014, October. Controlling an arduino robot using Brain Computer Interface. In Reliability, Infocom Technologies and Optimization (ICRITO)(Trends and Future Directions), 2014 3rd International Conference on (pp. 1-5). IEEE. Garner, T. and Grimshaw, M., 2011, September. A climate of fear: considerations for designing a virtual acoustic ecology of fear. InProceedings of the 6th Audio Mostly Conference: A Conference on Interaction with Sound (pp. 31-38). ACM. Garner, T., Grimshaw, M. and Nabi, D.A., 2010, September. A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game. In Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound (p. 10). ACM. Grierson, M. and Kiefer, C., 2011, May. Better brain interfacing for the masses: progress in eventrelated potential detection using commercial brain computer interfaces. In CHI'11 Extended Abstracts on Human Factors in Computing Systems (pp. 1681-1686). ACM. Grierson, M., 2008, July. Composing with brainwaves: minimal trial P300b recognition as an indication of subjective preference for the control of a musical instrument. In Proceedings of the ICMC. Grimshaw, M., 2009. The audio Uncanny Valley: Sound, fear and the horror game.

Hiemstra, G., 1983. You say you want a revolution?‘Information technology’in organizations. Communication yearbook, 7, pp.802-827.

IJsselsteijn, W.A., de Ridder, H., Freeman, J., Avons, S.E., 2000. Presence: concept, determinants and measurement. In: Proceedings of the SPIE, vol. 3959, pp. 520–529 Jennett, Charlene, et al. "Measuring and defining the experience of immersion in games."

International journal of human-computer studies 66.9 (2008): 641-661.

Jørgensen, K., 2006. On the functional aspects of computer game audio.

Jørgensen, K., 2010. Time for new terminology. Diegetic and nondiegetic sounds in computer games revisited. Teoksessa Grimshaw, Mark (toim.): Game Sound Technology and Player Interaction: Concepts and Developments. Hershey, PA: Information Science Reference, pp.78-97.

Karageorghis, C.I., Terry, P.C. and Lane, A.M., 1999. Development and initial validation of an instrument to assess the motivational qualities of music in exercise and sport: The Brunel Music Rating Inventory. Journal of sports sciences, 17(9), pp.713-724.

Katona, J., Farkas, I., Ujbanyi, T., Dukan, P. and Kovari, A., 2014, January. Evaluation of the NeuroSky MindFlex EEG headset brain waves data. InApplied Machine Intelligence and Informatics (SAMI), 2014 IEEE 12th International Symposium on (pp. 91-94). IEEE.

Katona, Jozsef, et al. "Evaluation of the NeuroSky MindFlex EEG headset brain waves data." Applied Machine Intelligence and Informatics (SAMI), 2014 IEEE 12th International Symposium on. IEEE, 2014.

Larsen, E.A., 2011. Classification of EEG Signals in a Brain-Computer Interface System.

27

Lewis, J.R. and Sauro, J., 2009. The factor structure of the system usability scale. In Human centered design (pp. 94-103). Springer Berlin Heidelberg.

Liljedahl, M., 2011. Sound for fantasy and freedom. Game Sound Technology and Player Interaction: Concepts and Developments, pp.22-44.

McAlpine, K.B., Bett, M. and Scanlan, J., 2009, February. Approaches to creating real-time adaptive music in interactive entertainment: A musical perspective. In Audio Engineering Society Conference: 35th International Conference: Audio for Games. Audio Engineering Society.

Miranda, E. and Brouse, A., 2005, May. Toward direct brain-computer musical interfaces.

In Proceedings of the 2005 conference on New interfaces for musical expression (pp. 216-219).

National University of Singapore.

Miranda, E., Brouse, A., Boskamp, B. and Mullaney, H., 2005. Plymouth brain-computer music interface project: Intelligent assistive technology for music-making. In International Computer Music Conference.

Miranda, E., Brouse, A., Boskamp, B. and Mullaney, H., 2005. Plymouth brain-computer music interface project: Intelligent assistive technology for music-making. In International Computer Music Conference.

Miranda, E.R., 2006. Brain-computer music interface for generative music. In Proceeds from the 6th International Conference on Disability, Virtual Reality and Associated Technologies, Esbjerg, Denmark.

Miranda, E.R., Roberts, S. and Stokes, M., 2004. On generating EEG for controlling musical systems. Biomedizinische Technik, 49(1), pp.75-76.

Mirza-Babaei, Pejman, et al. "Understanding the contribution of biometrics to games user research."

Proc. DiGRA. 2011.

Mouli, S., Palaniappan, R. and Sillitoe, I.P., ARDUINO BASED CONFIGURABLE LED STIMULUS DESIGN FOR MULTI-FREQUENCY SSVEP-BCI. Näätänen, R., 1990. The role of attention in auditory information processing as revealed by eventrelated potentials and other brain measures of cognitive function. Behavioral and Brain Sciences, 13(02), pp.201-233.

Nacke, L.E. and Grimshaw, M., 2010. Player-game interaction through affective sound. Game sound technology and player interaction: Concepts and developments, pp.264-285.

Nacke, Lennart E., Mark N. Grimshaw, and Craig A. Lindley. "More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game." Interacting with Computers 22.5 (2010): 336-343.

Noble, Ralph, et al. "Conditions of Engagement in Game Simulation: Contexts of Gender, Culture and Age." DIGRA Conf.. 2003.

Nunez, D. and Blake, E., 2006. Learning, experience, and cognitive factors in the presence experiences of gamers: An exploratory relational study.Presence: Teleoperators and Virtual Environments, 15(4), pp.373-380.

Otten, L.J. and Rugg, M.D., 2005. Interpreting event-related brain potentials.Event-related potentials: A methods handbook, pp.3-16.

Pates, J., Karageorghis, C.I., Fryer, R. and Maynard, I., 2003. Effects of asynchronous music on flow states and shooting performance among netball players. Psychology of Sport and Exercise, 4(4), pp.415-427.

28

Peretz, I. and Zatorre, R.J. eds., 2003. The cognitive neuroscience of music(p. xiii452). Oxford: Oxford University Press.

Reuderink, B., 2008. Games and brain-computer interfaces: The state of the art.

Reyes, C.E., Rugayan, J.L.C., Jason, C., Rullan, G., Oppus, C.M. and Tangonan, G.L., 2012, November. A study on ocular and facial muscle artifacts in EEG signals for BCI applications.

In TENCON 2012-2012 IEEE Region 10 Conference (pp. 1-6). IEEE. Riva, G., Davide, F. and IJsselsteijn, W.A., 2003. Being there: The experience of presence in mediated environments. Being there: Concepts, effects and measurement of user presence in synthetic environments, 5.

Rumsey, F., 2009. Sound and Music for Games: Prokofiev to Pac-Man to Guitar Hero. Journal of the Audio Engineering Society, 57(6), pp.455-459.

Rumsey, F., 2010. The Art and Business of Game Audio. Journal of the Audio Engineering Society, 58(4), pp.332-335.

Rumsey, F., 2011. Audio for Games. Journal of the Audio Engineering Society, 59(5), pp.341-345.

Rumsey, F., 2013. Game Audio: Transforming Simulation and Interactivity.Journal of the Audio Engineering Society, 61(7/8), pp.609-612.

Rumsey, Francis. "Game Audio: Generative Music, Emotions, and Realism." Journal of the Audio Engineering Society 63.4 (2015): 293-297.

Shiffrin, R.M. and Czerwinski, M.P., 1988. A model of automatic attention attraction when mapping is partially consistent. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(3), p.562.

Skalski, P. and Whitbred, R., 2010. Image versus Sound: A Comparison of Formal Feature Effects on Presence and Video Game Enjoyment.PsychNology Journal, 8(1), pp.67-84.

Stevens, R., Raybould, D. and McDermott, D., 2015, February. Extreme Ninjas Use Windows, Not Doors: Addressing Video Game Fidelity through Ludo-Narrative Music in the Stealth Genre. In Audio Engineering Society Conference: 56th International Conference: Audio for Games. Audio Engineering Society.

Stigsby, B., Rodenberg, J.C. and Moth, H.B., 1981. Electroencephalographic findings during mantra mediation (transcendental meditation). A controlled, quantitative study of experienced meditators. Electroencephalography and Clinical Neurophysiology, 51(4), pp.434-442.

Styles, E.A., 2005. Attention, perception and memory: An integrated introduction. Psychology Press.

Summers, T., 2011. Playing the tune: Video game music, gamers, and genre. Act-Zeitschrift fĂźr Musik & Performance, 2011(2), pp.2-27.

Sweetser, P. and Wyeth, P., 2005. GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment (CIE), 3(3), pp.3-3.

Vidal, J.J., 1973. Toward direct brain-computer communication. Annual review of Biophysics and Bioengineering, 2(1), pp.157-180.

Walker, J.L., 1977. Subjective reactions to music and brainwave rhythms.Physiological Psychology, 5(4), pp.483-489.

Webster, J., Trevino, L.K. and Ryan, L., 1994. The dimensionality and correlates of flow in humancomputer interactions. Computers in human behavior, 9(4), pp.411-426.

Webster, Jane, Linda Klebe Trevino, and Lisa Ryan. "The dimensionality and correlates of flow in human-computer interactions." Computers in human behavior 9.4 (1994): 411-426.

29

Whalen, Z., 2004. Play along-an approach to videogame music. Game studies, 4(1).

Wolpaw, J.R., 2010. Brain–computer interface research comes of age: traditional assumptions meet emerging realities. Journal of motor behavior,42(6), pp.351-353.

Woodman, G.F., 2010. A brief introduction to the use of event-related potentials in studies of perception and attention. Attention, Perception, & Psychophysics, 72(8), pp.2031-2046.

Wooller, R., Brown, A. R., Miranda, E., Berry, R. & Diederich, J. (2005). A framework for comparison of processes in algorithmic music systems. In Generative arts practice (pp. 109–124). Sydney: Creativity and Cognition Studios Press.

Wu, B., Horner, A. and Lee, C., 2014. The Correspondence of Music Emotion and Timbre in Sustained Musical Instrument Sounds. Journal of the Audio Engineering Society, 62(10), pp.663-675.

Appendix 1 Collected Test Results

30

31

Appendix 2 Individual Digitized Test Results with Participants Statements

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

Appendix 3 Participant Consent Form

71

Appendix 4 System Usability Testing Questionnaire

72

Appendix 5 AGRO Max MSP Patch Presentation wise the patches are still fairly messy, this will be addressed before the final presentation.

73

Appendix 6 MESS Max MSP Patch

74

Appendix 7 Additional Resources For copies of the Max MSP patches, Arduino C++ Sketches and copies of test results see the pen drive submitted with this dissertation.

75