Warren_Skipping Pong_Proofs copy

Int. J. Web Based Communities, Vol. X, No. Y, xxxx

Skipping Pong and moving straight to World of Warcraft: the challenge of research with complex games used for learning Scott J. Warren*, Greg Jones and Amy Trombley Department of Learning Technologies, University of North Texas, UNT Discovery Park, 3940 North Elm Street, G150, Denton, TX 76207-7102, USA E-mail: scott.warren@unt.edu E-mail: greg.jones@unt.edu E-mail: atrombley@unt.edu *Corresponding author Abstract: In 1972, Atari released a simple digital game in North America entitled Pong, the game simulated table tennis using two lines as paddles with a square to represent the ball. Video games have since expanded in complexity and player engagement. As a result of this engagement, educational models have been proposed that leverage video games. However, rather than begin with research on games as simple as Pong, many have focused on complex systems based on massively multiplayer online games. This article reviews approaches to designing or using video games for learning and identifies challenges to designing, using and researching games.

Keywords: games; learning; instruction; design; challenges; models of design; research; reporting; curriculum development.

Reference to this paper should be made as follows: Warren, S.J., Jones, G. and Trombley, A. (xxxx) ‘Skipping Pong and moving straight to World of Warcraft: the challenge of research with complex games used for learning’, Int. J. Web Based Communities, Vol. X, No. Y, pp.000–000.

Biographical notes: Scott Warren works as an Assistant Professor of Learning Technologies at the University of North Texas. His current research examines the use of emerging online technologies such as immersive digital learning environments, educational games and simulations, and open source course management tools in complex systems in K-20 settings. Prior to working in higher education as a Professor, Researcher and Designer, he taught both social studies and English for nearly a decade. He has also worked to the Quest Atlantis project and his current work with The Door and Broken Window alternate reality courses is partly funded with grants from the University of North Texas. He completed his PhD in Instructional Systems Technology at Indiana University-Bloomington.

Greg Jones works as an Associate Professor of Learning Technologies at the University of North Texas. His primary research interest is in using technology to further the creation and distribution of knowledge and learning. His research focuses on the combination of visualisation systems, virtual communities, telementoring, games, simulations and 3D online learning environments for

Copyright © 200x Inderscience Enterprises Ltd.

1

2

S.J. Warren et al. teaching and learning. These emerging technologies support learning by distributing interaction and feedback across time and space via interactive forms of multimedia. Teaching and learning relationships are no longer confined by space, but are defined by connectivity. He completed his PhD in Instructional Technology at the University of Texas Austin.

Amy Trombley is a Professor of Communication Studies at the Central Park Campus of Collin College. Her primary research interests include online disinhibition effect, online gender roles and technology-mediated communication. She obtained her MA in Communication from the Western Michigan University and is currently pursuing her Doctoral degree in Educational Computing at the University of North Texas.

1

Introduction to the problem

In 1972, Atari, Inc. released a simple video game in North America (Kent, 2001; Wikipedia, 2009). Titled Pong, the digital game simulated two players of table tennis using two long lines on either side of the screen with a square to represent the ball (Kent, 2001). This ball moved from side to side on the screen, while the player would move their line in order to ‘hit’ the ball, back to the opposing player, which was accompanied by a sound meant to represent that of a table tennis ball being struck. If you missed the ball, the opposing player scored. This score was kept on the screen and when one player achieved a predetermined score, they won and the game would begin again.

This game went on to be one of the most successful games ever released and helped launch the video game industry.

Since that time, video games have expanded greatly in scope and complexity with millions of players (Entertainment Software Association, 2007). Advances in media and hardware have resulted in high levels of engagement between player and game system, even rising to the level of sport in which thousands and even millions of dollars are at stake in competitions (Major League Gaming, 2008). As a result of this high level of engagement, especially with younger players, several models of education have been proposed that stem from the spread of the idea that video games should be used for education over the last decade. These models have included using a

3-D web browsers to develop multi-user virtual environments (MUVE) to develop curricula an approach leveraged by River City (Dede et al., 2006) and Quest Atlantis (Barab et al., 2008; Barab et al., 2007) to create new worlds using open-source or costly game engines

b

off-the-shelf video games and simulations with the purpose of wrapping outside learning activities around them as was the approach with using Civilization III (Squire, 2006, 2004)

c

newly developed educational games created by researchers and instructional designers (Foster, 2008; Saurbaum and Brown, 2006; Walker and Shelton, 2008).

The proponents of games for learning (Jenkins et al., 2003; Prensky, 2001; Squire, 2008) leave open several questions related to the use of these systems, our ability to research them, and the validity of findings related to current and past research due to the sheer

Skipping Pong and moving straight to World of Warcraft

3

complexity of the systems and, in some instances, our lack of knowledge about how the games are structured. The more complex a system is, the more difficult it becomes to control or understand.

The purpose of this article is to explore some of the challenges to designing and researching games for as they occur in several studies, including some conducted by the authors themselves. Further, we will propose some solutions that can help overcome these problems while allowing instructional designers to develop meaningful game products that foster community and learning. Many of these come from the simple decision to return to returning to designing and researching simpler educational games that allow us to study variables in isolation and building up the complexity.

Instead, we argue that rather than skipping the systematic study of simpler, early video games such as Pong to examine individual components that may benefit learners in favour of creating and implementing complex games based on massively multiplayer online games (MMOG) like Blizzard’s World of Warcraft (LeBlanc, 2008), we should return to simpler studies with discrete variables. These should be variables that can be readily tested and the outcomes can be clearly connected to a specific game stimulus rather than a large number of media enmeshed together in an impenetrable mess of confounding social, psychological and educational factors. Educators walk a dangerous ethical line when they implement games without really understanding how they function as psychological and learning tools. Evaluating what the possible ramifications of implementing them may be for instruction, learning, and the emotional well being of participants is essential, yet these outcomes associated with MMOGs are increasingly in question (Somaiya, 2009). Further, in order to do this, we argue that there needs to be a much stronger connection among researchers as a community in order to answer the basic questions that have thus far been neglected in the literature. However, there are some core challenges to either researching games for learning or building this community of researchers. Thus, we first address some of the main questions left unaddressed in the learning game literature. Just as importantly, we will note challenges to building a community of researchers to answer them as well as some suggestions for reaching the goal of building a community of researchers seeking to answer these neglected questions.

2

Challenges to games as instructional design models

2.1 What is a game?

Every instructional design model has its own strengths and weaknesses as a process for creating a set of instruction. However, games, for which we take a definition from Salen and Zimmerman (2004) are “a system in which players engage in an artificial conflict that is defined by rules and it results in a quantifiable outcome (italics added by author) (p.80).� A fourth component that these authors claim is optional but necessary with simulation games is the modelling reality characteristic. When working with educational games, this component is rarely optional as, for the practices of game play to have a meaningful relationship to the reality in which learners are to apply the acquired or constructed knowledge and skills taken from the game, the systems should in some way model that real system (Horn, 1977; Mitchell and Savill-Smith, 2004).

4

S.J. Warren et al.

Knowing what a game is, does not, however, explain its relationship to learning and instruction. Thus, from this definition of game, we synthesise an emergent definition of learning in the context of game as: “(a)n event that takes place in an intentionally designed contextual system that models reality and is defined by sets of rules in which learners are challenged by artificial cognitive conflicts set by the system or instructor in order to reach quantifiable educational outcomes.” Adapted from Warren and Dondlinger (2008, p.2)

Leveraging these two simple definitions, we have created several complex spaces for learning and instruction with varying degrees of success for learning (Barab et al., 2007; Jones and Warren, 2008; Warren et al., 2008; Warren and Dondlinger, 2008). However, they each also come with their own intrinsic affordances and challenges as identified by several authors.

2.2 Methods Using multiple research databases, the authors employed Boolean search including the terms ‘computer games’ AND ‘learning’ AND ‘education’ AND ‘research’, which yielded articles, books, book chapters, conference proceedings and dissertation studies from the each of the following databases: x

Digital Dissertations Education Research Complete

x

Directory of Open Access Journals

x

ERIC via FirstSearch

x

JSTOR

x

PsycArticles via EbscoHost

x

Sage Journals Online

x

ScienceDirect

x

Taylor and Francis via InformaWorld

x

Web of Science.

The results of this search were limited to those that resulted from studies that were then written as peer-reviewed publications and dissertation studies. Three of the databases yielded no results. We excluded those publications that were concerned only with pure simulations that failed to meet the definition of game provided by Salen and Zimmerman (2004) above, as they would fail to meet these criteria. Further, only those publications that reported research data were used, which eliminated articles that reported conjectural analysis, policy suggestions, and theoretical frameworks that were not directly supported by data.

Most important in this analysis were limitations to the research reports noted by the authors, specific flaws in the research methods (i.e., data collection, sample size) and/or analytical techniques employed, and threats to the validity of the studies. The goal of this analysis is to locate the most important challenges to employing social sciences research methods to study video games used for learning. As such, the resulting set of articles

Skipping Pong and moving straight to World of Warcraft

5

focused on games included 132 articles or dissertations items spanning the last 15 years, the majority being published in the last five. These results were then narrowed to 47 studies by examining the descriptions of the games to determine whether the study was of a ‘game’ as we have defined it or it the environment in which the study took place was instead another form of digital space such as a pure simulation or a 3-D world like Second Life or Quest Atlantis (QA), which we would classify as MUVEs because they lack designed artificial conflict. However, these environments were used as engines to create sub-spaces such as the Taiga world in QA that do qualify as games; thus, studies related to Taiga, Anytown, and similar game spaces were included.

3

Findings

3.1 Problem one: inherent flaws in the game systems As noted earlier, there are three main approaches to instructional designs that leverage games. Each has its own unique challenges in terms of creating designs that promote the affordances (Gibson, 1977) of ‘game’ and benefits to instructional designers and researchers. However, each also has a set of benefits that make one choice better than another while concurrently creating a new set of problems. This first problem is the genesis from which emerge other problems inherent to educational research that seeks to answer basic questions about using games for learning.

3.1.1 Off-the-shelf game products An off-the-shelf game product is one that anyone might purchase at a large retail chain store including games like World of Warcraft, The Sims, or Spore. These games have proven to be excellent sellers and are very popular among consumers with World of Warcraft boasting more than 11 million players who pay a monthly fee to access the MMOG world of Azeroth (Wikipedia, 2009). These games appear to be very motivating to players with some even resulting in addictive play resulting in negative consequences to themselves and others (MacMillan, 2006; Tanner, 2007).

Yet it is important to remember that the majority of these off-the-shelf games were not designed for educational use and have no data to support their use for learning. For example, ‘real-time simulations’ (RTS) games purchased at physical or online retailers like those in the Civilization, SimCity, and Flight Simulator series were not designed to ensure that players learn accurate depictions of the ancient or modern world; instead, they are more often designed to be replayed multiple times with differing results as entertainment. This makes it very difficult to convey accurate depictions of history, science or mathematics concepts for students; thus, the focus of learning designs is not on acquiring or constructing knowledge of content. Instead, the focus of supplementary instructional designs has been to support the use of off-the-shelf games to force them into the service of learning. Further, accompanying research has been is on the use of games has been on examining the game as a tool for mediating student understandings, social interaction, and engagement with system rather than on learning outcomes (Squire, 2004; Squire et al., 2005). Reviews of other instances in which such products were used were conducted by Hays (2005) and separately, Dondlinger (2007), who found infrequent statistically significant gains on learning outcomes when these products were used.

6

S.J. Warren et al.

Another example comes from Foti and Hannafin (2008) who conducted a study on curriculum that employed an off-the-shelf multiplayer online game called Club Penguin that was combined with multimedia that they had designed to support foreign language learning (FLL). The study examine whether the combination generated learning [t (1, 24) = 0.34, p = .37], confidence in language use [t (1, 24) = 0.94, p = .18], and whether or not students liked it [t (1, 24) = 0.12, p = .45]; however, none of the results reached significant differences. At the conclusion of their study, the authors noted that: “Unfortunately, we could not find an avatar-based online game that served our purpose for this learning unit. Club Penguin is not designed for foreign language learning. It did not, for example, provide guidance to help learners complete the task; nor were its avatar attributes ideal for the desired tasks to be completed.” (p.43)

We suggest that it would be better if off-the-shelf games used in education were designed specifically for education and that the underlying learning psychological structures, learning goals and specific objectives were revealed to both researchers and teachers.

Further, when designing simulation games, the accuracy of the presented ‘reality’ is paramount and students must be engaged with the computer system so that there are predictable behaviours and characteristics that the non-player characters (NPCs) exhibit.

For example, if the system in a civilization simulation acts randomly by assigning different characteristics to the behaviours of different NPCs (e.g., the Romans are peaceful in one simulation and aggressive in another), learner players are unable to engage in prediction and strategy as critical thinking by using their knowledge constructed from past game experience as a means of successfully constructing understanding of the world and apply them to alternative contexts. This becomes especially problematic when the authors played the game using the real ‘earth’ simulation. The Iroquois nation attacked our nation, the Aztecs, mercilessly, behaving in ways that are not in keeping with their behaviours that we understood from multiple historical texts about their particular goals, military styles, and other recorded behaviours.

In one instance, they built an empire across the USA and then developed ships that allowed them to colonise parts of Europe, Africa and Asia. Further, we, playing as Aztecs, were allowed to pursue a peaceful, diplomatic agenda, which was also out of keeping with historical records and, further, the two groups were not active concurrently with one another.

While this would not be the same problem if these changes were presented in a clearly depicted fantasy context, it becomes a difficult problem with young learners who may develop incorrect historical mental models. Squire (2004) notes that “simulation games can potentially distort the phenomena they are meant to model...this problem of simplification/distortion of reality in games is exacerbated by the fact that edutainment products first and then appropriated for use in classrooms second.” (p.11)

When placed in the context of the real world classroom and testing situations, the learners misunderstanding of real historical figures could easily problematic for students. If the students learn from the simulation game that the Iroquois nation were aggressive and warlike, they may misattribute the underlying causes of the French and Indian War (as the conflict is known in the USA) even when presenting with contrasting views from other sources such as textbooks or lectures. This may also lead to generalisations from

Skipping Pong and moving straight to World of Warcraft

7

their experiences with a single Native American tribe or group of tribes to believe that all Native American tribes were aggressive and warlike.

If the simulated part of the game presents an unrealistic and inaccurate model of social studies concepts, then the assumption presented that a learning simulation game presents a useful modelled reality is violated and, in fact, instead may lead to learning that is potentially harmful to the learner (Cubitt, 2001). Further, it slows the instructional process, because the teacher is now required to re-teach concepts that may have become entrenched facts in the minds of the learners, making the embedded cognitive constructs harder to change (Driscoll, 2000).

Underlying this problem from an instructional designer’s view is the lack of reporting in the related publications as to the underlying design of the game or supporting curriculum. In the publications we reviewed, only the dissertations spent more than three pages reporting the design of the complex game environment and some spent one page or less describing the learning tool. However, in our review, we found three articles that appear to be computer science and engineering publications that reported only the design of the game. While research reports in journals are important, many more expansive reports of the design of curricula supporting off-the-shelf and designed games are necessary. In some instances, this is occurring in advance of any research (Dickey, 2007, 2006); however, it is necessary for research to follow as well.

3.1.2 MUVEs as game tools MUVEs are not games. Instead, they are systems for building digital spaces such as buildings, avatars and other objects with which users can interact as well as communicate with each other to help generate experience. In addition, the open-access MUVE Second Life has gained in popularity and has been increasingly the focus of research and are being employed at universities to support instruction and learning management systems (De Lucia et al., 2009) as a supplementary communication and presentation tool with some positive survey results at the post-secondary level related to increased student satisfaction, increased their engagement and overall enjoyment in a course (Jarmona et al., 2008).

Other MUVE engines like Active Worlds have been used to develop instructional and learning spaces such as River City (Dede et al., 2006), Quest Atlantis (Barab et al., 2007) and have been increasingly researched in the last few years with a focus on science learning through inquiry into large, intractable problems using specific locales that mirror real-world situations. For example, one world in Quest Atlantis called Taiga (Barab et al., 2007) presents students with a small-scale state or national park in which the local fish population is dying off rapidly near where three different groups are working as fishermen and loggers. Students are challenged to develop a solution to this problem by first engaging in scientific inquiry related to possible pollutants in the water, temperature problems, and to gather other observable facts through interviews with stakeholders and examination of the simulated world. Similarly, River City asks students to locate the problem of a diseased well in a historical simulation focused on a city in the USA in the 18th century, modelled after a real outbreak of disease that sickened and killed several residents. By engaging in the use of the scientific method in order to locate the source of the problem, students are able to generate a simple solution that results in the highest possible benefit to the local populace.

8

S.J. Warren et al.

These MUVEs do share some aspects with games because, as noted in the earlier definition, they include systems, players, artificial conflict, rules and a quantifiable outcome. However, the conflict and accompanying rule set do not include the time component (e.g., you must complete the task in three minutes or you fail) that many video games include or the possibility of dire consequences (e.g., virtual death of their avatar) and therefore players may not recognise them as games. Instead, the simulation and instructional components come across so strongly that they more often appear to be more akin to Microsoft’s Flight Simulator series than what players of World of Warcraft would view as a game. More recent attempts by Quest Atlantis with worlds named Anytown and Taiga have sought to more clearly integrate game elements, though the very nature of a MUVE makes it difficult to design spaces that learners will intuitively recognise as games, regardless of their complexity (Warren and Dondlinger M., 2008; Warren et al., 2009a).

3.2 Problem two: confounding variables and the challenge of isolating multimedia, cognition and game factors Most, if not all, video games or game-like structures have a large number of variables that come from their nature as multimedia applications. Most of these are difficult to control for in experimental or quasi-experimental studies related to games and simulations because they tend to be conducted within intact classrooms in order to attempt to add authenticity to use. This stems from the fact that the average digital game includes many media affordances: video, audio and interaction between player and system (or multiple players).

With each of these often occurring simultaneously, how does a researcher discriminate which affordance is responsible for learning either at a given time during the implementation or at the conclusion of the study? If we study a learning game implementation holistically by comparing pre and post-test scores as most of the studies that employed quantitative quasi-experimental methods did, it is only possible to know that something that occurred during that time caused a change in the player, which resulted in an increased score. However, if the implementation lasted more than a week, which the majority of the studies did, how then do we control for each multimedia factor present in the game? A number of other questions also must be addressed: x

How do we know which was most effective, so that we can focus our future designs on improving that media and/or learning affordance and thus learning?

x

How do we know if it is a combination of factors? Does the game only work when all the multimedia factors are present in the game?

x

Are there cognitive factors present in some games that are lacking in others that must be present for high levels of engagement to exist?

The short answer is: we do not know. In order for us to understand what specifically exists in highly effective video games used for learning that can be used to help instructional designers and researchers create or use games that result increased learning, engagement, or other affective connection that supports learners. Some of the research by Tuzun (Tuzun, 2004; Tuzun et al., 2009) and Schneider et al. (2005) begin to examine issues of motivation in these games, but lack of discrimination among the multimedia

Skipping Pong and moving straight to World of Warcraft

9

variables leaves the reader without an understanding of which has the highest impact on learning.

3.3 Problem three: limited samples and generalisability As researchers, we have ourselves been guilty of conducting research on limited samples (Barab et al., 2007; Warren et al., 2008) due to the convenience of a group of students to our university, lack of access larger populations, and the challenge of recruiting an adequate number of participants due to our location in an isolated geographical region.

The majority of the studies we examined had sample sizes with fewer than 50 students making the power of the findings fairly weak at less than three. In addition, the samples tended to be drawn not from students who would be considered to be of median achievement; instead, many were gifted and talented (GT) or special education (Barab et al., 2005; Barab et al., 2007). By treating these groups as pilot samples and then expanding our studies to larger groups that more accurately represent the larger population, we can increase the validity and generalisability of findings.

3.4 Problem four: contextual problems in research reporting When we report research, how much are we required to reveal about the reality of the study? If we report only what is suggested in the average research textbook, we often only tell a small part of the story and leave gaping holes in the knowledge of the reader that lead to serious misunderstandings about the success of a designed implementation and what the statistical significance may have truly been influenced by. Was it the game?

Was it the teacher? How much time did students actually spend using the game?

How much teaching and re-teaching did the classroom teacher have to do in addition to students using the game? Without this information in the research report or article, it is difficult for readers to determine whether it was the educational game or direct instruction that resulted in statistically significant differences.

Further, due to the fact that many of these approaches to using games are very new to teachers and students there are often problems. As reported in the by Warren et al.

(2009a), there were major difficulties in the implementation of the Anytown literacy environment stemming from miscommunication between researcher and instructor, issues of power between the instructional environment and instructor, and systemic constraints at the school where the study was conducted. However, many studies fail to report this kind of information, possibly because of fear that a peer reviewer may interpret such reported data as ‘failure’ and result in lack of publication. As researchers ourselves, this definitely crossed our minds as we submitted the article. However, this form of information is invaluable to researchers and instructional designers seeking to develop their own educational games. If we withhold such valuable insights from our peers, we are doomed to continuously make the same mistakes, which slow the progress of the field and generation of useful knowledge.

4

Discussion

While this sort of development sounds promising, there are several difficulties with the approaches related to conducting research, interpreting findings, and designing game

10

S.J. Warren et al.

systems. Each of the identified problems noted here relate to recommendations that Ted Castranova (2001), creator of Arden: The World of William Shakespeare, contributor to Wired magazine, and well-known writer on video games for learning. After failing to find success with Arden, he concluded that: “It’s no fun…we failed to design a gripping experience” (Baker, 2008). Beyond the fun factor, we have identified several challenges to researching learning games.

Some of the challenges to conducting effective research on learning games include: 1

confounding variables that limit the findings

2

limited samples used for research

3

inherent flaws in the game systems related to learning

4

combinatory multimedia factors that resist isolation

5

contextual problems in research reporting.

Many of these problems and threats to the validity of accompanying research stem from the sheer complexity of the game environments, game engines, and instructional designs we have presented to learners that have been rushed into production because the funding was available or the innovative edge had to be pushed for a publications without really understanding the game systems and platforms through purposeful, systematic research.

Others simply arise from a lack of suitable participants for the research. Finally, others stem from a lack of time; the time to adequately test the designs, time to recruit subjects, and time to study the systems themselves to ensure their appropriateness to the learning goals.

4.1 Addressing the problems To begin, we need to return to analysing not the most complex game systems by breaking them down into their components as has been done in some of the studies here, but instead by starting with games that start with simpler components and build up by adding complexities of interaction, audio, video, and play one at a time. In Wired Magazine (Baker, 2008), Castranova identified “5 Tips for Making Games That Don’t Suck”.

1

do not be overly ambitious

2

go low tech

3

think about your audience

4

get a full-time staff

5

concede screwups (sic).

We have developed complimentary suggestions for addressing the research problems identified above.

1

Keep your research questions narrow, build over time, and start with usability testing.

It is easy to fall into the trap of wanting to answer the as many and the questions that are most important to a researcher first after spending hundreds or thousands of hours and (possibly dollars) designing and developing a game. However, none of the

Skipping Pong and moving straight to World of Warcraft

11

articles or chapters we reviewed here reported beginning with simple usability or play tests to determine whether or not the product met the needs of the learners and instructors or were cleared of glitches and bugs before widespread implementation.

Further, most of the research focused not on learning variables such as whether the amount of content was retained by the learners, but instead on affective variables that may be linked to learning. Even those that did report gains such as the studies conducted related to science (Barab et al., 2007; Dede et al., 2006) and language learning (Warren et al., 2008), it is impossible to distinguish which media or game affordances, or possible non-game teacher instruction may have been responsible for statistically significant learning gains.

2

Stick with simple games to keep variables low and build complexity As the authors note elsewhere and Castranova (Baker, 2008) does above, games do not have to be high technology and on the order of World of Warcraft in order to be useful for learning game research. In fact, high complexity games make it much more difficult to isolate the multimedia, game and learning variables to determine which is responsible for learning at a given time. By starting simple with a game like Pong, we can perform cognitive experiments that allow us to determine whether the game really requires the ding sound that occurs when the simulated ball strikes the simulated racquet for a player to understand the physics of tennis.

3

Spend a lot of time analysing your game choices for flaws and appropriateness of the subject to your audience.

Good instructional design begins with needs analysis; however, the majority of the articles and chapters we reviewed here fail to report that analysis and it is unclear whether it was conducted. In addition, there was extremely minimal reporting of what the game was in most articles. There are a number of questions that need to be answered prior to doing research.

4

x

What was the overall goal of the game?

x

What are the rules of the game?

x

What does it do that might allow for learning and how does it do it?

x

What did it look like and does that influence the learner?

x

Does the game leverage any psychological principles that you can identify easily and study such as intermittent reinforcement?

Choose or design games (with your users’ input) where individual media affordances (i.e., audio, video) can be isolated and studied independently.

This addresses the core suggestion we have here: choose simple games to study. By beginning with a simple game with only textual video components like Infocom’s Zork (Anderson, 1985), a completely text-based adventure game, it is much more valid to make claims about what in the game is responsible for the learner and less difficult to control the learning components. Once this is done, additional media components can be added such as graphical components by choosing a later game in the series like Zork Zero (Wikipedia, 2009), which adds a simple map interface that allows players to navigate. By taking this approach, researchers can more easily isolate individual factors and variables to study rather than being forced, to a large

12

S.J. Warren et al. extent, to examine the impacts of a complex game holistically, which increases the threats to the validity of many findings.

5

Report the instructional design elements and play elements of the games in detail; explain what did not work as well as what did.

With systems that are as complex as video games, it is even more important for instructional designers and researchers in the broader community to understand what the actual design is. Based on our reviews of many research reports, descriptions of the instructional design process for the learning game and the game products themselves are nearly non-existent, normally less than two pages of a 30-page article. While the report of research findings is very important, the sheer complexity of these systems warrants that journals be aware of the importance of giving equal time to reporting the instructional designs so that, if they are reported as successful, the designs can be used to guide future development. There is a need to allow other researchers to conduct replication studies to ensure that other researchers can vet these games or to further the research by implementing the games with different populations and under different circumstances to test their applicability in other settings. While a few of the researchers we reviewed have done, it is still not the norm. Increasing the value of these publications for tenure and promotion as well as on the editorial boards of journals will go a long way towards improving this situation and increasing the validity of the use of games for learning.

While these recommendations should be at the core of future research regarding the use of games for learning, building a community of researchers who are willing to use the recommendations to revive, renegotiate and revise existing research is fundamental to the overhaul. Too often academic writing is perceived as the culmination of the research process (Cutherbert and Spark, 2008), yet “there is value in an identity that is provided by the writing process not just the published product” [Murray and Newton, (2009), p.549].

This identity evolves from the exchange of ideas, knowledge, perspectives and understanding through the collaboration of inter-disciplinary researchers and builds a sense of community based on mutual negotiation and even competition to improve the body of research (Murray and Newton).

Building a new research community to start fixing the problems discussed above must start with the networking of games for education researchers. According to Shaffer and Anundsen (1993), there are six simple steps that anyone can take to build a community.

First, one must evaluate potential resources gathered from both past and present social connections. Researchers interested in games for education have gathered a lifetime of connections made through colleagues, conference participation, and personal interests.

Second, to avoid “false starts and disappointments” (p.39), Shaffer and Anundsen suggest determining the sort and scope of the community. Clearly identifying both the expected inputs and outputs of the group, in addition to desired attitudes and perceived needs, will improve the potential for group success. Third, a community starts with interpersonal relationships between individuals who bring their own strengths and weaknesses to the table. While building a community, members often need to reassess and develop personal attributes and skills necessary for effective participation in the new community. For researchers, this might mean letting go of old (possibly bad) habits and being open to new ways of thinking and thus new methods of research. Fourth, members of a new community often generate fears associated with “giving up too much time, individuality,

Skipping Pong and moving straight to World of Warcraft

13

or independence”, (Shaffer and Anundsen, p.42) therefore developing strategies for dealing with internal (and possible external) resistance is important to community development. The fifth step suggested for building a new community involves making personal contact with individuals identified in step one with whom easy connections can be made. When researchers attend conferences, such as those sponsored by the Association of for the Advancement of Computing in Education (AACE) and the American Educational Research Association (AERA), opportunities to verbalise the need for changes in research and seek support from colleagues are abundant. Last, it is important to remember that building a new community is a complex and multifaceted process. A community is built through the effort of many people, so it is important to allow enough time for the community to gel. If the community is not an instant success, re-evaluate the previous steps and recognise that “what you get may not last in exactly the form you had in mind” (Shaffer and Anundsen, p.46).

5

Conclusions

In conclusion, while we support the idea that video games may have promise in terms of engaging learners, and there is some research support for this, there is a broad array of concerns that we as researchers must address in a more systematic fashion. By doing so, we can address the concerns of peer researchers, parents and even students who may be resistant to using games for learning. However, before we can address these concerns, we must step back from the rush towards employing complex environments and examine simpler games like Pong or even more advanced games like Donkey Kong to learn what they can provide in terms of supporting learning before we add the chaos of factors that emerge from the use of a massively multiplayer game like World of Warcraft. Further, stepping back will help us to understand that these games do not exist in a vacuum and there are other components outside of them that need to be studied in depth whether they be the school system itself and its preparedness to use these games or the classroom teacher who is just trying to get her students to pass a standardised test. As Reiber and Noah (2008) note at the conclusion of their study with a simple simulation game: “An obvious conclusion is that teachers should not simply introduce “edutainment” software into the without careful consideration and planning as to how to make these highly interactive (and enjoyable) experiences relevant and meaningful from the educational point of view. Rather than replacing the teacher, this software elevates the role of informed teachers and emphasizes the importance of their contribution and influence.” (p.90)

By taking the focus off of what the game can do to support learning by itself, we can discover instead how it interacts with school system, teacher, student and curriculum to support learning. Instead of skipping the simpler Pong phase of our research in the field in a drive to reach the exciting Warcraft graphics and epic battles, we are prone to miss a lot and may potentially miss much of the value that video games have to offer the field of education.

References Anderson, T. (1985) ‘The history of Zork’, New Zork Times, Winter–Summer 1985.

14

S.J. Warren et al.

Baker, C. (2008) ‘Trying to design a truly entertaining game can defeat even a certified genius’, Wired, 24 March 2008, p.16.

Barab, S., Warren, S. and Ingram-Goble, A. (2008) ‘Conceptual play spaces’, in Fertig, R. (Ed.): Handbook of Research on Effective Electronic Gaming in Education, Idea Group Reference: IGI Global, Hershey, PA. Barab, S.A., Thomas, M., Dodge, T., Carteaux, R. and Tuzun, H. (2005) ‘Making learning fun: Quest Atlantis, a game without guns’, Educational Technology Research & Development, Vol. 53, No. 1, pp.86–107.

Barab, S.A., Zuiker, S., Warren, S., Hickey, D., Ingram-Goble, A., Kwon, E-J., et al. (2007) ‘Situationally embodied curriculum: relating formalisms and contexts’, Science Education, Vol. 91, No. 5, pp.750–782.

Castranova, E. (2001) ‘Virtual worlds: a first-hand account of market and society on the cyberian frontier’, In The Gruter Institute working papers on law, economics, and evolutionary biology, Vol. 2, The Gruter Institute.

Cubitt, S. (2001) Simulation and Social Theory, 1st ed., Sage Publications, London.

Cuthbert, D. and Spark, C. (2008) ‘Getting a GRIP: examining the outcomes of a pilot program to support graduate research students in writing for publication’, Studies in Higher Education, Vol. 33, No. 1, pp.77–88 De Lucia, A., Francese, R., Passero, I. and Tortora, G. (2009) ‘Development and evaluation of a virtual campus on second life: the case of SecondDMI’, Computers & Education, No. 52, pp.220–233.

Dede, C., Ketelhut, D. and Ruess, K. (2006) ‘Designing for motivation and usability in a museum-based multi-user virtual environment’, available at http://www.gse.harvard.edu/~dedech/muvees/documents/AELppr.pdf (accessed on 11 March 2006).

Dickey, M.D. (2006) ‘Game design narrative for learning: appropriating adventure game design narrative devices and techniquest for the design of interactive learning environments’, Educational Technology, Research and Development, Vol. 54, No. 3, pp.245–263.

Dickey, M.D. (2007) ‘Game design and learning: a conjectural analysis of how massively multiple online role-playing games (MMORPGs) foster intrinsic motivation’, Educational Technology Research & Development, Vol. 55, No. 3, pp.253–273.

Dondlinger, M.J. (2007) ‘Educational video game design: a review of the literature (electronic version)’, Journal of Applied Educational Technology, Vol. 4, pp.1–11, available at http://www.eduquery.com/jaet/index.htm (accessed on 19 July 2007).

Driscoll, M.P. (2000) ‘Psychology of learning for instruction’, Needham Heights, Allyn & Bacon, Massachussetts.

Entertainment Software Association (2007) ‘Facts and research’, available at http://theesa.com/facts/gamer_data.php (accessed on 7 January 2008).

Foster, A. (2008) ‘Games and motivation to learn science: personal identity, applicability, relevance and meaningfulness’, Journal of Interactive Learning Research, Vol. 12, No. 4, pp.597–614.

Foti, L. and Hannafin, R. (2008) ‘Using back-story in multimedia and avatar-based games to engage foreign language learners: a pilot study’, International Journal of Emerging Technologies in Learning, Vol. 3, No. 3, pp.40–44.

Gibson, J.J. (1977) ‘The theory of affordances’, in Shaw, R. and Bransford, J. (Eds.): Perceiving, Acting, and Knowing: Toward on Ecological Psychology, pp.67–82, Erlbaum and Associates, Hillsdale, NJ. Hays, R.T. (2005) ‘The effectiveness of instructional games: a literature review and discussion’, Technical Report No. 2005-004, Naval Air Warfare Center Training Systems Division, Orlando, FL. Horn, R. (Ed.) (1977) ‘The guide to simulations/games for education and training’, Vol. 1, Didactic Systems, Crawford, New Jersey.

Skipping Pong and moving straight to World of Warcraft

15

Jarmona, L., Traphaganb, T. and Mayrath, M. (2008) ‘Understanding project-based learning in second life with a pedagogy, training, and assessment trio’, Educational Media International, Vol. 45, No. 3, pp.157–176.

Jenkins, H., Squire, K. and Tan, P. (2003) ‘Entering the education arcade’, ACM Computers in Entertainment, Vol. 1, No. 1, pp.1–11.

Jones, J.G. and Warren, S.J. (2008) ‘Leveraging intelligent agents and directed narratives in online learning environments’, available at http://www.innovateonline.info/index.php?view=article&id=576 (accessed on 2 December 2008).

Kent, S.L. (2001) The Ultimate History of Video Games: From Pong to Pokemon: The Story Behind the Craze That Touched Our Lives and Changed the World, 1st ed., Three Rivers Press, Roseville, CA. LeBlanc, S. (2008) ‘Can ‘World of Warcraft’ make you smarter?’, Technology & Science Online, , 18 August 2008, available at http://www.msnbc.msn.com/id/26271240/ (accessed on 7 July 2009).

MacMillan, R. (2006) ‘Study: some internet addicts cover up habit (electronic version)’, MSNBC/Reuters, available at http://www.msnbc.msn.com/id/15309705/ (15 November 2007).

Major League Gaming (2008) available at http://gamebattles.com (accessed on 28 December 2008).

Mitchell, A. and Savill-Smith, C. (2004) ‘The use of computer and video games for learning: a review of the literature’, Learning and Skills Development Agency, London.

Murray, R. and Newton, M. (2009) ‘Writing retreat as structured intervention: margin or mainstream?’, Higher Education Research & Development, Vol. 28, No. 5, pp.541–553.

Prensky, M. (2001) Digital Game-Based Learning, McGraw-Hill, New York.

Rieber, L. and Noah, D. (2008) ‘Games, simulations, and visual metaphors in education: antagonism between enjoyment and learning’, Educational Media International, Vol. 45, No. 2, pp.77–92.

Salen, K. and Zimmerman, E. (2004) Rules of Play: Game Design Fundamentals, MIT Press, Cambridge, Mass.

Saurbaum, J. and Brown, R. (2006) ‘Econ 201: principles of microeconomics’, (accessed on 22 February 2007).

Schneider, M., Carley, K. and Moon, I-C. (2005) ‘Detailed comparison of America’s army game and unit of action experiments’, United States Army.

Shaffer, C.R. and Anundsen, K. (1993) Creating Community Anywhere, Tarcher/Perigee Books, NY. Somaiya, R. (2009) ‘Latest threat to student health: trolls and orcs’, The Guardian, 6 January 2009. p.1.

Squire, K. (2004) ‘Replaying history: learning world history through playing ‘Civilization III’, Unpublished dissertation, Indiana University-Bloomington, Bloomington, IN. Squire, K. (2006) ‘From content to context: videogames as designed experience’, Educational Researcher, Vol. 35, No. 8, pp.19–29.

Squire, K. (2008) ‘Video game-based learning: an emerging paradigm for instruction’, Performance Improvement Quarterly, Vol. 2, No. 2.

Squire, K., Giovanetto, L., Devane, B. and Durga, S. (2005) ‘From users to designers: building a self-organizing game-based learning environment’, Tech Trends, Vol. 49, No. 5, pp.34–42, 74.

Tanner, L. (2007) ‘Is video-game addiction a mental disorder? (electronic version)’, MSNBC/Associated Press, available at http://www.msnbc.msn.com/id/19354827/ (accessed on 15 November 2007).

Tuzun, H. (2004) ‘Motivating learners in educational computer games’, Unpublished dissertation, Indiana University, Bloomington, IN.

16

S.J. Warren et al.

Tuzun, H., Yilmaz-Soylu, M., Karakus, T., Inal, Y. and Kizilkaya, G. (2009) ‘The effects of computer games on primary school students’ achievement and motivation in geography learning’, Computers & Education, No. 52, pp.68–77.

Walker, A. and Shelton, B. (2008) ‘Problem-based educational games: connections, prescriptions, and assessment’, Journal of Interactive Learning Research, Vol. 12, No. 4, pp.663–684.

Warren, S. and Dondlinger, M. (2008) ‘Designing games for learning’, in Fertig, R. (Ed.): Handbook of Research on Effective Electronic Gaming in Education, Idea Group Reference: IGI Global, Hershey, PA. Warren, S., Barab, S. and Dondlinger, M. (2008) ‘A MUVE towards PBL writing: effects of a digital learning environment designed to improve elementary student writing’, Journal of Research on Technology in Education, Vol. 41, No. 1.

Warren, S., Dondlinger, M., Stein, R. and Barab, S. (2009a) ‘Educational game as supplemental learning tool: benefits, challenges, and tensions arising from use in an elementary school classroom’, Journal of Interactive Learning Research, Vol. 20, No. 4.

Warren, S., Stein, R., Dondlinger, M. and Barab, S. (2009b) ‘A look inside a design process: blending instructional design and game principles to target writing skills’, Journal of Educational Computing Research, Vol. 40, No. 3.

Warren, S.J. and Dondlinger, M.J. (2008) ‘Objective achievement in a PBL course: findings from a study on the first iteration of a problem-based learning course game’, Paper presented at the Association for Educational Communications and Technology, Orlando, FL. Wikipedia (2009) ‘Pong’, Wikipedia Online, available at http://en.wikipedia.org/wiki/Pong (accessed on 6 November 2009).

Wikipedia (2009) ‘World of Warcraft’, Wikipedia, available at http://en.wikipedia.org/wiki/World_of_warcraft (accessed on 6 November 2009).

Wikipedia (2009) ‘Zork Zero’, available at http://en.wikipedia.org/wiki/Zork_Zero (accessed on 7 January 2009).

Comment [t1]: Author: Please cite the reference in the text or delete from the list if not required.