Journal of Virtual Worlds Research - Volume 3, Number 1: The Researcher's Toolbox by Jeremiah Spence

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 Editor-in-Chief

Jeremiah Spence

Guest Editors

Tom Boellstorff, University of California, Irvine, USA Celia Pearce, Georgia Tech University, USA Dmitri Williams, University of Southern California, USA Thomas Malaby, University of Wisconsin, Milwaukee, USA Elizabeth Dean, RTI, USA Tracy Tuten, East Carolina University, USA

Technical Staff

John Brengle Sil Emerson

The Journal of Virtual Worlds Research is owned and published by the Virtual Worlds Institute, Inc. â&#x20AC;&#x201C; Austin, Texas, USA. The JVWR is an academic journal. As such, it is dedicated to the open exchange of information. For this reason, JVWR is freely available to individuals and institutions. Copies of this journal or articles in this journal may be distributed for research or educational purposes only free of charge and without permission. However, the JVWR does not grant permission for use of any content in advertisements or advertising supplements or in any manner that would imply an endorsement of any product or service. All uses beyond research or educational purposes require the written permission of the JVWR. Authors who publish in the Journal of Virtual Worlds Research will release their articles under the Creative Commons Attribution No Derivative Works 3.0 United States (cc-by-nd) license. The Journal of Virtual Worlds Research is funded by its sponsors and contributions from readers. If this material is useful.

Journal of Virtual Worlds Research Volume 3, Number 1 November 2010 “The Researcher’s Toolbox” ISSN: 1941-8477 Table of Contents • Virtual Worlds, the IRB and a User's Bill of Rights o Jeffrey M. Stanton, Syracuse University • The Neil A. Armstrong Library and Archives: That’s One Small Step for a Virtual World Library, One Giant Leap for Education! o Shannon Bohle (Archivist Llewellyn in Second Life), CoLab Library Director/Volunteer, NASA JPL/Caltech • Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds o Christopher A. Paul, Seattle University • Conducting Empirical Research in Virtual Worlds: Experiences from two projects in Second Life o Shailey Minocha, Centre for Research in Computing, The Open University, UK o Minh Quang Tran, Centre for Research in Computing, The Open University, UK o Ahmad John Reeves, Centre for Research in Computing, The Open University, UK • Interviews Within Experimental Frameworks: How to Make Sense of Sensemaking in Virtual Worlds o CarrieLynn D. Reinhard, Roskilde University, Denmark • dint u say that: Digital Discourse, Digital Natives and Gameplay o Theresa A. O’Connell, National Institute of Standards and Technology, United States o John Grantham, Systems Plus, United States o Wyatt Wong, Forterra Systems Inc., United States o Kevin Workman, Millersville University, United States

o Alexander Wang, Cornell University, United States • Applying Constant Comparative and Discourse Analyses to Virtual Worlds Research o Peter Leong, University of Hawaii at Manoa, United States o Samuel R. H. Joseph, University of Hawaii at Manoa, United States o Rachel Boulay, University of Hawaii at Manoa, United States • How to approach a many splendored thing: Proxy Technology Assessment as a methodological praxis to study virtual experience o Lizzy Bleumers, IBBT-VUB-SMIT, Belgium o Kris Naessens, IBBT-VUB-SMIT, Belgium o An Jacobs, IBBT-VUB-SMIT, Belgium • A Design Research Approach to Developing User Innovation Workshops in Second Life o Remko Helms, Utrecht University, The Netherlands o Elia Giovacchini, Utrecht University, The Netherlands o Robin Teigland, Stockholm School of Economics, Sweden o Thomas Kohler, Innsbruck University, Austria • What are users thinking in a virtual world lesson? Using stimulated recall interviews to report student cognition, and its triggers. o Lyn Henderson, James Cook University, Australia o Michael Henderson, Monash University, Australia o Scott Grant, Monash University, Australia o Hui Huang, Monash University, Australia • Using Design-Based Research for Virtual Worlds Research Projects o Antonio Santos, Universidad de las Americas Puebla, Mexico • eLab City: A Platform for Academic Research on Virtual Worlds o Thomas P. Novak, University of California, Riverside • Learning spaces, tasks and metrics for effective communication in Second Life within the context of programming LEGO NXT Mindstorms™ robots: towards a framework for design and implementation o Stewart Martin, Teesside University, UK. o Michael Vallance, Future University Hakodate, Japan. o Paul van Schaik, Teesside University, UK. o Charles Wiz, Yokohama National University, Japan

Volume 3, Number 1 The Researcher’s Toolbox November 2010

Virtual Worlds, the IRB and a User's Bill of Rights Jeffrey M. Stanton Syracuse University Abstract Virtual worlds provide a vibrant and exciting new venue for the conduct of social research. New social phenomena have emerged in the context of these worlds as individuals and groups use the worlds’ social communication tools to overcome limitations of time and distance. Yet these worlds also pose some novel ethical problems for social researchers who wish to conduct their work in these contexts. The present paper discusses ethical dimensions of research in virtual worlds in order to uncover some of these novel problems. The paper reports and analyzes a case study in which the author participated in a virtual world research study. The paper discusses a preliminary version of a virtual world subject’s bill of rights that can serve as the basis for further discussion among social researchers. The paper concludes with recommendations for virtual worlds researchers on strategies for working with their institutional review boards (human subjects ethics review) to obtain approval for their research proposals. Keywords: social research ethics; virtual worlds; human subjects review; bill of rights; institutional review board; IRB

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

Virtual Worlds, the IRB and a User's Bill of Rights Millions of individuals participate in one or more virtual worlds. Although the specific definition of the term virtual world is in flux, the term includes massively multiplayer online games (MMOGs) as well as general environments such as Second Life™. Because social interaction, textual and oral communication, and economic activity occur in these environments, they comprise a microcosm of the world at large. Whether behavior in virtual worlds is representative of behavior in the “real” world or not, these environments represent a fascinating venue in which to conduct social research (Bainbridge, 2007). A variety of standard, “real world” research areas have parallel manifestations in virtual worlds including small group research, prosocial behavior, counterproductive and deviant behavior, and behavioral economics. In addition, newly emergent areas of social research made possible by new technologies offer a range of novel research opportunities: e.g., virtual collaboration, online community formation, virtual property rights, simulated/game violence, as well as a variety of questions related to creation and management of online social identity. Just as the Internet expedited the process of doing survey research (Stanton & Rogelberg, 2001), virtual worlds have potential to substantially change the process of doing ethnographic, interview, and naturalistic research. Bainbridge (2007) also notes that in worlds that support building or scripting, one may construct facilities and apparatuses that are necessary for laboratory research. When researchers can instantly access a diverse set of virtual world participants, a range of benefits and pitfalls will appear. One pitfall that may appear pertains to ethical issues of conducting research in virtual worlds. We have reams of research and shelves of books on social science research ethics in traditional, “real world” environments. While some of this material will certainly inform resolution of the ethical dilemmas that arise in the context of virtual world research, other novel situations will arise that will tax our essential understanding of what activities are permissible and impermissible for researchers working in virtual world environments. This paper provides a first attempt at uncovering and organizing the relevant issues in this area and provides an “alpha” version of a virtual world subjects’ bill of rights. The Research Ethics Context in Brief Contemporary research ethics in the U.S. have been influenced by the Belmont Report (1978), which was commissioned by the federal government following passage of the 1974 National Research Act. The report, in turn, spurred the development of federal rules, which mandate the use of institutional review boards to examine and approve planned studies. The Belmont Report outlined three principles that apply equally to biomedical and behavioral research: respect for persons, beneficence, and justice. In turn, these principles, when applied to the conduct of research, each translate into an operational requirement: the use of informed consent, the application of risk/benefit analysis, and fairness in the selection of research subjects. Taking each in turn, the principle of respect for persons finds its operational match in the use of informed consent. Respect for persons derives from the Kantian moral imperative that individuals should only be treated as ends and never as means. The Belmont Report refers to

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

treating each person as an “autonomous agent” and relatedly, that individuals with diminished autonomy (e.g., children) must receive extra protection against unethical treatment. In operational terms, the use of informed consent ensures that individuals who volunteer to participate in research must receive adequate information on any adverse affects they may experience as a result of participation. Informed consent thus comprises disclosure of information about the research, measures that ensure the individual has comprehended the information, and the opportunity to voluntarily consent to participation. As a footnote to this consideration of informed consent, Title 45 of the U.S. Code of Federal Regulations, Part 46.116, indicates when the use of informed consent may be waived. These circumstances include situations where risk to subjects is minimal, where the waiver will not adversely affect the welfare of subjects, where the research would be impractical if informed consent were required, and where subjects can obtain additional information about the research afterwards (Kopelman, 2004). Note that this last element, referred to as debriefing, is a standard method of reducing risk by providing subjects with information to mitigate negatives impact that study procedures may have. The second principle – beneficence – refers to efforts to ensure the well being of research participants; it is paired with the operational activity of risk/benefit analysis. Beneficence, as used in the Belmont Report, means avoiding any activity known to cause direct harm as well as minimizing future potential harms and maximizing future benefits. Under this principle, researchers are obliged to analyze the methods and goals of their study and uncover all foreseeable harms that subjects may experience – physical and psychological. Then, the researchers must argue that the expected benefits of their research will substantially exceed the foreseeable harms. The standard baseline for risk in these assessments is the risk of harm from daily activities of normal living. Risks at or below this level are considered minimal, while risks above this level must be justified with respect to the benefits. Many social science studies exhibit only a weak promise of future benefits to individuals or society, so it is fortunate that the majority of those studies also bear minimal risk for participants. The third principle, justice, was considered critical in the Belmont Report because of a long U.S. history of mistreatment of members of minority groups. In one highly notable case, nearly 400 African American men living near Tuskeegee, Alabama were denied available treatments for syphilis over a period of 40 years as part of their involvement in a study examining the course of this disease. The race and socioeconomic class of these men the basis for their selection into the study. A leak to the press concerning this study in 1972 was a primary stimulus for the National Research Act and the Belmont Report. In response to this injustice, the Belmont Report explicitly states that all groups in society must equally bear the burden of human subjects research and obtain the benefits of that research. Selection processes for inclusion or exclusion of subjects must consider two interrelated concerns. First, an individual should not be recruited into a project based solely on easy availability or on manipulability (e.g., individuals with mental disabilities). Second, the Belmont Report advocates a kind of “affirmative action” for participant selection: studies should prioritize recruitment based on the risk/benefit ratio in the study. Those individuals who are less privileged

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

in society should obtain greater opportunities for studies with greater benefit and should be protected from overrepresentation in studies with greater risks. Summary: Following major ethical problems in research, passage of the National Research Act in 1974, led to the Belmont Report. This report guided development of rules guiding the conduct of research including institutional review boards (IRBs). IRBs now form the backbone of research ethics review in the U.S. Three basic principles inform IRB review: respect for persons, beneficence, and justice. Each principle reflects in an operational aspect of study review and conduct: informed consent, risk/benefit assessment, and subject selection. Applying the Belmont Principles to Virtual Worlds When a researcher decides to collect data in the context of a virtual world, a number of novel ethical issues arise. The novelty springs from fundamental differences between a person’s “existence” in the virtual world and conditions under which a person might participate in a traditional research environment. In the traditional environment, whether a laboratory or naturalistic environment, the researcher encounters the research participant physically. Even in research that uses mediated communications, the researcher has social cues about the research participant that establish a baseline level of assurance about the researcher-subject relationship. The researcher can verify the subject as an adult individual of a certain age and sex who is capable of voluntarily giving consent for participation, who has comprehended the study description, and who is available for debriefing. Likewise, the participant can judge the veracity of the researcher using cues such as the researcher’s dress and demeanor, setting, and appearance of materials or apparatuses. In contrast, the identity of a research subject who participates in the context of a virtual world is essentially unknowable to the researcher. The subject may be a child or adult, male or female, old or young. Likewise, it is impossible for the researcher to know if the individual is capable of making a consent decision, or alternatively is intoxicated or impaired. The researcher has no straightforward method of ascertaining whether the subject understands the study and no way of ensuring that the subject participates in debriefing. The subject, in turn, has no way of verifying the researcher’s identity or intentions. Dress and demeanor of the researcher are a function of how the researcher has designed his or her avatar. The setting is infinitely malleable, depending on the construction capabilities available in the virtual world, and may bear no resemblance whatsoever to a traditional university or laboratory environment. The apparatus for a study conducted in a virtual world may manifest in software that is invisible to the research participant. Even those objects that do have a recognizable manifestation may not have a function with which the research participant is familiar. In some respects, a virtual world study is akin to a postal mail or email study in which the researcher never has physical contact with research participants. This analogy is encouraging in the sense that these researchers have learned to overcome the disadvantages of distance and separation from their respondents. The analogy is also a warning in the sense that there are limitations to what kinds of studies are feasible in the absence of a direct physical encounter between researchers and subjects. For example, any study that contains substantial psychological risks (e.g., disturbing scenarios/questions about antisocial/illegal behavior) is difficult to conduct

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

by mail, because of the researcher’s inability to guarantee delivery of the research stimuli to the sampled individual, to gauge the participant’s reactions, or to provide debriefing or support after the study. Another analogy to explore is with the telephone interview. These interviews are synchronous and mediated, two similarities to researcher-subject interactions that may occur in virtual worlds. Telephone interviews usually involve just one subject at a time, however, and most virtual worlds by their nature have transcended this limitation. The telephone interview is rich with social information that the researcher and participant may perceive from vocal cues in the conversation. Virtual worlds that provide voice chat may offer the same benefits. These cues contain information relevant to the ethical conduct of research. The research may infer aspects of the participant’s state of mind (such as whether they are mentally capable of volunteering), whether the participant has understood informed consent information, and the progress and effectiveness of debriefing. Likewise, a researcher may use voice cues to know whether the participant is a child and unable to legally give consent. Risk/Benefit Assessment in Virtual World Studies. In one sense, virtual world studies have a benefit over “real world” studies: there is little potential for direct harm to the physical body. In a similar way, certain psychological risks may diminish, because the ability to teleport out of the research situation can provide an effective method of ending participation, without the necessity of physically extricating oneself from the research environment. Conversely, however, the risk of psychological harm may increase if the researcher inserts him or herself into an intact social environment to study the inhabitants. Consider the following example: Finn and Lavitt (1994) downloaded, analyzed and published notes from an online support group for sexual abuse survivors, making no request to the group members for permission. This data collection occurred even though the authors reported the existence of a note from the group’s moderator saying that interested people who were not sexual abuse survivors were discouraged from joining this group. The exact posting date and time of verbatims from group members, as well as the group’s name, appeared in the published results. As Flicker, Hans, and Skinner (2004) noted, one side effect of this publication was that the online group disbanded, because members believed that they had been “exposed” even though no individual identities had been revealed. A cautionary principle here is that a publicly available stream of (communication) behavior does not equal a dismissal of all expectations of privacy or an automatic granting of permission for research use of such data. Another cautionary principle is that communities of all types (including online) are potentially fragile, and researchers may detrimentally affect them through exposure of the community’s activities in research results. As Kraut et al. (2004) suggested, “Whether a person conversing online can reasonably expect the communication to be private depends upon legal regulation, social norms, and specific details of implementation, all of which are changing.” Fairness in Subject Selection: Despite increased dispersion of the Internet throughout the world, millions of people do not have access and are therefore absent from virtual worlds. In rural and low-income regions the infrastructure lags significantly behind other areas. Certain groups of potential Internet users (e.g., the disabled, older workers, single-parents) have, on average, lower degrees of access, and, in some cases, skills and knowledge about the Internet (Norris, 2001;

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

NTIA, 1999; Warschauer, 2004). Finally, many individuals have equipment that is incompatible (e.g., because of outdated equipment or slow Internet connections) with the demands of virtual world servers and clients. As a result, in parallel with traditional research, technologically underserved populations are least likely to enjoy benefits from virtual world research whether these benefits are tangible – e.g., participant incentives – or intangible. Of course, this exclusion has its converse: these populations are also unlikely to be exposed to harms from studies. Summary: The absence of a physical encounter between researcher and subject increases the difficulty for implementing the Belmont Principles. In traditional research, researchers have overcome some of these barriers. Using analogies of a mail and telephone surveys it is possible to draw some important lessons for the conduct of a virtual world study. These analogies break down when including the immersive and participative aspects of research in virtual worlds. Researchers whose avatars are “present” in an intact social situation must exert extra care not to cause harm. Finally, continuing issues of digital divide factor into sample selection in virtual world research, because technologically underserved populations are not able to participate. Case Study: Participation in a Virtual World In 2008, the author participated in an experimental study conducted in Second Life by a faculty member and two Ph.D. students from a research university. The study included an initial contact with a student researcher, exposure to an experimental stimulus, completion of a web-based research instrument, and a final brief contact with the student researcher. All interactions, as well as the research instrument itself, were in English and were textual (i.e., chat) rather than oral. Records of the interactions, primarily in the form of chat logs, provided the basis of the following impressions. The author located information about the research on a public website and contacted the named individual – a student researcher – by email as requested. The researcher set a mutually convenient time to meet in Second Life. The researcher self identified in the email by first name only and did not provide information by email about his or her university affiliation or the nature of the study. The author logged into Second Life at the appointed time and awaited a teleport request from the researcher, which occurred within the agreed time parameters. The author’s avatar followed the teleport request and arrived at a research facility – an unadorned room on an island sponsored by the researcher’s university – to meet the researcher’s avatar. The gender of the avatar and the typically construed gender of the first name on the email contact did not match; it was unclear whether this was part of the experiment. The researcher’s avatar gave some brief instructions by text chat about how to complete the study. No mention of participant incentives, time duration, or the nature of the study was provided. The researcher’s avatar requested, “It is essential for our survey that you do not discuss this survey with others because this would lead to falsification of the data.” The author’s avatar asked, “Who is running this study?” since that information was not included in the instructions. The researcher’s avatar replied, “I am, with the help of a colleague of mine, plus my research supervisor at our university.” (Note that avatar responses have been copy edited to protect the identity of the student.) This reply did not contain any further information about real life names

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights

or host institution, but the opening screen on the web-based survey did contain names and email links to two student researchers and a faculty member. The opening screen also contained an estimate of the completion time of the instrument, the number of questions on the instrument, general information about the topic, and the nature of the participation incentives. The opening screen did not describe any risks or benefits to participation. Prior to taking the web survey, the researcher’s avatar provided an assurance about anonymity: “In order for you to take part in the lottery you have to enter your avatar’s name. We will not link your avatar’s name to your responses, so the survey will remain anonymous.” Next, the author’s avatar viewed an experimental stimulus. Note that the stimulus was not identified as such by the researcher, but it was evident from the nature of the questions on the survey that it was indeed an experimentally manipulated stimulus, and that the experiment was a between subjects design. The experiment was consistent with the risks of everyday living – that is, the stimulus was similar to other stimuli that one might regularly encounter in Second Life, and was parallel to stimuli that one might regularly encounter in real life. Likewise, the web survey contained no questions of a troubling nature and did not request reports of illegal or unethical behavior. The survey was anonymous, although as noted above, the instrument did request the participant’s avatar name for purposes of providing the participant incentive. The survey comprised roughly a dozen initial questions, demographic inquiries including questions about Second Life usage, and manipulation check questions. The author met all eligibility criteria and completed all questions truthfully in order to avoid injecting false data into the study. Following completion of the survey, instructions indicated that the participant should return to Second Life. The author’s avatar returned to the student researcher’s location and had another brief encounter with him or her. The researcher’s avatar thanked the author for participating: “I wanted to thank you and also ask you if you might have any friends who might want to participate in this study?” The researcher’s avatar did not provide a debriefing or any other information about the nature of the study. The author’s avatar was dismissed from the facility and the researcher’s avatar went offline. Ethical Analysis of the Case A general analysis of the experiment described above and the questions in the web instrument suggest that this study contained minimal risk for participants as well as minimal direct benefit. The incentives for participation seemed non-coercive. Participation was voluntary. The sampling technique was open recruitment and snowball sampling, both of which in this case required access to the web and email. Additionally, completion of the experiment required installation and familiarity with the Second Life client application. With these characteristics, most U.S. institutional review boards would either classify the study as exempt from IRB review, or subject to expedited approval. In either case, it seems likely that a proposal describing the study would have passed IRB scrutiny, albeit with minor modifications. Experimental studies should generally use a debriefing, particularly when deception or disguise of purpose appears in the study design. This study included a mild disguise of purpose, in that the researcher never identified the study as an experiment and never noted the existence of the experimental stimulus. Relatedly, the researcher requested that the participant not discuss the experimental protocol with anyone else, a sure sign that the purpose of the manipulation needed

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights 10

to remain under wraps. As a result, best practices dictated that the researcher ought to have provided a debriefing. Even for an exempt study, the identification of the researchers and their institution was insufficient. Most U.S. IRBs require provision of telephone numbers of the researcher and the IRB, in addition to any other contact information provided. The student researcher who conducted the email and avatar contacts should have identified him or herself by first and last name, with a clear indication and provision of the sponsoring institution and faculty member. Given the capabilities of Second Life, it would have been sensible to provide the participant’s avatar with a “notecard” at the beginning of the experiment with full contact information and partial study information as well as at the end of the experiment with more complete study information and a debriefing. These notecards should have included information about the IRB approval file number for the study. Perhaps most importantly, the researcher never requested informed consent from the participant. If the study had been classified as exempt by the reviewing IRB, it is possible that the IRB would have approved “implied consent” – that is, one’s actual participation in the study would comprise an indication of voluntary consent. Some IRBs insist on actual consent for any study that includes an experimental manipulation, regardless of how low the risk of participation may be. Even if the reviewing IRB did not include such a requirement for this study, it would have been sensible and would have imposed minimal administrative overhead for the researcher to have stated that the study included minimal risk and that by choosing to continue, the participant was providing implied consent. Finally, the researcher should have provided information about participant incentives from the point of initial contact with the participant. This information should have included a statement of eligibility for the participant incentives. The researcher referred, obscurely, to a lottery; the survey itself later clarified this with specific information about the prizes offered. No indication appeared concerning the date, notification, or contact strategy for these awards. In short, the case study presented a number of minor ethical shortcomings that the research team could easily have rectified in designing their protocol. Interestingly, the case indicates that for experimental studies, the analogies between common “real life” practice and research conducted in a virtual world environment are strikingly clear. This provides an important basis for describing a “virtual world subjects’ bill of rights” that can provide guidance to researchers and participants with respect to how to conduct virtual world research. Virtual World Subjects’ Bill of Rights: Version 0.01 In the material below, I outline a set of basic rights based on ethical principles outlined in the Belmont report and informed by the case study. Consider these as a basis for discussion rather than a completed work – hence the designation Version 0.01. Undoubtedly, as the research community obtains more experience conducting research in virtual worlds, we will devise refinements to these rights that clarify their meaning, applicability, and functions. 1. The right to know that I am a subject: If you obtain data from me in a virtual world for research purposes, I have a right to know that I am in your study. Ethnographers and

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights 11

other naturalistic researchers will complain – with some justification – that the unobtrusive observation of public behavior does not warrant prior notification. Assuming the researcher is a legitimate witness to such public behavior, the threshold for unobtrusiveness is asking a research question. Before a researcher asks a subject a research question, the researcher must notify the participant that research is occurring. 2. The right to know you as a researcher: If I am a subject, you the researcher must represent yourself accurately so that I can confirm your identity. Although this obligation need not compel the researcher to use a photorealistic avatar, the subject must receive sufficient information to trace the avatar back to a specific person working in the context of a specific host institution. 3. The right to know who approved your study: Before participating in your study, I have the right to know what ethics body, if any, reviewed your research design. Virtual worlds lack international boundaries, and ethics review practices vary substantially across different countries. Researchers must inform subjects whether their research received ethics review. If a study did undergo review, the subject has a right to know what body performed the review and the results of that review, preferably in the form of an approval file number and a link to the homepage of the ethics review body. 4. The right to learn the risks: You must warn me if the study includes psychologically distressing material, if there is a risk that my avatar or I may be identified, if there may be a tangible or intangible costs to participation, or if other risks to me or my avatar exist. Although, the likelihood of physical harm from participating in a virtual world study is low in almost any conceivable research design, the possibilities for psychological harm have the same dimensions as in a telephone, postal mail, email, or web-based study. Further, to the extent that the individual’s avatar has a distinctive social identity within the context of the virtual world, there is a risk for disruption of that avatar’s social identity and status, particularly if the researcher revealed an avatar’s identity in research outputs. 5. The right to learn the benefits: I want to know why my avatar’s participation in the study is desirable, even if the benefits to me are indirect. In many instances, individuals participate in research expecting that they may receive a reward; they may have little concern for the scientific merits or outcomes of the study. In virtual worlds research it is also unlikely that any physical benefit – such as better health – can manifest as a benefit. Nonetheless, the subject has a right to learn about the full range of benefits from participation in the study, including such intangibles as improving scientific knowledge of a topic. 6. The right to know why my avatar was chosen: If researchers contacted my avatar, I want to know how they got my avatar’s name and what makes my avatar eligible to participate. Individuals recruited from sampling lists of any type should receive notification of the origins of the list, the permissions that the researchers obtained to use the list, and the eligibility criteria used in placing the subject on the list and selecting them from the list. In virtual worlds, greeter robots (which often record visitor lists) and

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights 12

group memberships serve as likely sources of research participants. Researchers need to provide the opportunity for subjects to obtain information about these origins of their lists and what, if any, selection criteria were used. 7. The right to participate as my avatar: If you recruit me for your virtual world study, I have the right to respond to your study in the identity and role I have selected for my avatar. One notable characteristic of a virtual world lies in the opportunity it provides for individuals to explore alternative identities and social roles. By conducting a study in a virtual world, the researcher has consciously chosen to study this environment and is thus obliged to let subjects maintain their identities and roles as they have construed them for that particular environment. In short, researchers should rarely admonish an avatar to “respond as you would in real life.” 8. The right to protect my group: If you are studying my social group, I have the right to protect the integrity and continued existence of my group. In principle, a researcher who studies an intact social group should have consent from all members of that group. Likewise, in principle, if members of the group object to the researcher’s presence or use of the group for research, those members should have veto power. In practice, it may be impractical for researchers to obtain active consent from every member of a large group, or from a group that has inactive members. In such cases, legitimate representatives of a group have the vested ability to speak for the group and to provide or revoke permission for any given research activity. 9. The right to teleport: When participating in your study, I reserve the right to teleport out of the research situation if I am uncomfortable with any of the procedures or questions. Standard language in informed consent states that the research participant has the right to withdraw from the study at any time without prejudice. In a virtual world, one mechanism of withdrawal is teleporting. Researchers must avoid imposing constraints on teleporting and should inform subjects that they may choose to teleport out of the research situation if they become uncomfortable. One concern raised by this right is that avatars may disappear for reasons other than discomfort – for example, because of a technical glitch. Researchers should develop and describe a method by which the subject’s avatar may resume or restart the study in case of such glitches. 10. The right to debriefing: If you use deception or disguise of purpose in the study, I deserve to learn about it afterwards. Given the ephemeral nature of “presence” in virtual worlds and the difficulty with ensuring comprehension of debriefings, researchers need to devise effective debriefing procedures in virtual world studies. 11. The right to be left alone: Following my avatar’s participation in your study, whether I completed it or not, I have the right to not be contacted again by the researchers. Similarly to email and instant messaging, electronic communication tools available in virtual worlds often lack the normal social boundaries that reduce unwanted contacts among people. Once an avatar name is known, a sender may send a message to that avatar, regardless of physical distance or social barriers between them. Researchers must avoid abusing this capability. If a study requires repeated contact with an avatar over

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights 13

time, this requirement appear at the beginning of the study, and the subject must actively give permission for repeated contacts. Researchers should keep written records of the granting of such permissions, e.g., as a time-stamped chat log. The catalog of rights listed above reflects some unique elements related to virtual world research and some common elements ubiquitous across all types of research. Presentation of this list does not imply that other rights from traditional research – or the procedures used to ensure those rights – should be neglected in virtual world research. Rather, one should view the list as supplemental food for thought, so that the unique aspects of virtual world research can find a reasonable mapping onto the ethical principles we already know and endorse. By generating rumination and discussion among the community of social researchers, this preliminary virtual world subjects’ bill of rights may help to ensure that virtual worlds become an active and vibrant source of information for rigorous and ethical social research. Working with Institutional Review Boards As the preliminary description of the virtual world subjects’ bill of rights above suggests, there are some important ethical considerations for research virtual worlds that do not typically arise in the course of traditional social research (Ess & Jones, 2004; Jankowski & van Selm, 2001). While the membership composition of institutional review boards (IRBs) varies substantially across different institutions, it is safe to say at this writing that few of the members of a typical IRB will have direct experience interacting in virtual worlds, let alone conducting research there. What follows is that virtual world researchers are well advised to educate the members of their IRB on the essential operational characteristics of virtual worlds. This knowledge will help IRB members to understand your arguments about the risks and benefits of research in virtual worlds. Such education can be accomplished in small measure in a research proposal sent to the IRB, but this misses an opportunity to work with the members of the board in advance, and thereby prepare the ground for when the board receives a relevant proposal. All certified IRBs in the U.S. have the capability of inviting researchers into board meetings to answer questions – usually when a problematic protocol has been received – but this same mechanism can be used to provide a brief tutorial on an emerging topic of interest to the members of the IRB (Amdur & Bankert, 2003). Researchers should contact the chair of the IRB or the lead administrator of the IRB to request a time to speak with the board and should then prepare a very brief presentation on the topic (e.g., how individuals communicate with one another in a virtual world) and be prepared to answer questions. One of the key questions that every board member considers when discussing the viability of a research protocol is risk. As described earlier, the standard baseline for risk in IRB discussions is the risk of harm from daily activities of normal living (Kopelman, 2004). Risks at or below this level are considered minimal, while risks above this level must be carefully justified and weighed against the benefits of the research. Proposed studies that have greater than minimal risk are scrutinized very carefully by IRBs and are likely to take the longest for review. Therefore it is a valuable strategy to design a study, wherever possible, to create minimal risk of harm for participants. As discussed above, harms from virtual world research are mainly psychological rather than physical. In this light, a researcher minimizes risk by not asking questions or observing behaviors that could cause research participants to be liable to civil or criminal prosecution, cause them to lose their jobs or financial standing, cause damage to their

Journal of Virtual Worlds Research – Virtual Worlds, the IRB and a User's Bill of Rights 14

reputations, or cause them to be socially stigmatized. Researchers can also help to avoid these harms by taking as many precautions as possible to protect the privacy of research participants. In the context of virtual worlds, it is not always easy to connect a researcher’s actions to these types of harms. For example, there are whole classes of illegal behavior – such as physical assault – that don’t have a direct analog in virtual worlds and therefore are not subject to researcher observation. On the other hand, adverse influence on a research participant’s reputational, finances, or employment may be quite salient, as there can be just a thin dividing line between a research participant’s work or home life and their “virtual” life. Researchers should take every opportunity to brainstorm with their colleagues – and ultimately even with the chair of the IRB – on inadvertent ways in which behavioral observation or questioning of an avatar may have an adverse influence on a research participant’s “first life.” In many cases when a problem is identified, a workaround will also become apparent, for example through improving privacy protections applied to collected data. To take a simple example, if a researcher was using screen capture to record the activities of a group of avatars, a simple adjustment to user interface preference could eliminate identifying information while keeping the data usable for analysis. Finally, actions associated with identifying oneself as a researcher and obtaining consent from research participants prior to involving them in research are central concerns to most IRBs. This issue was identified above in the very first element of the virtual world subjects’ bill of rights. Because the involvement of a researcher’s avatar in a virtual setting can seem so ephemeral – just pixels on a screen – it may be tempting to forgo consent and identification, particularly if the researcher is just observing and not interacting with other avatars. Countering this argument, though, is the power of teleportation to place oneself socially in an virtual setting where one is neither known nor welcome – like walking into a teacher’s classroom unannounced and uninvited. Unless a truly naturalistic research design is the only viable option for a particular research study, researchers should clearly describe in their IRB proposals how and when they will announce themselves as researchers and the mechanisms(s) they will use to obtain informed consent from participating avatars. Conclusion Virtual worlds provide a valuable new setting in which social research can be conducted as well a set of fascinating new social phenomena. At the same time, social researchers who wish to conduct their work in these contexts must be cognizant of novel ethical issues that may arise in their virtual world studies. The preliminary virtual world subject’s bill of rights presented in this paper may serve as a useful stimulant for discussion and brainstorming about how to create valuable and rigorous studies that carefully preserve the rights of human research participants. Educating the members of institutional review boards to these novel situations is one strategy for helping to ensure that high quality, relevant, and ethical research practices will be the norm in virtual worlds research.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Virtual Worlds, the IRB and a User's Bill of Rights 15

References Amdur, R., & Bankert, E. (2003). Institutional review board member handbook: Jones & Bartlett Publishers. Bainbridge, W. S. (2007). The Scientific Research Potential of Virtual Worlds. Science, 317(5837), 472-476. Ess, C., & Jones, S. (2004). Ethical decision making and Internet research: recommendations from the AoIR ethics working committee. Readings in virtual research ethics. Issues and controversies, 27-44. Finn, J., & Lavitt, M. (1994). Computer-Based Self-Help Groups for Sexual Abuse Survivors. Social Work with Groups, 17, 21-46. Flicker, S., Haans, D., & Skinner, H. (2004). Ethical Dilemmas in Research on Internet Communities Qualitative Health Research, 14(1), 124-134. Jankowski, N., & van Selm, M. (2001). Research ethics in a virtual world: Some guidelines and illustrations.

Retrieved

August

11,

2008,

from

http://collections.lib.uwm.edu/cipr/image/20.pdf Kopelman, L. (2004). Minimal risk as an international ethical standard in research. Journal of Medicine and Philosophy, 29(3), 351-378. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological Research Online: Report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59(2), 105-117. Norris, P. (2001). Digital Divide: Civic Engagement, Information Poverty, and the Internet Worldwide: Cambridge University Press. NTIA. (1999). Single-parent households at information disadvantage. Stanton, J.. (1978). The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research (No. DHEW Publication No. (OS) 78-0012). Washington, DC. Stanton, J. M., & Rogelberg, S. G. (2001). Using Internet/Intranet Web Pages to Collect Organizational Research Data. Organizational Research Methods, 4(3), 200. Warschauer, M. (2004). Technology And Social Inclusion: Rethinking the Digital Divide: MIT Press.

Volume 3, Number 1 The Researcher’s Toolbox November 2010

The Neil A. Armstrong Library and Archives: That’s One Small Step for a Virtual World Library, One Giant Leap for Education!1 Shannon Bohle (Archivist Llewellyn in Second Life) CoLab Library Director/Volunteer, NASA JPL/Caltech

Abstract Migration of library services has moved increasingly toward 2D web services and social interactivity and 3D virtual reference services. Patron satisfaction with reference services is a key tenant for determining a successful reference interaction. The study examines the roles of user identity, educational learning modalities, institutional sustainability, and credibility in achieving overall user satisfaction in reference social interactions within NASA JPL’s STEM education-focused library and other more generalized libraries within the virtual world Second Life. Keywords: Education, Library, Virtual Worlds

Revised and updated from the Virtual Worlds Best Practices in Education Conference, 2010

One Small Step for a Virtual World Library

The Neil A. Armstrong Library and Archives: Thatâ&#x20AC;&#x2122;s One Small Step for a Virtual World Library, One Giant Leap for Education! What is education without the library? Libraries are strategically positioned at the geographic center of college campuses for good reason. They represent the primary access point for students to obtain materials to meet their information needs and serve as social meeting places and centers for informal learning. Some would argue that during the Information Age the library has been gradually migrating from the â&#x20AC;&#x2DC;bricks and mortarâ&#x20AC;&#x2122; model of the 19th and 20th centuries to a borderless, networked, digital nexus. The wired campus provides students access to specialized online databases, scanned archival documents, and digitized books. Rather than students going to the library, the library has gone to the students. Whether through automated delivery of interlibrary loan articles or student email accounts, digitized course reserve materials or lectures downloadable to hard drives or I-pods, individualized RSS feeds for new articles on specific subject topics or new uses of social media to reach students, the library and its educational services has gone digital. To some degree, the library has now gone virtual as well. Perhaps the best example of the virtual library is the library project, the Library and Archives at NASA CoLab in Second Life. The Library is the first and only library or archive in a synthetic virtual environment recognized by the Library of Congress. Collaborative efforts have been interdisciplinary in nature, involving stakeholders in academia, business, and government. As academia moves toward greater integration with industry, partnerships with science, technology, and engineering academic faculty and students to government agencies like NASA are increasingly important. This project is an innovative use of virtual worlds as an example of best practices through bridging education to industry/government via virtual world technology.

One Small Step for a Virtual World Library

Figure 1. Virtual Doppelgänger (Photo Credit: Haplo Eberhart) The introductory photo shows the somewhat awkward juxtaposition of an avatar in a first life scenario. It is a visual reminder that our avatars are not ourselves, but "virtual doppelgängers." While many people feel their avatars are true extensions of themselves, there exists a dichotomy between our perception and others’ perceptions of the "self" and "other." I learned (after consultation with an attorney) that this difference is accentuated under the law. Avatars are neither recognized as legal entities nor as extensions of an individual. Liable or slander of an avatar’s name, for example, does not constitute liable or slander in the "real" world against the individual whom the avatar represents.2 Perhaps this is a problem with the law. Perhaps avatar names are in fact aliases. In any case, this brings to mind the question of how people as learners interact with the world, how the world perceives them, and how learning occurs within associated contexts and communities. In the 20th-century, Heidegger believed that "we can never...bring ourselves before ourselves as objects. That is, we can never conceptualize or objectify ourselves, see ourselves either from the outside or from the inside out."3 Yet, in the 21st-century, when existing and learning as avatars, people do just that. They create a digital version of themselves either how they perceive themselves or how they wish others to perceive 2

Attorneys may differ in their opinions. Ed. Groden,M., Kreiswirth,M., Szeman, I. “Heidegger, Martin.” Johns Hopkins Guide to Literary Theory and Criticism. Baltimore: Johns Hopkins UP, 2005.

One Small Step for a Virtual World Library

them. In Second Life, avatars find their virtual selves, as themselves objectified. In doing so, they are enabled to watch their behaviors from without in 360 degrees of latitude. Without "reallife" cues such as facial features and, in many cases, vocal intonations, viewpoints become amplified, and therefore virtual worlds are good examples where decision making and responses to psychological stressors can be tested by others. Virtual worlds can also help measure responses to change and the ability to adapt to different and unfamiliar circumstances. The military, for example, uses virtual worlds for training scenarios to measure and improve both of these types of responses to heighten situational awareness. Employers can use virtual worlds to train or improve their employees’ skills for personal interaction with a diverse, global consumer base. Additionally, people establish a sphere of familiarity, and when moving in and out of that sphere into different communities—whether virtually or in the "real" world—tension is created that can have social consequences relevant for learning. In such an amplified scenario, it may become possible for the individual learner, via virtual self-objectification, to gain selfactualization and achieve self-improvement. They do this by understanding the limitations of their own ability to cope with stress, to form relationships, to deal with change in a constructive manner, to perform new functions in role play and to coexist in a localized virtual social community and among different social virtual communities, just as they do in their "real life." Experiencing a virtual world can ideally bring about a greater awareness of one’s influence on others and one’s ability to successfully interact in (or withdraw from) the different cultures and communities they encounter. So too could it reveal hierarchies within those social communities to better understand the formation of power dynamics. What then is the role of the library in helping learners in this virtual construct? How can the library and its staff achieve the same library and educational standards in a virtual world it does in the "brick and mortar" world? That is what this talk addresses, using the Neil A. Armstrong Library and Archives as a case scenario, citing some of its patron reference questions, as well as integrating the results and feedback from an informal survey. In examining identity, the identity of virtual institutions as well as those of the individuals in virtual worlds (whether volunteers, employees, patrons, students, clients, or customers) comes under scrutiny. Oftentimes, lack of institutional buy-in from the "brick-andmortar" versions of virtual world iterations becomes an obstacle for entry as well as sustainability of virtual world presence. NASA is fortunate to be a leader in innovation and

One Small Step for a Virtual World Library

technology. It is not surprising, though, that even at NASA, delivering "real world" applications and acknowledgements are necessary for institutional funding and sustainability.

Figure 2. 2010 Linden Prize Finalist video filmed by Treet TV featured in the NASA Buzz Room (Source: http://buzzroom.nasa.gov/multimedia/videos/199/)

Figure 3. Library of Congress Listing

One Small Step for a Virtual World Library

Therefore, I applied to the Library of Congress for recognition of the Neil A. Armstrong Library and Archives at NASA JPL/Caltech. The application was accepted the library is the first and only virtual world library or archive to gain such recognition to date. In the Library of Congress screen capture, it is listed on equal footing with the "brick-and-mortar" libraries at NASA. Just as important as formal recognition as a "real" library has been the ability to migrate "real world" experiences into a virtual world environment. In every instance I have tried to implement best library and archival practices as I would have in a non-virtual environment. In the library, I posted a photo of my experience at NASA Glenn Research Center library and archives in my "first life" to help establish trust in me as a professional librarian by potential patrons. So too do I say in my profile that I am a credentialed librarian. As the Director, I participate in professional activities, such as presenting at "real world", "virtual world", and mixed reality events, publishing, and posting to professional listservs about the virtual library to promote its use and acceptance in the professional community. My avatar brings with her my "real life" experiences in education, both in learning and teaching. Educators are the first to realize that the ability to question is the foundation of learning. The NASA library is a subject specific library, and in that definition it strives to assist others to learn about NASA and its objectives in science, technology, engineering and mathematics (STEM).

One Small Step for a Virtual World Library

Figure 4. The Author in Real Life and Second Life The tangible value of the library revolves around access to materials, but its "intangible" value is perhaps more worthwhile; it is a place to stimulate interest and curiosity. To paraphrase Dr. Watson, with whom I worked as his archivist at Cold Spring Harbor Laboratory, teaching and learning science means asking the big questions. For him it was Schrödinger’s question, "What is Life?" and Darwin’s question, "How did life evolve?" He explained both, for which he won the Nobel Prize. How do we inspire students to think about and approach the big scientific questions like he did that still lay ahead, like "Where in the universe can life be found?"

One Small Step for a Virtual World Library

Figure 5. James D. Watson with the Author at the Cold Spring Harbor Laboratory Library and Archives Preservation and access to historical archival materials and published materials that encourage discovery, explain the scientific process, and allow personal engagement with NASA personnel and volunteers are the main functional goals of the library. Preservation of digitized and "born digital" scientific archival material is the subject of much discussion in the profession and is a concern in the scientific community. That is not to say that accomplishing these things is easy in a virtual library, or in a "brick-and-mortar" library for that matter and is a subject upon which I lectured at an international history of science conference Oxford.4 These questions are "leading edge," and applicability of best practices in a virtual world is very close—if not on—the "bleeding edge" of library and archival science. The theory of Multiple Intelligences came about in 1983 with the publication of Howard E. Gardner’s Frames of Mind, wherein it was recognized that intelligence is not limited to IQ. Gardner categorizes learning modalities (intelligences), as Linguistic, Musical, LogicalMathematical, Spatial, and Bodily-kinethestic. By design, the library attempts to facilitate different learning modalities.5 For example, students love to touch the protein molecule on

Bohle, S.. "Science Archives and History: Facilitating Discovery through Laboratory Notebooks" given at the Sixth Three Societies Conference: Connecting disciplines; University of Oxford, UK; 4-6 July 2008. 5 Gardner, Howard E. Frames of Mind. NY: Basic Books, 1983.

One Small Step for a Virtual World Library

display. They can listen to podcasts and short audio recordings of a Tuskegee Airman and Neil Armstrong.6

Figure 6. Students and their professor visiting the library One goal of a science library is to establish and maintain the trust and respect of patrons by establishing subject matter expertise achieved in a virtual world. Visibility and publications in the "real world" science community such as science educators, researchers, and students is paramount for trust and respect. The library displays are based on real science and real data. The protein p53 is a tumor suppressor. The original idea of the p53 molecule, for example, came from my experience writing a grant proposal at Cold Spring Harbor Laboratory (a National Cancer Institute Research Center) on 3d visualization of DNA and protein structures. I am pleased to announce that my article, "Studying the Causes of Cancer Creating the First 3d Model of p53 in a Synthetic Immersive Environment" about the p53 model on display is published on

Preston Pugh. Personal interview, Lima Public Library, 2009.

One Small Step for a Virtual World Library

the Nature website.7 In dualistic fashion, I must also appeal to the library and archival communities by writing for such publications as The Metropolitan Archivist.8

Figure 7. Nature Article (Source: http://blogs.nature.com/ub51cd45e/2010/03/12/studying-the-causes-of-cancer-creating-the-first3d-model-of-p53-in-a-synthetic-immersive-environmen)

Bohle, S.. "Studying the Causes of Cancer Creating the First 3d Model of p53 in a Synthetic Immersive Environment." Nature. http://blogs.nature.com/ub51cd45e/2010/03/12/studying-the-causes-of-cancer-creating-thefirst-3d-model-of-p53-in-a-synthetic-immersive-environmen. 12 March 2010. 8 Bohle, S.. â&#x20AC;&#x153;The Library and Archives at NASA CoLab in Second Life: A Virtual Co(-l)laboratoryâ&#x20AC;? http://www.nycarchivists.org/metro/2010_1.pdf. The Metropolitan Archivist. Winter 2010, p. 7-8.

One Small Step for a Virtual World Library

Figure

Cover

story

appearing

the

Metropolitan

Archivist

(Source:

http://www.nycarchivists.org/metro/2010_1.pdf) Just like good students, a good reference librarian never loses the ability to ask great questions. When Mark Sykes was on Science Friday here in Second Life, I could not help but ask a question related to NASA’s new policies, “In terms of lunar exploration, what about other countries, and what does this mean for geopolitics?”9 Taking it one step farther, I pursue a similar question during an interactive webcast, asking, “How will the ‘New NASA Plan’ handle international cooperation and make changes to ITAR (International Traffic in Arms Regulations)?” The four minute reply was from Bill Nye “The Science Guy" and Louis Friedman.10 As a librarian and former journalist, I have learned to approach people, speak up, and not to back down when asking uncomfortable questions, as well as to support Freedom of Information and an open and transparent government. That applies to a virtual, web-based or 9

Science Friday. “Rethinking the Human Future in Space.” National Public Radio. http://www.npr.org/templates/story/story.php?storyId=120613250&ps=rs. 20 November 2009. 10 Nye, B., Friedman, L. “The New NASA Plan.” The Planetary Society. http://www.ustream.tv/recorded/4822238. 18 February 2010.

One Small Step for a Virtual World Library

face-to-face interaction. So too did I participate in answering journalists’ questions such as those asked for a CNN article.11

Figure 9. Bill Nye and Lois Friedman Discuss the New NASA Plan (Source: http://www.ustream.tv/recorded/4822238) As a virtual librarian, I often refer patrons to web sources. I don’t consider myself, as one survey respondent stated, a "Google Helper." The respondent stated that the role of the virtual librarian appears limited to assisting people in navigating to a web page and that without database access can’t perform well. Perhaps a consortia of virtual libraries will one day purchase access to subscription databases. It is not a bad idea. But I would hardly call a good reference librarian a "Google Helper." For one thing, librarians know a wealth of information about evaluating resources. Directing patrons to the right resources means not just knowing how to manipulate a search tool, but becoming instinctively familiar with the right resource for the right job. They need to know how to evaluate new resources in terms of their reliability, their applicability, and their suitability for the question and purpose at hand, and they need to teach these skills during the reference interview through information literacy. 11

Eradicator. “The Next Giant Leap.” CNN. http://ireport.cnn.com/docs/DOC-301402. 20 July 2009.

One Small Step for a Virtual World Library

One of the great things about being a virtual world librarian is the ability to IM or talk to subject experts in the science community and ask questions on behalf of patrons, or to invite the expert to meet with the patron directly. Going to the source sometimes is not a web page or a virtual book, but connecting with a person. With complex scientific reference questions, I have occasionally requested the help of subject experts, like Dr. Mather, and scientists and science educators inworld.12

Figure 10. The author and avatar of John Mather, Nobel laureate in physics, in Second Life In many ways the library and archives is a celebration of and preservation of the past. Archives are often consulted in "brick and mortar settings" to prepare for celebrations like the 12

Bohle, S.. “Can results from Planck prove inflation paradigm?” The Nobel Prize Foundation. http://www.youtube.com/watch?v=aN9uaAEXnbY. 13 November 2009. Additional questions filmed at the library were asked of Nobel laureates Albert Fert ([the author]. “Are zeolite crystals grown in space viable for nanotechnology?” The Nobel Prize Foundation. http://www.youtube.com/watch?v=uek8-TapUPs. 13 April 2010” and David Gross ([the author]. “At the LHC, why collide lead ions instead of some other element?” The Nobel Prize Foundation http://www.youtube.com/watch?v=d4tbDBgX0HU. 25 June 2010, and [the author]. “Could you explain ‘compactification’ commonalities in string folding and protein folding?” http://www.youtube.com/watch?v=BLJ88MQ-6ak. 25 June 2010).

One Small Step for a Virtual World Library

40th Anniversary of the Apollo 11 moon landing. The Archive also supports the role of NASA CoLab to inspire future generations through role play, by providing virtual world shuttle suits and a working space shuttle model so visitors can experience being on board during a virtual shuttle launch and space trip. The future of education is now. Using virtual worlds and virtual world libraries for science education is no longer science fiction but science fact. There is a lot of work to be done to get the message out about the educational value of virtual world libraries and archives to both fellow professionals and the general public. NASA CoLab in Second Life is a place to network with NASA volunteers, employees, and contractors about the NASA work environment and the skills needed to work at NASA. In the Library visitors can find links to some "little known" resources to finding a job, grant, and contractor position with NASA. The Library also makes use of Web 2.0 applications like RSS feeds, Twitter, and Flickr groups.13 A quick rundown of the archival and subject specific library holdings after about 1.5 years of operation shows that there are: over 200 aeronautics and astronautics archival items, 5 years of unique, “born virtual” NASA meeting transcripts, donations, 15 full-text NASA digitized books (some over 300 pages long), 1 technical report, 1 serial publication (NASA News & Notes), 2 educator guides, and 12 reference desk pathfinders.14 As time goes by, more books, serials, and archival materials will be added. Presently, I am working on establishing at the library an archival finding guide and a Fedora OAI-compliant content management system to archive virtual world materials, both of which will be delegated to volunteers.

RSS feeds include: “NASA Breaking News,” “Current Space Shuttle Mission Updates,” and “Hubblecast.” Some of the Twitter links are grouped into the following categories: “Current Astronauts,” “Former Astronauts,” “International Astronauts,” NASA Centers,” and “CoLabLibrary.” With 215 members and 1,211 items, the most successful new Flickr group I started was called “Apollo 40th Anniversary Celebrations Around the World” 14 Most of the full-text books are made by the Library Assistant, Charlie Navarathna. Pathfinders include: “NASA Digital Collections,” “NASA Archives,” “NASA Technical Data,” “NASA Libraries,” “Tuskegee Airmen,” “NASA,” “NASA CoLab,” “Space Medicine,” “Women,” “SL Science Places and Events,” “Libraries, Archives, Museums,” and “Search SL.”

One Small Step for a Virtual World Library

Survey Results igure 11. For tech savvy library patrons, the learning curve is not excessive. Only 1 out of 36 respondents who said hey made the transition from web-based reference service to virtual world reference service thought it was hard.

“I don't really see it as a transition. Virtual reference in Second Life and other ‘chat reference’ services are just other options for finding information in addition to "live" reference at a real world library reference desk,” one respondent stated. Those who presently use “Ask a Librarian” or similar service might be the group of individuals for targeted marketing of virtual world reference services. “Ask a Librarian” patrons are good candidates to lead into the virtual world, but because that is the case, they will compare the two services’ ease of use. One patron commented on the ability, once in a virtual world, to locate the library for the assistance they required, “This difficulty in way finding exists throughout Second Life (and I understand most existing virtual worlds), compared to the now standard ‘Ask a Librarian’ or simply ‘Ask’ buttons, on library web pages.” This suggests that libraries with virtual world libraries might add similar links or buttons leading to their virtual world presence to their websites adjacent to the web-based service, along with their staffed service hours.

One Small Step for a Virtual World Library

Comparison of Informal Attrition Rates for Web and Virtual World Library Patrons Number of Visits

Virtual

Web

World 1-4

5-8

9-12

13+

SUBTOTAL

N/A

TOTAL

Looking at the number of visits, it is possible to informally determine attrition rates. If a patron tried the service, that is participation. If they returned, that is retention. If they did not return, that is attrition. To do the calculation, I first, determined the number of respondents who used a virtual or web reference service one or more times by adding them to determine the total number of people who participated in each category. Second, I calculated the total number of web AND virtual world participants who asked reference questions (that is, excluding those replying “Not Applicable”, which is 73). Some overlap of web and virtual questions by the same patron occurred. Next, I subtracted the number in each category from the total of 73 and that leaves how many respondents did not return for subsequent service. For “Virtual World” that was 38 and for “Web” it was 35. By dividing each of these by the participant total number of 73 and then multiplying by 100, the attrition rate was derived. Virtual World

20+1+3+11=35

73-35=38

38/73=.5205

.5205*100=52%

Web

15+8+1+14=38

73-38=35

35/73= .4795

.4795*100=48%

Figure 12. The reduced ratio of informal attrition rates, Virtual World to Web was about 13:12. So, patrons try the service and don’t find it terribly difficult to adjust, but will they be satisfied enough to return again and again? Virtual world reference services showed a very similar attrition rate as those seen for web-based reference services. Yet, because virtual world reference services are newer technologies with time to develop, improvements in virtual world delivery methodologies such as follow-up, group joining, automated delivery of announcements

One Small Step for a Virtual World Library

via subscriptions, expansion of service delivery audience, and improved and expanded reference services appear to hold the possibility to engage and retain more patrons than the web-based service model.

33 23

Figure

13.

The

traditional

Brick-and-Mortar,

In-Person

service

remains the most satisfying to patrons and should be emulated in virtual worlds.

delivery

model

One Small Step for a Virtual World Library

So, which model emerges as the most successful? The survey’s quantitative findings and 57 qualitative comments revealed a high level of satisfaction with certain aspects of the Brickand-Mortar, In-Person model which might be carried into the virtual world to improve service quality and retention rates. For example, the qualitative feedback comments suggested that virtual face-to-face scenarios should heighten and enhance discussion and personalization: “For my purposes, discussion is normally required, so in person with a librarian is great,” one patron said. “Virtual world reference was more direct with the person visible and doing things with people in front of me and the chat was more active, greater sense of presence,” commented another. Another respondent wrote, “In order to be more than just a ‘Google Helper’, virtual reference needs to make use of deep web resources.” Expanding services through greater access to informational resources from inside a virtual world such as the same subscription databases and publications offered at the "brick and mortar library" could improve retention. This could be achieved with links to subscription services from inside the virtual world, where a user ID and authentication number login is entered manually or automatically via a HUD or other device. Conclusion Highly successful models for library and archival reference services, like the Neil A. Armstrong Library and Archives, have been achieved in virtual worlds. As the internet permeates society, and the digital divide begins to close, people are becoming used to integrating technology into their daily lives. Some even enjoy the challenge of exploring new technologies. Of those who use the existing web services, like “Ask a Librarian,” they are finding they are able to transition successfully to virtual world reference services. Reaching new patrons and retaining existing patrons will be key to virtual world reference success. One of the major obstacles prevention initiation has been that many library computers (and patron home computers) lack the needed graphics cards required to run virtual worlds within the library or at home. The key to retention seems to be, as other LIS studies have indicated, patron satisfaction. Patron satisfaction is the primary objective regardless of format, for the successful reference interview. Many of the survey respondents indicated they had not yet tried virtual world reference service, but of those who had, virtual world reference service ranked high in patron satisfaction and low in patron dissatisfaction indicating it is a viable reference delivery service model that is ready to be implemented on a wider scale.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds Christopher A. Paul Seattle University, USA Abstract This essay addresses how rhetorical analysis can be applied to the study of virtual worlds. Rhetoric, which focuses on persuasion and ways of knowing, offers a strong, qualitative methodological approach for scholars engaged in virtual worlds research. A background in the history of rhetorical studies is complemented by analysis of how rhetoric has been applied to virtual worlds and game studies to develop a perspective for analysis of three kinds of texts found in the discourse of virtual worlds. These three types of texts: in-world play structures, or procedures; texts surrounding the worlds, or paratexts; and primary texts of virtual worlds, like narrative and images; provide rhetorical scholars plenty of surfaces to critique. Patch 3.3.0 of World of Warcraft is used as a case study to demonstrate how to engage in rhetorical criticism of virtual worlds and how the various texts are presented in practice. Keywords: rhetoric; process; procedural rhetoric; paratexts; text; World of Warcraft.

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 4!

Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds The primary purpose of this essay is to bring the tools of rhetorical analysis and extant bodies of research in virtual worlds and game studies into conversation. This means making a number of adjustments to existing literature, as most work in rhetorical studies is focused on analyzing public speech. Three key types of discourse within and surrounding virtual worlds offer texts rhetorical analysis is well suited to address. First, there is the actual in-world play that links to the processes of computer games and connects to Ian Bogost’s (2007) articulation of procedural rhetoric. Second, there are the texts surrounding worlds, like forums, blogs, and other extra-world discussion that exemplify what Gérard Genette (1997) deems as paratexts and are used by Mia Consalvo (2007) to examine cheating in games. Finally, there are the primary texts of virtual worlds, the images, words, and other elements that impact virtual worlds in dynamic ways, as in the case of Lisbeth Klastrup and Susana Tosca’s analysis of clothing in the World of Warcraft (WoW). Although rhetorical analysis can likely apply to phenomena in virtual worlds that fall outside of these three categories, I believe they are central to recognizing how discourse functions shape engagement in virtual worlds. Thus, they offer a strong framework for outlining ways in which scholars could pursue future projects that use elements of rhetorical criticism to analyze virtual worlds. Demonstrating the ways in which rhetorical analysis can come to bear on virtual worlds will take three steps. First, I will offer a review of relevant pieces of material in rhetorical studies, with a focus on defining what rhetoric is and developing it as a perspective for the analysis of virtual worlds. Second, I will engage in a discussion of relevant research in virtual worlds and game studies to demonstrate how scholars have pursued their projects. And third, I will use WoW patch 3.3.0, Fall of the Lich King, as a case study for the three kinds of texts found in virtual worlds. Rhetoric Rhetoric is a complicated, oft misunderstood subject that, when put in the title of a course, is almost guaranteed to scare students away. For the purposes of this essay, a brief historical overview will set the table to focus on contemporary rhetorical theory and how it can be used to analyze the study of virtual worlds. Rhetoric is an ancient term, dating back to Plato’s Gorgias, when it was roughly connected to the art of persuasion. The study of rhetoric was particularly important in ancient Greece, owing to the role of public speech in the governance of society. Although it was a contested term, with Plato positioning himself against the sophists - paid teachers of speech - the field of rhetoric developed standards and expectations for oratory. Aristotle added several categories for rhetoric, sorting oratory by approach and desired ends, uniting much of the work of Plato and the sophists.1 However, in contemporary times, rhetoric has taken on a negative connotation, often used to describe phenomena like talk without substance, “empty rhetoric,” or as part of a cover up of “real” facts, as when something is described as rhetorical posturing. In the popular vernacular, rhetoric is often symptomatic of empty promises, like the “mere rhetoric” of a politician’s campaign promises. It is within this backdrop, a history that stretches to classical times and contemporary misunderstanding of its subject matter, that the academic discipline of rhetoric is set. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! "

!#$%!&'!$()%(*)+!$,!-.&//*-&.!%0)1$%*-2!/))!34$5$/12!67789:!#$%!;)<10!*'!10)!0*/1$%=!$,!%0)1$%*-2!/))!34*>>)..!?! @)%>A)%52!"BB79:!#$%!*..C/1%&1*$'/!$'!%0)1$%*-&.!-%*1*-*/D2!/))!34C%5-0&%;12!"BBE9:!

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 5!

As could be expected of a discipline with this amount of history, there is difference of opinion about how to define the field. In the interests of an easily defensible starting point, Campbell and Huxman state “rhetoric is the study of what is persuasive” (2009, p. 5). In terms of the discipline, they argue that “the discipline of rhetoric examines the symbolic dimensions of human behavior in order to offer the most complete explanations of human influence… rhetoric is the study of the art of using symbols” (2009, p. 14). Rhetoric was initially used to understand and to further the study of oratory, and was tied to the analysis of what made particular appeals more or less effective. In contemporary times, many rhetorical scholars have moved beyond the study of speech, as speeches are less integral to contemporary culture than they were in ancient Greece and mediated communication plays an increasingly important role in how people are persuaded. As such, rhetoricians have begun to analyze elements of communication like images, online discussion, and virtual worlds. Twentieth-century rhetoric offers a number of modifications to classical rhetorical studies that are worthy of note in the journey to reaching Campbell and Huxman’s definition. Defined primarily in terms of “persuasive uses of language” (Campbell, 1970, p. 101) throughout its history, contemporary rhetorical studies are marked by a meta-discourse focused on redefining the role of the field to better address how symbol use fits within our lives. Much of this introspection was driven by the work of Kenneth Burke, who argued that language has a symbolic dimension and, as humans are defined as symbol-using animals, “the whole overall ‘picture’ [of reality] is but a construct of our symbol system” (1966, p. 5). For Burke, rhetoric is connected to language use and, for humans, language is “a symbolic means for inducing cooperation in beings that by nature respond to symbols” (1969, p. 43; emphasis in original). Burke adjusts the focus of rhetoric, often preferring to study its role in creating identification among people, rather than the persuasive force of symbol use. In doing so, he offers a means of analysis that could be quite useful to the analysis of virtual worlds because we can study elements of the “picture of reality” furthered by particular worlds and how the elements in those worlds facilitate identification. The study of language use is also addressed in the subfield of argumentation theory. Perelman and Olbrechts-Tyteca contend that “there is no neutral choice [in language]—but there is a choice that appears neutral… What term is neutral clearly depends on the environment” (1969, p. 149). This is a shift in looking at language, as it is the beginning of an approach where critics see the persuasive force in all things. Symbol and language use is deployed strategically to emphasize certain things and minimize others. Those following in the wake of Perelman and Olbrechts-Tyteca could study not only the clearly persuasive messages but also the subtly persuasive ones, as a truly neutral presentation, a language without rhetorical force, simply does not exist. Further, those languages that appear neutral may be even more interesting as objects of study, as we do not clearly see the power behind how those symbols are used. This broader notion of rhetoric is often connected to the work of R.L. Scott. For Scott, “rhetoric may be viewed not as a matter of giving effectiveness to truth but of creating truth” (1967, p. 13). Given this frame, rhetoric “is a way of knowing; it is epistemic” (Scott, 1967, p. 17). This idea has been expanded by other scholars, to the end that “one of the assumptions implicit in much of contemporary rhetorical theory is that there is no way to ground representations of reality (rhetoric) in a reality independent of discourse” (Cherwitz & Darwin, 1995, p. 192). Functionally, this means that rhetorical analysis shifted from identifying how to persuade to examining the strategic function of symbol use in a given context. In the wake of Scott’s work, “rhetoric is a unique cultural practice” that is predicated on “locating the substance

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 6!

of rhetorical knowledge in the creation of a situational truth” (Greene, 1998, para. 6). Scholarly focus shifts from “how one resides in a framework of meaning and interests” to “how one articulates and uses these” (Lyne, 1998, para. 14).2 The general perspective behind this way of looking at rhetoric is that “everything, or virtually everything, can be described as ‘rhetorical’” (Schiappa, 2001, p. 260). In the end, rhetoric changed. No longer relegated to advising on how to craft the best speeches or even necessarily placing the focus on persuasion, rhetorical analysis can be seen as a way of thinking about how knowledge is produced and deployed strategically. Rhetorical analysis is a tool that can be used to investigate how situational truths are constructed and, in turn, how those truths function. In addressing how rhetorical criticism works in the wake of this turn toward “big rhetoric”3, rhetoric can be seen more like a perspective than as a subject with a distinctive object of study, like speech. David Zarefsky writes that, as the number of “texts” rhetoricians have sought to analyze has expanded, the dynamic “in effect identifies rhetorical criticism with a mode or perspective of analysis, rather than with a distinctive critical object. Rhetorical critics bring to any object the focus of making arguments about how symbols influence people” (2008, p. 634). In the end, “rhetoric offers another perspective, one that accounts for the production, circulation, reception, and interpretation of messages” (Zarefsky, 2008, p. 635). Rhetorical analysis, particularly in the wake of the shift toward a broader understanding of text and performance, is well suited for analysis of virtual worlds. In the wake of “big rhetoric,” the tools of rhetorical analysis offer a perspective for scholars interested in studying how knowledge and situated truths are established in virtual worlds. Rhetoric can address the entire discursive environment of virtual worlds, as “virtually everything can be described as rhetorical.” This does not mean that all insights granted by rhetorical analysis of virtual worlds will necessarily result in papers worthy of publication, but it does mean that the many dynamics of virtual worlds can be commented upon by an understanding of how particular symbols have the power to persuade, shape meanings, and aid in the construction of our perspectives and beliefs. Rhetoric can look at the play in worlds and how the dynamics of design shape the ways in which a world is constructed. The paratexts are particularly appropriate for rhetorical analysis, from the blog posts that discuss a game to the impacts of the promotion of Second Life on its public image. Finally, the primary texts of the world, from the clothes to the lore, are all dynamics that help shape a particular way of viewing the world and ones that a rhetorician is well suited to critically analyze. Given this brief breakdown of key tenets in rhetorical studies, it is appropriate to turn to the second relevant body of literature that bears on how rhetoricians can engage virtual worlds, that of game studies.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 6

!F'!10*/!&%1*-.)!G=')!&./$!%&*/)/!10)!HC)/1*$'!10&1!/$,1+&%)!D&=!A)!)<*/1)D*-2!&/!I-$11!<$/1C.&1);!,$%!%0)1$%*-:!J0*/! D&=!A)!&!/*1)!$,!<$1)'1*&.!-$'')-1*$'!1$!4$5$/1K/!<%$-);C%&.!%0)1$%*-:! L J0)!0*/1$%=!$,!10*/!1C%'!1$+&%;!A*5!%0)1$%*-!-&'!A)!/))'!*'!A$10!I-0*&<<&!&';!3MC''2!677N9:!MC''!&./$!<%$(*;)/!&! %0)1$%*-&.!&'&.=/*/!$,!10)!;*/-*<.*')!&/!&!+0$.):!!

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 7!

Game Studies and Virtual Worlds Reaching beyond virtual worlds and looking to game studies for theoretical and critical analysis offers a platform upon which to refine and apply the tools of rhetoric to more completely illustrate how to analyze virtual worlds as rhetorical texts. There are four key pieces to reviewing this research. First, a general overview of methodology in game studies illustrates the lack of established methods in the field. Second, discussion of Ian Bogost’s procedural rhetoric will develop tools to analyze processes in virtual worlds. Third, I will outline existing research that uses elements of rhetoric to analyze either virtual worlds or games. Finally, I will address a handful of projects that could potentially be enhanced by the addition of a rhetorical perspective, yet do not necessarily claim it. As an emergent area of study, the analysis of virtual worlds and games do not yet have established methods upon which to rely in the criticism of texts. Further complicating things is that virtual worlds and games are more like collections of texts than a clear, stand alone discipline. Because of this, research generally borrows from existing methodologies in other disciplines, which are then applied to games or worlds. This dynamic is likely to change as more is published in the field, but there are several scholars who have addressed some of the ways in which games could be analyzed systematically. Jesper Juul analyzes the dynamics of computer games, arguing that the formal rule systems of games and programs make them fundamentally different than traditional narratives (2000). Interaction in a virtual world is subject to a certain set of rules that facilitate some interactions and prohibits others, whether that be Neal Stephenson’s Metaverse, where Hiro Protagonist is a preeminent sword fighter because he wrote the code (2000), or the inability to kill players of the same faction who are stealing your herbs in WoW. In discussing analysis of games, Espen Aarseth outlines three keys: study of game design, rules and mechanics; observation of others; and playing the game as part of research (2003). These criteria are quite broad, which makes them flexible enough for many different projects, including the study of virtual worlds. Consalvo and Dutton refine this approach, advocating four categories for qualitative analysis of games: Object Inventory, Interface Study, Interaction Map, and Gameplay Log (2006). Both of these articles are useful, but they are more aimed at identifying aspects of a world one needs to investigate than providing a clear way to engage in analysis. Further, while all mentioned elements deserve attention in the analysis of virtual worlds, the depth and detail of a world can be so much larger than an offline game that something like an object inventory may near impossibility. Perhaps the most appropriate guidance for approaching research comes in the form of these guidelines to which researchers should attend. Thomas Malaby and Timothy Burke add that “the empirical character of virtual worlds demands a multidisciplinary, methodologically polymorphous approach” (Malaby & Burke, 2009, p. 325) and Robert Brookey cautions that virtual worlds “require greater responsibility and renewed social concern” (2009, p. 103). Rhetoric offers both flexibility and critical force to attend to the ways in which texts are made to mean, while fulfilling the baseline expectations held by extant methodological guidelines. Ian Bogost’s conception of procedural rhetoric is the most established method of game studies analysis predicated on a rhetorical approach. Bogost defines procedural rhetoric as “a practice of using processes persuasively” (2007, p. 3). He argues that, in video games “the main representational mode is procedural, rather than verbal” (2006, p. 168), which necessitates developing a mode of procedural rhetoric to address texts that are not spoken or written in a conventional sense. Inspired by his descriptions of rhetoric, he contends that “Following the !

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 8!

classical model, procedural rhetoric entails persuasion—to change opinion or action. Following the contemporary model, procedural rhetoric entails expression—to convey ideas effectively” (2008, p. 125). To this end, a key take away from his argument is that “video games make claims about the world. But when they do so, they do it not with oral speech, nor in writing, nor even with images. Rather, video games make argument with processes” (2008, p. 125; emphasis in original). Procedural rhetoric seeks to analyze how games use processes to “dictate how actions can and cannot be carried out” (Bogost, 2007, p. 3) and how those processes persuade players. Although I think the notion of procedural rhetoric is quite useful in analyzing virtual worlds, I believe there are three key limitations in this approach, especially if one sought to make it the exclusive manner in which to use rhetoric to analyze games or virtual worlds. First, isolating procedures from other elements of discourse is problematic, specifically in the case of virtual worlds. In some ways, virtual worlds are richer than video games as they depend on a level of interaction with other people, which may happen outside of the processes of the game. Although something like the specific chat channels enabled or disabled by a virtual world could be considered an aspect of its procedural rhetoric, the discourse in those channels may be quite similar to traditional reading and writing. There are also dynamics inside and outside of the game that can shape how it is played, from Dragon Kill Points (DKP) (c.f. Silverman & Simon, 2009) to the clothes players wear (c.f. Klaustrup & Tosca, 2009) to theorycraft (c.f. Paul, 2009). Separately, one could read Bogost’s approach to procedural rhetoric as a paratext that reshapes how games are designed. Bogost presents a rearticulation of what games do, altering the structure of future processes. In effect, if his academic project is successful in changing how games are designed, his work is a demonstration of the rhetorical force of both paratexts and processes. Second, virtual worlds are iterative, malleable texts, rather than relatively stable, fixed ones. Although updates are relatively common for PC games and increasingly part of the console gaming experience, virtual worlds are almost defined by their patches, updates and expansions. As virtual worlds are in a state of flux the surrounding discourse can be quite influential in shaping the iterative construction of the procedures built into the game, creating an even stronger link between paratexts and process. Third, Bogost contends that “Video games have the power to make arguments, to persuade, to express ideas. But they do not do so inevitably” (2008, p. 137) and gives “the name persuasive games to videogames that mount procedural rhetorics effectively” (2006, p. 46; emphasis in original). This pair of statements is incompatible with the big rhetoric approach, which would hold that all games are persuasive and they are inevitably so. Whether or not those games and worlds are successfully persuasive or to what ends they persuade is secondary, but all games are persuasive, whether the persuasion is to buy the game, level in a certain way, follow a given narrative, buy extra goods in a “free-to-play” game, absorb the capitalist lessons of Animal Crossing (c.f. Bogost, 2008) or the political messages in Tax Invaders (c.f. Bogost, 2006). A handful of other scholars have also sought to employ the tools of rhetoric to analyze elements of games or virtual worlds. Gerald Voorhees argues that “the form of the player-game interaction has to be taken seriously if critics are to come to terms with the rhetorical force of Civilization,” (2009b, p. 256), as it is the player-game interaction that connects the game to the mind and actively ignores the constraints imposed by the body, internalizing the role of agent in the game. Voorhees also analyzes the changing character representations over the course of the Final Fantasy series, contending that “When every representation is in some way ideological it is not possible to speak about representation without also considering it rhetorical” (2009a, para. 9). Looking at online discussion boards, Moeller, Esplin and Conway contend it is the

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 9!

discussion boards where players “push the boundaries of what is considered ethical or sportsmanlike in a medium where testing the boundaries of an environment and the limits of the rules is encouraged and expected” (2009, para. 1). My work has also sought to apply elements of rhetorical analysis to virtual worlds, analyzing welfare epics and the reward system of WoW (Paul, 2010), how the development of a player originated notion of ‘theorycraft’ shapes play and design in WoW (Paul, 2009), and how Kenneth Burke’s concept of identification helps understand why guilds come together and fall apart (Paul & Philpott, 2009). The common thread through all of these projects is that they are interested in how certain elements of virtual worlds or games function to structure the way those texts are understood and encountered. All of these pieces can help scholars interested in virtual worlds take a deeper look at their research object of choice and articulate the ways in which virtual worlds persuade to a variety of different ends. There are a number of research projects studying elements of virtual worlds that offer potential links to a rhetorical perspective by critically analyzing discourse in a manner congruent with a rhetorical approach. The context for rhetorical analysis is largely set by understanding the vast number of interrelated forces that shape interactions in virtual worlds. In critiquing the concept of a magic circle, Mia Consalvo articulates what can be seen as a call for rhetorical analysis in the massively multiplayer online (MMO) game genre, stressing the importance of “the need to understand how players understand, contextualize, and challenge MMO games” (2009, p. 411). Consalvo goes on to explore how structures, real-life influences, and game play experience interact, stressing the importance of examining the interrelation of the various factors that shape play interactions in games and virtual worlds. Similarly, T.L. Taylor offers a list of interrelated factors that shape games and their play, including technological systems and software (including the imagined player embedded in them), the material world (including our bodies at the keyboard), the online space of the game (if any), game genre, and its histories, the social worlds that infuse the game and situate us outside of it, the emergent practices of communities, our interior lives, personal histories, and aesthetic experience, institutional structures that shape the game and our activities as players, legal structures, and indeed the broader culture around us with its conceptual frames and tropes (2009, p. 332). All of these dynamics offer surfaces on which rhetorical analysis could help to explain why or how key forces interact and the ways in which they remake or influence the virtual world. In doing this sort of analysis, rhetoric can help explicate the persuasive forces that help dictate how knowledge is created and how the terms are set for “what counts” in virtual worlds and games. Beyond the overarching context of worlds and games, several essays address specific instances of discourse where a rhetorical perspective might be useful. Silverman and Simon argue that DKP systems have a tangible impact on the way players interact in virtual worlds, arguing that “Players do not work together because the prospect of a reward gives them an incentive to do so. Rather, by playing a certain way (i.e., as a power gamer), they begin to perform a rational subjectivity that views the game in terms of incentives and rewards” (2009, p. 364). One could argue that DKP functions symbolically as a player generated rhetorical message that reshapes the way those participating in DKP systems see the game world, redefining the relative “truths” in the game. Certain games put players into situations where they must allocate scarce resources and the systems they derive function rhetorically to persuade players to alter the ways in which they play. Adam Ruch performs a critical reading of WoW’s End User License Agreement and Terms of Use documents to construct an argument about how Blizzard presents itself in relation to the players of the game. In doing so, WoW is constructed as a service within

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 10!

which items are accessed, rather than obtained. Ruch’s analysis is insightful and largely based on a legal reading of both documents, but a rhetorical reading may offer other insights. Lisa Nakamura argues that “Player-produced machinima accessed from warcraftmovies.com make arguments about race, labor, and the racialization of space” (2009, p. 135). In this case Nakamura argues that machinima, which is discourse from outside the game, shapes the ways in which players conceive of representation and behavior within the game. Nakamura’s work expands the kinds of texts that may be of interest to scholars analyzing virtual worlds, demonstrating how a paratext can function to influence interaction in online worlds. These three essays look at different aspects of online worlds, yet the common thread among them is that the texts they study shape the ways in which virtual worlds are encountered and could be complemented by a rhetorical perspective. There are a variety of ways for rhetorical analysis to aid in the analysis of virtual worlds. With all the surfaces, interactions and topics in these online environments, there is plenty of room for analysis of the ways in which persuasion occurs and how knowledge claims are shaped. Given the overview of what rhetoric is and pertinent research in extant analyses of virtual worlds and online games, the best way to give an example of some of the ways in which rhetoric provides a useful perspective for the analysis of virtual worlds is to look at a case study, WoW patch 3.3.0. Fall of the Lich King, Rise Of Rhetoric: WoW Patch 3.3.0 Patch 3.3.0 introduced a number of changes to WoW (WoWWiki, 2009b), some of which were regular bug fixes and several of which fundamentally alter the ways in which WoW works. This patch has been hailed as “maybe even the best [patch] Blizzard ever did” (Tobold, 2009b) and prompted the Best Patch Ever edition of a weekly feature on a prominent WoW-focused blog (Whitcomb, 2009). In addition to the generally positive response to the patch by commentators, the changes made by the patch illustrate each of the three types of discourse in virtual worlds: A reconstruction of the procedurality of WoW; paratexts which reshape the context for both the game, the patch, and the newly enabled procedures; and alterations to the core texts of the game. To show the greatest range of appeals for rhetoric I will give an overview of the changes in the patch, briefly discuss the ways in which a rhetorical approach could aid in analysis of each of the three kinds of discourse, and provide an extended example of how rhetorical analysis is particularly well suited to analyze interactions between process and paratext. The title of the patch, Fall of the Lich King, refers to a key piece of Warcraft lore that was central to the most recent expansion of the game, the opportunity for players to fight the most prominent enemy in Wrath of the Lich King, Arthas Menethil. 3.3.0 introduced Icecrown Citadel, which contains three new five person instances and a raid instance. The Ashen Verdict were added to the game, as a new faction dedicated to fighting Arthas. The patch made numerous adjustments to each class in the game and fixed several bugs. There were also ways in which core aspects of the game were altered, including the dynamics of leveling early in the game (Schramm, 2009b), the debut of a raid browser that makes it easier for players to find other players with whom to raid (Reece, 2009a), a new default user interface for quest tracking that is designed to make questing easier (Sacco, 2009a), and perhaps the most profound change, a new dungeon finder system (Reece, 2009b). The dungeon finder is supported by a “luck of the draw” buff, which makes groups more powerful when using the system to meet new people, encourage players to use the system to find pick up groups (PUGs) (Sacco, 2009b). In part, virtual worlds are interesting because they are not stable texts that are predefined and fixed; changes and

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 11!

updates are part of what makes them particularly important objects of study and patch 3.3.0 shows how a world can be altered in a single patch. The processes of WoW, the procedural dynamics of the game, were changed in many ways. The most straightforward, banal aspects of the changes were the class and bug fixes. Although some of them may close loopholes that players found a way to exploit, they do not typically reshape the procedural rhetoric in a substantive manner. However, the other changes to process, the dungeon finder, leveling and quest user interface changes, have a significant impact on how the game will be played. The dungeon finder facilitates temporary connections with new people, displacing the need to find and sustain connections on your own home server, as it enables cross-server matching to widen the pool of players from which the matchmaking system can draw. Leveling and quest changes make the game easier, especially for new players. In the case of the questing interface changes, Blizzard has integrated partial elements of player designed modifications into the default user experience to make it easier to move about the game world and complete quests.4 This may also displace the market for the external modifications, as they may become less relevant because Blizzard has filled the perceived need for information about how to complete quests. Although not all of these changes are necessarily worthy of a standalone essay about how they impact the procedural rhetoric of WoW, all do show how virtual worlds are malleable, which means the processes within them, thus the procedural rhetoric, is dynamic. As WoW has a lively, large community of players, any large-scale change in the game world creates a ripple effect throughout the paratexts surrounding WoW. Patches are an occasion for commentary and discussion about what works well and potential negative effects of what was done. The announcement of the patch created a flurry of analytical posts speculating on what the changes will mean for WoW and reflecting on the actual impacts of those changes. Paratexts ranged from topics like how to best maximize the opportunity to make more gold in the game (Carbon, 2009; Crashumbc, 2009) to documentation of various changes that were not mentioned in the patch notes (Kazanir, 2009) and ruminations on what the changes will mean for each of the classes in the game (c.f. WoW.com, 2009). All of these paratexts, and the many more that were spurred by the patch “work to shape the gameplay experience in particular ways” and demonstrate how “the creating of a flourishing paratexts has significantly shaped games and gamers in the process of creating new markets” (Consalvo, 2007, p. 9). Specific paratexts, especially those that focus on the debut of the dungeon finder are especially relevant in how they reshape what constitutes the procedural rhetoric of WoW. Generally these paratexts recontextualize WoW, constructing it as a ‘talk-about-able’ object, shifting the process of “playing” WoW from being contained within the frame of the game itself to extending into a discussion of how it is designed, how it works, and how changes will impact the ways in which different communities of players approach the game. In these cases, paratexts expand WoW and the discourse that could be analyzed by rhetoricians. Many of these texts may be interesting or informative, but certain strands could be used by a rhetorician to articulate how the discourse of WoW reconstructs the ways in which ‘proper’ knowledge is formed within the paratexts surrounding WoW. Although the textual changes in patch 3.3.0 were substantial and advance the narrative story surrounding WoW (c.f. Holisky, 2009), I am unable to come up with ways in which !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! O

!#$%!D$%)!&A$C1!0$+!C/)%!D$;*,*-&1*$'/!-0&'5)!10)!<.&=*'5!)P<)%*)'-)2!<.)&/)!/))!3Q&%;*2!67"79!&';!3J&=.$%2! 677R9:!

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 12!

analyzing the narrative alone would produce a sound, interesting rhetorical analysis worthy of a paper. This could be in symptomatic of a general inability to find strictly narratological approaches to game studies (c.f. Frasca, 2003), but in this case the narrative does not really alter the fundamental story of WoW. Certainly, there are new characters introduced and a new big bad to vanquish, but this chapter is not substantively different than any other in WoWâ&#x20AC;&#x2122;s story and has strong parallels to many other tales. The primary textual development in 3.3.0 extend the game and keep the story moving, rather than redefining how the story is structured. The stronger analyses of primary texts in virtual worlds may be less likely to rely exclusively on the stories and more likely to focus on other elements of the text, like the content of quest dialogue or ingame images, including gear and character design. However, those examples seem less likely to be fundamentally altered by a single patch, and are instead shaped over the design and run of the game. The relationship between procedure and paratexts are where I see the most utility for rhetorical analysis in critiquing how the intervention of patch 3.3.0 alters structures of knowledge in WoW. A particularly appropriate surface for this analysis is the new dungeon finder system, which is at the center of the praise for the patch and stands to alter the ways in which players come together in an allegedly massively multiplayer game. This combination of a change in the way WoW works, or process, and the reaction from players, or paratexts, can be addressed in three phases. First, it is necessary to provide a brief background of the history of grouping systems in WoW and an outline of the new system, dungeon finder. Second, I will outline some of the commentary about the dungeon finder, with a focus on how it alters social relationships in the world. Third, I will discuss how a rhetorical perspective offers a point of view that can articulate the magnitude of the change the dungeon finder could invoke on the world of WoW. Groups in WoW consist of five players who can join together to enter into dungeons, or instances, to fight powerful monsters and obtain various rewards. Typically, these groups contain a healer, a tank, and three damage dealers. In the launch version of the game there was no built in system by which players could meet others interested in running dungeons, beyond their guild or friends list. This put a particularly heavy emphasis on joining a guild and fostering relationships with other people in order to put oneself in a position to construct groups and reap the rewards of successfully completing dungeons. Since the launch of the game, Blizzard tried multiple iterations of systems to make finding other people with whom to work easier, although the interventions were plagued with problems or underuse.5 Groups found through these systems, PUGs, were derided by many as subpar, inferior, and risky. The ideal way to find a group was to gather people one already knew; these groups, premades, were seen as far more attractive by most WoW players, although they required being in either a prominent, active guild or having a lengthy list of friends who played a wide range of classes. The dungeon finder makes forming a PUG easy and smooth, quickly matching players from multiple realms together in groups. Although it does not ensure a successful run, it provides players with a quick way to find other people with whom to group. One WoW designer has gone so far to say that the dungeon finder is the coolest feature integrated into WoW (Gunslinger, 2009). There are several themes in the reaction to the dungeon finder, beyond a general appreciation for how much easier it makes finding a group to run dungeons. By making meeting !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! E

!S!10$%$C50!A%)&T;$+'!$,!10)!0*/1$%=!$,!5%$C<UD&T*'5!/=/1)D/!*/!A)=$';!10)!/-$<)!$,!10*/!<&<)%!;C)!1$!/<&-)2! AC1!&'!$()%(*)+!-&'!A)!,$C';!&1!3V$VV*T*2!677B&9:!

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 13!

random people easier, Blizzard has redefined how to play part of the game, which necessarily has collateral impacts on relationships within it. One observer praises the system, stating that it “finally managed to get rid of any need for friends to do dungeons” (Tipa, 2009), while another worries that the patch may threaten “the already weak social fabric of the game” (Syncaine, 2009). The change may also alter the skills of players, as the system could lead to a situation where “in a few months, there are going to be some sick skilled tanks available. Imagine leveling to max purely by running dungeons with PUGs. Those tanks who come out of that are going to be really good” (Rohan, 2009). It will also likely alter the ways in which reputation works within the game, as grouping in WoW becomes increasingly anonymous. Mike Schramm ponders that “Gone may be the days when you build up a good reputation by saying ‘remember me if you need a good DPS [damage dealer] at the end of a run. It’ll be interesting to see what methods we replace that with” (2009a). The lack of an ability to build up a positive reputation is also complicated by the ease of having to live with a negative one, with some hypothesizing that the system has led to more negative behavior, like ninja looting (Tobold, 2009a). Rhetorical analysis is useful to virtual worlds scholars because it can help to combine these two discrete kinds of discourse to develop a critical point of view with which to analyze elements of what makes procedures and paratexts meaningful. In this case, rhetoric proffers a position from which one can identify how the dungeon finder system fundamentally alters the ways in which grouping in WoW works, which aids in splicing the paratexts surrounding the implementation of patch 3.3.0. It also provides a point of view from which to analyze how those processes and paratexts construct persuasive appeals to shape the construction of knowledge in virtual worlds. In this case, it is likely that the dungeon finder system will resignify connotations of PUGs, encourage a different distribution of player classes in the game, as tanks and healers generally experience a shorter wait for instances than damage dealers, and relationships among players will be fundamentally altered. It is likely that the meanings of guild will change, particularly if the system is expanded with similar success to raiding encounters.6 The addition of the ‘luck of the draw’ buff and rewards for using the system are both introduced as new procedures that reshape how players perceive grouping with those they do not know. Working with random people is easier if everyone is made more powerful through a new systematic buff and there is a reason to use the system when given specific incentives for doing so. In the case of the dungeon finder, procedure and paratexts come together to reshape concepts fundamental to life in WoW’s virtual world, altering notions about how a group should be gathered, what classes to play, how to conceive of reputation in the game, and why one needs/wants to join a guild. Both procedure and paratexts function rhetorically, persuading players and arranging the terrain on which knowledge is built. Analyzing design and changes in process, as well as paratextual interventions, helps to explain how things work in the virtual worlds in which we participate and why certain interventions in them are particularly meaningful. Conclusion Rhetorical analysis offers virtual worlds a perspective for analysis of discourse, especially the procedural, paratextual, and textual discourse that typify virtual worlds. The tools of rhetoric help analyze how things work, what they do, and how these kinds of texts interact with each other to shape the context of virtual worlds. Rhetorical analysis has evolved to focus on how various appeals function to persuade, to construct knowledge, and shape how we perceive the !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! R

!J0)%)!+&/!&./$!&!')+!.$$T*'5!,$%!%&*;!*'1)%,&-)!&;;);!+*10!10)!<&1-0!3-:,:!W))/)2!677B&9:! 13

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 14!

world around us. Porting these tools to virtual worlds with care can give added perspective on the various dynamics that make worlds work and why we find them interesting and meaningful. The purpose of this piece is to provide a starting point for understanding how integrating rhetorical analysis into the methodological toolbox for analysis of virtual worlds can aid in research. As such, this essay is best read as a launch point, a beginning guide. Hopefully, this will contribute to an increasing application of a rhetorical approach into the analysis of virtual worlds that will continue to chart how to best engage in rhetorical criticism of processes, paratexts, and texts.

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 15!

References S&%/)102!X:!3677L9:!!"#$%&'()*+*#,-./(0*1.232"2'%-#"(#44,2#-.*+(12('#5*(#&#"$+%+:!Y&<)%!<%)/)'1);!&1! 10)!ZS[!-$',)%)'-)2!\).A$C%')2!SC/1%&.*&:!! 4*>>)..2!Y:2!?!@)%>A)%52!4:!3"BB79:!6.*().*12,%-#"(6,#3%1%2&/()*#3%&'+(7,25(8"#++%-#"(6%5*+(12(1.*(!,*+*&1:! Q)+!]$%T2!Q]^!4);,$%;!4$$T/!$,!I1:!\&%1*'_/!Y%)//:! 4$5$/12!F:!3677R9:!`*;)$5&D)/!&';!F;)$.$5*-&.!#%&D)/:!!249"#,(82559&%-#1%2&:(;3L92!"REU"NL:!! 4$5$/12! F:! 367789:! !*,+9#+%<*( =#5*+/( 6.*( >?4,*++%<*( !2@*,( 27( A%3*2'#5*+:! [&DA%*;5)2! \S^! J0)! \FJ! Y%)//:! 4$5$/12! F:! 3677N9:! J0)! W0)1$%*-! $,! `*;)$! M&D)/:! F'! a:! I&.)'! 3X;:92! 6.*( >-2"2'$( 27( =#5*+/( 82&&*-1%&'( B291.:(=#5*+(#&3(C*#,&%&'!3<<:!""8U"LB9:![&DA%*;5)2!\S^!J0)!\FJ!Y%)//:! 4%$$T)=2! W:! S:! 3677B9:! Y&%&;*/)! [%&/0);^! W)10*'T*'5! \\bWYM/! &';! b10)%! `*%1C&.! V$%.;/:! ! S'! F'1%$;C-1*$':!8,%1%-#"(D193%*+(%&(0*3%#(82559&%-#1%2&:(EF3692!"7"U"7L:!! 4C%5-0&%;12![:!W:!3"BBE9:!)*#3%&'+(%&().*12,%-#"(8,%1%-%+5:!I1&1)![$..)5)2!YS^!I1%&1&!YCA.*/0*'52!F'-:! 4C%T)2! a:! 3"BRR9:! C#&'9#'*( #+( D$5G2"%-( H-1%2&/( >++#$+( 2&( C%7*:( C%1*,#19,*( #&3( 0*1.23:! 4)%T).)=2! [S^! c'*()%/*1=!$,![&.*,$%'*&!Y%)//:! 4C%T)2!a:!3"BRB9:!H().*12,%-(27(021%<*+:!4)%T).)=2![S^!c'*()%/*1=!$,![&.*,$%'*&!Y%)//:! [&D<A)..2!a:!a:!3"B879:!J0)!b'1$.$5*-&.!#$C';&1*$'/!$,!W0)1$%*-&.!J0)$%=:!!.%"2+24.$(#&3().*12,%-:(I3692! B8U"7N:!! [&D<A)..2! a:! a:2! ?! @CPD&'2! I:! I:! 3677B9:! 6.*( ).*12,%-#"( H-1/( 6.%&J%&':( D4*#J%&'( #&3( K,%1%&'( 8,%1%-#""$:! 4).D$'12![S^!V&;/+$%10![)'5&5)!G)&%'*'5:! [&%A$':! 3677B2! E! b-1$A)%9:! L:L:7! Y&1-0! Q$1)/! ! W)1%*)();! "8! Z)-)DA)%! 677B2! ,%$D! 011<^dd67T.)().*'5:-$Dde\J[,$%CDd(*)+1$<*-:<0<f,g6B?1g"8R8! [0)%+*1>2! W:! S:2! ?! Z&%+*'2! J:! e:! 3"BBE9:! V0=! J0)! _X<*/1)D*-_! F'! X<*/1)D*-! W0)1$%*-f! ! J0)! Y&%&;$P! b,! W0)1$%*-!S/!Y)%,$%D&'-):!6*?1(#&3(!*,72,5#&-*(L9#,1*,"$:(MN2!"NBU67E:!! [$'/&.($2!\:!367789:!8.*#1*,+/(=#%&%&'(H3<#&1#'*(%&(A%3*2'#5*+:![&DA%*;5)2!\S^!J0)!\FJ!Y%)//:! [$'/&.($2!\:!3677B9:!J0)%)!*/!Q$!\&5*-![*%-.):!=#5*+(#&3(89"19,*:(;3O92!O7NUO"8:!! [$'/&.($2! \:2! ?! ZC11$'2! Q:! 3677R9:! M&D)! S'&.=/*/^! Z)().$<*'5! &! \)10$;$.$5*-&.! J$$.T*1! ,$%! 10)! hC&.*1&1*()!I1C;=!$,!M&D)/:!=#5*(D193%*+:(F3"9:!! [%&/0CDA-:! 3677B2! "8! Z)-)DA)%9:! L:L! 0*';/*501UU+0&1! ;$! =$C! +*/0! =$C! ;*;! ;*,,)%)'1.=f! ! W)1%*)();! "8! Z)-)DA)%2!677B2!,%$D!011<^dd67T.)().*'5:-$Dde\J[,$%CDd(*)+1$<*-:<0<f,g6B?1g6BOE! #%&/-&2! M:! 3677L9:! C932"2'%+1+( "2<*( +12,%*+:( 122/( &21*+( 7,25( #( 3*G#1*( 1.#1( &*<*,( 122J( 4"#-*:! Y&<)%! <%)/)'1);!&1!10)!G)().!c<![$',)%)'-)!$,!10)!Z*5*1&.!M&D)/!W)/)&%-0!S//$-*&1*$'2!c'*()%/*1=!$,! c1%)1-012!Q)10)%.&';/:!! M)')11)2!M:!3"BB89:!!#,#1*?1+/(1.,*+.2"3+(27(%&1*,4,*1#1%2&:![&DA%*;5)^![&DA%*;5)!c'*()%/*1=!Y%)//:! M%))')2!W:!3"BBN9:!J0)!S)/10)1*-!JC%'!&';!10)!W0)1$%*-&.!Y)%/<)-1*()!$'!S%5CD)'1&1*$':!H,'95*&1#1%2&( O(H3<2-#-$:(IN3"92!"B:!! MC''2! e:! 3677N9:! I*>)! \&11)%/^! Y$.=1$'*'5! W0)1$%*-_/! Y)%()%/)! S<$-&.=</):! ).*12,%-( D2-%*1$( L9#,1*,"$:( IP3"92!N6U"7N:!! MC'/.*'5)%2! `:! 3677B2! "L! Z)-)DA)%9:! XP-.C/*()! F'1)%(*)+! +*10! V$%.;! $,! V&%-%&,1! G)&;UZ)/*5')%! J$D! [0*.1$':! =#52&#Q3*! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+$+:5&D$'&:;)dD);*&d*'1)%(*)+/d)P-.C/*()U*'1)%(*)+U+*10U+$%.;U$,U+&%-%&,1U.)&;U ;)/*5')%U1$DU-0*.1$'di! @$.*/T=2!S:!3677B2!N!Z)-)DA)%9:!J0)!G$%)!$,!Y&1-0!L:L:!K2KQ-25!!W)1%*)();!"8!Z)-)DA)%2!677B2!,%$D! 011<^dd+++:+$+:-$Dd677Bd"6d7Nd10)U.$%)U$,U<&1-0ULULd! eCC.2!e:!367779:!K.#1(825491*,(=#5*+(8#&(#&3(8#&R1(S2:!Y&<)%!<%)/)'1);!&1!10)!Z*5*1&.!S%1/!&';![C.1C%)! [$',)%)'-)2!4)%5)':!!

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 16!

a&>&'*%:! 3677B9:! Y&1-0! L:LUUY&1-0! Q$1)/! &';! c';$-CD)'1);! [0&'5)/:! >"%1%+1( T*,J+! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd).*1*/1j)%T/:-$Dd,"Ed1NLONLU <&1-0kLkLk<&1-0k'$1)/kC';$-CD)'1);k-0&'5)/d! a.&C/1%C<2! G:2! ?! J$/-&2! I:! 3677B9:! l4)-&C/)! *1! eC/1! G$$T/! [$$.ml! #&/0*$'! &/! [0&%&-1)%! Y)%,$%D&'-)^! J0)! [&/)!$,!V$V:!T29,&#"(27(A%,19#"(K2,"3+()*+*#,-.:(M3L92!OU"8:!! G=')2! e:! 3"BBN9:! a'$+.);5)! &';! Y)%,$%D&'-)! *'! S%5CD)'1^! Z*/-*<.*'&%*1=! &';! Y%$1$UJ0)$%=:! H,'95*&1#1%2&(O(H3<2-#-$:(IN3"92!L:!! \&.&A=2! J:! \:2! ?! 4C%T)2! J:! 3677B9:! J0)! I0$%1! &';! @&<<=! G*,)! $,! F'1)%;*/-*<.*'&%*1=! *'! M&D)! I1C;*)/:! =#5*+(#&3(89"19,*:(;3O92!L6LULL7:!! \$)..)%2! W:! \:2! X/<.*'2! 4:2! ?! [$'+&=2! I:! 3677B9:! [0))/)%/2! YC..)%/2! &';! M.*1-0)%/^! J0)! W0)1$%*-! $,! I<$%1/D&'/0*<!&';!10)!Z*/-$C%/)!$,!b'.*')!I<$%1/!M&D)%/:!=#5*(D193%*+:(U369:!! Q&%;*2! 4:! 367"79:! 0$( C%7*( #+( #( V%'.1( >"7( !,%*+1/( H&( H&1.,242"2'%-#"( H--29&1( 27( K2,"3( 27( K#,-,#71:! S''! S%A$%2!\F^!c'*()%/*1=!$,!\*-0*5&'!Y%)//:! Y&C.2![:!S:!3677B9:!6.*2,$-,#71/(H(8,%1%-#"(S%+-29,+*(W*$2&3(1.*(=#5*:!Y&<)%!<%)/)'1);!&1!10)!S//$-*&1*$'! $,!F'1)%')1!W)/)&%-0)%/![$',)%)'-)2!\*.+&CT)):!! Y&C.2! [:! S:! 367"79:! V).,&%)! X<*-/f^! J0)! W0)1$%*-! $,! W)+&%;/! *'! V$%.;! $,! V&%-%&,1:! =#5*+( #&3( 89"19,*:( N3"9:!! Y&C.2![:!S:2!?!Y0*.<$112!e:!3677B9:! 6.*()%+*(#&3(X#""(27(86D/(Y*&&*1.(W9,J*(Z3*&1%7$%&'(@%1.(1.*(K2,"3(27( K#,-,#71Q! Y&<)%! <%)/)'1);! &1! 10)! Z*5*1&.! M&D)/! W)/)&%-0! S//$-*&1*$'! [$',)%)'-)2! 4%C').! c'*()%/*1=:! Y)%).D&'2! [:2! ?! b.A%)-01/UJ=1)-&2! G:! 3"BRB9:! 6.*( V*@( ).*12,%-/( H( 6,*#1%+*( 2&( H,'95*&1#1%2&:! Q$1%)! Z&D)2!FQ^!c'*()%/*1=!$,!Q$1%)!Z&D)!Y%)//:! W))-)2! M:! 3677B&2! L7! b-1$A)%9:! Y&1-0! L:L! YJW^! S! .$$T! &1! 10)! W&*;! 4%$+/)%:! K2KQ-25! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$+:-$Dd677Bd"7dL7d<&1-0ULULU<1%U&U.$$TU&1U10)U%&*;U A%$+/)%d! W))-)2!M:!3677BA2!6"!b-1$A)%9:!Y&1-0!L:L!YJW^!Q)+!G#M!*'1)%,&-):!K2KQ-25!!W)1%*)();!"8!Z)-)DA)%2! 677B2!,%$D!011<^dd+++:+$+:-$Dd677Bd"7d6"d<&1-0ULULU<1%U')+U.,5U*'1)%,&-)d! W$0&':! 3677B2! "7! Z)-)DA)%9:! Y&1-0! L:L! #*%/1! FD<%)//*$'/:! W"*++%&'( 27( Y%&'+! ! W)1%*)();! "8! Z)-)DA)%2! 677B2!,%$D!011<^ddA.)//*'5$,T*'5/:A.$5/<$1:-$Dd677Bd"6d<&1-0ULLU,*%/1U*D<%)//*$'/:01D.! WC-02!S:!3677B9:!V$%.;!$,!V&%-%&,1^!I)%(*-)!$%!I<&-)f!=#5*(D193%*+:(U369:!! I&--$2! \:! 3677B&2! "O! Q$()DA)%9:! Y&1-0! L:L^! 4.*>>&%;! <%)(*)+! ')+! HC)/1! 1%&-T*'5! cF:! K2KQ-25!! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$+:-$Dd677Bd""d"Od<&1-0ULULUA.*>>&%;U <%)(*)+/U')+UHC)/1U1%&-T*'5UC*d! I&--$2!\:!3677BA2!8!Q$()DA)%9:!Y&1-0!L:L^!GC-T!$,!10)!Z%&+!AC,,/!=$C!,$%!<C55*'5:!K2KQ-25!!W)1%*)();! "8! Z)-)D)A)%2! 677B2! ,%$D! 011<^dd+++:+$+:-$Dd677Bd""d78d<&1-0ULULU.C-TU$,U10)U;%&+U AC,,/U=$CU,$%U<C55*'5d! I-0*&<<&2! X:! 3677"9:! I)-$';! J0$C501/! $'! 10)! [%*1*HC)/! $,! 4*5! W0)1$%*-:! !.%"2+24.$( #&3( ).*12,%-:( I;3L92! 6R7U68O:!! I-0%&DD2! \:! 3677B&2! ""! Z)-)DA)%9:! @$+! %)<C1&1*$'! 5$()%'/! 10)! 5&D):! K2KQ-25! ! W)1%*)();! "8! Z)-)DA)%2!677B2!,%$D!011<^dd+++:+$+:-$Dd677Bd"6d""d0$+U%)<C1&1*$'U5$()%'/U10)U5&D)d! I-0%&DD2!\:!3677BA2!"!b-1$A)%9:!Y&1-0!L:L!YJW^!X&/*'5!10)!-C%()!,$%!A)5*'')%/:!K2KQ-25!!W)1%*)();!"8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$+:-$Dd677Bd"7d7"d<&1-0ULULU<1%U)&/*'5U10)U-C%()U,$%U A)5*'')%/d! I-$112!W:!G:!3"BR89:!b'!`*)+*'5!W0)1$%*-!&/!X<*/1)D*-:!8*&1,#"(D1#1*+(D4**-.(T29,&#":(MP2!BU"8:!! I*.()%D&'2!\:2!?!I*D$'2!4:!3677B9:!Z*/-*<.*')!&';!Z%&5$'!a*..!Y$*'1/!*'!10)!b'.*')!Y$+)%!M&D):!=#5*+( #&3(89"19,*:(;3O92!LELUL8N:!! I1)<0)'/$'2!Q:!367779:!D&2@(8,#+.:!V)/1D*'/1)%2!\Z^!4&'1&D!Z)..!YCA.*/0*'5!M%$C<:!

Journal of Virtual Worlds Research – Process, Paratexts, and Texts: Rhetorical Analysis and Virtual Worlds 17!

I='-&*'):! 3677B2! "E! Z)-)DA)%9:! S! A*1! $,! V$V! 0*/1$%=! 1$! )P<.&*'! 10)! G#M! -0&'5):! [#,3-2,*( 8#+9#"/( W,%&'%&'( G#-J( 1.*( -#,*G*#,( +1#,*! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd/='-&*'):+$%;<%)//:-$Dd677Bd"6d"Ed&UA*1U$,U+$+U0*/1$%=U1$U)P<.&*'U10)U.,5U-0&'5)d! J&=.$%2!J:!G:!3677R9:!Z$)/!V$V![0&'5)!X()%=10*'5f!!@$+!&!Y(Y!I)%()%2!\C.1*'&1*$'&.!Y.&=)%!4&/)2!&';! IC%()*..&'-)!\$;!I-)')![&C/);!\)!Y&C/):!=#5*+(#&3(89"19,*:(M3O92!L"NULL8:!! J&=.$%2!J:!G:!3677B9:!J0)!S//)DA.&5)!$,!Y.&=:!=#5*+(#&3(89"19,*:(;3O92!LL"ULLB:!! J*<&:! 3677B2! ""! Z)-)DA)%9:! V$%.;! $,! V&%-%&,1^! W&';$D! ZC'5)$'/! #JV:! K*+1( Y#,#&#! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+)/1T&%&'&:-$Dd*';)P:<0<d677Bd"6d""d+$%.;U$,U+&%-%&,1U %&';$DU;C'5)$'/U,1+d! J$A$.;:!3677B&2!"L!Z)-)DA)%9:!ZC'5)$'!#*';)%!&';!'*'j&U.$$1*'5:!62G2"3R+(00\)!=(W"2'!!W)1%*)();!"8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd1$A$.;/:A.$5/<$1:-$Dd677Bd"6d;C'5)$'U,*';)%U&';U'*'j&U .$$1*'5:01D.! J$A$.;:!3677BA2!""!Z)-)DA)%9:!W&'1*'5!&A$C1!<&1-0!L:L!10%)&;:! 62G2"3R+(00\)!=(W"2'!!W)1%*)();!"8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd1$A$.;/:A.$5/<$1:-$Dd677Bd"6d;C'5)$'U,*';)%U&';U'*'j&U .$$1*'5:01D.! `$$%0))/2! M:! S:! 3677B&9:! J0)! [0&%&-1)%! $,! Z*,,)%)'-)^! Y%$-);C%&.*1=2! W0)1$%*-2! &';! W$.)<.&=*'5! M&D)/:! =#5*(D193%*+:(U369:!! `$$%0))/2!M:!S:!3677BA9:!F!Y.&=!J0)%),$%)!F!SD^!I*;!\)*)%_/![*(*.*>&1*$'2!JC%'U4&/);!I1%&1)5=!M&D)/!&';! 10)![$5*1$:!=#5*+(#&3(89"19,*:(;3L92!6EOU68E:!! V0*1-$DA2! Z:! 3677B2! "E! Z)-)DA)%9:! JC)/;&=! \$%'*'5! Y$/1^! 4)/1! Y&1-0! X()%! X;*1*$':! K2KQ-25!! W)1%*)();!"8!Z)-)DA)%2!677B2!,%$D!011<^dd+++:+$+:-$Dd677Bd"6d"Ed1C)/;&=UD$%'*'5U<$/1U A)/1U<&1-0U)()%U);*1*$'d! V$V:-$D:! 3677B9:! V$V:-$D_/! MC*;)! 1$! Y&1-0! L:L:! K2KQ-25! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$+:-$Dd5C*;)U1$U<&1-0ULUL! V$VV*T*:! 3677B&9:! ZC'5)$'! #*';)%:! @%J%#( '#5%&'! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$++*T*:-$DdZC'5)$'k#*';)%! V$VV*T*:! 3677BA9:! Y&1-0! L:L:7:! @%J%#( '#5%&'! ! W)1%*)();! "8! Z)-)DA)%2! 677B2! ,%$D! 011<^dd+++:+$++*T*:-$DdY&1-0kL:L:7! n&%),/T=2!Z:!3677N9:!a'$+.);5)![.&*D/!*'!W0)1$%*-&.![%*1*-*/D:!T29,&#"(27(82559&%-#1%2&:(NP2!R6BURO7:!! !

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 Conducting Empirical Research in Virtual Worlds: Experiences from two projects in Second Life Shailey Minocha Centre for Research in Computing, The Open University, UK Minh Quang Tran Centre for Research in Computing, The Open University, UK Ahmad John Reeves Centre for Research in Computing, The Open University, UK Abstract At The Open University in the UK, we have conducted empirical qualitative research in Second Life, a virtual world. The first project is in the education domain in which we are investigating the designs of learning spaces in Second Life, while the second project is related to shopping and consumer behavior of users in Second Life. In this paper, we discuss our experiences of conducting empirical research in virtual worlds with a specific focus on the following aspects: ethical norms, real-world and virtual identities, privacy of the participants, communication modalities (voice, text and use of gestures), logistics of conducting user-based studies, and skills and training needs of researchers. Although our experiences are based on conducting empirical research in Second Life, we hope that our experiences and discussions in this paper will also be useful for researchers who aim to conduct research in avatar-based virtual worlds other than Second Life. Keywords: 3D learning spaces, communication modalities, consumer behavior, empirical research, ethical guidelines for 3D research, Second Life, three-dimensional learning spaces, virtual identity, virtual worlds, virtual world research.

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

Conducting Empirical Research in Virtual Worlds: Experiences from two projects in Second Life Introduction Three-dimensional virtual worlds, also called synthetic worlds, are multimedia, simulated environments, often managed over the Web, which users can “inhabit” and interact via their graphical self-representations known as “avatars.” In a virtual world, the users, represented as avatars, experience others as being present in the same environment, or “being there together” even though they are geographically distributed. Users converse in real time through gestures, audio, text-based chat and instant messaging communication (e.g. Meadows, 2008). These virtual worlds support communication and collaboration more effectively than two-dimensional web-based environments by extending the user’s ability to employ traditional communication cues of face-to-face interactions such as gestures, voice, having visual presence and mutual awareness in real time, and sounds, in a way that twodimensional (web-based) environments do not (Bronack, et al., 2008; Eschenbrenner, et al., 2008). Today virtual worlds are being used in many applications: education and training, social networking, entertainment, gaming, marketing, and commerce. There are several virtual worlds platforms such as Active Worlds, OLIVE, Protosphere, Second Life and Web Alive (Wilson, 2009). In this paper, our focus is on Second Life1. Users connect to the Second Life environment with a software program, called a client or viewer, which is responsible for displaying the virtual world and for negotiating user commands with a central server. The client displays the user’s avatar and surrounding environment consisting of other avatars, landscape, buildings, etc. Unlike Massively multiplayer online role-playing games, such as World of Warcraft that have a scripted plot or storyline, Second Life is not a “game” per se. The content and narrative in Second Life is constructed and owned by the residents (users) of Second Life, and Linden Labs, the company that created Second Life, provides the infrastructure, hardware, and software to support Second Life. At The Open University in the United Kingdom, we have been involved in carrying out empirical qualitative research within Second Life and on two Second Life projects. The first project is in the education domain in which we have been investigating the designs of learning spaces in Second Life, while the second project is related to shopping and consumer behavior in Second Life. The participants in the first educational project have been Second Life educators, designers and students. We have conducted inworld interviews, tours of Second Life islands followed by panel discussions, observations, and group interviews or focus groups with the participants. Our aim has been to elicit their experiences and perceptions of learning space designs in Second Life, and to find out which key characteristics of learning space designs were important to them; whether or how does the realism or non-realism of learning spaces influences student learning and engagement? The recruitment of the participants has been through calls in Second Life educational groups, a real-world survey, and also through contacting colleagues who are on the Second Life -related mailing lists. The aim of the second research project is to understand consumers’ shopping experiences in Second Life, factors that influence consumer satisfaction, and how the affordances of virtual 1

http: //www.secondlife.com/

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

worlds facilitate shopping in Second Life. We have applied the techniques of descriptive phenomenology and semi-structured interviews to elicit the Second Life shopping experiences and perceptions of consumers. The recruitment of participants has been within Second Life by meeting customers in popular shopping areas in Second Life and asking if they would be willing to participate in our research. In this paper, we discuss our experiences of conducting empirical research in virtual worlds and focus on the following aspects of research design and implementation: • • • • • • •

•

How to develop research materials such as the consent form, project summary sheet, and so on, to address the possible concerns of an institution’s ethics committee who may not be familiar with avatar-based interactions in virtual worlds. How the ethical guidelines of conducting research both in offline settings and on the Internet (or cyberspace) need to be considered and combined in the preparation of research materials for conducting research in virtual worlds. Identity rights of the avatar and how they are same or different from the identity rights of the person behind the avatar. How to combine complementary research techniques for the research questions, and how the techniques need to be adapted for conducting research in virtual worlds. Issues related to inworld recruitment of participants. Interviewing and focus group techniques in virtual worlds. Logistics of conducting empirical research in virtual worlds such as voice versus textbased data collection, developing a researcher identity or a consistent persona, choosing the locations for conducting interviews or focus groups, how to facilitate and manage the multiple channels of communication such as text (local chat and instant messaging), gestures and voice. Skills and training needs of researchers for conducting empirical research in virtual worlds.

Although our experiences are based on conducting empirical research in Second Life, we hope that the discussions in this paper will also be useful for researchers who aim to conduct research in avatar-based virtual worlds other than Second Life. In the next section, we present the challenges faced by researchers of social media and emerging virtual worlds, and we set out the motivation and focus of this paper. We then introduce two projects. Using a framework of a research process, we discuss the issues of conducting empirical research in Second Life through examples from these projects. Challenges for Researchers Investigating Virtual Worlds In the last decade, several technological changes have taken place such as easy access to computing power, the Internet becoming an integral part of people’s lives, mobile Internet becoming a reality, and the rise of social media. Social networking and social media are making significant inroads into education, marketing, and business practices. Educational institutions are adopting social software such as wikis, blogs, social bookmarking platforms and virtual worlds for facilitating socialization, collaborative learning and training. Companies are adopting social media (e.g. Twitter, discussion forums) as a tool to engage with consumers and provide consumer service (Grensing-Pophal, 2009). Virtual worlds are being employed for holding meetings and conferences, training (e.g. simulating crime scenes for law enforcement training), therapy (e.g. community-based therapy for military veterans),

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

and for demonstrating prototypes through simulations to get early stakeholder feedback before real-life production of products and services2. Investigating how people communicate, coordinate, and collaborate in these social networking environments and virtual worlds is important for the advancement of our understanding of human behavior and also for developing design principles which will enhance the capability of these spaces to support social and business needs. It is important to determine whether these emerging technologies are indeed meeting the intended social, business, or educational benefits. For example, social media and virtual worlds are facilitating “meaningful conversations” between businesses and their customers, but businesses are feeling loss of control and the risks of being transparent, and are keen to know about the competitive advantage of adopting these technologies in their business processes. Researchers of social media, online games and virtual worlds face a number of challenges. The characteristics of these technologies are in a constant state of flux, new social technologies and communities are emerging, and user behaviors are diverse and so are their online identities as they adopt different technologies over time. Although there has been a lot of discussion about conducting research in online spaces (e.g. Hine, 2005) and in gaming environments (e.g. Yee, 2008), there are few seminal sources to refer to about conducting research in non-gaming virtual worlds. In this paper, we will share our experiences from two Second Life research projects in which the empirical work has also been conducted within Second Life or inworld. The challenges of conducting research on Second Life and within Second Life are distinct from carrying out research of two-dimensional virtual environments, such as discussion forums, in that the characteristics of virtual worlds differ from two-dimensional virtual environments. One key difference is that the user’s avatar and the avatar’s appearance are additional dimensions of the user’s identity in a virtual world. In a three-dimensional space, the avatar can navigate (fly, walk, sit, teleport), encounter other avatars, and communicate with other avatars through gestures, voice, text, and instant messaging. These communication and interaction mechanisms in virtual worlds create a sense of “being together” in a “place” with other avatars. This sense of presence and sense of place in a virtual world is more similar to face-to-face (offline) situations in the real-world rather than interactions in a twodimensional virtual environment such as Facebook3, or over video conferencing. For empirical research on virtual worlds, this real-world-like environment and a face-to-facelike setting enable a perception of the interviews being conducted across a table with other avatar(s), or as if the researcher has “lived” in a community, or as if the researcher has conducted participant-observations in the participants’ contexts. Thus an advantage for a researcher of conducting empirical studies in virtual worlds, especially if the experience occurred in the virtual world, is that the participant will be situated in or near the context where the original event(s) took place. These situated contextual investigations have the ethos of the contextual inquiry method in social sciences research. Contextual inquiry reveals what people actually do, why they do it that way, latent needs, and core values (Beyer and Holtzblatt, 1997; see also http: //incontextdesign.com/). The inquiry involves one-on-one field interviews conducted in the user’s work or life space and focuses on observing and talking with users about their ongoing activities. In a virtual world, the researcher can “visit” the users in the users’ spaces and conduct observations and interviews. Hence, the 2 3

For example, http: //tinyurl.com/ykzlfb4 http://www.facebook.com/

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

researcher’s understanding of issues and solutions is better grounded in actual events and contexts. The research process in a virtual world is, therefore, influenced by codes of practice, etiquette, logistics, and ethical guidelines of conducting research in real-world (offline) and online. For example, logistics of setting up an interview location in Second Life are very similar to what would be done in a face-to-face setting: a pleasant location with furniture, and undertaking measures to avoid interruptions. Since Second Life is an online space, the codes of practice and ethical guidelines for conducting online research can be followed or adapted for carrying out research in Second Life. There are several sets of ethical guidelines which are available for carrying out research in offline and online spaces: for example, the most widely used in the United Kingdom are from the following organizations which publish their guidelines in full on their websites: British Educational Research Association (BERA), British Sociological Association (BSA), and the British Psychological Society (BPS). The Association of Internet Researchers (AOIR, 2002) provides ethical guidelines for conducting online research. Further, the user’s identity in Second Life is the user’s avatar (e.g. name, appearance, profile, pictures). Hence, the codes of research conduct for anonymity and privacy of the participants in offline and online spaces are also applicable for the avatars. Education and Consumer Behavior Projects In the next section, we describe the two Second Life research projects. The projects are from two different domains: education and consumer behavior. In the educational research project, a part of the recruitment was through a web-based survey and e-mails. A pre-interview questionnaire also enquired about their real-life learning and teaching (the course, university, how Second Life is embedded within the curriculum, etc.). Since the research involved the participants sharing their personal experiences with teaching and learning in virtual worlds, some of the participants took us to their Second Life learning spaces. Consequently, in this research project, we did get to know about the participants’ identities both in real-life and/or their institutional affiliations and in Second Life. But, since the empirical research was conducted in Second Life, we referred to the participants by their avatar names in our correspondence and interactions with them. In the second project related to consumer behavior within Second Life, both the recruitment and the empirical research were within Second Life. Since the project does not focus on real- world shopping activities, we did not ask about the real-life names. However, we enquired about gender, age, occupation and country of the participants because we needed a sample with maximum variation. In other words, we wanted to make sure all our participants were not only females, or only from a certain age group, etc. We used their avatar’s profile information only to match it with our criteria during recruitment but other than that we have not used their real-life information, even if available in their profiles, as a part of our data analysis. Once they qualified as a participant, we were no longer concerned about their real-life identity. Project: Design of Learning Spaces in Second Life Research project: In this project we are investigating the relationship between the design of Second Life learning spaces and the design of learning activities (Minocha and Reeves, 2010b). We have elicited educators’, designers’ and students’ perceptions of learning spaces

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

within Second Life. The research outcomes in the form of design guidance and examples are aimed to support educators and designers in designing learning spaces that foster students’ socialization, informal learning, collaboration, and creativity. Research question: Our key research question has been: “How should three-dimensional learning spaces be designed for student engagement?” Data collection techniques: A web-based survey; focus groups and one-to-one interviews with the participants in Second Life, tours with Second Life educators followed by panel discussions. Participants: The participants have been students who are attending their course-related activities in Second Life, educators who are carrying out a part or all of their teaching in Second Life, and Second Life designers (or builders) of learning spaces. In our study there has been an overlap in these three categories of participants – as some of the educators also design and build learning spaces. Research materials: Consent form; project summary sheet that accompanies the consent form which gives details of the project, how the data will be collected and stored, who will have access to the data, how the data and the project’s outcomes will be disseminated, and contact details of the research team; pre-interview questionnaire; semi-structured and focus group interview templates; and questions for the organized tours with Second Life educators. We created three different sets of materials: for Second Life educators, designers and students. Experiences with the Ethics Committee: The Open University’s Human Participants and Materials Ethics Committee4 (HPMEC) reviews and approves the research before recruitment and data collection can commence. The HPMEC proforma requires information about aspects such as: schedule of the project, funding source, justification of research, ethical guidelines to be followed, location of data collection, target participants, recruitment procedures, methodology, how the consent will be obtained, whether the project has been registered under the Data Protection Act (in the United Kingdom), and how the data will be stored and disposed of to comply with the Data Protection Act, recompense to participants, details of withholding any information from the participants, risk involved for the participants and researchers, and how the participants will be debriefed. The various research materials are attached with this proforma and submitted to HPMEC. Our Second Life research project was the first Second Life or virtual world-related project at The Open University. Therefore, prior to submitting the proposal, one of the project team arranged a face-to-face meeting with a panel member of HPMEC. This meeting enabled us to introduce Second Life through pictures of Second Life, which we had taken with us to the meeting. We also explained the research project, how we were proposing to carry out the recruitment and the study, and the benefits of our research to The Open University. The panel member asked us about avatar-based interactions, communication modalities in Second Life, and how the avatars are identified in Second Life. The panel member suggested that we give details of the interaction and communication mechanisms in our application to HPMEC as other panel members may have similar queries while they are scrutinizing the application. Next, we submitted a detailed application to HPMEC along with all the research materials. 4

http://www.open.ac.uk/research/ethics/index.shtml

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

HPMEC noted that, as Second Life residents and researchers, we should follow the Second Life Community Standards5 and asked us to include this aspect in the research protocol. Recruitment of participants: We recruited Second Life educators and designers through a web-based survey (on Survey Monkey6 when, along with other questions related to our research, we asked them if they would like to participate in interviews. The Second Life educational tours conducted by Second Life colleagues (Esme Qunhua and Janita Collins) helped us to network. We also sent messages to the various Second Life-educational groups. One of our research team regularly attended educational events as a way to network with the Second Life community and these contacts also directly and indirectly led us to prospective participants. Project: Investigating Consumer Behavior in Second Life Research project: The objective of our study was to understand consumers’ experiences in virtual worlds. Businesses in Second Life use retail strategies and store designs similar to real-world stores, while utilizing the affordances of a virtual world such as avatar-based interactions. Understanding consumer activities in Second Life can provide insights on the future of commerce and marketing in three-dimensional virtual environments and the possible blending of two-dimensional and three-dimensional e-commerce environments. Research question: Our research question has been: “What are the perceived experiences of consumers in three-dimensional virtual environments?” Data collection techniques: The study has involved open-ended and semi-structured Second Life interviews. The phenomenological or open-ended interviews (Langdridge, 2007) require consumers to describe any shopping experience they had in Second Life. The open-ended question gave participants an opportunity to describe the experience in their own words without preconceptions introduced by the researcher. This was followed by semi-structured questions to ensure participants discuss specific aspects of their shopping encounter. Participants: The participants for this study were consumers who had had shopping experiences in Second Life. Research materials: These included flow diagrams of the methodology and the recruitment strategy for the research team’s use, a project summary sheet, consent form, and questions for the open-ended and semi-structured interviews. Experiences with the Ethics Committee: Similar to the process described in the previous project, the proforma and the research materials were submitted to HPMEC. The committee asked us to provide satisfactory responses to the following comments before they could approve the research: how will the privacy and dignity of the avatars during any undisclosed observations in public and private spaces in Second Life shopping areas be ensured; the need for adding information to the project summary sheet about how the data will be stored, privacy, confidentiality and anonymity protocol, the conditions for publications of images or raw data, and to provide an additional contact name.

5 6

http: //secondlife.com/corporate/cs.php http://www.surveymonkey.com/

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

We assured the committee that, during observations in Second Life shopping areas, we will not be “shadowing” the shoppers without permission, recording the text chats, or taking pictures of the shopping areas that include avatars that are not participants in our research. We revised the project summary sheet (which accompanies the consent form) to include information about how the data will be stored, where it be stored, who would have access to it, and how the data, when disseminated, will not be directly attributed to the names of avatars. We also added a second contact name to the project summary sheet so that the participants have an additional contact point if they want to speak to someone else (other than the main researcher) about the research. We revised the consent form to seek permission for the images to be taken during the interviews. The revised materials and response to HPMEC’s queries were approved by HPMEC and we could then proceed with the data collection. Recruitment of participants: Recruitment was done inworld by approaching participants in Second Life shopping malls or stores. We used the “showcase” feature within Second Life that provides a list of popular shopping areas. Blogs that discuss Second Life also provide leads to shopping areas in Second Life. We visited a number of shops so that we were not concentrating on one or two stores or a particular set of customers. Second Life has a feature where you can make notes about people on their profile tab. We have used this feature to “tag” avatars to ensure that we do not approach the same person twice. Once we had found some shoppers, the next step involved deciding whom to approach for an interview. The avatar’s profile gives useful information that can enable a match with the criteria for the participants. Sometimes, information in the profile such as “I am friendly,” “I will be happy to talk to you” encouraged us to approach them. So an avatar’s profile is a real help during recruitment and a luxury we, as researchers, have in Second Life that does not exist in real-life. The final step was to initiate contact with the prospective participant. We sent an instant message inworld with a short introduction to start a conversation. Starting with a request or long introduction may not get a positive response. We asked the potential participant about being interviewed immediately or at another time. We mentioned how long the interview would take. In most cases, the interviews took place there and then after meeting them. We tended not to offer to take them to other locations for interviews and talked to them in the location where we met them. Qualitative Research in Second Life Based on our experiences in the two projects, this section describes guiding principles for conducting empirical research in virtual worlds. The principles are not meant to be prescriptive guidelines, but these are our reflections. For purposes of heuristic convenience, we use the framework similar to the one proposed by Knobel (2003) to describe the research process shown in Figure 1. Pre-study considerations • Participating in the community • Research design Before data collection • Creating a researcher’s identity • Preparing the research materials

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

• Applying for ethical approval Data collection • Recruiting participants • Seeking consent from the participants • Pre-interview preparations including choosing a location for the interview • Choosing a communication modality (voice vs. text communications) • Conducting interviews After data collection • Storage of the data • Formatting and anonymizing the data • Dissemination of the research outcomes Figure 1: Research process. Pre-study Considerations Participating in the community: The researcher should be a part of the online community for an extended period of time before formal data collection (Hine, 2005). It does not mean they should participate in all the activities, but they should be aware of what is going on in a general sense. Knobel (1993) states: The easy access to online communities afforded by the internet makes it tempting to practice hit-and-run research, where the researcher spends a few days or even a few hours observing the interactions of online participants in a given community, then writes about them as though everything to be known about the community…this kind of snatch-and-grab approach usually provokes scathing comments from the community members being studied (p. 192). We spent about eight months attending educational events and courses in Second Life before embarking on the project. Similarly, a member of the team (who later interviewed the shoppers) spent several evenings in shopping malls within Second Life. Participating in communities helped us in various ways: our skills of interacting with the Second Life interface improved; we discovered new places in Second Life; our understanding of the communities was enhanced. But being a part of the community means that the researcher has to be reflexive and should continuously question one’s biases during the course of research (Boyd, 2008). It is important to work in a team so that there are others who can also question the biases; this improves the validity of the results. Research design: A well-designed study (Knobel, 2003) should be grounded in a meaningful problem and should be framed by a well-formed and manageable research question and a workable theory or set of theories; it should involve a careful selection of data collection and data analysis tools and techniques that will produce the kinds of data and outcomes required to address the research questions. A well-planned study should also indicate the time the participants will need to commit to, whether the data collection will occur in two or three stages (e.g. an email interview followed by an interview in Second Life), and so on. A poorly planned study will be perceived as ad hoc by participants and may even undermine the confidence and trust in the researcher. Before Data Collection Creating a researcher’s identity: So as to maintain the participants’ confidence in the project and trust in the researcher, it is important that the researcher develops a researcher’s

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

identity. Hine (2000) calls it self-presentation in virtual ethnography contexts. It is important that the researcher completes the profile information of their avatar, i.e. giving some information of real-life (affiliation with their institution), details of the research project, and web links to the project. This information helps the prospective participants find out more about the researcher and the project before they consent. In Second Life, marketing companies and consumer research consultancies are also active. Therefore, in studies related to consumer behavior (as in our second project), it is useful for researchers to identify themselves as academic researchers as the prospective participants may be more sympathetic and not expect or demand incentives or monetary compensation for participation. It is important to maintain a consistent online persona throughout the study. In Second Life, this implies using the same avatar name throughout the study (Boellstorff, 2008). Second Life offers users several options for customizing an avatar: choosing the gender for a human avatar, or being a non-human avatar, and varying the size and shape, facial features, hairstyles, etc. An avatar’s customization from the default appearance is considered as an indication of the user’s expertise in Second Life or the time that they have spent in Second Life and that may influence their credibility as an Second Life researcher (Malaby, 2008; see also http: //tinyurl.com/ybtwhck in which the author discusses how, once the technological and Second Life-competence barriers are overcome, the user begins to feel empathy or care for their avatar and the avatar develops a recognizable identity in its own right). Researchers of games are known to “level up” their avatars before conducting interviews with the players, particularly in game-based virtual worlds (Brignall and Van Valley, 2007). In Second Life, the equivalent of “leveling up,” would be to customize the avatar suitably to demonstrate their Second Life expertise and also commitment as a participant in the community. Some researchers take it a step further and customize themselves to fit in with the community they are researching. For example, a researcher, Backe, assumed the role of a Neko before conducting research on Nekos in Second Life; Neko is a human avatar that wears cat ears and a tail, and typically affects a cat-like attitude in Second Life or role-play as cats (Backe, 2009). Preparing the research materials: The research materials constitute of: the flow diagram of the methodology (stages in data collection and data analysis), consent form, project summary sheet, interview protocols, and any other data elicitation materials based on the techniques that are being employed in the research design. The flow diagram of the methodology can help guide the research process and could be a useful document for review by the Ethics Committee. The interview questions should be short so that they can step through one at a time to ease the interactions when using instant messaging or text chat in Second Life. Also, overly formal language and research jargon should be avoided as it might intimidate or alienate the participants. Applying for ethical approval: Each institution has its own procedures and codes of practice for conducting research. There are two key general Open University governance documents that describe: (a) codes of practice for conducting research and (b) ethical principles for research involving human participants7. In the educational project, we followed the British Educational Research Association’s ethical guidelines for educational research8. In the consumer behavior project, we considered the guidelines of the British Psychological Society9. 7

http: //www.open.ac.uk/research/research-school/resources/policy-information-governance.php http: //www.bera.ac.uk/files/2008/09/ethica1.pdf 9 http: //www.bps.org.uk/the-society/code-of-conduct/code-of-conduct_home.cfm 8

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Conducting Empirical Research in Virtual Worlds

From our experiences on the two projects and other projects at The Open University, the following are some of the key aspects that the ethics committee are looking for: completeness of the project summary sheet so that the participants can give informed consent, participantâ&#x20AC;&#x2122;s right to withdraw; whether the incentives being offered can create a bias in the sampling or in participant responses; compliance with the legal requirements in relation to the storage and use of personal data. The ethics committee is concerned about eliciting emotions or altering the moods of participants. Ideally, a study should not make the participant feel any discomfort or anger that is more than what happens in their everyday interactions. Being completely honest about the study and giving an outline of how the interview will be carried out can avoid this. In Second Life, it is difficult to judge emotions, therefore, the best practice would be to use polite gestures and avoid using words that may be interpreted wrongly, and be very careful if humor is used. Data Collection Recruiting participants: Once a targeted set of participants has been defined, as discussed in the project-descriptions, there are various ways to send calls for participation such as inworld options (notice boards, group messages, instant messages to individual avatars) and realworld options (such as emails on mailing lists, web-based surveys, personal requests as in our educational project). When interacting with prospective participants, it can be reassuring for them to hear that the ethics committee of the institution has approved the research. The most direct way to recruit inworld is to talk to the users directly, as described in our consumer behavior project. If there is a specific subgroup of participants that is being targeted, for example consumers, artists, or educators, then the researcher should go to specific places and events where they are more likely to be present, such as malls, concerts or online campuses. When the researcher has found a place where users visit frequently, then it would be useful to plan some criteria to guide the selection of participants. Seeking consent from the participants: Before collecting data, acquiring informed and written consent from participants in which the participants understand and agree to their participation without any duress is a condition for research involving human participants. The difference between acquiring consent in person and through virtual worlds is the ability to get a handwritten signature. In our projects, we provide the participant the project summary sheet and consent form as note-cards inworld. If the participant acknowledges (through an instant message) that they understand and agree to participation, then that is taken as their consent. Pre-interview preparations including choosing a location for the interview: In the educational project, the interviews were pre-planned. As a preparation for the interviews and if the participants had not participated in the web-based survey, we asked them to fill in a preinterview questionnaire. This questionnaire captures the background information about the person; how long they have been in Second Life; avatar name, and some specific questions related to our project. We also ask the how they would like to be interviewed (Skype10/phone/inworld). We have interviewed 36 participants over the last year and we have had only two requests for phone interviews, the rest have all preferred inworld interviews. In our pre-interview questionnaire, we also check whether the participant has preference for audio or text. 10

http://www.skype.com/

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

There are other preparations before the planned interview session such as: visiting the interview location; testing the audio; checking that the seating arrangement works as most avatars prefer to sit during the interview; checking that the default seating postures on the seats are not too relaxing and match the context of the interview. A café area works well as it is semi-informal and friendly; but a setting that is too informal or too formal does not work very well. Some Second Life spaces have lot of background noise such as water flowing or wind blowing. It is useful to avoid such places; also choose locations that are not too busy so that you are not interrupted during the interview session. The land can also be made private, so that there is no public access, which will reduce the number of possible interruptions that could occur during the interview. It is useful to send an SLURL of the location to the participant where you are planning to meet. However, it is good to check the preference of the participant about where they would like to be interviewed. In our educational project, some participants offered to meet us on their islands so that they could take us for a tour of their learning spaces or islands during or after the interviews (designs of learning spaces was the focus of our project). In the shopping project, the interviews have been conducted through instant messages and in the shopping locations inworld. Choosing or arranging a location was not an issue as the interviews were conducted wherever we met the consumers. We have noted that some participants may not want you to be a “friend” or “contact” in Second Life. Therefore, we do not offer participants the opportunity to be contacts unless they ask. Choosing a communication modality (voice vs. text communications): In Second Life, there are ethical concerns about using voice and text. While voice can bring in the emotions of the speaker/participant, create the sense of co-presence, and may also help to establish rapport with the researcher, some participants prefer to use text. Voice projects information such as a user’s gender, ethnicity and age, thereby interfering with the anonymity which many users value in virtual worlds. These virtual worlds, with the emphasis on fantasy, offer users the possibility of experimentation with role(s) different from their real-life identity (Turkle, 1995; Ducheneaut et al., 2009). Users may, therefore, be concerned about revealing their real life identity to other users. Using voice reveals part of the user’s real-life identity that is at odds with the “fictional presence” they would like to maintain (Wadley et al., 2009). In our consumer behavior project we used inworld instant messaging, while in the educational project we provided the option to use voice as well, depending upon the participant’s preference. If the participant had the option of either audio or text (as in our educational project where we knew both their Second Life and real-life identities), it was useful to have the initial introductions using voice and then move to the text for the interview. The text provides a ready-made transcript for analysis. There is a concern that, for effective interviews, it is useful to develop a rapport with the participants (Seidman, 1998) and that instant messaging or text chat can make developing this rapport difficult (Voida et al., 2004). This concern could be less of a problem in a virtual world because the visual presence of the avatars may help to develop the initial understanding required.

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

The instant messaging window in Second Life provides privacy in text and voice communications between the participant and researcher, otherwise others near the interview location could pick up audio and text chat if it were in the local chat window. Through the preferences menu in Second Life viewer, it is useful to set up a Second Life folder for saving the chat or instant messages. Before the interview, it is helpful to check that the folder is actually recording the text. Conducting interviews: During a real-world interview, the researcher can show interest in the participant’s responses, usually by nodding, eye contact and other body language. Although Second Life has a set of gestures, we have found text more helpful in our conversations as gestures can get missed if the participant is not paying attention to the avatar. It is easier for the researcher to send prompts with simple statements such as “tell me more about [...].”, or type in phrases such as “fantastic,” “thank you,” “that was interesting,” “great,” and so on, at appropriate time-intervals during the interview to show our appreciation. We keep a text editor open during the interview so that when we are listening to or reading the participant’s answers, we can write any additional questions in the text editor and copy and paste these questions in the instant messaging window or ask the questions at the right time. The instant messaging window in Second Life provides an indication of awareness through prompts (e.g. <avatar name> is typing…) that the other user is writing in the instant messaging window. This helps the researcher to wait and pace the questions accordingly. Physical discomfort is also a risk if the interview is taking longer than expected, such as fatigue and eye strain for the participants. There should be opportunities for the participant to take breaks during the interview, and the researcher should not exceed the agreed length of the session. In any case the interview should not be more than an hour long – this is a realworld guideline but it is also useful in Second Life. In a Second Life focus group, the researcher take turns in asking questions from the participants, just as they would conduct focus groups in the real-world. If a researcher wants to take snapshots or screenshots (images which involve the participant), it is courteous to ask the participant for permission, even if a prior consent has been taken via the consent form. After Data Collection Storage of the data: There are practical, ethical and legal implications. The practical implication relates to how easily the data can be accessed for analysis. Ethical implications relate to the protection of a participant’s identity. A clear protocol for handling data gives participants peace of mind that their personal information will be handled securely and for its intended use. Unethical handling of data can also have legal consequences. Planning how data will be stored can also help later in the research process when it is time to do the analysis. In Second Life, logs of text conversations can be saved automatically on the researcher’s computer. Researchers should move these files to a secure/encrypted folder after the interview and delete the remaining unnecessary logs. In our projects, we have used encrypted USB drives (e.g. http: //www.ironkey.com) for movement of data between machines. We also create encrypted folders on the hard disk using the software http:

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

//www.truecrypt.org . We also store our data on the specific project spaces on the university servers (as backups). Formatting and anonymizing the data: Once the Second Life interview is over, it is important to remove the avatar name from the transcript (log) and to anonymize it suitably. In Second Life, the participant’s avatar name is an identifier and, therefore, the norms of anonymizing participant’s real names in real-life are the same for protecting their avatar names. It is useful to maintain a file with the avatar names and the corresponding identifiers (or pseudonyms) and to save this file in a secure place (e.g. in an encrypted folder on the computer or in a secure server) so that the participant can be easily traced back and contacted if a query arises during data analysis. Audio recordings should also be stored in secure folders. Dissemination of the research outcomes: Researchers are duty bound to represent the study participants fairly, respectfully and with dignity; for example, any transcript used in any reports or papers describing the research should have any grammatical or spelling mistakes corrected (Knobel, 2003). Other considerations Researcher skills and training: For conducting research in Second Life, the researcher should be skilled in using the Second Life interface: for example, navigation, communication (instant messaging, text chat, audio, through note-cards), giving and receiving objects inworld, and camera control. Spending some time on a regular basis in Second Life, and in the community which is the focus of the research, helps to develop Second Life skills as well as getting acquainted with the social norms, etiquette and rules of the community. Furthermore, the researcher should be familiar with the ethical guidelines of conducting online research. A number of resources for ethical guidelines have been suggested in this paper. If the researcher plans to conduct tours or group interviews, it would be useful to learn about setting up a group in Second Life and the communication protocol within an Second Life group. Some of the participants in a focus group or tour may not be familiar with the group interactions, so it would be helpful to ask the participants about their experience and set up individual or group training sessions ahead of the actual session. For a tour or focus group, it would be useful to have a facilitator, so that the researcher can ask the questions and run the session, while the facilitator can help with any technical problems that individual participants may experience during the session. The consent form and the project summary sheet should mention that presence of the facilitator in the session along with the main researcher. A virtual worlds researcher requires the skills and training of conducting both offline and online research since a three-dimensional space of a virtual world encompasses the characteristics of face-to-face or real-world interactions and yet the researcher may not have any offline or real-world interactions with the participant. Therefore, interviewing skills, skills of facilitating focus groups and conducting tours in real-world will help with research in virtual worlds. Similarly, developing skills of reading and analyzing online conversations in discussion groups and social networking sites such as Facebook and Twitter will also contribute towards a virtual worlds researcher’s skills and development. It is also useful to keep a diary/log of one’s experiences as a researcher for self-reflection and for sharing the experiences with other researchers.

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

Adhering to Linden Labs’ Community Standards of Practice: Community standards are regulations that govern the social behavior of Second Life users (http: //secondlife.com/corporate/cs.php). There are two of the six standards that researchers should be specifically aware of and which are related to (i) privacy, disclosure and anonymity of the avatars; and (ii) disturbing the peace. (i) There is an option for a user to disclose details of their real-life in their avatar’s profile. A Second Life user does not know when someone views their profile and there is no option to block others from viewing the profile. If real-life information is required for research purposes, we recommend researchers request a participant’s real-life information, even if it is available in their profile because the participant may not want to associate their real-life details with the research project in question. (ii) There are two sub-processes during research where there is a risk of disturbing the peace: advertising and recruitment. The researcher must be selective about where and how often they advertise and approach users. Researchers should be aware that advertising on someone’s privately owned land in Second Life, or during events could be seen as disturbing the peace. Therefore, advertising should be confined to public spaces, unless given permission beforehand, and, even when advertising is in a public space, researchers should self regulate. In our shopping study, we looked for participants in Second Life stores by directly approaching customers while they were shopping. To minimize our disruptive behavior, we did not visit the same store more than once a week, and we did not stay in the same store for more than one interview at a time. Recruitment was done via instant messaging, rather than broadcasting in the store’s local chat. It will be up to the discretion of the researcher how they perceive disruptive behavior. Conducting pilot studies: Evaluating the research materials with two or three colleagues is helpful to iron out problems such as ambiguity and completeness of the interview questions. Running pilots also helps to check whether the time it takes for, say, an interview does not exceed the planned time. Prioritization of questions, checking for repetitions, etc. can help to bring the interview protocols within the desired timings. During our pilots in the educational project, a colleague discussed usability of Second Life learning spaces and their impact on student engagement, an aspect which was an unanticipated thread of enquiry. Her inputs led us to change our interview materials and to develop an additional strand for data collection and analysis (Minocha and Reeves, 2010a). In the pilots of both projects, we went through the entire process of data collection, analysis, and interpretation to check whether our research questions were being answered. Pilots are time consuming but they provide a sense of confidence for the main study. Using inworld data collection tools: We have not covered the asynchronous inworld data collection tools in this paper. There are inworld survey tools, questionnaire kiosks, and so on, which can be used to collect data (e.g. see https: //www.xstreetsl.com/ or the Second Life Marketplace https://marketplace.secondlife.com/?lang=en-US which has a number of tools). Quantitative data on an avatar’s origin, behavior, and frequency of visits can be obtained via tracking systems available from inworld vendors. There are also spying devices which can be used to report conversations and record visitors’ movements even when the researcher is not

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Conducting Empirical Research in Virtual Worlds

present inworld and on site. It is, however, very important to emphasize that using such spying devices might infringe Community Standards of practice and should be used with caution and only if there is a research need and after telling the visitors that such devices will be used. It is important for researchers to be aware that such tracking tools and spying devices exist so that they can take some measures to counteract these devices. For example, in order to protect the anonymity of the participant and for the security of the data, it is important that audio and text conversations take place within the instant messaging window and not in the local chat window so as to prevent the conversations being picked up by any passers-by or spying devices. Combining offline and online interactions: Virtual worlds, like other online spaces such as newsgroups and discussion forums, allow users to represent themselves and socially interact in ways that are quite different from their offline personas (Turkle, 1995). The characters that users adopt in virtual environments are â&#x20AC;&#x153;self-madeâ&#x20AC;? people (Reid, 1995) that enable them to try out new identities or to express facets of their personality suppressed in offline life. Some researchers (e.g. Turkle, 1995; Orgad, 2005) have suggested triangulation of methods that combine offline and online interactions with the participants to increase the validity and for adding authenticity to the findings obtained in online spaces. However, we argue, similar to Orgad (2005) and Hine (2000) that conducting offline interactions with participants should not be driven by the assumption that the offline interactions would reveal more authentic or more accurate information than that generated by online interactions. Rather the rationale for combining offline and online interactions with participants should be grounded in the research context and goals. If the aim is to conduct research about phenomena in the threedimensional spaces in their own right, as in our two projects, then the question of offline interactions with the participants does not arise (Hine, 2000). Conclusion The guiding principles presented in this paper are based on our experiences of conducting qualitative empirical research, primarily through interviews and focus groups in Second Life. However, some of the principles relating to research design, ethics and maintaining anonymity and privacy of the participating avatars will also be useful for researchers who are applying quantitative research techniques in Second Life, and to researchers who are aiming to conduct research in virtual worlds other than Second Life.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Conducting Empirical Research in Virtual Worlds

References AOIR. (2002). Ethical decision-making and Internet research: recommendations from the AOIR ethics working committee, Retrieved from: http: //www.aoir.org/reports/ethics.pdf Backe, M. (2009). Neko culture in Second Life. In Proceedings, Under the Mask: Perspectives on the Gamer conference, University of Bedfordshire, UK, June 5-6. Retrieved from: http: //underthemask.wikidot.com/papers-2009 Beyer, H. and Holtzblatt, K. (1997). Contextual design: Defining customer-centred systems. San Francisco, CA: Morgan Kaufmann. Boellstroff, T. (2008). Coming of age in Second Life: An anthropologist explores the virtually human, New Jersey: Princeton University Press. Boyd, d. (2008). How Can Qualitative Internet Researchers Define the Boundaries of Their Projects: A Response to Christine Hine. Internet Inquiry: Conversations About Method (Annette Markham and Nancy Baym, eds.). Los Angeles: Sage, 26-32. Brignall, T.W. and Van Valley, T. L. (2007). An online community as a new tribalism: The World of Warcraft. Proceedings of the 40th Hawaii International Conference on System Sciences (HICSS 2007). Bronack, S.C., Cheney, A.L., Riedl, R.E., and Tashner, J.H. (2008). Designing virtual worlds to facilitate meaningful communication: issues, considerations and lessons learned. Technical Communication, 55(3): 261-269. Ducheneaut, N., Wen, M-H, Yee, N. and Wadley, G. (2009). Body and Mind: A Study of Avatar Personalization in Three Virtual Worlds. Proceedings of 27th international conference in Human factors in computing systems (CHI 2009), New York: ACM Press, 1151-1160. Eschenbrenner, B., Nah, F. F-Hoon, and Siau, K. (2008). 3-D virtual worlds in education: applications, benefits, issues and opportunities. Journal of Database Management, 19(4): 91-110. Grensing-Pophal, L. (2009). Social Media helps out the help desk. EConent, Available from: http: //www.econtentmag.com/Articles/Editorial/Feature/Social-Media-Helps-Outthe-Help-Desk-58056.htm Hine, C. (2000). Virtual Ethnography. London: Sage Publications. Hine, C. (ed.) (2005). Virtual Methods: Issues in Social Research on the Internet. London: Berg. Knobel, M. (2003). Rants, ratings and representation: ethical issues in researching online social practices. Education, Communication and Information 3(2): 187-210.

Journal of Virtual Worlds Research – Conducting Empirical Research in Virtual Worlds

Langdridge, D. (2007). Phenomenological Psychology: Theory, Research and Method. England: Pearson Education Limited. Malaby, T. (2009). Making Virtual Worlds: Linden Lab and Second Life. Ithaca: Cornell University Press. Meadows, M. S. (2008). I, Avatar: The Culture and Consequence of Having a Second Life. Berkeley, CA: New Riders. Minocha, S. and Reeves, A. J. (2010a). Interaction design and usability of learning spaces in 3D multi-user virtual worlds. In: Katre, Dinesh; Orngreen, Rikke; Yammiyavar, Pradeep and Clemmensen, Torkil eds. Human Work Interaction Design: Usability in Social, Cultural and Organizational Contexts. IFIP Advances in Information and Communication Technology. Springer, 157-167. Minocha, S. and Reeves, A.J. (2010b). Design of Learning Spaces in 3D Virtual Worlds: An Empirical Investigation of Second Life, special issue on Learning and researching in virtual worlds of the journal Learning, Media and Technology, 35(2): 111-137. Orgad, S. (2005). From online to offline and back: Moving from online to offline relationships with research informants, Virtual Methods: Issues in Social Research on the Internet (C. Hine, ed.). London: Berg. Reid, E. (1995). Virtual worlds: Culture and imagination. Virtual culture: Identity and communication in cybersociety (S.G. Jones, ed.). London: Sage. 164-183. Seidman, I. (1998). Interviewing as qualitative research. New York: Teacher’s College Press. Turkle, S. (1995). Life on the screen: Identity in the age of the Internet. London: Weidenfeld and Nicolson. Voida, A., Mynatt, E. D., Erickson, T. and Kellogg, W. A. (2004). Interviewing over Instant Messaging. Proceedings of the Conference on Human Factors in computing systems, CHI’04 extended abstracts, New York: ACM Press, 1344-1347. Wadley, G. Gibbs, M. and Ducheneaut, N. (2009). You can be too rich: Mediated communication in a virtual world. Proceedings of the 21st Annual Computer-Human Interaction Special Interest Group (CHISIG) of the Human Factors and Ergonomics Society of Australia (OzChi 2009), New York: ACM Press. Wilson, N. (2009). Virtual Worlds for Business. A report from Clever Zebra, Sandhurst, Berkshire, UK (personal communication). Yee, N. (2008). Maps of Digital Desires: Exploring the Topography of Gender and Play in Online Games. Beyond Barbie and Mortal Kombat: New Perspectives on Gender and Gaming (Y. Kafai, C. Heeter, J. Denner and J. Sun, eds.) Cambridge, MA: MIT Press, 83-96.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Conducting Empirical Research in Virtual Worlds

Acknowledgements The research presented in this paper has been supported by the JISCâ&#x20AC;&#x2122;s Learning and Teaching Innovation Grant, Centre for Open Learning of Mathematics, Science, Computing and Technology, one of the Centres for Excellence in Teaching and Learning at The Open University, UK, and the Faculty of Mathematics, Computing and Technology at The Open University, UK. We would like to express sincere thanks to all the participants in our research. We are grateful to our colleagues Dr. Anne Adams and Dr. Karen Kear of The Open University, UK for their comments on this research. Contact Information Correspondence concerning this paper can be sent to Dr. Shailey Minocha, Centre for Research in Computing, The Open University, Walton Hall, Milton Keynes MK7 6AA, UK. Address email to s.minocha@open.ac.uk

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010

Interviews Within Experimental Frameworks: How to Make Sense of Sense-making in Virtual Worlds CarrieLynn D. Reinhard Roskilde University, Denmark Abstract As virtual worlds become increasingly utilized for purposes of entertainment, information and retail, how people understand, think, feel, act and make decisions about them likewise become important research considerations. This essay reports on the methodology and methods used to study these sense-making processes in relatively inexperienced people as they engage with virtual worlds. In order to understand the sense-making of virtual worlds, a method to record the interpretive process, as well as physical actions, was required. In order to understand the sense-making processes involved in new experiences, an amount of control was required over the nature of those experiences. With these requirements, a hybrid study was designed by deconstructing the conceptualization of "the experiment" and utilizing both quantitative and qualitative methods. The resulting study involved the following: a within-subjects experimental design served as the framework for the study, while in-depth qualitative interviews were

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 4

employed alongside surveys and audio and video recording as the data collection methods. Data collection occurred while participants were engaging with the media products, via talk aloud protocols, and afterwards when they were asked to recall and compare these situations in open-ended questionnaires and interviews structured using Dervin's Sense-Making Methodology. Having completed the study using this mixed methodology approach, I discuss the effectiveness of this approach, and where the approach requires more work. Keywords: experiment, mixed methods, sense-making, entertainment, virtual worlds

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 5

Interviews Within Experimental Frameworks: How to Make Sense of Sense-making in Virtual Worlds

Introduction My main research interest is in understanding human sense-making processes, especially those involved in the engaging with a media product, whether categorized as "old" or "new". As part of the Virtual Worlds Research Group at Roskilde University, I conducted research on sensemaking with a recent innovation in media products, the virtual world. While there may be differing definitions of what is a virtual world, a common agreement discusses persistent online spaces where a network of computers generates a place for people to congregate via their digital representations, or avatars (Bell, 2008; Schroeder, 2008). This definition of virtual worlds provides for both gaming-based worlds, such as World of Warcraft and EverQuest, and socializing-based worlds, such as Second Life and SmallWorlds. The definition does not typically include console and computer games where the digital world in the game does not persist without the player's involvement. As a community of players around the world interact ingame or in-world at any time, virtual worlds change and evolve with or without the involvement of any single player or user. The ability for the world to evolve is what separates a virtual world from the more static environment of a single-player console or computer game. Given the intersecting interests of myself and the research group, I set forth to study people's sense-making in virtual worlds. What I discuss in this essay is the approach I took in my largest study for the project. In this study, the Virtual Worlds Entertainment Study, I sought to compare the engaging with virtual worlds to the engaging with other, older, media products, namely film and console games, for how sense-making related to evaluations of entertainment and desire to engage. As I will discuss here, my research interest and questions were instrumental in deciding how to study this phenomena. Instead of relying solely on either qualitative or quantitative methodologies, and related research methods, I determined the need for a mixed methodology approach. However, it was an approach predicated on the deconstruction of a traditional quantitative method, the experiment. In this essay I will discuss first this deconstruction of the experiment, and how the deconstruction allowed me to mix qualitative and quantitative research methods. This mixture was necessary as dictated by the nature of the research phenomena and questions of my study: a

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 6

necessity I hope to illustrate by discussing what was done in my virtual worlds study. Finally, I will discuss the disadvantages and advantages of this mixed methodology approach by detailing what I see as the successes and challenges of the data collection as I moved into data analysis. This essay, then, serves to explain my sense-making process as I decided how to design the study on my research phenomena. Deconstructing "The Experiment" Given that I saw the need to deconstruct the research method "the experiment" for my study, a place to begin this essay is with how an experiment has traditionally been defined. The standard definition relies on the manipulating of a variable while holding others in a system constant to observe the effect of the manipulation on predetermined, measured variable(s) in order to determine causality (Gjedde & Ingemann, 2001; Gorard, 2002; Roediger & McCabe, 2007; Winston & Blais, 1996; Shadish, Cook, & Campbell, 2001). The deconstruction begins when we consider the various components mentioned in this definition. First there is the "stimulus": the independent variable(s) that is manipulated. Second there are the "subjects": those being exposed to the stimuli under specified conditions, possibly with randomization determining which subjects are exposed to which stimuli. Third, there is the "system": the contextual conditions, control variable(s) and design structure in which the subjects are exposed to the stimuli. Finally, there are the "measurements" taken within this system on how exposure to the stimuli impacts the subjects. This deconstruction, and the relationship of the components, was expressed in Shadish, Cook and Campbell's (2001) discussion of Cronbach's deconstruction of experiments as having units (subjects) who receive the treatments (manipulation of stimulus) that are observed (measured) under certain settings (conditions of the system). The deconstruction continues further with confusion about what is "the experiment" when the perceived necessity of the various components differ across disciplines and researchers. Experiments in social sciences often differ from experiments in natural sciences. In physics there is more emphasis on measurement and testing theories than on manipulation and randomization, which psychology emphasizes (Winston & Blais, 1996) because of the need to account for variance between human subjects (Roediger & McCabe, 2007). Additionally, natural sciences

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 7

and social sciences are interested in how things work, and why, while design sciences are interested in how things differ based on different conditions (Gorard, 2002). Finally, the deconstruction arrives at a point where the experiment potentially becomes of no use when we consider its common criticisms, especially when it has been incorporated and applied by the social sciences, and psychology in particular. Both qualitatively and quantitatively minded scholars discuss the following critiques as to why experiments are not valid methods of research or how the results from such a method should be contextualized amidst these problems. The first critique focuses on the system component of experiments. The experiment is criticized as stressing internal validity, in its quest to establish causality, to the detriment of being able to provide external validity, also known as ecological validity or generalizability (Gillath, McCall, Shaver & Blascovich, 2008; Howe, 2004; Shadish et al, 2001). Because the experiment often involves an artificial system, the laboratory, it can be hard to generalize the findings of those conditions to other times and places. However, as long as in communicating their results researchers contextualize their findings to account for this limitation, Shadish, Cook and Campbell (2001) maintain that the experiment is not completely useless. Howe (2004), taking a qualitative stance, still considers the focus on causality, especially in social sciences, and the desire to make predictions about human activity, a fault of experiments, as such localized, artificial findings cannot adequately help us understand and influence the larger, broader, more complex systems that constitute human life. Others have argued that without ability to completely replicate the conditions of the system in the experiment, it can be hard for subsequent studies to verify causality (Gillath et al, 2008). Other common criticisms focus on the subjectâ&#x20AC;&#x2122;s component of experiments. These criticisms are particular to social sciences and how humans are handled as subjects. There are issues regarding researchers' overselling of the randomization of subjects as producing generalizable representative samples (Howe, 2004). Other critics take issue with the use of nonrepresentative samples, which adds to the problem of generalizability. As most social science experiments occur in university settings, researchers draw on the convenient testing population of college students; this sample, with characteristics unique to college students, cannot be seen as adequate to explain the wider human population (Gillath et al, 2008). Finally, there is the issue regarding being unable to prove causality in social sciences without knowing the "black box" of human interpretation (Howe, 2004; Potter & Tomasello,

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 8

2003). In psychology, the experiment was first primarily used within behavioral psychology's attempts to understand the stimulus-response mechanism. However, with the advent of psychological disciplines that focused more on the cognitive and affective mental processing of humans, the experiment was criticized for not addressing the subject's subjectivity when engaging with the stimuli. Without knowing this subjectivity, it is argued, whatever is learned in the experiment is not a result of completely interrogating all the processes that may have produced that result (Potter & Tomasello, 2003). My Deconstruction I have discussed in the previous section some deconstructions to the experiment as a concept that have occurred in previous literature. What I address in this section is the deconstruction I performed in order to see how to apply the experiment to the research phenomena and questions I had. My deconstruction is not limited to the experiment. Instead, it is a conceptualization of what is empirical research; I will explain it using the experiment, but I see it being applicable to other styles of empirical research, such as the survey or the ethnography. First, there is the "data collection framework": the foundational approach used to organize how information about the research phenomenon is collected to answer the research questions. Second, there are the "data collection methods": the means by which quantitative or qualitative information is gathered about the research phenomenon. Third, there are the "data analysis methods": the means by which the gathered information is explored, dissected, scrutinized and evaluated, usually synchronized with the approach used to gather the information; that is, qualitative data analysis methods often are employed with qualitative data collection methods, and the same with quantitative-based methods. However, this connection does not have to always-already occur; it is simply more likely that a researcher operating in one or the other approach tends to synchronize these three parts rather than consider how they could effectively mix and/or combine qualitative- and quantitative-based methods to study their phenomena and questions (Josephs, 2000). We often think of an experiment as a way to collect data; however, it can also be thought of as a way to organize the collection of data. The experiment provides an understanding of collecting data by determining a stimulus, sampling subjects, controlling the system, and

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 9

measuring what results from the subjects' exposure to the stimulus in the system. Data collection, then, occurs in the measurement component, as the framework determines what is being measured in whom and under what conditions. By seeing the experiment this way, the deconstruction acknowledges that the standard synchronization of quantitative-based framework, collection and analysis methods does not have to occur with the experiment. Instead, as the experiment provides the framework, collecting data can be done through quantitative methods, such as surveys and physiological recordings, or qualitative methods, such as interviews and participant observations. For me, the utility of deconstructing the experiment into these three aspects comes in how it frees up the experiment to be part of a mixed methodology approach. This approach to the experiment can be useful for several reasons. First, the use of qualitative data collection methods can address the criticism of the lack of measuring subject subjectivity (Howe, 2004; Potter & Tomasello, 2003; Williams, 2005). This reason was particularly important for my study, given the research interest in the phenomenon of sense-making. Second, using multiple methods of data collection and analysis can be a way to cross-reference the results obtained from the one or the other (Howe, 2004), thereby helping to explain the results obtained from either (see examples: Garau, M., Friedman, D., Widenfeld, H.R., Antley, A., Brogni, A. & Slater, M., 2008; Mandryk, Inkpen, & Calvert, 2006; Vinayagamoorthy, Steed, & Slater, 2008). Finally, from a more philosophical perspective, the mixed methodology approach can help the researcher and her study move away from the reductionism/determinism of a study based on either a qualitative or quantitative approach. Allegiance to just one approach often dictates the methods of data collection and analysis, possibly predetermining outcomes. Instead, as Josephs (2000) argues, researchers should construct a study based on the requirements of the phenomena being study. Advocating for a combined or mixed approach allows for recognition that no one method is better, epistemologically or ontologically, than another (Dervin & Foreman-Wernet, 2003; Gorard, 2002); rather, some are better at gathering certain types of information than others. As none offers complete information alone, it behooves the researcher to use a variety of methods in her study of the research phenomena. In this section, it has been my main assertion that by deconstructing what is the experiment into three main aspects, I could undertake a mixed methodology approach using the experiment as a data collection framework, hereafter called the experimental framework. This

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 10

deconstruction allowed me to construct a study to surround my research phenomena from a variety of perspectives theoretically needed to adequately study them. What I shall explain in the next section is how this deconstruction was informed by my study; that is, how the requirements of

the

study

dictated

the

construction

this

mixed

methodology

experiment.

Virtual Worlds Entertainment Study As I mentioned above, this deconstruction of the experiment was part of the sense-making process I went through for my study about how people make sense of virtual worlds, and how such sense-making relates to their evaluations of entertainment and desire to engage. In this section I will outline my specific research questions and how I conceptualized the phenomena of sense-making, entertainment, and desire to engage. After this discussion, I will focus on how my conceptualizations informed my decision to use the experiment as a data collection framework with quantitative and qualitative data collection methods. Research phenomena, questions Sense-making. There are various definitions of sense-making, but the primary focus is on the concept of people working to make meaning of themselves and the world as they move through their lives (Antonietti & Cantoia, 2000; Dervin, 2008; Dervin & Foreman-Wernet, 2003; Josephs, 2000). For my conceptualization, sense-making is understood as the internal behaviors, both affective and cognitive, and external, observable behaviors involved in working to understand one's self, life, and the world. Sense-making is a process of dealing with the situations of life, whether those situations are routine or novel, and whether the means of dealing with them are rote or unique (Reinhard & Dervin, 2010). While there may be outcomes of sensemaking that can be objectively observed, it is the internal processes that are the primary component. Thus, to properly understand an individual's situated sense-making, both subjective and objective measurements are required to be interrogated. Being entertained. My conceptualization of entertainment is not that it is simply due to the assignment by the producers to the media product, nor wholly in the hands of the media audience/user. My theoretical perspective is that the evaluation of entertainment is a complex, situated process of sense-makings by the individual as she regards: the media product, with

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 11

structural features of engaging that may or may not have been intended to be entertaining; herself and her lived experiences; and situational and socio-cultural contextual factors (Reinhard & Dervin, 2010). Of course, the structural features depends on the nature of the media product; the nature of the virtual world, or content, and the technology used to interface with it, or technological interface, can impact the determination of entertainment from that engaging. However, the nature of the media product is not the sole determining factor. As entertainment arises out of situations of media engaging, then it must be measured as part of the situated sensemaking process as the extent to which a person is being entertained. In this study, being entertained was not defined a priori; instead, the concept was left open, and what led the person to say he was entertained was measured in the study. Desiring to engage. The desire to engage with a media product is similar to the concept of consumer intentions in two ways. First, the idea is to understand the extent to which the individual desires to further her exposure to a particular media product with which she intentionally or unintentionally has engaged. In other words, the extent to which the individual would like to turn a onetime encounter into repeated encounters (Reinhard, 2008). Second, desiring to engage could refer to how much the person wishes to engage with a media product not yet experienced. This desire could arise from having engaged with similar media products, or having become aware of the media product from another source. For both types, the phenomenon is a desire â&#x20AC;&#x201C; an internal behavior that is a combination of cognitive and affective elements with the potential to determine a subsequent external behavior. Research question. My study sought to address the following question: how do people make sense of innovations to the structural features of media products meant to be entertainment when determining their own being entertained and desiring to engage? The focus on innovations addresses the rise of virtual worlds and their differentiation from previous media products, such as film, television, and even console and computer games, in terms of the technological interface. Looking at sense-making in relation to media products was similar to a question addressed by Antonietti and Cantoia (2000) in which the processes of sense-making were studied for how they were impacted by being in a virtual world to engage with a painting. My study had a similar intention, in that I wanted to know if engaging with different types of media products involves

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 12

different sense-making processes as the individuals determine the extent to which they are entertained and wish to continue engaging. How research intentions drove study's design In the following sections I will discuss how my research question, as focused on the research phenomena, were used to structure the study. First, I discuss how the intention of measuring the evaluations of entertainment and desire to engage informed the design of the experiment as a data collection framework to control the system, stimuli and subjects under study. Second, I discuss how the intention to understand the sense-making processes of engaging with the media products informed the data collection methods used within the experimental framework. The experimental data collection framework The purpose of the study was to compare sense-making processes and the outcomes of being entertained and desiring to engage for different media products to see if there is any impact on such processes and outcomes from the nature of the media product with which the individual engages. Deconstructing this purpose, using the procedures described above, there was an interest in understanding the impact of a stimulus (the nature of media products) on subjects' subjective reactions and objective actions. My intention was similar to one advocated by Gjedde and Ingemann (2001) for understanding media reception experiences. Because I was interested in comparing an individual's engaging with one media product to others, this meant designing a within-subject design as my experimental framework. In particular, four different sessions of media engagings were predetermined based on how media products differed regarding their requirements for the individual to engage with them. That is to say, the manipulation focused on the extent to which the individual had control over various aspects of engaging with the media product. This control could be over the content, or the technological interface, or both. The four types of media products chosen for this study were as follows: movie, console game, online role-playing game (MMOG), and social virtual world (SVW). With these as the main types, specific titles were chosen for each product. For the movie, 15 recent superhero titles were provided. For the console game, the Nintendo Wii game "Spider-Man: Friend or Foe". For the MMOG, "City of Heroes/City of Villains". For the SVW, an island was constructed in

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 13

Second Life, called Metrotopia: City of Superheroes. All media titles were chosen to provide novel experiences to the individuals; participants were instructed to choose a movie they had not seen, allowing some control over stimuli to them. The games were ones not played before; and the Second Life island was designed specifically for this study. One participant had played "City of Heroes,â&#x20AC;? but it was two years before this study, and the game had been updated repeatedly since then. Another participant had been in Second Life before, two years earlier, but had never been to the island Metrotopia prior to the study. The study was interested primarily with innovations on technological interface; while interface can affect content, it was decided to choose specific titles that would represent similar content. All of these media products had as a constant content that reflected superhero genre conventions. The movies were standard superhero stories; both games featured superhero main characters; the Second Life island was constructed to reflect specific superhero tropes, such as designing a costume and fighting villains. While some experiments involving virtual worlds have manipulated aspects of the content of the virtual environment (see examples: Arnold, Farrell, Pettifer & West, 2002; Gillath et al, 2008; Van Vugt, Konijn, Hoorn, Keur, & EliĂŤn; 2007; Yee & Bailenson, 2007), the focus of this study was on the manipulation of the structure of the interface with the media product, not the content; thus attempts were made to standardize the focus of the content, while recognizing that the information in each content cannot be equitable given the differences in narrative elements. Previous research has manipulated this interface aspect to test how changes to this particular structural features impacted the outcome of engaging. In these studies it appears more typical for the experimental framework to be a between-subject design, manipulating either the degree of difference in interface within same media technology or comparing different media technologies with completely different interfaces. Examples of the first type of manipulation would be: Barr, Noble and Biddle (2007) comparing console versus computer gaming interfaces; Downs and Oliver (2009) comparing the Wii's motion sensitive interface to button-symbolic interface; Klimmt, Hartmann & Frey (2007) manipulating the extent to which a game responded to user's control. Examples of the second type would be: Antoniettei and Cantoia (2000) comparing a static image to an immersive virtual environment; Chen and Raney (2009) comparing Wii, a Flash game, and a DVD clip; Mania and Chalmers (2001) comparing a

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 14

physical lecture, a desktop virtual environment, a head mounted virtual reality, and audio only; Suh and Chang (2006) comparing static images, video clips and a virtual environment. My manipulation of the interface was a combination of these approaches. I compared different interface designs within the same media technology by comparing the MMOG and the SVW. I compared the interfaces of different media technologies between film, console games and virtual worlds. Finally, I employed a within-subjects design to interrogate how the same person made sense of these differences, along with changes in content, amongst the media products. Unlike the "true" experiment, I allowed the participants to have more control in determining certain aspects of the system. First, while participants were brought into a laboratory for the sessions with the games and Second Life, they were allowed to watch the movie at a place and time of their choosing. Second, the length of the engaging was semi-controlled, in that they were required to engage with the media product for at least 30 minutes. Upon reaching that point, they were instructed that they could either stop the engaging, or continue until they wanted to stop, up to a point of 150 minutes (chosen because of the length of the longest movie in the study). Giving control over the length of engaging was used as a measurement of desiring to engage. Additionally, there was less strict control over the subjects in the study. Being a withinsubjects design, there was no attempt at randomization; all participants would be exposed to the same four types of media products. The primary control was to sample different experience levels. I was interested in novel engagings, hoping they would promote more self-awareness of sense-making processes within the participants to aid in later discussion. This meant the convenience sample was pre-screened to verify that the participants did not have high amounts of experience with the specific media titles being used in the study, as well as a range of experience with the types of media products. This requirement meant sampling was conducted to find people who had various amounts of experience with superhero movies, with video games, with computer games, with MMOs, and with Second Life. Additionally, recruiting sought out a range of ages, on the assumption that age influences exposure to the media technologies. These controls were done as an attempt to manage possible spurious effects that could account for differences in the sense-making processes.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 15

Thus far I have discussed how being entertained and desiring to engage were used to structure the data collection framework as a within-subjects experimental design. However, both of these phenomena were variables I was interested in measuring as a result of being exposed to the manipulation of type of media product. Along with the focus on sense-making, they informed the types of data collection methods the studied used. The mixed data collection methods From an experimental perspective, the focus on sense-making was both as a dependent variable, to be measured as a result of the exposure to the stimuli, and a mediating variable, a way into understanding the "black box" of human subjectivity that could give rise to the outcomes of entertainment and desire to engage. The other two phenomena were seen primarily as dependent variables; and, like sense-making, as evaluations of the engaging with the media product, they were primarily conceptualized as being subjective. However, while all these measured variables are subjective, there is argument for objective measurements to complement what is told in self report. The potential for measuring these phenomena both subjectively and objectively informed the decision for a mixing of data collection methods. While the need for such a combination exists in many studies, it was particularly required here given the intention to measure physical and interpretive interactivities. Engaging with digital games and virtual worlds involves physical interactivity: making sense of the technological interface in order to further the content, as the content will only be received as a result of the individual's actions with the media product. Typically the requirement for physical interactivity is used to differentiate the "new media" like digital games and virtual worlds from the "old media" like film. Such differentiation cannot be denied, and as mentioned above it was the manipulation used to determine the four stimuli. This physical interactivity can be measured objectively, through observing or logging, and subsequently coding, of any and all actions undertaken by the individual. Physical interactivity with games has been theorized as an important component for entertainment from that engaging (Klimmt, Hartmann & Frey, 2006). However, as these media products all involved content â&#x20AC;&#x201C; specifically superhero tropes â&#x20AC;&#x201C; another form of interactivity was also involved. All the engagings involved some type and amount of interpretive interactivity as the individuals had to make sense of the content in order for meaning to be generated from the experience. Games and social worlds are not so easily

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 16

comparable to the physical interactivity of other computer or digital applications because they have content within them that the engaging with is the purpose of using that technology (Barr, Noble & Biddle, 2007). I argue further that any engaging with a medium requires both physical interactivity via corporeal actions and psychological actions, such as cognitions and affectations, as the individual is sense-making his internal and external actions and reactions in the situation of the engaging (see Bucy, 2004, Salen & Zimmerman, 2004). Even watching a movie involves basic physical interactivity because actions to engage with the movie's content are undertaken in the physical world. Thus, in situations of engaging with stimuli like those used in this study, understanding interpretive interactivity becomes as important, if not more so, than physical interactivity. Interpretive interactivity could be gauged by emotive expressions and/or physiological measurements; however, there is often a need to match such objective measurement with a subjective measurement to insure that the researcher's interpretation of the observable behaviors coincide with the participant's (Mandryk, Inkpen, & Calvert, 2006; Vinayagamoorthy et al, 2008). Feldon and Karfai (2007) argue that studying virtual worlds calls for such a combined data collection given the individual's position in the interplay of behaviors in the virtual world and physical world. I extend this argument to digital games and films, as both involve some amount of physical and interpretive interactivities and negotiating between what happens in the content (the "virtual" diegesis) and what happens with the technological interface (the corporeal actions). The need for a combined data collection methods approach was furthered when considering that this study was not interested in usability issues regarding virtual worlds, but in how sense-making related to those issues. The study was interested in what are in part affective outcomes (Mandryk et al, 2006; ThĂźring & Mahlke, 2007): evaluations of being entertained and desiring to engage further. A focus on physical interactivity cannot provide the full understanding of such outcomes; but the addition of interpretive interactivity can account for the emotional experiences of engaging. Thus, to study sense-making processes and outcomes in engagings with media products, a variety of data collection methods were employed. Questionnaires were administered after each session to measure the amount of engagement and entertainment. The questionnaires consisted of close-ended and open-ended questions: numbered scale items with subsequent spaces asking

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 17

people to explain what led them to give the numerical answer they did. Observations were conducted during the laboratory sessions as I sat alongside the participant, taking notes, while video and audio recording captured what was done during the session. For the laboratory sessions, talk aloud protocols were followed, as participants were prompted to give voice to any reaction they were experiencing; additionally, specific questions were prepared ahead of time to probe specific features of each of the products. During the laboratory sessions, at the 30 minute mark, the session was paused for a mini-interview using Dervin's Sense-Making Methodology (SMM) to structure the questions (Dervin, 2008). During the movie session, participants were asked to pause the movie any time they had a reaction to it, and to record what the reaction was, when it occurred and what triggered it on a provided worksheet. After all the sessions were completed, an interview was conducted using Dervin's SMM to structure the questions used to have the participant compare the experiences and go in-depth into their experiences with each media product. This mixture of data collection methods was similar to the procedure used by Barr, Noble and Biddle (2007) to compare case studies of experiences with video games. In their research, they used observations, talk aloud protocols during game play, and semi-structured interviews after each game-playing session. One difference between our studies is that they focused on experienced players to see the results of having learned how to play these games, whereas I focused on inexperienced players to see process of learning, aka sense-making, as they engaged. By utilizing as many data collections methods as possible, I was able to amass a sizeable data corpus of information about my research phenomena that can be analyzed qualitatively and/or quantitatively. What I will discuss in the next session is how I see the successes versus the challenges of this approach in what my study attempted to do. Discussion of Approach The experimental design utilized in this study does not easily match unto the standard definition of the experiment. There was less rigidity of control over system, subject and stimuli components as I attempted to create more naturalistic engagings with media products to understand sensemaking processes. I was not interested in theorizing causality as beginning with the media product's structure, extending through sense-making processes, and resulting in being entertained and desiring to engage further. Instead, any interest in causality I had was based on the question

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 18

of do sense-making processes alter when faced with innovations in how one engages with different media products. As it can be hard to imagine such alterations not occurring, theorizing causality was not of primary concern; instead, I was concerned with fully mapping and understanding under what conditions do differences, or similarities, emerge amongst the sensemakings of these various situations. Without such a rigid focus on theorizing causality, my approach in this study closely aligned with Gjedde and Ingemann's (2001) experimental experience method and Gorard's (2002) discussion of design experiments. I employed the basic parameters of the experiment to structure four types of engagings with media products, then measured people's experiences as sense-making processes and outcomes in these engagings via quantitative and qualitative data collection methods. I am not the first to appropriate qualitative methods into the experiment; such work has been done to answer criticisms regarding the utility of results from the experiment. As those before me, I did find that the mixed methodology approach was useful in answering some of the criticisms of the experiment, and that the approach gathered information that can be used to more thoroughly interrogate the experiences of the participants. At the same time, there were issues regarding the way the experimental framework was designed that created challenges for my analysis of the information. However, the challenges can be addressed if, as Shadish, Campbell and Cook (2001) admonish, I remember to contextualize my results. In this next section I will address the successes and challenges from deconstructing the experiment and using a mixed methodology approach. Advantages/Disadvantages, Successes/Challenges Given the mixed methodology approach, what are seen as advantages and disadvantages depends on which side of the methodological fence one is: the qualitative side will see some aspects as advantages the quantitative side will see as disadvantages, and vice versa. For this study, these advantages and disadvantages became apparent in the data analysis; depending on what methods are used to interrogate the information, generate results, and communicate the findings, disadvantages may be just challenges to overcome. In terms of advantages, or successes, gathering the interpretive experiences along with recording their behaviors produced a complex and copious mixture of information. Preliminary

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 19

findings illustrate the utility of the mixture of data collection methods as information from one method can be used to reinforce and further elucidate information from another method. For example, an individual learning how to operate the Wii's motion sensitive controls for the first time: in the video recordings, she is seen trying to determine which hand should control the remote, and in the interview she discusses her thoughts and feelings about this physical interactivity issue. For all participants, after the sessions, the recollections in the interviews went into more depth, bringing in other lived experiences to discuss the engagings, such that the participants' expectations going into the session relate to what they did and how they made sense of what they did. The information gathered helped to show the variations possible amongst different people with different lived experiences dealing with similar situations. These variations also indicated the utility of pairing quantitative and qualitative data collection methods. Although the basic structure of each situation, in the laboratory at least, was held constant (same system, same stimuli), the between subject variance can be quite high, especially given the high probability that each person did not play the game in the exact same way. But even when the content was held constant, there was still between subject variance in the interpretive interactivity, such as when two different people watched the same movie. Although elements of the experimental sessions may be consistent, other elements are not, and this variation in physical and interpretive interactivities justifies the use of the mixed methodology approach as it delves into the "black box." Another advantage, or disadvantage, depending on one's perspective, is the size of the data corpus. The data corpus consists of 60.7 hours of audio recordings, 40.4 hours of video recordings, and 393 pages of questionnaires, reaction records, and transcriptions. For some, the amount of information gathered is daunting. However, this data corpus, with its subjective and objective data, can be used to probe the interpretive viewpoint of the participants during the experimental experiences. As indicated above, the in-depth information can be used to address the criticism of the experiment as failing to understand the "black box" sense-making processes. Two disadvantages, or challenges, align with common criticisms of experiments: the artificiality of the experiences as not being part of the participants' actual lived experiences with the media, as well as carryover effects in terms of learning how to engage. However, I stress that these disadvantages are more challenges that require I the researcher to be aware of the

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 20

limitations when it comes to deciding what data analytical method to employ, as well as how to communicate results. The first challenge, the artificiality of the experiences, can be answered from a qualitative approach, in particular Dervin's Sense-Making Methodology (Dervin & Foreman-Wernet, 2003). From an SMM perspective, each experience is a unique situation for sense-making, and at that point the experience can be seen for how it relates to other experiences in their past, present and future. Whether or not the experience occurred in the natural setting of the participant's home, or in the artificial setting of the laboratory, each experience still involves genuine sense-making from the participant, and this sense-making will in some way relate to the natural conditions of his everyday life. Indeed, the artificial conditions of the laboratory may help us better understand these sense-making processes. Josephs (2000) discussed the utility of the artificial lab environment "to slow down the meaning-making process in order to gain access to it." (p. 124). Of course, it is possible that a person's sense-making in a laboratory is affected by the pressure of such a situation (i.e. social desirability); however, an in-depth measurement of sense-making processes can illuminate any manifestations of the impact of the laboratory, and these manifestations must be provided in the communicating of the results to provide context for the other sense-making processes described. The second challenge is a common consideration for any within-subjects design, where each person experiences each experimental condition. In these designs there is the potential for carryover effect, such that in this study the experience with one media product may have influenced experiences with subsequent ones. There was indication of this, as some participants discussed how engaging with "City of Heroes" later impacted their engaging with Second Life due to similarities in their respective interfaces. Typically, carryover effect can be handled via the randomization of treatments, such that people are exposed to the stimuli in different orders. However, in the case of this study, such randomization would not have completely negated the carryover effect when considering how one individual made sense of each engaging; learning would have occurred regardless. Instead, as with the challenge of artificiality, participants' discussions that highlight the carryover effect must be drawn out in the analysis and communicating of the results. A final disadvantage is not a common criticism of experiments; in fact, it relates to a structuring element that is usually seen as one of the experiment's strengths. Perhaps the biggest

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 21

disadvantage, and most difficult challenge, involves the problematic comparison of sessions to one another, even within a particular participant's experience in the experiment. This problem is due to the differences in the conditions of engaging between the movie and the three other sessions; chiefly, how the former occurred outside of the controlled conditions of the laboratory, without the researcher present during the engaging. However, in addition, during the Wii session, I the researcher played with game with the participant, where the participant was the main character and I was the sidekick. This "playing with" did not occur in the other two laboratory sessions. Thus, across all four sessions, there were differences in the system as well as the stimuli. Such system differences are typically rigidly controlled for by the experimental framework so as to test causality hypotheses. As I mentioned above, I was not interested in studying causality, but in interrogating the processes and outcomes of situated sense-making. Thus I was less concerned with strict control of conditions and sought instead to create comfortable situations to promote a repertoire that would engender in-depth and honest reflection on sense-making processes. Movies were viewed at home because, as the first session, there was concern about social desirability; the idea that my presence while they watched the movie would impact their willingness to pause the movie and record every reaction they had to it. I played Wii game with them so as to promote a relaxed atmosphere for the think aloud aspects of data collection, especially as I would be there for that session and the following two. As part of the after sessions questionnaires for the laboratory sessions, I asked them to evaluate the impact of my presence on their engaging. Also, in the interviews, some participants discussed how the different conditions of the sessions related to their sense-making and evaluations of entertainment and desire to engage. As with the other challenges, all of this gathered information must be used to contextualize the results when they are communicated in the findings. Conclusion In conclusion, the mixed methodology approach was necessary given the research phenomena and question in my Virtual Worlds Entertainment Study. Deconstructing the experiment allowed me to conceptualize how I could proceed with my study. The experimental framework allowed for the comparison of different media products. The mixture of data collection methods has helped to measure the complexity of the engaging process, both the observable and non-

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 22

observable aspects of it. Thus far, the approach has yielded impressive data for in-depth, complex analysis, with recognition of the challenges generated by the experimental framework and the study's flaws. Future work should consider standardizing the conditions of the engagings, such that the movie occurs at the same location as the other engagings. The question is the extent to which a talk-aloud with a movie engaging could disrupt the sense-making process of the individual, not to mention alter the entertainment evaluation. However, as this is an empirical question, only more empirical experimental work can answer it.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 23

References Antonietti, A. & Cantoia, M. (2000). To see a painting versus to walk in a painting: An experiment on sense-making through virtual reality. Computers in Education, 34, p. 213223. Arnold, P., Farrell, M. J., Pettifer, S. & West, A.J. (2002). Performance of a skilled motor task in virtual and real environments. Ergonomics, 45(5), p. 348-361. Barr, P., Noble, J. & Biddle, R. (2007). Video game values: Human-computer interaction and games. Interacting with Computers, 19, p. 180-195. Bell, M. W. (2008). Toward a definition of "Virtual Worlds". Journal of Virtual Worlds Research,1(1), available at http://journals.tdl.org/jvwr/article/viewArticle/283. Bucy, E. (2004). Interactivity in society: Locating an elusive concept. The Information Society, 20, p. 373-383. Castronova, E., Williams, D., Shen, C., Ratan, R., Xiong, L., Huang, Y. & Keegan, B. (2009). As real as real? Macroeconomic behavior in a large-scale virtual world. New Media & Society, 11(5), p. 685-707. Chen, Y. S. & Raney, A. A. (2009). Mood management and highly interactive video games: An experimental examination of Wii playing on mood change and enjoyment. Paper presented at the International Communication Association Conference May 2009, Chicago. Dervin, B. (2008). "Interviewing as dialectical practice: Sense-Making Methodology as exemplar." Paper presented at International Association for Media and Communication Research (IAMCR) 2008 Annual Meeting, July 20-25, Stockholm, Sweden. Dervin, B. & Foreman-Wernet, L. (Eds.). (2003). Sense-Making methodology reader: Selected writings of Brenda Dervin. Cresskill, NJ: Hampton Press. Downs, E. & Oliver, M. B. (2009). "How can Wii learn from video games? Examining relationships between technological affordance and socio-cognitive determinates on affective

and

behavioral

outcomes."

Paper

presented

the

International

Communication Association Conference May 2009, Chicago. Evans, E. J. (2008). Character, audience agency and transmedia drama. Media, Culture & Society, 30, p. 197-213

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Interviews Within Experimental Frameworks 24

Feldon, D. F. & Kafai, Y. B. (2007). Mixed methods for mixed reality: Understanding users' avatar activities in virtual worlds. Educational Technology Research and Development, 56, p. 575-593. Garau, M., Friedman, D., Widenfeld, H.R., Antley, A., Brogni, A. & Slater, M. (2008). Temporal and spatial variations in presence: Qualitative analysis of interviews from an experiment on breaks in presence. Presence, 17(3), p. 293-309. Gillath, O., McCall, C., Shaver, P. R., & Blascovich, J. (2008). What can virtual reality teach us about prosocial tendencies in real and virtual environments? Media Psychology, 11(2), p. 259-282. Gjedde, L. & Ingemann, B. (2001). In the beginning was the experience: The experimental reception studies. Nordicom Review 2, p. 49-60. Gorard, S. (2002). Can we overcome the methodological schism? Four models for combining qualitative and quantitative evidence. Research Papers in Education, 17(4), p. 345-361. Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10(1), p. 42-61. Josephs, I. E. (2000). A psychological analysis of a psychological phenomenon: The dialogical construction of meaning. Social Science Information, 39, p. 115-129. Klimmt, C., Hartmann, T., & Frey, A. (2007). Effectance and control as determinants of videogame enjoyment. Cyberpsychology & Behavior, 10(6), p. 845-847. Mandryk, R. L., Inkpen, K. M., & Calvert, T. W. (2006). Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology, 25(2), p. 141-158. Mania, K. & Chalmers, A. (2001). The effects of levels of immersion on memory and presence in virtual environments: A reality centered approach. CyberPsychology & Behavior, 4(2), p. 247-264. Potter, W. J. & Tomasello, T. K. (2003). Building upon the experimental design in media violence research: The importance of including receiver interpretations. Journal of Communication, 53(2), p. 315-329. Reinhard, C. D.

(2008).

Gendered media engagings as user agency mediations with

sociocultural and media structures:

A Sense-Making Methodology study of the

situationality of gender divergences and convergences. Doctoral dissertation, Ohio State University.

Journal of Virtual Worlds Research – Interviews Within Experimental Frameworks 25

Reinhard, C. D. & Dervin, B. (2010). "Situations of media engagings: Comparing the new and the old through sense-making".

Paper presented at the Association of Internet

Researchers Conference 11.0, October 21-23, Gothenburg, Sweden. Roediger III, H. L. & McCabe, D. P. (2007). "Evaluating experimental research: Critical issues." In R. J. Sternberg, H. L. Roediger III, & D. F. Halpern (Eds). Critical Thinking in Psychology (pp. 15-36). Oxford, United Kingdom: Cambridge University Press. Salen, & K., Zimmerman, E., (2004). Rules of Play: Game Design Fundamentals. MIT Press, Cambridge, MA; London, England. Schroeder, R. (2008). Virtual worlds research: Past, present & future. Journal of Virtual Worlds Research, 1(1), available at http://journals.tdl.org/jvwr/article/view/294. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York, NY: Houghton Mifflin Company. Suh, K. S. & Chang, S. (2006). User interfaces and consumer perceptions of online stores: The role of telepresence. Behaviour & Information Technology, 25(2), p. 99-113. Thüring, M. & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4), p. 253-264. Van Vugt, H. C., Konijn, E. A., Hoorn, J. F., Keur, I. & Eliëns, A. (2007). Realism is not all! User engagement with trask-related interface characters. Interacting with Computers,19, p. 267-280. Vinayagamoorthy, V., Steed, A., & Slater, M. (2008). The impact of a character posture model on the communication of affect in an immersive virtual environment. IEEE Transactions on Visualization and Computer Graphics, 14(5), p. 965-981. Williams, D. (2006). Virtual cultivation: Online worlds, offline perceptions. Journal of Communication, 56, p. 69-87. Winston, A. S. & Blaise, D. J. (1996). What counts as an experiment? A transdisciplinary analysis of textbooks, 1930-1970. American Journal of Psychology, 109(4), p. 599-616. Yee, N. & Bailenson, J. (2007). The Proteus Effect: The effect of transformed selfrepresentation on behavior. Human Communication Research, 33, p. 271-290.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 dint u say that: Digital Discourse, Digital Natives and Gameplay Theresa A. Oâ&#x20AC;&#x2122;Connell National Institute of Standards and Technology, United States John Grantham Systems Plus, United States Wyatt Wong Forterra Systems Inc., United States Kevin Workman Millersville University, United States Alexander Wang Cornell University, United States Abstract Discourse analysis has the potential of providing insight into gameplay dynamics and team success. However, because of factors such as interrupted sequences, gameplay discourse does not easily lend itself to discourse analysis. Therefore, in addition to traditional methods, new and modified discourse analysis methods were applied to a corpus of 858 discrete gameplay discourse events disclosing discourse characteristics during collaborative problem solving. Four teams of four digital natives each played PanelPuzzle, a limited-time span, goal-oriented game, in a virtual environment. Discourse both reflected and impacted team dynamics. It manifested leadership. To promote team success, gameplay digital discourse tone was serious, showing little evidence of fun although players reported enjoying gameplay. Brevity, ill-formedness and distorted syntax were chief characteristics, but, because it was goal-oriented, it differed markedly from reported social digital discourse. Digital natives used digital discourse effectively to communicate, build community, collaborate and accomplish gameplay tasks. We conclude that gameplay digital discourse constitutes a distinct linguistic register which prioritizes efficiency over well-formedness. We characterize this register in a taxonomy and a meta-taxonomy.

Keywords: collaboration, computer mediated communication, digital discourse, digital natives, discourse analysis, games, roleplaying, usability engineering, collaborative virtual environments, virtual worlds, video games Acknowledgements: This work was supported in part by the Intelligence Advanced Research Projects Activity.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 4

dint u say that: Digital Discourse, Digital Natives and Gameplay Discourse analysis has the potential of providing insight into gameplay dynamics and team success. It also has the potential of uncovering whether digital discourse reflects players’ in-game experiences and interactions with each other while impacting gameplay outcomes. However, gameplay discourse among digital natives does not easily lend itself to traditional discourse analysis. Digital natives are people who have grown up surrounded by and using technology (Prensky, 2004). Lewis and Fabos (2005) note that a high degree of familiarity and comfort with technology can result in new ways of reading and writing while using technology. Digital natives enjoy a connectivity that fosters communication. Much of their digital discourse serves social purposes. With constant and immediate access to communication technologies such as cell phones, text messaging and instant messaging, they build community (Khoo & Zubek, 2002), e.g., by maintaining awareness of each others’ locations and activities. This awareness manifests in gaming environments. We examine gameplay experience from the perspective of digital discourse among digital natives playing PanelPuzzle, a game developed for this study. Steinkuehler (2003) demonstrates that massive, multiplayer online roleplaying gaming (MMORPG) is equivalent to engaging in discourse. Although our experiment was limited to 16 digital natives, playing under four independent conditions each limited to four players, characteristics that PanelPuzzle shares with MMORPGs make it possible to extend Steinkuehler’s (2003) observations on MMORPG discourse to apply to PanelPuzzle. These characteristics include role playing; interactivity; dependence on internet connectivity; a persistent virtual environment (VE); competition among teams; a timed quest; and teammate collaboration. In these aspects, PanelPuzzle also qualifies as a collaborative virtual environment (CVE) which provided an opportunity to study the relationship between digital discourse and collaboration. This work builds on two studies that examined in-game collaboration (O’Connell, Grantham, Workman & Wong, 2009; O’Connell, Choong, Grantham, Moriarty & Wong, 2008). Here, we focus on digital communication in American English (AmerEnglish) among digital natives using text chat during gameplay. In earlier analysis of data from this experiment, we saw that communication and roleplaying positively impact teams’ success and players’ satisfaction

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 5

(O’Connell, et al., 2009). Here, we investigate relationships between text chat and team dynamics; relationships between leadership and team discourse; and effective digital communication among teammates. We discuss characteristics of gameplay discourse. Within this paper, we define discourse as communication between one source (person sending a message) and one or more sinks (people receiving a message) typing natural language during video gaming. By detecting topic shifts, it is possible to divide discourse into parts (Brown & Yule, 1983). However, in a simple chat message box, the usual markers, such as paragraph breaks, do not occur. Interspersed parallel threads impose interruptions that give the appearance of topic changes when topics actually persist. Therefore, we used a discourse model constructed of discrete segments we call discourse events (DE). By a text chat DE, we mean a discrete body of text that proceeds from a sole source at a time-stamped start point and ends at publication in a text chat window where it is received by one or more sinks. A DE is bounded by the first and last characters typed. Its purpose is to convey meaning from source to sink. We limited discourse analysis to DEs among teammates during timed PanelPuzzle gameplay. Hypotheses A rich body of literature on social digital discourse sometimes addresses technology-driven characteristics of digital discourse among digital natives (e.g., Lewis & Fabos, 2005; and Greenfield & Subrahmanyam, 2003). There is a less rich body of literature on discourse during gaming (e.g., Tychsen, Smith, Hitchens & Tosca, 2006; Steinkuehler, 2003), little of it referring specifically to digital natives. We applied insights from this literature as well as our own observations to construct four hypotheses about teammates’ text chat discourse. H1. Text chat discourse will both reflect and impact team dynamics. By team dynamics, we mean interactions among teammates and the attitudes of teammates toward each other. We expected discourse to set the tone for team dynamics. Because team dynamics, in turn, would reflect team activities, we expected DEs to increase as players arrived at critical points in gameplay. We expected emotional statements to occur as players neared goal accomplishment. Building collaborative knowledge is characteristic of digital natives. Indeed, Prensky (2004) suggests that the motto of digital natives be “Sharing Knowledge is Power.” We expected teams to build collaborative knowledge through discourse.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 6

H2. When a team leadership role is played, text chat discourse proceeding from the team leader will promote and foster the team’s goal to win the game. Our earlier work aligned with the findings of Maybury (2001) who notes that a lack of leadership impedes successful group interactions in a CVE. Building on work showing that linguistic analysis can reveal social structures (e.g., Paolillo, 2001), we extended our earlier finding to a hypothesis that centers on team leaders’ statements and the impacts of leadership or the lack thereof on team dynamics. We expected team leaders’ statements to promote and foster winning by asserting authority; providing direction; organizing team activities; promoting construction of collaborative knowledge; and boosting morale. We expected leaders to build community. We expected a lack of leadership to coincide with a lack of community. Studying leadership in virtual teams, Sudweeks and Simoff (2005) note that leaders’ communication behaviors include firmness and early and frequent participation in team discourse. Therefore discourse can signal emergent leadership in conditions where leadership roles are not mandatory and disclose whether designated leaders do indeed exert leadership. We expected teammates’ discourse to be goal-oriented if their leader’s DEs manifested goal orientation. We expected that in conditions without a leader, discourse would not manifest an organized team strategy to win the game. We expected to see sequences of related DEs initiated by the team leader. H3. Text chat discourse during gameplay will be predominantly serious. We expected discourse to be predominantly serious, focusing on gameplay with little, if any, linguistic evidence of fun. H3 was based on our earlier observations of digital natives during timed gameplay; even when players were obviously having fun, there was little linguistic reflection of fun. Instead of banter and joking, we have observed serious task-oriented discourse. H4. The structure and vocabulary of text chat discourse will resemble the structure and vocabulary of the electronic messages that digital natives typically exchange. We expected DEs to exhibit properties discussed in literature on digital discourse. To inform H4, we examined computer-mediated communication (CMC) in general and five types of digital discourse: email; text messages (TM), including short message service (SMS) used for sending

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 7

TMs from cell phones; text chat, especially as used in MMORPGs; internet relay chat (IRC); and instant messages (IM). Herring (1999) examines the disrupted turn adjacency properties of CMCs where transmitted messages appear, as they did in PanelPuzzle, linearly in the order in which the system receives them, rather than in a logical discourse sequence. Herring (1999) notes adaptation to the CMC medium with innovative mechanisms to offset potential incoherence when strands of conversations juxtapose and interweave, forming complex, multidimensional text where overlaps and intentional interruptions are impossible. Mechanisms include minimal responses; signaling the end of the DE with an end-of-turn character or simulated raised hand; and addressivity (using an intended sinkâ&#x20AC;&#x2122;s name). During text chat, we expected players to import linguistic traits from emailing and TM, both popular in a survey of PanelPuzzle players. These included brevity, quick topic changes and deviations from Standard AmerEnglish (Khoo & Zubek, 2002; Zubek & Khoo, 2000). Aoki and Woodruff (2005) report that TMs are typically short. Causes include the difficulty of typing on small keypads and character limits imposed by service providers. Grinter, Palen and Eldridge (2006), studying teenage social use of SMS, report teens using short messages to plan and coordinate. Examining text chat in MMORPGs, Steinkuehler (2007) posits that limited display space leads to abbreviations (nvm for never mind), acronyms (lol for laughing out loud) lexical truncations (u for you), omissions of syntactic elements; onomatopoeia; and using alphanumeric characters and punctuation marks to convey gestures. She observes typing and grammatical errors and vocabulary customized to the game. Citing emoticons, i.e., using alphanumeric characters and punctuation marks to convey emotion, Greenfield and Subrahmanyam (2003) report that text chat has a visual aspect exploited by digital natives to facilitate communication. Emoticons replace extralinguistic signals such as smiling. Studying collaboration, Gergle, Millen, Kraut and Fussell (2004) note that collaborators with a persistent view of text chat tend to use short sequential DEs to describe puzzle pieces; because of the view, they need not request repetition. A linguistic register is a modification of language for a particular purpose or social context. Doell (1998) classifies IRC as a distinct register characterized by creative conventions that render it speechlike. Werry (1996) notes that IRC emulates paralinguistic traits of spoken

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 8

language through nonstandard capitalization for emphasis (go NOW); nonstandard spelling in colloquialisms, whutcha mean; or letter reduplication for emphasis (heeeeelp). He cites substituting punctuation for spoken cues (!!!) for a raised voice. He notes coded expression of gestures (*hugs*). Working with multinational English, including AmerEnglish IRCs, Paolillo (2001) finds DEs of three to six words and sources interrupting each other. Khoo and Zubek (Khoo & Zubek, 2002; Zubek & Khoo, 2000) note similarities between competitive gameplay text chat and IRC: misspellings; disconnected discourse; frequent topic changes; and simultaneous threads. Lewis and Fabos (2005) capture the structure of social IMs exchanged among digital native teens. They observe frequent use of visual elements to supplement discourse, e.g., ellipses to show that the source is thinking. They observe that correct spelling is important, but not proper punctuation or capitalization. Sources vary word choice and sentence structure to change conversational tone, e.g., they send shorter, more pointed answers to sinks with whom they are not interested in communicating. Sources often type several short, partial sentences in rapid succession to prevent topic changes. Independent Variables The source of a text chat message is always apparent, but the sink is not. Sometimes PanelPuzzle DEs addressed one teammate; other times, all three. So, the variables examined here were limited to the perspective of the source. The first independent variable was whether or not the source was playing the role of team leader. The second was whether the sourceâ&#x20AC;&#x2122;s role was taken on voluntarily (RV) or if roleplaying was mandatory (RM). The third was whether all sources in a condition were allowed to communicate with the game master (GM). Bravo

Charlie

Roles Voluntary (RV)

Roles Mandatory (RM)

All Sources Communicate with GM

GM Communicator/Leader Only Source to Communicate with GM

Delta

Echo

Roles Mandatory (RM)

Roles Voluntary (RV)

GM Communicator/Leader Only Source All Sources Communicate with GM

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 9

to Communicate with GM Table 1. Independent variables pertaining to role playing and communication with the GM. Players Players were fourteen male and two female digital natives between the ages of 18 and 29. Gameplaying experience varied among 11 players; five were not gamers. All gamers played interactive video games weekly: seven for 1-5 hours; three for 6-15 hours, and one for 16-25 hours. At the time of the study, all players were attending college or working in the US. All were fluent in AmerEnglish. One was a native speaker of French and three learned Spanish before English. All used telecommunication devices frequently. Chatrooms and forums, both important PanelPuzzle communication mechanisms, were among their least frequently used means of communicating with friends. Their favorite digital communication mechanisms were cell phones, Facebook®, IM and TM. Playing PanelPuzzle The goal of PanelPuzzle was to put together puzzles to earn points. The game was designed to cause players to communicate and collaborate. Players sought puzzle pieces distributed throughout a VE called Peninsula City. All teams experienced the same VE with the same 36 puzzle pieces for nine puzzles dispersed identically inside and outside of 100 buildings, 10 with interiors. Its 1,300 scaled kilometers spanned 36 city blocks, causing players to move independently throughout the city. Avatars walked or ran at a pace that scaled to real life. Teleportation immediately moved them to principal locations. Thus, avatars quickly dispersed beyond each others’ fields of view, causing players to communicate about their activities. Once a player reported a piece’s location and its unique numeric identifier to the team, that piece remained visible where it was found. Communication was necessary to promote gameplay efficiency, e.g., to prevent multiple reports of finding the same piece. Screens separated players to prevent eye contact, forcing players to depend on linguistic, rather than visual copresence during discourse. We discuss two PanelPuzzle communication channels: text chat, which is the topic of this paper, and forums, which are reported only as they relate to text chat. Players communicated synchronously using text chat, or asynchronously using the players forum. Designated players

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 10

communicated with the GM using a GM forum. Players manipulated their avatars and composed and sent text chat messages using a keyboard. Therefore, they were not able to post to the text chat or forums while moving their avatars. Teammates had to collaborate to learn which pieces belonged to which puzzles and to assemble pieces into panels in a horizontal, side-by-side arrangement. Peninsula City had no area large enough to display nine panels. Thus, the city center contained only three panels: each with sections to display only one three-piece, four-piece, or five-piece puzzle at a time. Each piece belonged in only one location within one panel. At any point during gameplay, teams usually had pieces for more than three puzzles. Therefore, they were forced to build collaborative knowledge about how pieces fit into both puzzles in view and puzzles that were not currently displayed. Four teams, Bravo, Charlie, Delta and Echo, competed to earn the most points. Winning depended upon building collaborative team knowledge by coordinating individual playersâ&#x20AC;&#x2122; distributed knowledge about which pieces were found and where those pieces fit into a panel. Four sessions each accommodated one team of four players. Players worked on any puzzle they desired at any point during 75 minutes of gameplay. had A random numerical identifier on each puzzle piece facilitated discussion. Notations on pieces indicated the size of the panels to which pieces belonged, but there was no indication of which puzzle a piece belonged to. Players had the option of displaying any piece in any panel at any point in gameplay by requesting insertion, movement or removal by the GM. For each panel size, there was one math, one word and one image puzzle. Common equations, e.g., the quadratic equation, appeared in the math puzzles. Everyday words or sentences constituted word puzzles. Image puzzles housed pictures easily recognizable by digital natives, e.g., a popular cartoon strip. To protect anonymity, players received gameplay names, e.g., Golf_Echo; the first part of the name referred to the player and the second to the team/condition to which the player was randomly assigned. In RV conditions, Bravo and Echo, players could opt to adopt no role, any one role, or multiple roles. RM conditions, Charlie and Delta, obliged players to assume only one role. Across all conditions, players chose among the same four roles. A GM Coordinator communicated with the GM. A Communications Coordinator managed the players forum. A Map Coordinator organized puzzle-piece searching. A Puzzle Piece Coordinator directed puzzle assembly. Prior work demonstrated that the GM Coordinator became the de facto team leader

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 11

(O’Connell et al, 2009). In RV conditions, players had the option of adopting an unofficial leader. Before gameplay, there was time to select roles and discuss strategies. A digital-native GM required players to communicate with him according to a strict format specifying the number of the puzzle piece, the panel size and the piece’s location in the panel. The GM was not considered a player. In Bravo, any player could communicate with the GM, but, only by using a GM forum. In Charlie, only the GM communicator could communicate with the GM, using only text chat. In Delta, only the GM coordinator could communicate with the GM, using only the GM forum. In Echo, any player could communicate with the GM, using only text chat. When these rules were met, the GM inserted, moved or removed pieces, even if a request specified the wrong panel or section. When players considered a puzzle correctly completed, the GM was informed. Using the GM forum or text chat, depending on the experimental condition, the GM responded with the number of points won: 50 points for a correct three-piece puzzle, 100 for a four-piece puzzle, and 200 for a five-piece puzzle. Experimental Environment Each player had a desktop computer running Windows XP Professional, Service Pack 2. Peripherals included a standard 101/102 keyboard; a three-button click/scroll-wheel mouse; and two monitors (19”, 20” or 21”) set to maximum resolution. For each player, one monitor displayed Peninsula City; the other the forums. Players used the keyboard and mouse to input to the in-game text chat and forums. Other software included Microsoft Internet Explorer 7 Web browser and Forterra’s On-Line Interactive Virtual Environment (OLIVE) version 2.0.1. OLIVE is a software platform for building persistent VEs for collaborating over networks. It employs a client-server architecture where PC clients connect to a central server via a network. OLIVE supports an array of capabilities and functionality in the baseline platform that support interactive VE operations, e.g., avatars, distributed physics, a session record and playback capability. The VE has a set of general three-dimensional art assets, e.g., avatar clothing, gestures, realistic faces, buildings and vegetation. The OLIVE client and server applications were modified to log text chat events. The OLIVE client redirected text messages to the log file and to the OLIVE chat window. The client log output captured the timestamp, source (avatar name), and content of all DEs.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 12

Figure 1. Text chat logs included three components: (A) date and Eastern Standard Time stamp; (B) text chat message identifier to separate DEs from other logged events; and (C) text chat message as seen by players, including transmission time and sourceâ&#x20AC;&#x2122;s game name. The text chat interface resided within the OLIVE graphical user interface. It comprised an input area for submitting and publishing DEs and a scrolling display of published DEs. Players activated the input area by clicking within it. A subsequent Enter keystroke submitted and published input. Nine lines of text, up to 40 characters wide, were visible at a time. It is characteristic of text chat that contributions to threads are not colocated or identified as belonging to any thread. Thus, it was possible for several interspersed threads to display at once. Text chat was persistent in that players could not delete each othersâ&#x20AC;&#x2122; input and could scroll back to earlier messages. However, DEs scrolled out of view, giving text chat a temporary aspect. Analysis Method Data consisted of a corpus of 858 DEs, teammate text chat collected under four experimental conditions within 75 minutes of gameplay. This time span excluded DEs during pregame training and familiarization. Data included alphanumeric characters and punctuation marks. It excluded all DEs where the GM was source or sink. Text chat does not easily lend itself to discourse analysis which focuses on coherent sequences of discrete communication acts. While PanelPuzzle DEs were discrete, four-way discourse among teammates who are sometimes collaborating and sometimes engaged in different activities is not always coherent. When possible, we applied traditional discourse analysis methods, isolating sources and sinks; examining DE function and pragmatic value in context; and analyzing DEs on the levels of morphemes and words (e.g., Brown, G. & Yule, G., 1988; Cook, 1969). In other cases, we devised methods to analyze the unique DEs that constituted typed communication among game-playing digital natives, e.g., treating emoticons as words and punctuation marks as having semiotic value beyond their traditional syntactic functions. We adapted traditional tagmemic analysis (Cook, 1969), treating sentence fragments as bona fide grammatical constructs and starting analysis at the level of the DE rather than the

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 13

sentence, then proceeding down to the morphemic level. Although PanelPuzzle DEs were not spoken, we examined their similarity to spoken AmerEnglish in seeking explanations of their deviations from standard written AmerEnglish. We applied a usability perspective, investigating efficiency, effectiveness and satisfaction (ISO, 1998). Text chat functionality forced turn taking in a way that differed from AmerEnglish vocal discourse where social protocols discourage interruptions and confer the role of source on one person at a time. Text chat functionality made deliberate interruptions impossible. However, multiple sources could send DEs simultaneously. DEs displayed in the order in which the system received them, not according to which player started to type first. Thus, although the playersâ&#x20AC;&#x2122; intent was synchronous communication, the technical reality was that in those cases where DEs were queued, text chat was asynchronous. In PanelPuzzle, with only four players and one GM active, this asynchrony was so minor that we could determine no effects such as forced pauses in gameplay or periods without discourse. Therefore, we considered text chat to be synchronous.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 14

Table 2. A taxonomy of PanelPuzzle discourse event characteristics.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 15

The non-sequential aspect of chat display made social protocols for turn allocation nonenforceable. In out-of-game everyday discourse, the source often uses addressivity to assign a sink the next turn as source and sinks often self-select themselves as the next source within social protocols for interruption. However, in text chat, the selection device for source was neither the current source nor self-selection. Source designation was confounded by DE length â&#x20AC;&#x201C; of two DEs started simultaneously, the one that was completed first was published first. Because the text chat record was always visible to all teammates, even in those cases where the intent was one-to-one communication, the effect could be one-to-many. There is another confounding factor: we have no way of verifying that every player read every DE. Therefore, we always know who the source is, but rarely can identify the sink(s). Nor can we verify that every DE did indeed have a sink. Sink identification is usually limited to those instances when a sink takes the next turn as source and the DEâ&#x20AC;&#x2122;s content indicates that it is a response. Thus, we argue that traditional discourse models that proceed from an identified source to an identified sink do not apply directly to text chat and, in discussing DEs, we usually refer only to the source. Discourse was analyzed both by humans and programmatically. Chat log analysis started with human categorization of DEs according to content characteristics. Categories arose from literature searches and the data set content. DE category identification resulted in taxonomies of digital discourse characteristics (Table 2.) and meta-properties (Table 3.) during PanelPuzzle play. We report analysis within taxonomy categories, but, due to space constraints, do not address each subcategory. Instances of most categories were identified by both digital natives and non-digital natives, then counted using Excel formulas. When, during categorization, the meaning of DEs was not clear, two or three digital natives, gamers and frequent digital communicators who had not been players in this study, served as subject-matter experts (SMEs), providing clarification.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 16

Table 3. A taxonomy of discourse event meta-properties. A modified shareware tool used within MS Word counted spelling and grammatical errors (Maxey, 2008). When the utility confirmed there were a number of grammatical errors worth investigating, human counts and analysis followed. A Perl script counted DEs; words and characters (excluding spaces) in each event; punctuation marks; instances of lower case i for the first person pronoun as well as iâ&#x20AC;&#x2122; and im; uses of all upper case; and sequential same-source DEs. Words were considered to be strings of characters bounded by either the start or stop of the DE or by blank spaces between the strings. Therefore rogue characters and emoticons were counted as words. Each character within the emoticon was counted, e.g., a smiley face, :-) constituted three characters. Disagreements between players were noted non-automatically. Because of the small sample size (16 players), we did not perform extensive statistical analyses. In some cases, we derived averages to support study of differences among teamsâ&#x20AC;&#x2122; discourse. Results We expected players to import linguistic traits from digital discourse technologies they commonly use. Chat analysis disclosed brevity, quick topic changes and deviations from Standard AmerEnglish. During gameplay, there were 858 DEs where source and sink(s) were

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 17

players. Bravo (0 points) had the least; this may be due in part to the fact that, early in gameplay, Bravo communicated with each other in the GM forum, a tactic not seen in other conditions. Bravo (RV)

Charlie (RM)

Delta (RM) Echo (RV)

Unique pieces found

Puzzles solved

Points earned

200

250

Total DEs

142

227

324

165

Total words

962

1,303

1,736

1,040

Average words per DE

Total characters

3484

1303

6761

4032

Average characters per DE

Table 4. Characteristics of discourse events and team success. We observed no statistical correlation between the number of words or types of discourse per DE, and solving puzzles throughout gameplay for any condition. However, human analysis of DEs gave insights into possible reasons for success. Although Delta tied with Echo in earning 250 points, Delta was considered the winning team because it was the most efficient, making the least number of placement requests (8), and the only team to make no wrong placement requests. Delta transmitted the most DEs and the most words; the length of their DEs was longer than those of other teams. Delta had the most instructional DEs (114); Bravo (57) the least. In Echo, the only condition where team dynamics were sometimes hostile and leadership was disputed, there were fewer DEs than in Delta or Charlie. All conditions sent the same top four categories of DEs: strategy, informational, cooperation, and questions. Across all conditions, these categories had the same ordering in the number of DEs from most to least. In all conditions, the top four DE taxonomy categories had relatively the same linear slope, suggesting that we could average to generalize the trends for these four categories. An average of 89% of all DEs was in one of these four categories.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 18

Figure 2. An average of DEs for all conditions across all categories shows that most DEs fell into one of four taxonomy categories: strategy, informational, cooperation or questions. Roles/Leadership DEs identified team leaders as they tended to send DEs early in gameplay. Even in Bravo (RV), with no official leader, Tango_Bravo evidenced emerging leadership by chatting first and firmly directing other players. In Charlie (RM), acknowledged leader, Lima_Charlie, was the second to chat at 12 seconds into gameplay. In Delta (RM), designated leader, Quebec_Delta, was first to chat with a sequence of three DEs. Zulu_Delta, who emerged to share leadership with QuebecDelta, immediately responded with a sequence of two DEs. The cooperative leadership in Delta can be considered a bond between two players. Paolillo (2001) posits that such a bond can be measured by counting DEs. Out of 324 Delta DEs, 125 constituted purposeful, albeit not always exclusive, communication between Quebec_Delta and Zulu_Delta. In Echo (RV), leadership was immediately evidenced in the first DE by Golf_Echo and challenged later by competing leader, X-Ray_Echo, who first used text chat at the sixth DE, one minute into gameplay. Golf_Echo sent the first DE at 10:40:57; early in gameplay, he gave instructions and clarified strategies. By 11:00:27, X-Ray_Echo began to give clarification. At 10:46:53 Zulu_Echo assumed the role of GM Communicator, sending two successive requests for piece insertions. Golf_Echo transmitted only 12 more DEs, four of them addressed

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 19

specifically to the new leader, X-Ray_Echo. In four sequential DEs between 11:01:50 and 11:05:43, X-Ray_Echoâ&#x20AC;&#x2122;s style manifested as aggressively assertive. By 11:32:24, Golf_Echo had ceded leadership, asking X-Ray_Echo for direction. X-Ray_Echo used sequential DEs to control discourse. By gameâ&#x20AC;&#x2122;s end, X-Ray_Echo was clearly in control, dealing harshly with competing Zulu_Echo when he contacted the GM, requesting insertions. In two conditions, leaders sent more sequential DEs than other players: Tango_Bravo (50 of a 76 team total) and Lima_Charlie (22/72). In Delta, DEs disclosed that Quebec_Delta (70/167) shared leadership with Zulu_Delta who had the most sequential DEs (77/167). In Echo, the earlier leader, Golf_Echo, had 13/70 and the later leader, X-Ray_Echo, had 24/70. Length and Flow Most DEs contained ten or fewer words across all content topics. The average words per DE was Bravo, 7; Charlie, 6; Delta, 5; and Echo, 6. In all, Bravo sent 962 words; Charlie 1,303; Delta 1,736; and Echo 1,040. Echo sent shorter, pointed answers to undesired sinks. Delta sent the most words, but its DEs had the shortest average length. Brevity promoted speed. DEs were often only one word.

Figure 3. DEs were typically short.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 20

Instead of a frenzy of activity immediately before puzzle submission and at gameplay’s end, the highest peaks of discourse activity occurred during the first quarter for Charlie, Delta and Echo as they set strategies and started finding pieces, and in the third quarter for Bravo whose strategy was not highly organized. Immediately prior to puzzle solving, discourse addressed methods to report found pieces and locate them in panels. For Charlie (200 points) and Echo (250), DEs decreased after puzzle submission. Conversely, for Delta (250), the number of DEs increased after submitting puzzles. DEs typically peaked about ten minutes before game’s end as emotions rose and players urged teammates to solve puzzles. Questions about piece locations increased as game end neared. As expected, team leaders initiated sequences of related DEs. Conforming to Lewis and Fabos’ (2005) findings, sources sent short DEs in uninterrupted sequence to focus discourse on topics of importance to the source. Sequential discourse was usually collaborative, e.g., 11:11:06 Lima_Charlie

1 left on 4 piece, 1 left on 3 piece and 2 left on 5 piece

11:11:56 Lima_Charlie

hmm these don't look like they go together

11:13:09 Lima_Charlie

does anyone know the quadratic equation?

11:13:20 Hotel_Charlie

yes

11:13:33 Hotel_Charlie

i know it

11:13:34 Foxtrot_Charlie i do

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 21

Figure 4. Squares indicate approximate points when teams submitted completed puzzles. Bravo completed none. Collaboration and Cheering Although seriously focused on gameplay, chatâ&#x20AC;&#x2122;s tone was casual, usually reflecting cooperative interactions. Across 858 DEs, 227 evidenced collaboration over seven dimensions. There were few disagreements, one resolved in Bravo and one unresolved in Echo. There were 29 instances of repair, i.e., players correcting misunderstandings. Message Categorization

Bravo

Charlie

Delta

Echo

TOTAL

Discourse Events

142

227

324

165

858

Points

200

250

Repair

Self-correction

Answering Question

Agreement

Suggestion

Helpfulness

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 22

Acknowledgement

Total Collaboration

227

Table 5. Collaboration instances over seven dimensions. Addressivity usually reflected collaboration. It occurred in Bravo 4 times; Charlie, 7; Delta, 21; Echo 14. In Echo, in at least two cases, addressivity was confrontational. In Delta, it always served team goals. When players perceived that Kilo_Delta was not participating in discourse, distorted addressivity paired with letter reduplication for emphasis, non-aggressively reengaged him. 15:26:05 Quebec_Delta KILOOOOO 15:26:08 Zulu_Delta

Kilooooooooooo

15:26:16 Kilo_Delta

what happend

15:26:18 Zulu_Delta

Snake snake!!!!!!!!!!

15:26:18 Quebec_Delta haha 15:26:21 Zulu_Delta

teleport to bank

15:26:36 Zulu_Delta

we are going to divide the map up

15:26:47 Yankee_Delta ill take the northeast quad 15:26:48 Kilo_Delta

i'm in the bank

There was no cheering in Bravo. Cheering occurred in Charlie (8 times; 5 by leader Lima_Charlie), Delta (7/3) and Echo (6), once by leader Golf_Echo and four times later by leader X-Ray_Echo. Lima_Charlie cheered immediately after a puzzle submission, â&#x20AC;&#x153;yay 500 points for the quad good job guys.â&#x20AC;? In other conditions, after submitting, players remained seriously focused on solving more puzzles. Cheering often mimicked spoken AmerEnglish, using terms such as yeah and sweet. Information Sharing Informational DEs typically reported or requested locations or the status of finding pieces. With 236 solicited and unsolicited informational DEs, almost 25% of DEs were informational; Charlie

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 23

(76) sent the most and Bravo (42) the least. Questions were answered 99 times; the average response time was 23.24 seconds. Bravo

Charlie

Delta

Echo

TOTAL

Discourse Events

142

227

324

165

858

Points

200

250

Status report

173

Location

Piece

149

Information Need

Total Statements

111

118

370

Confusion

Confirmation

Location

Pieces

Status

Instruction

Other

Total Questions

186

Statements

Questions

Table 6. Information-sharing discourse events. Strategy The 361 strategy DEs either directed or suggested that players take certain steps. Reflecting an understanding of communicationâ&#x20AC;&#x2122;s importance, Delta (250 points) had 21 and Echo (250) 23 DEs about communication strategy. Bravo (0) had two and Charlie (200) eight. Bravo

Charlie

Delta*

Echo

Total points

200

250

Total DEs

142

227

324

165

858

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 24

Game plan

Puzzle solving

Question (what to do) 15 Communication strategies

Totals

361

Table 7. Strategy DEs. Fun, Emotions and Emphasis Khoo and Zubek (2002) find that emotional involvement is key to gameplay enjoyment. Observations during gameplay and post-game interviews disclosed that all players enjoyed PanelPuzzle and were satisfied with their gameplay experience. Prior to gameplay start, discourse manifested fun, but once gameplay started, discourse rarely reflected fun. Two of 142 DEs evidenced fun for Bravo; 2/227 for Charlie; 2/324 for Delta. In Echo, DEs showed no evidence of fun. SMEs identified 38 expressions of emotions. Charlie celebrated twice after puzzle insertion. Teams showed excitement 16 times: Bravo; 1; Charlie, 5; Delta, 7; and Echo, 3. Charlie and Delta each expressed surprise once, using words from spoken AmerEnglish (yikes). Emotions did not rise as players neared goal accomplishment. However, as game end approached, emotions rose, use of upper case increased and profanity occurred. In Echo, discourse reflected discord. In 19 cases, the distinction between frustration and anger was not clear: Bravo, 6; Charlie, 2; Delta, 3; and Echo 8, e.g., 16:43:47 Tango_Bravo Now please do it in the GM forum so we don't have to do double work and time is running out!!! 16:44:24 Tango_Bravo PLEASE!!! Exclamation points emphasized DEs in Bravo (2); Delta (2); and Echo (2). Emphasis with all upper case letters occurred in Bravo (6); Delta (2) and Echo (1), e.g., as time ran out, 16:52:53 Tango_Bravo WE ONLY GOT 10 MINUTES LEFT, WE GOTTA TRY TO ORGANIZE SOME STUFF

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 25

Non-Standard Vocabulary Contrary to the literature, players did not often use vocabulary common to social IM and TM. Abbreviations were rare: Bravo, 0; Charlie 5; Delta, 7; Echo, 6. Bravo had the only case of an acronym, lol. Indeed, players often chose to spell out words for clarity. Equations in the puzzles contained the values, b2 and y2; players invented words, bsquared and ysquared, to avoid confusion from typing b2 or y2. Profanity occurred only once in Bravo and three times in Delta. The main evidence of onomatopoeia was four uses of haha. Other than the consistent use of numerals instead of written numbers, substituting symbols for words was rare. There were only four occurrences of using + for and, all by Quebec_ Bravo. Only Tango-Bravo used 2 for to, â&#x20AC;&#x153;try 2â&#x20AC;?. Golf_Echo and Yankee_Delta used # for number. Within 858 DEs, there were only nine emoticons (Bravo, 2; Charlie, 4; Delta, 2; Echo, 1) and two coded gestures, both in Delta. Players did not express gestures through alphanumeric characters and punctuation marks. DEs mimicked spoken AmerEnglish, using ok, yea (truncated yeah) for yes and distortions such as gotta for have to and gotcha for got you [I understand you]. This mimicry occurred in Bravo 14 times; Charlie, 56; Delta 97; Echo, 37. Of lexical truncations (Bravo, 6; Charlie, 7; Delta, 8; Echo, 4), the most common were bros for Mario Brothers, k for ok and u for you. Others, such as cause for because mimicked spoken AmerEnglish. Well-formedness DEs were not well-formed according to classic English grammar rules. Despite consistent access to a full standard keyboard, players rarely used initial caps or end-of-sentence punctuation. Discourse was characterized by missing punctuation (whats on it kilo); alternate uses of punctuation (yes!!!); and irregular capitalization (12 MINUTES). The pronoun I was typically typed in lower case (52 times each in Bravo and Charlie; 63, Delta; and 19, Echo); the only consistent exception was an occasional capital I starting a DE. While question marks occasionally appeared at the end of DEs, periods were rare (Bravo, 1, Charlie 4, Delta, 2 and Echo, 1). Players used ellipses, e.g., to indicate they were thinking, but often truncated them to two periods.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 26

DEs were concise. DEs containing at least one lexical deletion, e.g., word omission, were common (Bravo, 33; Charlie, 67; Delta, 97; Echo, 54). Players, having invented terms such as 3 piece or 3piece, then deleted piece, typing only 3. Although Lewis and Fabos (2005) observed that spelling correctness was important, we noted 415 misspellings: Bravo, 106 over 962 words; Charlie, 109/1,303; Delta, 139/1,736; and Echo, 61/1,040. Typing errors usually remained uncorrected. The SMEs reported a chatroom convention for marking self-corrected typos by prefixing or suffixing an asterisk. This clarification strategy occurred once in Charlie and Echo and five times in Delta, but never in Bravo. 16:16:19

Quebec_Delta at trai

16:16:24

Quebec_Delta *train

Discussion Discourse analysis showed that Delta, the winning team, was more active in sending DEs and the characteristics of its DEs differed from those of other teams. Bravo, the team with the lowest score, typically performed the lowest in taxonomy categories. H1. Text chat discourse will both reflect and impact team dynamics. DEs proved integral to success because they reflected and drove community dynamics. DEs reflected interactions among teammates and teammatesâ&#x20AC;&#x2122; attitudes toward each other. This occurred despite the fact that, in PanelPuzzle, DEs competed with gameplay while facilitating it because players could not control avatars and chat simultaneously. Delta, the most efficient and effective team in using DEs to strategize and to share information, was the most efficient and effective in solving puzzles. Echo experienced distraction disputing leadership, losing time and failing to solve another puzzle in time. Teams built collaborative knowledge in two ways. All teams used text chat primarily for tactical discourse. While this contributed to building collaborative knowledge, text chat scrolling out of view had a transitory aspect that did not lend itself to storing collaborative knowledge. Therefore, the players forum became the locus for

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 27

building collaborative knowledge. Consequently, there was a strong relationship between text chat and forum. Players used text chat to remind each other to post to the forum and text chat referred to forum information during piece assembly.

Figure 5. Differing ratios between number of words and number of multi-functional DEs reflect discourse differences between Bravo (0 points) and Delta (250 points).

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 28

Discourse analysis indicated that efficiently sharing and coordinating knowledge contributed to winning and that the content and sequences as well as the quantity of DEs promoted success. Out of 858 DEs, 285 efficiently fit into more than one taxonomy category (Bravo, 71; Charlie, 68; Delta, 68; and Echo, 78). Discussing collaborative strategy fostered success by empowering teamwork. The number of DEs that discussed strategy aligned with teams’ rankings: first place Delta (250 points), 60 DEs in strategy categories; second, Echo (250), 46 strategy DEs; third, Charlie (200), 45 strategy DEs; fourth, Bravo (0), 35 strategy DEs. We analyzed DE length and number to discern if there were discourse patterns that evidenced efficiency and effectiveness. Comparing Bravo, the least successful team, and Delta, the most successful, showed that Bravo had the least DEs (142) and Delta (324) the most, but their average DE lengths were approximately equal. Average and peak functions (taxonomy categories) per DE were higher for Bravo than Delta. So, Bravo was the most efficient team in terms of using the least number of words to average the highest number of multi-functional DEs. However, Delta sent more DEs, thus communicating more functions overall throughout gameplay to be more effective. DEs showed that teamwork characterized gameplay. Once gameplay started, there was no evidence of personal goals. Even in the Echo leadership dispute, interviews disclosed that each player felt he was right and that his leadership would benefit the team by serving the team goal of earning points. H2: When a team leadership role is played, text chat discourse proceeding from the team leader will promote and foster the team’s goal to win the game. Teams where discourse analysis showed firm leadership earned more points than Bravo where leadership was less obvious. Leadership emerged in all conditions. As Sudweeks and Simoff (2005) note, leadership emerged early. Authoritative and goal-oriented DEs with the leader as source were numerous. Team leaders’ DEs promoted and fostered winning by asserting authority; providing direction; building community; coordinating team activities; promoting building of collaborative knowledge; and boosting morale. (It is interesting that there was no cheering in Bravo.) DEs to these ends were

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 29

both informational and instructional. As expected, teammates’ discourse was goal-oriented, reflecting that of team leaders. The GM Coordinator organized team efforts by focusing discourse on completing and submitting puzzles. DEs disclosed emergent leadership in RV conditions and reflected incidents where RM leaders exerted leadership. Discourse reflected evolving and shared leadership. In Bravo (RV), where strong leadership did not emerge, discourse did not manifest an organized strategy to win the game. Although a GM Coordinator arose in Bravo (RV), any player could use the GM forum to communicate with the GM. In Echo (RV), where leadership was debated and anyone could use text chat to communicate with the GM, discord arose early and persisted until game’s end; a strong community did not develop. Discourse reflected roles. Murray (1997) views in-game roleplay as assuming an identity and gameplay discourse as a narrative of players’ roleplaying experiences. Immersion in gameplay identity parallels immersion in the game; disrupted narrative can signal that a player has shed an assumed role. PanelPuzzle DEs evidenced this phenomenon. Team leader X-Ray_Echo interrupted gameplaycentered discourse and acted out-of-role to chide Zulu_Echo, using language that contrasted with that of other gameplay DEs. 11:53:31

X-Ray_Echo

zulu, go away and look for more pieces

11:53:45

X-Ray_Echo

dude.. shut up

DEs disclosed that players tended to assume multiple roles even in RM conditions where this was prohibited, e.g., in Charlie (RM), Lima_Charlie, the designated GM Coordinator, filled that role, but also organized puzzles; started forum trends “to keep track;” and discussed locations. Tango_Bravo, while acting as leader, reported his location and discussed puzzle pieces.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 30

H3. Text chat discourse during gameplay will be predominantly serious. Text chat focused on successful gameplay, rarely expressing fun or emotions. At least four factors contributed to the serious nature of discourse. Gameplay time limitations left little time for joking and banter. Demand effects could have dampened spirits. Collaboration required serious discourse. Serious discourse served the serious goal of winning points. H4: The structure and vocabulary of text chat discourse will resemble the structure and vocabulary of the electronic messages that digital natives typically exchange. Once gameplay started, PanelPuzzle text chat displayed few of the expected properties although, prior to gameplay, as players explored the VE and practiced manipulating their avatars, DEs conformed to the literature. Linguistic evidence of fun diminished in three conditions and disappeared entirely in Echo. Structures and vocabulary common in CMCs were present only to a limited extent that contradicted expectations. A lack of well-formedness, usually to promote efficiency, was the principal characteristic of most DEs. On the levels of the word and of DEs, an abandonment of spelling and syntactic conventions, e.g., use of all lower case and omitting punctuation, promoted fast typing. This strategy served players’ goals to quickly communicate and coordinate. Despite linguistic distortions and adaptations, Steinkuehler (2006) asserts that gameplay text chat achieves the same full range of purposes as off-line written discourse. Our observations aligned with this. We posit a reason: when parts of a DE were malformed, others were not. Players were able to derive meaning from the context provided by other parts of the DE and the context of shared experience. For example, when Zulu_Echo typed, “i founf 5 pice near train station” teammates understood founf and pice because (1) other words in the DE, e.g., near train station, provided semantic and pragmatic clues that a piece was found; (2) finding pieces and reporting their locations were shared game goals; and (3) the term, 5 pice reflected vocabulary teammates had invented. Players effectively communicated strategies, tactics and needed information.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 31

Rather than using the expected vocabulary, DEs combined innovative linguistic strategies to accomplish efficient and effective discourse. Brief multi-functional DEs moved discourse efficiently. Communication succeeded even when one DE contained a variety of aberrations. Zulu-Delta’s “dint u say that” elicited an immediate turn-around response that verified the DE’s effectiveness. Players invented efficient ways to discuss puzzles. Metonymy, calling something by the name of one of its attributes, occurred in all conditions except Bravo: “we need one more mario” (Charlie); “how need mario?” (Delta); and “but is is mario” (Echo). Bravo players referred to “a super mario picture” and “a mario picture,” approximating metonymy by using the attribute name as an adjective. Because PanelPuzzle discourse focused on winning, it manifested expected characteristics of social digital discourse only to a limited degree even though it reflected digital natives’ practice of making others aware of what they are doing and their desire to know what others are doing. Devices such as abbreviations were rare. There was little evidence of turn-change signals. IM mechanisms such as ellipses and improper punctuation and capitalization were common. Short, rapid, sequential DEs from a single source kept discourse on topic. Even when discourse was disconnected with topic changes, it maintained a collaborative focus to win points. DEs had few of some expected visual aspects such as emoticons, but were rich in others such as untraditional capitalization. Players did adopt vocabulary customized to the game, e.g., role titles and panel, a term not usually associated with puzzles. They also invented terms such as 3 piece and 5 panel. The most successful team, Delta, were the most efficient communicators, e.g., with the most lexical deletions, lexical truncations, and mimicry of spoken AmerEnglish. Delta was also the most efficient, e.g., with the most self-corrected typos, and the most functions communicated. Conclusions This study analyzed 858 discrete DEs recorded during a controlled laboratory experiment. It is framed within two taxonomies that can be generalized for adaptation to general gameplay digital discourse. Discourse analysis disclosed that digital natives adapted lexical and syntactic

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 32

conventions, demonstrating and using to their advantage the malleability of natural language. In contorting typed AmerEnglish to build and sustain community and pursue team goals, players did indeed use a distinct register. Its DEs disclosed properties of digital native goal-oriented digital discourse, differing from social digital discourse described in the literature. It sacrificed traditional well-formedness to improve efficiency. Discourse analysis indicates that the more efficient and effective the communication among teammates, the more successful a team was. Doell (1998) posits that IRC conventions, e.g., unconventional spelling and emoticons, have no precedent in any register, and thus can only have emerged from the communicatorsâ&#x20AC;&#x2122; creativity and innovation. We observed that gameplay digital discourse also applied creativity and innovation to create a distinct register. Possibly, these phenomena constitute a step in the evolution of AmerEnglish, perhaps toward a trend where efficiency matters more than form. Digital discourse as expressed by digital natives in text chat differs from traditional written discourse, merging conventions from several digital discourse registers, abandoning others and inventing new conventions. Although it sometimes mimics spoken language, this register has no need for the full panoply of visual references and details required for successful spoken discourse. Its concise, truncated, malformed DEs communicate efficiently and effectively. Players coordinated collaboration, communicating their intentions, facilitating gameplay and sharing outcomes of their efforts. Murray (1997) foresaw gameplay discourse as narrative, the telling of the story of gameplay. PanelPuzzle DEs did indeed disclose the story of playersâ&#x20AC;&#x2122; experiences in Peninsula City. They also disclosed the power of digital discourse. We have seen that digital discourse, even with frequent topic changes, typos and syntactic and lexical distortions, is an effective communication mechanism. We posit that, in the case of PanelPuzzle gameplay discourse, reasons include shared understanding and shared references built on the shared experience of digital nativehood. As Lakoff and Johnson (1980) demonstrate in their seminal work, such sharing empowers communication. Our study disclosed that text chat relies more on context than syntax. The abbreviation bf can mean best friend or boyfriend. Even when chat syntax is the same, context imposes different meanings on bf. Without signals such as body language, gestures or tone of voice, when source and sink do not share context, the communication channel is obstructed and

misunderstanding can result, e.g., a source may type "ok" to convey

disagreement and a desire to end discourse, while a sink interprets "ok" to signify agreement.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 33

PanelPuzzle players conveyed shared context through DEs. When Lima_Charlie abruptly changed topics, requesting “a desc on the forum,” teammates knew the lexical truncation desc meant a puzzle piece description. As digital natives, they knew a forum was a logical place for this description. As goal-oriented teammates, they knew winning required organizing descriptions of pieces. The concept of a distinct register has implications for natural language research. What are the impacts of ill-formedness on machine translation? What are the impacts of this register and the communication behaviors it reflects on design of workplace tools and online training for digital natives? Such questions argue for continued research into digital communication among digital natives. We look forward to the community’s insights.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 34

BIBLIOGRAPHY Aoki, P. M. & Woodruff, A. (2005). Making space for stories: ambiguity in the design of personal communication systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 181-190. Brown, G. & Yule, G. (1988). Discourse analysis. Cambridge: Cambridge University Press. Cook, W. (1989). Introduction to tagmemic analysis. New York: Holt, Rinehart and Winston. Doell, W. (1998). The internet relay chat (IRC): Linguistic perspectives. Retrieved August 25 2009 from http://home.allgaeu.org/ndoell/work/ircpaper.htm Gergle, D., Millen, D. R., Kraut, R. E., & Fussell, S. R. (2004). Persistence matters: making the most of chat in tightly-coupled work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM: New York, 431-438. Greenfield, P. M., & Subrahmanyam, K. (2003). Online discourse in a teen chatroom: New codes and new modes of coherence in a visual medium. Journal of Applied Developmental Psychology, 24(6), 713-738. Grinter, R. E., Palen, L., & Eldridge, M. (2006). Chatting with teenagers: Considering the place of chat technologies in teen life. ACM Trans. Comput.-Hum. Interact. 13, (4), 423-447. Herring, S. (1999). Interactional coherence in CMC. Journal of Computer-Mediated Communication,

4(4).

Retrieved

August

14,

2009

from

http://jcmc.indiana.edu/vol4/issue4/herring.html ISO (International Organization for Standardization). (1998). Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. Khoo, A. & Zubek, R. (2002). Applying inexpensive AI techniques to computer games. IEEE Intelligent Systems, 17(4), 48–53. Lakoff, G. & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press. Lewis, C. & Fabos, B. (2005). Instant messaging, literacies, and social identities. Research Quarterly 40(4) pp. 470–501. Maxey,

(2008)

List

spelling

errors.

Retrieved

http://gregmaxey.mvps.org/List_Spelling_Errors.htm

March

11,

2009

from

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 35

Maybury, M. (2001). Collaborative virtual environments for analysis and decision support. Commun. ACM, 44(12), 51–54. Murray, J. (1997). Hamlet on the holodeck. Cambridge, MA: MIT Press. O’Connell, T., Choong, Y., Grantham, J., Moriarty, M. & Wong, W. (2008). From a video game in a virtual world to collaborative visual analytic tools. Journal of Virtual World Research,

1(1).

Retrieved

February

2009

from

http://journals.tdl.org/jvwr/article/view/292/246 O’Connell, T., Grantham, J., Workman, K. & Wong, W. (2009). Leveraging game-playing skills, expectations and behaviors of digital natives to improve visual analytic tools. Journal of Virtual

World

Research,

2(1).

Retrieved

April

2009

https://journals.tdl.org/jvwr/article/view/604/471 Paolillo, J. C. (2001). Language variation on internet relay chat: A social network approach. Journal of Sociolinguistics 5(2), 180-213. Prensky, M. (2004). The emerging online life of the digital native. Retrieved April 9, 2009 from http://www.marcprensky.com/writing/PrenskyThe_Emerging_Online_Life_of_the_Digital_Native-03.pdf Steinkuehler, C.A. (2007). Massively multiplayer online gaming as a constellation of literacy practices. E-Learning, 4(3), 297-318. Steinkuehler, C.A. (2006). Massively multiplayer online videogaming as participation in a discourse, Mind, Culture, & Activity, 13(1), 38-52. Steinkuehler, C. A. (2003). Videogaming as participation in a Discourse. Paper presented at the American Association for Applied Linguistics Annual Conference (AAAL), Arlington VA.

Retrieved

March

10,

2009

from

http://inkido.indiana.edu/onlinecom/Steinkuehler.pdf Sudweeks, F. & Simoff, S. (2005). Leading conversations: Communication behaviours of emergent leaders in virtual teams. Proceedings of the Thirty-Eighth Annual Hawaii International Conference on System Sciences, 3-6 January, Big Island, Hawaii, IEEE Society, CD-ROM. Tychsen, A., Smith, J., Hitchens, M. & Tosca, S. (2006). Communication in multi-player role playing games – The effect of medium. S. Göbel, Malkewitz, R. & Iurgel, I. (Eds.) TIDSE 2006, LNCS 4326. Heidelberg: Springer-Verlag Berlin , pp. 277 – 288.

Journal of Virtual Worlds Research - Gameplay Discourse Analysis 36

Werry, C. C. (1996). Linguistic and interactional features of Internet relay chat. In Herring, S. (Ed.), Computer-mediated communication: Linguistic, social, and cross-cultural perspectives (pp 47-63). Amsterdam: J. Benjamins. Zubek, R., & Khoo, A. (2000). Making the human care: On building engaging bots. AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment. AAAI Technical Report SS-02-01.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 Applying Constant Comparative and Discourse Analyses to Virtual Worlds Research Peter Leong University of Hawaii at Manoa, United States Samuel R. H. Joseph University of Hawaii at Manoa, United States Rachel Boulay University of Hawaii at Manoa, United States Abstract This paper presents the basics of the constant comparative analysis and discourse analysis methods to research in virtual worlds. Our data sources include video recordings of in-world class interactions; artifacts and documents such as studentsâ&#x20AC;&#x2122; blog, and asynchronous discussion postings. For data analysis, we use the constant comparative method as a tool for inductive analysis, and discourse analysis as a tool to discover patterns in discursive practices. The constant comparative method was originally developed for the use in grounded theory methodology, but is now more widely applied as a method of analysis in qualitative research. It requires the researcher to take one piece of data and compare it to all other pieces of data that are either similar or different, gradually identifying the salient differences. In contrast, discourse analysis examines the way in which sentences are combined in larger linguistic units, such as conversational exchanges or written texts. The critical analysis of discourse helps us extend and specify themes generated during constant comparative analysis. Keywords: virtual worlds; comparative analysis; discourse analysis

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 4

Applying Constant Comparative and Discourse Analyses to Virtual Worlds Research Virtual worlds are online environments, usually understood to provide a 3D environment with representations of multiple users. They are also referred to as three-dimensional Multi-User Virtual Environments (3-D MUVE), and examples include Second Life, World of Warcraft, Star Wars Galaxies and Blue Mars to name a few. Virtual worlds provide opportunities for many types of interaction that are not possible in purely text based or 2D environments. Virtual worlds are commonly used for recreation, commerce, education, and collaboration. While this is true of other online environments, it takes a 3D environment to allow multiple users to interact with multiple other 3D objects, each user perceiving the sets of available artifacts from their own unique perspective. In particular, virtual worlds support a form of deixis that is often absent from other environments. In a virtual world a user can pick up an object, display it to other users and refer to it in text and audio chat. This can have a powerful grounding effect that makes virtual worlds attractive for many pursuits. The fact that virtual worlds are used at all suggests that some individuals are deriving some sort of benefit from their online 3D pursuits, but the question remains whether the goals of those pursuits are being achieved as effectively in a virtual world as they would be in real life or in some other form of online environment. For example, given that educators use virtual worlds to deliver instruction, a natural question arising is how adding a virtual world component to a course affects the learning outcomes of the students. A critical research question for the field is: What analytical methods are best suited to answering the different domain specific research questions we might have about virtual worlds? Examples of domain specific research questions would include the following: What is the pedagogical effectiveness of virtual worlds when used for educational purposes? How effective are virtual worlds for use in supporting online research and collaboration? or What are the psychological consequences of extended interaction in virtual worlds? While this is a critical endeavor, much of the nitty gritty specifics of how to conduct rigorous data collection in virtual worlds still being understood, a goal this particular issue of the journal of virtual worlds' research is aiming to address. This study considered two different methods, those of constant comparative analysis and discourse analysis. There are, of course, many other methods that could be applied, but here the investigators hope to provide insight into the practical considerations and relative effectiveness

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 5

of these two methods with regards to different research questions. In particular the case study itself is focused on the following research questions: 1. What were student reactions to an experimental course taught online in Second Life? Using constant comparative method to determine student reactions, frustrations emerged as a salient theme. Therefore, the researchers employed discourse analysis to further investigate: 2. How the frequency of student frustrations changed throughout the course? and 3. What discourse patterns of response were typical when a student expressed a frustration? Background The case study described in this paper concerns Second Life, a virtual world where residents can explore, socialize, participate in individual and group activities, and create and trade virtual property and services with one another. SL includes three-dimensional modeling tools that allow residents to build scriptable virtual objects, which can be bought and sold in the SL currency, Linden dollars. As of October 2009, there were just over 16 million registered SL users and although there are no reliable figures for actual long term consistent usage, on average, 38,000 residents were logged in at any particular moment (Second Life, 2009). Due to the popularity of SL, a large number of colleges and universities, such as Cornell University, Harvard, Duke, Ohio State, University of California, Davis, and Berkley, Virginia Tech, and MIT have created virtual campuses and are offering courses in SL. For example, Harvard began offering a law course on SL in fall of 2006 (Foster, 2006). At least 300 universities around the world now teach courses or conduct research in SL (Michels, 2008). The case study concerns online courses taught using the University of Hawaii (UH) SL island, modeled as a replica of part of the UH Manoa campus and created in 2008. The island is available for use by all UH staff student and faculty for use as a home base, for virtual world projects and as a teaching space. Courses are being taught in Psychology, Library Science, Second Language Studies, Education and Music while research is being conducted in each of these areas, and also Computer Science and Aquaculture. Space is made available on UH SL

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 6

Island for any staff, student or faculty of Hawaii-based educational institutions interested in conducting research, teaching or presenting themselves and their work to their peers. While a considerable amount of research has been done on the sociology of virtual communities and virtual worlds, the body of knowledge on educational studies in virtual worlds is still at a relatively early stage (Campbell & Jones, 2008). Moreover, existing research tends to be descriptive in nature. For example, Johnson and Levine (2008) described their experiences and studies in virtual learning offered through the New Media Consortiumâ&#x20AC;&#x2122;s (NMC) Second Life campus. The NMC provides an immersive virtual learning experience which includes the teaching of atomic models by an Einstein avatar, teaching of Steinbeckâ&#x20AC;&#x2122;s of Mice and Men through role play, role playing and reenacting of famous court cases, language learning, and cultural learning. They assert that SL provides excellent opportunities for immersive learning, such as role playing and case studies. Other studies attempt to highlight the potential of SL for teaching. Jarmon, Traphagan, Mayrath and Trivedi (2009) demonstrated the effectiveness of the SL environment for a projectbased experiential learning approach, particularly because students were able to learn by doing and used what they learned in real life. Specifically, the students collaborated with real and virtual groups to create a Second Life version of two low-income model homes using sustainable design features designed by a real world non-profit agency. More recently, studies are beginning to address virtual world pedagogy. Twining (2009) contends that pedagogy in virtual worlds helps students be collaborative versus individualistic; learn about, by doing, and by becoming; and that avatars can be used as a reflective tool. Warburton (2009) argues that SL impacts pedagogy by allowing students and instructors to opportunity to experience rich interactions, visualization, authentic content and culture, identity play, immersion, simulation, community presence, and content production. He cautions about several barriers inhibiting wider spread adoption of SL, which include: technical, identity, culture, collaboration, time, economic, standards, scaffolding persistence and social discovery. In addition, Warburton highlights the current trends in virtual world development, which includes the development of open source virtual worlds and portable avatars between various virtual world platforms. Case Study

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 7

The case study presented in this paper is a six-week graduate educational technology elective summer course designed to provide students with the opportunity to explore the use of virtual worlds, primarily SL, for teaching and learning. Students identified and analyzed emerging research, as well as tools, pedagogy, SL teaching environments, content resources, and assessments for virtual world teaching. The course provided hands-on experiential learning and was designed to enable graduate students to design, develop, and evaluate instruction in SL. In addition to leveraging SL as a distance learning delivery tool, the students researched the various ways that SL can be leveraged for instructional purposes, such as exploring other educational SL builds (virtual field trips) and evaluating the design of educational simulations in SL. The course also covered basic fundamentals of building in SL with a focus on building simple educational objects (how to deliver content), as well as on how to promote interactivity in SL. There is a strong research component with students being required to research and compile a list of SL educational resources (both in-world and online) as well as develop an annotated bibliography of research on virtual world teaching and learning. There were also asynchronous components of this completely online course. UHâ&#x20AC;&#x2122;s course management system Laulima (Sakai) provided a document repository, while weekly asynchronous online discussions took place on a Ning social networking site where students could also post photos and videos. Furthermore, students were required to blog their SL learning experience using blogging tools of their choice. The final project was the creation of a class module by student teams. Data Collection The current study used two data sources: 1. classroom observations or video recordings of SL in-world class sessions 2. student discourse in written text, including student blogs, chat box text and asynchronous discussion postings. Requesting Human Subjects Approval In order to get institutional approval for the study we submitted the usual documentation to request approval for the participation of human subjects in a research project. This type of study, including recording class sessions and examining student discussion board postings, blogs and

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 8

text box chats, would not usually raise concern. However, the Committee on Human Subjects (CHS) was sufficiently surprised by the request to record video in an online world that they phoned to confirm details of the study. Much explanation of what is a 3-D MUVE is and how a course can be taught in this environment was needed. The committee said that the request was the “most eye-opening that month” and approved the study. One concern was getting students signatures on the experimental participation forms, but the CHS indicated that manual signatures were not required as long as the experimental participation forms were distributed. These forms were distributed via Second Life note card at the start of the course. Learning to Record Classroom Observations in SL Classroom observation is a common approach used to investigate what transpires during a class session. The training of individuals in conducting classroom observations may be covered in research methodology courses at many universities (see Patton, 1990, chapter 6, for a good introduction); but how to conduct classroom observations in courses being conducted entirely in 3-D MUVEs is far from well understood. We embarked and piloted different recording techniques in this 3-D MUVE. The desire was to record class sessions in SL in their entirety. In real life, class sessions are often video-taped because of the richness of data collected that can be reviewed later. As research in classroom behavior in 3-D MUVEs is still very exploratory, we wanted the option to return to any class session and re-examine the data, including the interaction of avatars, individual utterances of students and their relationship to spatial features in the 3-D MUVE, to name a couple of examples. In order to study students’ interactions and experiences in SL, we embarked on recording the class sessions initially by using a “non-participant” observer. At the beginning of the 6-week long course an avatar was introduced to students as an “observer” who would be recording the class sessions with their permission. In this same session students met an avatar who was the course instructor and another avatar who was the teaching assistant for the course. The SL in-world class sessions, totaling approximately 30 hours, were videotaped from a computer screen using the video-capturing software ScreenFlow. Various issues arose in the process of trying to video capture the online action. These challenges are shared in some depth as we hope our experience can inform others wishing to adapt common methodological approaches, such as classroom observations, to research within 3-D MUVEs. We wrestled with

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 9

understanding how these classroom observations in-world were similar and distinct from our prior expertise in using these strategies in real-world settings and share our practical steps to improve future recording attempts. Video-capturing Software While recording both the video and audio was challenging, success was achieved with ScreenFlow, after several other software products were tried. The Second Life client had originally included a “Movie to Disk” video recording option, but this was removed in 2008 due to technical problems. As an alternative Linden Labs provided a list of third party screen capture products (Second Life, 2010) for use in recording video of Second Life. A number of these (CamStudio, Snapz Pro X) were tried and led to only partial success in capturing classroom activity. In general the software would capture the screen activity well, but the main problem encountered was simultaneously capturing the audio. Initial recordings on both Windows Vista and XP appeared to suffer from software/soundcard incompatibilities, so that video was recorded but not audio. This problem was circumvented by playing the audio from the speakers and recording directly through an external microphone, but this led to the problem that other ambient noises were recorded. However other problems were encountered such as windows crashing after recording sessions of over an hour. Similar problems were not encountered when recording on the OSX platform, and ScreenFlow gave immediate options for recording screen, external audio and computer audio, making it the clear winner in terms of simplicity and reliability. The creation of large video files was always a storage issue, but the simple solution was a large capacity external storage drive. Nonetheless manipulating these files with other software to extract clips, or simply for transcription could lead to long file-processing wait times. A single two-hour class could easily generate to a multi-gigabyte ScreenFlow file. The general practice followed was to generate a half size .mov file from the ScreenFlow file (export function built into ScreenFlow) which would get the file size down to under a gigabyte with only an acceptable loss of quality. This file could then be used for transcription purposes, manipulated with QuickTime to extract clips, and the original multi-gigabyte ScreenFlow archived on the external drive. While ScreenFlow was successful for our needs, there were limitations imposed by the nature of the 3D-MUVE setup and the resources available to us. Ideally we would have been

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 10

recording not just from a single observer computer, but from all of the participants’ computers, capturing the interface views of instructors and students. However this would have required installing video recording software on all of the participants’ computers, which was impractical in the case study, largely because of the additional burden this would have placed on student’s already coping with the SL client software, but also because ScreenFlow is OSX only, and does place a non-trivial processing burden on the recording machine. Several students’ computers were already struggling with the SL client, making the running of additional software impractical. Practical Challenges While Recording in SL Starting the recording software was a trivial operation (ScreenFlow starts up asking the user what they would like to record), however since ScreenFlow and the SL client would both be running for a couple of hours it quickly became clear that it was expedient to restart the recording computer before attempting to record a session in order to maximize the available computer processing and memory resources, as well as checking the presence of sufficient disk capacity to store the initial multi-gigabyte ScreenFlow file. While being used for recording it was generally impractical to use the recording computer for much else other than making sure that the Second Life “camera person” was in the correct place. The “camera person” looked just like a normal, albeit inactive, SL avatar, and there was no particular indication that the particular avatar was videotaping or what they were recording. Both the computer recording the action and the Second Life client itself had timeouts that had to be disabled. In particular after a short period a Second Life avatar will slump over (see Figure 1a) in Away From Keyboard (AFK) mode which may suggest to the viewers that the “camera person” has gone to sleep. Further inactivity causes the Second Life client to log out. The solution was to use a camping setting in a Heads Up Display (HUD) called MystiHUD purchased within Second Life which prevents SL AFK status, and using Caffeine software on OSX to prevent the computer from going to sleep. While students seemed very comfortable with the recording of their class in SL, but the main challenge was the non-participant observer or “camera person” being “left behind.” Leaving the computer to record locks the view in SL, so if the focus of attention moves, for example, the students or instructor go somewhere else then they go out of shot, even if their audio is still recorded. In other words, if the instructor took students on a field trip where they left the class space and traveled to another island in SL, the “camera

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 11

person” was not traveling with the class or if the instructor walked to another area or the student avatar traveled out of the view of the observer, the field of view of the observer could be easily obstructed.

Figure 1(a,b). Screen shots of basic challenges while recording classroom observations in SL. a) Avatar slumping in AFK status, b) recording focused on empty classroom when participants had moved to another area.

This main challenge may be overcome through training and practice by those who wish to act in the traditional role of a non-participant observer recording class sessions in SL environments. However, we trialed another approach and had the camera person sit in a follow chair attached to the teaching assistant. In essence we converted the non-participant observer into a participant observer. Participant observation is when the observation is conducted by a participating member of the course. In real life, participant observation often reduces bias related to unnaturalness created by the presence of the non-member of the class. While the comfortableness of the students was not at issue, the non-participant was frequently “left behind” as the class traveled or shifted their physical space. A special chair, also available as part of the MystiHUD system, was set to follow the class teaching assistant and thus the view recorded was at least partially controlled by the teaching assistant. This allows situations like those shown in figure 1b above to be avoided, although other issues were now encountered. The main time that the class participants would move would be when the class took a field trip to another SL location. This generally involved teleporting and it seemed that the MystiHUD follow chair would generally not make it through the teleport process. In general field trips lasted between 10 and 30 minutes, and so the teaching assistant and the person responsible for recording the class session coordinated to check if a field trip would take place, and if so arrange for the follow chair to be set up once the field trip teleport had been completed. Overall this improved the data

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 12

collection so that fewer empty classrooms were recorded, and more fieldtrips were captured; however it became clear that ideally the camera person would be actively focused on the recording process for the entire duration of class movement or field trips. The teaching assistant pulling the follow chair could not see the recording view and so the class’s focus of activity might be missed if the camera person was not actively engaged to focus the recording SL client on whatever happened to be the center of attention (see Figure 2).

Figure 2. Screen shots of follow chair challenges while recording classroom observations in SL, here the camera person is capturing a view of some participants who are focused on activity taking place to the right that is not captured by this view, which had been set up and left as the optimal view for an earlier activity.

Transcribing Class Sessions The video-recorded class sessions were transcribed by two graduate students. One graduate student participated in the course as a teaching assistant. The other graduate student was from the University’s College of Language, Linguistics, and Literature and therefore had substantial training in transcription and provided an alternative outside perspective. The lead researcher met weekly with the graduate students to discuss progress and arising questions. The graduate students encountered several interesting issues as they began transcribing. One initial challenge was the difficulty in ascribing the audio to the correct avatar. The non-participant transcriber had difficulty identifying which avatars were speaking because they were either out of the view of the camera or the “talk” green icon flashing above their head was not easily visible. The non-

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 13

participant transcriber also had issues with many SL jargon used in the conversations. All of these points to the need for some training for non-participant transcribers. A recurring issue faced by both the participant and non-participant transcriber was the difficulty in following both the audio and text chat conversations. Very often the audio and text chat conversations were out of sync. In general, text chat comments or questions were addressed later in the audio conversations. Interestingly, although SL is voice-enabled, the majority of student participants preferred using the text chat feature instead of the voice feature. A decision was made to produce one transcript of all class sessions that marked the audio from the course in one color and marked the text from the public text chat in another color. Therefore, a mixed audio and chat transcript was created for all class sessions. It was felt this type of transcript of class sessions would provide a better base for investigating switches from audio to chat for different types of issues. Student Discourse in Written Text In addition to classroom observation data, student discourse in written text was used as a secondary data source. In this study, student discourse in written text consisted of student blogs, the public chat box text and asynchronous discussion postings. One of the assignments throughout the course was the student learning blog where students were required to blog about their SL learning experience using blogging tools of their choice. In addition, students participated in weekly asynchronous online discussions. Logs of public text chat were captured during the classroom sessions and incorporated into the transcriptions of the class sessions as described previously. The individual blogging assignment was designed to help students reflect upon their individual SL learning experience. Students were required to post weekly blogs of their SL learning experience (at least six blog postings in total). The blog postings took the form of free writing, i.e. students were given the freedom to write anything that related to their experience in SL, e.g. their perceptions, successes, problems, and insights. These various components produced student reactions about the course that could be compared to in-class discussions. Data Analysis

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 14

Two methods of data analysis were utilized in this study. Constant comparative analysis was used as an inductive data-driven analysis intended to find recurring patterns. Then discourse analysis was used to look more specifically at discursive practices surrounding a common pattern identified in the constant comparative analysis. Constant Comparative Analysis Data analysis for this qualitative research study was approached through multiple strategies. The researchers and two graduate students examined and reexamined some of the narrative transcriptions of the course sessions and student discourses, including blogs, text chats, and discussion postings to identify patterns and themes. Since the intent of the study was to identify some initial interesting student reactions to a course taught online in SL, only comments related to general student reactions were categorized. The constant comparative method was originally developed for the use in a grounded theory methodology and is now applied more widely as a method of analysis in qualitative research. It requires the researcher to take one piece of data and compare it to all other pieces of data. The qualitative comparative method of data analysis (Ragin, 1987) was used to construct categories and themes that captured the recurring patterns that emerged from the data. The analysis of the data was cyclical, consisting of initial coding, reflecting, and re-reading, then sorting and sifting through the codes to discover patterns and themes. These methods were used to triangulate the evidence of the data (Lincoln & Guba, 1985). Triangulation is a strategy commonly employed to strengthen the robustness of a qualitative study. Four kinds of triangulation contribute to verification and validation of qualitative analysis. Multiple investigators, multiple data sources, multiple theories, and multiple data collection methods to confirm findings are all strategies for reducing systematic bias in the data. In each case the strategy involves checking findings against other sources and perspectives. Triangulation is a process by which the researcher can guard against the possibility that a study's findings are simply an artifact of a single method, a single source, or a single investigator's biases (Patton, 1999). In this study, multiple investigators, data sources and data collection methods were used to triangulate the findings. Discourse Analysis

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 15

The same sources of data were analyzed through a different data analysis approach. Constant comparative analysis was used with an inductive approach to determine initial salient themes of student reactions. Then discourse analysis was used to focus in on one prominent theme and investigate that particular theme in more detail. The investigators felt discourse analysis provided a more fine-tuned approach to understanding a specific issue; whereas constant comparative method was more useful for exploratory research. The investigators hope that sharing these steps of using constant comparative method followed by discourse analysis will provide a useful methodological example to others who are interested in how to sequentially use different methodologies in virtual world research to investigate issues that arise in their experimentation with teaching in 3-D MUVEs. According to Wikipedia, discourse analysis is “… a general term for a number of approaches to analyzing written, spoken or signed language use … The objects of discourse analysis—discourse, writing, talk, conversation, communicative event, etc.—are variously defined in terms of coherent sequences of sentences, propositions, speech acts or turns-at-talk. Contrary to much of traditional linguistics, discourse analysts not only study language use 'beyond the sentence boundary', but also prefer to analyze 'naturally occurring' language use, and not invented examples.” (Discourse Analysis, 2009) Gumperz (1982) contends “that to understand the role of language in education and in social processes in general, we need to begin with a closer understanding of how linguistic signs interact with social knowledge in discourse" (p. 29). The approach in this study mirrors conversation analysis. Having created categories and identified common patterns using the constant comparative method, the next step was to select “frustrations” as a salient theme. The coders then reviewed the entire set of transcripts for all instances related to the category frustrations. The number of references to frustrations were counted over the 6-week course duration. Frequency data over the entire course encourages a more longitudinal look that can help answer whether the frequency of these occurrences is maintained, decreases or increases. Further, the video transcripts reveal which individuals were involved in exchanges allowing one to explore whether individuals are taking on certain roles within the discourse. Specific instances of frustrations were examined for

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 16

the subsequent discursive response to determine the patterns common in how frustrations are responded to in a course setting in SL. Results & Discussion from Constant Comparative Analysis While the preliminary analysis of the data indicates that students’ perceptions and learning experiences were very diverse, three common themes emerged from the data collected and initial analyses. Similarities were found among the students’ experience concerning a) frustrations with technical issues and learning functionalities of SL, b) need for socialization and sense of presence, and c) appreciation for the potential of SL for teaching and learning. Technical Issues and Steep Learning Curve. One major recurring theme in the data sources was about students’ frustration with technical issues and the steep learning curve of learning the functionalities of SL: [Blog posting]: I restarted the computer, tried to log back in the SL – SL crashed again. I tried again, and got an error message saying the region was logging out – something like that… What I took away from this were the potential problems associated with a SL technical failure. [Video transcript]: I see [avatar’s name] has come online again. So it is pretty common I guess people will drop in and drop out [of SL] because of lack of bandwidth. [Blog posting]: For something I want to learn so badly, it certainly has me in a state of constant frustration! One thing I realized this week is that the speakeasy HUD that includes directions for building we are doing... is completely distracting! Whatever it is I am building gets lost behind the text. Trying to read the notecards, with text flying by... ugh! [Blog posting]:

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 17

But it does take a lot of time and effort to build materials that could be used in a classroom. I am sure it gets easier with more practice, but it does take a lot of time and effort to build one object because I would either forget a step or think of a better way to make it. Sanchez (2009) interviewed a focus group of students to understand their experience in a Fall 2006 world literature course that had some activities during the semester in SL. The majority of students registered negative emotions with their experience, ranging from boredom to frustration and anger. Sanchez grouped these negative experiences into four categories: technical, interface, user expectations, and time. The counter intuitiveness of the SL controls was a predominant complaint in this area. To the extent students were familiar with computer games such as the Sims or Grand Theft Auto, the expectations that SL would have a similarly easy interface and goals contributed to their frustration. The time-consuming nature of SL also alienated some students. One thing generally agreed upon by students interviewed was the high learning curve that exists for new users of SL. However, Sanchez identified three components that help to create a positive student learning experience: creating and customizing ones avatar; challenging students to think visually and creatively; and opportunities to play. Similarly, Skiba (2009) presented three main challenges of using SL: mastering the SL environment, by learning how to build, how to purchase, how to find scripts, and gaining additional skills; securing an appropriate level of hardware and access; and lastly, enabling the students to master the skills necessary to use SL. According to Skiba, students’ reaction to using SL was described as mixed initially, with students registering a more positive experience over time. This was also reflected in data: [Blog posting]: I just re-read my first blog that I wrote during the first week of this course, and I remember how discouraged I felt because I fell so behind while building the objects. However, now at the end of the course, although I am no where near to being an expert builder, I feel confident in using prims, scripts, and the other building elements to construct objects in Second Life. As I mentioned in my very first blog, “I cannot wait to look back on this blog at the conclusion of this course, and think, “ACCOMPLISHED!”

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 18

well, six weeks later, I honestly do feel accomplished as I was able to create a virtual learning environments in Second Life with my partner… a task that I once thought was a very, very, very far-fetched goal finally became a reality. Socialization & Social Presence Another theme that emerged from our preliminary analyses is of the need for socialization in SL and the ability of SL to provide learners with a sense of presence: [Blog posting]: It seemed to me that SL was one big social wasteland, good for elaborate but lifeless buildings and sites. My community in SL started to take root with this class. In addition to meeting class members, I began to get acquainted with others in SL, such as [avatar’s name], who took me on a tour of the UH site, and [avatar’s name], a librarian whose RL lecture I attended. Some of my friends from Twitter are also in SL. As my community grew, so did my interest in SL. [Blog posting]: A couple of observations…co-presence or ambient awareness – the sense of being there and connected, is really evident when we’re in Second Life. I mean it literally seems we’re meeting face-to-face when we’re in class and meeting with our groups. I wonder if it’s because, subconsciously, we know that someone is controlling each avatar we see in class? It was funny when Mark, Cheryl and I met a few nights ago in Second Life. At the end of our meeting both Mark and I complemented Cheryl on the dress she was wearing – her Second Life dress. It was a beautiful dress! The lines are getting blurred. Edirisingha, Nie, Pluciennik, and Young (2009) contend that the SL environment creates a sense of immediacy and social presence, reducing the sense of distance through the use of highbandwidth technology. This in-world socialization could be extended to network building in real life among the participants and a positive learning experience overall. They argue that their research supports the notion that the ability of a medium to engender social presence is

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 19

dependent on its bandwidth. The availability of high bandwidth networks creates the potential for socialization to occur at a distance in 3-D MUVE. Appreciation of SL’s Educational Potential Additionally, many students expressed their appreciation and enthusiasm for SL as an educational innovation with tremendous educational potential: [Blog posting]: How valuable do I think SL can be for education? Well, I’ve been looking to buy some land and start establishing my space where I can further explore how best to incorporate SL into what I do at work. More than that, I want to look at how I can begin doing new things in new ways. Only time will tell where that leads – I sure am excited about the journey. [Blog posting]: The in-class experiences with building objects has taught me to be appreciative of the objects in Second Life environments. A simple 3D tree takes skill and time to create, so I can imagine how much effort and time it takes to build an entire environment or sim. Nevertheless, the power of sims for educational purposes is significant. Sims allow participants to experience and to be immersed in settings and situations that may be difficult to achieve in real life, if not impossible. It also adds convenience as participants can visit these virtual locations any time of day and anywhere they may physically be, as long as a computer with Internet connection and Second Life is accessible. Furthermore, participants do not need to worry about crowds, expensive costs, and even dangers that would be related to performing certain activities in real life. Second Life can take educational opportunities above and beyond! The constant comparative analysis highlights the importance of addressing the issues of students’ frustrations with technical problems, the steep learning curve of SL and the need for socialization in order to get students engaged in SL. More importantly, this analysis has implications on the

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 20

changing roles of instructors in virtual worlds (Berge, 2008). With the change in how students learn in virtual worlds, the instructor’s role needs to change as well. Results & Discussion from Discourse Analysis Figure 3 provides a graph of the number of frustrations that student participants in this study experienced over the duration of their six-week course as reflected in their weekly blog postings.

Figure 3. Frequency of frustration occurrence over the six-week course duration as reflected in students’ weekly blog postings.

As one can see the majority of concerns was apparent at the beginning of the new course and rapidly decreased as the course progressed. There was a spike in week 5 because a specific problem occurring. During one of the in-world sessions in week 5, SL experienced a lost of its voice capability. Fortunately, the instructor had access to an audio conferencing system and by getting the students to log into the audio conferencing system, the instructor and students were able to conduct audio conversations. However, running two band-width intensive software (SL and an audio conferencing system) simultaneously created numerous technical problems for many students. In addition to numbers of instances, patterns of discourse were examined in the video transcripts. When a frustration was expressed, what happened? Did the instructor address it or

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 21

comment on it? Did peers respond? Firstly, frustrations or concerns were typically expressed using the text chat rather than voice. As mentioned earlier, students seemed more comfortable communicating using text chat and that text chat comments or questions were typically addressed later in the audio conversations. The delay is due to the fact that attention was focused on the more prominent audio conversations. Secondly, when frustration or concern was expressed (typically through text chat), the first person to respond was usually either the teaching assistant or a fellow student. The instructor was usually preoccupied with leading the class through a build exercise or a lecture and would only become aware of the situation later because of the text chat interactions. Here is an exchange from the video transcripts (red text indicates text chat conversation while audio conversation is in black text): Instructor: So today, I just want to give you a quick introduction…all that you have experienced in the second life environment so far, okay. Student A: [argh… I can’t sit… Instructor: I am sure all of you know what the second life is, but I just want to clarify that it is technically known as a multi-user virtual Teaching Asst: [then choose relax Instructor: environment, m-u-v-e-, Student B: [choose “relax.” Teaching Asst: [there we go :) This exchange illustrates a typically response to an expressed frustration where Student A was unable to “sit” on a floor cushion in SL. While the instructor is preoccupied with presenting an overview lecture, the teaching assistant and Student B were quick to respond to Student A’s frustration. Another interesting observation was the occurrence of students using the instant messaging (IM) feature in SL to help troubleshoot their peer’s technical problems. Students who are more comfortable in SL tended to use IM to reduce the local text chat “clutter.” These private IMs are not captured in the video recording. However, detailed analysis of the video transcripts revealed that there are probably instances of such occurrences:

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 22

Student C: [I still don’t have it… Instructor: … educational purposes. It’s not building a building, per se… Student C: [I think… Instructor: … although you can though once you have the ability… Student C: […I have a grey dialog box Instructor: Do you get the idea? Yeah, you now need to think in three dimensions, we are no longer thinking in just two dimensions. So going… Student C: [Is there another box? Instructor: …forward, you are thinking in three dimensions, and then you will be able to see things all from 360 degree view, ok? I know you are not familiar with this and I can’t emphasize enough, and I will repeat over and over again, how important it is for you to get used to using the alt zoom feature Student C: [Got it Student C: [Thanks This exchange seems to indicate that Student C was talking to himself. Student C was having problems with a build exercise but his problems seemed to be “miraculously” solved. Although we are unable to see the private IMs, it is highly probable that Student C obtained some help from another student via IM as indicated by subsequent interactions with the teaching assistant and instructor. The presence of the instances described above is indicative of the relative ease with which the users of this virtual environment make use of the different media available to them, integrating information from text, audio and visual channels. The challenge in terms of trying to address individual research questions is to develop a theoretical framework that connects the frequency of certain interactions with higher level concepts such as learning outcomes. Arguably peer interactions are excellent opportunities for generation and resolution of cognitive conflict (Tudge & Rogoff, 1999). The process of repair observed in the discourse can be seen as an indication of learning opportunities. Too much conflict and the learner becomes demotivated and gives up in despair (Wlodlowski, 2008). Tracking the instances of peer interaction, particularly

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 23

those that involve repair will likely provide insight into how the virtual environment supports effective learning, and perhaps provide prescriptions for increasing their effectiveness. Conclusion A graduate education course was taught in Second Life as an experimental course and served as a case study for investigation. While the goal was to investigate students’ reactions to their course experience, the researchers learned many valuable lessons about the practical steps to recording in-world class sessions, transcribing those classroom dialogues, and utilizing them as one source of data. From using ScreenFlow as the video-capturing software to disabling timeout features of SL or Mac OS systems, the researchers believed it was important to share the many tips and tricks learned with the community of individuals interested in research in virtual worlds to improve logistical data collection in future studies. As much of the research in virtual worlds is still developmental, the investigators of this study chose to combine multiple data sources and multiple data analysis methods to triangulate findings. Classroom observations were conducted in the in-world SL classroom space originally by a non-participant “camera person.” Similar to observation methods recommended in real world settings, converting the “camera person” to a participant observer proved more fruitful in the capturing of the action in-world of the instructor and students. Using multiple data analyses allowed the researchers to best target different types of research questions. Constant comparative analysis proved very useful in identifying salient themes from a broad exploratory approach. Using multiple coders and data sources addressed common concerns with this approach with regards to rigor of data collection and analysis (Patton, 1999). With student reactions as the focus, the three most salient themes emerged as: frustrations with technical issues and learning curve in SL, socialization and social presence, and appreciation of SL’s educational potential. The researchers then choose to select one of those themes and explore it from a different perspective using discourse analysis. Student frustrations was investigated as it is likely to be common in any course taught in SL and most broadly useful to a larger audience. Frequency findings indicate that frustrations are much higher at the beginning of a course and, in general, decrease over the duration of a course. However, new frequency spikes of frustrations are likely to emerge when a new technology pitfall is encountered. In the particular example in this study, the audio was very troublesome during a

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 24

particular class session. Using discourse analysis, the investigators also began to explore roles and sequence of responders to student frustrations. Student frustrations were predominantly expressed in text box chat versus a student making a verbal comment in classroom discussion. The particular course examined was fortunate enough to have a teaching assistant. This individual was the primary responder to student frustrations. However, instances were found of peers assisting peers. The roles of responders to frustrations discovered in this study, strongly suggest the utility of a teaching assistant in a SL course, especially one in which students and instructor may have less experience with courses in SL. The investigators found the sequential pairing of constant comparative method followed by discourse analysis a useful combination.

Journal of Virtual Worlds Research –Comparative and Discourse Analyses 25

References Berge, Z.L. (2008). Changing instructor’s roles in virtual worlds. Quarterly Review of Distance Education, 9(4), 407–414. Campbell, C. & Jones, S. (2008). Using second life as an educational instructional tool with preservice education students: A work in progress. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 3638-3642). Chesapeake, VA: AACE. Discourse Analysis. (2009). In Wikipedia, the free encyclopedia. Retrieved November 15, 2009, from http://en.wikipedia.org/wiki/Discourse_analysis. Edirisingha, P., Nie, M., Pluciennik, M., & Young, R. (2009). Socialization for learning at a distance in a 3-D multi-user virtual environment. British Journal of Educational Technology, 40(3), 458-479. Foster, A. L. (2006). Harvard to offer law course in 'virtual world'. The Chronicle of Higher Education, 53(3), A.29. Glaser, B. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4). California: University of California Press. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine Degruyter. Gumperz, J. J. (1982). Language and social identity. Cambridge: Cambridge University Press. Jarmon, L., Traphagan, T., Mayrath, M., & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life. Computers & Education, 53, 169-182. Johnson, L. F., & Levine, A. H. (2008). Virtual worlds: Inherently immersive, highly social learning spaces. Theory Into Practice, 47(2), 161-170. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage. Michels, P. (2008). Universities Use Second Life to Teach Complex Concepts. Government Technology. Retrieved from http://www.govtech.com/gt/252550. Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.

Journal of Virtual Worlds Research â&#x20AC;&#x201C;Comparative and Discourse Analyses 26

Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Service Research, 34(5), 1189â&#x20AC;&#x201C;1208. Ragin, C. (1987). The comparative method: Moving beyond qualitative and quantitative strategies. CA: University of California Press. Sanchez, J. (2009). Barriers to student learning in second life. Library Technology Reports, 45(2), 29-34. Second Life. (2009). In Wikipedia, the free encyclopedia. Retrieved from Wikipedia November 15, 2009: http://en.wikipedia.org/wiki/Second_Life. Second Life. (2010). Video resources. Retrieved from Second Life Wiki January 4, 2010: http://wiki.Second Life.com/wiki/Movie_Recording Skiba, D. J. (2009). Nursing education 2.0: A second look at second life. Nursing Education Perspectives, 30(2), 3. Tudge, J., & Rogoff, B. (1999). Peer influences on cognitive development: Piagetian and Vygotskian influences. In P. Lloyd & C. Fernyhough (Eds.), Lev Vygotsky: Critical Assessments, Vol. 3: The zone of proximal development. London: Routledge. [Reprinted from Bornstein & Bruner (Eds.), Interaction in human development.] Twining, P. (2009). Exploring the educational potential of virtual worlds, Some reflections from the SPP. British Journal of Educational Technology, 40, 496-514. Warburton, S. (2009). Second Life in higher education: Assessing the potential for and the barriers to deploying virtual worlds in learning and teaching. British Journal of Educational Technology, 40(3), 414-426. Wlodkowski R. J. (2008) Enhancing adult motivation to learn: A comprehensive guide for teaching all adults, 3rd edition. Jodsey-Bass, Wiley.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 How to approach a many splendored thing: Proxy Technology Assessment as a methodological praxis to study virtual experience Lizzy Bleumers IBBT-VUB-SMIT, Belgium Kris Naessens IBBT-VUB-SMIT, Belgium An Jacobs IBBT-VUB-SMIT, Belgium Abstract This article introduces Proxy Technology Assessment (PTA) as a methodological approach that can widen the scope of virtual worlds and gamesâ&#x20AC;&#x2122; research. Studies of how people experience virtual worlds and games often focus on individual in-world or ingame experiences. However, people do not perceive these worlds and games in isolation. They are embedded within a social context that has strongly intertwined online and offline components. Studying virtual experiences while accounting for these interconnections calls for new methodological approaches. PTA answers this call. Combining several methods, PTA can be used to investigate how new technology may impact and settle within people's everyday life (Pierson et al., 2006). It involves introducing related devices or applications, available today, to users in their natural setting and studying the context-embedded practices they alter or evoke. This allows researchers to detect social and functional requirements to improve the design of new technologies. These requirements, like the practices under investigation, do not stop at the outlines of a magic circle (cf. Huizinga, 1955).We will start this article by contextualizing and defining PTA. Next, we will describe the practical implementation of PTA. Each step of the procedure will be illustrated with examples and supplemented with lessons learned from two interdisciplinary scientific projects, Hi-Masquerade and Teleon, concerned with how people perceive and use virtual worlds and games respectively. Keywords: Proxy Technology Assessment; virtual worlds; games; methodology; ethnography

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 4

How to approach a many splendored thing: Proxy Technology Assessment as a methodological praxis to study virtual experience Introduction The study of virtual worlds and games is quite a young field that is not yet fully institutionalized. No real “schools of studies” have been established (Mäyrä, 2008), in part because of the large variety in backgrounds of virtual worlds and games researchers, such as literature, psychology and media studies. The definition of shared concepts and the specification of methodological approaches remain in their infancy. To capture the user experience of virtual media, researchers tend to adopt frameworks that reduce the complexity of these media. Konzack (2007) for instance identified eight rhetorics dominating game studies, which he compiles into three categories: product-oriented (technology, economy), socio-culturally oriented (anxiety, learning, gender, ideology) and aesthetically oriented (narratology, ludology). These rhetorics shape how researchers approach games and heavily influence research choices such as participant selection and applied methodological angles. Sticking to a single rhetoric is problematic, as a broad and open view on subjects and methods is mandatory in this rapidly evolving field. Mäyrä explains the undesirability of a “master theory” in game studies as follows (2008, p. 10): “Games, players and their interactions are too complex and interesting in their diversity to allow for all too powerful simplifications.” A powerful simplification, often applied in studies of new media, is the strict distinction between the real and the virtual. An example of this reasoning is the application of Huizinga’s (1955) well-known concept of the magic circle to gaming and virtual worlds, in which the game world is seen as an entity closed off from the outside world, a place where acts of a different kind take place. However, as Castronova (2005) argues, the magic circle surrounding social and game virtual worlds is far from impenetrable. In fact, it cannot be impenetrable, given that people move back and forth across this border taking their impressions, their beliefs and habits with them wherever they go. Tuszynski (2008) uses the same argument in a critical reflection on the distinction between virtual and real communication. On the one hand, real-world referents seep into virtual realms through people’s activity. On the other hand, virtual realms change what we know and do in our everyday life. Ultimately, a strict separation of the real and the virtual cannot 4

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 5

be upheld. Doing so can even be problematic, as it obscures potential areas of study such as the mutual shaping of virtual media and the people who use these media and their everyday settings. The goal of this paper is to describe a methodological approach, called Proxy Technology Assessment (PTA), which enables researchers to study mutual shaping with respect to new media use, and to describe how this approach can be applied to the study of games and virtual worlds. Virtual worlds and games were, especially at the end of the 20th century, studied in terms of effects, with its audience considered to be passive. In recent years, ethnographic research of virtual media has grown exponentially, taking into account how people also actively create meaning from experiences with these media. PTA combines several interpretative and ethnographic methods for investigating people’s experiences with and use of games and virtual worlds whilst taking into account the social context in which these users are embedded. Introducing Proxy Technology Assessment Proxy Technology Assessment (PTA) is a methodological approach that is used to gain understanding about whether and how new technology and new applications may become part of people’s everyday life. PTA is conducted by giving future users experience with one or more related technologies (i.e. hardware and/or software) that already exist today. Crucial is that these technologies share as many characteristics as possible with the technology under development. These types of technology are described as proxy technologies. Both the way in which the proxy technology is appropriated and the users’ experience-based reflections on these technologies can be used to inform and inspire the development of new technologies in an early stage. PTA is an operationalization of domestication theory, which puts the user in the center of innovation (Pierson et al., 2006). According to the domestication approach, innovations are not situated at the level of technology as such, but at the level of the everyday practices of the people that use that technology. The entrance of a new technology starts a dynamic process in which the technology, the user and his or her social and physical surroundings undergo changes until a new equilibrium sets in. New practices arise and old practices may be disrupted. For instance, the use of the technology within the home can be the result of a heated negotiation between the family members. It is impossible to capture this process by conducting a single individual lab study.

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 6

PTA has been developed to capture this social dynamic by investigating technology use in the natural everyday environment of the user over a period of time. PTA shows resemblance to technology probes (Hutchinson et al., 2003). In both cases technologies are deployed in the natural surroundings of users to study the adapted and new practices surrounding these technologies. Both PTA and technology probes involve giving users an experience as a reference point to reflect on. They are both applied to gain insights in users’ behavior per se and in relation to technology during the early stages of development. They differ in terms of their role in the development process. While technology probes are intended to obtain design ideas, proxy technology assessment requires that a basic idea or concept for a technology has been defined. The primary aim of PTA is to help anticipate whether and how the future technology would fit in with the users’ lives in order to guide further development. The procedure used in PTA can be split into four stages. First, in the preparatory stage, one or more suitable proxy technologies are selected. Consecutively or in parallel, the researcher selects and recruits participants and conducts a careful study of their natural setting. Second, the proxy technologies are introduced to these participants in their natural setting. In the third stage, data are gathered on the use of and experiences with the proxy technology. Preferably, PTA is combined with different methods to allow both method and data triangulation. Finally, the gathered data are analyzed and the results are reported in the form of guidelines or recommendations. In the following section, we will describe the procedure of PTA in more detail. We will make each step more concrete by giving examples from user research that we conducted within the scope of two projects, Hi-Masquerade and Teleon. Both projects are interdisciplinary projects – funded by the Flemish Institute for Broadband Technology (IBBT) – in which technological research (both by academic and industrial partners) is complemented with user research. The goal of the user research in Hi-Masquerade was to inform the development of a virtual world application to support remote interaction between family members. In the Teleon project, the aim was to make a broad examination of users’ experiences of computer and videogames, in order to optimize specific features of future games produced by the project partners. In both projects, the end goal of the project was not a finished product, but rather a proof-of-concept. However, PTA

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 7

can be applied in both cases, as it is not dependent on the actual realization of the future technology. How to conduct Proxy Technology Assessment At the start of Proxy Technology Assessment (PTA), a concept or notion of the future technology use is required. In particular, three factors should be determined: the characteristics of the future technology, the users for which the technology is intended for, and the setting in which the technology will supposedly be used. These three choices are often made in collaboration with other project partners and can be more or less informed by prior user research. For instance, the selection of certain features may have been inspired by prior participatory design or may have been fixed by the developers as must-haves. One way to delineate the concept of the future technology is through a scenario that describes project partners’ beliefs of what the application would be like, who would use it, where, when and how it would be used. When not immediately informed by prior user research, this story is an idealization, or what Arnowitz, Arent and Berger (2007, p. 301) refer to as a “sunny-day scenario.” In any case, the scenario gets revised as user research progresses. Storyboards can also be used to illustrate a (potential) application and the context in which it is (to be) used in a graphical manner (Truong, Hayes & Abowd, 2006). They can be used complementary to the scenario, for instance, to present the scenario to project partners or research participants. They allow research participants in particular to empathize with the story characters and to reflect on the application’s functionality and what it might mean for them (Van Der Lelie, 2006).

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 8

Figure 1. Fragment of the storyboard visualizing the first scenario of Hi-Masquerade. In Hi-Masquerade, two scenarios were written in collaboration with all the project partners, intended to guide the development of two demonstrators in the project. Both scenarios, illustrated by storyboards (see Figure 1), described the use of a virtual world application by a child and his grandmother in their respective homes, each with a different focus. While the emphasis of the first scenario was more on sharing past experiences with family members, the second scenario was rather on virtual play and cooperation between them. In Teleon, the project partners broadly described the application the project had to generate. Its focus was a game world, where users could navigate and play games, with emphasis on easy implementation of new mini-games, storytelling and the formation of a dynamic user community. The application would first be PC-based only, but would have to be able to migrate to other platforms. The application had no specified audience in mind, as the developers wished to reach different audience segments in terms of demographics and playing experience. Preparation Prior to the actual introduction of the proxy technology, the researcher needs to undertake three actions: Choosing one or more appropriate proxy technologies, selecting, recruiting and profiling participants, and finally capturing the setting to which the proxy technology will be introduced. Each of these research steps is guided by the initial conceptualization of the use of the new technology. Selecting Suitable Proxy Technologies The concept of the future technology allows the researcher to compile a set of basic functionalities that the proxy technologies should possess. Aside from these functional criteria, additional criteria are specified. These are more general properties such as ease of use, technical reliability and language. The selected applications should have a transparent interface and function properly. We emphasize that PTA is not a usability evaluation method. The focus is on how and why people would use an application, not on establishing which usability problems

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 9

prevent or discourage them from using it. Language can also be a barrier for use; a mismatch with the native language of the users should be avoided. In Hi-Masquerade, we distilled a set of functional criteria from the scenarios that were written, such as the fact that the application had to be a virtual world that allowed remote interaction and customization. Examples of additional criteria were ease of use, which was particularly important because of the different generations involved, the fact that the application should run both on PC and MAC laptops and desktops and the use of native language, which turned out not to be straightforward. After discarding a number of applications, we found an online web application that featured a small virtual world targeted at family members of various ages and that ran on MAC and PC. Unfortunately, the application was not available in Dutch. However, it was the most optimal application given our set of criteria. In Teleon, as mentioned previously, the initial concept was rather broad. To deal with the limited set of prefixed features, we chose a large variety of games to address what various user groups would like and what they would not like in such an application. We selected 6 platforms (PC, Microsoft Xbox 360, Sony Playstation 2, Sony PSP, Nintendo DS, Nintendo Wii) and 18 games. Differences in technological features such as mobile vs. domestic hardware, strength of graphics and sound and controlling devices influenced the selection of proxies. The selection of games covered most popular genres such as sports, strategy, MMORPGs, action, platform and sandbox games. The 18 games allowed eliciting participants’ reactions to a range of proxies with great variety. Selecting, Recruiting and Profiling Participants In preparation of PTA, each envisioned user group has to be described in terms of demographic and/or socio-economic characteristics and other distinguishing features, such as degree of familiarity and experience with related technologies. All these features will be the core of the selection criteria. This means that sampling is done purposively (Bernhard, 2006) instead of probabilistically. There is not a magic number of participants to engage in PTA. As the goal of PTA is to generate a rich description of how a technology settles within people’s everyday life, the approach is grounded on the use of qualitative, interpretative and ethnographic methods (Pierson 9

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 10

et al., 2006), which will be described in more detail further on. PTA does not exclude the use of quantitative methods; it simply cannot be based solely on a quantitative methodology. The intensity of data collection in this type of research limits the number of participants that can be studied properly within a given amount of time. In our view, PTA research can best be carried out within a small sample of participants. Recruiting participants for PTA research can be very difficult. First, researchers need to identify participants with specific characteristics, whose numbers are usually limited. Also, participation in ethnographic research requires great effort from respondents, since active input is demanded for a longer period of time. Indeed, many people are hesitant to participate in longterm in-depth research. Recruiting by using snowball sampling therefore proves to be highly valuable for this approach. Once you locate a suitable participant who is willing to participate in the study, he or she often knows friends or family with similar characteristics. This participant can then become an informant for your study, either convincing other suitable participants to take part or bringing you into contact with these participants. Both in Hi-Masquerade and Teleon, snowball sampling was essential because we wanted to recruit user groups, instead of individual users. In Hi-Masquerade, we needed to recruit family members across households (with internet access at each household). In particular, we were looking for family members across three generations: a child between eight and twelve years old, one of its parents and one of its grandparents. We approached grandparents and parents, asking them whether they would be willing to participate with two relatives in a research project of about one year and a half1. In this way, three groups of family members were recruited. In Teleon, the goal was to involve single households (friends sharing a residence, couples, families with children) in which one of the household members played games. The recruitment of the households took place through a first contact with one household member, who was not necessarily the household member that played games. The total number of participants in Teleon amounted to 15, distributed over 5 households. These 15 participants were followed in their use of the offered proxy technologies for some five months. As PTA requires great effort and is vulnerable to dropout of respondents, it is vital to keep participants interested in the study. By having the same researcher in contact with the 1

Participants were informed that they would be contacted maximally six times within this timeframe. 10

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 11

participants during the whole period, from recruiting until the end of the study, a strong social tie is created that can motivate participants to stay engaged. In addition, incentives can be offered to keep participants motivated. For instance, in Hi-Masquerade, family members received tickets for the zoo. Next to money or gifts, PTA research offers participants the unique incentive of being able to use the proxy technology for the duration of the field study. In Teleon, for example, participants had the opportunity to use several gaming devices and games at home together with their friends and family over a period of five months. This was particularly a motivator for the participants that were already more experienced with digital gaming. Once the participants are recruited, their profile information is gathered, which can be done through a questionnaire (Dreessen & Pierson, 2007). This can help to confirm that the participants meet the selection criteria, particularly when recruitment was done by an external agency. However, the questionnaire also enables the researchers to collect additional information on the social context of the participants, their access and use of particular media, and their answers to project-specific questions (Lievens et al., 2008). In Hi-Masquerade and Teleon participants were asked to fill in basic demographic and socio-economic data as well as answer questions about media access, use and preferences. Mapping Usersâ&#x20AC;&#x2122; Setting In line with the domestication approach, an accurate interpretation of the use of and experiences with the proxy technology necessitates a thorough understanding of the setting in which the proxy technology is deployed. Indeed, peopleâ&#x20AC;&#x2122;s behavior, including their technology use, is intimately connected with the spaces they live in. The setting in which technologies are used influences what users do with them and how they operate them. It is because of this intimate connection that Kuznar and Werner (2001) emphasize that understanding human behavior should begin by drawing a map of the environment in which this behavior is played out. This is exactly what is done in the preparatory stage of PTA research. The setting that needs to be mapped is determined in advance. It can be a fixed setting such as the home or workplace when studying technologies or applications with limited mobility (e.g. home cinema system, IDTV programs requiring connection to a cable network). However,

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 12

the setting can also be dynamic. For mobile technologies and applications for instance, it includes means of transport for daily commutes to and from the homes of friends and family. With regard to what should be mapped, especially the presence of technologies with resemblance to the future technology is highly relevant. The mapping itself can proceed in two ways. The researcher can draw up a geographical map containing the location of all relevant context elements or let participants draw up a map themselves. In both cases, it is recommended that researchers take photographs of the setting to enrich the map (see Figure 2). Baillie and Benyon (2008) present a way for researchers to map out both the physical spaces within people’s homes and the way people perceive these spaces. Their approach is to create maps based on what they call “technology tours.” During a technology tour, researchers walk through the house, both with the entire household and with each of the individual household members in order to establish which technology is present in each room and where it is located, who of the family members uses it and what for. The resulting maps showed that the different household members clearly had differing views concerning the purpose of certain spaces. This approach was applied in Teleon. In the Teleon project, we produced a geographical mapping of the participants’ homes, describing the basic architecture, which media technologies they owned and where these were placed in their residences. Placement was often critical, as media devices in participants’ private rooms were seen as personal and very often not to be used by other household members. This ownership and familiarity with certain devices acted as a point of reference for the researcher during the data collection and the analysis process. Instead of drawing up a map of the users’ environment, the researcher can also ask participants to draw up a map. Such a map is a direct representation of how users see their environment. Hasbrouck (2007) has effectively used cognitive mapping to gain insight into domestic practices and their relationship with domestic technologies. Together with fellow researchers he asked people to draw maps of their homes and mark the location of objects that take on an important role in their daily activities. Their creations were taken as the starting point for additional questions. This mapping procedure is less invasive. However, there is a risk that participants will neglect or fail to mention the presence of certain technologies, for instance, out of forgetfulness or because of social desirability. In Hi-Masquerade, we let family members draw a map of their home and mark the location of media such as the television. This map was 12

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 13

discussed with them and supplemented with photographs of the media taken by one of the researchers during the following visit. These photographs helped to “verify” the map of one of the households.

Figure 2. Photograph of participant’s living room in Teleon project. Distribution of Proxy Technology When introducing a proxy technology to the participants, researchers should install and configure the technology properly and give a demonstration. When conducting PTA, researchers are mainly interested in the actual appropriation of the proxy technology and not in the preceding phase of figuring out how to use it. By equipping participants with operational technology and basic knowledge on how to use it, researchers ensure that the participants smoothly pass this initial phase and are not demotivated by a failure to use it. It is important to explain to participants what is expected from them with regard to using the proxy technology. For instance, during the Teleon project, households were provided with gaming platforms and households members were asked to try the accompanying games at least once. However, this request was very informal and not bound to rules or deadlines. It is important not to force participants to use the proxy technology, because the lack of use by certain participants can be highly informative. Also during this introduction, practical topics such as making appointments and how they should correctly fill out diaries can be addressed. It is likely that users cannot keep the proxy technology after the fieldwork has ended. If this is the case, this should be clearly indicated during the introduction. The withdrawal of the 13

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 14

proxy technology is a sensitive issue, as participants may get attached to it. In Hi-Masquerade, we needed to proceed even more cautiously because some of our participants were children. We chose an existing free online web application as proxy technology, which eliminated the issue of having to hand over something and remove it later on. A related effect of the proxy technology introduction on participants was noted in Teleon, where after the field study some participants purchased one or more of the gaming platforms and games that were offered to them as proxy technology. Studying Proxy Technology Use PTA studies involve participants using proxy technology over a longer period of time. It is essential to capture as many relevant data as possible during this time. Data can be gathered both while the use of the proxy technology is ongoing, and after the use of the proxy technology has ended. For each time period, different methods are warranted to gather relevant data. During the first period, the use of the technology has to be monitored. During the second period, methods are applied to encourage reflection among participants about the technology and how they used it. At each point in time, the study of the proxy technology use can be supplemented with complementary methods that can help situate the results. We will now describe these methods in more detail. During Proxy Technology Period: Monitoring Use In our experience, a period of four weeks is a minimum for fully capturing the use of the proxy technology. When the technology is first introduced, participants will first experiment with the application. Although this already gives them an experience to reflect on, this type of use does not yet show whether the participants are genuinely interested and whether the application truly fits in with their everyday life. It is what happens after this first experimentation phase that is truly interesting for the researcher: is the application abandoned or is it appropriated and how does this take place? While the proxy technology is in use, it can be monitored in two ways: via automated logging or by means of diaries or logbooks. The advantage of automated logging is its high accuracy, as software in the technology register how the user deals with the technology. When 14

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 15

using diaries, participants have to answer questions about the use of the proxy technology as well as more general questions. These diaries can consist of writing in a paper booklet or answering the questions online, depending on the user group under investigation. In Hi-Masquerade, for instance, where we dealt with young children and older adults, we distributed paper and pencil logbooks in which we kept the amount of writing that had to be done to a minimum. Aside from questions regarding the proxy technology, we also inquired about the relationship between children and grandparents. We encouraged them to fill in their logbook by including a section, in which they shared personal information about each other, that they would be able to exchange after the research was finished. In addition, we gave the logbook a playful layout (see Figure 3).

Figure 3. Extract of logbook used in Hi-Masquerade. Although diaries or logbooks are a less accurate instrument for keeping track of frequency of use than automated logging, they have the added advantage of allowing researchers to probe usersâ&#x20AC;&#x2122; attitudes and experiences with the proxy technology and other relevant aspects that are not directly related to the technology. In addition, automated logging may simply be impossible, as was the case in Hi-Masquerade and Teleon. In Teleon, we asked participants to fill in a daily diary for seven days during each month of the field study. These diaries included questions about how many hours they used various media and the content they saw, next to 15

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 16

several questions about gaming (such as games played, hours played, goals attained, single or multiplayer). When use of the proxy technology may be infrequent, an alternative to a daily diary is a logbook in which participants answer questions each time they use the proxy technology. This was an effective approach in Hi-Masquerade, in which we asked participants for each session to fill in when, where, how and with whom they had used the application and what they had and had not liked about the session. During the period in which the proxy technology is in the field, researchers can also observe participants while these are using the technology to complement the self-reports. These observations should not only focus on how a participant deals with the provided technology, but also on what happens in the participant’s surroundings. Only the Teleon project involved such observations. We asked participants to play some of the provided games and try to attain a certain goal (e.g. get to the end of level one). Not only did we observe physical and emotional reactions (e.g. screaming at the screen when losing), but also how other members of the household acted when others played. Participants with much gaming experience for instance were very concentrated and showed little emotion while playing, while parents of the participating households often commented on the games their children were playing. After the Proxy Technology Period: Eliciting Reflection After the proxy technology period has ended, participants gather with the researchers to reflect on the proxy technology in a post-interview. If a logbook was used, this also can be used as a reference point for the discussion. Interviews in PTA research tend to be semi-structured. A topic list is compiled based on the project-specific research questions, but room is left to discuss the participants’ particular experiences. Indeed, in case of non-use, some topics can simply not be handled. The semi-structured interview leaves room for finding out why certain participants did not use the proxy technology. For instance, in Hi-Masquerade a topic list was prepared to probe the participants’ experience of presence while using the virtual world. However, one of the family members had not used the application. Hence, we inquired ad hoc about his reasons not to use the application and then extended the discussion to the experience of presence with other media (for a comprehensive review on presence, see Lombard & Ditton, 1997).

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 17

Both in Hi-Masquerade as well as in Teleon, we interviewed the participants together as a group. Although both individual and group interviews can be conducted within the scope of PTA, we felt that group interviews were more appropriate here. They allowed us not only to ask what each participant had liked and disliked about the provided proxy technologies, but also to discuss what place the proxy technologies got in the household or family as a whole. In HiMasquerade, we found that group interviews had an additional advantage: older family members stimulated the children to give their point of view. In part because the children felt more at ease with their close relatives around, but also because the older family members would quickly notice when a particular question was difficult to answer and then reframed it. This advantage outweighed the social desirability effect that the presence of these family members had. Complementary Methods We have found that two methods can be very useful in supplementing the findings obtained by PTA, namely co-creation and gathering feedback through storyboards. In the broadest sense, co-creation refers to a joint creative activity (Sanders & Stappers, 2008; Sanders, 2006). In the context of designing and developing new technologies, it can be applied to actively engage users in the idea generation process. In this case, co-creation becomes an instrument for participatory design and is referred to as co-design (Sanders & Stappers, 2008). Within the framework of PTA, the researcher uses projective techniques such as making and discussing drawings to elicit desires, dreams and needs with regard to the new technology, its functions and use. Participants are handed tools to produce something that expresses what they think about the technology. Tools suited for this goal can be both electronic (e.g. animation programs), as well as more tangible (e.g. paper, pencils). Because an active creative input is required from the participants, the collected data will be richer than for instance the diary entries. In addition, co-design can help to gather data from participants who were not interested in using the proxy technology. In essence, the artifacts that are created and particularly the users’ interpretation of these artifacts will enhance the researchers’ understanding of the users’ needs and desires and thereby support the formulation of requirements and recommendations that PTA aims for.

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 18

The advantages of conducting co-design sessions were clearly noted in Teleon, where we ended the fieldwork with such sessions. Participants were asked to make a collage with various materials, such as pencils, magazine articles and post-it notes. The collage needed to represent all the characteristics of what they conceived to be an ideal game. This creative process was stimulated by the various proxy technologies they had used in the past months of fieldwork. For participants who had used the proxy technology intensively during the fieldwork, we saw highly similar results between the outcomes of the co-design and other applied methods. For those participants who had not or barely used the offered proxy technologies, taking part in the codesign sessions sparked ideas previously not uttered during the field study. The likes and dislikes with proxy technologies were not targeted as this was a central subject of other data collection methods. Instead, the goal was to give users the opportunity to look beyond certain games and let their imagination run free without restraints. For this specific group, we found needs and wants pointing towards being able to control the environment (e.g. no time limit) and easy to use controls. In Hi-Masquerade, co-design sessions have not been conducted. Because the concept for the future technology was more strongly delineated, the initial focus was on the evaluation of this concept, rather than on enriching it with new ideas. Nevertheless, if time allows, it would be valuable to probe the participants’ view of an ideal remote interaction application or virtual world. In particular, we have used storyboards that illustrate a conceptualization of the future technology use to gather feedback from participants as a complement to PTA. Storyboards can be used to capture participants’ attitudes towards the future technology as a whole and to gather feedback on specific issues. For instance, in Hi-Masquerade, project partners wanted to know more about how users feel about being embodied by an avatar in the virtual world. Furthermore, storyboards can be used to illustrate the use of features that are absent in the proxy technology. In Hi-Masquerade, for example, the project partners envisioned a 3D visualization of the virtual world. Similarly to storyboards, other forms of prototypes of the future technology can be shown. In Hi-Masquerade, participants were invited to explore the use of an initial demonstrator of the future technology. As a demonstrator it was not yet suitable for home use, but the family members’ responses to this technology were a valuable supplement to the findings gathered through PTA. We were able to observe the family members’ behavior while using the virtual 18

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 19

world application and their responses to features they had no previous experience with, such as the integration of a video-image of themselves in a computer-generated virtual world. Analyzing Data and Reporting Results If applied effectively, PTA yields a “thick description” (Geertz, 1973) of the proxy technology use based on a variety of data (text and audiovisual material). A “thick description” is a description that allows the researcher to situate the behavior in the social context in which it arose, to understand the purpose and intentions behind it (Ponterotto, 2006). The goal of analysis is to arrive at such a rich interpretation (or “thick interpretation” as Ponterotto calls it), which warrants a qualitative approach. In the past, we have successfully applied an approach that is based on Grounded Theory (Glaser & Strauss, 1967) to conduct our qualitative analyses. In this approach, the researcher first identifies categories and concepts that emerge from text, and then links the concepts into substantial and formal theories (Bernard, 2006). By iteratively analyzing the data, new concepts and findings emerge. Due to the large data output PTA research usually generates, it is necessary to have a clear strategy on how data will be analyzed throughout the research process. In addition, it is recommendable to start analyzing data during the field study. This qualitative analysis is ideally done with the help of specialized software that supports researchers in organizing, coding and linking the data. The findings obtained for Hi-Masquerade and Teleon illustrate the rich interpretations that can be obtained through PTA. In Hi-Masquerade, we established that family members found the proxy technology that we introduced to them unsuitable for remote family interaction. Through PTA, we were able to look beyond the functionality of the application per se at pinpointing the reasons for this “misfit.”.” For instance, the busy and very different schedules of the family members made it difficult to arrange for a remote session. We were also able to establish how the use of the technology affected these family members’ everyday lives. The grandparents that used the application appreciated the fact that because they were trying to use the virtual world application together, they had had more actual contact with their grandchildren. It had given them a common ground.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 20

In Teleon, providing a diverse amount of platforms and games proved fruitful, as these proxy technologies sensitized participants about what virtual worlds and games currently offer. The combined use of various methods made it possible to build up a clear understanding of what each of the selected user groups liked and disliked about virtual worlds and games. The combination of interviews, observations, diaries, context information and co-design sessions gave insights in both personal factors (such as experience, time willing to devote to gaming) as well as social and contextual factors (such as social control of family, motivations to play together, see Authors, 2009) contributing to how the technology was used. Once the analysis is finished, the findings are integrated in the development process by reporting them to those involved in this process: the project partners in general, and the developers, engineers and designers in particular. This requires that the rich interpretation of the researcher is translated into a format that can be understood and put to use by the intended audience. We have found that formulating the findings in terms of the usersâ&#x20AC;&#x2122; requirements (needs, wishes, concerns, desires, etc.) and recommendations (on what the technology should incorporate to support, facilitate, prevent, etc.) is a suitable format. However, it is a challenge to make recommendations sufficiently concrete to be workable; and our experience has shown that recommendations certainly benefit from discussion with the technical partners (Lievens et al., 2008). Conclusion In this article, we have described and illustrated Proxy Technology Assessment as a methodological approach for studying the use of and experiences with virtual worlds and games in order to guide the development of new virtual world and game technologies. Through the investigation of the use of proxy technologies (i.e., technologies that share characteristics with the technology under development) in usersâ&#x20AC;&#x2122; natural settings, PTA allows researchers to anticipate whether and how the targeted users are likely to appropriate these future technologies. Based on the insights obtained through PTA, requirements and recommendations for new virtual worlds and games can be formulated. Although Proxy Technology Assessment can be used on a wide variety of technologies and applications, in this article, we have focused on its particular usefulness for the study of 20

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 21

virtual worlds and games. Drawing from qualitative-ethnographic methodology, domestication theory and Grounded Theory, its strength lies in its potential to draw a comprehensive and highly complete picture of the use of virtual worlds and games. PTA offers researchers the possibility to include both online and offline, technology-related and contextual factors in their studies. Considering the desire to attract new user groups to virtual worlds and games (cf. the success of Nintendo Wii) as well as the wish to better address the needs and wants of existing users, this complete picture of the interplay between personal, contextual and technology-related factors can bring valuable insights for the design of new virtual world and game applications and technologies or even for redesign. The key component of Proxy Technology Assessment is the combination of multiple methods and techniques within the same research design to understand the (non-)use of a technology (through monitoring tools and observation) within the natural environment of the user (through geographical mapping) and in the light of who the users are and the social web that they are part of (through questionnaires, group interviews, co-design, storyboard sessions). Method and data triangulation is, as such, central within PTA research. Researchers need to link the data provided by the different research methods and need to keep doing so during the research process to constantly generate fresh insights. PTA can be characterized as a relatively open approach in the sense that it allows researchers to add methods depending on the project scope and the technology under development. While qualitative-ethnographic methods are particularly suitable for PTA, quantitative methods may also be applied but are likely to require the recruitment of additional larger user samples. How to integrate quantitative approaches within a PTA framework sets up an interesting topic for future research. The inherent nature of using multi-method approaches within PTA research places great demands on its participants. Although samples are usually limited due to the specific scope, PTA studies require a prolonged effort of participants to take part, with activities such as giving interviews, being observed, filling in diaries, letting researchers into their residence and participating in co-creation study. As in other longitudinal research, one of the challenges of PTA research is to maintain participantsâ&#x20AC;&#x2122; interest in the study. There are several specific ways to support this. First, being able to use the proxy technology for several weeks can motivate participants. Also, keeping close contact with users that are highly motivated can be useful. This 21

Journal of Virtual Worlds Research â&#x20AC;&#x201C; How to Approach a Many Splendored Thing 22

group of participants often provides new participants that can be included in the studied sample and can also stimulate these participants not to drop out. We would like to stress the explorative nature of Proxy Technology Assessment. First, although the approach is multi-layered and delivers a broad perspective, it does not enable the researcher to foresee all issues related to the use of the future technologies. A fraction of uncertainty always remains when dealing with human behavior. Second, PTA starts out from a concept or vision of a future technology that is prone to change and may even not be implemented. In essence, this is not problematic for user experience researchers. In any case, a grounded understanding of the proxy technology use is obtained. This openness of the PTA approach gives researchers ample opportunities to study the various and ever-changing aspects surrounding the use of virtual worlds and games.

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 23

References Arnowitz, J., Arent, M., & Berger, N. (2007). Effective prototyping for software makers. San Francisco, CA, USA: Morgan Kaufmann. Authors (2009). The place of TV and other media in common gaming practices: A fluent marriage? Paper presented at EuroITV2009, Leuven, 3-5 June 2009. Bernard, H. R. (2006). Research methods in anthropology: Qualitative and quantitative methods, fourth edition. Oxford: Altamira Press. Castranova, E. (2005). Synthetic worlds: The business and culture of online games. Chicago/London. University of Chicago Press. Dreessen, K., & Pierson, J. (2007). Study 3: Co-creation and social change in off-line communities (IBBT). In EU FP6 CITIZEN MEDIA – Initial context, user and social requirements for the CITIZEN MEDIA-applications. Retrieved from http://www.istcitizenmedia.org:8080/download/attachments/270/D1.2.1_InitialContextUserAndSocialR equirementsForTheCitizenMediaApplications_V1.0.pdf Geertz, C. (1973). Thick description: Toward an interpretive theory of culture. In C. Geertz (Ed.). The interpretation of cultures (pp. 298-320). New York: Harper. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine. Hasbrouck, J. (2007). Mapping the digital home: Making cultural sense of domestic space and place. Intel Technology Journal, 11(01), p.89-90. Huizinga, J. (1955). Homo ludens: A study of the play element in culture. Boston, MA: The Beacon Press. Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., et al. (2003). Technology probes: Inspiring design for and with families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 17-24). Ft. Lauderdale, FL, USA: ACM. Konzack, L. (2007). Rhetorics of computer and video game research. In J. P. Williams & J. H. Smith (Eds..), The players’ realm: Studies on the culture of video games and gaming (p. 110-130). Jefferson, NC: McFarland & Company, Inc. 23

Journal of Virtual Worlds Research – How to Approach a Many Splendored Thing 24

Kuznar, L. A., & Werner, O. (2001a). Ethnographic mapmaking: Part I – Principles. Field Methods, 13(2), 204-213. Lievens, B., Torben-Nielsen, K., Pierson, J., & Jacobs, A. (2008). Proxy technology assessment as a multi-method approach for identifying social requirements of co-creative media. Paper presented at NordiCHI 2008 Workshop: New approaches to requirements elicitation.

Lund,

20-22

October.

Retrieved

from

http://www.ist-

citizenmedia.org:8080/download/attachments/3834785/NordiCHI2008+WS+final+proceedings.pdf?version =1

Lombard, M., & Ditton, T. (1997). At the heart of it all: The concept of presence. Journal of Computer

Mediated

Communication,

3(2),

Retrieved

from:

http://www.ascusc.org/jcmc/vol3/issue2/lombard.html Mäyrä, F. (2008). An Introduction to game studies: Games in culture. London: Sage. Pierson, J., Jacobs, A., Dreessen, K., Van Den Broeck, I., Lievens, B., & Van den Broeck, W. (2006). Walking the interface: uncovering practices through proxy technology assessment. In Proc. of EPIC 2006 (pp. 40-54). Portland, OR, USA: National Association for the Practice of Anthropology. Ponterotto, J. G. (2006). Brief note on the origins, evolution, and meaning of the qualitative research concept “Thick description..”.” The Qualitative Report, 11, 538–549. Sanders, E. B. N. (2006). Design research in 2006. Design Research Quarterly, 1(1), 1-8. Sanders, E. B.N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign: International Journal of CoCreation in Design and the Arts, 4(1), 5-18. Van der Lelie, C. (2006). The value of storyboards in the product design process. Personal Ubiquitous Comput., 10(2-3), 159-162.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 A Design Research Approach to Developing User Innovation Workshops in Second Life Remko Helms Utrecht University, The Netherlands Elia Giovacchini Utrecht University, The Netherlands Robin Teigland Stockholm School of Economics, Sweden Thomas Kohler Innsbruck University, Austria Abstract The Design Science Research approach is increasingly being applied in the field of Information Systems (IS) research. The philosophy behind design science research is that new scientific knowledge can be generated by means of constructing an artifact, and the core of this approach is a problem-solving process used to develop the artifact. As virtual worlds are a relatively new IS medium, limited attention has been paid to investigating the use of design research in virtual worlds. Nevertheless, it is considered a relevant approach as much research in the field of virtual worlds involves the design of virtual spaces to support some kind of business activity. As such, the research purpose of this paper is to investigate the use of the design research approach in virtual worlds. In this paper, we describe and take a practical perspective of a specific case study in which design research was developed and used for a specific project. The specific project in focus is the development of a user innovation workshop inside Second Life for a start-up company interested in gaining insights and ideas for the development of its product. Keywords: virtual world; design science research; user innovation; entrepreneurship

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

A Design Research Approach to Developing User Innovation Workshops in Second Life Introduction Virtual worlds or immersive environments, such as Second Life and Active Worlds, have been around for some years now. After the initial hype that virtual worlds would unleash new unlimited commercial success, the focus is now on more pragmatic and serious applications of virtual environments. Several authors have identified a classification for how both profit and non-profit organizations can use virtual environments, including areas such as marketing and PR, support for mass customization, virtual markets/shopping, communication and collaboration, consumer research, innovation, virtual education and learning, and recruitment (e.g., Barnetta, 2008; Breuer, 2007; Grosser and Klapp, 2008). Whether it is running shoes to be displayed or new courses for teaching students, in both cases a virtual space is needed for the virtual activities to take place. In most cases, it requires that a new virtual space be created in a virtual world, such as Second Life, that suits the organizationâ&#x20AC;&#x2122;s specific needs. Such a virtual space consists of a certain landscape and or building, e.g. shoe display or classroom, and several objects, e.g., shoes or discussion boards, with which avatars might interact. However, the design aspects of virtual spaces in the virtual environment have not received much attention in the literature. Still this is a relevant aspect as transferring something from the physical world to the digital world might require a different approach (Parsons et al., 2008). Moreover, the design process itself requires attention to ensure that the various design needs are inventoried, several alternatives explored, and that finally a virtual space is created addressing the initial needs. Such an approach is valuable for practitioners but also for researchers who focus on the creation of an artifact to create new knowledge. In this paper, we turn to the Design Science Research (DSR) approach that has gained considerable popularity in the IS domain as a research method in which the IS development method itself or the outcome of the development process is the subject of study (Hevner, March, Park, & Ram, 2004). We believe that the DSR as a research method can also be useful for those researchers developing virtual spaces or objects within these spaces. This leads us to the following research question for our research: RQ: Can we use Design Science Research as an approach for developing virtual spaces in virtual worlds?

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

To investigate our research question, we describe a case study in which we applied the DSR approach to build a virtual space and virtual objects with the purpose of conducting virtual user innovation workshops in Second Life. The case study company that is involved is a Swedish internet startup, RunAlong, in the e-services industry. RunAlong is developing an international web community, primarily for women joggers, that is entirely designed through a user-driven innovation approach based on a series of physical user innovation workshops. The beta site was launched in Sweden in the summer of 2009. This paper describes the design (and application) of virtual user innovation workshops that were based on the physical workshops with the purpose of collecting further information from runners from across the globe. The paper is structured as follows. In section 2 an overview is provided of what is known in the literature about the design of virtual spaces for virtual worlds, with a focus on the design process. Next, the Design Science Research approach is discussed in section 3 describing the background, design, and examples. In section 4 the application of the DSR approach is demonstrated and we explain how we designed the virtual user innovation workshops and how it has been evaluated in several iterations to improve the design. Finally, the conclusions, limitations, and implications for practice and theory are discussed in the conclusions. Design of Virtual Environments Regardless of the promising opportunities provided by virtual worlds for real world companies, one major challenge impeding development is the lack of interest in virtual corporate places among avatars. On a general level, many reports point toward nascent presences being ghost towns (Rose, 2007), and the SL community is more interested in their own homegrown activities (Au, 2006). To encourage visits and engagement, the design of the virtual environment is critical. Only when avatars experience an environment that features immersive and appealing surroundings as well as interactive and engaging objects will they visit the place, spend time there, and return again. Previous researches (Donovan & Rossiter, 1982; Crosby, Evans, & Cowles, 1990) have also found evidence of the influence upon user and customer behavior generated by the layout of a physical environment as well as by the objects present in it. One study by Bitner (1992),

Journal of Virtual Worlds Research – Design Research Approach

introduced a model focusing on the environment influences upon the behaviors of customers and employees in the service industry and stated among their conclusion that “the physical setting can aid or hinder the accomplishment of both internal organizational goals and external marketing goals.” A different angle was investigated by McCoy and Evans (2002), who looked at the relation between the physical environment and creativity, identifying the existence of a relation between the creative potential of a subject and the features present in an environment (e.g. the view of a natural environment promote creative performance). In view of this established body of knowledge about the importance of the environment, it is no surprise to find an advanced literature about the design of web-based systems, including also a number of useful design principles. However, virtual world research that has emerged only recently has not yet been accompanied by any theoretical development that directly informs the conceptualization of virtual environments and guides their design. For the web-based context, Nambisan and his colleagues’ work (Nambisan & Baron, 2007; Nambisan & Nambisan, 2008) contribute to the knowledge base that is the basis for this research. The authors studied customers’ interaction experiences in the context of online product forums and proposed an analytical framework suggesting that virtual co-creation systems have to consider four experience dimensions – pragmatic, sociability, usability, and hedonic – in order to serve participants needs. The first aspect relates to the customer’s experience in realizing product-related informational goals in a virtual customer environment, while the underlying social and relational aspects of such interactions form the sociability component. The usability dimension is defined by the quality of the human-computer interactions. Finally, interactions in virtual environments can be mentally stimulating or entertaining, referring to the hedonic component. Based on these four components, Nambisan & Nambisan (2008) suggest a set of implementation principles and strategies commonly used in online environments such as 1) design to encourage customer innovation, 2) link the external to the internal, 3) manage customer expectations, and 4) embed the virtual customer environment in CRM activities. In a seminal article, Hoffman and Novak (1996) proposed that creating a compelling website depends on facilitating a state of flow. Flow is the term introduced by Csikszentmihaly (1977) to describe a highly enjoyable and rewarding ‘optimal’ experience, where challenge and skills match. Flow has been applied to various online activities, such as browsing (Novak et al.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

2000), playing games (Chen 2007; Hsu and Lu 2004) or engaging in computer-mediated communications (Ghani and Desphande 1994; Trevino and Webster 1992). Csikszentmihalyi (2002) identified the following elements that determine flow: clear goals, immediate feedback, balance between challenges and skills, merge of action and awareness, exclusion of distractions from consciousness, no worry of failure, self-consciousness disappears, distortion of sense of time, and an autotelic activity. For the internet context, additionally telepresence and interactivity are considered antecedents of flow (Hoffman and Novak 1996). However, in a recent update Hoffman and Novak (2007, p.17) point out that â&#x20AC;&#x153;In examining flow in virtual worlds such as Second Life, there are a number of ways in which our original conceptual model (Hoffman and Novak 1996) could be augmented.â&#x20AC;? Indeed, while the application of the findings of web-based customer integration research to virtual worlds may provide some interesting insights, the translation is difficult as virtual worlds are in some respects significantly different from the traditional web. Navigation in a 3-D environment, avatar-mediated communication and the interactivity with virtual tools pose unique issues for virtual design. Only few recent articles directly inform the design process of virtual environments. For instance, Murphy, Owens, Khazanchi, Zigurs & Davis (2009) take a social-technical stance and present a conceptual model for metaverse (i.e. virtual worlds) research consisting of five components: 1) the metaverse 2) avatars, 3) behaviors, 4) metaverse technology capabilities, and 5) outcomes. The authors highlight the interaction of avatars and name a range of topics such as representation, presence, and immersion. However, the article neither provides concrete insight about the design of the virtual environment nor about the design process itself. Similarly, Drettakis, Roussou, Reche & Tsingos (2007) do not consider creating a particular virtual environment but focus on the tools needed to create a virtual environment in the field of architecture and urban planning. One important criterion for improving such tools is the level of realism provided. The study provides useful insights in the design process of such a tool, as they explicate how they specified requirements, developed, and improved a prototype based on testing and evaluation. A third article by Parsons et al. (2008) explores the usefulness of virtual worlds, to create a learning environment. Before building the learning environment, the researcher should develop an analytical framework to guide the thinking about developing virtual learning environments.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Looking at the design process of the virtual university, the authors recognized the potential of virtual environments and developed the analytical framework for that purpose. Although, the actual design process of the virtual environment has not been made explicit, one can identify concrete steps such as needs analysis and development. While this early research sheds some initial light about the design of virtual environments, to date the designers of virtual world spaces have a hard time finding useful information that inform their design decisions. As such, our intent in this paper is to investigate whether we can use Design Science Research as an approach for developing virtual world spaces. The Design Science Research Approach In this research, we propose to investigate the application of a more structured and rigorous approach, i.e. Design Science Research (DSR), towards the design and development of virtual spaces in virtual worlds. The Design Science Research approach stems from the Engineering Sciences where design methodology has been studied for a long time (Cross, 1984; Eekels & Roozenburg, 1991). The first publications about the application of design principles to IS research date back to the early 1990s with publications from researchers such as Nunamaker, Chen & Purdin (1991) and March & Smith (1995). Through publications of Hevner et al. (2004), Vaishnavi and Keuchler (2004), Peffers Tuunanen, Rothenberger & Chatterjee (2008) and Jones & Gregor (2007), this approach has increased of late in popularity among researchers in the IS domain. At its core, DSR is a problem solving process that is used to develop an artifact similar to design in the engineering sciences. Furthermore, the philosophy behind DSR is that scientific knowledge can be generated by means of constructing an artifact (Guba & Lincoln, 1994; Hevner et al., 2004; Vaishnavi & Keuchler, 2004). According to Hevner et al. (2004), this approach is supposed to bring more rigor to the IS domain that is focused on studying new IT artifacts and its applications. These IT artifacts can be in the form of constructs, models, methods, instantiations, or better theories and are developed to enable a better understanding of the development, implementation, and use of information systems (Vaishnavi & Keuchler, 2004).

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Although there are many contributions to the domain of Design Science Research, Peffers et al. (2007) are the first to propose a comprehensive design science research methodology. They define a methodology, based on a definition by DM Review, as â&#x20AC;&#x153;a system of principles, practices, and procedures applied to a specific branch of knowledge.â&#x20AC;? Principles of DSR are already described above to some extent and are further elaborated in the several publications that are referenced. The Practice Rules are according to Peffers et al. (2007) very well described by Hevner et al. (2004) who provide seven Design Guidelines. Finally, procedures have been defined amongst others by Vaishnavi & Keuchler (2004), and Peffers et al. (2007) further extend this process description and demonstrate its use in four case studies. Both the practice rules and procedures will be elaborated in more detail because it concerns the methodology that we propose to apply to the design of processes and corresponding environments in virtual worlds. As mentioned above, the practice rules as defined by Hevner et al. (2004) consist of seven guidelines for DSR. The first two guidelines concern the outcome of the design research and problem awareness. In other words, a new and innovative artifact needs to be developed (guideline1) that is a response to a clear and relevant business problem that is identified by the researchers (guideline 2). The next four guidelines concern the design and development of the IT artifact. First of all, the utility, quality and efficacy of a design artifact must be demonstrated through evaluation methods (guideline 3). This requires that proper evaluation measures and methods are defined before the start of the development phase. Second, the outcome of the research should have a clear research contribution and should not be limited to an artifact that is only useful for practitioners (guideline 4). Third, proper research methods should be applied in developing the artifact (guideline 5). Fourth, the design process is typically a searching process in which several iterations are needed to tweak the artifact to the initial requirements (guideline 6). Thus, it is important in the research design to plan for such iterations. It should be more than just a trial and error process and should have a good theoretical basis and use established data collection and analysis methods such as for instance case study research. Finally, the last guideline is about communication of the research (guideline 7). As with other research, its results should be published in order to create a cumulative knowledge base.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Vaishnavi & Keuchler (2004) describe the procedures of the DSR methodology through five steps that they adopted from Takeda, Veerkamp, Tomiyama & Yoshikawam (1990) as they are very descriptive and accentuate the problem solving nature of DSR to. The five steps are presented in figure 1, and although it suggests a sequential order, there can be some overlap in the steps and several iterations can also take place, especially in the development and evaluation phase.

Figure 1. Design Science Research general steps (Vaishnavi & Kuechler, 2004).

In a generic way, the steps can be explained as follows. The first step, problem awareness, is the realization that there is a particular problem in business, society or science. Once the problem has been defined, one can start to investigate the problem at hand a bit further and search for any available literature and then to suggest a possible design solution in the form of an artifact â&#x20AC;&#x201C; step 2. In step 3, the artifact, which should solve the identified problem, is developed. After building the (prototype of the) artifact, it needs to be evaluated against predefined evaluation criteria (step 4). During the process of developing and evaluating the artifact, questions might be raised that require a re-formulation of the problem resulting in further iterations (step 5). Moreover, the development and evaluation process are iterative, as the developed artifact is not expected to be right the first time. Below we map the seven guidelines

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

onto the five steps (table 1) and in the next section, we elaborate on this five-step process using examples that are relevant to our research.

Table 1. Mapping of seven DSR guidelines onto the five steps.

Two examples of Design Science Research are by Gemmil et al. (2004) and by Levantakis, Helms & Spruit (2008). The research by Gemmil et al. (2004) concerns the design, development and deployment of directory services middleware for scalable multimedia conferencing applications. Although they do not refer to the formal application of a Design Science Approach in their paper, it clearly complies with at least five of the seven DSR guidelines mentioned above. They start with a clear problem description and formulate design challenges, e.g., related to security in large-scale conferencing applications, which are the main criteria for their new design. Then they actually build their design in a test-bed and test to what extent their design solves the identified problems. Their paper shows the result is an artifact, they follow a problem solving approach, and they publish about the research in a ViDe H.350

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

â&#x20AC;&#x2DC;cookbookâ&#x20AC;&#x2122; to make their results widely available. The second example involves the application of a reference method for knowledge auditing as the artifact of the research (Levantakis et al., 2008). For developing the reference method, they use a DSR approach based on the DSR process mentioned above and combine it with Method Engineering techniques. Based on a literature review, several current information and knowledge auditing methods are found. The creation of the reference method is based on the idea of the smallest common denominator of the other methods. After developing the reference method, it was then tested in a case study organization to demonstrate its value and possible shortcomings. In this case, the results were also published in a separate report containing a full description of the reference method. Case study: Design process of a user innovation environment We now turn to our research in which we investigate the application of DSR to a case study. Below we describe how we applied the five steps of the DSR approach to the development of a user innovation workshop in Second Life that included the design of the virtual space in which the workshop takes place as well as virtual objects. Step 1: Problem Awareness The first step concerns awareness about a relevant business problem. In this research, problem awareness was initiated by an entrepreneur in the context of her start-up, RunAlong.se, a web community for joggers. The entrepreneur had adopted a user innovation approach for her venture, involving the users from the ideation phase through the service release process in order to tailor the product to user needs. During the process of involving the users in a series of workshops in the physical world in Stockholm, Sweden, the entrepreneur discovered several significant shortcomings with this process. The workshops were onerous and more importantly they were limited in terms of insights because investigating only very local users gave a very restricted view. Due to limited financial resources, the entrepreneur was therefore looking for potential ICT supported solutions to bring runners together from all over the world to discover their specific needs and values and to incorporate these into the development of her web community for joggers. Participation by one of the authors in the physical workshops in

Journal of Virtual Worlds Research – Design Research Approach

Stockholm led to discussions with the entrepreneur regarding the limitations of these physical workshops and her interest in investigating an alternative approach to user innovation. Step 2: Suggestion In this step, potential areas for solving the problem are researched and a suggestion concerning a solution is made. Based on the input of RunAlong.se, we searched the existing literature concerning user innovation (c.f. von Hippel) and co-creation processes supported by ICT and also studied the practices of international companies such as Nokia, Philips, Coca-Cola, and Toyota in this area. The latter three aforementioned companies were chosen as they have been experimenting of late with the user innovation/co-creation process in virtual worlds, e.g., showing new products to (potential) customers and collecting feedback that is then fed into the innovation process. We found that the approach towards innovation in virtual worlds has been typically based on the traditional literature on user involvement in innovation in physical worlds. Two relevant contributions in this field are the Lead User Method (LUM) and the Co-creation approach, where the first is the more formalized of the two approaches (Herstatt & von Hippel, 1992; Olson & Bakke, 2001; Hienerth & Pötz, 2006). The core element of LUM is the involvement of lead users, or those users who face needs that will be in a marketplace before the bulk of that marketplace realizes these needs. The most recent version of the lead user method consists of four steps (Hienerth & Pötz, 2006), the last being the lead user workshop (LUW) where lead users from diverse fields and domains selected in the previous steps are brought together in one location to generate innovative ideas. The second approach is co-creation, which doesn’t present a formalized method yet. Co-creation is less concerned with the type of users involved, it rather focuses on orchestrating high-quality interactions, also called experiences, as the key to unleashing innovation (Prahalad & Ramaswamy, 2004). We considered both approaches in designing our user innovation approach within a virtual world, providing valuable structures upon which to elaborate. One common element we found between the two approaches was the use of workshops to unleash user creativity. The use of workshops is seen as a key element in successful innovation processes since they enable and support interaction between users as well as different ways of involving them. Additionally, the way in which users are involved in the innovation process is argued to have a strong influence on

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

the innovation outcomes (Gruner & Homburg, 2000), suggesting that design activities that maximize the engagement of the users (Magnusson, Matthing & Kristensson, 2003) lead to the most innovative outcomes. Diversity of user backgrounds is also acknowledged as an element supporting innovation because it represents an opportunity to create new interdisciplinary insights, especially when supported by a conducive process and a stimulating environment (Grant, 1996; Swan, Newell, Scarbrough & Hislop, 1999). Virtual worlds can greatly facilitate both these conditions as engagement can be created through various means, such as simulation or role play, and the diversity of user backgrounds is possible due to the ubiquity of virtual worlds (Ondrejka, 2007). As a result, we were of the opinion that the design of virtual user innovation workshops would have a clear research contribution to the area of user innovation in addition to being of use to the entrepreneur. The potential of workshops within virtual worlds for supporting user innovation led us then to suggest this approach to the entrepreneur as a solution to her problem uncovered in step 1. The entrepreneur immediately accepted our proposal as she felt that it solved her problem very well. Subsequently, we then focused on developing proper evaluation measures and methods regarding the utility, quality and efficacy of the virtual workshop and environment Step 3: Development Taking into account RunAlongâ&#x20AC;&#x2122;s needs as well as the insights from the literature, we developed the virtual world user innovation process outlined in Table 2. The process comprises four steps, where the name of each step indicates its goal, complemented with a customized environment within the virtual world of Second Life. According to the practice of LUW (Hienerth & PĂśtz, 2006), the process is an intertwining of discussion in small groups as well as a plenum discussion to favor the interaction without overlooking the knowledge transfer. The duration of the entire process was set to about 75 minutes, leaving a time buffer before exceeding the allotted 90 minutes, generally considered the upper-limit within virtual worlds after which the participation and contribution tend to decrease sharply.

Journal of Virtual Worlds Research – Design Research Approach

Table 2. Process steps and literature references.

We chose the virtual world of Second Life as the “site” for our workshop due to its relative user-friendliness and the ability to create a process easily and relatively cheaply as well as the ability to reach out to a diverse set of users across the globe. For developing the workshop environment in Second Life, we carefully evaluated the principles guiding the design of digital environments, especially taking into consideration the characteristics of virtual worlds. A relevant study is the work by Nambisan & Nambisan (2008) mentioned above, concerning the design of innovation activities in virtual customer environments (VCEs), based on the four components constituting the customer experience: namely, pragmatic, sociability, usability, and hedonic. According to Nambisan & Nambisan (2008), the effectiveness of customer participation in a VCE can be greatly increased through leveraging and emphasizing the digital environment’s features as well as its design. Among the principal actions suggested to favor the customer experience are features that favor social cues offering clear guidance about the process, thus allowing for a high degree of autonomy. These features are to then be complemented by the use of simulation capabilities that increase the user experience. Recognizing that Nambisan & Nambisan’s (2008) research focused on the traditional web, we needed to make additional adjustments accounting for the specifics of virtual worlds. Ondrejka (2007) indicated four critical areas of intervention: 1) process duration, 2) availability of multiple communication channels, 3) a playful environment that 4) solicited the user to participate and being active during the process. The combination of the work by Nambisan & Nambisan (2008) and that of Ondrejka (2007) provides key design principles for designing virtual spaces that are applied in this research.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Below we discuss first the basics of the virtual workshop environment in Second Life and then the workshop process we designed to take place in this environment. The Digital Environment in Second Life As suggested by both Nambisan & Nambisan (2008) and Ondrejika (2007), participants within digital environments benefit greatly from the presence of visual cues. In the case of a threedimensional environment, each location and object can carry a specific meaning that can be used to guide the users through the process. In view of this opportunity, the workshop environment was designed on two levels and further divided into smaller areas to support the user focus during the process (Figure 2). The first level of the environment featured a welcome area with informative boards showing details about the process and a running track. The latter was built based on one of the VCE design criteria (Nambisan & Nambisan, 2008), suggesting that â&#x20AC;&#x153;flow technologiesâ&#x20AC;? create stimulating experiences leading to creativity. The second level of the environment is composed of three platforms: one main platform and two elevated smaller platforms. The main platform is dedicated to the plenum activities of the introduction and the concluding activities of discussion and voting with the support of an interactive board. The two side platforms are allocated to break-out sessions for brainstorming in an environment equipped with questioning tables, sitting pods and inspiring posters showing pictures of runners exercising in different places and weather conditions.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Figure 2. RunAlong Innovation Workshop Area.

The Workshop: User innovation approach within a virtual world Step 1: Get Inspired When the participants arrived, they were presented with several informative panels providing a summary of the workshop steps. In addition, they were invited to receive some running gear consisting of running shorts, a RunAlong t-shirt and running shoes. The giving away of running gear was designed to create a feeling of group spirit by dressing in the same gear as well as to provide an ice-breaking moment for people to interact with one another. The avatars were then invited to take an immersive run (Figure 3) on the track around the workshop area. These activities played a great function in the desirability dimension, stimulating the playfulness of the participants and enabling them to experience the situation, as suggested by Gladwell (2005) as crucial in gaining insights otherwise difficult to acquire only through thoughts. In addition, these first activities were useful in creating interaction and interest in the event since workshop participants did not all arrive at the scheduled time. Individuals thus had something to do while they waited for the event to start once all the participants had arrived. Once all participants had arrived and had received the running gear as well as taken a run, all participants were asked to move to the plenum area on level 2. When all were in place in the plenum area, the workshop

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

moderators then provided a brief introduction of the workshop purpose and some insights into the upcoming activities. Step 1 lasted around 10 minutes.

Figure 3. Get Inspired, step 1.

Step 2: Get active After Step 1, the moderators split the participants into two groups based on where the avatars were sitting on the plenum platform, i.e. those who were sitting on the left side of the platform were asked to move to the left break out session area and those on the right to the right break out platform. The break out sessions (Figure 4) took place on the dedicated break out platforms with a group of participants and a facilitator. Discussions were driven by the questions that appeared on the discussion table controlled by the moderator. The questions focused on the challenges faced by the participants and as well on emerging trends they have experienced or have learned about in their local environments, thus unleashing knowledge for discussion and new ideas (Table 3). Before the start of the discussion, one of the participants was invited to be the note taker for each group for the session. The facilitator moderated this discussion as well â&#x20AC;&#x201C; encouraging individuals to contribute as well as to keep the discussion moving. Step 2 took about 15 minutes.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Figure 4. Get Active, step 2.

Step 3: Get creative. The participants continued working in the break out setting (Figure 5) and were invited to discuss in detail the aspects concerning the development of a web community for runners (Table 3). The values were the first theme of discussion, asking what participants see as the most appropriate values to be represented in the web community with which they can identify themselves. Then the discussion moved forward to brainstorming about features that the participants would consider useful and attractive for perspective runners, i.e. users of the web community. The facilitator continued to moderate this discussion. Additionally, the note taker was given the option to continue to take notes or to pass this task to another participant. Step 3 took about 15 minutes.

Table 3. Break out session questions.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Figure 5. Get creative, step 3.

Step 4: Get critical. After finishing the discussions in the breakout sessions, participants gathered together on the main platform, where the note takers summarized the outcomes of their discussion group for the other group. Here the goal was to exchange ideas and to create a moment of discussion between the two groups. The group was then introduced to a tool called the BrainBoard with a trial session in which the facilitators demonstrated how to use the board. The purpose of this session was to encourage the participants to brainstorm around two areas for the entrepreneur: 1) what values the website should embody and 2) what functions the website should offer. Through the use of the BrainBoard, the participants were able to write their values and ideas features onto the board through the Second Life chat function as well as to vote on which ideas and then to vote on their favorite ones. The purpose of the Brainboard was to enhance interactivity in the session, allowing the users to interact with a tool and to easily display their own ideas as well as to summarize the outcomes of the workshop (Figure 6). The session concluded with a short wrap up from the entrepreneur commenting on the session and the ideas proposed. This final step took about 20 minutes.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Figure 6. Get critical, step 4.

Step 4: Evaluation. To evaluate the user innovation workshop process and the Second Life environment supporting it, four workshop sessions were organized, evenly distributed between the months of August and October 2009. The recruitment of participants for these workshops targeted enthusiastic runners and experts in web-design, experience design, and community development as well as individuals interested in innovation in virtual worlds. Contrary to the lead user method (von Hippel, 2005), selection of the participants relied on the potential of users to self-select themselves, driven by their self-interest in sharing their knowledge to potentially benefit from the results (Harhoff, Henkel & von Hippel, 2003). The potential participants were invited to join the workshops through multiple digital communication channels: advertising banners on numerous sites and group notices within Second Life, announcements posted to related groups on social media sites like Facebook and LinkedIn, Twitter, and announcements on traditional internet channels such as mailing lists dedicated to Second Life users and information scientists. The recruitment attracted a total of 21 participants, mainly from the United States and Europe, mostly sharing a passion for running and the enthusiasm in supporting the development of the webcommunity. Noticeably, the sample presented a large number of academics and researchers, likely due to their higher involvement within virtual worlds. The total number of participants was comparable to the number of participants in RunAlongâ&#x20AC;&#x2122;s physical workshops and to user innovation workshops in general (Herstatt & von Hippel, 1992; Hienerth & PĂśtz, 2006).

Journal of Virtual Worlds Research – Design Research Approach

In line with Drettakis et al. (2007), each workshop was evaluated by the authors through direct observation during the workshop execution. In addition, at the end of each workshop, the participants were invited to take an online survey through clicking on a presentation board that then opened a web browser with an online survey (using Survey Monkey). Participants were encouraged to answer the survey through the offer of 500 Lindens (just under USD 2) for a completed survey. The first survey section covered questions relating to the participant’s demographics: location, age, occupation, running experience, SL experience, and innovation experience. The core of the survey was then based on the four components of Nambisan and Nambisan’s (2008) VCE experience to better understand how well our workshop fulfilled these components: 1) pragmatic, 2) sociability, 3) usability, and 4) hedonic. In addition, we asked one question related to “flow” (Csikszentmihalyi, 1996), which is tightly linked to creativity (Amabile et al., 1996) as previously explained. The interview questions are presented in the appendix. In total we received 18 completed questionnaires out of 21 total for a response rate of 86%. After the series of workshops, a number of participants were also contacted to further discuss the process as well as their experience during forty-minute interviews based on a semistructured template. The interview questions can be found in the appendix. In total, we interviewed 10 participants as well as the entrepreneur. Evaluation Results The process evaluation considered the participants’ perspective, collected through a total of 18 completed questionnaires and 10 interviews among a representative sample of the participants selected on the bases of their answers to the survey as well as to reach participants who did not complete the survey. The data collected from the participants showed a very positive sentiment with regard to the capacity of pursuing user innovation within virtual worlds. One participant commenting on the workshop stated: “I did not expect the second life platform to be so suitable as it was for this kind of workshops. Also, the voice chat and chatbox-chat ran alongside each other really nice. Before the workshop I did not expect it to run so smoothly.” The analysis of the results aims to shed light on the strength of the process, virtual space and hence the virtual world in fulfilling the VCE dimensions. Our analysis shows that the three aspects of hedonic,

Journal of Virtual Worlds Research – Design Research Approach

pragmatic, and usability were considered valuable during the idea generation and screening while the aspect of sociability was not discussed to a high degree. Hedonic. Several participants described the process of taking a run on the track as fun and even if there was some initial skepticism, they found themselves recalling one of their usual runs in real life. This was a first indicator of hedonic fulfillment but was not considered the only one. Receiving running gear was commented on by several people as supportive for the attitude and perceptions of pursuing a common purpose. One person commented, “I liked the clothing that everybody was wearing – taking everybody into the spirit of the fact that there was an overarching goal and a consulting capacity.” However, one of the participants, noticeably using SL for the first time, indicated having little inspiration from this activities suggesting from his perspective the use of a short movie about running. The activity that, however, was found more stimulating to the hedonic dimension, regardless of the participant’s background, was the break out sessions because the storytelling encouraged people to think about their own experiences while making connections with other participants, unleashing their creativity and a sense of pleasure from having contributed. Pragmatic. We found a prevalent trend of people who recognized an overall easiness in appropriating new knowledge in the workshop, and numerous ideas emerged in the first part of the break out sessions in which participants discussed the needs and trends of runners in their local environments. The opportunity to learn more about the situations encountered in different countries was appreciated by all the participants, in line with the finding of Harhoff et al. (2003) who proposed the beneficial effect of diversity in supporting discussions. In very few occasions did language appear to be an issue in terms of hindering some non-native speakers from contributing more actively. However, some of the participants felt like they were unprepared for some of the discussions as one participant commented, “Well, I could contribute what I knew, I like running, but I don't know how much I actually know about the needs and trends in my home market. So my contribution was based on the observations while running around.” This was, however, in line with the research team’s expectations and provided the ground to seed further discussions. Few participants were inclined to have an initial presentation illustrating the current web community for runners but recognized the likelihood to inhibit the generation of novel ideas.

Journal of Virtual Worlds Research – Design Research Approach

Usability. The design of the environment was highly appreciated among the participants, who indicated that it was easy to navigate due to the wide use of signs and boards conveying the right amount of information and at the right time to support the process. A second aspect that greatly impacted the usability of the process was the presence of facilitators, considered crucial in orchestrating the break out sessions and supporting people encountering either technical or practical difficulties. However, some problems did emerge, fundamentally related to the use of the two communication channels, chat and voice, at the same time and with the use of the BrainBoard. The two-communication channels, created an initial “frightening” feeling among the inexperienced participants, but after they became familiarized with them, they found them useful because they allowed participants to share ideas at anytime without the risk of forgetting them. The second issue concerned the use of the Brainboard, a new tool for the majority of the people, requiring a small learning curve to understand the functioning and in few cases requiring the participant to ask for help from the facilitator to input their ideas because they could not successfully interact with it. VCE dimensions, however, provide only a partial view of the ability of virtual worlds to support innovation. As mentioned above, research suggests that the state of “flow” is an important element in designing innovation processes (Csikszentmihalyi, 1995). During the interviews as well as in the survey, we found mixed answers with respect to flow. Interestingly, those who did not experience flow were those who admitted being busy in the physical world with other activities simultaneously. For example, one person even said that she was multitasking as we heard her phone ring on several occasions, and she said that she had to participate in a real life telephone conference at the same time. The inexperience of some participants in managing their avatar was also found by few as a cause of distraction. However, while this slightly affected the flow feeling, it is a clear indication of engagement with the technology. In some cases, a strong feeling of a state of flow was noted in the survey answers. For example, one person stated, “At one point, I suddenly realized that it was a virtual discussion, somehow it did feel like a real-world discussion.” Our last step in the evaluation was to discuss with the entrepreneur, highlighting aspects of the workshop process itself as well as the outcomes of the workshop (e.g., values and features) gained through interviews and the analysis of the results. Overall, the entrepreneur was

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

enthusiastic about the workshop process and was in agreement with the other participants that the breakout sessions were the most absorbing activities. The absence of a drawing tool in the workshops appeared to challenge the entrepreneur, who considers such a tool as a valuable complement for the participants to visualize ideas concerning web design. Looking at the outcomes generated, the entrepreneur found several valuable inputs with great market potential that were not developed in the physical workshops. A comparison between the features selected by the participants and the ones discussed during the break out sessions highlighted a rather high capacity of the group in contributing emerging potentially relevant ideas. Step 5: Conclusions. While we did not find that the problem revealed in step 1 needs to be reformulated, in our design and running of the workshops, we did find that these virtual workshops can work very well in terms of facilitating user innovation and do help to solve the problem identified in step 1. This is supported by the interviews with some of the workshop participants and the entrepreneur who confirms that the outcome is a workable avatar based innovation process that produced results that are considered valuable by the entrepreneur. In terms of evaluating and refining the workshop, our evaluation approach in which we both conducted interviews as well as online surveys enabled us to refine the workshop three times. For example, one refinement concerned the note taking activity, which initially was done on note cards, but observing the participants during the first workshop we realized that the local chat could have been a more suitable tool for the purpose. This is because it allows sharing the salient points of the discussion with all the participants, who can then directly contribute and reflect upon them, increasing the interactivity and also stimulating the hedonic aspect of the project. In general, the improvements were more of a fine-tuning nature than of an overhauling one. An issue that remains after the iterations is the technology hurdle. During the workshop, it became clear that for some participants it was their first or second experience in Second Life meaning that they were still struggling to find out how to move and communicate. Furthermore, some participants experienced problems because they could not connect to Second Life, for example from their work office, because firewalls blocked required Second Life ports.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Consequently, some Second Life experience is therefore desirable for running a workshop in the future. In addition, if we look to the literature that we reviewed and applied in our development of the workshops, we find that this did provide a sound basis for designing the virtual workshops. The aspects of VCEs: hedonic, pragmatic, sociability, and usability, as well as flow enabled us to ensure that we considered all these aspects in building the environment. However, as mentioned above, the aspect of sociability was not discussed that much by the participants in the evaluation step. One of the reasons for giving the participants the same t-shirt and shoes was to promote sociability by giving them a feeling of identity and sense of belonging, as discussed in the literature on communities (Wenger & Snyder, 2000). Perhaps this act was insufficient to create this sense of sociability or perhaps the aspect of sociability is not of that much importance in user innovation environments. Having said the above, one final question we need to ask is whether the approach of virtual innovation workshops was the appropriate approach to choose. Could the entrepreneur achieve the same or better results through a different approach, e.g., bringing people from across the globe physically, conducting a video conference, or even just an online questionnaire? Turning to the international entrepreneurship literature, we find that one of the major challenges to internationalization is the ability to access foreign market knowledge. These knowledgegathering activities are particularly challenging for small firms, which compared with large firms have limited financial and managerial resources and limited network and information resources as well as a lack of experience in such activities (Coviello and McAuley, 1999, Melen 2009). In recent years, attention has been turned to the internet as a means to gather foreign market knowledge; however, there is considerable debate as to whether the traditional internet actually can be used to gain experiential foreign market knowledge. Thus, reflecting on the positive outcomes in terms of results from our virtual workshops as well as the inability of the entrepreneur to afford bringing together participants physically, we posit that the development of virtual world innovation workshops was the right approach to take. Discussion and Conclusion

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Returning to our research intent to investigate whether we can use Design Science Research as an approach for developing virtual world spaces, we find that this approach greatly facilitated the development of virtual innovation workshops through a clear structure. Using this structure enabled us to first critically evaluate the problem â&#x20AC;&#x201C; was it really a problem that was faced by the entrepreneur? When we turned to step 2, the guidelines of building in evaluation as well as making a research contribution encouraged us to review various bodies of literature, e.g. user innovation and virtual customer environments, and then use these to help design the environment and the objects and then to build the structure of the evaluation. Step 3 of development encouraged us to ensure that we built in the proper research methods and step 4 enabled the continuous improvement of the workshop between iterations. The iterative nature of the design process is considered to be essential in improving the design in several cycles. During the evaluation, we found it important to involve the different stakeholders and to use different methods for collecting feedback during and after the workshops. Interestingly, step 5 encouraged us to write this paper so that we could disseminate our findings to others interested in building innovation environments. Moreover, we plan to continue our research in this area and build on these findings. Summarizing, the contribution of this research is twofold. In the first place, this research contributes to the design process of virtual spaces. It demonstrated that the DSR approach is a useful, but generic, approach and that still relatively little is known about the particular needs and criteria for designing virtual spaces. Nevertheless, we think that our research shows how the DSR approach can exert more rigor and relevance to research concerning applications in virtual worlds (of which our user innovation application is just one example). In the second place, the research resulted in an avatar-based innovation process and supporting virtual space in the virtual world of Second Life. This process is tested in several iterations, and both the feedback of the participants and entrepreneur indicated that it is a workable solution. We realize that our study also has some limitations. First of all, it was already mentioned that the DSR approach provides a generic process for the design of virtual spaces. Hence, the process should be adapted to the specific situation of the design of virtual spaces. Furthermore, we have conducted only one case study in which we created only one artifact. Further research should investigate the application of the DSR approach to other virtual world situations to

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

determine if it can be applied or how it should be adapted. While there are significant limitations, we would nevertheless like to suggest that the use of the DSR approach in virtual worlds by practitioners could enable them to better design virtual environments. Moreover, our study has revealed that the application of Nambisan and Nambisanâ&#x20AC;&#x2122;s components of Virtual Customer Environments as well as the concept of flow are valuable in the design of virtual innovation environments. We would like to then suggest five aspects of virtual innovation environments: hedonic, pragmatic, sociability, usability, and flow. Clearly, one area for further investigation is the relative impact and interplay of the five aspects to determine to what degree each impacts the user innovation experience outcomes as well as if there are other aspects that should be included. Finally, another area for further research is in the area of entrepreneurship. For example, to what degree are virtual worlds conducive to the acquisition of valuable experiential foreign market knowledge by entrepreneurs in the physical world. Another question is to what degree avapreneurs, or entrepreneurs whose primary entrepreneurial activity is in virtual worlds (Teigland 2009), can involve customers in the development of new products and services through user innovation activities in-world.

Journal of Virtual Worlds Research – Design Research Approach

References Amabile, T.M., Conti, R., Coon, H., Lazenby, J. & Herron, M. (1996). Assessing the Work Environment for Creativity. The Academy of Management Journal, 39, 1154-1184. Au,

(2006).

The

mixed

success

mixed

reality.

Retrieved

from

http://nwn.blogs.com/nwn/2006/10/why_mixed_reali.html, 23 October 2006. Au, W. J. (2006b, August 20). Adidas, Toyota come to Second Life. GIGAom. Retrieved December 10, 2009, from http://gigaom.com/2006/08/20/adidas-toyota-come-to-secondlife/. Barnetta, A. (2009). Fortune 500 companies in Second Life – Activities, their success measurement and the satisfaction level of their projects. Master thesis ETH Zürich. Bitner, M. (1992). Servicescapes: the impact of physical surroundings on customers and employees. The Journal of Marketing, 56(2), 57-71. Breuer, M. (2007). Second Life und Business in virtuellen Welten. Whitepaper. Accessed on December 3, 2009 at: http://www.pixelpark.com/de/pixelpark/_ressourcen/attachments/ publikationen/0703_White_Paper_Second_Life_e7_Pixelpark.pdf. Coviello, N. E. and McAuley, A. (1999). Internationalisation and the smaller firm: A review of contemporary empirical research. Management International Review, 39 (3), 223-256. Crosby, L., Evans, K., & Cowles, D. (1990). Relationship quality in services selling: an interpersonal influence perspective. The Journal of Marketing, 54(3), 68-81. Cross, N. (1984). Developments in Design Methodology. Chichester, UK: John Wiley & Sons Ltd. Csikszentmihalyi, M., Sawyer, K. (1995). Creative insight: The social dimension of a solitary moment. The nature of insight, p. 329–363. Donovan, R., & Rossiter, J. (1982). Store atmosphere: An experimental psychology approach. Journal of Retailing, 58(1), 34-57. Drettakis, G., Roussou, M., Reche, A., & Tsingos, N. (2007). Design and evaluation of a realworld Virtual Environment for architecture and urban planning. PRESENCETELEOPERATORS AND VIRTUAL ENVIRONMENTS, 16(3), 318-332.

Journal of Virtual Worlds Research – Design Research Approach

Eekels, J., & Roozenburg, N. (1991). A methodological comparison of the structures of scientific research and engineering design: Their similarities and differences. Design Studies, 12(4), 197-203. Gemmil, J., Srinivasan, A., Lynn, J., Chatterjee, S., Tulu, B., & Abhichandani, T. (2004). Middleware for Scalable Real-time Multimedia Cyberinfrastructure. Journal of Internet Technology, 5(4), 99-114. Gladwell, M. (2005). Blink: The Power of Thinking Without Thinking. Little, Brown and Company. Grant, R.M. (1996). Prospering in Dynamically-Competitive Environments: Organizational Capability as Knowledge Integration. Organization Science, 7, 375-387. Grosser, U., Klapp, M. (2008). Second Life. Unpublished working paper, University of St. Gallen, St. Gallen, Switzerland. Gruner, K.E. & Homburg, C (2000). Does Customer Interaction Enhance New Product Success? Journal of Business Research, 49, 1-14. Guba, E., & Lincoln, Y. (1994). Competing Paradigms in Qualitative Research. In N. Denzin & Y. Lincoln (Eds.), The Handbook of Qualitative Research (pp. 105-117). Thousand Oaks, CA: Sage. Harhoff, D., Henkel, J. & von Hippel, E. (2003). Profiting from voluntary information spillovers: how users benefit by freely revealing their innovations. Research Policy, 32, 1753-1769. Herstatt, C. & von

Hippel, E. (1992). FROM EXPERIENCE: Developing New Product

Concepts Via the Lead User Method: A Case Study in a "Low-Tech" Field. Journal of Product Innovation Management, 9, 213-221. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. Management Information Systems Quarterly, 28(1), 75–106. Hienerth, C. & Pötz, M. (2006). Making the lead user idea-generation process a standard tool for new product development. Jones, D., & Gregor, S. (2007). The Anatomy of a Design Theory. Journal of the AIS, 8(5), 312335. Levantakis, T., Helms, R., & Spruit, M. (2008). Developing a Reference Method for Knowledge Auditing. In Proceedings of the 7th Conference of Practical Aspects on Knowledge

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Management, Lecture Notes in Artificial Intelligence (Vol. 5345, pp. 147-159). Presented at the 7th Conference of Practical Aspects on Knowledge Management, Yokohama, Japan: Berlin Heidelberg: Springer-Verlag. Magnusson, P.R., Matthing, J. & Kristensson, P. (2003). Managing User Involvement in Service Innovation: Experiments with Innovating End Users. Journal of Service Research, 6, 111124. March, S., & Smith, G. (1995). Design and Natural Science Research on Information Technology. Decision Support Systems, 15, 251-266. McCoy, J., & Evans, G. (2002). The potential role of the physical environment in fostering creativity. Creativity Research Journal, 14(3&4), 409-426. Melen, S. (2009). New Insights on the Internationalization Process of SMEs: A Study of Foreign Market Knowledge Development. Published doctoral dissertation. Stockholm: Stockholm School of Economics. Murphy, J., Owens, D., Khazanchi, D., Zigurs, I., & Davis, A. (2009). Avatars, People, and Virtual Worlds: Foundations for Research in Metaverses. Journal of the Association for Information Systems, 10(2), 91-117. Nambisan, S., & Baron, R. A. (2007). Interactions in virtual customer environments: Implications for product support and customer relationship management. Journal of Interactive Marketing, 21(2), 42-62. Nambisan, S., & Nambisan, P. (2008). How to Profit From a Better Virtual Customer Environment. MIT Sloan Management Review, 49(3), 53-61. Nunamaker, J., Chen, M., & Purdin, T. (1991). Systems Development in Information Systems Research. Journal of Management Information Systems, 7(3), 89-106. Olson, E.L. & Bakke, G. (2001). Implementing the lead user method in a high technology firm: A longitudinal study of intentions versus actions. Journal of Product Innovation Management, 18, 388-395. Ondrejka, C. (2007). Collapsing Geography - Second Life, Innovation, and the Future of National Power. Innovations: Technology, Governance, Globalization, 2, p. 27-54. Parsons, D., Stockdale, R., Bowles, J., & Kamble, V. (2008). If We Build It Will They Come? Creating a Virtual Classroom in Second Life. In Proceedings of the 19th Australasian

Journal of Virtual Worlds Research – Design Research Approach

Conference on Information Systems (pp. 720-729). Presented at the 19th Australasian Conference on Information Systems, Christchurch, New Zealand. Peffers, K., Tuunanen, T., Rothenberger, M., & Chatterjee, S. (2008). A Design Science Research Methodology for Information Systems Research. J. Manage. Inf. Syst., 24(3), 45-77. Prahalad, C.K. & Ramaswamy, V. (2004). The Future of Competition: Co-Creating Unique Value with Customers. Boston, MA: Harvard Business School Press. Rose, F. (2007). How Madison Avenue is wasting millions on a deserted Second Life. Wired Magazine, 15(8). Swan, J., Newell, S., Scarbrough, H. & Hislop, D. (1999). Knowledge management and innovation: networks and networking. Journal of Knowledge Management, 3, 262 - 275. Takeda, H., Veerkamp, P., Tomiyama, T., & Yoshikawam, H. (1990). Modeling Design Processes. AI magazine, (Winter). Teigland, R. (2009). Born Virtuals and Avapreneurship: A case study of achieving successful outcomes in Peace Train – a Second Life organization. J. of Virtual Worlds Research, January 2009. Vaishnavi, V., & Keuchler, W. (2004, January 20). Design Research in Information Systems. Association

for

Information

Systems.

Retrieved

December

11,

2007,

from

http://home.aisnet.org/displaycommon.cfm?an=1&subarticlenbr=279. von Hippel, E. (2005). Democratizing Innovation. The MIT Press. Wenger, E., & Snyder, W. (2000). Communities of practice: The organizational frontier. Harvard business review, 78(1), 139–146.

Journal of Virtual Worlds Research – Design Research Approach

Appendix A – Participant’s Interviews

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Journal of Virtual Worlds Research – Design Research Approach

Appendix B – Participant’s Interviews

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Design Research Approach

Volume 3, Number 1 The Researcher’s Toolbox November 2010 What are users thinking in a virtual world lesson? Using stimulated recall interviews to report student cognition, and its triggers. Lyn Henderson James Cook University, Australia Michael Henderson Monash University, Australia Scott Grant Monash University, Australia Hui Huang Monash University, Australia Abstract: Stimulated recall is an empirically rigorous introspection data collection tool that allows the interviewer to elicit, identify and explore participants’ thinking. In this study it was used to identify the types of thinking skills and strategies employed by first year university students engaged in a Chinese language and culture lesson in Second Life. A valuable affordance of this technique is the ability to account for stimuli from both the virtual and physical environments, thus strengthening the researchers’ claims about the relationship between thinking and instructional design. This was accomplished through the use of screen capture software to record both the avatar’s on-screen activity in Second Life as well as the face of the participant (via the web camera). This data was then used during the interview, within strict methodological protocols, to stimulate participants’ recall of their thinking at the time of carrying out the activity. The value of stimulated recall over other introspection tools, within the context of this study, is discussed. In addition, methodological concerns, especially those relating to reliability and validity of data, are outlined in this article; we also use data from the study to explicate strategies to minimize those concerns. Keywords: Stimulated recall, Introspective process tracing methodology, Second Life, Virtual World, Thinking skills, Think aloud, language acquisition, education, tertiary

Journal of Virtual Worlds Research – Stimulated Recall Interviews

What are users thinking in a virtual world lesson? Using stimulated recall interviews to report student cognition, and its triggers. This article details and exemplifies the utilization of an empirically rigorous, qualitative methodology called stimulated recall. The aim of the study was to identify the types of thinking skills and strategies used by first year university students engaged in a Chinese language and culture lesson in Second Life (SL). The study also intended to identify the triggers or stimuli, both in-world and in-class, which resulted in those thinking skills and strategies. In achieving these aims a stimulated recall approach was selected because it has been validated as a reliable introspection tool and consequently allows researchers to make claims about participant in-situ activity; for instance, what participants were thinking when completing a particular action or response in SL. The study employed the stimulated recall method of using video to record participants during the lesson and then replaying it during each individual’s retrospective interview. During the interview the video was paused by the student when recalling a specific thought and/or by the researcher when he considered a pause might elicit mediating processes. A major strength of the stimulated recall interview is using the video to explore the visual cues, such as frowning or taking notes, that have been found as major indicators of mediating processes (e.g., Gass & Mackey, 2000; Henderson & Tallman, 2006). While the stimulated recall interview holds many advantages over other introspection tools (think aloud, questionnaire, and interview), no method is without limitations. The key methodological concerns—especially those relating to reliability and validity of data—are outlined in this article along with strategies to minimize those concerns. This article also provides an unbroken trail from theoretical and methodological choice and continuing through to analysis and discussion of findings. The implications will be of interest to researchers trying to understand participant activity—in this article, cognition—while engaged in a virtual world as well as to discretely account for simultaneous real life and inworld stimuli triggers. The aim of this paper is to provide readers with a useful introduction to using stimulated recall within the context of research in virtual worlds. It uses the context and data from a study of tertiary level students enrolled in a Chinese language and culture subject to illustrate the discussion about the research design and process. While some of the findings are presented here, this paper primarily concerns itself with a clear description of how researchers can enable their own research utilising a stimulated recall approach. Research Context Universities and other educational institutions continue to invest heavily in acquiring virtual worlds. However, rigorous research exploring the value of these environments in terms of cognition is scant. In response, this qualitative empirical research gathered evidence of student cognitive skills and strategies utilised during one of a series of 90 minute SL lessons in a first year university core Chinese language and culture subject. Apart from the use of SL, the subject consisted of the traditional weekly lectures and tutorial, independent study based on textbooks, and their associated language DVD. Of the 153 students enrolled in the subject, 11 students volunteered to be video recorded and interviewed.

Journal of Virtual Worlds Research – Stimulated Recall Interviews

The lesson under scrutiny in this study was specifically designed to reinforce and extend the language and culture content previously covered in the lectures and textbook. In this lesson the students had a number of learning objectives aligned with those of the subject which was contextualised within a collaborative learning activity—identifying and ordering appropriate food in Mandarin in a Chinese restaurant setting (see Figure 1). In a dynamic semispontaneous scenario, communication was centered around, but not limited to, practicing key vocabulary and phrases related to ordering food learned from the textbook and to expand on the textbook through new phrases introduced by the tutor or robots (automated avatars). Successful completion of the SL lesson could only be achieved through reading, writing and negotiating choices in Chinese text. The interaction between avatars was limited to text because one of the lesson objectives was to strengthen student use of Pinyin to input Chinese characters on a computer as well as to be able to read Chinese characters.

Figure 1. Chinese restaurant in Second Life Theory and Methodology Stimulated recall is a tried and tested data collection tool that allows the interviewer to elicit, identify and explore participants’ thinking (Gass & Mackey, 2000; Henderson & Tallman, 2006). For instance, this approach has been used to study: • teacher librarians’ mental models (Henderson & Tallman, 2006) • first year medical students’ engagement with interactive multimedia (Pausawasdi, 2001) • the relationship between teachers’ beliefs and classroom practices (Meade & McMeniman, 1992) • physician reasoning (Barrows, Norman, Neufeld & Feightner, 1982) • sports coaches’ decision making (Gilbert, Trudel & Haughian, 1999) • second language acquisition (Gass & Mackey, 2000).

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Stimulated Recall Interviews

While stimulated recall is versatile, it also has a number of limitations. Consequently, in order to strengthen validity and reduce concerns over limitations, such as reliability, it is important for researchers to recognise that stimulated recall is inherently grounded in information processing theory, the mediating process paradigm and introspection process tracing methodology. Figure 1 illustrates the relationship between the theory, paradigm and methodology that are discussed in greater detail below.

Figure 2. Diagrammatic Model of Theoretical and Methodological Framework for Stimulated Recall (adapted from Henderson & Tallman, 2006; Pauswasdi, 2001). In brief, the information processing theory delineates how learning and remembering occur by examining the way we take in, discard, act on, store, and retrieve instructional input (e.g., from the instructionally designed Chinese restaurant environment in SL) to produce an outcome (e.g., correctly using Mandarin to order appropriate food) (Anderson, 1990; Atkinson & Shiffron, 1968; Craik & Lockhart, 1972; Miller, 1956). As depicted in the

Journal of Virtual Worlds Research – Stimulated Recall Interviews

simplified diagram (Figure 2), the stimuli are sent to sensory memory where it is forgotten, discarded, or given attention and forwarded to the short term working memory for processing. Next, after being processed, it is discarded or sent to long term memory to be categorized, stored and retrieved back into working memory. Finally, the student would deliver their cognitions as learning outcomes. The mediating processes paradigm allows identification of the thinking skills, strategies, and processes of participants when carrying out a task in a virtual world. These mediating processes mediate, that is, come between stimuli (e.g., lesson task) and outcomes (e.g., selecting appropriate dishes) and are “the fine-grained elements of cognition through which, and by which, learning outcomes are realized” (Henderson, Putt, Ainge, & Coombs, 1997, p. 103). The mediating process paradigm is in contrast to the prevalent process-product paradigm in which thinking processes (e.g., recall) are assumed to have occurred because an outcome has been observed (e.g., ordering the correct meal in Mandarin). Consequently, the process-product paradigm is flawed as it cannot capture the choices, decisions and judgments made by students during a learning activity nor what actually triggered these cognitive processes (Marland, Patching, & Putt, 1992; Shulman, 1986). Introspective process tracing methodology is utilized to access mediating processes based on the tested assumption that a participant can access these at some level and verbalize their thoughts (Ericsson & Simon, 1999; Gass & Mackey, 2000; Henderson & Tallman, 2006; Lyons, 1986; Shavelson, Webb, & Burstein, 1986; Vermersch, 1999). Mediating processes can be accessed through verbal protocol analysis including: (a) introspection think aloud, talk aloud or, as it is sometimes referred to, self-revelation, (b) self-observation (retrospective interview) and (c) the retrospective (as it is occurring after the event), stimulated recall introspective-prompted interview (in short, stimulated recall interview). (a) Think-aloud Introspection Method The “think aloud” method requires the participant to voice his thoughts as a running commentary when performing a task in SL (or other situation). However, thinking aloud is not a straightforward task as it does not come easily for the participant (Branch, 2000, Nielsen & Yssing, 2003, 2004). There are three major problems reported in the literature. One: think aloud is alleged to be the most “invasive method” (Karasti, 2000) as it interferes with the participant’s normal thought routines, processes and actions. If we had utilized “think aloud” as the data collection method in our SL study, then the participants’ perceptions and actions … [would] be transferred to talk, and even if the speech is immediate and run concurrently with the thoughts – users’ attention has to shift focus from … interacting with a multi-modal interface: colours, layout, sound, graphics, animations … [that is, from] visual perception … to understanding to verbalization (Nielsen & Yssing, 2003, p.69). Thus, think aloud generates an artificial situation that adds cognitive strain in working memory (Branch, 2000; Karsenty, 2001). Two: Afflerbach and Johnston (1984), Ericsson and Simon (1984) and Langer (1993) pointed out decades ago that participants are unable to verbalize as fast as they think or act, unless they are slowed down by a particularized problem. Hence, think aloud is inappropriate for research on synchronous collaborative activities such as group work in Second Life.

Journal of Virtual Worlds Research – Stimulated Recall Interviews

Three: think aloud is seen to require participant training (Gass & Mackey, 2000). For instance, Freeman (2003) described a study in which participants had to undergo “intensive training to ensure reliability of results ... providing at least 10,000” separate verbalized think alouds before being judged reliable (Freeman, 2003, p. 11)! Although this demonstrates a fanatical adherence to clinical protocols, participant training remains an issue of validity and reliability in concurrent think aloud research (Bielaczyc, Piroll, Brown & Brow, 1995; Branch, 2000; Gass & Mackey, 2000; Hertzum, Hansen, & Andersen, 2009). These three weaknesses supported the decision to discount using the concurrent think aloud method. The study aimed to identify the thinking skills and strategies of students engaged in the lesson that required in-world interaction, including collaborative decision making. An intrusive approach that slows down the interaction would seriously influence the data being collected. In addition, since the study participants were university students with limited time, there would have been little or no opportunity for training to strengthen reliability. (b) Retrospective Self-Observation Method The retrospective self-observation method—usually an interview or written/online questionnaire based on the researcher’s interests—requires participants to report what they had being doing, thinking, strategizing, and/or feeling during a past activity, such as a SL session. It relies heavily on the participant’s memory and, without a visual (e.g., video of, or actual picture or diary used), aural (audiotape) or written (e.g., notes taken during a lecture) artifact reminder, it is open to charges of non-reliability and non-validity (Ericsson & Simon, 1993; Meade & McMeniman, 1994). This is confirmed by current research. For instance, Clarke, Fieberg and Gerdtham (2008) reported that, although retrospective self-observation can deliver large amounts of information, “the longer the period before recall, the greater the likelihood of error” (p.1275). One example is the study of 144 undergraduate students who increasingly made inferential errors about events and causes as the time increased from the original event (Hannigan & Reinitz, 2001). Similarly, Ottesen, Foss and Gronhaug (2004) found substantial perceptual errors made by their participants while Andersen and Mikkelsen’s (2008) research indicated considerable under-reporting—by 82 participants—in a retrospective questionnaire of occupational injuries over the previous month. Of particular relevance to this study is the research by Harris and Wear (1993) in which 15 undergraduate education students were asked to recall when they deviated from their planned lessons and when their students deviated from the lesson topic. Despite the short time between lesson and the retrospective self-observation these pre-service teachers averaged only 42 percent accuracy for their own deviations and 35 percent for their students’ deviations. This was considered to be a serious concern for the current study because, unlike Harris and Wear’s studies that simply required participants to recall their behaviour, it required participants to recall their specific thinking processes while completing activities inworld. One of the memory dangers with retrospective self-observation without an artifact stimulus trigger is that participants may infer plausible stories without consciously realizing that they are doing so. One reason is because we are “essentially sense-making beings and tend to create explanations” (Gass & Mackey, 2000, p.6), such as filling in omissions that were not present because they are typical of that situation (Hannigan & Reinitz, 2001; Waldmann, Holyoak, & Fratianne, 1995). This weakness has been well documented, especially the way in which details from one event or context intrude upon, and are incorporated into, memories

Journal of Virtual Worlds Research – Stimulated Recall Interviews

of a completely different event and, consequently, result in erroneous data (Allen & Lindsay, 1998; Hannigan & Reinitz, 2003; Lindsay, Allen, Chan, & Dahl, 2004). (c) Stimulated Recall Introspection Interviews Stimulated recall is also referred to as retrospective stimulated recall (as it occurs after the event to be introspected), “retrospective think aloud” (e.g., Guan, Lee, Cuddihy, & Ramey, 2006) and “retrospective user-reported” interview (e.g., Card, Pirolli, Van Der Wege, Morrison, Reeder, Schraedley, & Boshart, 2001). The goal of stimulated recall, and the other introspection tools, is to record the participant’s reported thoughts. It does not require them to become knowledgeable about their own subjective experience of reliving and reporting the initial experience. Reliability and Validity Issues. No methodology is without critics and no methodological tool is without limitations. It would be as foolish to claim infallibility for introspective reports (Lyons, 1986) as it would for other perceptual reporting. Stimulated recall has been criticized most notably on the issue of reliability (see Ericsson & Simon, 1993; Lyons, 1986; Nisbett & Wilson, 1977; Smagorinsky, 1994). In response, Henderson and Tallman (2006) proposed six issues that need consideration in order to strengthen reliability and validity: (1) timing of the stimulated recall interview, (2) capturing the data, (3) conduct of the interview, (4) interviewer training, (5) interviewee and interviewer fatigue, and (6) coding and categorizing the data. (1) Timing of the stimulated recall interview Recall accuracy is a reliability issue. Bloom’s (1954) stimulated recall study is particularly relevant. Prompting participant recall with aids within two days of the task, recall was 95 percent accurate but declined to 65 percent two weeks later. Garner (1988) also demonstrated significantly fewer recalls 48 hours later than on the day of the event. As mentioned previously, there is greater likelihood of plausible, schematic and/or causal-inferential gapfilling errors, the longer the timeframe between the event and the recall. Consequently in our study, ten of the eleven stimulated recall interviews were conducted within 15 minutes of the video-recorded SL lesson. The eleventh occurred 70 minutes after the lesson. (2) Capturing the data This study followed the well established method of video recording each participant during the SL computer lab lesson and then replaying the video during the participant’s stimulated recall interview. Eliciting participant thoughts through a video prompt has the advantage of staying close to the actual events in the sequence they occurred. Perceived as a memory retrieval cue with high associative strength, it is more likely to trigger accurate memory recall, thus avoiding guessing or reporting thoughts that they think they might have had [or what they think the interviewer would like to hear]. In fact, the process of recalling one´s thoughts appears to strengthen the overall memory of the immediate past event (Ericsson, 2002, p.984). Guan, Lee, Cuddihy, & Ramey (2006), after comparing their 24 undergraduate participants’ stimulated recall verbalizations with their actual eye movements during the task, affirmed the “validity and reliability … of stimulated RTA [retrospective think aloud] as it provides a valid account of what people attended to in completing tasks, it has a low risk of introducing fabrications, and its validity isn't affected by task complexity” (p.1253).

Journal of Virtual Worlds Research – Stimulated Recall Interviews 10

A significant enhancement in this study was the use of innovative screen capture software (Screen Flow) to record both the on-screen SL activity as well as the face of the participant (via the inbuilt web camera). Unlike previous use of videotape technology, this allowed full resolution capture of the screen, crystal clear pause during the interview, full-facial expression capture, synchronization of face and screen video capture, as well as audio capture from both the SL-world and participants’ in-class learning site. Additionally, Screen Flow did not require rendering time and allowed instant visual scrubbing (fast forward and rewind) along the video. Consequently, the participant or interviewer could move quickly forwards, backwards, jump to places of interest or return to points in the timeline as the need arose. Additionally, Screen Flow allowed for a particularly non-invasive data collection process as opposed to participant head or lapel microphones, video cameras with tripods, and roving camera operator. It provides the distinct advantage of unobtrusively capturing rich data of what students are doing in SL synchronously with their verbals and non-verbals in-class. It also captured other stimuli, such as the lecturer giving instructions to the class in the computer laboratory. (3) Conduct of the stimulated recall interview The validity and reliability of the data being collected is highly dependent on the way in which the stimulated recall interview is conducted. Three issues need consideration: (a) promoting interviewee confidence, (b) instructions to the interviewee and (c) interviewer prompts. (a) Promoting interviewee confidence: Establish rapport, mutual trust, and respect with the interviewee by being supportive, a good listener, interested in, and non-judgemental of, what the interviewee is saying. Becoming aware that they are capable of telling the interviewer about their thoughts and feelings allows interviewee realization that they are the authority on their cognitive processes during the activity (e.g., Henderson & Tallman, 2006; Marland, et al., 1992). (b) Instructions to the interviewee: These are simple, minimal, and implement the previous stipulation. At the beginning of each stimulated recall interview, there are two general statements. The first statement situates what is required within of the purpose of the interview: “What we want you to do is tell us what you were thinking during the lesson. [Interviewer pauses.] If you cannot remember what you were thinking or if you were thinking about something other than the lesson, either is okay. Just tell us.” Specific instructions about what to recall are not permissible as this contaminates the validity of the data. For instance, Ethell and McMeniman (2000) undermined the reliability of their collected data by requiring multiple tasks of the interviewee who was asked to walk the researcher through the lesson, stopping the video at any point where he would like to elaborate on what he was thinking, how he was feeling, or what thinking, beliefs, knowledge, or theories influenced any in-class decisions and observable teacher practices (p.93). As Ericsson and Simon (1993) cautioned, asking participants to vocalize their thoughts, seek out specific targets (e.g., theories), and explain them, increases cognitive load in short term memory and consequently impedes memory and recall. The second statement delineates who is to pause the video. For example, “As you are watching the replay, when you remember what you were thinking, please pause the video and

Journal of Virtual Worlds Research – Stimulated Recall Interviews 11

tell me those thoughts. I will also pause the video and prompt you to tell me what you were thinking and feeling.” This instruction reduces researcher bias of determining when the student reported (Gass & Mackey, 2000; Henderson & Tallman, 2006; Marland, et al., 1992). (c) Interviewer stimulated recall prompts: Gass and Mackey (2000) contend that researchers’ questions and responses are “the most serious of difficulties” (p. 89) as they have the potential to compromise the data. Exposure of the participant’s thoughts through adherence to tested protocols helps maximize reliability and credibility of recall. These protocols involve attention to non-directive questioning, interviewer versus participant initiation of responses, the timing of questions, and checking to ascertain recall versus the plausible story. The nature of the interviewer’s prompt is an important validity issue in participant recall and researcher bias. Prompts are used to focus respondents so that they can access their memory and then vocalize what it was that they were thinking during the activity. There is a high likelihood of obtaining validity and accuracy of recall if the interviewer’s prompts (a) are non-directive, “What were you thinking then?”; (b) also include a specific context that is being viewed, “When you heard the pronunciation of the food, what were you thinking?”; and (c) seek confirmation: “Did you think that just now or when you were in the SL lesson?” If an interviewee’s response is “I can’t remember” or “No thoughts!” then asking, “So what were you doing there [interviewer points to the screen where the video has been paused]” is usually an effective memory aid that provides valuable thoughts to the follow-up question, “Okay, can you remember what you were thinking when you did that?” If prompts are more directive, such as those used in Sim’s (2004) study – “What is the basis for your using this technique/directing the lesson in this way” (p. 353) – then the prompts are keying the participant to report those thoughts that align with the research focus. There is a high likelihood of the participant reporting and creating plausible stories as explanation rather than recalling what they actually do remember thinking. Notwithstanding their emphasis on non-directedness, Gass and Mackey (2000) commented that “unstructured situations do not always result in useful data” (p. 55). In our research, this has not been the case. Our prompts followed the strict protocols adapted from Marland, et al. (1992) and authenticated in other research (Burton, 2006, Henderson & Tallman, 2006; Patching, Putt & Henderson, 1996; Pausawasdi, 2001). What this research and our experience suggest is that researchers can take confidence in open non-directedness in terms of instructions and prompts rather than trade off reliability and validity against the fear of not obtaining information about their specific research focus. A solution to this trade-off is to conduct a modified self-observation interview (Henderson & Tallman, 2006), that immediately follows the stimulated recall interview. Questions, usually asked in a survey or interview, are now permissible. Examples are: “Can you tell me something about your relationship with your avatar?”; “What are your thoughts about learning with SL?” Notice that these deliberately remain non-directive, unlike “What strategies did you use in your SL lesson, today?” which was also included to confirm and augment those identified from the stimulated recall data. (4) Interviewer Training In contrast to the participant training in the think aloud introspective interview, interviewers need to be trained to maximise reliability and accuracy of recall. Some guidelines are proposed by various researchers (e.g., Ericsson & Simon, 1999; Henderson & Tallman, 2006; Marland, et al., 1992) as well as in this study. Three critical strategies are advocated. Two

Journal of Virtual Worlds Research – Stimulated Recall Interviews 12

have been covered above: promoting interviewee confidence and conduct of the stimulated recall interview. The third examines interviewer training. The literature generally does not identify how the researchers should be trained. However, based on the authors’ experiences in this and other research projects, it is recommended that interviewer training occurs at least twice, and that one of those sessions is scheduled just before the actual research is conducted as a timely refresher. To promote optimum interviewer success, training includes a video replay of both the neophyte and skilled stimulated recall researchers carrying out a short task (e.g., 5-8 minutes is adequate). First, the inductee is interviewed by the researcher. Then the roles are reversed. This allows a “walking in my moccasins” experience in both roles. In this research project there were 3 interviewers (one experienced in stimulated recall). After each practice the interviewers compared notes, talked through issues such as establishing rapport, appropriate prompts, and interview fatigue. (5) Interviewee and interviewer fatigue Fatigue in both interviewee and interviewer, but especially interviewer, are infrequently discussed in stimulated recall or other research literature, despite having a potentially dramatic influence on the reliability and validity of the data. The process of stimulated recall can be particularly fatiguing since the one-on-one interview typically lasts, at minimum, twice as long as the activity under study (Henderson & Tallman, 2006). The interviewee and interviewer are required to watch the video replay as well as to pause the video to allow participant verbalization of their thinking. In addition, back-to-back interviews may cause increasing fatigue across the interviews. As interviewers become fatigued their adherence to protocols and observation skills can deteriorate, not to mention their willingness to follow up potentially significant actions or utterances from the participants. Similarly, participants who are fatigued are more likely to make recall errors and overlook potentially valuable incidents. The risk of fatigue needs to be weighed against how much of the video needs to be played in order to provide sufficient data to achieve the research aims. Options are: to play all of the video; only particular segments; or to use a fast-play option wherein the play-back is set at a slightly faster speed than normal. The researchers in this study chose the fast-play option. This allowed the interview to cover more of the lesson while still allowing the interviewee and interviewer ample time to slow down, pause, or rewind the video at points of interest. (6) Coding and categorizing the data An accurate transcription of the interview is important since some utterances can convey significant implications in terms of categorizing thinking skills. The interviews are audiorecorded, usually professionally transcribed, and the transcription checked by the researchers. The video recording allows confirmation. Coding reliability is critical. In the coding process, there is constant referral to a coding schema based on the research aim. Our study followed a well established list of thinking processes, including their definitions (see Henderson, 1996; Marland, et.al., 1992; Table 1). When there is a difference, consensus among the researchers occurs through justification of the rationale for that categorization. If there is no consensus, then that mediating process is excluded as usable data. In this study, guided by the lead researcher, all researchers coded the same interview transcript and subsequently debated coding differences. Coding reliability

Journal of Virtual Worlds Research – Stimulated Recall Interviews 13

was further strengthened by every transcript being coded a second time by a different researcher; differences were debated. This means that every identified thinking skill reported in this research was approved by at least two researchers. Table 1: Cognitive Skills and Strategies Utilized by Participants with Total Number Used and Percentage of Total Type Affect

Definition of Cognitive Skill * reports feelings aroused by SL and by real-world stimuli

Analysing

reduces, breaks down whole (e.g., problem, task, issue) into parts

Anticipating

wonders about the possibility of an event, relevance of material, or content; predicts

Applying

considers the use of an idea, tactic in a different context. sorts items, ideas, skills into different groups

Categorising Comparing

Confirming

Deliberating

Diagnosing

Evaluating

identifies similarities, differences between two statements, concepts, ideas, models, situations, points-of-view, etc. judges that the ideas in SL support own tactics, practices; verifies actions, thoughts, and significance of SL engage in “thinking” about a topic, segment, etc., (type of thinking not disclosed): identifies strengths and/or weaknesses in idea, strategies, pointsof-view worthwhileness of SL, activities, own strategies, graphics, and issues of

Interviewee Example “I was feeling sort of excited!” “[When he said, ‘You have to write in Chinese’], I was really scared.” “Is this going to come back and is it going to come up with my name and say what I put down and will I be assessed as to what I’d done.” “I thought: ‘What if I click on something and it does something wrong?’ ” “I thought, ‘I wonder what kind of dishes is there?’” “I’m going to order this dish in Mandarin when I go to my favourite Chinese restaurant.” “I was thinking, that dish is some kind of meat because roe is meat.” “They know it and I don’t.”

Thinking % Skills # # 145 19.7

1.9

4.5

0.1

2.5

4.6

“I already have it – the non- 24 spicy dish!” “Oh, yeah, I’m in the right Second Life place.”

3.3

“I’m just wondering if I need to 6 talk with the waitress.”

0.8

“Huh, that didn’t work but I 24 still got something.”

3.3

“It’s hard. It’s too hard for me 107 to understand properly.” “I thought, ‘She’s sitting on the

14.6

Journal of Virtual Worlds Research – Stimulated Recall Interviews 14

Generating

Imaging

morality and ethical behavior in SL formulates one's own questions, examples, ideas, or problems; interpolating; beyond what was in SL & class creates a mental image to gain a fuller understanding

Justifying

rationalizing, explaining, and providing reasons for their actions and thoughts

Linking

associates or brings together two or more ideas, topics, experiences, tasks thinks about, reflects on, evaluates their certainty of understanding; directs own thinking, ways to troubleshoot lack of comprehension

Metacognising

Recalling

brings back into working memory an idea, opinion, fact stored in long-term memory

Reflecting

general indication of consideration over past action and response

Selecting

identifies key material, gist material, or that which is relevant to assessment. plans ways for processing and carrying out the lesson tasks, manoeuvring in SL, and

Strategy Planning

food! Yeah, well, it’s still inappropriate’!” “I was thinking: ‘If I ever go to China, if I ever go and order something in Chinese, even if I don’t like this meal, this is the one I’m going to order’.” “I’m a very picture person. I remember, ‘this one [picture of a dish] was here on the food counter and this one was over here … and because I have a memory picture, I draw connections’.” (see Fig.1) “I was thinking: ‘Where is this stupid thing? I have so many things in my inventory, how am I supposed to find one thing?’” “[I heard the voices] and then I was wondering, ‘Which teachers of the Chinese faculty they are’ [sic].” “I thought: “Don’t expect to learn anything straight away. You know, like, a little bit, but don’t expect to know the dishes at the end or remember what they were made of. Come and do it again before the exams.” (I.6) “At the start he [the lecturer] said, ‘You should ask other people’.” “What were the different names of food [in the text book]?” “I was thinking, ‘I didn’t realise I was typing something wrong! [Then I did realise and] I thought, ‘Oh, no! I’ve done it four times’.” “I thought, ‘I know those dishes don’t have the win sign, so don’t order them’.”

4.4

0.1

8.6

2.0

9.9

1.9

0.3

0.1

“If I don’t finish I can do it, 121 like, double check at home.” (I3) “Try to find some food that

16.5

Journal of Virtual Worlds Research – Stimulated Recall Interviews 15

subsequent strategies Translating

you know in Mandarin, like Mee-fun. It’s only rice, so go there and order some rice.” using their own words to “I was thinking, ‘She’s not 7 interpret, explicate, taking it too seriously; that’s and/or clarify what is irritating. It’s a difficult task happening and we really have to coordinate properly and we’re at different stages’.”

Thinking Skills TOTAL

733

1.0

100

* definitions adapted from Henderson (1996); Marland, et.al. (1992). # the numerical tally and percentage of the total number are discussed below (see Analysis). There are two issues that require close attention in order to maximize validity and reliability of the coding. These are: types of thoughts accessed and leading questions by the interviewer Types of thoughts accessed. Both the video and interviewer prompts trigger the participant’s thoughts. As depicted in Figure 3, there are two types of thoughts reported during stimulated recall interviews. These are “recall thoughts” and “hindsight reports”.

Figure 3: Types of interviewee thoughts and researcher action Stimulated recall thoughts occur during the performance of the task and are the “back there and then during the SL lesson” thoughts. For instance, “Can you tell me what you were thinking when you were trying to find a way around it?”, elicits “I was thinking, I’m never going to find this, and what am I going to do if everyone else is speaking in Chinese characters and I’m not?” Thus they are introspective thoughts and therefore coded and tallied. In contrast, hindsight thoughts only occur during the interview, and can be labelled “here and now” thoughts and are sometimes explanatory, as is seen in Figure 4 (see participant statement labelled “A”). Such hindsight thoughts may be interesting but they cannot be coded as stimulated recall.

Journal of Virtual Worlds Research – Stimulated Recall Interviews 16

Figure 4: Excerpt from an interviewee’s transcript demonstrating a leading question. Leading the interviewee. This excerpt from a transcript (Figure 4) demonstrates, in the first “Q” (meaning Interviewer Question) a non-leading prompt, “So any thoughts as you’re moving around now?” that only elicits explanatory “here and now” thoughts (Figure 3). The interviewer then asks a leading question—“Right, so you felt annoyed back then?” (Figure 4)—and consequently the interviewee’s subsequent thought of feeling annoyance had to be excluded from the “affective” tally (see Table 1). In the following excerpt (Figure 5), the typed section is the verbatim transcript from an audio recorded stimulated recall interview. The left hand margin is used to record the thinking skills and strategies utilized by the student during the SL lesson. The underlined text delineates the thinking skill and/or the SL and real world triggers. The right hand margin identifies the trigger—those from the SL Chinese restaurant and those from the real world of the SL lesson in the computer lab.

Figure 5: Delineation of cognitive skills and SL and real world triggers Analysis The coding process resulted in the identification of 733 instances of thinking skills or strategies used by the 11 participants (see Table 1). A more detailed table of each participants’ thought types and their associated triggers (not presented here due to limited space) reveals considerable variability between the participants. For instance, Interviewee 7

Journal of Virtual Worlds Research – Stimulated Recall Interviews 17

reporting 178 thoughts and Interviewee 6, a mere 23, that had occurred while they were working through the SL lesson. Although each student engaged in the same instructional activity in SL they varied in the number and types of thinking skills employed. Such a variation across the 11 participants clearly begged to be investigated. One such line of inquiry for the researchers was to see if the instances of one type of thought process was related to a particular trigger and/or occurred in conjunction with another type of thought process. For instance, while a substantial number of complex thinking skills (e.g., metacognition and strategy planning) were indicated by the participants, the most common category of thinking was Affect, defined as: feelings aroused by SL and real-world stimuli (e.g., “I’m happy that I now have three dishes”). The second most common thinking skill was Strategy Planning, for example, devising tactics to accomplish the lesson objectives and for passing the exam. As a result the researchers were prompted to consider if the frequency of Affective thinking was a result of instructional stimuli (e.g., an effective learning task in which success elicits an affective state), or is related in some way to the more complex Strategy Planning, or is perhaps explained by some other theory. The data provided by stimulated recall allows for this kind of exploration. The value of stimulated recall becomes particularly evident when the types of thinking skills and strategies are linked to their triggers (the event that made them think, such as an automated waitress greeting them at the door of the restaurant) and then compared over time with a specific participant as well as across the participants. This enables the researchers to explore such things as the influence of types of in-world instructional stimuli, in-world collaborative activity, real life lecturer instructions, audio and text media, and pace. For instance, researchers interested in instructional design of SL lessons could match the trigger with the mediating process (cognitive skill) providing them with possible teaching and learning design strategies that cognitively engage students to utilize more complex thinking skills. For example, generative thinking (see Table 1) was triggered in all of the participants at the time of choosing and ordering dishes from the Chinese restaurant menu for people with particular dietary requirements, such as a vegetarian or a diabetic. Another example of the ways in which stimulated recall can inform pedagogy is how it highlighted the effect of the lecturer moving around the class to work with students. The researchers were able to analyse if the lecturer’s movement and interaction triggered thinking across the gamut of students. In this case it was revealing to note that some of the students reported feeling self-conscious when he stopped near them in the real world of the classroom (e.g., Figure 5 includes the example of a student who said “[the lecturer] was next to me then, and I was really careful about what I said because I didn’t want to say the wrong think and look silly”). However, no student indicated similar thinking being triggered when the lecturer’s avatar stood next to the students’ avatar. A more detailed analysis of this research data will be conducted in another forum. The intent of this paper has been to demonstrate some of the ways in which data provided by stimulated recall can support analysis. Although the number of participants in stimulated recall research is usually small due to the intense amount of work, the data is rich and it is possible to use statistical analysis if there are sufficiently detailed coded examples of both the type of mediating process and its triggers. Additionally, one transcript affords various analyses, such as identifying and explicating: the SL and real-world triggers of the participants’ cognitive skills and strategies; identity of self and avatar when learning in a SL world; the matches and mismatches between the lecturer’s thoughts and actions and those of the students during the

Journal of Virtual Worlds Research – Stimulated Recall Interviews 18

SL session; ascertaining if a deliberate lesson design elicits certain cognitive processes and collaboration; charting whether the SL lesson outcomes are affected by the novelty value of one SL lesson compared with two or more SL lessons. Conclusion To reiterate, stimulated recall is an empirically rigorous introspection data collection tool that allows the interviewer to elicit, identify, and explore participants’ thinking. In this study it was used to identify the types of thinking skills and strategies employed by first year university students while engaged in a lesson in Second Life. Stimulated recall enabled the researchers to make claims about the kind of thinking triggered by the virtual world environment in general and the instructional design in particular. In addition, a valuable affordance of this technique is the ability to account for stimuli from not just the virtual, but also from the physical environment, thus strengthening the researchers’ claims about the relationship between participants’ cognitive processes and instructional design. This article described how stimulated recall could be used by other researchers. The key methodological concerns, especially those relating to reliability and validity of data, have been outlined along with strategies to minimize those concerns. Of particular note is that stimulated recall is embedded within an information processing-mediating processesintrospection theoretical framework and requires adherence to strict methodological practices to maximize reliability and validity of the data and analysis. However, it less intrusive on thought processes than the concurrent verbal reporting of think-aloud methods and is more reliable than other introspection reporting tools.

Journal of Virtual Worlds Research – Stimulated Recall Interviews 19

References Afferbach, P., & Johnston, P. (1984). On the use of verbal reports in reading research. Journal of Reading Behavior, 16(4), 307-322. Allen, B.P., & Lindsay, D.S. (1998). Amalgamations of memories: Intrusions of information from one event into reports of another. Applied Cognitive Psychology, 12, 277-285. Anderson, J.R. (1990). Cognitive psychology and its implications. (3rd ed.). New York: Freeman. Anderson, L.P., & Mikkelsen, K.L. (2008). Recall of occupational injuries: A comparison of questionnaire and diary data. Safety Science, 46, 255-260. Atkinson, R., & Shiffron, R. (1968). Human memory: A proposed system and its control processes. In K. Spence & J. Spence (Eds.), The psychology of learning and motivation. Advances in research and theory (Vol. 2, pp. 89-195). New York: Academic Press. Barrows, H., Norman, G., Neufeld, V., & Feightner, J. (1982). The clinical reasoning of randomly selected physicians in general medical practice. Clinical Investigative Medicine, 5(1), 49-55. Bielaczyc, K., Piroll, P.L., & Brown, A.L. (1995). Training in self-explanation and selfregulation strategies: Investigating the effects of knowledge acquisition activities on problem solving. Cognition and Instruction, 13(2), 221-252 Bloom, B.S. (1954). The thought process of students in discussion. In S.J. French (Ed.), Accent on teaching: Experiments in general education (pp.23-46). New York: Harper & Row. Branch, J.L. (2000). Investigating the information-seeking processes of adolescents: The value of using think-alouds and think afters. Library & Information Science Research, 22(4), 371-392. Burton, C. (2006). M-Trilogy: An investigation of mediating processes, metacognition and mismatches in an ICT education subject. B.Ed.(hons) thesis. Townsville: James Cook University. Card, S., Pirolli, P., Van der Wege, M., Morrison, J., Reeder, R., Schraedley, P., & Boshart, J. (2001). Information scent as a driver of web behavior graphs: Results of a protocol. Analysis method for web usability. Proceedings of SIGCHI’01, March 31-April 4, Computer Human Interaction. ACM Press pp. 498–504. Clarke, P.M., Fiebig, D.G., & Gerdtham, U.G. (2008). Optimal recall length in survey design. Journal of Health Economics, 27(5), 1275-1284. Craik, F. & Lockhart, R. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Stimulated Recall Interviews 20

Ericsson, K.A. (2002). Towards a procedure for eliciting verbal expression of non-verbal experience without reactivity: Interpreting the verbal overshadowing effect within the theoretical framework for protocol analysis. Applied Cognitive Psychology, 16(8), 981â&#x20AC;&#x201C; 987 Ericsson, K., & Simon, H. (1984; 1993; 1999). Protocol analysis: Verbal reports as Data. (1st ed. 1984; 2nd ed. 1993; 3rd ed. 1999). Cambridge, MA: MIT Press. Ethell, R., & McMeniman, M. (2000). Unlocking the knowledge in action of an expert practitioner. Journal of Teacher Education, 51(2), 87-101. Freeman, A. (2003). Consciousness: A guide to the debates. Santa Barbara, CA.: ABC CLIO Inc. Garner, R. (1988). Verbal-report data on cognitive and metacognitive strategies. In C. Weinstein, E. Goetz, and P. Alexander (Eds.), Learning and study strategies: Issues in assessment, instruction and evaluation (pp. 63-76). New York: Academic Press. Gass, S.M., & Mackey, A. (2000). Stimulated Recall Methodology in Second Language Research. Mahwah, New Jersey: Lawrence Erlbaum. Gilbert, W., Trudel, P., & Haughian, L.P. (1999). Interactive decision making factors considered by coaches of youth ice hockey during games. Journal of Teaching in Physical Education, 18(3), 290-311. Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective think-aloud method as measured by eye tracking. Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 1253-1262). Hannigan, S.L., & Reinitz, M.T. (2003). Migration of objects and inferences across episodes. Memory & Cognition, 31(3), 434-444. Hannigan, S.L., & Reinitz, M.T. (2001). A demonstration and comparison of two types of interference-based memory errors. Journal of Experimental Psychology: Learning, Memory and Cognition, 27(4), 931-940. Harris, J.C., & Wear, S.B. (1993). Creating the reflective practitioner: Sources and causes of student teacher deviation from planned lessons. EDRS Document ED365 715. Henderson, L., & Tallman, J. (2006). Stimulated recall and mental models: Tools for teaching and learning computer information literacy. Lanham, MD: The Scarecrow Press. Henderson, L., Putt, I., Ainge, D., & Coombs, G. (1997). Comparison of students' thinking processes when studying with WWW, IMM and text-based materials. In F. Verdejo & G. Davies (Eds.), Virtual campus: Trends for higher education and training (pp. 124162). London: Chapman and Hall.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Stimulated Recall Interviews 21

Henderson, M. (1996). Multimedia interactivity: An investigation into learners' mediating processes during click-drag activities. B.Ed. Hons: James Cook University, Townsville. Hertzum, M., Hansen, K.D., Andersen, Hans H. (2009). Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology, 28(2), 165-181. Karasti, H. (2000). Using video to join analysis of work practice and system design: A study of an experimental teleradiology system and its redesign. IRIS 2000 Conference. Retrieved January 1, 2005, from http://iris.informatikgu.se/conference/ iris20/31.htm. Karsenty, L. (2001). Adapting verbal protocol methods to investigate speech systems use. Applied Ergonomics, 32(1), pp. 15-22. Langer, J. (1993). Approaches toward meaning in low- and high-rated readers. Report Series 2.2 Albany, NY: National Center on Literature Teaching and Learning. Retrieved December 20, 2009, from http://cela.albany.edu/meaning/index.html Lindsay, D.S., Allen, B.P., Chan, J.C.K., & Dahl, L.C. (2004). Eyewitness suggestibility and source similarity: Intrusions of details from one event into memory reports of another event. Journal of Memory and Language, 50, 96â&#x20AC;&#x201C;111. Lyons, W. (1986). The disappearance of introspection. Cambridge, MA: MIT Press. Marland, P. W., Patching, W.G., & Putt, I. (1992). Learning from text: Glimpses inside the minds of distance learners. Townsville: James Cook University. Meade, P., & McMeniman, M. (1992). Stimulated recall: An effective methodology for examining successful teaching in science. Australian Educational Researcher, 19(3), 118. Meade, P., & McMeniman, M. (1994). Teacher knowledge in action: The theoretical base of effective teaching. Paper presented at Australian Association for Research in Education Conference (AARE). Retrieved January 1, 2005, from https://www.aare.edu.au/94pap/meadp94404.txt Miller, G. (1956). The magical number sesven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. Nielsen, J. & Yssing, C. (2003). "Getting beyond the disruptive effect of Think Aloud". In M. Hertzum & S. Heilesen (eds.), Proceedings of the third Danish human-computer interaction research symposium (pp. 67-71) Datalogiske Skrifter, Roskilde University. Nielsen, J., & Yssing, C. (2004). What kind of information does an HCI expert want - on concurrent usability texting? Working Party, HCI Research Group, NORDICHI. Tampere: NORDICHI. Retrieved January 4, 2010 from http://openarchive.cbs.dk/bitstream/handle/10398/6465/15-2004.pdf?sequence=1

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Stimulated Recall Interviews 22

Nisbett, R., & Wilson, T. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259. Ottesen, G.G., Foss, L., Gronhaug, K. (2004). Exploring the accuracy of SME managers' network perceptions. European Journal of Marketing, 38(5), 593-607. Parsons, J. M., Graham. N. & Honess, T. (1983). A teacher's implicit model of how children learn. British Educational Research Journal, 9(1), 91-101. Pausawasdi, N. (2001). Studentsâ&#x20AC;&#x2122; engagement and disengagement when learning with IMM in mass lectures. PhD thesis, James Cook University. Shavelson, R.,Webb, N., & Burstein, L. (1986). Measurement in teaching. In M. Wittock (Ed.), Handbook of research on teaching (3rd ed.) (pp.50-91). New York: MacMillan. Shulman, L. (1986). Paradigms and research programs in the study of teaching: A contemporary perspective. In M. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 3-36). New York: MacMillan. Sim, C. (2004). The personal as pedagogical practice. Teachers and teaching: Theory and practice, 10(4), 351-364. Smagorinsky, P. (Ed.). (1994). Speaking about writing: Reflections on research methodology. Thousand Oaks, CA: Sage Publications. Vermersch, P. (1999). Introspection as practice. Journal of Consciousness Studies, 6(2-3), 17-42. Waldmann, M., Holyoak, K., & Fratianne, A. (1995). Causal models and the acquisition of category structure. Journal of Experimental Psychology: General, 124, 181-206.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 Using Design-Based Research for Virtual Worlds Research Projects Antonio Santos Universidad de las Americas Puebla, Mexico Abstract Due to the characteristics of educational virtual worlds, this manuscript underlines the need for a research model that considers the social context as part of its unit of analysis instead of just the individualâ&#x20AC;&#x2122;s cognitive process and learning. It is proposed that such a research approach could be design-based research (DBR), because the methodology employed by the DBR perspective thoroughly meets the challenges related to understanding how learning occurs inside a complex context of activities and interactions like those that usually take place inside an educational virtual environment. To accomplish this, the DBR employs an iterative methodology, which consists of repeating cycles of design, implementation, analysis, and redesign. This systematic procedure allows theory to emerge during the process; thus, using DBR, a researcher not only understands how to improve the quality of a certain virtual world, but also addresses issues regarding the theoretical background on which her design was based to revise and extend it. The main objectives of this article are to propose the use of designbased research as a viable methodology to do research in a virtual world like Second Life and to describe in detail how to do it. First, the design-based research approach is explained in terms of its origins, its methodological resources, and its theoretical underpinnings. Secondly, considering the characteristics and affordances of virtual worlds, an adaptation is proposed and explained to employ it for a virtual world research project. Finally, an example of a research project is built to show how the proposed design-based research methodology can be applied to plan it and revise its underlying theory. Keywords: Design-based research; Multi-user virtual environments; Constructivist pedagogy; Situated cognitive paradigm

Using Design-Based Research for Virtual Worlds Research Projects Introduction From a situated learning perspective, learning in a virtual world is conceptualized as a process distributed across avatars, the environment, and the learning activities in which they engage. If we accept this assumption, we must employ a research paradigm that considers the social context as part of its unit of analysis instead of just the individualâ&#x20AC;&#x2122;s cognitive process and learning to understand more profoundly how cognition and learning occur within a virtual world. One such research approach is design-based research that was developed with the assumption that the context affects learning; it also offers a methodological tool kit to systematically and iteratively test and improve a designed learning environment. Most importantly, as conceived by Ann Brown (1992), a researcher uses design-based research (DBR) not only to test a certain design to find out how it is working, but also to generate theory. This perspective maintains that theory development is linked to practice (Brown & Campione, 1996, cited by Wang & Hannafin, 2005). That is, through the DBR methodology a designer of an educational experience within a virtual world can address issues regarding the theoretical background on which her design was based to revise and extend it. The methods however, differ from those used in laboratory-based experiments to control variables and test hypotheses. Design-based research, in contrast, looks for multiple dependent variables to encompass most of the complexity of a natural context. Its basic aim is to explore how a design works in practice (Dede, Nelson, Ketelhut, Clarke, & Bowman, 2004); in this way, it â&#x20AC;&#x153;lay[s] open and problematize[s] the completed design and resultant implementation in a way that provides insight into the local dynamics,â&#x20AC;? (Barab & Squire, 2004, p. 8). In view of these arguments, the main objective of this manuscript is to propose the use of DBR as a viable methodology to do research in a virtual world such as Second Life and to describe in detail how to do it. First, the DBR approach will be explained in terms of its origins, its methodological resources, and its theoretical underpinnings. Secondly, considering the characteristics and affordances of virtual worlds, an adaptation will be proposed and explained as

used for a virtual world research project. Finally, an example of a research project will be constructed to show how the proposed DBR methodology can be applied to plan its evaluation and revise its underlying theory. Why Design-Based Research? During the past two decades, a new model for research has been emerging to study learning in natural contexts called design-based research. Two factors have promoted its emergence: 1) concerns about the low impact of educational research on educational practice 2) the extended use of constructivist situated theories. DBR basically emerged as an attempt to: first, address the general concern that educational research has not been dealing with the problems and issues related with educational practice and all its complexities, and secondly, to acknowledge that many of its findings have had a rather small impact on educational practice and theory (Dede, 2005a; Design-Based Research Collective, 2003; van den Akker, Gravemeijer, McKenney, & Nieveen, 2006). This situation is considered to be true in the field of educational technology as well because its research has also been characterized by some as being trivial, with poor ties to theory and practice, and use of inappropriate methods (Squire, 2005). Several authors consider using the DBR model in education to address these issues as a promising solution, because it addresses practical issues related to human learning in real-world contexts, and its aim is precisely to modify educational practice. DBR is indeed a “methodology for understanding how, when, and why educational innovations work in practice” (Design-Based Research Collective, 2003, p. 5). Answering these questions, it addresses both empirical and theoretical issues. In fact, as indicated by Dede (2005b), DBR has been addressing research weaknesses in the educational technology field, and has contributed to the growth and increased quality of its body of research. The other cause that has justified the use of the DBR methodology in education is the fact that during these past two decades—also as a response to the growing dissatisfaction with the education systems, as Tobias and Duffy (2009) affirm—an important approach to the design of instruction has been the use of theories grounded in the epistemological perspective of constructivism and its pedagogical variants like the situated cognition paradigm and the design

and use of effective learning environments. The situated perspective affirms that culture affects knowledge building, the context and the activity in which it occurs (Brown, Collins, & Duguid, 1989). Literature defines learning environments as contexts where students can socially build knowledge by collaboratively performing authentic learning practices to acquire high-level cognitive abilities (Jonassen, 1999; Perkins, 1992; Wilson, 1996). Both concepts stress the central role of context in the learning process; thus, to study instructional artifacts based in the constructivist view, like a new instructional methodology, a distance-learning course or an educational virtual simulation, we need a research model that includes context as one of its central variables. From this perspective, isolating variables in a laboratory to understand learning seems like a very limited lens through which to analyze and encompass the whole experience of cognition in context. For this reason Collins, Joseph, and Bielaczyc (2004) have concerns about the narrowness of measuring single variables such as learning content or skills with a multiple choice test—for example—to explain some rather complex aspects of human interaction during learning. Considering these constructivist issues, several authors propose that the methodology employed by the DBR perspective thoroughly meets the challenges related to understanding how learning occurs inside a complex context of activities and interactions like the one that usually takes place inside a constructivist learning environment. In this sense, Squire (2005) affirms that DBR “provides a useful framework for developing technology-enhanced learning environments and better pedagogical theory” (p. 9). In addition, Reeves (2005) states that design-based researchers “make a fundamental commitment to developing interactive learning environments in the contexts in which they will be implemented” (p. 49). That is, DBR deals with theoretical questions regarding the nature of learning in the real world (Collins et al. 2004) and sees context as one of its central variables (Barab & Squire, 2004). Of course, the DBR methodology can also be used to study designs based on other theories besides the constructivist view because education is generally accepted as being an applied science. That is, educational practitioners, no matter their theoretical background, are commonly interested in designing interventions to improve learning, applying their designs in real world settings to understand how they work in practice and in refining the theory on which the designs were based. However, as we stated before, several researchers believe that, since DBR methodology is focused on understanding learning within the messiness of real-world

teaching and learning practices which include context as an essential variable (Barab & Squire, 2004), it is well-suited for the constructivist theoretical underpinnings. It is worth mentioning here that some other authors (see Willis, 2009) believe that DBR, strictly speaking, cannot be considered constructivist in nature because one of its main goals is the generalization of its findings in the form of design principles; this is a fundamental objective of the positivist position and, thus, incompatible with the constructivist paradigm. However, Willis accepts that DBR is an appealing option because its basic characteristics (like addressing problems in real contexts, conducting rigourus and iterative studies, and defining new design principles) depart, in fact, from the traditional positivist approach; it also envisions a middle position between the constructivist research posture and the generalization interests of DBR. That is, a posture where, on the one hand, it is accepted that what is learned in a study can be found useful by other researchers and, on the other, that the final findings are not to become universal laws in the positivist sense of the term. What is Design-Based Research? A comprehensive list of characteristics of DBR is proposed by Wang and Hannafin (2005). They say that DBR is: 1) Pragmatic, because its aim is to address practical issues regarding the teaching and learning process. 2) Grounded, because it is conducted in real-world settings maintaining a multivariable perspective while using quantitative and qualitative methods, and supported in relevant educational research, theory, and practice. 3) Interactive, because during the research process researchers collaborate with the local practitioners addressing contextual issues. 4) Iterative, because it is done in iterative cycles of design, implementation, analysis, and redesign, the idea being that after each cycle findings are used as input for the following cycle so that, during this systematic process, data accumulates and theory emerges. 5) Flexible, because the instructional artifact, the object of research, is not fully developed initially so that, during the cycles, necessary changes emerge. 6) Integrative, because it combines methods according to how the research evolves, like observations, tests, surveys, document analysis, etc., always observing scientific rigor.

7) Contextual, because it stresses relating findings to the research process itself and local context where it is conducted, always documenting the whole process as much as possible. In an effort to summarize DBR characteristics van den Akker et al. (2006) propose the following list of attributes: •

Interventionist: the research aims at designing an intervention in the real world.

•

Iterative: the research incorporates a cyclic approach of design, evaluation, and revision.

•

Process oriented: a black box model of input-output measurement is avoided, the focus is on understanding and improving interventions.

•

Utility oriented: the merit of a design is measured, in part, by its practicality for users in real contexts.

•

Theory oriented: the design is (at least partly) based upon theoretical propositions and field-testing of the design contributes to theory building (p. 5). Cobb, Confrey, diSessa, Lehrer, and Schauble (2003) identify several DBR features that

complement the definition.These authors also agree that DBR results in theories related to the teaching and learning process, although they talk of humble theories because they are related to the particular research circumstances in the setting where it is conducted, like a school classroom. They remark that its methodology is very interventionist because it involves the investigation of how a new design improves education, and explain its iterative characteristic in terms of testing a set of initial conjectures, resulting in the creation of new conjectures, which are also subject to evaluation, and so on and so forth. Wang and Hannafin (2005) offer a rather complete and thorough definition of DBR that includes most of the characteristics and features stated before, when they define it as: A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextuallysensitive design principles and theories. (p. 6)

We see from the previous definitions of DBR, that when an educational researcher has a design perspective, his or her main interest is not to carry out studies where variables are isolated and controlled to prove a certain hypothesis and, in this way contribute to the generalization of results and establish universal principles and laws. Instead, his central interest is to identify practice-driven problems and test solutions (designs) while working collaboratively with the local participants and considering other solutions to similar problems published by researchers working in the field. The assumption is that in this way, systematically, a rich body of usable disciplinary knowledge develops. DBR Methodology The research methods advocated by DBR arise from the concepts explained in the previous paragraphs. They come from the writings of Ann Brown, who focused on the complexities of intervening in real-world settings and proposed that researchers systematically and iteratively implement and change their designs so that during the process, theory is created in naturalistic contexts (Barab & Squire, 2004). That is why the methodology proposed by most researchers using DBR follows these lines. For example, Collins et al. (2004) recommend a set of guidelines for doing DBR, which well ground Brownâ&#x20AC;&#x2122;s ideas. In this section, the six guidelines proposed by Collins et al. (2004) will be explained in detail because they will be used later as the base for this manuscriptâ&#x20AC;&#x2122;s proposal. The cited authors propose the following guidelines for DBR: 1) Implementing a design. Each setting where a design is applied is different, thus, it is important to match the designâ&#x20AC;&#x2122;s elements to the particularities of each case. 2) Modifying a design. Part of the procedure of DBR is to apply a design iteratively so that it can be systematically modified and improved; thus, after each cycle, design failures are identified and corrected. This is why documenting the whole process extensively is crucial. Nelson, Ketelhut, Clarke, Bowman, and Dede (2005) recommend documenting each cycle of the iteration in terms of describing implementation, findings and implications.

3) Multiple ways of analyzing the design. To better understand the design, Collins et al. (2004) suggest evaluation at the following levels: •

Cognitive level. The students’ previous knowledge and how it changes during the process. This can be assessed by analyzing students’ representations (visual) and explanations (verbal); this data would help researchers understand the students’ thought process.

•

Interpersonal level. Refers to student-student and student-teacher interactions. Qualitative research methods are used to observe people in action.

•

Group level. The dynamics (structure, power relationships, etc.) of the group as a whole. Qualitative research methods are also used to observe the group (e.g., in a classroom).

•

Resource level. The resources available to students and how well they integrate into their learning process. Assessed using structured interviews and surveys.

•

Institutional level. Refers to how the institution’s organization (e.g., administrators) and outside community (e.g., parents) affect the design’s implementation process. Assessed using structured interviews and surveys.

4) Measuring dependent variables. The authors propose assessing three types of dependent variables: •

Climate variables. For example, students’ engagement, cooperation and effort. Assessed through qualitative methods like participative observation or video analysis.

•

Learning variables. For example, students’ learning of content, reasoning and dispositions. To assess these variables qualitative and quantitative assessment designs are used, like observation, and pretest-posttest.

•

Systemic variables. For example, adoption and sustainability of the design, and its diffusion to others in the institution. Structured interviews and surveys are used.

5) Measuring independent variables. These are variables related to the context in which the research is being conducted. Some are: •

Setting. e.g., school, workplace, museum

•

Nature of learners. e.g., age, socioeconomic status.

•

Required resources and support for implementation. e.g., technical, administrative support.

•

Professional development. e.g., teachers’ training, practices, workshops.

•

Financial requirements. e.g., equipment and service costs.

•

Implementation path, variables related to the implementation process. e.g., initiation, duration, etc.

6) 6. Reporting on design research. The final research report should have the following sections: •

Detailed description of goals and elements of the design.

•

Description of the setting where the design was implemented.

•

Description of findings and changes done during each cycle in the iterative process.

•

A final integration of all the findings describing how the design evolved including limitations and failures.

Adapting the DBR methodology to a Virtual World research project The objective of this section is to propose a methodology to carry out a research project to study an educational virtual world applying the concepts and research methods of the DBR model discussed so far. In Figure 1, the basic structure of the methodology is represented. The methodology is organized as a procedure including three steps: 1) describing the virtual world; 2) iterating and documenting cycles of implementation-findings-implications; and 3) writing a final report integrating the findings of all cycles.

Figure 1: DBR methodology for a virtual world research project. Most of the methods described herein are subject to adaptations and modifications according to each researcherâ&#x20AC;&#x2122;s background and interests. Step 1) Describing the virtual world

Before starting the research process, it is assumed that the researcher is interested in testing a virtual world and in revising the theory that underpins it. It is expected that she would be comprehensively familiar with it; perhaps because she was part of the team that created the virtual world in question or because she uses it in her teaching practice. Therefore, as part of the first step, she is able to describe the educational virtual world in detail in terms of the: •

Theory that supports the virtual world.

•

Teaching and learning principles used to design the virtual world.

•

Settings where the virtual world would be used.

•

Relevant learners’ characteristics.

•

Stated learning objectives.

•

World’s contextual characteristics.

•

Learning activities that users are expected to perform inside the learning environment.

•

Goals that users are expected to achieve performing the learning activities.

•

Resources needed.

•

Financial requirements.

Step 2) Iterating cycles of implementation-findings-implications As part of the planning done to initiate the first implementation, it is helpful to create a table like the one shown in Table 1. As can be seen, the first column lists the research questions. Of course, these questions would vary according to the type of virtual world we are investigating; that is, according to its objectives, theoretical base, and technical characteristics. In Table 1, I included as an example, a set of research questions relevant to the study of a virtual world which was designed using the situated learning paradigm as its theoretical base. In the second column, each question is categorized, according to Collins et al. (2004), in terms of the type of dependent variable that it is addressing. The third column registers the level of analysis. Notice that a contextual type of variable is added which refers more to the physical ambience. In this case, it is considered relevant because we are investigating a virtual world based on constructivism theory. Finally, in the fourth and fifth columns, the research methods and instruments employed to gather data are stated. From the table, the researcher can further develop the specific procedures for the research and create the necessary materials to carry out the first implementation of the project, such as instruments to gather the data, chronograms, and logistics. The idea is to develop

the table before initiating the first cycle; in addition, after each cycle it is important to revise the table and make the necessary changes so that, before each cycle of investigation, a revised table is generated to serve as its methodological base. Research Questions

Variable

How much previous Learning knowledge do users have about the stated learning content? How is users’ level of Learning engagement during learning?

How do users solve the Learning problem-related activities presented to them? How is the users’ motivation Learning during learning?

Level of Methods analysis Cognitive Pretest – posttest comparison Cognitive Observation (video recording, or movie making if inside Second Life, could also be used) Cognitive Observation

Instruments Pretest and posttest Observation guide

Observation Guide

Cognitive

Structured interviews

Structured Interview guide Observation guide Unstructured interview guide Observation guide Unstructured interview guide Observation guide

How do the teacher-student Climate interactions affect learning?

Interpersonal

Observation Unstructured interviews

How do the student-student Climate interactions affect learning?

Interpersonal

Observation Unstructured interviews

How were the collaborative Climate learning activities performed?

Group

Observation

How is the social process Climate within the virtual world?

Group

Observation

Observation guide

What are the different social Climate roles that users take during activities? How do the virtual world Contextu resources (physical, al instructional, etc.) scaffold

Group

Observation

Observation guide

Resource

Observation

Observation guide

learning? How has the use of the virtual Systemic world diffused to other teachers?

Institutional

Structured survey

Survey

Table 1. DBR design After finishing the first implementation, which included carefully documenting the implementation process itself, the findings, and the relevant implications, the DBR methodology suggests repeating the process several times to revise the design and let the theory emerge. Thus, the set of implications that are reported as the product of the first cycle is carefully studied to plan the next cycle of implementation-findings-implications. Before doing another cycle, it is important to reflect upon the reasons for going into it. The number of cycles is a function of the nature of the research project itself, the researcher’s interests, and time and financial limitations. For example, Nelson et al. (2005) report four cycles when they made a study to investigate the multi-user virtual environment (MUVE) called River City, which is a very complex and wellfunded project. For the first cycle, researchers focused on investigating usability, motivation, learning, and the virtual world’s classroom application. They gathered data using quantitative and qualitative methods and instruments such as tests, questionnaires, narrative analysis, and observations. After analyzing the data, the researchers found that the MUVE was motivating and usable, but found weaknesses that led them to refine content and pedagogy through changing several of the MUVE’s attributes, including the ways students interact with the virtual world’s citizens, and how they move inside the virtual environment. Then, the revised MUVE was implemented a second time, focusing in this occasion on investigating how the changes affected the students’ responses. Again, they learned new and relevant aspects about how students were learning inside River City, such as how students became easily lost in the virtual world. They identified further modifications and repeated the process of implementation-findingsimplications for a third and fourth cycle. With each cycle, the researchers were better able to focus their research questions and to gain more comprehensive information about how the MUVE worked with the selected students and teachers, and about the theory behind it, such as students’ motivation and self-efficacy while learning inside virtual worlds.

Considering the iterative dynamic of this methodology, which evolves over time, it is crucial to document systematically the whole process because it is very important to identify the changes as they occur. Researchers use several methods to do this. For example, they use extensive writing of field notes and research diaries, or even video recording the process as much as possible. Step 3) Writing a final report integrating findings from all cycles. The main objective of the final research report should be to integrate and interpret the findings and implications that emerged during the research process into a cohesive whole. Hence, the report can have the following sections: •

Introduction The main objectives of the research are stated and the virtual world in question is described as was done in Step 1 of this methodology. That is, state: 1. the theory underlying the virtual world, underscoring the teaching and learning principles used; 2. the settings where the virtual world was used (a classroom, a training industrial environment, a museum exhibition, a teachers’ home, etc.); 3. relevant learners’ characteristics; 4. specified learning objectives; 5. world’s contextual characteristics; 6. learning activities; 7. goals that users are expected to achieve performing the learning activities; 8. resources needed; 9. financial requirements; 10. the design and procedures of the research.

•

Cycles of investigation Each process of implementation-findings-implications is described extensively corresponding with each cycle of investigation that was carried out. For each cycle, research questions, objectives, failures, and successes are described.

•

Conclusion

Present a final integration and interpretation of all findings and implications, describing how the design evolved—including limitations and failures—and state general conclusions that would clarify what was learned about how the virtual world works, about its underlying theory, and about the teaching and learning principles employed for its instructional design. Using DBR methodology in a Virtual World research project In this section, an example of the application of the methodology proposed in the previous section will be presented. The methodology will be used to build a plan to investigate the prototype of an educational virtual world still under construction in Second Life called the “SimEscuela” (Spanish for SimSchool). Due to the nature of the present manuscript, a briefer version of the research design will be described; however, enough details will be presented while following the three stated steps, so that the reader can comprehend how the methodology could be applied. Step 1) Describing the virtual world: The theory that supports the virtual world The SimEscuela virtual world was created to offer teachers a virtual learning environment where they could build a learning community to acquire and refine their teaching skills. Its instructional design was based on the situated cognitive paradigm, which proposes a different view of the nature of human learning. Briefly, this paradigm claims that the processes of cognition and learning are modified by the situation in which they happen. For this constructivist perspective, doing and learning are intricately related, or as Maturana and Varela (1987) put it: “All doing is knowing and all knowing is doing” (p.27). That is, our knowledge building is affected by the culture, the context and the type of activities we engage in when using it (Brown et al., 1989). Thus, when our students are at school, they are not only learning content, but are also going through a process of enculturation because they are, in addition, relating with those cultural aspects that define the communities of practitioners, mathematicians, historians, etc. that created the content. In this way, a student learns their values, the symbols they use to communicate, and the tools they employ when performing their professional practices. From this view, learning is reconceptualized in social, cultural, and historical terms (Lave & Wenger, 1991) and as participation in a community of practice (Wenger, 1998). From this definition of situated

learning, it is possible to understand how well this theoretical perspective lends itself to produce a virtual world. (For more details in using this paradigm to design virtual worlds see Author, in press). 2) Instructional Principles Some of the situated learning instructional principles that were adapted to form the theoretical base for the SimEscuela were: •

Users learn in the SimEscuela performing problem-related activities collaboratively. During these activities they freely negotiate meanings, and discuss, reflect, and share information.

•

SimEscuela simulates a traditional Mexican school so that participants can involve themselves as members of the community of practice formed by Mexican teachers.

•

The problems that users solve in the virtual world are authentic; that is, they are very similar to the type of problems that teachers solve in their everyday practices.

•

The objective is to solve a problem collaboratively and gain information during the process.

•

The environment constantly fosters reflection.

•

New users should be allowed to interact with experts as legitimate peripheral participants so that they can learn with and from them.

•

The teacher performs activities for the learners so that they can model expert performing.

•

The teacher closely supervises the participants’ activities and gives relevant feedback; however, he reduces his supervision as users become more experts.

•

The environment offers resources so that participants can easily store the information that they produce during their learning activities.

•

The users’ performance should be constantly evaluated so that participants know how they are learning in the virtual world.

3) Settings where the virtual world would be used Because the main objective of the SimEscuela is to serve as the learning platform for a community of Mexican teachers in Second Life, it is expected to have participants from all over

the country. The virtual environment was planned for K-12 teachers that belong to the public sector of education. Thus, the setting where it will be most often used is the computer lab of the different schools that form the public educational system in Mexico. Working individually in the computer lab, a teacher will access the SimEscuela and meet there with colleagues from other Mexican states to work collaboratively. The strategy will be to disseminate its use as much as possible among the teachers using the official channels. Teachers are usually involved in several types of training, therefore, it is expected that the use of the SimEscuela will be integrated into their training programs. 4) Relevant learners’ characteristics •

All the teachers have at least an undergraduate degree in education with majors in the different domains of content taught at the schools, e.g., mathematics, history, etc.

•

Many have completed graduate studies.

•

Technological literacy is low.

•

Most have access to a computer with wideband Internet in their school computer labs.

5) Stated learning objectives It is expected that teachers interacting inside the virtual world will: •

Develop the abilities to perform as an active member of a learning community with the purpose of building teaching knowledge collaboratively.

•

Develop the higher order cognitive abilities to solve problems collaboratively.

•

Apply their learning in their corresponding real life classrooms.

6) World’s contextual characteristics The virtual world SimEscuela was built to simulate a traditional Mexican school (See Figure 2). Its first version (the prototype) was created with only two classrooms because it is going to be tested with a small sample of teachers. One of the classrooms serves as the place where they can meet (in groups of five due to limitations regarding avatar movement in Second Life) to interact, share information, reflect with the facilitators, access relevant contents, and identify the learning problem-solving activities that they will need to perform to achieve their learning objectives. The other room simulates a traditional classroom where teachers can go to practice their newly

acquired teaching abilities doing role-playing games with the other members of their team. Moreover, in the school’s courtyard, there is an auditorium with a screen where teachers can display and explain their solutions to larger audiences.

Figure 2. Snapshots taken of the virtual world SimEscuela in Second Life 7) Learning activities that users are expected to perform inside the learning environment •

Meet together to identify a common problem; for example, a group of five teachers can decide that they need to better understand how to integrate the use of the computer in their teaching practices.

•

Build a learning plan including a chronogram.

•

Read relevant materials at the SimEscuela.

•

Investigate relevant information from other sources.

•

Perform tasks preplanned by the facilitators to learn specific content, such as the differences between learning “about”, “from”, and “with” technology.

•

Present their final products to other teams to obtain feedback.

•

Ask facilitators to evaluate their learning and do the corresponding evaluation activities.

•

Go to small workshops where they can learn specific abilities, like how to move inside Second Life.

8) Goals that users are expected to achieve performing the learning activities •

Identify a common problem related to their teaching practice.

•

Solve the selected problem working collaboratively.

•

Transfer the learned abilities in the SimEscuela to their everyday teaching practices.

9) Resources needed •

Technical support to aid researchers and participants.

•

Graduate student assistance.

•

Administrative support.

•

Materials.

10) Financial requirements •

The salary of at least one expert technician for computer equipment.

•

Funds will be needed to offer graduate students academic units.

•

The costs related to the production of all the materials.

Step 2) Iterating cycles of implementation-findings-implications To carry out the implementation of the first cycle of investigation according to the second step of the proposed methodology, the following table was created: Research Questions

Variable

Level

of Methods

Instruments

analysis How is users’ engagement Learning

Cognitive

during learning? How

users

problem-related

solve

the Learning

Cognitive

activities

Observatio

Observation

guide

Observatio

Observation

guide

Structured

interviews

Interview

presented to them? How is the users’ motivation Learning

Cognitive

during learning?

guide How do the teacher-student Climate interactions affect learning?

Interpersonal

Observatio

Observation

guide

Unstructure Unstructured d

interview

How do the student-student Climate

Interpersonal

interactions affect learning?

interviews

guide

Observatio

Observation

guide

Unstructure Unstructured

How were the collaborative Climate

Group

learning activities performed? How is the social process Climate

Group

within the virtual world? What are the different social Climate

Group

roles that users take during

interview

interviews

guide

Observatio

Observation

guide

Observatio

Observation

guide

Observatio

Observation

guide

Observatio

Observation

guide

Structured

Survey

activities? How do the virtual world Contextu resources instructional,

Resource

(physical, al etc.)

scaffold

learning? How has the use of the virtual Systemic

Institutional

world diffused to the teachersâ&#x20AC;&#x2122;

survey

real life practices? Table 2. Research design for the SimEscuela prototype As can be seen, Table 2 offers enough information to plan the procedure for this research project. In this case, at least three iterations are planned. The first implementation will be done with a small sample of five local teachers since, initially, we are interested in doing an overall test of everything, and we want to be able to closely watch how every user interacts with the SimEscuela and with other teachers. During this first cycle, all the designed instruments will be evaluated and teachers will be constantly interviewed during their learning process to identify any major failures. With the information gathered, the following two cycles will be planned. The second and third cycles will be done with larger samples from all over the country.

After the three cycles have been finished, the final research report will be written making an effort to conclude relevant issues related to the virtual worldsâ&#x20AC;&#x2122; general performance and about the stated teaching and learning principles and theories. The report should include suggestions of how to improve the virtual world SimEscuela for future versions. Conclusions Globally, the use of virtual technologies is already playing a relevant and promising part in our educational endeavors. However, we are still at an early stage of adaptation of this technology (Richardson and Molka-Danielsen, 2009) and have questions that have not been adequately addressed by current research regarding how we can take better advantage of its educational potential. I agree with the authors that propose that, at present, multi-user virtual environments are still mostly used to replicate real life experiences and are seldom developed to teach significant learning objectives (Berge, 2008; Dieterle and Clarke, 2005). To reverse this tendency, we need more potent theory that could, on the one hand, aid our practices as designers and creators of educational virtual environments and, on the other, explain how humans learn engaging in the types of social interactions that are possible between them (Gunawardena et al., 2009). An important part of the literature believes, as has been discussed in this manuscript, that a great aid to develop this type of theoretical body could be the design-based research model because it begins with the premise that theory and practice are closely related. With these ideas in mind, in this article, the theoretical underpinnings and methodological proposals of the Design-based research model were described to be able to use them as the base for research projects interested in investigating virtual worlds. Then, a methodology to use DBR in virtual worlds was proposed and a case was built to explain how to apply it. We still have a long way to go before we can satisfactorily conclude that the use of the DBR methodology in education in general, and in instructional technology in particular, produces the results that we are looking for to advance the field in the right direction. However, due to this research modelâ&#x20AC;&#x2122;s innovative characteristics, I believe that its use is indeed very promising and that it is worth the effort to carry out further research projects based on its methodology to study educational virtual worlds.

References Barab, S. and Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14. Berge, Z. L. (2008). Multi-user environments for education and training? A critical review of Second Life. Educational Technology, 48(3), 27 â&#x20AC;&#x201C; 31. Brown, A.L. (1992). Design Experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141178. Brown, J., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42. Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42. Dede, C., Nelson, B., Ketelhut, D., Clarke, J., & Bowman, C. (2004). Design-based research strategies for studying situated learning in a multi-user virtual environment. Paper presented at the 2004 International Conference on Learning Sciences, Mahweh, NJ. Dede, C. (2005a). Why design-based research is both important and difficult. Educational Technology, 45(1), 5-8. Dede, C. (2005b). Commentary: The growing utilization of design-based research. Contemporary Issues in Technology and Teacher Education, 5(3/4), 345-348. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Dieterle, E., & Clarke, J. (2005). Multi-user virtual environments for teaching and learning. In M. Pagani (Ed.), Encyclopedia of multimedia technology and networking (2nd ed). Hershey, PA: Idea Group, Inc. Gunawardena, C.N., Hermans, M.B., Sanchez, D., Richmond, C., Bohley, M., & Tuttle, R. (2009). A theretical framework for building online communities of practice with social networking tools. Educational Media International, 46(1), 3-16.

Jonassen, D. H. (1999). Designing constructivist learning environments. In C.M. Reigeluth (Ed.), Instructional-design theories and models, 2nd ed. (pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates. Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Maturana, H. & Varela, F. (1987). The tree of knowledge. Boston, MA: Shambhala Publications. Nelson, B., Ketelhut, D., Clarke, J., Bowman, C., & Dede, C. (2005). Design-Based research strategies for developing a scientific inquiry curriculum in a multi-user virtual environment. Educational Technology , 45(1), 21-34. Perkins, D. N. (1992). Technology meets constructivism: Do they make a marriage? In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation. Hillsdale, NJ: Lawrence Erlbaum Associates. Originally in Educational Technology, 1991, 31(5). Reeves, T. (2005). Design-based research in educational technology: Progress made, challenges remain. Educational Technology, 45(1), 48-52. Richardson, D. & Molka-Danielsen, J. (2009). Assesing student performance. In J. MolkaDanielsen & M. Deutschmann (Eds.), Learning and teaching in the virtual world of Second Life (pp. 45-60). Trondheim, Norway: Tapir Academic Press. Santos, A. (in press). Learning in Virtual Worlds: A Situated Perspective. In G. Vincenti y J. Braman (Eds.), Teaching through Multi-User Virtual Environments: Applying Dynamic Elements to the Modern Classroom. Hershey, PA: IGI Global. Squire, K. (2005). Resusciting research in educational technology: Using game-based learning research as a lens for looking at design-based research. Educational Technology, 45(1), 8-14. Tobias, S. & Duffy, T. (2009). The success and failure of constructivist instruction. In T. Duffy & S. Tobias (Eds.), Constructivis instruction: Success or failure. New York, NY: Routledge. van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Introducing educational design research. In J. van den Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 3-7). Abingdon, Great Britain: Routledge.

Wang, F., & Hannafin, M. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development , 53(4), 5-23. Wenger, E. (1998). Communities of practice: Learning, meaning and Identity. Cambridge, UK: Cambridge University Press. Willis, J. (2009). Constructivist instructional design (C-ID). Charlotte, NC: Information Age Publishing. Wilson, B. (Ed.). (1996). Constructivist learning environments. Englewood Cliffs, NJ: Educational Technology Publications.

Volume 3, Number 1 The Researcherâ&#x20AC;&#x2122;s Toolbox November 2010 eLab City: A Platform for Academic Research on Virtual Worlds Thomas P. Novak University of California, Riverside Abstract The eLab City project in Second Life is a laboratory environment for the study of user behavior in virtual worlds. This paper describes the origin and development of the eLab City project, which includes virtual infrastructure constructed in Second Life, a panel of Second Life users who have agreed to participate in research studies, tools for observational data collection, and procedures for fielding research projects. The eLab City panel is described in detail, with discussion of the panel signup process, recruitment, and panelist demographics. Cooperation rates for a series of research studies that used subjects from the elab City panel are presented, comparing studies fielded in Second Life to Web-based studies. We conclude with a discussion of lessons learned and next steps for this research project. Keywords: cooperation rates, online research, Second Life, virtual worlds

Acknowledgments. The eLab City project would not have been possible without the assistance of many individuals. The author thanks Mario Menti and Dietmar Walter for respectively providing panel programming support and access to the GMI Net-MR product, Nico Agosto for his LSL scripting and database support, Melah Carter, Khushbu Gokalgandhi, Erica Lopez, Tanya Manifold, and Mark Manalang for their research support, Andrea Godrey, Francesco Massara, Xing Pan, and Michael Trusov for their collaborative efforts on various research projects, Cezary Ostrowski for his design and development work in single handedly constructing eLab City, and Nadia Hennrich for her conceptual work and artistic direction.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 4

eLab City: A Platform for Academic Research on Virtual Worlds

Overview of the eLab City project

The eLab City project was launched in Second Life in 2008 as a multi-faceted laboratory environment for studying user behavior in virtual worlds (Novak 2007, 2008). The objective of this paper is to describe the motivation for creating eLab City and to explain the various research components of the project, paying particular attention to the eLab City Panel. Over the course of a year, we have assembled a survey panel of over 5,000 Second Life users who have visited the eLab City project, many of whom have subsequently participated in a series of Web-based studies as well as experiments that were programmed and fielded in Second Life. The conception and development of the eLab City project and the eLab City Panel raised a host of methodological and operational issues which we identify and address. We trust that our experiences over the past two years in developing and launching eLab City will be valuable to other researchers who are contemplating using virtual worlds as a platform for academic research. 1.1 Virtual worlds as research environments. Virtual worlds can arguably provide a sound basis for business study, given the ability of virtual worlds to simulate both the laws of physics as well as economic systems (Bloomfield 2007). These worlds are inhabited by virtual representations of real world consumers; considerable research has established that social, psycho-physical, and economic behaviors enacted in virtual worlds are consistent with real life behaviors (e.g. Miller 2007; Chesney et al. 2007; Gorini, Gaggioli and Riva 2007; Yee et al. 2007). Given behavioral consistency across real and virtual environments, virtual worlds can serve as laboratory environments for furthering consumer knowledge (Hemp 2006; Novak 2007; 2008) and more generally can serve as a platform for scientific research (Bainbridge 2007, Djorgovski et al 2009, Miller 2007). Some authors have argued that it is important for research about virtual worlds to be conducted from within the virtual world itself, rather than, for example, by inviting respondents from a virtual world to complete a Web-based survey, since â&#x20AC;&#x2022;moving out of the virtual world to answer a survey creates a break in immersion and thus potentially prevents accurate recall of the virtual environmentâ&#x20AC;&#x2013; (Bell, Castronova and Wagner 2009, p3). Of course, if the virtual environment is not itself the topic of the research study, requesting that participants exit a virtual world and complete a survey in a Web browser would not necessarily pose such a dilemma. Furthermore, fielding research studies in Web browsers may offer significant advantages considering that current technology for implementing surveys in virtual worlds (i.e.Second Life) is extremely

Journal of Virtual Worlds Research – eLab City 5

crude by comparison to the available tools for Web-based surveys such as the two survey research tools we used in the eLab City project—Qualtrics1 and GMI-Net-MR2. Still, there are many situations in which it is critical that research be conducted within a virtual world, rather than in the real world or in a Web browser. One such stream of research—which requires data collection occuring within a virtual world—focuses upon demonstrating that virtual world behavior mirrors real world behavior. For example, Slater et al (2006) replicated Milgram’s (2006) obedience study in a projection based virtual reality environment; Yee et al. (2007) replicated findings from the physical world regarding interpersonal distance in the virtual world Second Life; and Eastwick and Gardner (2008) replicated physical world findings regarding the ―foot-in-the-door‖ technique in the virtual world There.com. In such cases, since virtual world behavior itself is of interest, it is necessary for the research to be fielded and data collected within the context of a virtual world. In other situations, it is equally critical that research not be conducted within a virtual world, but be conducted in the real world (outside the virtual world). Such situations include research on the impact of virtual world behavior on subsequent real world behavior. For example, Yee and Bailenson (2007) found that participants who were given taller avatars in a virtual environment negotiated more aggressively in the virtual environment, but also negotiated more aggressively in subsequent real world interactions. Similarly, Fox and Bailenson (in press) reported that participants who watched their own avatar run on a virtual treadmill were more likely to voluntarily exercise the next day, in the real world. In such settings, at least some data must be collected outside the virtual world, in the real world, from subjects who had some prior interaction with a virtual world. Based upon the previous examples, we can reasonably conclude that academic research on user behavior in virtual worlds should allow for various data collection possibilities. Potential participants may need to be initially contacted either within a virtual world, or within the real world. The study itself may need to be fielded either within a virtual world, or within the real world (including a Web browser). Examples of virtual world participant contact include:   

1 2

Instant messages sent within a virtual world to a list of avatar names Intercept surveys in which an avatar—or an automated avatar ―survey bot‖ (Menti 2007)—approaches another avatar, or Survey devices which can be placed around a virtual world and encountered by virtual world visitors as they explore the virtual environment (Bell, Castronova and Wagner 2009).

http://qualtrics.com/ http://www.gmi-mr.com/

Journal of Virtual Worlds Research – eLab City 6

Real world participant contact includes:   

Emails sent to an existing sampling frame of email addresses, Posts on Web pages, or Social networks informing and directing potential participants about a research study.

In the next section, we introduce the eLab City project and describe various modes of data collection in the eLab City project. From our discussion so far, we note that eLab City is intended as a virtual world research platform that allows flexibility in whether research subjects are contacted—in the virtual world or the real world (i.e. via email)—for participation in research studies—fielded either within the virtual world or the real world (i.e. via a Web browser). 1.2 eLab City concept and development A concept plan for the eLab City project was released in November 2007, and bids were solicited from Second Life developers to construct a two-region academic build in Second Life.3 Construction began in late 2007, was completed in Spring 2008, and the first research activities began in Summer 2008. Originally envisioned as a ―live-work-play‖ community, eLab City was designed to provide a microcosm of many key social aspects of Second Life. However, eLab City’s primary functions were to be a working laboratory and a mechanism for building a subject pool of participants for academic research studies. Construction of eLab City was guided by three principles: form, function, and community. In hindsight, it became apparent that these three principles were ordered from least to most difficult to achieve. Form, in the sense of high quality design, was the first principle. Most academic builds in Second Life are not particularly inspired; the same can be said for much of the user generated content in Second Life itself. eLab City was designed to be a destination with strong design considerations and a compelling sense of place and we believe it is highly successful on the form dimension. The physical infrastructure of eLab City includes, as mentioned above, live-work-play aspects. Figure 1 shows an annotated screenshot of the eLab City project. The ―live‖ aspect includes a section of roughly one dozen pre-built unfurnished apartments intended for student workers and others involved with the eLab City project. As in other Second Life locations, residents quickly displayed a sense of ownership. For example, when the first set of student residents moved in, they changed the name from the non-descript ―Building A‖ and ―Building B‖ to ―Oceanside Suites North‖ and ―Oceanside Suites South,‖ and they remain so today. The ―work‖ section of

The original concept plan for eLab City is available at: http://elabresearch.ucr.edu/novak/elabcity/eLab City Concept Plan (Nov 5 2007).pdf

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 7

Cubicles Outdoor Stage

Research Labs Cinema

Dance Club Library

Park

Welcome Center

Shops Housing

Meeting Rooms

Housing

Shops Auditorium

Shops

Design Center

Housing

Dance Club Housing

Survey kiosk locations

Figure 1. The eLab City Project

Journal of Virtual Worlds Research – eLab City 8

eLab City is a complex of research-related buildings. These include 14 cubicles in a virtual behavioral lab that can be configured for experimental studies and mapped to parallel physical world research cubicles in our department’s real-world behavioral lab. There is also a large office building intended for use by faculty researchers, an auditorium, several conference rooms, library, visitor’s center, and survey sign-up kiosks. The ―play‖ section includes two dance halls, a multipurpose exhibition hall, resident-run shops, a cinema, outdoor stage (later converted to a drive in movie screen), and garden and park space. Function, the second principle, focuses on the role of eLab City as an environment for academic research. As an academic research facility and platform, eLab City combines a panel of Second Life users, survey and experimental research capabilities, and tools for unobtrusively tracking user visit behavior. This functional aspect of eLab City as a means to support academic research is described in detail in subsequent sections. Again, we have found eLab City to be highly successful on the function dimension. The third principle is community. At the most basic level, eLab City needs people to agree to participate in research studies for the project to succeed. As described below, we have been very successful in building a research panel of over 5000 Second Life users who have agreed to be contacted when research studies are available. However, beyond this rather narrow definition of community, we hoped to encourage a core group of Second Life users to make eLab City one of their primary destinations in Second Life, and to bring others to this destination. By providing modest financial support to Second Life residents who operate two dance clubs on eLab City, we have attracted a loyal group who visited the project for the purpose of socializing with their friends at these clubs. However, as discussed later, we are not as satisfied with the community building aspect of eLab City as we are with its form and function. Given limited resources, we found most of our time consumed with the research function, and found the task of building a real community to be especially challenging. 1.3 eLab City research process The eLab City research process consists of three main functions – the eLab City panel, a base for fielding research studies, and a means for collecting observational data. Figure 2 provides an overview of these three functions. Section 2 describes the eLab City panel signup process, recruitment, and demographics of panelists. Section 3 provides a brief discussion of the observational data that is collected at eLab City, and Section 4 presents information on cooperation rates for eLab City research studies that have been fielded with either email or Second Life IM invitations, and that have been fielded either with a traditional Web browser or within Second Life. Some comparisons of cooperation rates for the eLab City panel of Second Life users with a more general global panel of Web users, for the same research study, are also made.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 9

Research Studies Invite participant via Second Life IM

Conduct study in Second Life Study incentive payment

Invite participant via email

Conduct study in Web browser

Select sample for research study

Valid user check

eLab City Panel SL ads for panel

Panel signup kiosks

Monitor study completion rates

Challenge response email

Signup survey

Add panelist to GMI Net-MR panel database

Signup incentive payment

Abuse detection

Relate completion rates to demographics

Monitor panel demographics

Relate signups and demographics to visit dynamics

Observational Data eLab City Visitors

Figure 2. eLab City Research Process

Visit tracking sensors

Maya Realities service

Monitor XYZ coordinate timestamp data

Journal of Virtual Worlds Research – eLab City 10

The purpose of Figure 2 is to provide a high-level overview of the steps involved in running eLab City from a research perspective. The eLab City panel lies at the heart of the research process. We directed much of our initial attention toward obtaining and identifying high quality panellists. The data reported in the following section represent 5265 panel signups in a one year period from August 14, 2008 through August 30, 2009. 2. eLab City panel 2.1 Overview description The eLab City panel is a group of Second Life users who have agreed to participate in academic research studies fielded by the Sloan Center for Internet Retailing at the University of California, Riverside. Users who opt-in to the panel are contacted a few times a year and invited to participate in research studies related to virtual worlds. Incentives of L$250 (approximately $1 US) are provided to those who join the panel, and additional incentives of L$250-$L500 (approximately $1-2 US) are paid for each study completed after joining the panel. There are a number of similar panels in Second Life; for example the Second Life research panel maintained by Market Truths4, the First Opinions Panel maintained by the Social Research Foundation 5, and the Repères Second Life Panel6. While the eLab City panel is similar to these other panels, it is unique in that it is devoted to non-commercial, academic research, and that it is only used for research studies fielded by university researchers affiliated with the eLab City project. All research studies that use participants from the elab City panel have filed University human research review board (IRB) protocols. Setting up and maintaining a research panel is a substantial undertaking. Owning one’s own research panel provides researchers with much greater control over the signup and research process, but the trade-off is in the time and financial cost required to build and maintain the panel. Maintaining a research panel, however, does allow researchers to investigate problems involving the operation of the panel itself, such as methodology for devising optimal contact strategies that maximize survey cooperation rates of panel members (Neslin, Novak, Baker, and Hoffman 2009). The basic operation of the eLab City Panel is described in the next section. 2.2 Panel process Signup Kiosks. Five panel signup kiosks are located throughout eLab City, with locations shown in Figure 1. Each kiosk contains a main ―click to join‖ sign describing the basic terms of the panel. An FAQ sign, when clicked, delivers a Second Life notecard with detailed information about why and how to join the panel, what information is collected, how to remove yourself from the panel once you’ve joined, privacy considerations, and contact information for the human research review board. 4

http://sl.markettruths.com/panel/default.asp http://www.socialresearchfoundation.org/terms_conditions.html 6 http://www.reperes-secondlife.com/privacy.asp 5

Journal of Virtual Worlds Research – eLab City 11

Adding New Panelists. Clicking the main kiosk sign triggers a set of steps summarized in Figure 2. To minimize the possibility of Second Life users creating multiple avatars (alts) for the purpose of joining the panel and receiving a signup payment, a Linden Scripting Language (LSL) script checks to see if the Second Life avatar who is trying to join the panel has been in Second Life for at least 30 days. A check is also made to verify that the avatar has not previously joined the panel. All panelist data is stored in the GMI Net-MR 5.5 panel management system. Communication between LSL scripts and Net-MR through HTTP requests allows the Net-MR database to be accessed within Second Life, an example being a query of whether a potential panelist is already in the panel. If the avatar is at least 30 days old and not already enrolled in the panel, the avatar is considered a valid user and asked if they would like to receive an invitation to join the eLab City panel. If they answer yes, a form opens in a browser window with the avatar’s own name supplied and asking for an email address. Submission of this form triggers a challenge response email; an email is sent to the email address that was entered; the responder clicks a URL in this email to open the panel signup form in a Web browser. A check is made at this point to ensure that the email address is not already in use in the panel. The signup form asks for basic demographic information – respondent gender, avatar gender, and respondent birth year, education level, country, and native language. Although the signup form is Web-based, the signup process can only be initiated by an avatar within Second Life. The new panelist is requested to provide a password, which can be later used to sign into a panelist portal to change personal information or opt out of the panel. Abuse detection and payment. One of the problems with any online panel is the possibility of abuse. The likelihood of abuse increases when panelists are provided financial incentives for joining the panel. eLab City panel policies specify only one signup is allowed from a given real life person or IP address. Of course, Second Life users maintain multiple identities or ―alt‖ accounts, and when one is awarded Linden dollars for joining a survey panel, the temptation to sign up multiple alts is sometimes too much for some users to resist. In the initial days of the eLab City Panel, scripts automatically paid a signup incentive immediately after an avatar name was added to the panel. After a few days, we discovered that one individual had signed up—and had been paid—over 20 times, using 20 different accounts. A number of clearly suspicious patterns made this abuse obvious. First, in a short period of time there were over 20 emails from the same domain, with user names ―a,‖ ―b,‖ ―c‖ and so on. Second, most passwords for these suspicious signups were identical. Third, the signups all came from Second Life avatars with identical first names that ended in digits ―1,‖ ―2,‖ ―3,‖ etc. This abuse incident was reported to Linden Lab, but as with all abuse reports, the reporter of the incident is not notified as to how the situation was handled, as per the rather abrupt notice supplied by Linden Lab as a ―solution‖ to this case: ―For reasons of privacy, the outcome of reported incidents will not be public. Please do not expect any update on the situation. If you are

Journal of Virtual Worlds Research – eLab City 12

still experiencing abuse, please report it via in-world tools.‖7 In response to a question about whether this situation was considered a violation of Linden Lab’s Terms of Service (TOS), Linden Lab responded, ―if someone is attempting to fraudulently obtain linden – by whatever means, that is against TOS.‖8 While one would hope that such a violation of TOS would result in negative consequences for the perpetrator of the abuse, a year and a half after this incident, every one of the suspicious avatar names that fraudulently joined the panel is still listed as an active Second Life user when searched for in Second Life. It is clear that the researcher must bear responsibility for being alerted to such cases of abuse; it is equally clear that Linden Lab maintains a hand-off approach in such cases. Following this incident of abuse, we began collecting IP address, limited signups to one per IP address, and modified Net-MR programming to flag all signups which had an IP address identical to that of someone who was already in the panel. If there are multiple signups from the same IP address, then only the first of those signups is enabled as a valid panelist, and all subsequent signups from that IP address are disabled. As a result, multiple unique individuals from the same household who share a router are not allowed to join the eLab City panel. One advantage of limiting signups to one per household is preventing panelists who have been invited to the same study from comparing notes face to face. We also instituted a procedure of manually reviewing and approving new signups before payment, and disabling those signups suspected of abuse. Payments are now made about 3 times a week, and panelists are told to expect payment 24 to 48 hours after joining the panel. An avatar key is collected as part of the signup process, and used by a GMI Net-MR script to pay L$250 signup payments to approved new panelists. Of 5950 signups from August 2008 through August 2009, the percentage disabled and not paid because of suspected abuse was 11.5%. While is not insignificant, neither is it an alarmingly large concern. However, we should note that our current screening procedures do not eliminate all cases of multiple signups from a single individual. 2.3 Recruitment We used Second Life classified advertising as the primary method for building our panel. Second Life maintains a classified advertising system which can be accessed either through a Web browser, or by the Second Life client while the user is logged into Second Life. Page position in both Web and in-world classified ad listings is determined by the amount that a user pays per week to place an advertisement. We placed advertisements in two sections, Employment (L$3500/week) and Wanted (L$1300/week), paying placement fees that assured our ads remained in the top five ads for that section. The ads described the eLab City panel, noted that we paid a L$250 incentive for answering a brief signup questionnaire, and provided a teleport to a signup kiosk location in eLab City. 7 8

Solution provided to Linden Lab support ticket #4051-5096926, August 6, 2008. Belinda Linden, Second Life concierge support live chat log, August 1, 2008.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 13

The ads, as described above, ran from November 1, 2008 through August 30, 2009. The ad in the Employment section produced 4524 teleports, and the ad in the Wanted section produced 1076 teleports. Note that we had a total of 5265 panel signups from August 14, 2008 through August 30, 2009. Over the 42 weeks the ads ran, this translates to a cost per teleport of L$38 (approximately 15 cents, US). This is a very reasonable pay-for-performance rate for delivering potential panelists to the signup kiosks. The total number of panel signups tracks closely with the number of teleports from the classified ads, though we cannot determine whether a teleport directly resulted in a signup. Prior to November 1, we experimented with various advertisements. Figure 3 displays the number of new signups per week. Once the classified ads were put in place, the number of new signups was reasonably constant from week to week. 2.4 eLab City Panel demographics Basic demographic characteristics collected when Second Life visitors join the eLab City panel are reported in Table 1. Column a shows demographics for the 5265 valid enabled eLab City panelists. There is a skew toward more female panelists (55% female), and even more reporting using female avatars (57% female avatars). Instances of reported gender switching are relatively rare and are more likely for men than for women. Of those who reported their gender as male, 7.3% reported using a female avatar, while of those who reported their gender as female, only 2.2% reported using a male avatar. Education is roughly uniformly distributed, with fairly equal proportions of high school, some college, and college education, and slightly fewer reporting at least some post-graduate education. Avatar age is evenly divided between three groups: a third have been in Second Life for three months or less a third from 3-12 months, and a third a year or more. Age skews young, with a third of the panel between 18-24, 40% between 24-34, and only 10% age 45 or older. The eLab City Panel has a similar age distribution to previous surveys of Second Life users. A 2007 Global Market Insite survey of 10,000 Second Life users9 found that 51% were 29 or younger, as compared to 55.8% of eLab City panelists. Mean age of eLab City panelists is 30.7 years, compared to a mean age of 32 reported by Linden Lab in 2006.10 Column (b) of Table 1 reports demographic statistics for avatars whose accounts were disabled because of suspected abuse. While the differences between enabled and disable accounts are not too striking, Second Life visitors whose panel membership was disabled because of suspected abuse were slightly more female, slightly less educated, more likely to be from 25-34 (although mean age is the same: 30.8 years for disabled vs. 30.7 years for enabled), and more likely to have joined Second Life recently (217.5 days for disabled vs. 304.2 days for enabled).

http://nwn.blogs.com/nwn/2007/04/second_life_dem.html#more http://www.secretlair.com/index.php?/clickableculture/entry/second_life_stats_early_2006/

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 14

Figure 3. Distribution of Join Dates for eLab City Panel

Journal of Virtual Worlds Research – eLab City 15

Table 1 – eLab City Panel Demographics

GENDER Female Male Not stated AVATAR GENDER Female Male EDUCATION Less than High School High School Some College College Some Post-Graduate Post-Graduate Not stated

(a) eLab City Panel Enabled (n=5265)

(b) eLab City Panel Disabled (n=685)

55.5% 45.0% n/a

58.1% 41.9% n/a

58.9% 38.8% 2.3%

57.0% 43.0%

58.2% 41.8%

n/a n/a

4.3% 25.5% 24.3% 27.6% 7.3% 11.0% n/a

6.4% 27.6% 19.3% 27.6% 9.8% 9.3% n/a

1.0% 13.2% 23.8% 30.2% 10.8% 15.9% 5.2%

33.5% 40.2% 16.7% 6.7% 2.2% 1.2%

27.9% 48.3% 15.5% 5.0% 1.6% 1.8%

9.1% 26.9% 24.7% 22.7% 12.8% 3.7%

AGE 18-24 25-34 35-44 45-54 55-64 65+

Mean age 30.7 30.8 AVATAR AGE WHEN JOINED ELAB CITY PANEL < 3 months 33.6% 40.9% 3-6 months 15.1% 20.6% 6-12 months 17.8% 17.7% 1-2 years 22.6% 16.6% 2+ year 10.8% 4.2%

41.1

Mean avatar age

n/a

304.2

217.5

n/a n/a n/a n/a n/a

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 16

How does the eLab City panel compare demographically with a panel of general Web users? We also maintain a separate eLab global panel, which consists of Web users (likely not Second Life users) who opt in to receive email invitations to Web based research studies. Column c of Table 1 reports demographic statistics for the eLab global panel. Compared to the eLab City panel (column a), the global panel of Web users (column c) is more likely to be female, more likely to have higher education, and more likely to be older. Table 2 reports the top 10 native languages and the top 10 countries for enabled eLab City panelists (column a), disabled eLab City panelists (column b), and eLab global panelists (column c). For the eLab City panel, Table 2 also reports country as determined by geolocation of IP address using the WorldIP free geolocation database.11 Of enabled panelists, 62.7% report English as their native language, which is 10 percentage points higher than for disabled panelists (52.7%), and almost 10 percentage points lower than the eLab global panel (71.6%). Comparing disabled vs. enabled panelists, those whose accounts were disabled for suspected abuse were more likely to speak Portuguese, Spanish and Russian. Results for the top 10 countries largely parallel native language, with four English speaking countries (US, UK, Canada, Australia) listed in the top 10 among enabled eLab City panelists. Comparing the top 10 countries obtained by panelist self-report with the top 10 countries as determined by geolocation of IP address shows some discrepancies. For enabled panelists, 44.7% report their country as United States, but only 39.5% of their IP addresses return a US country code. The pattern is much more extreme for disabled panelists, of whom 39.4% report their country as United States, but only 25.7% of their IP addresses return a US country code. This discrepancy could be because some international individuals are using proxy servers hosted outside the United States, or because some expatriate US citizens living in other countries report their country as United States. However, a portion of the discrepancy can also simply be from people lying about their true country, an explanation which seems likely given the discrepancy is greater for those panelists whose accounts were disabled because of suspected abuse. Further analysis of enabled panelists identified countries which had the greatest mismatch between self-reported country and country as determined by IP address. Russia was the most problematic, in that only 76% IP addresses with a Russian country code were from panelists who said their country was Russia (18% of IP addresses with Russian country codes were from panelists who said their country was the United States). Again, while it is possible there is a large expatriate US Second Life community living in Russia, this discrepancy may point to suspect accounts. Other IP addresses with low percentages of geolocated country codes matched selfreported country are Germany (81.9%), Turkey (82.45), Portugal (82.5%), Argentina (85.2%), and Brazil (87.9%). In contrast, 99.5% of IP address with a United States geolocated country code were from panelists who self-reported their country as the United States. 11

http://www.wipmania.com

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 17

Table 2 â&#x20AC;&#x201C; eLab City Panel Native Language and Country (a) eLab City Panel Enabled (n=5265)

(b) eLab City Panel Disabled (n=685)

TOP 10 NATIVE LANGUAGES English Portuguese Spanish Russian French Romanian German Turkish Italian Dutch 35 other languages

62.7% 12.4% 3.6% 3.6% 2.9% 2.0% 1.9% 1.4% 1.4% 1.3% 6.8%

English Portuguese Spanish Russian French Turkish Czech Romanian Italian German 19 other languages

52.7% 21.5% 7.0% 6.1% 2.8% 2.2% 1.6% 1.2% .6% .4% 3.9%

English Spanish Russian Chinese Tagalog Indonesian Romanian Portuguese Arabic German 30 other languages

71.6% 3.1% 2.3% 1.6% 1.6% 1.1% .7% .7% .6% .6% 16.1%

TOP 10 COUNTRIES (AS REPORTED BY PANELIST) United States 44.7% United States Brazil 10.0% Brazil United Kingdom 6.7% United Kingdom Canada 4.4% Russia Russia 2.8% Canada France 2.5% Argentina Portugal 2.4% Turkey Romania 2.0% Portugal Germany 1.9% France Australia 1.7% Ukraine 108 other countries 20.8% 47 other countries TOP 10 COUNTRIES (IP LOOKUP) United States 39.6% United States Brazil 11.3% Brazil United Kingdom 6.3% United Kingdom Canada 4.9% Canada Russia 3.5% Russia Portugal 2.6% Argentina France 2.4% Turkey Romania 2.4% Czech Republic Germany 2.2% Portugal Romania 2.1% Ukraine 92 other countries 22.8% 32 other countries 12

Native Language was not asked of 3002 panelists

39.4% 18.5% 4.4% 4.1% 3.8% 3.6% 3.4% 2.3% 2.0% 1.9% 16.5% 25.7% 23.2% 5.0% 4.8% 4.8% 4.7% 4.1% 3.8% 3.8% 3.8% 16.4%

United States India Canada Australia United Kingdom South Africa Malaysia Philippines Singapore Ukraine 110 other countries

56.5% 9.0% 7.9% 3.0% 2.2% 1.8% 1.1% 1.0% .7% .7% 13.4% n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 18

Avatar location data

Description of location data. An additional source of input is avatar location data, consisting of minute-by-minute observations of XYZ location coordinates and timestamps, together with avatar names. Locations in each 256x256 meter region in Second Life correspond to X and Y coordinates between 0 and 256, with the Z coordinate indicating the distance in meters from a zero elevation. The Maya Realities service was used to collect these data for the two eLab City regions, using a set of sensors placed throughout the eLab City property in Second Life. Each minute, the sensors scan to detect the presence of avatars within a fixed radius of each sensor. Sensors are located in an overlapping array that permits coverage of the entire area of eLab City. This paper reports avatar location data for the time period from August 14, 2008 through December 31, 2008. A total of over 250,000 location observations were recorded during this time period from 7423 distinct avatars who visited eLab City during this time period. Of the 5265 enabled eLab City panelists who joined by August 31, 2009, avatar location data is available on a subset of 1275 panelists who joined through December 31, 2008. Figure 4 plots the XY coordinates for over 250,000 avatar location observations between August and December 2008. Labels from Figure 1, identifying key eLab City landmarks, are overlaid on the plot. The location observations concentrate in certain key areas, especially the two dance clubs. Shopping areas, housing, and survey kiosk locations also display visible concentrations of traffic. While avatars can fly in Second Life and are not constrained in terms of where they can go, the locations chosen by visiting avatars tend to conform to the physical structures or walkways in eLab City. Location statistics. Of the 7423 avatars who visited eLab City sometime between August and December 2008, a total of 1275, or 19% of all visitors, successfully joined the eLab City panel. An additional 134 avatars attempted to join the panel during this period, but had their accounts disabled because of suspected abuse. Using the Maya Realities avatar location data, we can determine the total number of minutes each visiting avatar spent in eLab city between August and December 2008, by summing up the number of per-minute scans in which an avatarâ&#x20AC;&#x2122;s presence was detected. Enabled panelists spent 49.0 minutes on average in eLab City (n=1275), disabled panelists spent 15.8 minutes (n=134), and those visitors who never joined the panel spent a mean of 23.0 minutes (n=6014) in eLab City. The differences among the means of these groups was statistically significant (p=.007). Mean differences are somewhat misleading, however, since the distribution of time spent in eLab City is highly skewed. While just over half of all visitors spent five minutes or less in eLab City, a relatively small number of individuals spent a very long time there. One avatar alone spent 190 hours in eLab City, while 12 other avatars each spent over 50 hours in eLab City, during this four

Journal of Virtual Worlds Research – eLab City 19

0 Dance Club

Cubicles

Park

Cinema

Outdoor Stage

Library

Housing Housing

120

Shops

Housing

Research Labs Housing

180

Welcome Center

Shops

Design Center

Housing Shops

Meeting Rooms

Dance Club

240

Auditorium

480 Survey kiosk locations

360

240 X

Figure 4. eLab City Visiting Avatar Locations from August 2008 – December 2008

120

Journal of Virtual Worlds Research – eLab City 20

and a half month period. Table 3 categorizes the time spent in eLab City and compares enabled panelists, disabled panelists, and those who never joined the eLab City panel. The mean time for those who never joined the panel is inflated from having a few avatars who spent a long time in eLab City, but who never joined the panel. The frequency distributions show a clear progression, where 26.6% of enabled panelists spent less than 5 minutes, 41% of disabled panelists spent less than 5 minutes, and 59.2% of those who never joined the eLab City panel spent less than 5 minutes. Table 3 – Time Spent in eLab City between August 2008 and December 2008 Time spent: 1-2 minutes 3-5 minutes 6-10 minutes 11-15 minutes 16-30 minutes 31-60 minutes 1-6 hours 6+ hours

Mean time spent

Enabled Panelists (n=1275) 6.4% 20.2% 24.0% 15.0% 19.7% 9.2% 4.3% 1.3% 100%

Disabled Panelists (n=134) 21.6% 19.4% 25.4% 13.4% 12.7% 3.7% 3.0% .7% 100%

Never Joined Panel (n=6014) 30.8% 28.4% 17.3% 7.7% 7.9% 4.5% 2.7% .6% 100%

49.0

15.8

23.0

Avatar location data can also be used to compare the exact locations in eLab City that these three groups visited. The XYZ coordinates were coded into various specific areas in eLab City. Five of these areas define panel signup kiosk locations and 15 areas correspond to major buildings or outdoor areas in eLab City. Table 4 reports the percentage of enabled panelists, disabled panelists, and non-joiners who visited each of these 20 specific areas at least once between August 2008 and December 2008. The majority of the signups to the panel seem to be coming from visits to the central and east kiosks, which corresponds with the fact that Second Life advertising directed visitors to these two kiosk locations. Since a greater proportion of the disabled panelists visited the east kiosk, we might hypothesize that the classified ads placed in the ―Wanted‖ classifieds which directed panelists to the east kiosk attracted avatars more prone to abuse than the classified ads places in the ―Employment‖ classifieds which directed panelists to the central kiosk. Very few panelists who joined at either of the two signup kiosks located at one of the eLab City clubs wound up being disabled due to suspected abuse. The percent of avatars visiting the 15 non-kiosk locations show a very interesting result. In most cases, enabled panelists were more likely to have visited eLab City locations than those visitors who never joined the panel. For example, 56.4% of all enabled panelists visited the main

Journal of Virtual Worlds Research – eLab City 21

walkway in eLab City at least once, compared to only 28.2% of those visitors who never joined the panel. There are, however, three key exceptions. Non-joiners were more likely than enabled panelists to have visited Club Sky (21.1% vs. 6.9%), Club Alcatraz (15.6% vs. 6.4%), and the main shopping mall (28.5% vs. 13.6%). Since the two dance clubs and the mall shops were intended, at least in part, to drive traffic to the eLab City panel, we can conclude that they were not very successful in doing so. Table 4 – Percent Visiting Selected eLab City Areas between August 2008 and December 2008 Enabled Disabled Never Joined peLab City Area Panelists Panelists Panel (n=6014) value (n=1275) (n=134) Kiosk - Central 54.2% 43.3% 24.1% .000 Kiosk - East 40.9% 47.8% 12.1% .000 Kiosk - West 14.8% 10.4% 3.1% .000 Kiosk - Club Sky 4.4% 0.7% 3.0% .009 Kiosk - Club Alcatraz 2.1% 0.0% 1.2% .013 Walkway - Mall Walkway - Residential District Shops - Old Town Shops - Mall Club Sky Club Alcatraz Walkway – Design Center Welcome Center Residential District Auditorium Research Labs Design Center Cinema/Library Great Lawn Capital Hill Park

56.4%

44.8%

28.2%

.000

43.8%

49.3%

17.3%

.000

20.7% 13.6% 6.9% 6.4% 6.2% 5.8% 5.4% 5.3% 4.5% 4.2% 3.5% 3.5% 3.3%

15.7% 10.6% 3.7% 0.7% 4.5% 3.0% 4.5% 3.7% 1.5% 1.5% 0.7% 3.0% 0.7%

11.7% 28.4% 21.1% 15.6% 4.0% 3.4% 5.3% 2.3% 2.0% 3.0% 1.8% 2.2% 2.8%

.000 .000 .000 .000 .002 .000 .899 .000 .000 .043 .000 .021 .219

Location data caveats. While avatar location data is extremely useful, there are some caveats regarding its collection and use. Multiple overlapping sensors may pick up an avatar within the same one minute scan, so the data needs to be cleaned to eliminate such duplicates. To minimize lag, scans are conducted at one minute intervals; however, during the period between consecutive one minute scans, any additional movement of an avatar cannot be detected. To capture real-time interaction, fixed interval location scans can be supplemented by object

Journal of Virtual Worlds Research – eLab City 22

interaction sensors, which record the exact time of interactions with an object (touches, collisions, sits, etc.) at the time the interaction occurs. Object interaction sensors were installed in panel signup and FAQ signs. They provide useful additional information – for example, for every enabled panelist who touched the ―click to join‖ sign in a signup kiosk, there is almost exactly one non-joiner who also touched the sign. While only half of those who touch the kiosk signup signs join the panel, many of those who touch and do not join are informed by the script in the signup kiosk that their avatar is less than 30 days old and thus not eligible to join. Last, sensors are only useful if they are in place and active. In Second Life, the Maya Realities sensor must be deeded to the group that owns the parcel of land on which the sensor is placed. In practice, however, sensors have unfortunately been mistakenly deleted by members of the group who did not know what they were. 4. Research studies Fielding studies within virtual worlds offers the potential to solicit panel members either via traditional email or via instant messaging within the virtual world itself. We consider the merits of both methods and provide empirical evidence of difference in cooperation rates. Based upon comparable studies fielded to both our Second Life and our global Web panels, we also compare cooperation rates and data quality between Web-based studies administered to virtual world and web-based users. We begin, however, with a more general discussion of the types of research studies in progress and planned for eLab City. 4.1 Types of research studies As noted earlier, Web based surveys offer advantages of sophisticated pre-existing systems for data collection, such as the Qualtrics product used to field eLab City research studies. Samples of email addresses drawn from the eLab City panel are uploaded into Qualtrics, which is used to email the sample. Avatar name can be uploaded as well and used to customize the solicitation email. Those who complete a Web-based survey are paid an incentive, typically L$250-L$500. This payment is made in Second Life using a script that pays a list of avatar names, using the avatar key collected during initial signup to the eLab City panel. A series of in-world studies have been fielded in eLab City. These are still in progress, and only general details are provided here. One series of studies involves ―choice tasks‖—described below—in which the participant inspects a series of stimuli and is asked to select which stimulus they like best. Versions of the study have included 2, 4, and 16 stimulus options. The stimuli are presented as images which can be clearly seen only when the participant stands on a mat in front of the image. Stepping onto the map triggers an LSL script which swaps in a high resolution version of the image that clearly shows the stimulus; stepping off of the map swaps in a blurry unclear version of the image. Various events, with timestamp and avatar name, are transmitted to a MySQL database: study start, step on to mat, step off of mat, choice, study completion. In

Journal of Virtual Worlds Research – eLab City 23

addition, following the choice, a short survey is administered in Second Life through a dialogue menu, and these responses are also sent to the MySQL database. Another series of studies present a slideshow of stimuli in Second Life, and then re-present the stimuli one at a time, together with false stimuli that have not been presented, and ask participants if they recognize a stimulus from the original slideshow. This is a classic signal detection task, and we have used this in a series of ongoing studies that look at recall accuracy (Massara and Novak 2010). An additional study in progress uses student research assistants as in-person interviewers in Second Life. These studies involve a precise way that questions are to be asked in different experimental conditions. To ensure these questions are asked in a uniform manner, we have developed ―interview HUDs‖ (heads-up displays) that allow the interviewee to click a series of buttons that triggers avatar movement and questions that are asked. The timing of individual questions, and responses provided by the interviewee, are passed into a MySQL database. The interviewer HUD also scans the environment and passes information about key events, such as the presence of other avatars, and the location of the interviewer and interviewee, to the MySQL database as well. In the next section, cooperation rates for a set of Web-based and in-world studies are reported. 4.2 Cooperation Rates One can build a panel of Second Life users, but to what extent will panelists actually participate in research studies? To date, we have fielded three substantial research studies using the eLab City Panel. Because these studies are still in progress, we refer to them here as ―Study A,‖ ―Study B,‖ and ―Study C.‖ Study A is a choice experiment, Study B is a survey of avatar personality, and Study C uses a signal detection task to test recall accuracy. In this section we report cooperation rates for these studies. In addition, as some of the studies were administered in different ways (i.e. with email invitations vs. Second Life instant message used as invitations), and with different panels (i.e. eLab City panels vs. eLab global panel), we can see if the mode of administration or nature of panel seem to influence cooperation rates. Study A. Study A was administered in four different formats: 1) email invitations to a Webbased study were sent to 1041 eLab City panelists, 2) email invitations to a Second Life (inworld) study were sent to 155 eLab City panelists, 3) Second Life IM invitations to a Second Life study were sent to 1281 eLab City panelists, and 4) email invitations to a Web-based study were sent to 2270 eLab global panelists. Study B was administered in one format, where email invitations to a Web based study were sent to 976 eLab City panelists. Study C was administered in two formats: 1) email invitations to a Second Life study were sent to 1548 eLab City panelists, and 2) email invitations to a Web-based study were sent to 1006 eLab Global panelists.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 24

Table 5a reports completion rates for all studies that used email invitations. Across the six variants of Studies A, B and C, completion rates (defined as number of completes divided by number of email invitations) ranged from 15% to 32%. These cooperation rates are in line with other recent studies using email invitation to Web based studies, using similar panels. For example, of the most recent 20 studies reported by Neslin, Novak, Baker and Hoffman (2009, p. 730), seven studies had completion rates between 10-20%, eight studies had completion rates between 20-30%, and five studies had completion rates between 30-40%. Considering Study A, we see very similar completion rates for email invitations to a Web based study sent to the eLab City panel (32%, Study A.1) and to the eLab global panel (30%, Study A.3). When email invitations to an SLURL for a Second Life study were sent to the eLab City panel (Study A.2), completion rates were unchanged at 30%. For this study, completion rates were not affected by the sample, or whether the study was administered in a Web browser or in an experimental setting in Second Life itself. Table 5a â&#x20AC;&#x201C; Cooperation Rates for Email Invitations

eLab City Panel Study A.1 Email Invite + Web Study eLab City Panel Study A.2 Email Invite + SL Study eLab City Panel Study B: Email Invite + Web Study eLab City Panel Study C.1 Email Invite + SL Study eLab Global Panel Study A.3 Email Invite + Web Study eLab Global Panel Study C.2 Email Invite + Web Study

# of email invites

# of completes

Complete rate # completes / # email invites

1041

335

32%

155

30%

976

189

19%

1548

234

15%

2279

685

30%

1006

202

20%

Table 5b reports one additional mode of administration for Study A (see Study A.4). Here, we sent instant messages in Second Life to invite member of the eLab City panel to complete a study in Second Life. There was one additional step in this mode of administration. A script went, in sequence, through each name on a contact list of 1281 panelists. The script first determined if the target panelist on the list was currently logged into Second Life. If the target panelist was logged into Second Life, an IM invitation was sent, and the script paused for 10 minutes and then continued with the next target name on the list. The 10 minute pause was designed to allow the recipient time to teleport to the experiment location and complete the study. In effect, we attempted to regulate the number of panelists receiving IM invitations so that

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 25

not many would be at the experiment site at any given time. If the target panelist was not logged into Second Life, the script proceeded to the next name on the list. Once the script reached the bottom of the list, it returned to the top and repeated the processes, ignoring anyone who had already been sent an IM invitation. The script ran for approximately two weeks, until it was consistently unable to find names remaining on the contact list who were logged into Second Life. Table 5b â&#x20AC;&#x201C; Cooperation Rates for IM Invitations

eLab City Panel Study A.4 IM Invite + SL Study

# on IM contact list

# of completes

Complete rate # completes / # on IM list

1281

245

19%

Reach rate # reached / # on contact list

Conditional Complete rate # completes / # reached

42%

45%

note: Study A (IP), Study B (GT), Study C (RA) The overall completion rate for Study A.4, defined as the number who completed the study divided by the number of names on the IM contact list, is 19%, which is appreciably lower than the 30% - 32% completion rates found for Studies A.1 through A.3. This suggests that email may be a more efficient way to reach eLab City panelists than instant messages in Second Life. However, while email invitations were used in Study A.2, one problem is that it is much more difficult to control when the email recipient will choose to respond to the invitation and proceed to the SLURL. Thus, the lower overall completion rate is offset to some degree by greater experimental control over the timing of when the participant arrives to begin the study in Second Life. For the IM invitation method, however, completion rate is not the only story. From Table 3b, we see that during the two weeks the script was running, only 42% of the names on the contact list were found to be logged into Second Life. This does not mean that the remaining names were not logged into Second Life at all during these two weeks, but that these names were not logged in at those points in time the script attempted to locate them. If we consider those names the script was able to reach and IM because they were logged in to Second Life, the conditional completion rate is 45% (defined as number of completes divided by number of IMs sent). This presents a more favourable picture. When we sent an IM to someone who was logged into Second Life, the completion rate was 45%, compared to 30-32% using email invitations. Studies B and C. A second study, Study B, was only fielded with email invitations sent to the eLab City panel for a Web based study. This was a more involved study than Study A and

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 26

completion rate for this study was only 19%. Study C was fielded in two different formats. Completion rate was 15% for email invitations to the eLab City panel sent to a Second Life study, and was 20% for email invitations sent to the eLab Global panel for a Web based study. The Second Life version of the study required more of the participant, in that they needed to navigate a series of lab rooms and operate a virtual slideshow, while the Web based version was a relatively straightforward survey. Thus, the difference in completion rates is not surprising. 4.3 Factors Affecting Cooperation Rates One question is whether completion rates can be predicted by demographic characteristics of panelists, and whether there may be differences in which demographic characteristics predict completion rates for different types of study administration, or different panels. Table 6 reports results of a series of multiple regression models, predicting a binary dependent variable of whether the study was completed or not completed from a set of demographic variables. Table 6 â&#x20AC;&#x201C; Factors Affecting Cooperation Rates note: dependent variable is complete/not complete study eLab City Panel (a) Study A.1 Email Invite Web Study (n=1041) Std Beta p

eLab Global Panel

(b) Study A.2 Email Invite SL Study (n=155) Std Beta p

(d) Study B Email Invite Web Study (n=976) Std Beta p

(e) Study C.1 Email Invite SL Study (n=1548) Std Beta p

(f) Study A.3 Email Invite Web Study (n=2279) Std Beta p

(g) Study C.2 Email Invite Web Study (n=1006) Std Beta p

Gender

-.055 .073 .101 .208 -.067 .017 -.088 .006 -.021 .406 -.003 .900 .023 .524

Education Age Avatar Age Country 1a Country 2b

.028 .393 -.075 .382 -.043 .151 .061 .061 .056 .035 .077 .000 .058 .108 .102 .002 .214 .012 .059 .046 .139 .000 .065 .015 .057 .010 .005 .881 .136 .000 .052 .527 -.032 .256 .051 .107 .075 .003 n/a

n/a

-.031 .319 -.101 .209 .040 .162 .095 .004 .022 .393 -.006 .790 -.074 .051 p

.040 .000 .095 .021 .011 .032 .051 .000 .017 .000 .009 .002 .008 .249

from Canada, UK, Australia, New Zealand from any country except USA, Canada, UK, Australia, New Zealand

n/a

.032 .297 -.146 .071 -.013 .653 .073 .023 -.021 .423 -.001 .974 -.002 .962

Full Model

n/a

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 27

Table 6 reports standardized regression coefficients (Beta) together with the p-value for each demographic term in the model. R2 statistics for most models are quote low, although R2 is .095 in column b (Study A.2). With the exception of column g, where R2 is not significant, panelist age is a significant predictor of study completion, with older panelists more likely to complete. In two cases, each of the following predictors is significant: women are more likely to complete (columns c and d), older avatars are more likely to complete (columns a and e), and those panelists with a higher education are more likely to complete (columns e and f). The weak prediction of study completion by demographic characteristics is not surprising, as completion of previous studies and time since last invitation have been found to be stronger predictors of completion rates than user demographics (Neslin, Novak, Hoffman and Baker, 2009). 5.

Lessons Learned and Next Steps

In this final section, we consider a range of broader issues related to the eLab City project ranging from lessons learned during the course of development, to thoughts about conducting research in virtual worlds and next steps Sustainability. We learned a number of lessons during the development and subsequent use of the eLab City project in Second Life. We began with a detailed set of specifications for eLab City, but construction in Second Life is an iterative process and it is very difficult, if not impossible, to completely describe in advance all objects and structures that are to be built and the scripting functionally of each item. Learning to build and script in Second Life is not terribly difficult if one is willing to spend the time learning and practicing; learning to build in an aesthetically pleasing manner, though, is much more difficult. While building skills can be learned, artistic and design talent is less readily acquired, and it is here that a suitably chosen developer can make a major difference. Once a build is constructed, however, it is time consuming and challenging to sustain human presence in Second Life. As has been noted by many commentators, Second Life often feels empty and deserted (Rose 2007). From Figure 4, it can be seen that while some sections of eLab City were well visited, such as the two dance clubs, many other areas were thinly visited and many areas were underutilized. The reason the dance clubs were heavily visited is that they were effectively managed in Second Life by an avatar hired as club manager who employed two assistant managers and a dozen hostesses/hosts. The manager, assistant managers, and hosts/hostesses were paid wages in Linden dollars. Total combined hours worked/week by club staff were about 100 hours, with a combined payroll of approximately L$15,000/week (just under $60 US). While $60 goes a long way in paying wages in Second Life, over the course of months these expenses add up. While the work-for-hire model could, in theory, have been used to also support other areas in eLab City such as the library, cinema, auditorium, outdoor events, and so on, the cost would have diverted funds away from resources for supporting academic

Journal of Virtual Worlds Research – eLab City 28

research. Because of cost considerations, financial support for the clubs was terminated in the summer of 2009, and visits to eLab City dropped accordingly. A natural alternative to hiring staff is to rely on user generated content and volunteer labor. While we attempted to do this, there is a significant time cost to identifying and maintaining oversight over volunteer workers to ensure that the project is meeting its academic objectives. Nothing runs itself in Second Life – human intervention is required to nurture social interaction. Over a year into the eLab City project, we are now rethinking the original purpose of much of the construction. One of the motivations for writing this paper, in fact, is the hope that some interested readers may contact the author with ideas for academic collaboration that utilize the eLab City infrastructure in a way that creates sustainable visitor traffic and real community. Object permissions. All objects in Second Life are built from basic building blocks called ―prims.‖ Each region supports a maximum of 15,000 prims. This may sound like a lot, but major structures in eLab City, such as the design center or the main research lab building consist of over 2,000 prims. This is a well-known limitation of Second Life, but a bigger problem is the permission structure. Each prim has user-specified permissions associated with it that specify whether the prim, as well as objects created from a set of prims, can be 1) modified, 2) copied, or 3) resold/given away. Each of these permissions is a binary switch that can be set to yes or no. For example, with all permissions set to ―yes‖ we have a ―full perm‖ object can be freely modified, copied, or resold or given away. Developers typically charge more for full perm construction because the creator relinquishes control and a full perm object represents profit potential for whoever owns it. Consider, for example a building in eLab City that avatar A constructs and sells to avatar B as full perm. Avatar B could then sell copies of the building, with permissions changed to no modify/no copy/no resell, to other avatars for profit, without any compensation to avatar A. Since the developer would have no way to profit from subsequent reselling of the building by avatar B, the developer would typically want a higher price for a full perm item than for an item that that could not be resold or given away. Because of such cost considerations, all objects created by the eLab City developer were received with permissions set to modify/copy/no resell. In retrospect, this was not ideal, and we would recommend that academic researchers only receive full permission objects from Second Life developers, and negotiate up front to ensure this. The problem with objects having less than full permission comes up with collaborative work, in that it is significantly more difficult for multiple people to work on a project if items can’t be copied and given from one person to another. In addition, full permission on objects is also required to transfer items from Second Life into OpenSim, which as we discuss later in this section, is a cost-effective alternative to Second Life.

Journal of Virtual Worlds Research – eLab City 29

Even with full permission objects, Second Life itself presents major limitations that get in the way of true collaborative work. Even though an object has full permissions, only the owner of the object can make changes to it. It is possible to deed objects to a group so that multiple people can work on it, but this approach has its own problems and what is really needed is a more flexible system that allows the owner of an object to specify the exact permissions for a list of avatar names that are authorized to work on the object. Value of research in virtual worlds. Things take longer in Second Life. Virtual construction and scripting take longer than expected. Communication via IM is less efficient than talking by phone or email. I recall my first experience renting real estate in Second Life. An avatar working as a real estate agent took me to view several properties she thought I might like. The process of finding an apartment took about three hours and I recall thinking that if I were driving around with a real estate agent in the real world, I would have been done by now. I’ve lectured in eLab City on a number of occasions to various student groups from other universities. Again, things took longer than expected, there were difficulties with audio connections, and the experience of trying to get a group of a two dozen avatars to move in tandem throughout eLab City on a tour was akin to herding cats. I’ve expressed this frustration to various people I’ve talked to, that things just seem to take longer in Second Life than in the real world. At a recent conference, I mentioned this again, and was asked, ―so why do you do it if it’s so hard?‖ Despite the difficulty of doing research in an environment like Second Life, the challenge alone is part of the appeal, as the challenges themselves raise research questions. In the real world, things may go more quickly, but what happens when the communication process is interrupted – for example when the user talks with an avatar real estate agent who is likely trying to simultaneously juggle multiple clients and keep them all satisfied? How do we communicate effectively with students in a classroom setting in a virtual world when we don’t have any real idea what the person behind the avatar is actually doing – or even if they are at their computer? How do sellers and buyers of virtual goods value objects which have permissions that vary on the dimensions of modify, copy and resell? All of the quirks of Second Life get us to think about research in a different way, and often suggest important research questions to be answered. Experience with eLab City also raises research questions. Beyond sample composition and cooperation rates for studies conducted within virtual worlds vs. traditional Web environments, how might the environment within which a study is administered affect data quality as well as the way participants respond to the study itself? What factors impact data quality and validity of results in virtual worlds? It was noted that avatar locations in Figure 4 largely correspond to physical structures and walkways in eLab City. eLab City is a highly realistic virtual environment. In a less realistic

Journal of Virtual Worlds Research – eLab City 30

environment, would avatar visit locations also correspond to structures and walkways? More generally, what impacts navigation and wayfinding in virtual worlds, and how can navigation be influenced? A practical application of influencing navigation would be direct eLab City club visitors who didn’t join the panel to signup kiosk locations. This was unsuccessfully attempted by placing ads in the clubs, but what appeal would be successful? Conducting academic research in virtual worlds such as Second Life involves significant programming effort, since the tools for fielding research studies in Second Life are not native to the Second Life platform and must be developed. The development of toolkits to facilitate academic research in Second Life and other virtual worlds would be very useful. As an example, Yee and Bailenson (2008) have provided LSL/PHP scripts and MySQL database schemas for a system for longitudinal tracking of a broad range of avatar-related variables, collected every 30 seconds, as an avatar wearing a device containing these scripts moves about and interacts with Second Life. In addition to such tools for real-time observation of avatar behavior, a reasonably complete toolkit for fielding academic research studies in Second Life would include:     

Scripts for inviting avatars to studies using IM invitations and controlling the number of avatars present at the site of the research study in-world, Systems for fielding surveys within Second Life (e.g. Bell, Castronova, and Wagner 2009), Payment scripts for awarding incentives in Linden dollars upon completion of a research study, Scripts for tracking object interaction, avatar location, and avatar behavior during inworld studies, and Scripts for random assignment to experimental conditions, and for rezzing different versions of a study corresponding to a different experimental conditions.

What’s next? What is the future of eLab City? Given that initial purchase and development costs during the first year are significantly higher than the cost to maintain the eLab City project subequent years, it is relatively cost-effective to maintain the project once it has been developed. We expect to maintain eLab City at least through 2011, and possibly beyond. The future of eLab City is largely tied with the future of Second Life. While usage of Second Life increased dramatically from 2006 to 2007, Table 7 shows that since 2007 the number of users actually logged into Second Life has been relatively flat, with the number of users logged in over the two month period from October to November decreasing from 2007 to 2009. At the same time, the number of users logged in during a one week period increased from 2007 to 2009, suggesting that those people who are using Second Life are using it a bit more frequently. While the overall picture suggests growth in usage is flat, there appears to be a core group of loyal users providing a more than sufficient critical mass for academic research purposes.

Journal of Virtual Worlds Research – eLab City 31

Table 7 – Number of Users Logged Into Second Life Residents Logged-In During Last 7 Days 571,652 538,373 475,383 169,272

Residents Logged-In During Last 14 Days 738,872 729,293 616,591 273,824

Residents Logged-In During Last 30 Days 1,000,468 1,039,501 981,494 452,222

Residents Logged-In During Last 60 Days 1,378,079 1,426,294 1,498,415 690,800

Date: December 1, 2009 December 3, 2008 December 1, 2007 December 4, 2006

Source: http://secondlife.com/statistics/economy-data.php

From an academic research perspective, the OpenSimulator Project (OpenSim) provides an alternative to Second Life and possible future migration path for eLab City, although the largest public grid currently has only 21,000 users.13 OpenSim supports LSL scripting, with some limitations, but currently lacks a large centralized user base and a built-in economy that enables building a research panel and making incentive payments as was done in Second Life with eLab City. However, OpenSim is considerably less expensive than Second Life. Educational pricing for a Second Life region is $147.50/month, plus a one time setup fee of $700. In contrast, educational pricing for an OpenSim region hosted by ReactionGrid is only $25/month, plus a one time setup fee of $50.14 The lower cost, plus the ability to upload content created on the OpenSim platform to Second Life, make OpenSim an attractive development platform for academic research on virtual worlds. Academic researchers interested in fielding research studies in virtual worlds may also want to explore new next generaton platforms, such as Blue Mars, that are much more visually compelling than Second Life and which support significantly larger numbers of avatars. While a Second Life region is 256 by 256 meters in size, and supports approximately 40 simulanous avatars, the largest unit of land that can be purchased in Blue Mars, a ―city,‖ can be up to 16 by 16 kilometers in size, and can support up to 1500 concurrent users. Despite the emerging alternatives, Second Life currently remains the most fully realized example of a general purpose virtual world, with a substantial loyal user base, a functioning economy, social and cultural nuances, and user-created infrastructure. While real-world businesses have experimented with and largely abandoned Second Life, from the perspective of a platform for academic research, Second Life remains a very attractive environment.

13 14

http://opensimulator.org/ http://www.reactiongrid.com/

Journal of Virtual Worlds Research – eLab City 32

References Bainbridge, W.S. (2007). ―The Scientific Research Potential of Virtual Worlds.,‖ Science 27 (July), 472-476. Bell, M. W., Castronova, E., & Wagner, G. G. (2009). Surveying the Virtual World: A Large Scale Survey in Second LIfe Using the Virtual Data Collection Interface (VCDI). Retrieved from http://ssrn.com/abstract=1418562. Bloomfield, Robert (2007), Worlds For Study: Invitation – Virtual Worlds for Studying RealWorld Business. Working paper, Johnson Graduate School of Management, Cornell University. Retrieved from (http://ssrn.com/abstract=988984). Chesney T., S. Chuah and R. Hoffmann (2007), ―Virtual World Experimentation: An Exploratory Study.,‖ Retrieved from http://ssrn.com/abstract=1068225. Djorgovski, S.G., P. Hut, S. McMillan, E. Vesperini, R. Knop, W. Farr and M.J. Graham (2009). ―Exploring the Use of Virtual Worlds as a Scientific Research Platform: The MetaInstitute for Computational Astrophysics (MICA). F. Lehman-Grube, J. Sablating, et. al (Eds), Proceedings of Facets of Virtual Environments. Berlin: Springer Verlag. Eastwick, P. W., & Gardner, W. L. (2008). Is it a Game? Evidence for Social Influence in the Virtual World. Social Influence , 4 (1), 18-32. Fox, J., & Bailenson, J. (in press). Virtual Self-Modeling: The Effects of Vicarious Reinforcement and Identification on Exercise Behaviors. Media Psychology 12(1), 1-25. Gorini A., Gaggioli A. and G. Riva (2007), Virtual Worlds, Real Healing. Science, 318 (5856), 1549. Hemp, Paul (2006), Avatar-Based Marketing. Harvard Business Review, June, 48-57. Massara, F. and T.P. Novak (2010). Context-specific Information Processing: Investigating Circumstances that Improve the Retention of Message Content. Paper presented at the INFORMS Marketing Science Conference 2010 , Cologne, Germany, June 16-19. Menti, M. (2007), Conducting Research in Second Life. Quirks Marketing Research Review. May, 44-48. Milgram, S. (1963). Behavioral Study of Obedience. Journal of Abnormal and Social Psychology. 67, 371-378. Miller G. (2007), The Promise of Parallel Universes. Science, 317 (5842), 1341-1343.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; eLab City 33

Neslin, S.A., T.P. Novak, K.R. Baker, and D.L. Hoffman (2009), An Optimal Contact Model for Maximizing Online Panel Response Rates. Management Science, 55 (May), 727-737. Novak T.P., (2007), Consumer Behavior Research in Second Life: Issues and Approaches. Association for Consumer Research Pre-Conference. Memphis, TN, October 25 2007. Novak, T.P. (2008), eLab City: A Platform for Consumer Behavior Research in Virtual Worlds. Marketing Science Institute Conference, Leveraging Online Media and Online Marketing. Palm Springs, CA, February 6-8. Rose, F. (2007). How Madison Avenue is Wasting Millions on a Deserted Second Life. Wired, July 24. Slater, M., Antley, A., Davison, A., Swapp, D., Guger, C., Barker, C., et al. (2006). A Virtual Reprise of the Stanley Milgram Obedience Experiments. PLoS One , 1, e39. Yee, N., & Bailenson, J. N. (2007). The Proteus Effect: The Effect of Transformed SelfRepresentation on Behavior. Human Communication Research , 33 (3), 271-290. Yee, N. & Bailenson, J.N. (2008). A Method for Longitudinal Behavioral Data Collection in Second Life. Presence, 17. Retrieved from http://www.nickyee.com/pubs/secondlife.html Yee N., J.N. Bailenson, M. Urbanek, F. Chang, and D. Merget (2007), The Unbearable Likeness of Being Digital: The Persistence of Nonverbal Social Norms in Online Virtual Environments. Journal of CyberPsychology and Behavior, 10 (1), 115-121.

! "#$%&'!()!*%&+',!-! ./'!0'1'2,3/',41!.##$+#5!! *#6'&+',!78-8!

Learning spaces, tasks and metrics for effective communication in Second Life within the context of programming LEGO NXT Mindstormsâ&#x201E;˘ robots: towards a framework for design and implementation Stewart Martin Teesside University, UK. Michael Vallance Future University Hakodate, Japan. Paul van Schaik Teesside University, UK. Charles Wiz Yokohama National University, Japan. Abstract Science education is concerned with the meaningful pursuit of comprehension, knowledge and understanding of scientific concepts and processes. In Vygotskian social! constructivist learning, personal interpretation, decision-making and community cooperation foster long"term understanding and transference of learned concepts. In short, the construction of knowledge requires learners to be actively involved in the process of learning. This paper presents a discussion of the method used to assess and define effective measurements. We use these measurements to evaluate strategies for communicating science by using LEGO robots and a Mindstormsâ&#x201E;˘ RCX controller that are collaboratively constructed and programmed by students in a virtual technology (Second Life). This occurs while the students are physically situated in different locations. Keywords: learning spaces; social constructivism; pedagogy

Journal of Virtual Worlds Research – Learning Spaces in Second Life

Learning spaces, tasks and metrics for effective communication in Second Life within the context of programming LEGO NXT Mindstorms™ robots: towards a framework for design and implementation. Context Our findings originate from an international collaborative project tasked to develop metrics for establishing the success of learning within immersive environments. Within this project we propose approaches to designing effective educational tasks composed of different learning objectives and include a framework for them; we also present an effective learning and teaching pedagogy. We describe the development of these metrics and our method and also discuss how they were applied to a specific project, including an evaluation of strengths and weaknesses. Science education is concerned with the meaningful pursuit of comprehension, knowledge and understanding of scientific concepts and processes. This paper presents a discussion of the method that has been applied to assessing and defining effective measurements that evaluate strategies for communicating science. We use LEGO robots and a Mindstorms™ RCX controller collaboratively constructed and programmed by students using a virtual technology (Second Life) while physically situated in different locations. This paper reports a set of experimental protocols that were designed to elicit the metrics which may be applied to bring about successful learning within such an environment. Ours is an international collaborative project between three universities: one in the UK and two in Japan. We were to develop metrics for establishing the success of learning tasks within immersive environments as well as to propose approaches for designing effective educational tasks for different learning objectives. Then the team was to design a framework for these objectives and tasks and for effective learning and teaching pedagogy. The research team adopted a mixed#methods approach to combine data from in#world transcripts, reflective diaries, individual interviews with subjects, psychometrics, robotic programming data, and transcriptions and coding from video capture of interaction sessions between participants. Participants were required to discuss, design and solve real#world and in#world programming challenges. Beyond traditional practice Research in the informed use of technology for educational purposes highlights the need to go beyond the replication of traditional, didactic practices to an appropriation of digital communication (Warschauer, 1999) facilitated by a constructivist pedagogy (Jonassen & Land, 2000) to support purposeful tasks (Martin & Vallance, 2008). The convergence of instructivism, constructionism, and social and collaborative learning towards a cohesive ‘Conversational Framework’ (Laurillard, 2002) provides opportunities for learners to take, “a more active role in learning and for tutors to support learning activities in multimodal ways” (deFreitas & Griffiths, 2008, p.17). Technologically enhanced learning makes possible a “shift from merely exposing the learners to the material, to transforming the learning environment” (Dror, 2007, p.5). This paper reports a set of experimental protocols which were designed to elicit the metrics that may be applied to bring about successful learning within such an environment as part of an ongoing project1. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1

See http://web.me.com/mvallance/PM12/PM12/PM12.html for project background

Journal of Virtual Worlds Research – Learning Spaces in Second Life

Virtual worlds are ideally positioned to enable educational transformation—they permit the creation of new learning environments with unique tools and communication opportunities, all of which can support inter#disciplinary and intra#disciplinary collaboration and build bridges between education and experience (Jarmon et al, 2009; Field, Vallance & Yamamoto, 2008). The first phase of adoption often attempts to create engaging experiences in these new learning environments by replicating familiar real world buildings and institutions (Jennings & Collins, 2008) together with their associated approaches to the learning experience. This transposition of existing practice to these virtual world environments may not be appropriate and does not effectively utilize their unique design and communication opportunities in meaningful ways. One informed educator in the Second Life (SL) virtual world summarizes her experiences of this approach: After attending numerous educational meetings in SL, my tolerance for mapping real world activities directly into the virtual world fell into sad decline. Sitting in a closed classroom space while being silent, waiting to raise my hand to speak is in direct contrast to how I teach and interact in a virtual world. Many favor these practices, and perhaps there are more examples of traditional methods than is apparent at first glance. My penchant for employing less traditional learning spaces does not mean that I advocate classroom chaos, but that I like to leverage virtual world capabilities beyond what we can do at our brick#and#mortar campus and in our online course management systems (Calongne 2007, p.111). Innovative articulation of virtual world collaboration requires a uniqueness of contribution by participants (personally or anonymously), synchronously and/or asynchronously (whichever is most comfortable for the user) with a democratization of the process that can lead to a sum product greater than individual contributions (Vallance & Wiz, 2008). This represents a move from the commonly seen replication of existing practice towards the exploitation of the unique pedagogical affordances offered by emerging technologies; a move from first to second order change (Cuban, 1992). To implement this change, tasks are required that promote learner activities which would be difficult, if not impossible, in the real world. As deFreitas (2008) asserts, “In order to achieve this next step two related aspects are required: the first is developing better metrics for evaluating virtual world learning experiences, and the second is developing better techniques for creating virtual learning experiences (e.g. frameworks, approaches and models)” (p.11). The challenge posed by deFreitas is for educational researchers to develop valid, reliable and transferable metrics for assessing the teaching and learning effectiveness of virtual worlds. Transformation of learning and of pedagogical practices can only be accomplished from evidence that supports and highlights the resulting benefits of such change. Theoretical context Teaching takes place within the context of educational philosophies based on theories of how people learn. Furthermore, epistemological considerations influence course design and pedagogy regarding the strategy teachers use to facilitate learning. While these theories and resulting teaching techniques vary, the idea of learning as acquisition and as participation has underpinned much educational thought (Rovai et al, 2009, p.2).

Journal of Virtual Worlds Research – Learning Spaces in Second Life

Bloom provides a taxonomy that has allowed educators to visualize teaching objectives and perceived learning and the associated notation, categorization and assessment of aims (Bloom, 1956; Anderson et al, 2001). Anderson et al (2001) posit that “(a)ll frameworks such as the taxonomy are abstractions of reality that simplify in order to facilitate perceptions of underlying orderliness” (p.259). From the development of its original format in 1956 to its revision in 2001, Bloom’s taxonomy (Figure 1) has been widely adopted and extensively cited as a useful way of framing what happens in a potential learning situation. It was adopted as the theoretical context for our study in line with a widespread academic consensus that “(i)n a field marked by wide pendulum swings, the likelihood of finding an idea, concept, or point of view that has remained constant in its acceptance and application is small indeed. Without doubt, the taxonomy is one of these rarities” (Anderson & Sosniak, 1994, p.viii).

Figure 1. Bloom’s Rose. Aainsqatsi, 2008; after Manuel, 2000.

In Bloom’s taxonomy, a range of learning objectives are presented as cognitive functions (Anderson et al, 2001) that enable cognitive learning, that is, “… recall or recognition of knowledge and the development of intellectual abilities and skills” (Bloom, 1956, p.7). The six

Journal of Virtual Worlds Research – Learning Spaces in Second Life

categories associated with the cognitive domains identified in the revised taxonomy are: (1) remember—retrieve relevant information from long#term memory; (2) understand—construct meaning from instructional messages, including oral, written, and graphic communication; (3) apply—carry out or use a procedure in a given situation; (4) analyze—break material into constituent parts and determine how parts relate to one another and to an overall structure or purpose; (5) evaluate—make judgments based on criteria and standards; and (6) create—put elements together to form a coherent or functional whole, reorganize elements into a new pattern or structure (Anderson et al, 2001). The verb sub#sets associated with these cognitive processes are: (1) remember—recognize, recall; (2) understand—interpret, exemplify, classify, summarize, infer, compare, explain; (3) apply—execute, implement; (4) analyze—differentiate, organize, attribute; (5) evaluate—check, critique; and (6) create—generate, plan, produce. Supporting the cognitive processes are four general types of knowledge that include: (1) factual knowledge— knowledge of discrete, isolated, content elements; (2) conceptual knowledge—knowledge of more complex, organized forms such as classifications, categories, principles, generalizations, theories, models and structures; (3) procedural knowledge—knowledge of how to do something; and (4) metacognitive knowledge—knowledge about cognition in general, as well as awareness of and knowledge about one’s own cognition (ibid). Cognitive Process Knowledge Remember Understand Apply Factual Conceptual Procedural Metacognitive

Analyze

Evaluate

Create

Table 1. The taxonomy table; after Anderson et al, 2001.

A strength of Bloom’s taxonomy is that it provides a visualization of a relationship between both cognitive processes and knowledge (Table 1). It allows researchers and practitioners to sort out complexities and identify gaps where none may have been previously acknowledged. By adopting the revised Bloom’s taxonomy and the associated descriptive verbs, educational practitioners can use research outcomes to develop advice for creating assignments intended to have certain desired objectives. The clarity offered by the taxonomy has potential to make virtual worlds more accessible to educators and to allow exploration of the ways in which they differ from more traditional learning contexts. The activities of students during the process of virtual task implementation can be assessed in terms of their success in meeting the specified objectives and the alignment between ‘objective – activity – assessment’ should make virtual world tasks more integrated, transparent and coherent to both learner and educator. Using constructivist pedagogy to support purposeful tasks In Vygotskian social#constructivist learning, the learner’s interpretation, decision#making, and community cooperation fosters long#term understanding and transference of learned concepts. Learning is considered to be a ‘‘process whereby knowledge is created through the transformation of experience” (Kolb, 1984, p.41) and the construction of knowledge thus requires learners to be actively involved as participants in the process of learning. This requires an instructor’s pedagogical approach to be anchored in such a way that students meaningfully

Journal of Virtual Worlds Research – Learning Spaces in Second Life

‘experience’ their learning. Learning, then, is deemed to rely upon active learner engagement; virtual worlds provide an opportunity to explore a new space for the cognitive process. The student can do this by facilitating active learning through the transfer of control – as Dror (2007, p. 5) says: “providing control to the learners helps to achieve active and motivated learners, and when they are involved, participating, engaged, and interacting with the material, then learning is maximized.” deFreitas and Neumann (2009) suggest that the appeal, immersivity, and immediacy of virtual worlds in education requires a re#consideration of how, what, when, and where we learn and offer their Explanatory Learning Model (ELM) based upon constructivist experiential learning as a way forward. The ELM extends Kolb’s (1984) model of experiential learning to provide a ‘cycle of learning’ (Mayes & deFreitas, 2007) consisting of: • • •

situative learning—where learners engage in communities of practice, cognitive learning—which builds upon experience, reflection, experimentation and abstraction, and associative learning—employing feedback and transfer.

deFreitas and Neumann (2009) also use Dewey’s concept of inquiry (pre#reflection, reflection, and post#reflection) to posit that learners’ virtual experiences, their use of multiple media, the transactions and activities between peers, and the facilitation of learner control between them lead to ‘transactional learning’. The ELM “aims to support deeper reflection upon the practices of learning and teaching” which arguably leads to “wider opportunities for experiential learning” (ibid p.346). Two recent studies suggest that this concept has merit. Hobbs, Brown, and Gordon (2006) studied students’ interactions within Second Life while completing a series of complex, open#ended tasks and found that, ‘‘with careful planning the intrinsic properties of the virtual world can inform transferable skills and provide a rich case study for learning” (p. 9). These conclusions are supported by Jarmon, Traphagan, Mayrath and Trivedi’s (2009) study, using a mixed methods approach demonstrating the effectiveness of Second Life in a project#based experiential learning approach where students learn by doing and then apply ‘virtually’ learned concepts to the real world. In constructivist learning students are engaged in active cognitive processing (as opposed to simply recalling or memorizing information); they are paying attention to relevant incoming information but are also organizing this into a coherent representation and integrating it with existing knowledge (Mayer, 1999). A principal characteristic of constructivist learning is collaboration among learners but while virtual worlds can facilitate interactions between instructors and students, not all individuals or groups automatically possess the knowledge, attitudes and sensibilities to work together effectively online (Vallance, Towndrow & Wiz, 2010). Some learners may require induction and practice before they are able to leverage benefits from collaborative learning; initial exposure to immersive collaborative environments may therefore not advantage all students equally. Towndrow and Kannan (2002) observed that individuals, regardless of specific task objectives, collaborate best when they have shared knowledge and interests. These preconditions are met in two ways: when online collaborators

Journal of Virtual Worlds Research – Learning Spaces in Second Life

build on prior face#to#face working relationships; and when inputs from collaborating partners are balanced in terms of vision and effort. The aim of many learning situations is to enhance knowledge and understanding which, in Constructivist theory, are not external aspects to be captured but are uniquely internally constructed by individuals; each makes meaning of their own experiences. Constructivism does not prescribe planned strategies of instruction but advocates the creation of learning environments that provide opportunities to engage learners in meaning making. Collaboration thus occurs when learners effectively communicate their understanding, engage with the views of others and reflect upon their encounters (Miller & Miller, 1999), but does not assume all students are similarly equipped to do so. Experimental design When created, effective metrics for designing and evaluating effective learning in immersive virtual environments (IVEs) may draw upon a range of factors and different data types. We adopted a mixed#methods approach therefore, to provide access to rich qualitative and quantitative data enabling different possible relationships and hypotheses to be explored in a context that seemed more likely to offer opportunities to discover relevant factors. The educational tasks provided for students within the virtual environment required a representative range of authentic activities found in real world scenarios, while being broadly typical of problem#solving activities used in engineering and science classrooms, all the while being sufficiently representative of those learning situations within other subject areas to permit generalization. The two characteristics selected were the intrinsic level of task difficulty and the availability of information relevant to the task. In both real world and classroom#based contexts some projects will be more challenging than others and in every case either more or less information relating to the task will be available. Relatively easy tasks can become more challenging if little information about their context or relevant variables is available while otherwise challenging tasks can become less so if, for example, significant information about important constraints and relationships is provided. Task difficulty and available information were therefore applied in our study as key design parameters. We are interested in exploring the degree to which learning activities in virtual worlds are likely to provoke behaviors which can be located within Bloom’s revised taxonomy. The current project was conceived to facilitate an exploration of this by studying the communicative exchanges between and within teams during problem solving activity. A closed and highly defined task seemed most likely to provide the necessary comparability and empirical data to determine the success of task completion. We adopted the programming of robots to navigate mazes of varying complexity for this purpose. Programming the Lego robots used in this study requires—as a minimum for each step—the determination of an action and a vector. The first important variable of task difficulty in this context (Barker and Ansorge, 2007), we defined as the minimum number of discrete maneuvers required to successfully navigate a given maze. A maze requiring five distinct maneuvers such as a forward move, a left turn, a forward move, a right turn and then a final forward move, we defined as a maze of complexity level five. Our group determined that successfully navigating

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Learning Spaces in Second Life 10

this maze would be no different in level of intrinsic difficulty to navigating a maze requiring a right turn, a forward move, a right turn, a forward move and then a left turn. Mazes with differing levels of intrinsic difficulty could therefore be provided for participants to facilitate true comparisons of like with like and to act as the problem specification dependant variable. Available information was our second design parameter. Any real#world, designed learning or problem solving activity is accompanied by some amount of information about the learning required or the problem to be solved. In the context of the current study we anticipated that the amount of information provided about a maze to be navigated would significantly affect the nature and volume of communication between the two collaborating teams. Tasks of equivalent complexity (intrinsic difficulty) were therefore employed together with task information of three kinds from those deemed relatively rich and complete to those which were relatively poor and incomplete: a scale image of the maze to be navigated together with its specification (numbers and types of maneuver) would be a rich source of information relevant to designing a program to successfully navigate the robot through it. Having an image only would provide less information; whilst having the specification only (for example the unordered number and type of turns and straights) would represent relatively sparse information. Sets of mazes of equivalent complexity were paired for learning/teaching with the participant groups using these three levels of relevant information as the complexity level rose. The â&#x20AC;&#x2DC;available informationâ&#x20AC;&#x2122; parameter was defined as the communicative constraint imposed upon the problem. These variables may be related in a mutually inclusive grid and the study presented here focused on the exploration of six cells (Table 2). Ongoing studies are extending exploration into the remaining cells.

Image, specification, avatar posture, Mindstorms icon manipulation, in!world chat/text

Configuration, complexity and sub!task sequence prescribed Task X complexity only prescribed Unprescribed task

Communication constraint

Image, specification, in!world chat/text

Image, in!world chat/text

Specification, in!world chat/text

In!world chat/text only

High

Low

Problem specification

High

Journal of Virtual Worlds Research – Learning Spaces in Second Life 11

Table 2. Task design matrix showing communication constraint, problem specification and intersection cells explored in the present study.

We applied overarching constraints to interactions between two physically separated teams consisting of two participants during experimental runs. We designed these constraints to encourage more and richer communicative exchanges and encourage questioning, innovative thinking, and hypothesis formation. In one constraint we did not allow students to design mazes with consecutive identical robot maneuvers (e.g. a left turn may not be followed immediately by another left turn). A second constraint was placed on communicative exchanges. In paired runs each team designed a maze of the given level of complexity; then one team first acted as the ‘teachers’ while the other team (the ‘learners’) attempted to create a program to navigate the robot through this successfully. Then teams exchanged roles. For each level of maze complexity all of the communication constraints were applied in sets of paired runs (Table 2). Participants were told that when responding as ‘teachers’ to requests for feedback from ‘learners’ they should respond as helpfully as possible. However, they were not permitted to offer unsolicited information or advice or to reveal the correct action/vector required by the robot at any given point to make further progress. In line with the project’s adopted methodological context, teaching teams were therefore tasked to develop imaginative strategies to help learning teams succeed, but those strategies could not include supplying them with the most productive questions or ‘correct answers’.

Journal of Virtual Worlds Research – Learning Spaces in Second Life 12

We anticipate that from our data, relationships to elements within Bloom’s taxonomy will emerge between communicative constraint and problem specification. It seems probable, for example, that the amount of available information about a task of given difficulty may influence the appearance and frequency of different cognitive processes (analyzing, evaluating, etc.) and their application to particular types of knowledge (factual, procedural, etc.). Such relationships could inform metrics for evaluating and designing learning scenarios in immersive virtual worlds, thereby promoting effective learning outcomes by applying certain parameters and constraints. The data from measurements of flow, video, text/chat, diaries, and interviews is expected to further illuminate the discovered potential relationships and enrich our understanding of how these relationships may correlate to individual and group learning in our collaborative scenarios. Materials, procedure and data collection Materials We used two computer configurations: desktop PCs running Windows XP Service Pack 3, using 3.0 GHz Intel Core 2 Duo processors (model E8400), with 2 GB RAM, ATI X1300 graphics cards, and 17” LCD monitors with screen resolution set to 1280´1024 pixels; and Apple MacBook Pro laptops running OSX version 10.5.X, using 2.26 GHz Intel Core 2 Duo processors, with 2 GB RAM, NVIDIA GeForce 9400M graphics cards, and the 13” inch laptop display set to 1280 x 800 pixels. We chose Lego Mindstorms NXT software version 1.1 to create robot programs. The design of the robot followed instructions 8527 of Quickstart#Mindstorms (see http://preview.tinyurl.com/yfw75s2) and was adopted due to its simplicity and its potential for sensors to be added as the research and task framework is further developed (Figure 2).

Journal of Virtual Worlds Research – Learning Spaces in Second Life 13

Figure 2. Robot design displayed within the Second Life experimental environment.

We used the Second Life interactive 3#D Internet environment to support inter#team communication and record ‘chat’ communication, using a private secure island leased from Linden Labs. Two video cameras were used to record intra#team communication for each of the teams. Additionally, a tailored Visual Basic program was employed to measure flow using Guo and Poole’s (2009) psychometric inventory. Flow, in this context, is defined as ‘a state of consciousness that is sometimes experienced by people who are deeply involved in an enjoyable activity’ (Pace, 2004). The flow scale measures three pre#conditions of flow (clarity of goal; fast, unambiguous feedback; and perceived balance of challenge and skill) and six dimensions of flow (concentration, perceived control, mergence of action and awareness, transformation of time, transcendence of self, and autotelic experience). The scale has good psychometric properties in terms of both convergent and discriminant validity and of reliability (Guo & Poole, 2009). Procedure Problem#solving with online communication was studied during the consecutive run of sessions for given tasks. In each session the two teams were located in physically separate computer laboratories within the same building. The teams communicated through Second Life’s chat facility and through the behavior of participants’ avatars in the environment. Each team had to design the course of an identical robot on the floor of their laboratory using adhesive tape. Next, one team’s task was to act as ‘learners’ and create a robot program (using the MindStorms software) to follow the course that the other (teaching) team had designed, drawing on the information provided within the strictures of the communicative constraints being applied. The learning team used the information provided in an attempt to solve the robot#programming !

Journal of Virtual Worlds Research – Learning Spaces in Second Life 14

problem. Once the learning team had designed what they judged to be a suitable program, they sent it by email to the teaching team. The teaching team used their robot to run the program on the taped course to establish its success. They were encouraged to offer feedback to the learning team—via the Second Life environment—when the robot executed an incorrect maneuver and to answer any questions posed. However, the communication between the teams was mediated by the protocols pre#set for each experimental run (Table 2) as described above. The implications of this are discussed below. The cycle of learning and teaching through communication, programming, and testing was repeated. The task was terminated when the teaching team concluded that the learning team’s executed program for navigating the designed course was successful, after allowing for uncontrolled variables such as mechanical differences between the two robot devices in traction, drive variability, and other variance due to the manufacturing tolerances inherent in mass produced robotics kits. Then the teams switched roles and the new learning team (the teaching team in the first run) created a robot program to solve the course designed by the new teaching team (the learning team in the first run). Each paired series was run with escalating difficulty; each with all three variants of communicative constraint (Table 2). Data collection During each session, the activities and intra#team communication of each team were recorded on video; the inter#team chat communication in Second Life was recorded as a log file in ASCII format; and the robot programs that were created were copied. Additionally, the experience of the members of each team was measured—usually at the end of a session and also sometimes immediately before a break within a session—using the flow scale. Participants also kept a personal diary of their experience of the problem#solving sessions. Data analysis Pre#processing of the video recordings of the intra#team communication was necessary to produce data that could be analyzed on all of the computer platforms employed by the research team. We transcribed and stored this communication in word#processed format. We used thematic content analysis (Riffe et al, 2005) to analyze the between#team and intra#team communication data and diary data. We also analyzed the assignments in terms of time#on#task and number of attempts to create a correct robot program. Robot programs were analyzed in terms of their complexity. Flow#scale data were analyzed using statistical techniques. Process We followed a well#established process of content analysis for the inter#team communication data (Riffe et al, 2005). First, the content of each participant’s verbal report was divided into units of thoughts (Gardial et al, 1994). Each unit is the smallest set of words that are meaningful outside its context, which allows coders to interpret the meaning of a statement without reading the preceding or following text where further division of a unit would render it meaningless. We used Bloom’s updated taxonomy (Anderson et al, 2001) to adopt and further develop a coding scheme for the communication in second worlds (Vallance et al, 2010). The following communicative functions were targeted for coding into one or more of the communicative categories:

Journal of Virtual Worlds Research – Learning Spaces in Second Life 15

• • • • • • • • • • • •

independently providing information; asking questions; confirming information previously provided; answering a question with an affirmative statement; expressing an observation; making a suggestion; giving a direct instruction; praising; making a request; answering a question with a negative statement; apologizing; greeting.

Five independent coders pretested an initial subset of data within these categories using a randomly selected subsample of 10% of the units as training material. We revised the definitions of categories where needed and continued pretesting until intra#team coding was consistent. A second randomly selected subsample was coded and reliability of coding was established calculating the percentage of coding agreement. The process of selecting subsamples and coding was repeated until we reached 90% coding agreement. We followed the same process for the content analysis of intra#team communication data and diary data. The coding scheme for intra#team communication was developed from Vallance et al’s (2009) scheme and the scheme for the diary data was developed based on an initial reading of participants’ diaries from which we extracted reflective experience. We used the number of steps in a task to define its complexity. We analyzed psychometric flow data for factor structure using factor analysis and for reliability using Cronbach’s alpha reliability coefficient. We calculated correlations with time#on#task, attempts required to create a successful robot program, and robot#program complexity (lines of code) by using validity analysis. Data matching and cross referencing For each session, we created displays including course complexity, constraints imposed on communication, the results of content analysis of inter# and intra#team communication per team, flow per participant and per team, time#on#task, attempts required to create a successful robot program, and robot#program complexity. We used both of the allowed within#case displays and cross#case displays for exploratory and explanatory data analysis (Miles & Huberman, 1994). Within#case displays allowed the cross#validation of measures by identification of patterns of association between different types of datum (e.g. inter#team communication and intra#team communication). Using cross#case displays, we established the consistency of cross#validation across sessions by determining to what extent the same patterns occurred over sessions. In addition, we used cross#case analysis to establish the effect of manipulations of course complexity and communication constraints on the different types of datum. Deriving metrics Our group derived metrics for each type of datum and made distinctions between measures that were involved in patterns of association discovered in cross#validation and those that were not. The reason for this differentiation was that those measures that were involved in patterns were

Journal of Virtual Worlds Research – Learning Spaces in Second Life 16

considered to show evidence of validity (through association with other measures) or sensitivity (through association with experimental manipulations). Proposed measures of inter# and intra#team communication were the communicative functions (presence or frequency of functions) identified from content analysis. We used measures of flow for dimensions of the flow state scale, and measures of reflective experience for thematic codes (presence or frequency of functions) which we identified from content analysis. Using the thematic coding schemes created in the current study may allow for the development of automatic analysis of (transcribed) communication data and (reflective) diary data stored in digital form. For this aspect, we used taxonomical approaches such as WordNet (Miller, 1995) based on human#coded taxonomies using synonyms and statistical techniques that employ unsupervised machine learning and can reproduce human performance accurately (Landauer & Dumais, 1997). Examples include Latent Semantic Analysis (Landauer & Dumais, 1997), Point#wise Mutual Information (Manning & Schutze, 1999), and Hyperspace Analogue to Language (Lund & Burgess, 1996). Research in other domains has shown that statistical techniques produce more valid results than taxonomic techniques (Kaur & Hornof, 2005; Landauer & Dumais, 1997) although further research, including the present study, may challenge this. Discussion The need to promote scientific enquiry among students and the application of approaches that use immersive virtual environments seems likely to grow. However, the use of new technologies needs to be accompanied by clear methods for measuring learning and learning outcomes. Establishing metrics, not only for science learning but also for learning that occurs within virtual environments, is necessary if educators hope to accomplish more than simply using this new technology because it is readily available. This project attempted to explore how learning could be measured while students programmed robots to solve a series of problems. Our initial findings indicate that the development of clear and explicit metrics will be possible but that further research will be needed to ensure their validity. After agreeing on the design and implementation of the tasks, we made several assumptions: • • •

that the data sources (text chat logs, video logs of participants, reflective diaries, and interviews) would be sufficient for establishing a series of metrics; that the structure of the tasks and their subsequent iterations would be related to the level and nature of communication between team members and teams; that by varying task difficulty, constraining communication and the modalities for communicating, a range of strategies for problem solving, negotiation of meaning, communication repair and hypothesis formation would emerge.

These assumptions we found to be generally true; when images of the mazes were provided, team members communicated less with each other and dialogue between teams tended to be limited to defining the parameters of each programming action. Providing only information about the number of turns in the maze, however, increased communication between team members and led to a slight increase in dialogue between teams. When we changed the task either to provide no image, or to give less descriptive help, we experienced a marked jump in intra- and inter-team communication.

Journal of Virtual Worlds Research – Learning Spaces in Second Life 17

However, we observed a number of unanticipated outcomes. One of the most significant outcomes was the variation in the amount of discussion between team members. One team consisted of members who were relatively communicative but had different levels of technical knowledge. A second team consisted of one communicative member with a high level of technical knowledge and another quieter member with limited technical knowledge. While we originally assumed that there would be a disparity in the amount of communication between and within the two teams, the initial review of the video recordings showed that far less discussion took place between the latter team members than was expected. An examination of the text log chats verified this disparity. The design and completion of the tasks led to constant discussions between the participants and the researchers. While we provided instructions to the participants, they nevertheless consistently asked questions and negotiated for modification of the constraints. This conversational framework (Laurillard, 2002) between participants and researchers occurred on a daily basis and led to modifications not only in the daily tasks, but in the reformulation of the final task as well as changes in the communication parameters. However, the nature of the tasks themselves created a conversational framework between teams. Members of the teams discussed task goals, debated perceptions of their learning goals, took responsibility for directing the focus of the discussions, and provided imaginative feedback to each other (Britain & Liber, 1999). The tasks also required the participants to take an active role in their learning. The teams of participants received minimal programming instructions for the robots except for a general orientation to the software. They were responsible for learning how to program each step and determining the range of variables to be used. Since each task (except for the final one) consisted of a problem with a set solution, the problem space was limited but contained enough variation for the participants to experiment and formulate hypotheses. This co#construction of programming knowledge between participants enabled them to develop autonomy and not to rely on a teacher or tutor to provide them with guidance. Also, the use of an immersive virtual environment (IVE) created a space that was separate from the shared physical space where the researchers could also be present. We hypothesized that this “spatial differentiation” increased the participants’ reliance on one another rather than a teacher or tutor who would be present in the room. Since the problem and the solution were situated “in world”, and the mode of communication was text chat, participants tended to ask each other questions and solve their own problems before asking a member of the research team for answers. The basic communication mode of IVEs is text chat, although voice communication is available. When using an IVE such as Second Life, residents/students/learners can define which mode they use and switch between them. This allows educators to either limit learning modalities—for example by only allowing chat—or to increase the range of learning modalities used either by allowing or by requiring both chat and text. Also, IVEs are virtual 3D worlds with embodied participants represented by avatars who move within a world. This simulation of movement can possibly give advantage to learners who are kinesthetically oriented. One imaginative and unanticipated strategy developed by participants in the present study exemplified this advantage when teams in their ‘teaching’ role made use of

Journal of Virtual Worlds Research – Learning Spaces in Second Life 18

avatar gestures— using feedback to the ‘learning’ team—to signal the required successive robot maneuvers necessary to navigate their maze. Additionally, artifacts are built and constructed in IVEs, providing opportunities for students who learn through manipulation of objects. These environments provide multimodal learning opportunities that can be incrementally altered, allowing learners not only to use their preferred learning modes and strategies but also to explore and develop the use of new modalities. Through this project we seek to explore how educational tasks can be designed for different learning objectives. Effective tasks should not only accomplish specified learning goals, but should also provide opportunities for exploration and discovery. At the same time, the tasks must take into account individual differences. Our approach has been to create a set of problems that need to be solved in both the real and virtual worlds. Different learning objectives often require different design components within teaching. Rather than creating unique tasks for multiple learning objectives, we feel it is possible to deploy similar tasks to achieve a variety of goals in immersive virtual environments. For example, if an instructor is interested in maximizing communication, then limiting the amount of provided information may lead to increased negotiation between participants (given an appropriate scenario). If hypothesis formation is the goal, then the instructor may instead need to vary the level of problem complexity. Conclusions One of the significant educational benefits of using immersive virtual environments for achieving learning objectives is that they allow individual and cooperative convergence as well as collaborative learning to take place. Learning can occur not only as an individual act of obtaining and deploying knowledge, but also through the co#construction of understanding through participation in a learning community; these aspects are enhanced when students are actively involved in and responsible for their learning. However, the most effective learning is situated in contexts that are meaningful; immersive virtual environments allow educators and educational institutions to create simulations that meet these situations. Creating a relevant and meaningful context is possible in virtual reality by designing a space that best facilitates learning. For example, if students are exploring the concept of acceleration, educators are able to create a virtual laboratory where simulated objects can be observed and measured. However, it is also possible to model otherwise inaccessible objects such as planets or high speed vehicles that move in 3D, and then to change a wide range of variables so that learners can formulate and test their hypotheses under different conditions. For effective learning and teaching to take place, collaborative learners must be actively involved in meaningful activities that have meaningful contexts. Digital communication has created new opportunities, but often traditional teaching and learning methodologies have been employed without due consideration of their suitability. Teachers and universities should require a theoretical framework to better leverage the benefits and affordances of immersive virtual technologies. The conversational framework, for instance, is an approach that lends itself to networked learning, as do collaborative and constructivist learning models. However, we suggest in this study that approaches employing concepts of task difficulty and communicative richness and constraint must inform metrics to better design and evaluate effective learning within immersive virtual environments. By combining these approaches with Vygotskian social

Journal of Virtual Worlds Research – Learning Spaces in Second Life 19

constructivist learning models and the concepts of purposeful tasks, collaborative learning, and meaningful contexts, educators can develop a more robust understanding of what constitutes effective learning processes in immersive virtual environments and thereby facilitate better opportunities for learning within them. Initial indications from our data suggest that the nature and defined difficulty of learning tasks can—together with measures of communicative constraint and of the information provided about the task—be used to create metrics for designing and evaluating learning scenarios in immersive virtual environments that can be articulated within Bloom’s revised taxonomy. We propose that the revealed dynamics between these taxonomic elements and the developed metrics will provide insights into the nature of effective pedagogy in these new learning and teaching environments. Such dynamics will facilitate their fuller educational exploitation, especially considering the affordances they offer which are unavailable in more traditional contexts. A subsequent paper will present the completed analysis and derived metrics from our data and discuss the pedagogic model we develop from our experiments. Ours is an international project involving collaboration between four academics in three Higher Education institutions from two countries with significantly different cultures. The research team’s backgrounds and expertise cover the fields of education, media, psychology, science, and engineering. One of the strengths of such collaborations is the rich experience and diversity that they contribute to the research enquiry. Other strengths are the opportunities they offer for drawing upon a broad sweep of pedagogy, methodology, and social perspective in different environmental and economic contexts which nonetheless broadly share common economic, scientific, engineering and educational aspirations. These projects are not, however, without practical problems in addition to those most immediately obvious as a result of the wide geographical separation of researchers. Additional challenges emerge when creating common research protocols that: • • •

must function in institutions which apply different procurement constraints; use dissimilar and sometimes incompatible IT infrastructures; and are driven by policies with very different embedded institutional and cultural assumptions about the nature of academic roles and responsibilities including those towards and for students and research participants.

International research into the effective use of immersive virtual environments, by its very nature, offers the opportunity to annihilate conventional frames of space and time. Such research also involves successfully navigating a combination of real#world and in#world assumptions, limitations, and obstacles while exploiting their many opportunities and resources for exploration, innovation, and progress. Work proposed for the next phase of our project will not only build upon our existing experimental context to complete the exploration of our task design matrix (Table 2), but will also be developed to target alternative learning scenarios and curriculum areas. This is an attempt to demonstrate the degree of generalizability of our findings, metrics, and proposed pedagogy within the two cultural contexts.

Journal of Virtual Worlds Research – Learning Spaces in Second Life 20

Acknowledgment The authors gratefully acknowledge the financial support from the British Council for this research project under ‘The Digital Economy’ (Prime Minister’s Initiative) funding stream and the Japan Society for the Promotion of Science (grant number 20500758).

Journal of Virtual Worlds Research – Learning Spaces in Second Life 21

Bibliography Aainsqatsi, K. (2008). Retrieved from http://en.wikipedia.org/wiki/File:Blooms_rose.svg Anderson , L.W., & Sosniak, L.A. (Eds.), (1994). Bloom's taxonomy: a forty#year retrospective. Ninety-third yearbook of the National Society for the Study of Education. Chicago: University of Chicago Press. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruicshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J. & Wittrock, M.C. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy of educational objectives (Abridged ed.). New York: Longman. Barker, S.B. & Ansorge, J. (2007). Robotics as a means to increase achievement scores in an informal learning environment. Journal of Research on Technology in Education, 39(3), 229#243. Bloom, B.S. (Ed.) (1956). Taxonomy of Educational Objectives, the classification of educational goals. Handbook 1: Cognitive Domain. New York: McKay. Britain, S., & Liber, O. (1999). A framework for pedagogical evaluation of virtual learning environments.

JISC

Technology

Applications

Programme.

Retrieved

from

http://www.leeds.ac.uk/educol/documents/00001237.htm#_Toc463843836 Calongne, C. (2007). A View from Second Life’s Trenches: Are You a Pioneer or a Settler? The New

Media

Consortium

Summer

Conference

Proceedings.

Retrieved

from

http://www.nmc.org/2007#proceedings Cuban, L. (1992). Curriculum Stability and Change. In P.W. Jackson (Ed.) Handbook of Research on Curriculum. New York: Macmillan. deFreitas, S. & Neumann, T. (2009). The use of ‘exploratory learning’ for supporting immersive learning in virtual environments. Computers & Education, 52, 343#352. deFreitas, S. (2008). Serious Virtual Worlds: a Scoping Study. Bristol: Joint Information Systems Committee. deFreitas, S., & Griffiths, M. (2008). The convergence of gaming practices with other media forms: what potential for learning? A review of the literature. Learning, media and technology, 33(1), 11#20. Dror, I.E. (2007). Gold mines and land mines in cognitive technology. In Itiel E. Dror (Ed), Cognitive Technologies and the Pragmatics of Cognition. Amsterdam: John Benjamins. !

Journal of Virtual Worlds Research – Learning Spaces in Second Life 22

Field, M., Vallance, M. & Yamamoto, T. (2008). Crossing the Discipline Divide: Building Learning Contexts. In: Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 84#89. Chesapeake, VA: AACE. Gardial, S., Clemons, D., Woodruff, R., Schumann, D. & Burns, M. (1994). Comparing consumers’ recall of prepurchase and postpurchase product evaluation experiences. Journal of Consumer Research, 20, 548#560. Guo, M. & Poole, M. (2009). Antecedents of flow in online shopping: a test of alternative models. Information Systems Research, 19, 369#390. Hobbs, M., Brown, E., & Gordon, M. (2006). Using a virtual world for transferable skills in gaming education: The Higher Education Academy. ITALICS 5(3). Retrieved from http://www.ics.heacademy.ac.uk/italics/vol5iss3/hobbsbrowngordon.pdf. Jarmon, L., Traphagan, T., Mayrath, M. & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life. Computers & Education. 53, 169#182. Jennings, N. & Collins, C. (2008) Virtual or Virtually U: educational institutions in Second Life. International Journal of Social Sciences, 2(3), 180–186. Jonassen, D.H., & Land, S.M. (2000). Theoretical foundations of learning environments. Mahwah, N.J: Lawrence Erlbaum Associates. Kaur, I. & Hornof, A.J. (2005). A comparison of LSA, WordNet and PMI#IR for predicting user click behavior. Proceedings of the SIGCHI: Conference on Human Factors in Computing Systems, 51–60. Portland, Oregon: ACM. Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. New Jersey: Prentice#Hall. Landauer, T.K. & Dumais, S. (1997). A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211#240. Laurillard, D. (2002). Rethinking University Teaching. A conversational framework for the effective use of learning technologies. New York: Routledge Lund, K. & Burgess, C. (1996). Producing high#dimensional semantic spaces from lexical co#occurrence. Behavior Research Methods, Instrumentation, and Computers, 28, 203#208.

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Learning Spaces in Second Life 23

Manning, C. & Schutze, H. (1999). Foundations of Statistical Natural Language Processing. Cambridge, MA: The MIT Press. Manuel, J. (2000) From http://en.wikipedia.org/wiki/User:JohnManuel Martin, S. and Vallance, M. (2008) The Impact of Synchronous Inter#Networked Teacher Training in Information and Communication Technology Integration. Computers and Education, 51(1), 34#53. Mayer, R. H. (1999) Designing instruction for constructivist learning. In C. M. Reigeluth (Ed), Instructional design theories and models: A new paradigm of instructional theory, Volume II,141#160. Mahwah, NJ: Lawrence Erlbaum Associates. Mayes, T. & deFreitas, S. (2007) Learning and e#Learning: The role of theory. In H. Beetham & R. Sharpe (Eds) Rethinking pedagogy in the digital age. London: Routledge. Miles, M. & Huberman, M. (1994). Qualitative data analysis: an expanded sourcebook. Thousand Oaks, CA: Sage. Miller, G.A. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11), 39#41. Miller, S.M. & Miller, L.M. (1999) Using Instructional Theory to Facilitate Communication in Web#based Courses. Educational Technology & Society, 2(3). Pace, S. (2004). A grounded theory of the flow experiences of web users. International Journal of Human-Computer Studies, 60, 327#363. Riffe, D., Lacy, S., & Fico, F. (2005). Analyzing media messages: using quantitative content analysis in research. Mahwah, NJ: Erlbaum. Rovai, A.P., Wighting, M.J., Baker, J.D. & Grooms, L.D. (2009) Development of an instrument to measure perceived cognitive, affective, and psychomotor learning in traditional and virtual classroom higher education settings. Internet and Higher Education, 12, 7â&#x20AC;&#x201C;13. Towndrow, P. A., & Kannan, J. (2002) The role of collaboration in the design and production of on#line English language learning materials. Teaching and Learning, 23(1), 77#84. Vallance, M. & Wiz, C. (2008) The Realities of Working in Virtual Worlds. In: Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, 3085#3090. Chesapeake, VA: AACE. Vallance, M., Martin, S., Wiz, C. & Schaik, P. van (2010). Designing defective spaces, tasks and metrics for communication in second life within the context of programming LEGO NXT

Journal of Virtual Worlds Research â&#x20AC;&#x201C; Learning Spaces in Second Life 24

Mindstorms tm Robots. International Journal of Virtual and Personal Learning Environments, 1(1), 20-37. Vallance, M., Towndrow, P.A. & Wiz, C. (2010) Conditions for Successful Online Document Collaboration. Tech Trends, 54(1), 20#23. Warschauer, M. (1999). Electronic literacies: language, culture, and power in online education. Mahwah, N.J: Lawrence Erlbaum Associates.