CUbRIK First GWAP and implicit user information Techniques

FIRST GWAP AND IMPLICIT USER INFORMATION TECHNIQUES Human-enhanced time-aware multimedia search

CUBRIK Project IST-287704 Deliverable D3.1 WP3

Deliverable Version 1.0 – 31 December 2012 Document. ref.: cubrik.D31.QMUL.WP3.V1.0

Programme Name: ........................ IST Project Number: ............................. 287704 Project Title: ................................... CUbRIK Partners:........................................ Coordinator: ENG (IT) Contractors: UNITN, TUD, QMUL, LUH, POLMI, CERTH, NXT, MICT, ATN, FRH, INN, HOM, CVCE, EIPCM Document Number: ..................... cubrik.D31.QMUL.WP3.V1.0.doc Work-Package:............................... WP3 Deliverable Type: .......................... Report Contractual Date of Delivery: ........ 31 December 2012 Actual Date of Delivery: ................. 31 December 2012 Title of Document: ....................... First GWAP and Implicit User Information Techniques Author(s): ..................................... Navid Hajimirza, Naeem Ramzan, Ebroul Izquierdo (QMUL), Luca Galli (POLMI), Aliaksandr Autayeu (UNITN), Raynor Vliegendhart (TUD) Approval of this report ................... Executive Committee Summary of this report: .............. Design and implementation of a game with a purpose in the CUBRIK framework and current status of the Games under development and implicit user derived information History: ........................................... First draft 31st Oct 2012 (Naeem Ramzan), Second draft 30th Nov 2012 (Navid Mirza and Luca Galli), first revision 19th Dec 2012 (Ebroul Izquierdo), Final revision 28th Dec 2012 (Navid Mirza and Ebroul Izquierdo) Keyword List: ................................. Games with a Purpose, conflict manager in the Content Processing Tier of CUbRIK, General architecture of the CUbRIK Gaming Framework, structure of an achievement system, Image Annotation and Sketchness, implicit user information and implicit feedback Availability .................................... This report is: public

This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License.

This work is partially funded by the EU under grant IST-FP7-287704

CUbRIK First GWAP and Implicit User Information Techniques

D3.1 Version 1.0

Disclaimer This document contains confidential information in the form of the CUbRIK project findings, work and products and its use is strictly regulated by the CUbRIK Consortium Agreement and by Contract no. FP7- ICT-287704.

Neither the CUbRIK Consortium nor any of its officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein.

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2011-7) under grant agreement n째 287704.

The contents of this document are the sole responsibility of the CUbRIK consortium and can in no way be taken to reflect the views of the European Union.

CUbRIK First GWAP and Implicit User Information Techniques

D3.1 Version 1.0

Table of Contents EXECUTIVE SUMMARY 1.

GWAP PLATFORM MODEL IN CUBRIK FRAMEWORK 2.

GWAP: A SURVEY OF RELATED WORK 2.1 MAIN STRUCTURAL GAME MECHANICS 2.2 EXISTING GAME-BASE FRAMEWORKS 2.2.1 Games for object annotation 2.2.2 Games for image segmentation 2.2.3 Games for Gesture based Image Tagging 2.2.4 Games for image rating 3.

CUBRIK GAMING FRAMEWORK AND INITIAL GAME INSTANTIATIONS 3.1 ACHIEVEMENT DEFINITION AND DESIGN 3.2 OPEN ACHIEVEMENT FRAMEWORK 4.

GWAP FRAMEWORK FOR IMAGE ANNOTATION 4.1 SYSTEM OVERVIEW 4.2 PAYOFF CALCULATION AND DECISION MAKING UNIT 4.3 PLAYER OUTCOME PREDICTION UNIT 4.4 IMPLEMENTATION OF THE FRAMEWORK 4.5 GRAPHICAL USER INTERFACE 4.6 EXPERIMENTAL SETUPS 5.

GWAP FOR SEGMENTATION: SKETCHNESS 5.1 GAME STRUCTURE 5.1.1 Scoring 5.1.2 Winning 5.2 SKETCHNESS: HC TASK ANALYSIS 6.

GWAP FOR METADATA VERIFICATION: CROSSWORDS 6.1 CROSSWORD PUZZLE GENERATION 6.2 CROSSWORD PUZZLE SOLVING 6.3 FEEDBACK COLLECTION 6.4 MEDIA METADATA CORRECTION 7.

IMPLICIT USER INFORMATION TECHNIQUES 7.1 IMPROVING PEOPLE RECOGNITION BY IMPLICIT FEEDBACK 7.1.1 Introduction to the People Recognition Framework 7.1.2 Application Context 7.1.3 Dataset 7.1.4 Technical Contribution 7.1.5 Incorporating Implicit Feedback: Overview 7.1.6 Detailed Description of how Implicit Feedback is incorporated 7.2 LIKELINES: COLLECTING IMPLICIT USER INFORMATION 7.2.1 Use of implicit playback behaviour 7.2.2 Technical details of the LikeLines player component 8.

TOWARDS POTENTIAL FUTURE EXTENSIONS 9.

REFERENCES

CUbRIK First GWAP and Implicit User Information Techniques

1 2 6 6 8 8 9 10 10 11 11 14 17 17 19 19 20 20 23 24 25 25 26 26 27 27 29 29 30 31 31 31 32 32 32 32 34 34 34 35 36 38

D3.1 Version 1.0

Executive Summary In this document the Games with a Purpose (GWAP) platform model in CUbRIK is described, and its application in the following two key tasks presented: •

Generation of high level metadata using human input.

Interaction with the conflict manager in the Content Processing Tier of CUbRIK for improved conflict resolution.

The general architecture of the CUbRIK Gaming Framework is provided. This framework architecture is designed targeting easy development, improvement and maintenance of available games, as well as design of additional GWAPs. Furthermore, the structure of an achievement system, i.e. a set of tasks defined by a designer, for the player to achieve a milestone and to track progress in the system, is discussed. It is explained how and why it is possible to implement such an achievement system in the CUbRIK platform. In a nut shell, the corresponding unit is used to provide incentives for the user, enabling retention and engaging capabilities.

Three instantiations of the framework are described: Image Annotation, Sketchness and Crosswords. These are games implemented within the GWAP modules of CUbRIK. They belong to Output agreement and Inversion problem game classes. The aim of these games is to provide annotations for images and verification of metadata, while users are playing and having fun with the games.

Finally, a method for managing implicit user information is described. It identifies people in a photo collection by using implicit feedback obtained through the dedicated feedback frameworks of the CUbRIK system. •

CUbRIK First GWAP and Implicit User Information Techniques

Page 1

D3.1 Version 1.0

1.

GWAP Platform Model in CUbRIK Framework

The aim of the CUbRIK project is to develop a modular framework and distributed system architecture for flexible design and implementation of multimedia search applications allowing easy reuse of existing components and multimedia processing workflow, their extension with domain-specific elements, and the incorporation of human computation for tasks requiring human intelligence in the solution process.

Traditional multimedia search engines are still not able to leverage the full potential of the entertainment capabilities offered by the technological advancement to drive the increased amount of users that are willing to work to improve the results of their search. With the increased need of human contribution to tailor and improve search results and provided media content, new paradigms able to encourage and reward the improvements brought by the users are needed.

The CUbRIK project aims at exploiting Games with A Purpose (GWAPs), digital games where players generate useful data as a by-product of play. GWAPs are usually applied to tasks where the problem to solve is out of the reach of traditional machine-learning algorithms, such as common sense elicitation and content tagging for multimedia search, thanks to the fact that humans have superior capacity for understanding complex content.

As stated in the DoW this will be achieved by "revising the proposed patterns in the literature (e.g., input agreement games, inversion-problem games, output optimization games, etc.

[Kam2009]) and extending the works to deliver cutting-edge techniques for the most prominent challenges in multimedia search."

It is said that “a picture is worth a thousand words”. This refers to the idea that complex scenarios can be represented by just a single image. Human beings are all capable of obtaining a majority of information in the real world by visual sense and this includes entities that can be visualized, such as images and videos. Recent developments in social networks and an increasing number of portable electronic devices, such as cameras and camera embedded mobile phones, have contributed to the already large quantity of digital multimedia content on the World Wide Web. As a consequence, the following question arises, do people label the content? If so, how often do they do so? With the increase of digital media, problems of automated classification, annotation, indexing, retrieving, and aggregating become critical for the provision of useful and user friendly multimedia systems. Reacting to these and other similar questions, researchers around the world have designed a considerable number of algorithms and frameworks with the capabilities of automated image annotation.

Over the last decade, a number of research directions have been explored addressing the semantic gap problem. One such approach is crowdsourcing (or manual annotation), which has been successfully used for harvesting multimedia annotations. For instance, very promising results have been reported for the well-known ESP game (see references and details in the next section). It has been shown that this particular game can be modified to annotate different types of multimedia materials or features. Thus, games like this are called “Games with a Purpose”. They are at the centre of the research and development described in this document.

Since the ESP game was introduced, a number of similar approaches to address the semantic gap problem have been proposed. The most basic approach in engaging human attention is designing interactive frameworks with multiplayer game strategies. It has been shown to be fun and entertaining. As a result, public attention is drawn into playing the game and its real purpose, image annotation, goes largely unnoticed. Although the literature is full of game-based approaches, little research has been conducted on standalone games and Game Theory based approaches for image annotation. For that reason these two aspects are directly addressed in the underlying CUbRIK framework reported in this document.

CUbRIK First GWAP and Implicit User Information Techniques

Page 2

D3.1 Version 1.0

Figure 1: GWAP framework in the CUbRIK platform.

CUbRIK First GWAP and Implicit User Information Techniques

Page 3

D3.1 Version 1.0

One major issue of this kind of application derives from the fact that the games are designed and tailored over the specific task that has to be solved on an ad-hoc basis. This can lead to an experience that may still be perceived by the users as work and not as entertaining as other interactive media applications that follow established game design patterns and incentives for the players. For these reasons, it is necessary to investigate the design of game mechanics and motivation techniques in games in order to solve human computation tasks and define a methodology for the assignment of human computation tasks to the right players based on their profile information and past history. These problems are addressed in CUbRIK with the use of a Gaming Framework that provides a set of tools and guidelines that can ease the development of novel engaging applications able to exploit human contributors.

In the CUbRIK Framework, GWAPS are classified as applications and their main contribution is to bring humans in the loop of the search process: the Gaming Framework is in charge of improving the platform services, namely, Task Management Support, Incentive Support and User Interface Support.

The main task of the Gaming Framework in CUbRIK is solving tasks to be executed with the aid of users in situations in which other processes (pipelines) have failed or for which no known software component can be used. Figure 1 shows the standing of GWAP and its corresponding Gaming Framework inside the CUbRIK platform. In this diagram, GWAP is classified as: •

Human annotation apps;

• Conflict resolution apps.

Whenever a player is using an application that makes use of the gaming framework, he/she will communicate with the GWAP servers where their profile, gaming scores, history, level of trust, etc. are updated constantly in real-time. When a user is considered as trustworthy (on the basis of her previous history and live annotation), his/her input is accepted as valid metadata and stored for the relevant multimedia content.

GWAPs either directly provide the high level metadata that can only be generated by means of human contribution or aid the conflict manager in the Content Processing Tier to resolve the conflicts that machines cannot handle alone. The gaming applications as part of Conflict Resolution Applications in the CUbRIK platform can resolve the problem of potential conflicts by using human resources. The advantage of the framework is to delegate certain processes to humans while keeping the task to be resolved engaging. This includes image annotation, bridging the semantic gap (the difference between human understanding and machine understanding of the multimedia content) and understanding the media when the human level interpretation is required. In order to do so, the Gaming Framework is structured to: •

Exploit the power of media enhancement techniques namely image annotation, object recognition, media clustering, event detection, etc;

•

Keep the users motivated for their contribution by advanced rewarding systems;

• Filter the users based on their trustworthiness in providing the metadata content.

Figure 2 shows the architecture of the Gaming Framework. As shown, multiple tasks are introduced into the system. These tasks are either the ones that cannot be performed by other processing units of CUbRIK without the human in the loop or the ones that are referred by the Conflict Manager Unit and need human judgment to remove the conflict. The Gamification system is able to suggest the logic that can be used in the design and development of new GWAPs that are well suited to solve the specified task or to suggest a game genre that can be used to inject the task as part of the gaming experience. After selecting the task for conversion, the task is converted to a game either by gamification of the task itself or by injecting the task into existing provided games with minor adjustments. By then, the gaming framework is used to manage the game with the use of its components.

CUbRIK First GWAP and Implicit User Information Techniques

Page 4

D3.1 Version 1.0

These components include: - Reward System o Responsible for inducing motivation and increasing participation of the users.

- Visualization Components o This includes Graphical User Interfaces (GUI) and their supporting components.

- Player Management o Responsible for registration, management, authentication and authorization of players within the CUbRIK’s platform.

- Validation System: o Responsible for confirming the results of user inputs with other available resources.

- Gameplay: o It maintains a list of running games, active users and checked out task instances and it uses this information to orchestrate the results and to assign suitable players to running gaming sessions.

Figure 2: GWAP and Gaming Framework Architecture.

CUbRIK First GWAP and Implicit User Information Techniques

Page 5

D3.1 Version 1.0

2.

GWAP: A Survey of Related Work

In this section the main structural template of games is introduced which include output agreement games, inversion problem games and input agreement games. A review of the existing GWAP base frameworks which include games for object annotation, image segmentation, gesture-based image tagging and image rating is then proposed. A review of achievements systems in gaming is described and how they can help to increase the number of gamers in CUbRIK and keep them interested to the platform is shown.

As a follow-up to this section, three games are introduced in sections 4, 5 and 6. The first two solve the problem of recognising the semantic contents of the images with the help of the users, while the third game acts as a data validator. These games are included in the Human Annotation and Conflict resolution apps units of the CUbRIK platform in Figure 1 and are used for Task Injection in Figure 2. These games are: •

GWAP Framework for image annotation;

•

GWAP Framework for image Sketchness;

• GWAP Framework for metadata verification: Crosswords.

The first game is classified as an Output Agreement game, the second as an Inversion Problem game, while the third is classified as a single player game.

2.1

Main Structural Game Mechanics

In [Lvon2008] three game structural game mechanics are listed to generalize successful instances of human computation games; output agreement games, inversion problem games and input-agreement games.

• Output agreement games Output agreement games [King2009] are a generalization of the ESP game. Here, two players are chosen randomly among a large group of players and will be given the same content as the input. Players are asked to provide outputs based on the given input. In Figure 3 an example of the output-agreement game is shown.

Figure 3: Output agreement game mechanism.

CUbRIK First GWAP and Implicit User Information Techniques

Page 6

D3.1 Version 1.0

• Inversion-Problem games Inversion-Problem games [King2009] choose players randomly from a large set of players.

Here, one player is assigned the role of “describer” and the other player is assigned as the role of “guesser”. The game chooses the input content and gives it to the describer. The describer produces output (in many games a single word or sentence) based on this input.

The objective of the describer is to help the guessers to produce the original input. In Figure 4 an example of the inversion-problem is shown.

Figure 4: Inversion problem game mechanism • Input-agreement games Input-agreement games [Law2009] choose players randomly. In each round, both players are given inputs that are to be the same or different, known by the game itself but not by the players. The players are told to describe their inputs, so their partners are able to assess whether their inputs are the same or different. Both players In Figure 5 the mechanism of the input-agreement game is shown.

Figure 5: Input agreement game mechanism.

CUbRIK First GWAP and Implicit User Information Techniques

Page 7

D3.1 Version 1.0

2.2

Existing game-base frameworks

This subsection reviews some well-known GWAPs played by thousands of people. It mainly focuses on the games that are developed for image annotation purposes which are one of the main tasks of the GWAP unit in the CUbRIK platform.

GWAP (Image Tagging)

Object Wise

Segmenting

Scene wise

Likeness

ESP

Squigl

Phetch

Matching

Figure 6: Overview of “GWAPs”.

Figure 6 shows an overview of media annotation, mainly image tagging. It is divided in four main sections as follow:

2.2.1

Games for object annotation

There are a number of games introduced in the literature for object wise image annotation.

Among them, the ESP is the first and the most popular game that annotates images based on human perception. This game was introduced in 2003 and was played by 13,630 individuals [Ahn2004] within the first four months. The game is designed using a java applet and the applet is connected to a main server for the purposes of data handling and monitoring.

ESP The ESP game is designed to be played by two partners and is meant to be played online by a large number of pairs at once. Players are randomly selected from among all the people playing the game and are not told who their partners are, nor are they allowed to communicate with them. The only thing partners have in common is an image they can both see. From the player’s point of view, the goal of the ESP game is to guess what their partner is typing for each image. Once both players have typed the same key-word or string, they move on to the next image. Here, both players do not have to type the string at the same time, but each must type the same string at some point while the image is on the screen.

Every time the players agree on an image, they will be rewarded with a certain number of points, encouraging them to play more.

This game uses numerous techniques to prevent cheating. The IP addresses of players are recorded and allocated differently from that of their partner to make it difficult for players to be paired with themselves. To prevent global agreement of a strategy such as typing ‘a’ for every image, the game uses pre-recorded game-play. If a massive agreement strategy is detected, the game insets a large number of bots to make it harder for cheating.

KissKissBan (KKB) KissKissBan (KKB), for image annotation, is a different game from other human computation games. Here, the game is designed to be played by three online players. One of the players is called the “blocker” and the other two players are called the “couple”. With the same image presented, the couples try to match (Kiss) with each other by typing the same word and the

CUbRIK First GWAP and Implicit User Information Techniques

Page 8

D3.1 Version 1.0

blocker tries to stop couples from matching (Ban). The blocker is only given seven seconds to act in each round and he/she is able to see every word the couples are typing during the game.

Monitoring the actions of the couples not only makes the waiting process fun, but provides the blocker with an opportunity to stop the couples from achieving some unified strategy. For example, the blocker could give “a” as the blocked word if he/she found the couples trying to match on “a” in every round. The objective of the couples is to guess what the partner is typing. However, unlike the players in the ESP Game, the couples in KKB cannot see what the blocked words are. Therefore, the couples are encouraged to guess harder words to avoid guessing the word in the blocked words list.

Annotation by key-sentences Image describing by key-sentence is another method to tackle the semantic gap problem. In [Tuulos2007] and [Ahn2007], authors have addressed the benefits of this initiative by introducing 2 different games, namely Manhattan Story Mashup (MSM) and Phetch.

Manhattan Story Mashup Manhattan Story Mashup is a large-scale pervasive game, which combines the web, mobile phones and one of the world’s largest public displays in Times Square. Here, the web players used a storytelling tool at the game website to mash up stories, either by writing new sentences or by re-using already given sentences. A noun from each new sentence was sent to a street player for illustration. The street player had to shoot a photo which represents the word within 90 seconds. The photo was then sent to two other street players who had to guess what the photo depicts amongst four nouns, including the correct one. If the photonoun pair was guessed correctly, the original sentence was illustrated with the new photo and it was turned into an ingredient for new stories. Here, players will be rewarded by displaying the best story on the Reuters Sign in Times Square in real-time. This game was deployed as a part of SensorPlanet project at Nokia Research Centre to examine the player’s creativity by exploiting ambiguity and how the players were engaged in a fast-paced competition.

Phetch Phetch is an online game played by three to five players. Initially, the game chooses one of the players as the “Describer” while the others are “Seekers.” The Describer is given an image and helps the Seekers find it by textually describing it. Only the Describer can see the image and communication is one-sided: the Describer can broadcast a description to the Seekers but they cannot communicate back. Given the Describer’s text, the Seekers can find the image using an image database which contains a large number of images. However, they are not cued as to how to extract a search query from the given text. The first Seeker to find the image obtains points and becomes the Describer for the next round. The Describer also gains points if the image is found. Unthinkingly, by observing the Describer’s text, a collection of natural language descriptions of images are obtained. Here, the main disadvantage is that the Describer’s text could contain unrelated textual descriptions, which is being posted among the related descriptions to generate false annotations.

2.2.2

Games for image segmentation

PeekaBoom and Squigl are two different games introduced in the literature for image segmentation. These games are designed to be played by two players that are randomly chosen.

PeekaBoom PeekaBoom, players are named as ‘Peek’ and ‘Boom’. Initially, Peek starts out with a blank screen, while Boom starts with an image and a word related to it. The goal of the game is for Boom to reveal parts of the image to Peek so that Peek can guess the associated word.

Boom reveals circular areas of the image by clicking. A click reveals an area with a 20-pixel radius. Peek, on the other hand, can enter guesses of what Boom’s word is. Boom can see Peek’s guesses and can indicate whether they are hot or cold. For example, if the image CUbRIK First GWAP and Implicit User Information Techniques

Page 9

D3.1 Version 1.0

contains a car and a dog and the word associated to the image is “dog,” then Boom will reveal only those parts of the image that contain the dog. Thus, given an image-word pair, data from the game yield the area of the image pertaining to the word. If Peek managed to guess the correct key-word, both players will be given some points that encourage them to play further.

Squigl Squigl is another type of GWAP introduced for image segmentation. This game is designed to be played by two players, where players are given the same image associated with a keyword. Here, players are supposed to draw the contour of the key-object. Based on both player outcomes, the similarities are analysed by the framework. Depending on the similarity, players are assigned game points that encourage them to play the game further. The main purpose of this game is to generate a database of segmented objects that can be used in machine learning.

2.2.3

Games for Gesture based Image Tagging

The ASAA is the first game designed for gesture based image annotation. The game consists of a combination of manual and automatic image annotation, with interaction by means of gestural signs in front of a camera. Here, the game interface provides a three dimensional game, where people move tags and images, using a motion detection algorithm applied to the captured (user) image. The ASAA game uses semantic image annotation by means of a set of concepts previously trained for image classification. This information is used to calculate the score and the annotated images are used to refine the semantic concept models.

2.2.4

Games for image rating

Matchin [Hacker2009] is another GWAP used to annotate images based on the likeness. This game is a two player model that gives both players two images for voting. If the players vote for the same image, they will be given some points encouraging further engagement in gaming. The objective of this game is to create a large database of images based on image likenesses. In [Lee2008] another approach is introduced for rating people based on pictures.

The approach is called “HOT or NOT” which is a social entertainment website launched in the year 2000 and has been successfully subscribed by millions of members. Figure 7 shows the screen shots of some of the mentioned GWAPs.

Figure 7: Visual representation of current GWAP approaches.

CUbRIK First GWAP and Implicit User Information Techniques

Page 10

D3.1 Version 1.0

3. CUbRIK gaming framework and initial game instantiations In today’s gaming world, the word “Achievements”, even if rooted in the gaming history, has become extremely popular. The spread of broadband connections and the introduction of multiplayer interactions as core components of a videogame have brought to life to a number of social platforms like Xbox Live!, Playstation Network, Steam and Kongregate, in which the players can track their progress along different game titles and compete among each other.

Unfortunately, even if such platforms share similar features, the way in which they manage the aspects of user profiling and statistic tracking is different, leaving the architectural and development aspects of an achievement system tied to the implementation of each vendor.

Achievement is a word that has become main stream in the gaming field in recent years. The term refers to a task that the player has to complete in order to obtain recognition for his effort and thus “unlock the achievement”. This notion is encountered in several heterogeneous systems under different synonyms, like achievements, badges, trophies, challenges, rewards.

The growing popularity of the achievement concept is witnessed by the fact that it is difficult nowadays to find a game that has not some kind of reward or motivation mechanism; even in gamification [Deterding2011], that is the use of game design techniques and game mechanics to enhance non-game contexts, achievements cover a fundamental role and are employed as a way to retain customers or improve learning [Evans2011]. Despite the growing popularity in practical gaming, the literature on game design has paid little attention to achievements, even if there is a general consensus that their proper design is core when driving gamers through their digital game experiences.

In the following, an insight about achievements and their design will be provided, along with a description of the architecture of an open achievement system able to describe all the current commercial systems available. An achievement system is under development as a component of the Reward System of the CUbRIK’s Gaming Framework. The Reward System is designed to increase the participation and engagement of CUbRIK’s users with the use of incentive mechanisms already well-established for games and gamified applications. The Reward system, besides its motivational features, allows also gathering the expertise of players in a formalized and persistent way and rendering it explicitly available for assigning tasks to the most suited users.

3.1

Achievement definition and design

Achievements are now so popular in the gaming culture that the reasons for which they have been introduced are often overlooked; however to make a reward system effective, it is necessary to keep in mind the purpose for which it has been developed. As stated by Björk and Holopainen [Björk2005], “Games do not work without incentives for the players to perform actions and to strive towards their goals”, while Juul [Juul2010] claims that “Players play for personal goals, are aware of the goals of other players, and the shared understanding of intentionality makes game actions socially meaningful”.

Before describing the topic in deeper details, it is necessary to provide some definitions useful to clarify the context: An Achievement is a set of tasks, defined by a designer, for the player to fulfill so to achieve a milestone and track the progress in a system.

Achievements range from simple actions that the player would do anyway, as common gameplay actions, to more difficult challenges even against other players, to recognition for sharing contents among a community.

A Badge is an artifact associated to the completion of an achievement and given to a player after its completion, or, in gaming terms, after “unlocking the achievement”.

An Achievement System, also called Reward System, is a component of an entertainment CUbRIK First GWAP and Implicit User Information Techniques

Page 11

D3.1 Version 1.0

platform used to offer, present, manage and share achievements, at a global scale and across multiple games or entertainment systems. It offers the developers a set of functionalities and APIs to define gaming tasks that can be converted into achievements for their games; it also offers to players a custom statistical information panel that summarizes their gaming history, also called player dossier, which records everything they have accomplished across the games they have played.

An achievement is usually defined by four components: •

Title: The title is a unique identifier used to suggest a theme or hint the player about the action he/she is expected to perform.

•

Icon: the icon is a visual representation of the badge that can be obtained after completing an achievement. An icon is usually an evocative, self-descriptive image that can hint the player on the actions to be performed or be sufficiently attractive to create interest among the other players of a community. The icons for an achievement can appear just after the completion of it or they can come in two different versions: a grey version is used when the player has not yet completed the achievement and a colour version is used when the player has obtained the corresponding badge.

•

Description: the description is used to describe the conditions that must hold in order for the player to complete an achievement or it may just provide hints about a possible action that can be performed in the game. It may also be used to provide information regarding possible rewards upon the completion of the achievement.

•

Points: the achievement’s difficulty may be measured through points assigned to the player upon the completion of the related achievement. Accumulated points can then be used as a measure of the effort or the ability of a specific player by displaying them in his or her player dossier. Not all the achievement system makes use of points.

Figure 8: Achievement for the game: Gears of War 2 (Epic Games) with highlighted components Figure 8 shows an example of these components for an achievement in the Xbox Live!

Achievement system. Every achievement must also have one or more completion criteria that can be defined through event-condition-action (ECA) rules [Ceri1997].

The event may be a player action, a system event, the occurrence of a specific condition of the gamestate or a combination of the three that may trigger the achievement completion.

The condition is the set of pre-requirements on the present state or on past actions that must hold in order for the completion of the achievement to be attributable.

The action is the unlocking of the achievement, which entails the generation of a badge for the user that has completed the achievement and the assignment of digital or real world rewards. The acquired badges implicitly store valuable user information that can be used for profiling purposes, such as his favourite games and genres, his mastered skills and past gaming history.

Achievements allow others to recognize what the player has attained and enhance games by providing lasting rewards. This leads to a sense of affirmation given by the fictional status that the player has created for himself and the expectation that others will look with admiration someone who has undertaken the action stated in the achievement.

The gamer score is a synthetic mean for quantifying a player's skill. While the obtained CUbRIK First GWAP and Implicit User Information Techniques

Page 12

D3.1 Version 1.0

badges can represent the specific game mechanics that a player has been able to master, the numerical score is an immediate and recognizable indication of the gamer's experience.

The last component needed to a fully operational achievement system is a statistical information dossier about the player. In recent years, games are rarely played in isolation, as players often discuss online their mastery of the game, including any goals they have completed. A detailed dossier of the gaming history of a player, including the game he/she has played, the badges he/she has obtained, the level he/she has achieved and his/her score along with a friend list and the social gaming groups he/she is subscribed to is therefore crucial, because it is the feature that enables the social value of the game rewards.

The design of achievements in a game is an aspect often overlooked by game designers but is one of the key factors that are needed to motivate a player. In the following, meaningful guidelines to be followed when designing achievements are provided.

Achievements Design Guidelines In the following there are some guidelines presented on achievements design based on the best practices provided by Greg McClanahan, achievement designer at Kongregate, in [McClanahan2009]. They can be used to avoid designing achievements that are not appealing for the players or those are not meaningful for the gaming experience.

•

A player is motivated by the need of completing all the achievements that the game has to offer, thus he/she will always try to use the most efficient method to earn them. An achievement has to be designed by evaluating this strategy to avoid creating a repetitive and alienating task.

•

If several achievements have been designed to reward ending the game at different difficulty levels, it is good practice to acknowledge the highest difficulty at which something has been accomplished, plus all the implied easier achievements.

•

Achievements should always be earnable without compromising the game progression; a player should not be forced to restart the entire game from the beginning just because he/she has missed an achievement.

•

Achievements hints must be findable; players must be able to get know that the game contains secret features or side quests and be accompanied in their exploration of the gameplay to get to them.

•

Unlikely situations that can happen during the gameplay should be rewarded with a special badge in order for the player to remember the moment. These tasks have not to be random or too difficult to be obtained; otherwise committed players will just spend hours trying to achieve that particular result.

•

It is better to avoid designing achievements that reward getting the highest ranking among a too large group of people, reaching a top spot on some leader board or winning a tournament. Only few selected or committed players will be able to obtain them, while the others will feel like they will not be able to complete the game with the risk of abandoning it.

•

It is better to avoid designing “hidden” achievements that do not state in a clear way which tasks the player has to perform: achievements are well stated goals that the player has to reach; if the tasks are not given the player will just obtain an achievement by chance.

•

Awarding badges upon failures, such as losing a certain number of matches or suffering a brutal death, has to be avoided. Players are not satisfied when losing or being considered low skilled; remarking this fact in their gaming history may have a negative effect.

•

Repetitive tasks should fit with an action that the player would naturally do anyway.

If completing a specific task in the game would be considered as a normal behaviour even without an achievement attached to it, then an achievement can be designed around it. Otherwise the achievement forces a player to change his behaviour, which goes against the principle that an achievement should contribute to a better experience for the player.

CUbRIK First GWAP and Implicit User Information Techniques

Page 13

D3.1 Version 1.0

3.2

Open achievement framework

In this section a platform-independent architectural model is proposed for the development of an achievement system. The model of the achievement system is a component of the more general architecture model for games. The game architecture model is illustrated in Figure 9.

The Gameplay Foundations sub-system includes components of the game engine used to implement the game mechanics and logic of a game. The Front End represents the interface for the players. The architecture is completed with Game-Specific Subsystems, which embody functionalities that depend on the specific game, and with the Online Multiplayer Management sub-system, which, if present, controls the synchronization of multi-player games. The player interacts with the Game-Specific sub-system, thus modifying the internal status of the game; he/she can also interact with the Front End by navigating menus and requesting information regarding his gaming history.

In this model, the Achievement System is a cross-cutting module connected with all the other main sub-systems. Its goal is to receive Gameplay Events from a running game played by a specific user, process them and return as output the updated gaming history data for that player, including the badges he/she may have acquired and an updated profile.

Gameplay Events represent the occurrence of meaningful game states, for instance reaching a specified number of collected objects during a gameplay session. Once the requirements for the completion of an achievement are reached, the achievement system signals the change to the Game Specific Flow System and to the High Level Game Flow System of the engine if there are consequences within the gameplay such as rewarding a player with an ingame item.

Figure 9: Gaming Architecture with Achievement System Component The signal is also sent to the Front End, to inform the user about the reward he/she has obtained and about his new gaming history report, and to the Online Multiplayer Management components, if present, in order to enhance the interaction by pairing players with same skills together, encouraging contacts from people with the same tastes and other social features.

An Achievement System is structured into the components and data flows shown in Figure 10

CUbRIK First GWAP and Implicit User Information Techniques

Page 14

D3.1 Version 1.0

Figure 10: Architectural model for an Achievement System •

The Action Detection module filters raw events and returns only the meaningful achievement actions. The Action Detection performs the filtering and composition of the events through the use of Achievement Action Patterns, i.e., expressions defined by the game developer denoting the sequence of events that trigger the notification of an action meaningful for the unlocking of the achievement.

•

Achievement actions patterns can be simple predicates selecting events of a specific type, e.g., “Level Completion”, or more complex expressions, that detect an action based on a sequence of gameplay events, such as “Find the key, open the chest and return the crown to the King”.

•

Achievement actions are processed by a GameStat Updater Module, which tracks and persists the monitored actions as Gameplay Statistics.

Achievement actions and player statistics are the input to the Achievement Detector, which checks if the required conditions for a particular achievement are met, assigns the associated badge to the player and outputs the updated profile data regarding him.

Figure 11 illustrates a possible schema of the database supporting game data persistence.

Game is the core entity: It has a Mode attribute that represents the gameplay modes (Single Player, Multi Player, and Cooperative), while the Genre attribute identifies its genre (e.g.

Puzzle, Educational).

An Achievement has an Icon, which describes it in a visual way, a Category that specifies the task (Instructor, Grinder), an attribute PointsGiven, which contains the amount of points to be granted, and a Boolean attribute OfTheDay defining whether the achievement has to be completed on a specific day in order to obtain virtual goods, more points, or increased levels.

The Player entity accommodates game-specific personal and social features. Avatar and Nickname allow the user to be recognizable by using a custom image or a unique fictional name, while Motto, Biography and GamingRig convey customization.

To enable social interactions, attributes like Friends and Fans keep track of the players the user likes to play with and the players that appreciate the user’s performance, while the Status attribute denotes if the player is online, offline, occupied or the game he is playing.

Reputation in online gaming communities is fundamental and a distinctive feature of any player: being able to recognize whether a player is bad-mannered, prone to cheating, or unpleasant to play with is of utter importance to assure a satisfying gaming experience for the user of an entertainment platform. It is usually measured as an integer number ranging from 0 to 5.

•

CUbRIK First GWAP and Implicit User Information Techniques

Page 15

D3.1 Version 1.0

Figure 11 UML Class Diagram for an Achievement System ScreenShot and Videos show salient moments worth sharing with the community. A player can also choose his FavouriteGame.

The model describes also the game-relevant statistics: the Level represents the proficiency and the experience of a player, by aggregating in a compact way such indicators as points gathered, hours spent playing or particularly difficult tasks completed.

The Points attribute stores all the points that have been achieved in all the games.

ObtainedBadges represent the achievements that have been unlocked. PlayerTitle is a special recognition given to the player for his actions, like a chivalry role, while the PlayerType (e.g. Achievers, Explorers, etc.) associates the player with a particular behavioural category.

A GameBadge relates a Player with the Achievement he/she has obtained. The CompletionPercentage field shows how much the player has already achieved in a specific task. StartDate and EndDate record the dates in which the player has started to work on the achievement's goals and the date in which he/she obtained it. The TrialsN attribute tracks how many times the user tried to fulfil the achievement. The GameStats relationship denotes the meaningful statistics that the developer has designed for a game, for example the HoursPlayed by a Player or the Score he/she has obtained. Finally, a GamePlayAction, associated with a specific Gameplay of a player, records the StartDate and EndDate of the gaming session and the actual actions performed by the player on that specific time frame and the Role defines which are the allowed actions in the game for the role associated to a player.

CUbRIK First GWAP and Implicit User Information Techniques

Page 16

D3.1 Version 1.0

4.

GWAP Framework for Image Annotation

This section describes the design and implementation of a framework combining several paradigms for archiving more realistic image annotation. The proposed framework is a standalone game (single player), used to annotate images purely based on the players intention. Throughout the whole gaming experience the client side (user side) communicates with CUbRIK servers to provide the best possible solutions for the problem tasks. This is to evaluate the user’s truthfulness level based on their historical inputs and provide the appropriate image classes (namely fully annotated, partially annotated and non-annotated) to the user. The underlying approach is designed based on a two-player game model where the user is competing with their history. It gives the independence to combine a number of different paradigms, such as player outcome prediction algorithms, Game theory based decision making concepts and the players overall contribution in annotation. In this section, a novel approach for image annotation based on Game Theory (GT) is presented. GT and its driven mathematical models are introduced to make decisions on the player’s outcome, i.e. to accept or to reject the player’s annotation.

The proposed approach follows the well-known crowdsourcing paradigm, in which a given problem is tackled by exploiting the power of users in a widely distributed way. In our case, the aim is to harvest the power of millions of computer gamers for the purpose of annotation of digital multimedia as in Flickr1, Facebook2 and Dailybooth3.

Crowdsourcing has been successfully used for harvesting multimedia annotations. A commonality of all these use at least two-players interacting remotely through the Internet so as to prevent cheating and control a potential flow of misleading annotations into the metadata base. A more critical issue related to cheating prevention in ESP-like games is the latent possibility of remote gamers agreeing on a strategy that can be used to provide quick useless annotations but yet obtaining high scores in the game. In contrast to ESP-like strategies, the approach proposed here can be instantiated as a standalone game or be deployed over the internet as well.

The proposed framework can be initiated by single standalone games and is based on a twoplayer model. In this model, the gamer (user) takes the role of Player 1 while the machine takes the role of Player 2. The underpinning model considers two different types of gamers: rationally minded and malicious or deceptive players. It uses an outcome prediction mechanism to expose the player to the most suitable multimedia material, i.e. fully annotated (images that are fully annotated by a paid human operator), partially annotated (images that have obtained one or more annotations) or non-annotated (images that have no annotations at all) contents.

4.1

System Overview

A diagrammatic overview of the proposed approach is given by Figure 12. The system relies on a small number of previously annotated images and a transitional database for storing partially annotated images. That is, the entire image database consists of three subsets: fully annotated, partially annotated and non-annotated images. Initially, the fully annotated subset would consist of a small number of images previously annotated by a human operator and the set of partially annotated images is empty. Once the game is deployed, it is expected that both fully and partially annotated image sets start to grow as semantic metadata is obtained through the game. Thus, the complete framework comprises two main modules. The first module (right in Figure 12) handles fully-annotated multimedia units, while the second module (left in Figure 12) deals with partially and non-annotated multimedia units. The first

1 http://www.flickr.com/ 2 http://www.facebook.com/ 3 http://dailybooth.com/ CUbRIK First GWAP and Implicit User Information Techniques

Page 17

D3.1 Version 1.0

module is used to understand the player’s behaviour, confirm results from statistical inference, as well as estimate model parameters and the shape of its payoff functions. The second module is the actual annotation engine providing semantic metadata for nonannotated content.

Figure 12: A complete block diagram of the framework.

In our case, two generic types of gamers are considered: rationally minded and malicious or deceptive players. The first type of player plays the game in a fair way trying to achieve high scores by correctly annotating content. This type of player is called “rational” in the sequel.

The second type of player contains all those who try to achieve high scores by cutting corners and cheating. These types of player are called “malicious” in the sequel. Clearly, there will be users that change behaviour while playing. Thus, the system assumes that a rational player can become malicious and vice-versa. At the start, a small set of fully annotated content is fed to the game to initiate the process of learning player’s behaviour and model parameters. Here, a transition matrix (the one used by the proposed Markov model prediction) is used to measure player’s contributions to the game, i.e. the player’s potential to provide correct and meaningful annotations; and the player’s cost, i.e. the effect of incorrect or misleading annotations. Next, content is extracted from one of the three available databases (fully, partially or non-annotated) and uploaded into the system.

Database selection for content extraction depends on the predicted player’s behaviour as detailed in 4.3. In subsequent steps, this prediction is done by taking into account the previous outcomes of the player for a series of fully annotated contents. When the player prediction module expects an incorrect annotation with high probability, then it exposes a fully annotated unit to the player. On the other hand, when it predicts a valid annotation with high probability, it loads a partially annotated or a non-annotated piece of content, based on the outcome of equilibrium analysis module. However, a module referred to as a Random Content Selection module that forces extraction of content from the fully annotated multimedia database at random time intervals is also used. Given that, in practice, players change their behaviour often and rational-minded players could thus become malicious, the Random Content Selection module addresses this problem by exposing the player to a number of fully annotated contents at random time intervals. The outcomes for these images are used to update the state of the MM, with the aim to assist MM in representing the player’s latest behaviour in gaming.

CUbRIK First GWAP and Implicit User Information Techniques

Page 18

D3.1 Version 1.0

The visual interface is the window of the instantiated game. Its design depends on the game strategy. However, it has two fundamental tasks: to expose content to the player and to enable input of character or strings to be associated with the exposed content. Two different game strategies are described in Section 4.5, to illustrate the corresponding visual interfaces and test the performance of the proposed model.

Although Figure 12 shows that the proposed framework is developed using a number of modules, the entire process can be summarised using two major units, namely, the Payoff Calculation and Decision making unit and the Player Outcome Prediction unit.

4.2

Payoff Calculation and Decision making Unit

Payoff calculation and decision making is one of the important units in this framework. This unit mainly relies on the player’s outcome, i.e. accuracy of the player annotations. The fundamental algorithm is designed to measure the player’s contribution in gaming in order to expose them to the most suitable, i.e. fully annotated, partially annotated or non-annotated content, as well as to decide whether to accept or to decline the player’s outcome. In order to do so, two payoff functions are constructed and represent both players’ contribution in gaming. In the beginning of each game, a player will be exposed to a number of fully annotated images. Outcomes are then used to form a transition matrix. This matrix is used to measure the Player 1’s overall contribution in gaming as well as the cost. Here, player’s overall payoff is measured by subtracting the player’s bad contribution from the good one.

Player 2 in this game is a virtual player and therefore his contribution is measured based on a number of different aspects. Here, the payoff function used to measure Player 2’s payoff is designed to aggregate number of different key factors; Player 1’s payoff, previously recorded players contributions and image classification outcomes. Since, in this game, players are not fully independent, and given that the objective of the machine (that takes the role of Player 2) is to encourage Player 1 to produce correct annotations, it is fair to use Player 1’s information, i.e. payoff or any other information to measure the Player 2’s payoff. More formally, if the machine motivates Player 1, it can be assumed that the probability of entering a previously recorded annotation by Player 1 would increase. This further confirms the suitability of using Player 1’s information to assign correct weights the Player 2’s payoff. In addition, the Player 2’s payoff is weighted by a SVM classifier, whereby the classifier selects the most optimal trained concept from a set of pre-trained concepts based on the player’s input keyword. The probabilistic outcome (the probability of an image being relevant to the trained concept) from the classifier is used as a factor for weighting the Player 2’s payoff.

For partially annotated and non-annotated contents, the Player 2’s cost is calculated based on the number of different annotations that have been obtained by an image. In practice, if the framework performs well, annotations from cheating oriented players will be recognised.

As a result, the framework would accept a few different annotations, i.e. those from trustworthy players. Thus, using the number of different annotations assigned to an image for calculating the Player 2’s cost is the most optimal solution.

The proposed framework uses game theories, in particular Nash Equilibrium based decision making techniques for exposing the player to the most suitable image content, i.e. fully annotated or non-annotated. Since game theories postulate that decisions should be based on primitive actions, two different primitive, yet influential, game actions are introduced to the system. One of these actions represents the short-term contribution of the Player 1 in gaming and the other action represents the long-term contribution of Player 2 in gaming.

4.3

Player Outcome Prediction Unit

Player outcome prediction is the second most important unit in this framework. It is used to predict the player’s outcome prior to exposing images, fully annotated, partially annotated or non-annotated. As a result, it is used to improve performance of this framework. In here, two different outcome prediction algorithms are introduced. One is based on the well-known Markov chains and the other one is based on Sequential Sampling plans. Since human CUbRIK First GWAP and Implicit User Information Techniques

Page 19

D3.1 Version 1.0

outcomes are dynamic and do not follow any sequence, it can say that prediction based on present intention is the most practical approach to predict future outcomes. Here, it says that there is a high potential that human behaviour depends on current intention and is not based on past performances. Since Markov chains predict future events based on the outcome of the present event, Markov chains are used to predict human outcomes. To compare and evaluate the Markov prediction approach, we also used sampling algorithms to predict human outcomes, in particular Sequential Sampling, where the prediction and decision making is influenced by examining the entire distribution, not only based on the present outcome.

Unlike the Markov approach, SS is well known and the involved risk of accepting a defective sample is what makes it admired when compared to the Markov approaches.

4.4

Implementation of the framework

The main goal of the framework is to satisfy a large number of game players, thus could obtain a large number of valid annotations for a given set of images. As a result, the following major features were constructed in devising the process: •

An easy to use and attractive human-machine interface.

•

Support for the storage of metadata.

• An easily adopted method for annotating various type of multimedia content.

Currently backbone structure and basic modules of the targeted system have been implemented. Based on these, an experimental environment has been composed for image annotation. The proposed framework is mainly implemented using C++. Implementation of GUI (Graphical User Interface) employs the OpenGL Application Programmable Interface (API) development environments. This API is the interface implemented for the game application which allows the other applications to communicate with it. OpenGL is an open source toolkit designed to provide efficient, portable access to the user interface facilitated by the operating systems on which it is implemented. It is a premier environment for developing portable, interactive 2D/3D graphic applications. OpenGL has become the industry’s most widely used and supported 2D and 3D API. OpenGL fosters innovation and speeds application development by incorporating a broad set of rendering texture mapping, special effects and other powerful visualization features.

4.5

Graphical User Interface

For testing purposes a graphical interface was developed, Figure 13 as a temporary interface. This was to verify the output of the developed backend algorithms for trusting a user hence accepting their annotation.

This interface was based on a design of a simple game scenario. Players were asked to annotate images by typing a key word. Here, the image subject for annotation were randomly displayed in one of 6 displays and the player was asked to steer himself towards the image and enter a keyword. In the case of a player bumping into given obstacles or unable to complete annotations in a given time frame, they would be given a life penalty.

CUbRIK First GWAP and Implicit User Information Techniques

Page 20

D3.1 Version 1.0

Figure 13: Sample screenshots of the developed temporary game.

Figure 14: Steam Pilots game’s entry form.

At the current stage a new game, called Steam Pilots, for image annotation is developed which is a light weight web application and is accessible via all of the popular web browsers.

This gives the application the benefit of attracting high volumes of crowd to play the game hence providing large number of annotation.

In this game first a player can sign in or register through a provided form, Figure 14, by entering their selected username and password. Then if they have history in the CUbRIK’s servers their profile is retrieved and loaded to the game client running on their web browser.

Next, the users have the options to play the game, view their profile or view the leader board, Figure 15. The player’s profile includes their highest score in this game so far, their ranking in the leader board and their achievements. Also the leader board option displays the players with highest score amongst others.

If the players choose to play the game, they can select between different balloons as their air ship with different properties such as strong, fast, etc. At the start of the game the basic instructions are given to the player about how to control their air ship. The goal of each level of the game is to find a golden key to open the door to the next level. The golden key itself is locked in a box and needs silver keys to become accessible. To find all of the required keys the players have to explore the game arena and pass through different hurdles.

CUbRIK First GWAP and Implicit User Information Techniques

Page 21

D3.1 Version 1.0

Figure 15: Steam Pilots’ information page.

Figure 16: Steam Pilots’ gameplay screenshots .

CUbRIK First GWAP and Implicit User Information Techniques

Page 22

D3.1 Version 1.0

At different stages of the game a door opens that puts an image on display. A text field with a send button is provided under the door by which the player should annotate the displayed image. The entered text is sent to the CUbRIK’s servers and it is processed by the backend algorithms which are described earlier in this report. Based on the results, the player’s score is updated and it is decided what type of image (annotated, non-annotated or semiannotated) should be displayed to the player for the next stage. Finally the selected image and the new player’s score are sent back to the game’s client on the player’s machine to proceed with the game. Figure 16 shows the sample screenshots of the developed Steam Pilots game.

4.6

Experimental Setups

A general experimental environment is constructed by implementing frameworks as mentioned earlier. It contains different setups for specific experiments on proposed algorithms. Since this framework depends on human players, we used a number of different players to evaluate the framework. Moreover, we used three natural image databases for these experiments, namely, ESP, Caltech and Corel image datasets. Details of these datasets are given as follows.

ESP dataset The first dataset is a small set containing 200 images selected from the ESP dataset, which is referred as the ESP dataset. Here, manual labelling of the ground-truth for 100 images were conducted prior the experiments. These images contain complex scenes and scenarios with large numbers of objects present, such as busy streets, seaside, landscape, office environments etc. Therefore, they cannot be categorised into a particular semantic category.

Since the ground-truth for this dataset is known, player outcomes from these images have been used to measure the player confidence in image annotation.

Caltech 101 dataset The second dataset is a small set containing 200 images selected from the Caltech 101 dataset, which is referred as the Caltech dataset. Here, manual labelling of the ground-truth for 100 images were conducted prior to the experiments. This dataset contains a higher level of ground truth based on semantic meaning. Images belonging to the same class illustrate the same concepts, however, their visual appearance is different. This dataset consisted of 101 object categories which do not overlap with any other concepts.

Corel dataset The third dataset is a small set containing 200 images selected from the Corel dataset, which is referred as the Corel dataset. Here, manual labelling of the ground-truths for 100 images were conducted prior the experiments. This dataset contains a higher level of ground truth based on semantic meaning. Images belonging to the same class illustrate the same concepts, however, their visual appearance is different in practice. The dataset consists of seven concepts, namely, Car, Lion, Tiger, Cloud, Elephant, Building and Vegetation.

CUbRIK First GWAP and Implicit User Information Techniques

Page 23

D3.1 Version 1.0

5.

GWAP for Segmentation: Sketchness

Sketchness is a multiplayer puzzle game in which the players take turns to draw the shapes of objects in a provided image in order to make the other players guess the underlying object.

If the right word is spotted both the drawer and the players that were able to spot the word will receive points based also on the time the response was submitted. After a certain number of rounds in which every player is asked to draw on different images, the winner will be the player which has achieved the highest score. Sketchness is a multiplayer inversion-problem game that takes inspiration from guess-drawing games like the famous board game Pictionary.

Inversion-Problem Games are games in which a player has access to the whole problem and gives hint to the other players to make a guess. If one of the other players is able to guess the secret, it can be assumed that the hints given by the first player are correct. The game consists of 10 rounds where one person is given a word and an underlying image on which to draw the object stated in the word, while the other players will try to guess it within a designated time.

In each round a player is chosen at random to be the Sketcher while all the others will play as Guessers.

Figure 17 an example for the Guesser’s view If the current role for a player is the Guesser, he/she is asked to type his guesses in a provided text input box. The guesses are visible to the other players except when the player have typed a word that is close to the one requested or have guessed correctly. When a player has guessed correctly, his status will be changed (changing the name color in the list of the players) in order to show to everyone that he/she was able to answer correctly. The quicker the player answers correctly, the higher the score he/she will obtain.

If the current role for a player is the Sketcher, he/she will be provided with an image coming from a set of annotated or not annotated fashion images with low confidence.

CUbRIK First GWAP and Implicit User Information Techniques

Page 24

D3.1 Version 1.0

Figure 18 an example for Sketcher's view He/she will be the only player with the rights to see the image, while the image will be hidden to the other players. The sketcher is asked to provide a tag for a garment visible in the image, such as “tie” or “trousers” or he/she will be given a tag generated from previous matches.

Once the tag has been added, the Sketcher is asked to draw the selected object by tracing its contour.

He/she will be able to see the guesses given by the other players and he/she will also be able to skip the drawing of the image object if he/she cannot/does not want to play for that particular image.

The round will end when any of the following occurs: •

Time available to the Sketcher runs out.

•

The Sketcher decides to end the round by clicking skip.

•

The Sketcher doesn't choose the tag within the first 20 seconds.

•

After a set time (usually 20 seconds) following the first correct guess.

•

If enough players press the "Warn Artist" button when the artist violates any drawing rule.

•

If any rule is violated and the artist is skipped over by an administrator.

5.1

Game Structure

5.1.1

Scoring

Sketcher: The artist receives 10 points for the first correct guess. One (1) point for each additional guess is awarded, up to a maximum of 5, and a total earning potential of 15 points per round. In case the image fetched to the sketcher is damaged or somehow unusable, he/she will be able to skip the turn, in this case no points are awarded. If the sketcher decides that the image is usable and starts sketching, he/she will receive a negative score if no one of the other players guesses correctly in the given time.

Guesser: The first guesser is awarded 10 points. The 2nd guesser is awarded 9 points, 3rd guesser gets 8 points and so on, with a minimum of 5 points. The scoring system ensures a big payout for guessing quickly, drawing effectively and getting as many correct guessers as you can.

CUbRIK First GWAP and Implicit User Information Techniques

Page 25

D3.1 Version 1.0

5.1.2

Winning

When all 10 rounds have been played, the player with the highest score will win the game.

5.2

Sketchness: HC Task Analysis

The purpose of the game is to perform segmentation on the images provided as input, which derive from fashion images that has been analyzed by specialized garment detection algorithms but have provided low confidence results. This objective has been achieved by suitably modifying the game mechanics of a well-known and appreciated game category, called “drawing and guessing games”, to implicitly solve segmentation problems while playing. We call the process of hiding human computation tasks beneath existing game mechanics “Task Injection”.

The proposed game differs from the existing drawing and guessing games since, while the traditional game mechanics rely on the imagination of the player to draw the requested subject, in the GWAP that is being developed the player uses an underlying image to draw the contour of the object to be recognized; this is the feature that enable the task solving capabilities of the game.

Players may solve two different tasks during their gameplay: they may tag the provided image by identifying garments and they are asked to segment the picture with the contour of the object stated in the tag. The tag for the image can also be provided as an input by the previous components of CUbRIK, in order to specify the garments or areas of the image that the system has not recognized and thus requiring segmentation. The image itself is an immediate hint for the player that will just need to trace a contour of the tagged object within the image to get a representation good enough for the other players to guess it. The game generates two kinds of annotation as a byproduct of play: the polylines related to the segments traced by the player and bounding boxes identifying the position of the object within the image. While the bounding boxes can be useful to get rid of outliers, the polylines are used to create a punctual representation of the contour of the objects.

The correspondence between the segmented part of the image and the required one is enforced by the answer of the guessers: if they agree with the tag that is available only to the Sketcher, it can be assumed that the contour drawn by the Sketcher truly represents the target object to be segmented in the image.

Considering the fact that the results provided by a single player could not be considered accurate, several output aggregation techniques are being analyzed to understand which technique could validate the results provided by several rounds of play.

In particular, bounding boxes able to enclose the entire segment traced by the player are created for each image and for each game round. Since the image can be considered as a matrix of pixels, we first create an integer matrix of the same dimensions as the original image and we fill it with “0”. The area covered by the bounding box is then identified in this matrix by marking each pixel with “1”.

All the collected bounding boxes for the same image-tag pair are then aggregated by adding their corresponding matrices. This aggregation phase produce a new matrix containing the number of different players that have identified the pixel corresponding to the stated object.

On this combined matrix, we apply a variable threshold X, depending on the quality of the results that we want to obtain and the number of available recorded games; this means that each integer in the combined matrix that has a value less than X (the number of players that have identified the area as meaningful for the object description) will be substituted with 0 and every value greater than X will be untouched.

This will provide a matrix able to identify all the pixels that have been revealed by at least X players. The pixels are then clustered and the new aggregated bounding boxes are generated by taking, for each cluster, the leftmost, rightmost, topmost and bottommost points.

The generation of the bounding boxes is in development while aggregation techniques to be applied on the polylines are still under research.

CUbRIK First GWAP and Implicit User Information Techniques

Page 26

D3.1 Version 1.0

6.

GWAP for metadata verification: Crosswords

Crosswords is a single player game in which the player solves a crossword puzzle. The puzzle is generated on purpose using the metadata which needs to be verified. For every textual piece of metadata that needs to be verified or confirmed a clue-answer pair is generated. This pair is then built in a crossword puzzle. It is possible to built-in more than one pair which needs to be verified into a puzzle, but the total amount should not exceed certain threshold to be established experimentally. The puzzle is solved by multiple players. In the process of solving the puzzle each of the players either solves the puzzle without any problem and therefore confirms the metadata or experiences difficulty in solving the puzzle and stumbles upon that particular piece of metadata which is incorrect and therefore does not allow to complete the puzzle. At this moment the player have a chance to submit error correction feedback.

6.1

Crossword puzzle generation

Full crossword puzzle generation consists of two steps. First, we generate the puzzle grid.

Second, we generate the clues.

To generate the crossword puzzle grid, we follow four basic steps: 1. select a layout, 2. select a slot, 3. pick a word, and 4. check consistency.

Different terms that we use here are: •

Slot: Sequence of cells that together form a word either across or down.

•

Word pattern: A word pattern is a partially filled slot.

•

Compatible word: A word is compatible if it matches the word pattern and it has enough information to generate at least one clue.

•

Consistent Grid: A grid is consistent if there are compatible words for all of its unfilled slots.

Select a Layout We have some predefined layouts to generate crossword puzzles. From that list, we randomly select a layout that matches our expected row number and column counts.

Select a Slot Our target is to select a slot that maximizes our future choices. To select a slot, we use the most constrained slot. This method is used in [Meehan1997]. To do this, we take all slots that can be filled. For each of them, we count number of compatible words in our metadata repository. Then we select a slot that has least compatible words.

Pick a Word After selecting a slot, we select a word to fill it. We want to pick a word for that slot, which increases the probability to choose more words for the rest of the slots. We take first ‘n’ compatible words for the selected slot. Then for each word, we count the number of compatible words in slots that intersect the selected slot. Finally, we pick the word for which we have maximum number of compatible words. Taking first ‘n’ words strategy is used in [Cheng2009] and taking word that has the most compatible words strategy is used in [Meehan1997].

Check Consistency While picking a word for a slot, we pick a word for which the grid is consistent. From first ‘n’ CUbRIK First GWAP and Implicit User Information Techniques

Page 27

D3.1 Version 1.0

compatible words, we select a word, put it in the selected slot, and count compatible words for all slots that intersect the selected slot. If there is an intersecting slot for which there is no compatible word, then the grid is inconsistent for that word and we try for next word. This strategy is followed in [Meehan1997] and [Cheng2009] where they call it arc-consistency.

At the end of this algorithm the puzzle grid is generated. We proceed with generating the clues for the words in the grid.

Clue Generation For each word, namely an entity name coming from a piece of metadata which needs to be verified, that is used to fill a crossword grid, we need a summary about the entity provided the metadata. Summary is not the only technique, other clue techniques can be used, such as “fill in the blank� clues. Crossword generation uses two steps for clue generation. First, we check that sufficient information is available already for an entity to generate a clue and second, we generate the clue based on that information. Each entity in our metadata repository has an associated etype, which is the entity type. There are several dozen etypes available in our repository. However, some of them such as Location, Person and Organization have most instances in our repository. We know which attributes each entity type may have. We use these attributes for clue generation. For example, we use partOf information to generate clue for entities of location etype. PartOf is a relational attribute of location etype and points to another location, which contains the original location. For locations we look for partOf information to generate clues and to check that clue generation is possible for a specific entity.

Check Clue Information For each entity, we first take its etype information. Then based on that etype, we look for its attributes, which can be used to generate clue. If expected attribute is present for that entity, clue generation is possible for that entity. A combination of an entity type, attributes available and a text composing them into a human readable clue makes a clue template.

Figure 19 Clue generation Generate Clue Similar to clue information checking, we take etype information of an entity for which we want to generate clue. Then we extract attributes for that entity based on the entity etype. Finally, we check which templates suit the particular case and we use attributes to generate a clue for the entity using a template. The clue generation scheme is depicted in Figure 19 Clue generation.

CUbRIK First GWAP and Implicit User Information Techniques

Page 28

D3.1 Version 1.0

6.2

Crossword puzzle solving

The generated puzzle is solved using a usual process of solving a crossword puzzle. From the point of view of the player it does not differ from solving any other kind of puzzle. Figure 20 Crossword Puzzle Gameplay depicts such a process in the middle of it.

Figure 20 Crossword Puzzle Gameplay As could be noted, the difference with a standard crossword puzzle is in the “Report error” button, which provides the player with an option to give feedback and correct the metadata.

6.3

Feedback collection

The actual task of metadata verification starts to happen when the player comes to solving the clue generated for the embedded piece of metadata to be verified. The player solves the clue, and fills the answers. At this moment, if the metadata is actually verified, the player fills the correct value in and it fits the grid and the rest of the puzzle. Otherwise, if the player’s knowledge does not confirm the metadata in question, the player cannot fit the answer into the grid: it either does not fit in the slot or does not fit with the rest of the grid. In case of insignificant differences, such as 1 letter spelling mistakes, to exclude to possibility of putting important letters into an unverifiable place, such as an isolated cell (no crosses with other slots), it is possible to embed the same piece metadata into different grids achieving full coverage of the word by complementing grids. For example, grid 1 can cross the particular name at letters 1,3 and 5, while grid 2 crosses the same name to be verified at letters 2 and 4.

When the player is stuck with an impossibility to fit the suspected answer into a designated slot, the player can use the “Report Error” option, “Reveal Letter”, “Reveal Word” or a combination thereof to clear the situation and continue solving the puzzle. The “Reveal Letter”, “Reveal Word” options are provided to keep player satisfaction at high level.

Figure 21 Crossword Puzzle Feedback Mechanism depicts how the feedback (a correction to an erroneous piece of metadata) is provided and handled. The player click “Report Error” button and in a special dialogue provides a correction and a justification for the correction.

The feedback is then aggregated, weighted and applied.

CUbRIK First GWAP and Implicit User Information Techniques

Page 29

D3.1 Version 1.0

Figure 21 Crossword Puzzle Feedback Mechanism

6.4

Media metadata correction

Crosswords might seem to be an exclusively textual game, but it is not. There is a modification of crosswords which is well suited for verification of metadata associated with media is depicted in Figure 22. The picture demonstrates a way to verify metadata for visual and audio data. For these kinds of data the clue is the data itself: a picture, a piece of audio or video. And the answer is again the piece of metadata which needs to be verified. This modification is among planned future extensions.

Figure 22 Crosswords for Media Metadata Verification

CUbRIK First GWAP and Implicit User Information Techniques

Page 30

D3.1 Version 1.0

7.

Implicit User Information Techniques

Thanks to the advances in technology, the population of earth is storing a massive amount of visual information (i.e. image and video) in online databases. Today it is normal for a person to take a photo of some event with his/her smartphone and without any effort upload it to a host computer. For later quick access, the stored data needs to be indexed by providing metadata for the stored content. Location tags and low level features including colour and edge histograms are examples of the metadata that are currently stored along with the visual content. However, there still remains the challenge to provide appropriate captions for the semantics of the images which are not understandable by the machines. As part of the CUbRIK platform, implicit user information, provided by different frameworks, works as a bridge for closing the gap between machine understanding and human understanding. The goal is to find an approach which employs the computational power of machines and integrates it with the precision of human to achieve a fast and accurate system for image annotation and retrieval.

As part of CUbRIK a method is introduced to identify people in a photo collection by considering implicit feedback gained through a dedicated feedback framework.

7.1

Improving People Recognition by Implicit Feedback

One of the use cases of provided implicit feedback is to identify people in a photo collection by considering implicit feedback gained through a dedicated feedback framework.

7.1.1

Introduction to the People Recognition Framework

The interest is in recognizing people by exploiting context - especially relevance feedback.

The application focus is on Consumer Photo Collections, which are collections of personal photos from only one or few related individuals. Such collections might contain large amounts of photos, possibly spanning over multiple years. Examples of the rich context available in such photo collections are interrelationships among depicted people (family, friends, etc.), clothing people wear, scene and location in which people are depicted, date and time photos were captured and events like Wedding Party that photos are associated with.

The initial focus is on recognizing people primarily based on their faces. The development of a framework is being investigated that detects faces and then discriminates these faces.

Instead of a traditional classifier-based approach a graphical model together with a distancebased face description method are envisioned. The graphical model is used primarily to more flexibly incorporate context like social semantics. Most of the processing is done offline. In other words, the retrieval only implies looking up results that are computed beforehand. A reason for this is that the recognition within a single photo considers - and the same time depends on - information from other photos. Another reason is to incorporate implicit feedback that might not be gained immediately.

The people recognition framework is divided into several modules; therefore, it is possible to integrate the framework in steps: •

Detector Module Detects specific parts like faces within a given photo

•

Feature Extractor Module Extracts features from detected parts

•

Recognition Module Performs people recognition offline and stores results

•

Feedback Module Leverages implicit feedback by exchanging information with a dedicated feedback framework, which in turn communicates with clients that present a game to players

CUbRIK First GWAP and Implicit User Information Techniques

Page 31

D3.1 Version 1.0

•

Presentation Module Looks up precompiled results and displays them

7.1.2

Application Context

The people recognition framework fits into several application contexts. For example, its usecases could be demonstrated as an application that tells apart related people in personal photo collections or that identifies popular people (celebrities) in photos gathered from online news. The latter, however, could also be applied to historical archives, for instance to associate depicted people in older newspaper photos that have accompanying captions.

7.1.3

Dataset

Depending on the application context, a dataset in the form of a photo collection shall be used to evaluate the framework. Since we are interested in recognizing people, the photos shall mainly depict people. The recognition framework does however not rely on normalized face shots. Thus, photos taken in uncontrolled environments are fine.

Generally, the recognition framework targets photo collections depicting few individual people in overall. Photos with people that frequently appear together would be of interest to research social semantics. In the latter case, the dataset should preferably come from a single source.

Lastly, any dataset should be accompanied by a ground truth. In particular, such a ground truth should provide for each depicted person in every photo a class labels and some annotations relating to face markings (for example eye-centre).

7.1.4

Technical Contribution

The technical contribution of the people recognition framework is the detection of people (faces), the extraction and pre-processing of features, and the identification of people in photo collections in such a way, that additional contextual cues and implicit feedback is incorporated.

We focus predominately on faces as a mean of detecting and recognizing people. Other contextual cues like time or social semantics are possible as well. A key-characteristic of the researched graph-based recognition framework compared to traditional classifier-based approaches is the support for constraints; in particular, the exclusivity or uniqueness constraint wherein multiple faces in a photo cannot relate to the same individual person. Such a constraint is especially useful for photos that depict groups of people, and can notably increase recognition accuracy.

Lastly, the use and integration of implicit feedback itself will form a major part in the overall contribution.

7.1.5

Incorporating Implicit Feedback: Overview

We envision a web-based user interface that allows a user to browse, query and retrieve people from a photo collection.

For example, a user may wish to retrieve a person by name or select (click) on a person in a gallery overview. In that case, all appearances of the searched person within an imported photo collection shall be presented as a result.

CUbRIK First GWAP and Implicit User Information Techniques

Page 32

D3.1 Version 1.0

To achieve this, an implicit feedback framework is incorporated to help gather relevant training information.

The people recognition application communicates with the server of a feedback framework.

The people recognition framework will send faces, and receive information about this faces in return. The server of the feedback framework distributes the faces to clients, which present a game to players. One possible scenario in this game is that the players walk through a virtual world, and need to interact with the faces displayed to them. The most straightforward question is to ask a player for a person’s name:

A possible extension is to present to the player two faces and ask if these faces are associated with the same person.

CUbRIK First GWAP and Implicit User Information Techniques

Page 33

D3.1 Version 1.0

7.1.6 1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

7.2

Detailed Description of how Implicit Feedback is incorporated A new photo is added by importing it and storing it locally A dedicated component notifies the recognition framework about the presence of this new photo The new photo is processed: faces are detected and face descriptors are extracted from it Intermediate results are stored locally Detected faces within the new photo are sent to the server of the feedback framework The server of the feedback framework distributes the received faces among its clients The clients implement a game, and by playing this game, an implicit feedback is gained about the faces The clients send back any gained implicit feedback to the server of the feedback framework, which processes it, then saves some intermediate results, and finally sends it further to the people recognition framework The people recognition framework received the implicit feedback and incorporates it with the aim of improving recognition accuracy The faces contained within the added photos are recognized and results are stored locally Finally, the results are presented to the user

LikeLines: Collecting implicit user information

User interactions with multimedia items in a collection can be used to improve their representation within the information retrieval system’s index. The LikeLines component developed in the project in task "Implicit User-Derived Information" is used for collecting these user interactions and is described within this section. The work carried out in this task further supports work on "Pipelines for relevance feedback", wherein LikeLines will be a core component of the pipeline.

7.2.1

Use of implicit playback behaviour

In order to perform multimedia retrieval at the fragment level, it is necessary to know what the interesting parts of a multimedia item are. The LikeLines component focuses on finding the interesting bits in a video by capturing the user’s playback behaviour based on the assumption that a user only likely wants to watch the parts of a video that are interesting, potentially multiple times, and skips those parts that are uninteresting. By recording which parts are watched and which parts are skipped by each user through playback events (such as “play”, “pause”, and “seek”), LikeLines infers what is considered by viewers to be CUbRIK First GWAP and Implicit User Information Techniques

Page 34

D3.1 Version 1.0

interesting and visualizes this as a heat map (“like line”) below the video.

To power this inference, large quantities of user interactions are needed in order to understand how each type of contribution needs to be interpreted. Therefore, LikeLines is made to target a large collection of existing videos on the Web and to be incorporated in any Web site. By having an open component, it can be deployed beyond conventional lab settings and have a greater reach.

Additionally, the collected user interactions are used as feedback for improving automatic analysis of video content.

7.2.2

Technical details of the LikeLines player component

This section originally appeared partly in [LARS2012] - LikeLines: Collecting timecode-level feedback for web videos through user interactions. The LikeLines system consists of two main components: a Web video player component that resides in a browser on the user’s system and a server component:

Figure 23 Users’ interactions User interactions (such as play, pause, etc.) with the LikeLines player are stored at a server and are aggregated into a heat map. Viewers can use this heat map to jump to a particular point in the video. Content analysis of the video can seed the heat map.

The user only directly interacts with the player component. This component is implemented in JavaScript and uses HTML5 or Flash for video playback. User interactions such as playing and pausing the video are captured by the LikeLines player and are sent to the server component. The server component is responsible for storing and aggregating all these user interactions. The player component communicates with the server using the HTTP protocol and can make the following requests: a) Create a new interaction session for a video; b) Add new interactions to an existing session; and c) Aggregate content analysis and all sessions for a particular video to compute a heat map.

The server’s reply messages are encoded in the JSON or JSONP format. The heat map is computed by representing an interaction session for an n seconds video as n bins. Each bin is initially set to 0 and each interaction can contribute, possibly negatively, to a bin’s value.

Content analysis of a video is modelled as an interaction session as well. The heat map is then obtained by aggregating all sessions and mapping each bin’s accumulated value to a colour.

CUbRIK First GWAP and Implicit User Information Techniques

Page 35

D3.1 Version 1.0

8.

Towards potential future extensions

Games with a Purpose (GWAP) are instruments for gathering annotations of improve the existing annotations by making the process entertaining for human. The main purpose of this kind of games is to include the human in the loop of metadata production for multimedia while encouraging them to remain truthful by providing various incentives throughout the gaming experience including the scoring system, high score table, achievements, badges, etc.

In CUbRIK the GWAPs and their corresponding Gaming Frameworks are classified as the units that either directly provide the human annotations or they help the conflict manger unit to resolve the conflicts that other parts of the platform could not handle without human contribution. For this purpose the conflicts are broken into tasks where each task is either injected to an existing game or a new game is automatically produced for it.

When the gamers play a game they are connected to the CUbRIK servers in which their profile along with achievement records are stored and is updated constantly in real-time.

Three gaming frameworks were introduced that help with acquiring user data by means of gaming. These are: •

The Image annotation game: The purpose of this game is to annotate images directly by the user and 1) to update their score based on the quality annotation 2) expose them to the best suitable image database (either fully annotated, partially annotated or non-annotated) based on their trustworthiness. The processing system of this game can be summarised by using two major units, namely, payoff calculation and decision making unit and player outcome prediction unit. The payoff calculation unit is designed to measure the player’s contribution in gaming in order to decide which image database to use. In other words, the optimal fully annotated, partially annotated or non-annotated content is selected based on the Nash Equilibrium based decision-making process. The player outcome prediction unit, on the other hand, enhances the performance of the payoff calculation and decision making unit by predicting the player’s outcome. Here, a temporarily graphical user interfaces was developed for testing purposes. In this practice, the annotation is achieved by offering the image subject to the player through the interface and prompting the players to comment on it using a string of characters. This string is subsequently analysed by the dictionary analysis module to establish whether the player has entered a valid keyword. Following the keyword search, the payoff calculation and equilibrium analysis unit will measure each player’s payoff and finally the score computation module will calculate the scores of both players. This process will continue until the game session ends. .

•

Sketchness game: This is a multiplayer puzzle game that makes use of the inversion-problem mechanic.

The game tries to 1) understand the identity of the different objects inside an image by exploiting human cognition and 2) provide meaningful traces of the contour of the object as they have been provided by several users. In this game only one player can see the subject image and they have to sketch their object of interest inside the image over the image while other players can only see the resulting sketch; the score is achieved based on the correctness of the guess of the blind player and the timing to spot the word.

•

Crosswords game:

This is a single player game that uses the Output agreement technique. The game tries to verify the metadata by formulating the clues and the answers, presenting them to multiple players and having players confirm or correct the presented metadata. The game collects players’ contributions in the form of explicit metadata fixes and implicit confirmations (votes) in favour of metadata correctness.

At the current stage, the games are using a temporary user interface and are working CUbRIK First GWAP and Implicit User Information Techniques

Page 36

D3.1 Version 1.0

independently. The future goal is to develop an appealing gaming interface for the Image Annotation game in the distributed environments which is in direct contact with CUbRIK servers and working in harmony with other games including Sketchness. The target is to synchronise the games in a way that they play complementary roles for each other to augment data that the other games are no able to acquire. In addition to improve the quality of the annotation it is intended to combine a number of different key paradigms including Computer gaming, Game theoretic principles and Machine learning techniques.

At the current stage the game theory based decision making and Nash equilibrium based decision making techniques are used to expose the player to the most suitable multimedia content. These methods are only applied on the Image Annotation game. After testing and validating the results these methods will be centralised so that other games will be able to apply them in their structure. The goal is to obtain mmore realistic annotations by predicting the player’s behavior. This will be done by: - Providing a Centralised user profiling unit where each game has to synchronise with CUbRIK gaming server to authenticate the user profiles, managed also by the achievement system that is being implemented based on the specifications that have been provided.

- Providing synchronized multimedia: All the annotations and intelligent mechanism to show un-annotated images to certain player will be on CUbRIK gaming server in a synchronised way.

- Frontend of GWAP for image annotation: GWAP for image annotation has advanced backend mechanism with very basic frontend that will be improved in the 2nd year of the project.

CUbRIK First GWAP and Implicit User Information Techniques

Page 37

D3.1 Version 1.0

9.

References

[Ahn2004] L. von Ahn and L. Dabbish, "Labelling images with a computer game," in proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 2004.

[Ahn2007] L. von Ahn, S. Ginosar, M. Kedia, and M. Blum, "Improving Image Search with PHETCH," in International Conference on Acoustics, Speech and Signal Processing, 2007 [Björk2005] Björk, S, Holopainen, J (2005). Patterns in game design. Hingham, MA: Charles River Media. pp. 15-20.

[Ceri1997] Ceri S, Fraternali P (1997) Designing Database Applications with Objects and Rules: The IDEA Methodology. Addison-Wesley.

[Cheng2009] D. Cheng, N. Dhulekar (2009) Crossword puzzle generation. CS44 Artificial Intelligence Course Assignment. Dartmouth College.

[Deterding2011] Deterding S, Sicart M, Nacke L, O’Hara K, Dixon D (2011) Gamification using game-design elements in non-gaming contexts. In CHI Extended Abstracts, D. S. Tan, S. Amershi, B. Begole, W. A. Kellogg, and M. Tungare, Eds. ACM, pp. 2425–2428 [Evans2011] Evans M J, Jennings E B, Andreen M (2011) Assessment through achievement systems: A framework for educational game design , IJGBL, vol. 1, no. 3, pp. 16–29 [Fullerton2008] Fullerton T, Swain C, Hoffman S (2008) Game Design Workshop: A Playcentric Approach to Creating Innovative Games. Elsevier Morgan Kaufmann.

[Hacker2009] S. Hacker and L. von Ahn, "Matchin: Eliciting User Preferences with an Online Game," in proceedings of ACM Conference on Human Factors in Computing Systems (CHI), 2009, pp. 1207-1216.

[Hamari2011] Hamari J, Eranti V (2011) Framework for designing and evaluating game achievements. In Think Design Play: The 5th Int. Conf. of the Digital Research Association, C. Marinka, K. Helen, and W. Annika, Eds. Hilversum, the Netherlands, p. 20.

[Juul2005] Juul J (2005) Half-real: video games between real rules and fictional worlds. MIT Press [Juul2010] Juul J (2010) In search of lost time: on game goals and failure costs. In Proceedings of the Fifth [Kam2009] Kam Tong Chan, Irwin King, Man-Ching Yuen: Mathematical Modeling of Social Games. CSE (4) 2009, pp. 1205-1210 [King2009] I. King, Li. Jiexing, and K. T. Chan, "A Brief Survey of Computational Approaches in Social Computing," in Proceedings of International Joint Conference on Neural Networks, 2009.

[Lars2012] R. Vliegendhart, M. Larson, and A. Hanjalic. LikeLines: Collecting timecode-level feedback for web videos through user interactions. In Proceedings of the 20th ACM international conference on Multimedia. ACM, 2012 [Law2007] E. L. M. Law, L. von Ahn, R. B. Dannenberg, and M. Crawford, "TAGATUNE: A Game for Music and Sound Annotation," in proceedings of 8th International Conference on Music Information Retrieval (ISMIR), 2007.

[Law2009] E. Law and L. von Ahn, "Input-Agreement: A New Mechanism for Collecting Data Using Human Computation Games," in Conference on Human Factors in Computing Systems, 2009.

[Lee2008] L. Lee, G. Loewenstein, D. Ariely, J. Hong, and J. Young, "If I'm Not Hot, Are You Hot or Not? Physical-Attractiveness Evaluations and Dating Preferences as a Function of CUbRIK First GWAP and Implicit User Information Techniques

Page 38

D3.1 Version 1.0

One's Own Attractiveness," Psychological Science, vol. 19, no. 7, pp. 669-677, 2008.

[Luca] Galli, Luca, and Piero Fraternali. "ACHIEVEMENT SYSTEMS EXPLAINED." [Lvon2008] L. von Ahn and L. Dabbish, "Designing games with a purpose," Communications of the ACM, vol. 51, no. 8, 2008.

[McClanahan2009] McClanahan G (2009) Achievement Design 101. http://www.gamasutra.com/blogs/GregMcClanahan/20091202/3709/Achievement Design 101.php.

[Meehan1997] G. Meehan, P. Gray (1997) Constructing crossword grid: use of heuristics vs constraints. In: Proceedings of Expert Systems 97: Research and Development in Expert Systems XIV, SGES [Ps3Trophies.org2006] Ps3Trophies.org (2006) Ps3Trophies. http://www.ps3trophies.org/.

Accessed 17 Aug 2012 [Salen2004]. Salen K, Zimmerman E (2004) Rules of Play: Game Design Fundamentals. Mit Press.

[Tuulos2007] V. Tuulos, J. Scheible, and H. Nyholm, "Combining Web, Mobile Phones and Public Displays in Large-Scale: Manhattan Story Mashup," in proceedings of the 5th international conference on Pervasive computing, 2007, pp. 37-54.

[Xbox360Achievements.org2006] Xbox360Achievements.org (2006) Xbox360Achievements. http://www.xbox360achievements.org/. Accessed 17 Aug 2012

CUbRIK First GWAP and Implicit User Information Techniques

Page 39

D3.1 Version 1.0