cubrik tutorial at icwe 2013: part 2 - introduction to games with a purpose

AN INTRODUCTION TO

HUMAN COMPUTATION &

GAMES WITH A PURPOSE

ALESSANDRO BOZZON DELFT UNIVERSITY OF TECHNOLOGY

LUCA GALLI POLITECNICO DI MILANO

ABOUT THE TUTORIAL •  Crowdsourcing, Human Computation, and GWAPs are hot topics

•  “Human Computation” => more than 3000 papers •  400 in 2013

•  “Crowd Sourcing” => more than 36000 papers •  4800 in 2013

•  “Games With A Purpose” => more than 1400 papers •  162 in 2013

•  This short tutorial is necessarily shallow, but •  Concrete Examples •  Lot of references and links •  An outlook on the future

•  Slides and additional materials available •  http://hcgwap.blogspot.com

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose 2

ABOUT THE SPEAKERS ALESSANDRO BOZZON Assistant Professor - TU Delft http://www.alessandrobozzon.com

[email protected]

LUCA GALLI Ph.D. Student - Politecnico di Milano http://www.lucagalli.me

[email protected]


•  RESEARCH BACKGROUND AND INTERESTS

•  Web Data Management •  Crowdsourcing and Human Computation •  Game Design •  Web Engineering and Model Driven Development

AGENDA

4 ICWE 2013 - An Introduction To Human Computation and Games With a Purpose

AGENDA •  PART 1 => CrowdSourcing and Human Computation

•  Introduction •  Design of Human Computation Tasks •  Frameworks For/With Human Computation •  The Future of Human Computation

•  PART 2 => Games With a Purpose •  Play vs Games •  Introduction to Game Design •  Games with a Purpose Design Guidelines


PART 2 AN INTRODUCTION TO GAMES WITH A PURPOSE



Something really bad is going to happen…


Or not?


MAGIC CIRCLE

“All play moves and has its being within a playground marked off beforehand materially or ideally, deliberately or as a matter of course… The arena, the card-table, the magic circle, the temple, the stage, the screen, the tennis court, the court of justice, etc., are all in form and function play-grounds, i.e., forbidden spots, isolated, hedged round, hallowed, within which special rules obtain. All are temporary worlds within the ordinary world, dedicated

to the performance of an act apart.”

Johan Huizinga, “Homo Ludens”, (1938-1950)

WHAT IS PLAY?


Autonomy: play is a voluntary ac2vity.

Safety: during play there are radically reduced serious consequences in what we do

A<unement: being interested in the same things temporarily while playing together

Explora2on: the possibility to experiment and try out new things

Mastery: the will to improve one’s own skills

WHICH ARE THE BENEFITS?


“Play is an enjoyable ac/vity that allows us to train and test our skills in a safe environment.”

Fosters crea2vity, flexibility, and learning

An2dote to loneliness, isola2on, anxiety, and depression Teaches perseverance

Hints on how to cooperate with others Increases energy and prevents burnout

Stuart Brown, “Play: How it Shapes the Brain, Opens the Imagination, and Invigorates the Soul”, 2009.

GAMES VS PLAY


Games are dis/nguished from play

– Play is free-‐form

– Games are rule-‐based A game is a closed, formal system that ·∙ Engages players in structured conflict and ·∙ Resolves its uncertainty in an unequal outcome.

Fullerton, T.; Swain, C. & Hoffman, S. Game Design Workshop: A playcentric approach to creating innovative games, 2008

CLASSIC GAME DEFINITION


1.  Rules • Games are rule-‐based.

2. Variable, quan/fiable outcome

• Games have variable, quan/fiable outcomes.

3. Valoriza/on of outcome

• The different poten/al outcomes of the game are assigned different values, some posi/ve and some nega/ve

CLASSIC GAME DEFINITION


4. Player effort

• The player exerts effort in order to influence the outcome (games are challenging).

5. Player aVached to outcome

• The player is emo/onally aVached to the outcome of the game in the sense that a player will be winner and “happy” in case of a posi/ve outcome, but a loser and “unhappy” in case of a nega/ve outcome.

6. Nego/able consequences

• The same game [set of rules] can be played with or without real-‐life consequences

Juul, J. “Half-R

eal: Video Gam

es Betw

een Real

Rules and Fictional W

orlds”, 2005

INCENTIVES


Money, Passion, Ease of Participation, Altruism, Appeal to one’s own knowledge, Reputation, Fun

Organisciak, Piotr. Why Bother?: Examining the Motivations of Users in Large-scale Crowd-powered Online Initiatives. Diss. University of Alberta, 2010.

INCENTIVES


Intrinsic mo2va2on is defined as doing an ac2vity for its inherent sa2sfac2ons rather than for some separable consequence.

Extrinsic mo2va2on is defined as doing an ac2vity to a<ain some separable outcome

Deci, Edward L., and Richard M. Ryan. Self‐Determination. ,1985.

GAMES WITH A PURPOSE


Games with a Purpose (GWAPs) are digital games that generate useful data as a by-‐product of play. [vA06,LvA09]

The design of a GWAP requires to create a game so that its structure encourages computa2on, correctness of the output and players reten2on.

WHAT THEY ARE USED FOR


Krause, M

. & S

meddinck, J. (2011b). H

uman C

omputation

Gam

es: A Survey. In: P

roceedings of the 19th European

Signal P

rocessing Conference (E

US

IPC

O-2011).

Intui2ve Decisions

Aesthe2c Judgment

Contextual Reasoning

Embodiment Issues

ARE THEY WORKING?


ESP Game

Purposes: Let players determine the contents of images by submitting meaningful labels they can agree on.

Results: A total of 13,630 people played the game during the test phase, generating 1,271,451 labels for 293,760 different images.

Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game.

ARE THEY WORKING?


Foldit

Purposes: The objective of the game is to fold the structure of selected proteins to the best of the player's ability, using various tools provided within the game to discover new structural configurations.

Results: In 2011, players of Foldit helped to decipher the crystal structure of an AIDS-causing monkey virus. While the puzzle was available to play for a period of three weeks, players produced an accurate 3D model of the enzyme in just ten days. The problem of how to configure the structure of the enzyme had stumped scientists for 15 years.

Khatib, F.; Dimaio, F.; Cooper, et al. (2011). "Crystal structure of a monomeric retroviral protease solved by protein folding game players"

GAMIFICATION


“The use of game design techniques and game mechanics to enhance non-‐game contexts”

S. Deterding, M. Sicart, L. Nacke, K. O’Hara, and D. Dixon, “Gamification. Using game-design elements in non-gaming contexts”

Actually ques/onable, we will see why

Sebastian Deterding

Miguel Sicart

WHAT IS USED FOR


IS IT WORKING?


Samsung Nation

Purposes: Samsung Nation, a social loyalty program that lets users earn badges for activities as writing reviews and watching videos and compete for rewards.

Results: 500% increase in customers product reviews 66% increase in site visitors 30% increase in comments reduced marketing costs reduced product support costs

IS IT WORKING?


MTV Italia MyChart

Purposes: User generated video chart based on various “game dynamics” like avatars, points and leaderboards to drive users from Facebook to TV and to loyalize existing TV audience. Results: More than 500.000 votes in less then three months More than 150.000 videos viewed

COMMONALITIES AND DIFFERENCES


One or more objec2ves or tasks that users have to accomplish.

A gaming experience, defining challenges to overcome and rewards for their solu2on.

One or more players, the users of the applica2on, who are profiled and monitored in their ac2vi2es.

But...


INTRODUCTION TO GAME DESIGN


Mancala, 7th century AD

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose

30

Unreal Tournament 3, Epic Games, 2007


Turn based boardgame vs Real /me ac/on shooter

Handmade physical board vs Personal Computer

Public domain rules vs Copyrighted

1 - PLAYERS


Number of players Roles of the players Interac2ons among players and the game:

Single Player/Mul2 Single Unilateral/Mul2lateral Compe22on

Coopera2ve Team compe22on

2 - OBJECTIVES


Objec2ves: specific goals for the player to accomplish

Capture: capture or destroy a resource Chase: catch an opponent or elude one Race: reach a goal before the other players Alignment: arrange game objects in a spa/al or conceptual

configura/on Rescue\Escape: bring a unit to safety Construc2on: build, maintain, manage objects Explora2on: explore the environment where the game take place Solu2on: solve a problem or puzzle before (or more accurately) the compe//on or certain constraints Outwit: gain and use knowledge to defeat other players

2 – OBJECTIVES: EXAMPLES


Alignment: arrange game objects in a spa2al or conceptual configura2on

Tetris, Alexey Pajitnov, 1984 Puzzle Bobble, Taito Corporation, 1994

2 – OBJECTIVES: EXAMPLES


Solu2on: solve a problem or puzzle before (or more accurately) than the compe22on or following certain constraints

Connect Four, Milton Bradley, 1974

Professor Layton and the Miracle Mask, Level-5, 2011

3 - PROCEDURES


Procedures: the methods of play and the ac/ons that players can take to achieve the game objec/ves. They are used to define who does what, where and how.

Typically a game is composed of: Star%ng ac%on: How to put a game into play. Progression of ac%on: Ongoing procedures a_er the star/ng ac/on. Special ac%ons: Available condi/onal to other elements or game state. Resolving ac%ons: Bring gameplay to an end.

3 – PROCEDURES: EXAMPLE


Star%ng ac%on: Choose a player to go first. Each player chooses a color: red or black.

Progression of ac%on: On each turn, a player drops one colored checker down any of the slots in the top of the grid.

Resolving ac%ons: The play alternates un/l one of the players gets four checkers of one color in a row. The row can be horizontal, ver/cal, or diagonal.

Connect Four

4 - RULES


•  Define Objects •  Restrict Ac/ons •  Determine Effects (ECA rules) Chess: A player cannot move her king into check. Poker: A straight is five consecu8vely ranked cards; a straight flush is five consecu8vely ranked cards of the same suit. WarCrad II: To create knight units, a player must have upgraded to a keep and built a stable.

5 - RESOURCES


Resources: valuable objects that can help the players to achieve their goals but are scarce in the system.

Lives: number of “trials” that can be aVempted to reach a goal Units: mul/ple objects that has to be managed instead of lives Health: represent the status of loss or near loss of lives and units Points: numerical value that represents a measure of the skill and progression of a player Ac2ons: number of possible dis/nct choices that a player can make within a defined /meframe Power-‐ups: object that gives a boost to the player Items: used by the player to accomplish an objec/ve, made scarce by the system Turns: the number of game phases within which a player must accomplish the objec/ves Time: restricts player ac/ons or phases in periods of /me.

5 – RESOURCES: EXAMPLE


Card Hunter, Blue Manchu Pty Ltd, TBR

Time

Points

Ac/ons

6 - CONFLICTS


Conflicts: emerges from the players trying to accomplish the goals of the game within its rules and boundaries, since procedures and rules tend to deter players from accomplishing goals directly or make players work against each other. The most common conflicts are generated by: Obstacles, objects or rules that limit the freedom of the players Opponents, since they are usually trying to achieve an objec/ve faster than us Meaningful Choices, players have to make choices that will influence the outcome of the game

7 - BOUNDARIES


Boundaries are what separate the game from everything that is not the game and defines the physical or virtual scopes in which the game is performed.

Example: Football would not be the same game if the boundaries of the football field were not defined.

8 - OUTCOME


Outcome: the outcome of the game (if present) is uncertain, since it is not possible to predict the results ahead. If an outcome can be obtained it has to be quan/fiable with respect to the defined goals. It is different than the objec/ve since all the player can reach the same objec/ve.


Derived by modeling the guidelines and best practices found in: Fullerton, T.; Swain, C. & Hoffman, S.

Game Design Workshop: A playcentric approach to creating innovative games, 2008

Crowford, C. The Art of Computer Game Design, 1984

GAME DEVELOPMENT PROCESS

GWAPS AS SERIOUS GAMES


Ben

Saw

yer,

Pet

er S

mith

: S

erio

us G

ames

Ta

xono

my,

200

8


Katharina Siorpaes and Martin Hepp. 2008. OntoGame: weaving the semantic web by online games.

GAMES WITH A PURPOSE: CURRENT ISSUES


●  “Ad-‐Hoc” Design

●  Engagement and reten/on mechanisms.

●  Mapping from task to game mechanics

●  Valida/on techniques and strategies

E.G. Real user comments about OntoGalaxy


1) PURPOSE

2) TASK DEFINITION

3) GAME MECHANICS DESIGN

4) INSTANCE MATCHING

5) OUTPUT VALIDATION

6) PERFORMANCE EVALUATION

7) ENGAGEMENT SRATEGY

GWAPS DIMENSIONS

GWAP DEVELOPMENT PROCESS


REQUIREMENTS SPECIFICATION


uTask Task Description

Input Objects

Output Objects

Execution Interface

Operations Output Validation

INPUT OBJECTS

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose 52 Adapted from“Mathematical Modeling of social games, IrwinKing, SIAG2009”

An input object is an object with data t23e and a set of finite a67ibutes denoted as

tex9, images, audio seg=ents, video seg=ents, other unst7@ct@red data, st7@ct@red data

where the data t23e is the media t23e presented by

Each a67ibute has a relationship to a set of values . Each is an object with its own data t23e that represent the metadata of

DEFINING OPERATIONS



Input Objects

Output Objects

Execution Interface


OPERATION TYPES


They fall in two broad categories: Genera/ve tasks Decision tasks

A possible (non-‐exhaus/ve) list of human computa/on tasks may include: •  Object Recogni/on/Iden/fica/on/Detec/on •  Sor/ng (Clustering/Ordering) •  Natural Language Processing •  State Space Explora/on •  Content Genera/on/Submission •  User preference/opinion elicita/on

ENSURE OUTPUT QUALITY



Input Objects

Output Objects

Execution Interface


Twofold purpose: ensure output correctness and counter player collusion.

OUTPUT VALIDATION


Player Tes2ng: compare the output provided by the user with known correct outputs

Repe22on:collect more than one solu/on for the same input object, then aggregate the solu/ons

Taboo Output: limit the possible solu/ons that can be submiVed

Random Pairing: pairing players at random

TASK EXECUTION



Input Objects

Output Objects

Execution Interface


TASK TO GAME MECHANICS MATCHING


Does a game with game mechanics similar to the task exists? If so, integrate the task within the game If not, custom game mechanics has to be implemented. Clear and Transparent game mechanics

SOLUTION MECHANICS


Pa<ern Recogni2on

Wikipedia stands to knowledge as BoardGameGeek stands to games.

Benng/Wagering

Line-‐Drawing

Tile-‐Placement Pa<ern Building

Memory

Hand Management

Ac2on Programming

Auc2on/Bidding

Partnership

Possible, meaningful game mechanics:

Take Inspira/on!

TILE PLACEMENT


Tile Placement games feature placing a piece to score points, based on adjacent pieces or pieces in the same group/cluster, taking into consideration non-spatial properties like color, "feature completion", cluster size etc.

PATTERN RECOGNITION APPLIED TO GWAPS

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose 61 Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, et al. (2012) Phylo: A Citizen

Science Approach for Improving Multiple Sequence Alignment

Purpose: Used to arrange the sequences of D.N.A, R.N.A or proteins to iden/fy regions of similarity

Game Mechanic: align the sequence contained in each row in order to obtain the greatest number of columns with matching colors.

LINE DRAWING


Games that make use of this mechanics involve drawing drawing of lines in one way or another.

LINE DRAWING APPLIED TO GWAPS


Purpose: Segment images Game Mechanic: draw the shapes of objects in a provided image in order to make the other players guess the underlying object.

MEMORY


Games that use the Memory mechanic require players to recall previous game events or information in order to reach an objective.

MEMORY APPLIED TO GWAPS


Purpose: Find similar Images Game Mechanic: Exploit the visual memory of players to find images which are similar.

PARTNERSHIP


Games with partnerships offer players a set of rules for alliances and teams. Partners are o_en able to win as a team, or penal/es are enforced for not respec/ng alliances.

PARTNERSHIP APPLIED TO GWAPS


Purpose: Collect “common-‐sense” fact for specific words. Game Mechanic: provide hints to the partner in order to let him guess the secret word

GWAPS VALIDATION MECHANICS


Bad solu/ons due to: incompetence mischief behavior plain mistakes

Find or adapt game mechanics in order to validate the results.

GWAP FORMAL MODEL

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose 69 Based on Mathematical Modeling of social games, Irwin King, SIAG2009

is the set of uTask that are associated with the problem, with the uTask defined as we have previously seen.

is the answer domain; it contains the set of possible cer9ain answers fi (based on the input D) if the out3ut is contained within a defined domain, or empt2 if the answers are provided by the players

is a f@nction that deter=ines if a par9icular out3ut is a valid solution to the problem

is the GWAP Domain, that maps the input to the out3ut of a GWAP

MULTIPLAYER: INPUT AGREEMENT


Same or different input Based on each other’s hints, player must agree if they are dealing with the same input

Given a correct answer set for the problem

players have a set of poten/al outputs which cardinality influence the probability of agreeing upon the input

Results Submission: One Trial

Suggested for: Subjective Information

Same or different input, same roles

INPUT AGREEMENT: TAG A TUNE


Same Input: Audio file

Two different shared sets of descrip/ons, one for each player

Result: same input or different input

Results Submission: One Trial

MULTIPLAYER: OUTPUT AGREEMENT


Same Input Agreeing on common output without communica/on

Same Input, Same Roles.

Given a correct answer set for the problem,

players have a set of poten/al outputs

the probability to obtain an agreement depends on the shared output

Results Submission: Mul/ple Trials Suggested for: Objec/ve Informa/on

OUTPUT AGREEMENT: ESP GAME


Same Input: Image

List of possible guesses, hidden to the other players

Result: Meaningful tag for the provided image

Results Submission: Mul/ple Trials

MULTIPLAYER: INVERSION PROBLEM


One players knows the whole problem and gives hints to the other players. The others try to unveil the secrets based on the hints.

Input to just one player

the output of one player is the input for the others

The correct answer set for the problem is unknown

The probability of guessing depends on the cardinality of the hints

Result: User generated content, solu/ons for complex data structures


INVERSION PROBLEM: IMAGE LABELING GAME


Describer: Image

Seekers: Textual description given by the describer Results Submission: Multiple trials Results: Meaningful descriptions for the image

INSUFFICIENT PLAYERS: PRERECORDED GAMES


Fake players based on the logs of previous matches Pros: The gwap can be played anytime Cons: The actions in the game has to be modeled and stored Inversion problem games can be difficult to simulate Requires a bootstrapping phase to acquire the initial traces

SINGLE PLAYER: ALGORITHMIC EVALUATION


Automa/c algorithm able to par2ally assess the quality of the provided results.

For each instance of the game the input is provided to just one player

The correct answer set for the problem is defined

The probability to obtain meaningful results is applica/on specific,based on the algorithm that is used to make the evalua/on

ALGORITHMIC EVALUATION: TYPE ATTACK


Jovian

, L.T.; Am

prim

o, O., "O

CR Correc8on

via Hum

an

Compu

ta8o

nal G

ame," S

ystem Sciences (HICSS), 2011 44th

Hawaii Interna

8ona

l Con

ference on

, vol., no., pp.1,10, 4-‐7

Jan. 2011

Improving the results of OCR in digitalizing ar/cles old newspapers.

Human players transcribe snippet of text and their performance is measured by comparing the number of similar words in the player’s output and the text from the library’s OCR transcrip/on.

HYBRID


Combines the previous approaches Input data based on previous results or user generated content

Inputs to one or more players with same or different roles

Answer set depending on the nature of the problem or provided by a player

The probability of obtaining a solu/on depends on the combina/on of the strategies

HYBRID: SKETCHNESS


Input: One image to a player Hints to the others

Segments traced by one player, possible tags submiVed by the others

Result: Meaningful tag + segmenta/on


INSTANCE MATCHING


How to assign tasks (instances) to players? • Random Strategy: assign a random task

Pros: Easy to implement Cons: Does not take into considera/on player skills

• Past History: assign a task based on past performances

Pros: Can improve the quality Cons: Keeping track of gaming history Performance measures must be defined

• Selec2ve Assignment: assign a specific task based on skills and past performances

Pros: Can improve the quality Cons: Same as Past History Needs profiling of the player to assess skills

PERFORMANCE EVALUTION


Luis von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM 51, 8

Throughput = average number of problem instances solved per human hour

ALP = average (across all people who play the game) overall amount of time the game will be played by an individual player

Expected contribution = throughput multiplied by ALP.

ENGAGEMENT STRATEGY


Core activity must be already engaging for the user. Gamification involves defining clear and definitive objectives that last even past the gaming experience, and are either set by, or negotiated between the user and the game along with a progressive path of short and intermediate goals leading to it. Onboarding Retention Virality Social Dimensions Additional Mechanics

POINTS


Points or Player Scores are a numerical value that represents a measure of the skill of a player. •  Immediate and las/ng feedback •  External display of progression •  May determine the win state • Connec/on between progress in the game and rewards

Galli, L., Fraternali, P. “Achievement Systems Explained“ SGSC2012, Singapore

Werbach, K. & Hunter, D. For the Win: How Game Thinking Can Revolutionize Your Business Wharton Digital Press, 2012

LEADERBOARDS


A Leaderboard is an ordered list of players based on the scores they have obtained in a specific game or system. •  Relates the performance of a player to the others •  Fosters compe//on and par/cipa/on •  Risky: May be demo/va/ng.


Werbach, K. & Hunter, D. For the Win: How Game Thinking Can Revolutionize Your Business Wharton Digital Press, 2012

ACHIEVEMENTS AND BADGES


An Achievement is a set of tasks, defined by a designer, for the player to fulfill so to achieve a milestone and track the progress in a system. A Badge is an ar/fact associated to the comple/on of an achievement and given to a player a_er its comple/on, or, in gaming terms, a_er “unlocking the achievement”. •  Define goals •  Onboarding tool •  Visual markers for reputa/on, •  Provide las/ng rewards


THIS IS JUST A GLIMPSE OF WHAT GAMIFICATION IS...


EXAMPLE: ZAMZEE GAMIFICATION


Personalization Rewards

Challenges Social Status

CASE STUDY: FASHION TREND MINING


Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture)

Use case: identifying trends in collections of images of people and garments Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years

Color descriptors

Texture descriptors

coarse (sub-)image similarity

THE FASHION TREND MINING PIPELINE


Male, 24 Female, 22 Female?, ??

REQUIREMENTS SPECIFICATION


Task Description: Recognize if a particular garment is present within a picture or define a new one and outline its contours.

Input Objects: A fashion image, an optional tag defining the garment to identify.

Aggregation Strategy: assign a value of 0 to each pixel outside the contour and 1 to each pixel contained within the contour, sum all the contribution and apply a threshold based on the number of players.

Output Data: For each submitted task the game has to provide the contour of the garment within the image (Polyline) and a tag defining the garment that has been segmented

GWAP DESIGN


Solution Mechanics: Pattern Recognition

Established genre: Draw and Guess

Inversion Problem Mechanic

PLAYERS:Number >=2

Multilateral Competition

Two different roles: Sketcher: has to draw the contours of the stated garment Guesser: has to guess the garment drawn by the sketcher

PLAYER ROLE: SKETCHER


●  The only player to see the low confidence image

●  “May” be asked to provide a tag for the image

●  Is asked to draw the contour of the object for which the tag is provided within the allotted time

●  Goal of the Sketcher is to let the other players guess the tag within a time slot without providing any other hints than the contour

PLAYER ROLE: GUESSER


●  Any other player in the game

●  His/Her goal is to guess the object for which the Sketcher has provided the contour

●  Not allowed to draw on the whiteboard, just to type in the chat box the probable answer as fast as possible

●  Scoring:

●  Sketcher: 10 pts + 1 for each guesser

●  Guesser: 10 pts to the first, then decreasing down to five

FRAMEWORKS: WEB GAMES


FRAMEWORKS: GAMIFICATION


Google Play Game Services

Mozilla Open Badges User Infuser

THE CUBRIK PROJECT


●  CUbRIK is a research project financed by the European Union

●  Goals:

●  Advance the architecture of multimedia search

●  Exploit the human contribution in multimedia search

●  Use open-source components provided by the community

●  Start up a search business ecosystem

●  http://www.cubrikproject.eu/

cubrik tutorial at icwe 2013: part 2 - introduction to games with a purpose

Technology