A Draw-and-Guess Game to Segment Images

Luca Galli, Piero FraternaliDavide Martinenghi, Marco TagliasacchiPolitecnico di Milano, ItalyEmail: first.last@polimi.it

Jasminko NovakEuropean Institute for Participatory Media,GermanyEmail: j.novak@eipcm.org

Luca Galli

● CUbRIK is a research project financed by the European Union

● Goals:

● Advance the architecture of multimedia search

● Exploit the human contribution in multimedia search

● Use open-source components provided by the community

● Start up a search business ecosystem

● http://www.cubrikproject.eu/

The CUbRIK Project

Luca Galli

Humans in Multimedia Information Retrieval

● Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features

● Solution: humans have superior capacity for understanding the content of audiovisual material

● State of the art: humans replace automatic feature extraction processes (human annotations)

● Our contribution: integration of human judgment and algorithms

– Goal: improve the performance of multimedia content processing

Luca Galli

4

● Problem statement: segment fashion images for mining trends based on visual features of garments (e.g. color and texture)

● Use case: identifying trends in collections of images of people and garments

● Applications: retrieving similar garments, inspect clothing trends in image collections, analyzing trends change in the years

The Fashion Trend Mining Scenario

Color descriptors

Texture descriptors

coarse(sub-)image similarity

Luca Galli

5

Problems in automatic feature detection: Body part and feature recognition is affected by the posture

of the portrayed subject

Problems arise when part of the body is occluded or in pictures with complex human poses

Issues of traditional approaches

?

Luca Galli

6

● Body part recognition

● Garments segmentation

● Gender identification

Solution: Human Contribution Issues are solved by augmenting the capabilities of

algorithms with human intervention. Humans have superior capacity to understand complex

content and to perform high level abstractions on visual representations

For these reasons, they are suitable for tasks such as:

Luca Galli

The CUbRIK Architecture

Content and users registration in the system

Managing tasks and processing contents

Provides front end for issuing queries and viewing results

Search Engines used to access the content and annotations

Luca Galli

8

Users Motivation: Games with a Purpose

● Users need motivation to complete a task in a human computation platform, such as monetary rewards, prestige, entertainment.

● Games with a Purpose (GWAPs) are digital games that generate useful data as a by-product of play.

● The design of a GWAP requires to create a game so that its structure encourages computation, correctness of the output and players retention.

● Introduced by Luis Von Ahn with the ESP Game, several GWAPs have been developed with a number of different purposes

Luca Galli

9

The Fashion Trend Mining Pipeline Male, 24 Female, 22 Female?, ??

Luca Galli

10

Gwap Design: Sketchness!

● Multiplayer Puzzle Game

● Established genre: Guess and Draw(Pictionary, iSketch…)

● Inversion Problem Mechanic

● Players take turns into drawing the shapes of objects inside an image in order to make the other players guess the underlying object.

● Two different roles:SketcherGuesser

● Objective: Garments tagging, body partsegmentation, garment segmentation

Luca Galli

11

Player Role: Sketcher● The only player to see the

low confidence image

● “May” be asked to provide atag for the image

● Is asked to draw the contourof the object for which the tag is provided within theallotted time

● Goal of the Sketcher is to let the other players guessthe tag without providingany other hints than thecontour

Tag of the target object

Fashion image provided by the pipeline

Contour of the garment provided by the player

Luca Galli

12

Player Role: Guesser

● Any other player in the game

● His/Her goal is to guess the object for which the Sketcherhas provided the contour

● Not allowed to draw on the whiteboard, just to typein the chat box the probableanswer as fast as possible

● Scoring:

● Sketcher: 10 pts + 1 for each guesser

● Guesser: 10 pts to the first, thendecreasing down to five

Contour of the garment provided by the Sketcher

Luca Galli

13

Task Injection: new approach to GWAPs

● Traditional gwaps developed on ad-hoc basis and tailored for a specific task

● Resulting gwaps may be not fun, unappealing to players or provide an experience that may still perceived by the users as work

● The proposed GWAP exploits suitable modified game mechanics of a well known and appreciated category of games, called “drawing and guessing games” to implicitly solve segmentation problems while playing

Luca Galli

14

● Task Injection:

● Does the task injection approach guarantee the same level of entertainment and participation for the players as the original instance of the game?

● How to measure entertainment anyhow?

● Output Aggregation:

● How the results provided by different players on the same input be processed and grouped together in order to obtain a unique result?

● Majority voting not suitable for lines!

● Integration within CUbRIK’s Gaming Framework:● Exploiting the features provided by CUbRIK in order to manage

players and their abilities and provide retention and reward features through gamification techniques.

Future Directions● The proposed game concept is still under development and there are

still several open questions and future improvements: