methods and techniques (2/6)
DESCRIPTION
ISWC 2010: TUTORIAL: Ten Ways to Make your Semantic App AddictiveTRANSCRIPT
Methods and techniques to analyze and design incentivized
semantic applications
www.insemtives.eu
Roberta Cuel, University of Trento, IT and Markus Rohde, University of Siegen, DE
ISWC 2010
www.insemtives.eu04/14/2023 2
Why is my app not as successful as Facebook…
04/14/2023 www.insemtives.eu 2
• Motivation and incentives– Reciprocity – Reputation – Competition– Altruism– Self-esteem – Fun– Money
• This session is about the methods and techniques you can use – To study your scenario prior to
application design– To evaluate your incentives strategy
and adjust your application
www.insemtives.eu
Overview
• Basic notions and definitions• Methods and techniques for analysis, design
and evaluation• Case studies
04/14/2023 3
www.insemtives.eu
BASIC NOTIONS AND DEFINITIONS
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 4
www.insemtives.eu
Starting with motivation…
• Basic tenets of organizational behavior – Performance : f(ability*motivation)
04/14/2023 5
www.insemtives.eu
Motivation and incentives
• Intrinsic vs extrinsic motivations• Incentives are ‘rewards’ assigned to performer
to make him/her commit– Can be totally uncorrelated to the nature of the
task – Need to be compliant with the values and beliefs
of the acting person
04/14/2023 604/14/2023 www.insemtives.eu 6
www.insemtives.eu
Example: DBPedia
The DBpedia community collaborates using the following tools– Mailing lists, Facebook group– Blog, quality assurance and bug tracking– Development
• “The framework is easily extended and we offer the possibility to do this yourself, increasing the likelihood of your desired data being included in the next DBpedia release.”
– DBpedia consulting (money)
04/14/2023 7
www.insemtives.eu04/14/2023 8
Typology of motivationsMotivations Internal
(embedded in structure, e.g., task, tools)
External(additional to structure, external re-inforcements)
Intrinsic(predispositioned in person, e.g., drives, needs, desires )
Fun, joy, gaming,interest, satisfaction, self-actualization, self-re-inforcement
Social appreciation, reputation, love, trust, social capital, community support
Extrinsic(additional to personal predispositions, extern re-inforcements )
Usability, sociability,Design-for-fun, curiosity,community-building support
Material/financial capital, money, rewards, prices, medals, credit points
Structure
Person
Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)
www.insemtives.eu
METHODS AND TECHNIQUES FOR ANALYSIS, DESIGN AND EVALUATION
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 9
www.insemtives.eu
Overview of methods and techniques
• To analyze your scenario and design your application– Game theory, mechanism design– Participatory design and end-user development
• To evaluate an incentives strategy in its context– Interviews and questionnaires and field experiments – Heuristic evaluation, guideline reviews– Cognitive and pluralistic walk-throughs– Participatory/User evaluation
04/14/2023 1004/14/2023 www.insemtives.eu 10
www.insemtives.eu
Game theory
• Game theory is a formal way to analyze interaction among a number of rational agents who behave strategically– The rational agents: players involved in the situation (best choice)– A number of players: more than one– Rationality/payoffs: what are the players’ preferences over the
outcomes of the game– The interactions: one player’s behavior affects another – The rules: who moves when, what do they know, what can they do– The outcomes: what is the outcome of the game (for each move)
04/14/2023 11
www.insemtives.eu
Mechanism design
• Mechanism design is about how to translate game theory in effective behavior– To design rules such that a desired set of outcomes
happens– Alignment of interests between parties and production of
maximum social welfare
• Relevant variables– Structural and personal motivations– Goal and nature of good produced– Tasks and skills required– Social structure
04/14/2023 12
www.insemtives.eu
Analysis matrix
• A multidimensional tool that allows for an analysis of a particular task with respect to the variables relevant for mechanism design
04/14/2023 13
Goal Tasks Social structure Nature of good produced
Skill variety/level
Communication level
Variety of Hierarchy-neutral
Private good Skilled ability
Participation level
Specificity of Hierarchical Public good Competence
Clearness level Identification with
Common resourceClub good
www.insemtives.eu04/14/2023 14
The matrix in practice: semantic annotation
• Goal of the annotation or ontology population exercise• Task, or more typically, an ordered collection of tasks into
which the annotation exercise can be broken down• Social structure, a stylized and simplified set of social
relationships among the subjects participating in the exercise• Nature of good, a stylized description, in game-theoretical
terms, of the relationship between what good is produced and who consumes it
• Required skills of the agents to complete the annotation task
04/14/2023 www.insemtives.eu 14
www.insemtives.eu
The matrix in practice: crowd-sourcing of ontology evolution
• Goal: communication and participation level– Coordination (free-open)
• Interests are already aligned (free riding)• Private and public benefit coincide
• Task: variety vs specificity • Social structure: hierarchy neutral (large vs small
groups)• Nature of good: public good non-rival (use does not
limit use of others), non-exclusive (open access)• Skill variety/level: Skilled ability
04/14/2023 1504/14/2023 www.insemtives.eu 15
www.insemtives.eu
Participatory Design (PD)
• To develop software for the participation of end-users, we propose a participatory way of designing these software tools, integrating potential users by participatory design methods
• Participatory Design– Improvement of the participation of workers in software development
processes– The cooperation between software developers and end-users– Participate in IT development projects as experts of their own work
04/14/2023 16
www.insemtives.eu04/14/2023 17
Methods for PD
• Participatory-design projects combine– Design-by-doing methods – PD workshops– Scenarios – Different forms of prototyping (mockups, rapid
prototypes)– Work organization games – Ethnographic methods
www.insemtives.eu04/14/2023 18
End-User Development (EUD)
• EUD is about designing highly flexible systems that enable users to participate during the use of the system by adapting and modifying the tools according to their needs/preferences
• Goal: Empower end-users to develop and adapt systems themselves by designing these systems to be easy to– Understand– Learn– Use– Teach
www.insemtives.eu04/14/2023 19
How to design an incentivized applicationIdeally: field desk lab field
Analyze the domain and find yourselves in the matrixes Find the relevant point of that situation (goal and tasks)Focus on a small group of individuals (social structure)Analyze their motivation (internal/external intrinsic/extrinsic)Analyze the other relevant variables (nature of good being produced, kill variety/level)
Design a simplest possible model that can effectively support contributorsTest and get feedback Fine-tune the experiment and add other elements
www.insemtives.eu04/14/2023 20
Methods
• Domain observations (second-hand data)• Ethnography or qualitative face-to-face interviews • Questionnaires• Observations with selected individuals • Quantitative analysis (data collections)• Usability-design methods
04/14/2023 www.insemtives.eu 20
www.insemtives.eu
CASE STUDIES
Basic notions and definitionsMethods and techniques for analysis, design and evaluation
Case studies
04/14/2023 21
www.insemtives.eu
Examples from our project
• Telefónica Investigación y Desarrollo (TID - Spain): Knowledge Intranet Platform
• Pepper’s Ghost (PGP-UK): Virtual Games/Worlds• Seekda! (Austria): Web Services Search Portal
04/14/2023 22
www.insemtives.eu04/14/2023 23
Field work: TID
• Domain analysis– Site visit, semi-structured, qualitative interviews
• Communication processes• Existing usage practices and problems• Existing tools/solutions• Semantic annotation solutions
– Tape recording, transcription– Data analysis per ex-post categorization
• Focus group discussion– Usability lab tests– Expert walkthroughs
www.insemtives.eu04/14/2023 24
Field work: TID (2)
04/14/2023 www.insemtives.eu 24
• Find TID in the matrix– Goal: communication and participation (contribute to the
knowledge portal) – Social structure (various structures co-exist)
• Strongly hierarchical organization (control is an issue) • working groups and community of experts
– Nature of good: public good vs. private, club goods– Skill variety/level: Skilled ability (knowledge workers)– Motivations: fun, visibility, reputation, promotion, money
www.insemtives.eu04/14/2023 25
Field work: PGP• Domain analysis– Data analysis (Benchmark: Galaxyzoo, Moonzoo)– Usability lab tests and expert walkthroughs– Collect data on users such as logs files
• Find PGP in the matrix– Goal: participation level– Task: specific– Social structure: hierarchy neutral– Nature of good: public good non-rival (use does not limit
use of others), non-exclusive (open access)– Skill variety/level: non skilled ability
www.insemtives.eu04/14/2023 26
Field work: SEEKDA
• Domain analysis– Interviews and workshops – Replicating Telefonica model
• Find SEEKDA in the matrix:– Goal: communication and participation level– Task: variety vs. specificity – Social structure: hierarchy neutral (mainly)– Nature of good: public good non-rival (use does not limit
use of others), exclusive, or club good– Skill variety/level: Skilled ability