On the inaccuracy of numerical ratings: A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto [email protected]
Dpto. Lenguajes y Sistemas InformáticosUNED
Outline
1. Introduction
2. Motivation
3. From opinion ratings to pairwise queries: PWRM
4. Towards ranking resources in MOOCs
5. Conclusions & future work
2
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review Systems) Reputation
many types: professional links, friendships, purchases, ...complex: dynamism, complexity of the social structure, many nodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume? (more and more available information online)Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review Systems)
Reputation
many types: professional links, friendships, purchases, ...complex: dynamism, complexity of the social structure, many nodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume? (more and more available information online)Not enough experience so.. online review systems a (Yelp, Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review Systems)
Reputation
TrustReputation
opinions of third parties
Confidencelocal experiences
many typescomplexnodes (users, entities, ..)how can we identify and locate appropriate entities/services to consume?Not enough experience so.. Tripadvisor, ..) as a means to obtain opinions, rankings, etc…
Social Networks (Online Review Systems) & Reputation
4
1. Introduction
Social Networks (Online Review Systems)
Reputation
TrustReputation
opinions of third parties
Confidencelocal experiences
objective: extract reputation of entities (users, objects, …)how: gathering and aggregating opinionsexamples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
objective: how: examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
objective: how: examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
objective: how: examples:
5
1. Introduction
Reputation Mechanisms in Social Networks
Reputation Mechanisms
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
PROBLEM 1
‣DIFICULT TO MAP PREFERENCES INTO NUMERICAL OPINIONS‣SUBJECTIVITY!!!
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGR
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGRPROBLEM 2
BIAS PROBLEMS!!!
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGRDie Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGRDie Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Die Hard 3
Gone with the wind 0
Ben-Hur 4
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGRDie Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Capturing preferences through numerical opinions
passive: expecting users’ opinionscapturing opinions through numerical ratings + textual informationestimate reputation based on aggregating ratings (average ratings)
7
2. Motivation
Reputation Mechanisms (traditionally)…
AGRDie Hard 1
Gone with the wind 2
Ben-Hur 3
Ben-Hur ≻ Gone with the wind ≻ Die Hard Ben-Hur ≻ Die Hard ≻ Gone with the wind
Ben-Hur ≻ Die Hard ≻ Gone with the wind
Die Hard 0.2
Gone with the wind 0.1
Ben-Hur 1.0
Ben-Hur ≻ Gone with the wind ≻ Die Hard
Capturing preferences through numerical opinions
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000 most reviewed movies by users
clearly biased to positive ratings
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics
not influenced and biased by others’ ratings
8
2. Motivation
best model fitting VS best fit to a normal distribution
Bias problems derived from numerical ratings
Analyzing the distribution of average ratings from HetRec-2011 dataset
average rating distribution of the 100, 1.000 and 4.000 most reviewed movies given by top critics
not influenced and biased by others’ ratings
PROBLEM CONFIRMATION
potential bias problems when mapping opinions onto numerical values, reputation
rankings may vary; and likely to cause differences between the true quality of an
entity and its rating aggregated from opinions
9
2. Motivation
FROMReputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Solution proposed
9
2. Motivation
FROMReputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Reputation Rankings:
pro-active method + comparative opinions + comparative aggregationTO
Solution proposed
9
2. Motivation
FROMReputation Rankings:
passive method + numerical opinions + opinions aggregation (average ratings)
Reputation Rankings:
pro-active method + comparative opinions + comparative aggregation
Pairwise preference elicitation
Aggregation mechanism
TO
Solution proposed
11
3. From opinion ratings to pairwise queries: PWRMComparative opinions: Pairwise preference elicitation
Based on pairwise queries:
11
3. From opinion ratings to pairwise queries: PWRMComparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROMBen-Hur
[0..1][Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone withthe wind
:15. Ben-Hur 4.3
:23. Gone with the wind 4.1
:
11
3. From opinion ratings to pairwise queries: PWRMComparative opinions: Pairwise preference elicitation
Based on pairwise queries:
FROMBen-Hur
[0..1][Awful, fairly bad, It’s OK,
Will enjoy, Must see]
Gone withthe wind
:15. Ben-Hur 4.3
:23. Gone with the wind 4.1
:
TO Which movie do you prefer, Ben-Hur or Gone with the wind?
:15. Ben-Hur
:23. Gone with the wind
:
easier for users to state opinions when the queries compare objects in a pairwise fashion…
“… between these two objects, which one do you prefer?”
12
3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (I)
Reputation (opinions aggregation) as an iterative process based on …
12
3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …
12
3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (I)
Knock-Out tournaments:
Reputation (opinions aggregation) as an iterative process based on …
A
B
C
D
A
D
D
Match: pairwise comparison between two entities
Dynamics: every match sent to a set of users that reply to the query
Policies:‣ Entity selection‣ Tournament schedule‣ Users selection‣ Winner determination
13
3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (II)
Policies
Entities selection: which pair of entities should I select to be compared?‣ Mixed: current ranking vs new objects (Exploitation Vs Exploration)‣ Random‣ Domain-dependent: objects with no information/fuzzy positions
Tournament schedule: how to initialize the tournament‣ Random schedule (iterative process)
14
3. From opinion ratings to pairwise queries: PWRMPairwise comparison dynamics (III)
Policies
Users selection: who receives the queries (matches)?‣ Random selection‣Clustering of users by their preferences (representative users)‣ Using (social) network properties: degree distribution, centrality of nodes, …
Winner determination: how to decide which entity wins in a match‣ Voting procedures: preference replies from users count as votes‣ Alternatives: absolute majority / full agreement (voting protocols)‣ If there is no winner, no object gets through the next round
15
3. From opinion ratings to pairwise queries: PWRMComparative aggregation: from matches to a ranking
When: After each match, the ranking is updated (iterative method)How: Adaptation of a method for aggregating partial pairwise comparison results into a ranking (Negahban et al., 2012)
‣Ranking approximation = random walk on G (weighted graph):‣ An edge <ei,ej> if the pair has already been compared‣ The weights define the outcome of the comparisons‣ Random walk uses a transition matrix P where:‣ It moves from state ei to state ej with probability equal to the chance that entity ej is preferred over entity ei
‣ Under these conditions, a vector w is a valid stationary distribution for matrix P (wT
t+1 = wT · P)‣ w defines the scores for each entity => ranking
16
3. From opinion ratings to pairwise queries: PWRMPWRM’s iterative process for building a reputation ranking
Require: a social network G = (U,E,LU , LE)
Require: a subset of E
0 ✓ E entities to be evaluated
1: for t 2 � time do
2: Ei EntitiesSelectionPolicy.selectEntitiesToEvaluate(E
0)
3: KTEi scheduleTournament(Ei)
4: for m 2 matches(KTEi) do
5: nb UsersSelectionPolicy.getUsersToAsk(U)
6: send(m,nb)
7: votes receive()
8: winner WinnerDeterminationPolicy.getWinner(votes)
9: Ri AggregationMechanism.updateRanking(m,winner)
10: setWinnerNextRound(winner,KTE0)
11: end for
12: end for
13: return Ri where E
0are ranked by their estimated reputation
19
4. Towards ranking resources in MOOCsApplying PWRM into MOOCs
‣MOOCs modeled as a Social Network (Online Review Systems)
‣Apply PWRM for ranking learning resources in MOOCs
‣Allowing users (students/teachers) to find the best resources
‣Formalize a MOOC from a peer based system point of view
Idea:
Let M = hU,R,LR, LU i be a MOOC, where:
• U = {u1, . . . , un} is a set of users (teachers or students);
• R = {r1, . . . , rm} is the set of learning resources uploaded in the course;
• LR = {hui, rji/ui 2 U ; rj 2 R} is the set of links among users and re-sources, representing that user ui has uploaded the resource rj in thecourse;
• LU = {huk, rmi/uk 2 U ; rm 2 R} is the set of links also between usersand resources representing that user uk has used the resource rm.
20
4. Towards ranking resources in MOOCsPWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so..
M = hU,R,LR, LU , ranki
• rank : R0 ⇥ O ! {1, . . . , |R0|} is a function in charge of defining a totalordering (ranking) over a subset of resources R0 2 R, taking into accountthe set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rjiand representing a pairwise query sent to a set of users participating inthe MOOC, where learning resources ri and rj are compared.
20
4. Towards ranking resources in MOOCsPWRM as a function
‣ Objective: to query users about resources to give their opinions in order to build a global ranking of resources regarding their reputation, so..
M = hU,R,LR, LU , ranki
• rank : R0 ⇥ O ! {1, . . . , |R0|} is a function in charge of defining a totalordering (ranking) over a subset of resources R0 2 R, taking into accountthe set of opinions O given by users;
• oi 2 O represents an opinion given into a MOOC, modeled as oi = hri, rjiand representing a pairwise query sent to a set of users participating inthe MOOC, where learning resources ri and rj are compared.
rank = PWRM
21
4. Towards ranking resources in MOOCsPWRM algorithm in MOOCs
Require: a MOOC M = hU,R,LR, LU , rankiRequire: a subset of R0 ✓ R of learning resources to be ranked1: for t 2 � time do
2: Ri ResourcesSelectionPolicy.selectResourcesToEvaluate(R0)3: KTRi scheduleTournament(Ri)4: for m 2 matches(KTRi) do5: Ui UsersSelectionPolicy.getUsersToAsk(U)6: send(m,Ui)7: Oi ReceiveOpinions()8: winner WinnerDeterminationPolicy.getWinner(votes,Oi)9: Ranki AggregationMechanism.updateRanking(Oi, winner)
10: promoteResourceWinnerToNextRound(winner,KTRi)11: end for
12: end for
13: return Ranki where the subset R0 of learning resources are ranked by their
reputation
22
4. Towards ranking resources in MOOCsPWRM algorithm in MOOCs: Policies
‣ Resource selection policy:
- resources clustered regarding their typology (e.g. videos, recorded class…)
- regarding the number of opinions received by each resource (lowest/highest number)
- opinions in terms of the result of each match (matches with tight results)
‣ User selection policy:
- taking advantage of the underlying structure generated by interactions between users and resources
‣ Winner determination policy:
- voting theory: simple majority, complete agreement, …
24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems due to difficulty/subjectivity to map opinions into numerical values
Our Approach: PWRM (1) based on comparative(2) preference aggregation in reputation rankings (iterative process - tournaments)(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work
24
Contributions
Current reputation mechanisms
✓ follow a very passive/static and quantitative dependent approach
‣ easy to manipulate
‣ bias problems
Our Approach: PWRM (1) based on comparative opinions, elicited through pairwise preference request(2) preference aggregation in reputation rankings (iterative process - tournaments)(3) applied to MOOCs (ranking learning resources)
5. Conclusions & Future work
25
Future work5. Conclusions & Future work
‣ Adding social network properties:
- cluster users, centrality, betweenness, …
‣ Partial cooperative users:
- incentive mechanisms fostering cooperation (“what do you think users prefer, A or B?”)
‣ Reputation of MOOCs:
- resources = courses, finding opinions in other opinions sites: twitter, Facebook, forums, etc..
‣ Individual recommendation:
- resources/courses: from global reputation ranking to individual recommendations
On the inaccuracy of numerical ratings: A pairwise based reputation
mechanism in MOOCs
July 1st, 2015
Roberto [email protected]
Dpto. Lenguajes y Sistemas InformáticosUNED