Semantic Stability in Social Tagging Streams

Claudia Wagner, Philipp Singer, Markus Strohmaier and Bernardo Huberman

2

Folksonomies

Ontologies

Formal, shared and stableNot formal but shared

and stable?

4

1970

1990

2010

http://schwarzenegger.com/

5

How can we measure semantic stability?

How can we compare the semantic stabilization process in different systems?

What impacts semantic stability?

6

Measuring Semantic StabilityState of the Art

• Relative tag proportions per resource become stable with increasing number of tag assignments [Golder and Huberman, 2006]

• KL-divergence of rank-ordered tag frequency distribution per resource at different time points converges towards zero [Halpin et al., 2007]

• Power Law distributions [Cattuto et al., 2006] – Scale invariance property ensures that regardless how large the system grows the shape of the distribution stays the same

7

Some Limitations• Don’t allow comparing the semantic stabilization

process of different systems • Prune tag distributions to top-k tags

– Cannot handle non-conjoint lists of tags• Random tagging process also produces “stable”

description– Tag assignment at timepoint t+1 has less impact on the

tag distribution of a resource than a tag at timepoint t

8

ExampleKL-Divergence

• KL-divergence converges towards zero.

• But random baseline also converges towards zero if we assume a constant tagging rate.

• We do not always know the top k tags!

9

ExampleRelative Tag Proportion

Intuition and Approach• Some descriptors are

more important than others.

• Ranking of (top) descriptors remains stable over time

• All descriptors are equally important.

• Ranking of (top) descriptors changes over time

Schwarz

eneg

geract

or

terminato

r

Hollywood

bodybuild

ing0

0.10.20.3

P(T)

Schwarz

eneg

ger

actor

terminato

r

Hollywood

bodybuild

ing0

0.10.20.3

00.20.4

P(T)

00.20.4

stable

less stable

tn tn+m

tn tn+m

Intuition and Approach• Some descriptors are

more important than others.

• Ranking of (top) descriptors remains stable over time

• All descriptors are equally important.

• Ranking of (top) descriptors changes over time

Schwarz

eneg

geract

or

terminato

r

Hollywood

bodybuild

ing0

0.10.20.3

P(T)

Schwarz

eneg

ger

actor

terminato

r

Hollywood

bodybuild

ing0

0.10.20.3

stable

less stable

tn tn+m

tn tn+m

gove

rnor

California

republican CA

00.20.4

00.20.4

P(T)

13

Requirements• Rank agreement of the descriptors of a resources

over time

• Weighted rank agreement

• Non-conjoint lists of descriptors

• Random Baseline

14

Rank Biased Overlap (RBO)[Webber et al., 2010]

• RBO falls in the range [0, 1], where 0 means disjoint, and 1 means identical

• p lies between 0 and 1 and determines how steep the decline in weights is

• The smaller p, the more top-weighted the metric

15

Example

novel

fiction sf

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

novel sf

fiction

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

Overlap at depth 1 = 1

P(T) P(T)

tntn+m

16

Example

novel

fiction sf

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

novel sf

fiction

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

Overlap at depth 2 = 0.5

P(T) P(T)

tntn+m

17

Example

novel

fiction sf

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

novel sf

fiction

book

London

00.05

0.10.15

0.20.25

0.30.35

0.4

Overlap at depth 3 = 1

P(T) P(T)

tntn+m

18

Effect of the Paramter p

19

Tie correction for Rank Biased Overlap

• RBO does not penalize ties• We want to penalize ties since they show that users have

not agreed on a ranking

• Sum only over those depths which occur in at least one of the two rankings

Same concordant pairs: (A,D) and (B,D) and (C,D)

A B C D0

102030405060708090

C B A D0

102030405060708090

RBOorig = 0.2RBOmod= 0.2

A B C D0

102030405060708090

A B C D0

102030405060708090

RBOorig = 0.34RBOmod= 0.17

No Ties Ties

tn tn+m tn tn+m

R1 R2

A B C D C B A D A B C D C B A D

Freq

uenc

y

Freq

uenc

y

23

Semantic Stabilization on a Resource Level

• Tag distributions of Twitter users become semantically stable between 1k and 2k tag assignments

• The RBO values of random tagging distributions increase slower and are significantly lower

24

Semantic Stabilization on a System Level

• How can we compare the semantic stabilization process in different systems?

• We call a resource description semantically stable after tn+m tag assignments, if the RBO value between its tag distribution at point tn and tn+m is equal or greater than k.

25

Semantic Stabilization on a System Level

After 1250 tag assignments 90% of all resources have a stability above 0.61

26

Empirical StudyTwitter

Medium level of semantic stability is reached after 1k-2k tag assignments

27

Empirical StudyTwitter and Delicious

Tag streams in Delicious stabelize faster and sign.

higher than in Twitter

28

Empirical StudyTwitter, Delicious and LibraryThing

Same is true for tag streams of books in

LibraryBook

29

Empirical StudyRandom Baseline

30

Difference between tag and word streams?

31

What causes semantic stability?

• Simulations based on the epistemic tagging model [Dellschaft and Staab, 2008].

• Use parameter I as imitation rate and produce tag distributions for I=0, 0.1, ... 1

33

What causes stability?

Medium levels of semantic stability are

reached after 1k-2k tag assignments

34

What causes stability?

Same is true if we combine BK and imitation

when BK is dominant

35

What causes stability?

If imitation and BK are combined an imitation is dominant higher levels of

semantic stability are reached faster

36

What causes stability?

• Combination of shared background knowledge and imitation behaviour (where imitation is more important) leads to the fastest and highest stabilization.

• Natural language systems show similar stabilization as social tagging systems where no imitation is supported

37

Conclusions & Implications• Attempt to formalize semantic stability in social streams• Novel approach to measure and compare the semantic

stabilization process in different social streams

Why is that useful?• Identify social streams (e.g. tag stream of URL or word stream

of hashtags) which are semantically stable – Extract shared and agreed-upon semantic knowledge from social

streams• Select systems that provide semantically stable streams

40

References• D. Bollen and H. Halpin. The role of tag suggestions in folksonomies. In Proceedings of the 20th ACM

conference on Hypertext and hypermedia, HT ’09, pages 359–360, New York, NY, USA, 2009. ACM.• C. Cattuto, Semiotic dynamics on social tagging communities. The European Physical Journal C - Particles

and Fields August 2006, Volume 46, Issue 2 Supplement, pp 33-37• A. Clauset, C. R. Shalizi, and M. E. J. Newman. Power-law distributions in empirical data. SIAM Rev.,

51(4):661–703, Nov. 2009.• K. Dellschaft and S. Staab. An epistemic dynamic model for tagging systems. In HT ’08: Proceedings of the

nineteenth ACM conference on Hypertext and hypermedia, pages 71–80, New York, NY, USA, 2008. ACM.• S. Golder and B. A. Huberman. Usage patterns of collaborative tagging systems. Journal of Information

Science, 32(2):198–208, April 2006.• H. Halpin, V. Robu, and H. Shepherd. The complex dynamics of collaborative tagging. In Proceedings of the

16th international conference on World Wide Web, WWW ’07, pages 211–220, New York, NY, USA, 2007. ACM.

• A. Hotho, R. Jäschke, C. Schmitz, and G. Stumme. Bibsonomy: A social bookmark and publication sharing system. In Proceedings of the Conceptual Structures Tool Interoperability Workshop at the 14th International Conference on Conceptual Structures, pages 87-102, 2006.

• C. T. Kello, G. D. A. Brown, R. Ferrer-i Cancho, J. G. Holden, K. Linkenkaer-Hansen, T. Rhodes, and G. C. Van Orden. Scaling laws in cognitive sciences. Trends in Cognitive Sciences, 14(5):223{232, May 2010.

• W. Webber, A. Moat, and J. Zobel. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4):20:1{20:38, Nov. 2010.

41

Thank you!

Special thanks to my collaborators (2/3 of them are here):

42

Limitations and Future Work• RBO measures ranking but ignores the differences

in the frequencies

• Decay function to weight tag counts– old tag assignments are less important than new ones

• Number and diversity of users who tag a resource might impact the semantic stabilization process

43

Alternatives to RBO• Unweighted and conjoint measures

– Kendall tau, Spearman rho• Weighted and conjoint measures

– Weighted Kendall tau• Unweighted and non-conjoint measures

– Intersection metric• Weighted and conjoint

– Cumulative overlap at increasing depths

44

Dataset

45

Categories of Semantically Unstable Resources

• Entity to which a resource refers changes• Resource (i.e. website) changes • Entity/Topic to which a resource refers is controversial

– website refers to controversial entity/topic on which different viewpoints exist

• External conditions which impact viewpoints on entity/topic change– Website remains stable but viewpoint of taggers on the

entity or topic related with the site change

46

Relative Tag Proportion [Golder and Huberman, 2006]

tn+mtn

stableless stable

47

Relative Tag Proportion [Golder and Huberman, 2006]

48

KL-Divergence [Halpin et al., 2007]

• KL divergence between the rank-ordered frequency distribution of the top 25 tags at different time points

tn+mtn

stableless stable

49

KL-Divergence

50

Power Law [Cattuto, 2006]

• Is the rank-ordered frequency distribution a power law distribution?

• Is the frequency y of a tag inversely proportional to it's rank r?

tn+mtn

51

Power Law [Cattuto, 2006]

• Is it really power law?– Very likely yes according to the maximum

likelihood estimator and Kolmogorov-Smirnov statistic [Clauset et al., 2010]

– Estimate alpha and xmin over some reasonable range

– Compare power law fit to the fit of the exponential function, the lognormal function and the stretched exponential (Weibull) function. Use the log-likelihood ratios to indicate which fit is better.

– We do not find significant differences between the power law fit and the lognormal fit

52

RBO

53

Stablilization going beyond Baseline Stability

54

Stablilization not going beyond Baseline Stability