ec-tel 2016: which algorithms suit which learning environments?

W I S S E N T E C H N I K L E I D E N S C H A F T

http://kti.tugraz.at

Which Algorithms Suit Which LearningEnvironments? A Comparative Study ofRecommender Systems in TELS. Kopeinik, D. Kowald, E. Lex,Graz University of Technology, AustriaKnowledge Technologies Institute, Cognitive Science SectionOctober 24, 2016

2 Outline

A study comparing a variety of recommendation strategieson 6 empirical TEL datasetsConsidering 2 application casesFindings:

The performance of algorithms strongly depends on thecharacteristics of the datasetsThe number of users per resource is a crucial factorA hybrid combination of a cognitive-inspired and apopularity based approach works best for tagrecommendations

S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016

3

Introduction

Recommender Systems (RS)

... are software components that suggest items of interest or ofrelevance to a user’s needs [Kon, 2004, Ricci et al., 2011].

Recommendations are related to decision making processes:

Ease information overloadSales assistance

Popular examples: Amazon.com, YouTube, Netflix, Tripadvisor,Last.fm, ...


4

Introduction

RS in Technology Enhanced Learning

. . . are adaptational tasks to fit the learner’sneeds [Hamalainen and Vinni, 2010].

Typical recommendation services include:

Peer recommendationsActivity recommendationsLearning resource recommendationsTag recommendations


5

Introduction

Motivation

So far there are no generally suggested or commonlyapplied recommender system in TEL [Khribi et al., 2015]Learning data is sparse, especially in informal learningenvironments [Manouselis et al., 2011]Available data varies greatly, but available implicit usagedata typically includes learner ids, information aboutlearning resources, timestamps [Verbert et al., 2012]

Research Question 1

How accurate do state-of-the-art resource recommendation algorithms,using only implicit usage data, perform on different TEL datasets?


6

Introduction

Motivation

Lack of learning object meta-dataShifting the task to the crowd [Bateman et al., 2007]

Tagging is a mechanism to collectively annotatelearning objects [Xu et al., 2006]fosters reflection and deep learning[Kuhn et al., 2012]needs to be done regularly and thoroughly

Research Question 2

Which computationally inexpensive state-of-the-art tag recommendationalgorithm performs best on TEL datasets?


7

Evaluation

Study Setup

Evaluation of two recommender application cases

a) Recommendation of learning resourcesb) Recommendation of tags

For each dataset

1. Sort user activities in chronological order(timestamp)

2. Split data into training and test set

Application Training Set Test SetResources 80% 20%Tags n-1 1


8

Evaluation

Algorithms

Well-established, computationally inexpensive tag andresource recommendation strategies

Most Popular (MP) [Jaschke et al., 2007]. . . counts frequency of occurrence

Collaborative Filtering (CF) [Schafer et al., 2007]. . . calculates neighbourhood of users or items

Content-based Filtering (CB) [Basilico and Hofmann, 2004]. . . calculates similarity of user profiles and itemcontent


9

Evaluation

Algorithms

Approaches that have been suggested in the context of TEL

Usage Context-based Similarity (UCbSim)[Niemann and Wolpers, 2013]. . . calculates item similarities based onco-occurrences in user sessions

Base Level Learning Equation (BLLAC) [Kowald et al., 2015]. . . mimics human semantic memory retrieval as afunction of recency and frequency of tag use

Sustain [Seitlinger et al., 2015]. . . simulates category learning as a dynamicclustering approach


10

Evaluation

Metrics [Marinho et al., 2012, Sakai, 2007]

Recall (R)The proportion of correctly recommended items toall items relevant to the user.

Precision (P)The proportion of correctly recommended items toall recommended items.

F-measure (F)The harmonic mean of R and P.

Discounted Cumulative Gain (nDCG)A ranking quality metric that calculates usefulnessscores of items based on relevance and position.


11

Evaluation

Datasets

Six datasets from different application domains:

BibSonomy, CiteULike (Social Bookmarking)KDD15 (MOOCs)Mace, TravelWell (Open Social Learning)Aposdle (Workplace Learning)

|P| |U| |R| |T | |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp

BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0


12

Discussion

Results: Resource Recommender

Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU

BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541

CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717

KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608

TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220

MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215

AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0


13

Discussion

Datasets

|ATr | |ATpr | |ARu| |AUr |BibSonomy 4.1 0 33.8 3CiteULike 3.5 0 14.7 2.5KDD15 0 1.8 17.2 49.4TravelWell 3.5 1.7 26.5 1.4MACE 2.4 0 36.7 1.9Aposdle 0 1.1 74.8 1


14

Discussion

Results: Resource Recommender

Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU

BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541

CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717

KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608

TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220

MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215

AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0


15

Discussion

Findings

1. The performance of most algorithms (F@5) stronglycorrelates (.958) with the average number of users perresource

2. Good performance values can only be reached for theMOOCs dataset

3. Algorithms based on implicit usage data don’t satisfy therequirements of small-scale environments like Aposdle

4. The performance of algorithms strongly depends on thecharacteristics of the datasets


16

Discussion

Results: Tag Recommender

Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR

BibSonomyP@5 .1991 .0572 .2221 .2066 .2207 .2359F@5 .2535 .0688 .2814 .2606 .2795 .2987nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022

CiteULikeP@5 .1687 .0323 .1829 .1698 .1897 .2003F@5 .2310 .0427 .2497 .2315 .2597 .2738nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140

TravelWellP@5 .1000 .0366 .1333 .0800 .1300 .1400F@5 .1376 .0484 .1724 .1096 .1708 .1872nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615

MACEP@5 .0576 .0173 .0618 .0631 .0812 .0812F@5 .0799 .0259 .0869 .0893 .1114 .1138nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734


17

Discussion

Conclusion

Learning Resource Recommendation:

A dense user resource matrix is crucialThe performance of most algorithms (F@5) strongly correlates (.958) with the averagenumber of users per resource

For small-scale learning environments, a thoroughdescription of user and learning resources is necessaryAlgorithms based on implicit usage data don’t satisfy the requirements of small-scaleenvironments like Aposdle

MOOCs are not representative for other, typically sparseTEL environmentsGood performance values can only be reached for the MOOCs dataset

Tag Recommendation:BLLAC+ MPR clearly outperforms the remaining algorithmsMPU,R, an alternative for runtime-sensitive environments


18

Discussion

Acknowledgement

Special thanks are dedicated to Katja Niemann who providedus with the datasets MACE and TravelWell. For the KDD15data, we would like to gratefully acknowledge the organizers ofKDD Cup 2015 as well as XuetangX for making the datasetsavailable. This work is funded by the Know-Center and theEU-IP Learning Layers (Grant Agreement: 318209). TheKnow-Center is funded within the Austrian COMET Programunder the auspices of the Austrian Ministry of Transport,Innovation and Technology, the Austrian Ministry of Economicsand Labor and by the State of Styria.


19

Discussion

References I

[Kon, 2004] (2004).

Introduction to recommender systems: Algorithms and evaluation.

ACM Trans. Inf. Syst., 22(1):1–4.

[Basilico and Hofmann, 2004] Basilico, J. and Hofmann, T. (2004).

Unifying collaborative and content-based filtering.

In Proc. of ICML’04, page 9. ACM.

[Bateman et al., 2007] Bateman, S., Brooks, C., Mccalla, G., and Brusilovsky, P. (2007).

Applying collaborative tagging to e-learning.

In Proceedings of the 16th international world wide web conference (WWW2007).

[Hamalainen and Vinni, 2010] Hamalainen, W. and Vinni, M. (2010).

Classifiers for educational data mining.

Handbook of Educational Data Mining, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series, pages 57–71.

[Jaschke et al., 2007] Jaschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., and Stumme, G. (2007).

Tag recommendations in folksonomies.

In Proc. of PKDD’07, pages 506–514. Springer.


20

Discussion

References II

[Khribi et al., 2015] Khribi, M. K., Jemni, M., and Nasraoui, O. (2015).

Recommendation systems for personalized technology-enhanced learning.

In Ubiquitous Learning Environments and Technologies, pages 159–180. Springer.

[Kowald et al., 2015] Kowald, D., Kopeinik, S., Seitlinger, P., Ley, T., Albert, D., and Trattner, C. (2015).

Refining frequency-based tag reuse predictions by means of time and semantic context.

In Mining, Modeling, and Recommending’Things’ in Social Media, pages 55–74. Springer.

[Kuhn et al., 2012] Kuhn, A., McNally, B., Schmoll, S., Cahill, C., Lo, W.-T., Quintana, C., and Delen, I. (2012).

How students find, evaluate and utilize peer-collected annotated multimedia data in science inquiry with zydeco.

In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3061–3070. ACM.

[Manouselis et al., 2011] Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., and Koper, R. (2011).

Recommender systems in technology enhanced learning.

In Recommender systems handbook, pages 387–415. Springer.


21

Discussion

References III

[Marinho et al., 2012] Marinho, L. B., Hotho, A., Jaschke, R., Nanopoulos, A., Rendle, S., Schmidt-Thieme, L., Stumme, G.,and Symeonidis, P. (2012).

Recommender systems for social tagging systems.

Springer Science & Business Media.

[Niemann and Wolpers, 2013] Niemann, K. and Wolpers, M. (2013).

Usage context-boosted filtering for recommender systems in tel.

In Scaling up Learning for Sustained Impact, pages 246–259. Springer.

[Ricci et al., 2011] Ricci, F., Rokach, L., and Shapira, B. (2011).

Introduction to recommender systems handbook.

Springer.

[Sakai, 2007] Sakai, T. (2007).

On the reliability of information retrieval metrics based on graded relevance.

Information processing & management, 43(2):531–548.


22

Discussion

References IV

[Schafer et al., 2007] Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. (2007).

Collaborative filtering recommender systems.

In The adaptive web, pages 291–324. Springer.

[Seitlinger et al., 2015] Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Ley, T., and Lex, E. (2015).

Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics.

arXiv preprint arXiv:1501.07716.

[Verbert et al., 2012] Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (2012).

Dataset-driven research to support learning and knowledge analytics.

Educational Technology & Society, 15(3):133–148.

[Xu et al., 2006] Xu, Z., Fu, Y., Mao, J., and Su, D. (2006).

Towards the semantic web: Collaborative tag suggestions.

In Collaborative web tagging workshop at WWW2006, Edinburgh, Scotland.


ec-tel 2016: which algorithms suit which learning environments?

Internet