Competence Center Information Retrieval & Machine Learning

10. April 2023

UMAP‘13 Doctoral Consortium

Evaluation of Cross-Domain News Article Recommendations

Benjamin Kille

210. April 2023

Agenda

► Problem description► Challenges in News Article Recommendation

Sparsity Dynamic item collection Evaluation

► Research Questions► Data outline► Preliminary results► Conclusions► Next steps

310. April 2023

Problem description

► Information overload amount of on-line accessible news articles increases limited user perception limited time capacity

► Solution: Recommender System filtering news articles with respect to relevance/utility

► Special challenges for news recommender systems Sparsity Dynamics

► General challenges for recommender systems Evaluation strategy

410. April 2023

Problem formalization

► Set of domains each referring to a publisher► Each domain comprises a set of users , along with a set of items ► We observe preferences as users interact with items ► Let denote the set of items the target user interacted with, and

denote the set of users who interacted with a specific item► We define the recommendation task as

Rank the set of items according to estimated relevance/utility

510. April 2023

Sparsity

► Cold-Start Problem ( and/or )Cacheda, F. et al., 2011. Comparison of collaborative filtering algorithms. ACM Transactions on the Web, 5(1), pp.1–33.► Providing preferences better recommendations (trade-off)Cremonesi, P., Milano, P. & Turrin, R., 2012. User Effort vs . Accuracy in Rating-based Elicitation. In 6th ACM Conferene on Recommender Systems. pp. 27–34.► News websites usually avoid log-in requirements to attract

larger user segments Incomplete user profiles Inconsistent user profiles

610. April 2023

Dynamics

► News dynamic contentBillsus, D. & Pazzani, M.J., 2007. Adaptive News Access. In P. Brusilovsky, A. Kobsa, & W. Nejdl, eds. The Adaptive Web. Springer, pp. 550–570.► Unlike music or movies rarely re-consumed► For instance: Deutsche Presse Agentur (DPA)

750 messages 220k words 1,5k images

http://www.dpa.de/Zahlen-Fakten.152.0.html

710. April 2023

Evaluation

► Strategy on-line: A/B testing (user-centric) off-line: data set (data-centric)

► Numerous facets utility relevance novelty serendipity …

► Dependending on the model formulation preference prediction (requires numerical preference data) item ranking

810. April 2023

Evaluation (cont‘d)

► on-line

Dispatcher

● recommendation request click

●

●

●

●

●

●

CTR  = #  clicks#  requests

910. April 2023

Evaluation (cont‘d)

► off-line (replay)

Dispatcher

● recommendation request click

● ● ● ● ●●

CTR  = #  clicks#  requests

[𝑖1 ,𝑖2 , 𝑖3 ] [𝑖1 ,𝑖2 , 𝑖3 ]

? ?

Li, L. et al., 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining - WSDM ’11. p. 297.

1010. April 2023

Cross-domain setting

D1

D2

U

U

I

ID1

D2

U

U

I

I

D1

D2

U

U

I

I

D1

D2

U

U

I

I

no o

verla

p

Use

r ove

rlap

Item

ove

rlap

full

over

lap

Cremonesi, P., Tripodi, A. & Turrin, R., 2011. Cross-Domain Recommender Systems. In 2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, pp. 496–503.

1110. April 2023

Research Questions

► How can other publishers' user interactions contribute to

decrease sparsity for the target publisher?

► What characteristics must recommender algorithms exhibit to

successfully cope with dynamically changing item collections?

► How to evaluate cross-domain recommender systems with

dynamically changing item collections? How do standard

evaluation metrics compare to the observed clicks?

1210. April 2023

Data outline

► > 1-2M impressions by 12 publishers (general news, local news, finance, information technology, sports, etc.) on a daily basis

► user features such as browser ISP OS device

► news article features such as title text URL Image

► http://www.dai-labor.de/en/irml/epen/

►Real interactions with actual users!

1310. April 2023

Preliminary results

► Sparsity► Histogram of the relative frequency of user interactions

1410. April 2023

Preliminary results (cont‘d)

► Dynamics

1510. April 2023

Preliminary results (cont‘d)

► Popularity

1610. April 2023

Conclusions

► News recommender systems must handle enormous sparsity► Dynamic item collections

Continuously, news items enter the system Continuously, items stop to be relevant/read

► We observe a serious popularity bias► We suppose that considering user preferences on additional

domains will decrease sparsity ► We suppose that identifying general preference pattern will

allow us to deal with item collection dynamics

1710. April 2023

Next steps

► Implementation of existing cross-domain recommender algorithms

► Evaluating recommender algorithms with respect to CTR novelty diversity

► Investigate UI effects► Analyze applicability of context-sensitive recommendations► User/Item clustering to speed-up computation time

1810. April 2023

Thank you for the attention!

Questions???

1910. April 2023

Announcement: NRS 2013

► International News Recommender Systems Workshop and Challenge► In conjunction with ACM RecSys 2013IMPORTANT DATES

July 21, 2013 paper submission deadline July 1, 2013 data set release August 15, 2013 on-line challenge kick-off

HIGHLIGHTS Access to a real recommender system Real-time requirements Big Data Cross-domain Implicit feedback

Website: https://sites.google.com/site/newsrec2013/homeTwitter: @NRSws2013

Competence Center Information Retrieval &Machine Learning

20

www.dai-labor.de

FonFax

+49 (0) 30 / 314 – 74+49 (0) 30 / 314 – 74 003

DAI-Labor

Technische Universität BerlinFakultät IV – Elektrontechnik & Informatik

Sekretariat TEL 14Ernst-Reuter-Platz 710587 Berlin, Deutschland

10. April 2023

Benjamin KilleResearcher / PhD student

benjamin.kille@dai-labor.de74128