ELIS – Multimedia Lab

Tom De Nies Supervised by: Erik Mannens and Rik Van de Walle

Ghent University – iMinds – MMLab

http://users.ugent.be/~tdenies

@TomDeNies (this presentation has links, click them to find out more!)

Assessing Content Value for Digital Publishing through Relevance and

Provenance-based Trust

Presented at ISWC 2013 Doctoral Consortium

2

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Digital publishers are facing an information overload … but they get less time to deal with it!

They need a way to select content that is valuable to themselves, and to their target audience!

Problem Description

3

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Most work on quality assessment of machine-observed data

Only limited number of works[1] for human-generated content

State of the Art

Very diverse: worthiness, novelty, readability, pageranking, …

No integrated, automated approach to assess content value through multiple aspects.

4

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Research Questions and Hypotheses

How can we automatically assess the value of content on the Web?

Relevance assessment? Trustworthiness assessment?

Through provenance + reputation?

Retrieval? Reconstruction?

Through context and semantic similarity

5

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Proposed Approach

A contextual model is used to generate the content's relevance, reconstruct its provenance, and assess its trustworthiness.

6

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Content Value depends on end-user and use case

makes a general evaluation infeasible

representative use cases! E.g. online news

Relevance: benchmarks and established metrics (e.g. MediaEval)

Provenance reconstruction:

Currently no benchmarks or gold standard datasets!

-> used preliminary dataset -> make our own (e.g. with Git2PROV)

Trust & Value: human evaluation & crowdsourcing (e.g. Amazon MT)

Evaluation Plan

7

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Preliminary Results

- Linking videos based on textual content - Supervised method (2012): MAP 0.17 - Unsupervised method (2013): MAP 0.0375

HOWEVER: MAP is wrong metric! - Precision: 35% of top-10 links relevant

[APROVeD] - 420 news items - Reconstruct their provenance

(find the original source) - Using our clustering method - 73% of sources found - With 68.2% precision

[AVALON]

- 224 news abstracts - Newsworthiness criteria identified

(from expert literature) - Detect these criteria automatically - Detected with 83,9% precision

8

ELIS – Multimedia Lab

Assessing Content Value for Digital Publishing through Relevance and Provenance-based Trust Tom De Nies 22/10/2013

Reflections

Trust Relevance

valu

e

Combination Fine-grained Cross-domain

Benchmarks

- Our points of attention:

+ Our strong points:

Annotate uncertainty! [2] Supervised vs. unsupervised