1

Time-Sensitive Web Image Ranking and Retrieval via Dynamic Multi-Task

Regression

Gunhee Kim Eric P. Xing

School of Computer Science, Carnegie Mellon University

February 6, 2013

Image Ranking and Retrieval

Goal: Find the images for a given query

ex. Cardinal Text-based image retrieval

2

Image Ranking and Retrieval

ex. Cardinal

northern_cardinal_glamour.jpgFile name

http://www.allaboutbirds.org/guide/Northern_Cardinal/id

Url

3

Text-based image retrieval

• Ambiguity and noise due to mismatch.

• Scalable and successful so far

Goal: Find the images for a given query

Recent Image Ranking and Retrieval

Various efforts to improve text-based image search

User relevance feedback [Wang et al. CVPR 11]

Text-based search by apple

chosen by a user

Reranking on visual features

Pseudo-relevance feedback[Liu et al. CVPR 11]

Human labeled training data[Yang et al. MM10]

Image click data [Jain et al. WWW11]

4

5

Time-Sensitive Image Ranking and Retrieval

From experiments of 7.5 millions of Flickr images of 30 topicswe found three good reasons …

Discovery of temporal patterns of Web image collections

• [D08] Dakka et al. CIKM 2008• [M09] Metzler et al. SIGIR 2009• [K10] Kulkani et al, WSDM 2011

• [V11] Amodeo et al, CIKM2011• [R12] Radinsky et al, WWW 2012• …..

No previous work using temporal info on image retrieval

[Related work] Exploring temporal dynamics of Web queries• Popular search keywords and relevant documents change over time.• ex) Keyword search, Product search, News recommendation

6

Why Time-Sensitive Image Retrieval? (1/3)

1. Knowing when search takes place is useful to infer users' implicit intents.

Cardinal: (1) the red bird in America.

Fall to Winter (Sep. ~ Feb.)

Google

Bing

(2) Arizona cardinals (football) (3) St. Louis cardinals (baseball)

Spring to Fall (Mar. ~ Oct.)

• Severely redundant. Almost identical all year long.

7

Why Time-Sensitive Image Retrieval? (1/3)

1. Knowing when search takes place is useful to infer users' implicit intents.

at May 4, 2009

at Feb. 7, 2009 Football

Google

Bing

Ourresults

baseball

• Diversity can make search interesting.

8

Why Time-Sensitive Image Retrieval? (2/3)

2. Timing suitability can be used as a complementary attribute to relevance.

Google

Bing

at May 4, 2009

at Feb. 7, 2009

Ourresults

• There are so many almost equally good images.Background: snow

Background: Green Baby birds or eggs

9

Why Time-Sensitive Image Retrieval? (3/3)

3. Temporal information is synergetic in personalizedimage retrieval.

Louisville Men's College Basketball

At Nov. 7, 2009 for user 30033302

Each user’ term usages are relatively stationary, and predictable once they are learned.

10

Algorithm

Regularized multi-task regression on multivariate point process

• Goal: Scalably learn temporal models for each topic keyword.

• Multi-task framework: allows multiple image descriptors.

• Several regularization schemes

• Personalization by locally-weighted learning

11

Thank you !Stop by our poster!

12

Multivariate Point Process Models

Given a stream of hornet pictures up to T

Clustering by descriptor 1 Clustering by descriptor 2

Time t1 t2 t3 t5 t6 t7 t9 t10

1st descriptor (v1) 2nd descriptor (v2)

13

Regularized GLM on Point Processes

Given a stream of hornet pictures up to T

Time t1 t2 t3 t5 t6 t7 t9 t10

Formulate a regression between occurrence rates and covariates.

Covariates: any likely factors to be associated with image occurrence (ex. Time, season, and other external events)

Compute sparse regularized MLE solutions For each visual cluster, we select only a small number of strong factors.

(v1,v2) (3, 2) (3, 2) (3, 2) (2, 1) (1, 3) (1, 4) (2, 3) (2, 1)

14

A Toy Example of Image Reranking

Peaked in summer

(Aquarium)

(Sea tour)

(Ice hockey) Peaked in January

Covariates: only year and months

Stationary