Timeline: A Dynamic Hierarchical Dirichlet Process Model for Recovering Birth/Death and Evolution of

Topics in Text Stream(UAI 2010)

Amr Ahmed and Eric P. Xing

Presented by Bo Chen

Dynamic Non-Parametric Mixture Models and The Recurrent Chinese Restaurant Process

(SDM, 2008)

Outiline

• 1. Dirichlet process (DP)• 2. Temporal Dirichlet process mixture model

and the recurrent Chinese restaurant process (RCRP)

• 3. Hierarchical Dirichlet process (HDP)• 3. Dynamic hierarchical Dirichlet process • 4. Inference• 5. Experimental results• 6. Conclusions

Dirichlet Process (DP)

If we integrate out G,

The posterior of G

Perspectives: • Chinese restaurant process or Polya urn model• Stick-breaking process• A limit of finite mixture models.• Normalized Gamma process

The Temporal Dirichlet Process Mixture ModelIntuition: The TDPM seek to model cluster parameters evolution over time using any time series model, and to capture cluster popularity evolution over time via the rich-gets-richer effect, i.e. the popularity of cluster k at time t is proportionable to how many data points were associated with cluster k at time t-1.

Notations:

Recurrent Chinese Restaurant Process (RCRP)

Modeling Higher-Order DependenciesOne problem with the above construction of the TDPM is that it forgets too quickly especially when it models cluster popularity at time t + 1 based on its usage pattern at time t, while ignoring all previous information before time epoch t. Moreover, once a cluster is dead, it can no longer be revived again. Clearly, in some applications one might want to give a slack for a cluster before declaring it dead.

Non-Parametric Dynamic Topic Model

Cons: The above simple DTM allows each document x to be generatedfrom a single component (topic), thus making it suboptimal in modeling multi-topic documents.

Experimental Results

Hierarchical Dirichlet Processes (HDP)

Recurrent Chinese Restaurant Franchise Process (RCRFP)

Recurrent Chinese Restaurant Franchise Process (RCRFP)

Building Infinite Dynamic Topic Models (iDTM)

Use the RCRF process as a prior over word assignment to topics in a mixed-membership model, we get the infinite dynamic topic model (iDTM).

Inference via Gibbs Sampling (1)

In order to calculate the last two lines, the authors use the following approximation:

Inference via Gibbs Sampling (2)

Sample the whole topic chain via M-H:

Simulation ResultsEpochs: 20; Vocabulary: 16; Topics: 8; Docs in each epoch: 100; Words in each doc: 50

Timeline of the NIPS Conference (1)Years: 1987-1999; Vocabulary: 3379; Documents: 1740

Timeline of the NIPS Conference (2)

Timeline of the NIPS Conference (2)

Timeline of the NIPS Conference (3)

Conclusions and Future Works

• 1. To addressed the problem of modeling time-varying document collections topic model, iDTM, that can adapt the number of topics, the word distributions of topics, and the topics' trend over time.

• 2. Extend the Gibbs sampler to sample all the hyperparameters of the model.

• 3. Extend the model to evolve an HDP at various levels, for instance, lower levels might correspond to conferences, and the highest level to time. This framework will enable us to understand topic evolution within and across different conferences or disciplines.