learning with local consistency - 浙江大学cad&cg
TRANSCRIPT
![Page 1: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/1.jpg)
Deng Cai (蔡登)
College of Computer ScienceZhejiang University
Learning with Local Consistency
Chinese Workshop on Machine Learning and Applications 20101
![Page 2: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/2.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
What is Local Consistency?
2
![Page 3: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/3.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency Assumption
3
A
BC
![Page 4: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/4.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency Assumption
Put edges between neighbors (nearby data points)
Two nodes in the graph connected by an edge share similar properties.
4
![Page 5: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/5.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency Assumption
5 M. Belkin, and P. Niyogi. Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering, NIPS 2001.
![Page 6: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/6.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency and Manifold Learning
6
Manifold learning
We only need local consistency
![Page 7: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/7.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
How to use the local consistency idea?
7
![Page 8: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/8.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency in Semi-Supervised Learning
8 M. Belkin, P. Niyogi, and V. Sindhwani. Manifold Regularization: a Geometric Framework for Learning from Labeled and Unlabeled Examples, Journal of Machine Learning Research, 7(Nov):2399-2434, 2006.
![Page 9: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/9.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Manifold Regularization
9 M. Belkin, P. Niyogi, and V. Sindhwani. Manifold Regularization: a Geometric Framework for Learning from Labeled and Unlabeled Examples, Journal of Machine Learning Research, 7(Nov):2399-2434, 2006.
![Page 10: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/10.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
How to use the local consistency idea?
Matrix factorizationNon-negative matrix factorization
Topic modelingProbabilistic latent semantic analysis
ClusteringGaussian mixture model
10
![Page 11: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/11.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Matrix Factorization (Decomposition)
approximation left factor right factor
11
![Page 12: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/12.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Matrix Factorization (Decomposition)
111 21
212 22
13 23 3
1 2
n
n
n
m m nm
xx xxx x
x x x
x x x
1 1 2 2i i i ki kv v v
x u u u
m
n111
212
13 3
1
k
k
k
m km
uuuu
u u
u u
11 12 1
1 2
n
k k kn
v v v
v v v
m
nk
k
12
![Page 13: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/13.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Singular Value Decomposition
Orthonormal columns
Singular values (ordered)
C. Eckart, G. Young, The approximation of a matrix by another of lower rank. Psychometrika, 1, 211-218, 1936.
![Page 14: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/14.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
14
The LSA via SVD can be summarized as follows:
Document similarity
Folding-in queries
Latent Semantic Analysis (Indexing)
documents
...
...
terms
...
LSA documentvectors
... LSA termvectors=
< , >
M. Berry, S. Dumais, and G. O'Brien. Using linear algebra for intelligent information retrieval. SIAM Review, 37(4):573-595, 1995.
![Page 15: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/15.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Non-negative Matrix Factorization
15 D. D. Lee and H. S. Seung, Algorithms for non-negative matrix factorization, NIPS 13, pp. 556-562, 2001.
![Page 16: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/16.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Locally Consistent NMF
1 1 2 2i i i ki kv v v
x u u u
1 1 2 2j j j kj kv v v
x u u u
Neighbor: prior knowledge, label information, p-nearest neighbors …
16 D. Cai, X. He, J. Han, and T. Huang, Graph regularized Non-negative Matrix Factorization for Data Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear.
![Page 17: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/17.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Locally Consistent NMF
1 1 2 2i i i ki kv v v
x u u u
1 1 2 2j j j kj kv v v
x u u u
17 D. Cai, X. He, J. Han, and T. Huang, Graph regularized Non-negative Matrix Factorization for Data Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear.
![Page 18: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/18.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Objective Function
18 D. Cai, X. He, J. Han, and T. Huang, Graph regularized Non-negative Matrix Factorization for Data Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear.
GNMF:
Graph regularized NMF
![Page 19: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/19.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Clustering Results
Please check our papers for more details.
http://www.zjucadcg.cn/dengcai/GNMF/index.html
19
COIL20
TDT2
![Page 20: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/20.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
How to use the local consistency idea?
Matrix factorizationNon-negative matrix factorization
Topic modelingProbabilistic latent semantic analysis
ClusteringGaussian mixture model
20
![Page 21: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/21.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
What is Topic Modeling
Text Collection
ProbabilisticTopic Modeling
web 0.21search 0.10link 0.08 graph 0.05…
…
term 0.16relevance 0.08weight 0.07 feedback 0.04…
Topic models(Multinomial distributions)
Powerful tool for text mining
Topic discovery, Summarization, Opinion mining, Many more …
21
![Page 22: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/22.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
22
Language Model Paradigm in IR
Probabilistic relevance model
Random variables
Bayes’ ruleprior probability of relevance for document d (e.g. quality, popularity)
probability of generating a query q to ask for relevant d
probability that document d is relevant for query q
J. Ponte and W.B. Croft, A Language Model Approach to Information Retrieval, ACM SIGIR, 1998.
![Page 23: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/23.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
23
Language Model Paradigm
First contribution: prior probability of relevance
simplest case: uniform (drops out for ranking) popularity: document usage statistics (e.g. library circulation
records, download or access statistics, hyperlink structure)
Second contribution: query likelihood
query terms q are treated as a sample drawn from an (unknown) relevant document
12
1
2
![Page 24: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/24.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Query Likelihood
![Page 25: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/25.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
25
Naive Approach
TermsDocuments
Maximum Likelihood Estimation
number of occurrencesof term w in document d
Zero frequency problem: terms not occurring in a document get zero probability
![Page 26: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/26.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
26
Estimation Problem
Crucial question: In which way can the document collection be utilized to improve probability estimates?
document estimation
(i.i.d) sample
other documents
learning from otherdocuments in a collection ?
![Page 27: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/27.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
27
Probabilistic Latent Semantic Analysis
Latent Concepts
TermsDocuments
TRADE
economic
imports
tradeModel fitting
T. Hofmann, Probabilistic Latent Semantic Analysis, UAI 1999.
![Page 28: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/28.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Log-Likelihood
Goal: Find model parameters that maximize the log-likelihood, i.e. maximize the average predictive probability for observed word occurrences (non-convex optimization problem)
pLSA via Likelihood Maximization
![Page 29: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/29.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010 E step: posterior probability of latent variables (“concepts”)
Expectation Maximization Algorithm
Probability that the occurence of term w in document d can be “explained“ by concept z
M step: parameter estimation based on “completed” statistics
A.P. Dempster, N.M. Laird, and D.B. Rubin, Maximum Likelihood from Incomplete Data via the EM Algorithm, Journal of Royal Statistical Society B, vol. 39, no. 1, pp. 1-38, 1977
![Page 30: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/30.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Local Consistency ?
Put edges between neighbors (nearby data points);
Two nodes in the graph connected by an edge share similar properties.
Network data
Co-author network, facebook, webpage30
![Page 31: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/31.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Text Collections with Network Structure
Literature + coauthor/citation network
Email + sender/receiver network
…31
Blog articles + friend network News + geographic network
Web page + hyperlink structure
![Page 32: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/32.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Importance of Topic Modeling on Network
32
Information Retrieval +Data Mining +Machine Learning, …
=Domain Review +Algorithm +Evaluation, …
orComputer Science Literature
?
![Page 33: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/33.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Intuitions
People working on the same topic belong to the same “topical community”
Good community: coherent topic + well connected
A topic is semantically coherent if people working on this topic also collaborate a lot
33
IR
IR
IR?More likely to be an IR person or a compiler person?
Intuition: my topics are similar to my neighbors
![Page 34: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/34.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Social Network Context for Topic Modeling
Context = author
Coauthor = similar contexts
Intuition: I work on similar topics to my neighbors
34
Smoothed Topic distributions P(θj|author)
e.g. coauthor network
D. Cai, X. Wang, and X. He, Probabilistic Dyadic Data Analysis with Local and Global Consistency, ICML’09.Q. Mei, D. Cai, D. Zhang, and C. Zhai, Topic Modeling with Network Regularization, WWW’08.
![Page 35: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/35.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Objective Function
35 D. Cai, X. Wang, and X. He, Probabilistic Dyadic Data Analysis with Local and Global Consistency, ICML’09.
![Page 36: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/36.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010 E step: posterior probability of latent variables (“concepts”)
Parameter Estimation via EM
M step: parameter estimation based on “completed” statistics
Same as PLSA
Same as PLSA
?|k iP z d
36 D. Cai, X. Wang, and X. He, Probabilistic Dyadic Data Analysis with Local and Global Consistency, ICML’09.
![Page 37: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/37.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Parameter Estimation via EM
M step: parameter estimation based on “completed” statistics
1 111
12 2 21
1
, | ,( | )( | ) , | ,
( | ) , | ,
Mj k jj
kM
k j k jj
Mk NN j k N jj
n d w P z d wP z dP z d n d w P z d w
L
P z d n d w P z d w
1
N
n d
n d
,
Graph LaplacianL D W
1( | ) , | , /M
k i i j k i j ijP z d n d w P z d w n d
Same as PLSA
If λ = 0
37 D. Cai, X. Wang, and X. He, Probabilistic Dyadic Data Analysis with Local and Global Consistency, ICML’09.
![Page 38: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/38.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Experiments
Bibliography data and coauthor
networks
DBLP: text = titles; network = coauthors Four conferences (expect 4 topics):
SIGIR, KDD, NIPS, WWW
38
![Page 39: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/39.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Topical Communities with PLSA
Topic 1 Topic 2 Topic 3 Topic 4
term 0.02 peer 0.02 visual 0.02 interface 0.02
question 0.02 patterns 0.01 analog 0.02 towards 0.02
protein 0.01 mining 0.01 neurons 0.02 browsing 0.02
training 0.01 clusters 0.01 vlsi 0.01 xml 0.01
weighting 0.01 stream 0.01 motion 0.01 generation 0.01
multiple 0.01 frequent 0.01 chip 0.01 design 0.01
recognition 0.01 e 0.01 natural 0.01 engine 0.01
relations 0.01 page 0.01 cortex 0.01 service 0.01
library 0.01 gene 0.01 spike 0.01 social 0.01
39
? ?? ?
Noisy community assignment
![Page 40: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/40.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Topical Communities with NetPLSA
40
Topic 1 Topic 2 Topic 3 Topic 4
retrieval 0.13 mining 0.11 neural 0.06 web 0.05
information 0.05 data 0.06 learning 0.02 services 0.03
document 0.03 discovery 0.03 networks 0.02 semantic 0.03
query 0.03 databases 0.02 recognition 0.02 services 0.03
text 0.03 rules 0.02 analog 0.01 peer 0.02
search 0.03 association 0.02 vlsi 0.01 ontologies 0.02
evaluation 0.02 patterns 0.02 neurons 0.01 rdf 0.02
user 0.02 frequent 0.01 gaussian 0.01 management 0.01
relevance 0.02 streams 0.01 network 0.01 ontology 0.01
Information Retrieval
Data mining Machine learning
Web
Coherent community assignment
Q. Mei, D. Cai, D. Zhang, and C. Zhai, Topic Modeling with Network Regularization, WWW’08.
![Page 41: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/41.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Coherent Topical Communities
41
Semantics of community:
“Data Mining(KDD) ”
NetPLSA
mining 0.11
data 0.06
discovery 0.03
databases 0.02
rules 0.02
association 0.02
patterns 0.02
frequent 0.01
streams 0.01
PLSA
peer 0.02
patterns 0.01
mining 0.01
clusters 0.01
stream 0.01
frequent 0.01
e 0.01
page 0.01
gene 0.01
PLSA
visual 0.02
analog 0.02
neurons 0.02
vlsi 0.01
motion 0.01
chip 0.01
natural 0.01
cortex 0.01
spike 0.01
NetPLSA
neural 0.06
learning 0.02
networks 0.02
recognition 0.02
analog 0.01
vlsi 0.01
neurons 0.01
gaussian 0.01
network 0.01
Semantics of community:
“machine learning (NIPS)”
Q. Mei, D. Cai, D. Zhang, and C. Zhai, Topic Modeling with Network Regularization, WWW’08.
![Page 42: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/42.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
For More Detials
Please check our papers
http://www.zjucadcg.cn/dengcai/LapPLSA/index.html
42 D. Cai, X. Wang, and X. He, Probabilistic Dyadic Data Analysis with Local and Global Consistency, ICML’09.Q. Mei, D. Cai, D. Zhang, and C. Zhai, Topic Modeling with Network Regularization, WWW’08.
![Page 43: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/43.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
How to use the local consistency idea?
Matrix factorizationNon-negative matrix factorization
Topic modelingProbabilistic latent semantic analysis
ClusteringGaussian mixture model
43
![Page 44: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/44.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Gaussian Mixture Model (GMM) is one of the most popular clustering methods which can be viewed as a linear combination of different Gaussian components.
Gaussian Mixture Model
![Page 45: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/45.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Gaussian Mixture Model
![Page 46: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/46.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Gaussian Mixture Model
![Page 47: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/47.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Gaussian Mixture Model
0 0.5 10
0.5
1
(a)0 0.5 1
0
0.5
1
(b)
![Page 48: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/48.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Gaussian Mixture Model
![Page 49: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/49.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
![Page 50: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/50.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Objective Function
50 J. Liu, D. Cai, and X. He, Gaussian Mixture Model with Local Consistency, AAAI’10.
![Page 51: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/51.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
• E-step:
• M-step:
EM Equations
1
( | , )( | )( | , )
k i k kk i K
j i j jj
xP c xx
N
N
, ,,, 11
( | ) ( | )( | )
2
( )( )NN
k i k j i k j k ijk i i ki ji
kk k
P c x P c x S S WP c x S
N N
, 11
( | ) ( | () )( | )
2
-( )NN
k i k j i j iji k ii ji
kk k
P c x P c x x x Wx P c x
N N
1
( | )N
k ii
k
P c x
N
, ( )( )T
i k i k i kS x x
1
( | )N
k k ii
N P c x
original GMM part51
![Page 52: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/52.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010 7 Real Data sets :
•The Yale face image database.
•The Waveform model described in “The Elements of Statistical Learning” .
•The Vowels data set which has steady state vowels of British English.
•The Libras movement data set containing hand movement pictures.
•The Control Charts data set consisting control charts.
•The Cloud data set is a simple 2 classes problem.
•The Breast Cancer Wisconsin data set computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.
Experiment
52
![Page 53: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/53.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Clustering Results
Data set LCGMM GMM K-means Ncut size # of features
# of classes
Yale 54.3 29.1 51.5 54.6 165 4096 15
Libras 50.8 35.8 44.1 48.6 800 21 3
Chart 70.0 56.8 61.5 58.8 990 10 11
Cloud 100.0 96.2 74.4 61.5 360 90 15
Breast 95.5 94.7 85.4 88.9 600 60 6
Vowel 36.6 31.9 29.0 29.1 2048 10 2
Waveform 75.3 76.3 51.9 52.3 569 30 2
53
![Page 54: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/54.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
The Take-home Messages
54
![Page 55: Learning with Local Consistency - 浙江大学CAD&CG](https://reader030.vdocuments.net/reader030/viewer/2022012807/61bd481e61276e740b113b59/html5/thumbnails/55.jpg)
© D
eng
Cai,
Col
lege
of
Com
pute
r Sc
ienc
e, Z
heji
ang
Uni
vers
ity
Chin
ese
Wor
ksho
p on
Mac
hine
Lea
rnin
g an
d Ap
plic
atio
ns,
Nan
jing
, N
ov.
2010
Thanks!
55