a convergent solution to tensor subspace learning
TRANSCRIPT
A Convergent Solution to Tensor Subspace Learning
1 2 2 1,3Huan Wang Shuicheng Yan Thomas Huang Xiaoou Tang
1 2 3
Chinese University University of Illinois Microsoft Research
of Hong Kong at Urbana Champaign Asia
Concept
Tensor Subspace Learning . Concept• Tensor: multi-dimensional (or multi-way) arrays of components
Application
Tensor Subspace Learning . Application
• real-world data are affected by multifarious factors
for the person identification, we may have facial images of different
► views and poses
► lightening conditions
► expressions
• the observed data evolve differently along the variation of different factors
► image columns and rows
Application
Tensor Subspace Learning . Application
• it is desirable to dig through the intrinsic connections among different affection factors of the data.
• Tensor provides a concise and effective representation.
Illumination
pose
expression
Image columns
Image rows
Images
Tensor Subspace Learning algorithms
Traditional Tensor Discriminant algorithms
• Tensor Subspace Analysis He et.al
• Two-dimensional Linear Discriminant Analysis
• Discriminant Analysis with Tensor RepresentationYe et.al
Yan et.al
• project the tensor along different dimensions or ways
• projection matrices for different dimensions are derived iteratively
• solve an trace ratio optimization problem
• DO NOT CONVERGE !
Tensor Subspace Learning algorithms
Graph Embedding – a general framework
• An undirected intrinsic graph G={X,W} is constructed to represent the pairwise similarities over sample data.
• A penalty graph or a scale normalization item is constructed to impose extra constraints on the transform.
intrinsic graph penalty graph
Discriminant Analysis Objective
Solve the projection matrices iteratively: leave one projection matrix as variable while keeping others as constant.
1
21
2| 1
( ) |arg ax
( ) |k nk
k n pi j k k iji j
k nU i j k k iji j
X X U Wm
X X U W
• No closed form solution
Mode-k unfolding of the tensor
2
2arg ax
T T
T Tk
k k k k pi j iji j
k k k kU i j iji j
U Y U Y Wm
U Y U Y W
kiY
~1 1 1
1 1 1... ...k k ni i k k nY X U U U U
~
iY
Objective Deduction
Discriminant Analysis Objective
Trace Ratio: General Formulation for the objectives of the Discriminant Analysis based Algorithms.
( )arg ax
( )
T
Tk
k p kk
k k kU
Tr U S Um
Tr U S U
( )( )k k k k k T
ij i j i ji jS Y Y Y YW
( )( )k k k k T
i j i ji j
p pk ijS Y Y Y YW
DATER:
TSA:
kS Within Class Scatter of the unfolded data
pkS
Between Class Scatter of the unfolded data
W pS Diagonal Matrix with weightsConstructed from Image Manifold
Disagreement between the Objective and the Optimization Process
Why do previous algorithms not converge?
11
1
1 111
( )arg ax
( )
T k
k
T k kk
k p
kU
Tr U S Um
Tr U S U1 1 1 1 1
11
1arg ax (( ) )T T
k
k k k k kpk
U
m Tr U S U U S U
GEVD
22
2
2 222
( )arg ax
( )
T k
k
T k kk
k p
kU
Tr U S Um
Tr U S U
2 2 2 2 2
22
1arg ax (( ) )T T
k
k k k k kpk
U
m Tr U S U U S U
( )
( )
Tr A
Tr B1( )Tr B A
The conversion from Trace Ratio to Ratio Trace induces an inconsistency among the objectives of different dimensions!
from Trace Ratio to Trace Difference
What will we do? from Trace Ratio to Trace Difference
( )arg ax
( )
T
Tk
k p kk
k k kU
Tr U S Um
Tr U S UObjective:
Define( )
( )
T
T
k p kt k t
k k kt t
Tr U S U
Tr U S U
Then
( ( ) ) 0Tk p k kt k tTr U S S U
( )ktg U
( ) ( ( ) )T p kkg U Tr U S S U
Trace Ratio Trace Difference
Find ( ) ( )ktg U g U
So that
( ( ) ) ( ) 0p k kk tTr U S S U g U
( )
( )
T pk
T k
Tr U S U
Tr U S U
from Trace Ratio to Trace Difference
What will we do? from Trace Ratio to Trace Difference
Constraint TU U I
Let '1 1 2[ , ,..., ]k
kt m
U u u u
We have
1( ) ( ) 0k kt tg U g U
Thus
1 1
1 1
( )
( )
T
T
k p kt k t
k k kt t
Tr U S U
Tr U S U
The Objective rises monotonously!
Projection matrices of different dimensions share the same objective
( ) ( ( ) )T p kkg U Tr U S S U
Where '1 2, ,...,km
u u u are the leading
eigen vectors of .( )p kkS S
Hightlights of the Trace Ratio based algorithm
Highlights of our algorithm
• The objective value is guaranteed to monotonously increase; and the multiple projection matrices are proved to converge.
• Only eigenvalue decomposition method is applied for iterative optimization, which makes the algorithm extremely efficient.
• Enhanced potential classification capability of the derived low-dimensional representation from the subspace learning algorithms.
• The algorithm does not suffer from the singularity problem that is often encountered by the traditional generalized eigenvalue decomposition method used to solve the ratio trace optimization problem.
Monotony of the Objective
Experimental Results
The traditional ratio trace based procedure does not converge, while our new solution procedure guarantees the monotonous increase of the objective function value and commonly our new procedure will converge after about 4-10 iterations. Moreover, the final converged value of the objective function from our new procedure is much larger than the value of the objective function for any iteration of the ratio trace based procedure.
Convergency of the Projection Matrices
Experimental Results
The projection matrices converge after 4-10 iterations for our new solution procedure;while for the traditional procedure, heavy oscillations exist and the solution does not converge.
Face Recognition Results
Experimental Results
1. TMFA TR mostly outperforms all the other methods concerned in this work, with only one exception for the case G5P5 on the CMU PIE database.2. For vector-based algorithms, the trace ratio based formulation is consistently superior to the ratio trace based one for subspace learning.3. Tensor representation has the potential to improve the classification performance for both trace ratio and ratio trace formulations of subspace learning.
Summary
Summary
• A novel iterative procedure was proposed to directly optimize the objective function of general subspace learning based on tensor representation.
• The convergence of the projection matrices and the monotony property of the objective function value were proven.
• The first work to give a convergent solution for the general tensor-based subspace learning.
Thank You!