Tracking by Regression

Tomas Svoboda

Department of Cybernetics, Center for Machine Perception

March 31, 2014

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 1 / 48

Linear mapping for tracking

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 2 / 48

Optimization vs. regression based tracking

KLT:∆p∗ = argmin

∆p∈S

∑x

[I (W(x;p + ∆p))− T (x)]2

Optimization in general:

p∗ = argminp∈S

f (p; I ,p0),

Regresssion based tracking:

p∗ = ϕ(I ,p0)

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 3 / 48

Optimization vs. regression based tracking

KLT:∆p∗ = argmin

∆p∈S

∑x

[I (W(x;p + ∆p))− T (x)]2

Optimization in general:

p∗ = argminp∈S

f (p; I ,p0),

Regresssion based tracking:

p∗ = ϕ(I ,p0)

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 4 / 48

Optimization vs. regression based tracking

KLT:∆p∗ = argmin

∆p∈S

∑x

[I (W(x;p + ∆p))− T (x)]2

Optimization in general:

p∗ = argminp∈S

f (p; I ,p0),

Regresssion based tracking:

p∗ = ϕ(I ,p0)

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 5 / 48

Regression learned from data - sample image

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 6 / 48

Regression learning

ϕ∗ = argminϕ

∑p

‖ϕ(I(p ◦ x

))− p‖.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 7 / 48

Connection to KLT

∆p = H−1∑x

[∇I ∂W

∂p

]>[T (x)− I (W(x;p))]

where:

H =∑x

[∇I ∂W

∂p

]>[∇I ∂W

∂p

]

Reformulating regression:

p = ϕ(I (x)

)= H(I (x)− T (x)

)How to get H?

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 8 / 48

Connection to KLT

∆p = H−1∑x

[∇I ∂W

∂p

]>[T (x)− I (W(x;p))]

where:

H =∑x

[∇I ∂W

∂p

]>[∇I ∂W

∂p

]

Reformulating regression:

p = ϕ(I (x)

)= H(I (x)− T (x)

)How to get H?

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 9 / 48

Connection to KLT

∆p = H−1∑x

[∇I ∂W

∂p

]>[T (x)− I (W(x;p))]

where:

H =∑x

[∇I ∂W

∂p

]>[∇I ∂W

∂p

]

Reformulating regression:

p = ϕ(I (x)

)= H(I (x)− T (x)

)How to get H?

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 10 / 48

Collecting training data

Image template: T = T (x)Collected training pairs (I i , pi ) (i = 1 . . . d) of observed intensitiesI i and corresponding motion parameters pi , which align the objectwith current frame.

Training set is an ordered pair (I, P), such thatI = [I1 − T, I2 − T, . . . Id − T] and P = [p1,p2, . . .pd ].

The bolds I,T are vectorized I ,T . Think about: I = I(:)

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 11 / 48

Collecting training data

Image template: T = T (x)Collected training pairs (I i , pi ) (i = 1 . . . d) of observed intensitiesI i and corresponding motion parameters pi , which align the objectwith current frame.

Training set is an ordered pair (I, P), such thatI = [I1 − T, I2 − T, . . . Id − T] and P = [p1,p2, . . .pd ].

The bolds I,T are vectorized I ,T . Think about: I = I(:)

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 12 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 13 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 14 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 15 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 16 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 17 / 48

Learning alignment for one predictor

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

I ϕ( )= (0, 0)>

I ϕ( )= (−25, 0)>

I ϕ( )= (25,−15)>

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 18 / 48

Generating training examples

[ ]

t − motioni I = I(x ,...x ) − observation1 ci

examples

displacementestimation

generating

+2

+7+9

+4

−8

−4

training set: +2 −4 +4+9 +7 −8

T= I=

Training set: (I, P)I = [I1 − T, I2 − T, . . . Id − T] and P = [p1,p2, . . .pd ].

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 19 / 48

Learning H from data

H∗ = argminH

d∑i=1

‖H(Ii − T)− pi‖22 = argmin

H‖HI− P‖2

F =

= argminH

trace(HI− P)(HI− P)> =

= argminH

trace(HII>H> − 2HIP> + PP>).

Next its derivative is set equal to zero:

2H∗II> − 2PI> = 0

H∗II> = PI>

H∗ = P I>(II>)−1︸ ︷︷ ︸I+

= PI+.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 20 / 48

Learning H from data

H∗ = argminH

d∑i=1

‖H(Ii − T)− pi‖22 = argmin

H‖HI− P‖2

F =

= argminH

trace(HI− P)(HI− P)> =

= argminH

trace(HII>H> − 2HIP> + PP>).

Next its derivative is set equal to zero:

2H∗II> − 2PI> = 0

H∗II> = PI>

H∗ = P I>(II>)−1︸ ︷︷ ︸I+

= PI+.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 21 / 48

LS learning - summary

ϕ∗ = argminϕ∑p

‖ϕ(I(p ◦ x

))− p‖2.

Minimizes sum of square errors over all training set. Leads tomatrix pseudoinverse computation.

Example for linear mapping:

H∗ = argminH

d∑i=1

‖H(Ii − T)− ti‖22 = argmin

H‖HI− P‖2

F

after some derivation

H∗ = P I>(II>)−1︸ ︷︷ ︸I+

= PI+.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 22 / 48

Min-max learning

ϕ∗ = argminϕ maxp‖ϕ(I(p ◦ x

))− p‖∞.

Minimizes the worst case (the biggest estimation error) in thetraining set. Leads to linear programming.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 23 / 48

Tracking of a single point by a sequence of predictors

X

t =1 ϕ1

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 24 / 48

Tracking of a single point by a sequence of predictors

X

o

1

t =1 ϕ1

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 25 / 48

Tracking of a single point by a sequence of predictors

X

o

1

t =1 ϕ1

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 26 / 48

Tracking of a single point by a sequence of predictors

X

o

1

t =1 ϕ1

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 27 / 48

Tracking of a single point by a sequence of predictors

X

o

1

t =1 ϕ1

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 28 / 48

Tracking of a single point by a sequence of predictors

X

o

o2

1

t =1 ϕ1

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 29 / 48

Tracking of a single point by a sequence of predictors

X

o

o2

1

t =1 ϕ1

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 30 / 48

Tracking of a single point by a sequence of predictors

X

o

o2

1

t =1 ϕ1

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 31 / 48

Tracking of a single point by a sequence of predictors

X

o

o2

1

t =1 ϕ1

t =3 ϕ3

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 32 / 48

Tracking of a single point by a sequence of predictors

X

o

o

o

2

1

3 t =1 ϕ1

t =3 ϕ3

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 33 / 48

Tracking of a single point by a sequence of predictors

X

o

o

o

2

1

3

New position

t =1 ϕ1

t =3 ϕ3

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 34 / 48

Tracking of a single point by a sequence of predictors

1 2Φ=(ϕ,ϕ,ϕ)

3

X

o

o

o

2

1

3

New position

t =1 ϕ1

Motion

t =3 ϕ3

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 35 / 48

Tracking of a single point by a sequence of predictors

1 2Φ=(ϕ,ϕ,ϕ)

3

Ranges

X

o

o

o

2

1

3

New position

t =1 ϕ1

Motion

t =3 ϕ3

t =2 ϕ2

Old position

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 36 / 48

Learning of sequential predictor

I Learning - searching for the sequence with predefined range,accuracy and minimal computational cost.

I [Zimmermann-PAMI-2009] - Dynamic programming estimatesthe optimal sequence of linear predictors.

I [Zimmermann-IVC-2009] - Branch & bound search allows fortime constrained learning (demo in MATLAB).

[Zimmermann-PAMI-2009] K.Zimmermann, J.Matas, T.Svoboda. Tracking by anOptimal Sequence of Linear Predictors, in IEEE Transactions on Pattern Analysis andMachine Intelligence, IEEE computer society, 2009, vol. 31, No 4, pp 677–692.[Zimmermann-IVC-2009] K.Zimmermann, T.Svoboda, J.Matas. Anytime learning forthe NoSLLiP tracker. Image and Vision Computing, vol. 27, No 11

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 37 / 48

Learning of sequential predictor

I Learning - searching for the sequence with predefined range,accuracy and minimal computational cost.

I [Zimmermann-PAMI-2009] - Dynamic programming estimatesthe optimal sequence of linear predictors.

I [Zimmermann-IVC-2009] - Branch & bound search allows fortime constrained learning (demo in MATLAB).

[Zimmermann-PAMI-2009] K.Zimmermann, J.Matas, T.Svoboda. Tracking by anOptimal Sequence of Linear Predictors, in IEEE Transactions on Pattern Analysis andMachine Intelligence, IEEE computer society, 2009, vol. 31, No 4, pp 677–692.[Zimmermann-IVC-2009] K.Zimmermann, T.Svoboda, J.Matas. Anytime learning forthe NoSLLiP tracker. Image and Vision Computing, vol. 27, No 11

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 38 / 48

Learning of sequential predictor

I Learning - searching for the sequence with predefined range,accuracy and minimal computational cost.

I [Zimmermann-PAMI-2009] - Dynamic programming estimatesthe optimal sequence of linear predictors.

I [Zimmermann-IVC-2009] - Branch & bound search allows fortime constrained learning (demo in MATLAB).

[Zimmermann-PAMI-2009] K.Zimmermann, J.Matas, T.Svoboda. Tracking by anOptimal Sequence of Linear Predictors, in IEEE Transactions on Pattern Analysis andMachine Intelligence, IEEE computer society, 2009, vol. 31, No 4, pp 677–692.[Zimmermann-IVC-2009] K.Zimmermann, T.Svoboda, J.Matas. Anytime learning forthe NoSLLiP tracker. Image and Vision Computing, vol. 27, No 11

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 39 / 48

Learning of the optimal sequence of linear predictors

Range

Complexity

Uncertaintyregion

I Range: the set of admissible motions, r .I Complexity: cardinality of support set, c .I Uncertainty region: the region within which all predictions

lie, λ. Small red circles show acceptable uncertainty.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 40 / 48

Learning of the optimal sequence of linear predictors

0

20

40

60

80

100

0

50

100

150

200

0

10

20

30

40

50

r − range

c − complexity

λ −

unc

erta

inty

are

a

Achievablepredictors ω

Inachievablepredictors

λ−minimal predictors

I Range: the set of admissible motions, r .I Complexity: cardinality of support set, c .I Uncertainty region: the region within which all predictions

lie, λ. Small red circles show acceptable uncertainty.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 41 / 48

Learning of the optimal sequence of linear predictors

I Range: the set of admissible motions, r .I Complexity: cardinality of support set, c .I Uncertainty region: the region within which all predictions

lie, λ. Small red circles show acceptable uncertainty.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 42 / 48

Learning of the optimal sequence of linear predictors

I Range: the set of admissible motions, r .I Complexity: cardinality of support set, c .I Uncertainty region: the region within which all predictions

lie, λ. Small red circles show acceptable uncertainty.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 43 / 48

Branch and Bound

0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Graph depth

Pre

dic

tion e

rror

c=600c=320 c=360

c=80

0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Graph depth

Pre

dic

tion e

rror

c=100

c=600c=320

c=380

Don’t forget to show the live demo!

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 44 / 48

Branch and Bound

0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Graph depth

Pre

dic

tion e

rror

c=600c=320 c=360

c=80

0 1 2 3 4 50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Graph depth

Pre

dic

tion e

rror

c=100

c=600c=320

c=380

Don’t forget to show the live demo!

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 45 / 48

Tracking with one linear predictor.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 46 / 48

Modeling motion by number of linear predictors.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 47 / 48

Motion blur, fast motion, views from acute angles andother image distortions.

T. Svoboda / Department of Cybernetics, CMP / Tracking by Regression 48 / 48