Dictionary Learning for Graph Signals

Yael Yankelevsky

22.12.2019

236862 – Introduction to Sparse and

Redundant Representations joint work with

Prof. Michael Elad

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The Sparseland Model

2

Model assumption: All data vectors are linear combinations of FEW (𝑇 ≪ 𝑁) atoms from 𝐃

=

Dictionary Learning:

Dictionary Learning for Graph Signals By: Yael Yankelevsky

.

≈ .

… … X Y D

Initialize Dictionary

Sparse Coding Using OMP

Dictionary Update

Atom-by-atom + coeffs.

For the j-th atom:

K-SVD Algorithm Overview [Aharon et al. ’06]

3

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Energy Networks

Biological Networks

Transportation Networks Social Networks

Meshes & Point Clouds

Data is often structured…

4

Dictionary Learning for Graph Signals By: Yael Yankelevsky

5

What happens for non-conventionally structured signals?

Can dictionary learning work well for such signals as well?

The general idea: Model the underlying structure as a graph and incorporate it in the dictionary learning algorithm

Dictionary Learning for Graph Signals By: Yael Yankelevsky

We are given a graph:

The 𝑖𝑡ℎ node is characterized by a feature vector 𝑣𝑖

The edge between the 𝑖𝑡ℎ and

𝑗𝑡ℎ nodes carries a weight

𝑤𝑖𝑗 ∝ 𝑑 𝑣𝑖 , 𝑣𝑗−1

The degree matrix:

Graph Laplacian:

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣8

𝑣7

𝑣9

𝑣12

𝑣11

𝑣10

𝑣13

6

Basic Notations

Dictionary Learning for Graph Signals By: Yael Yankelevsky

𝑓1

𝑓2

𝑓3

𝑓4

𝑓5 𝑓6

𝑓8

𝑓7

𝑓9

𝑓12 𝑓11

𝑓10

𝑓13

The 𝑖𝑡ℎ node has a value 𝑓𝑖

f = graph signal The combinatorial Laplacian is a

differential operator:

Defines global regularity on the graph (Dirichlet energy):

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣8

𝑣7

𝑣9

𝑣12

𝑣11

𝑣10

𝑣13

7

Basic Notations

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Ignore structure (MOD, K-SVD) [Engan et al. ’99],[Aharon et al. ’06]

Analytic transforms

Graph Fourier transform [Sandryhaila & Moura ’13]

Windowed Graph Fourier transform [Shuman et al. ’12]

Graph Wavelets [Coifman & Maggioni ’06],[Gavish et al. ’10],[Hammond

et al. ’11],[Ram et al. ’12],[Shuman et al. ’16],…

Structured learned dictionaries [Zhang et al. ’12],[Thanou et al. ’14]

Related Work: Dictionaries for Graph Signals

8

Our solution: Graph Regularized Dictionary Learning

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Construct 2 graphs capturing the feature dependencies and the data manifold structure

The Basic Concept

9

Lc

L

Y

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Example: Traffic Dataset

10

Y

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Introduce graph regularization terms that preserve these structures

Imposed smoothness (graph Dirichlet energy):

Dual Graph Regularized Dictionary Learning

11

Dictionary Learning for Graph Signals By: Yael Yankelevsky

A good estimation of L is crucial!

We can learn L and adapt it to promote the

desired smoothness [Hu et al. ’13], [Dong et al. ‘15],

[Kalofolias ’16], [Segarra et al. ‘17],…

12

The Importance of the Underlying Graph

[Shuman et al. ’13]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Dual Graph Regularized Dictionary Learning

13

Key idea: Dictionary atoms preserve feature similarities Similar signals have similar sparse representations The graph is adapted to promote the desired smoothness

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The DGRDL Algorithm

14

Initialize Dictionary

Sparse Coding Using ADMM

pursuit

Dictionary Update Atom-by-atom + coeffs. (modified update rule)

Graph Update

Dictionary Learning for Graph Signals By: Yael Yankelevsky

The DGRDL Algorithm

15

Initialize Dictionary

Sparse Coding Using ADMM

pursuit

Dictionary Update Atom-by-atom + coeffs. (modified update rule)

Graph Update

For the j-th atom:

?

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph Regularized Pursuit

16

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph Regularized Pursuit

17

ADMM: [Boyd et al. ’11]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

ADMM: [Boyd et al. ’11]

Graph Regularized Pursuit

18

graph SC [Zheng et al. ’11]

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Classical sparse theory:

Theoretical Guarantees

19

Theorem: If the true representation 𝐱 satisfies

𝐱 0 = s <1

21 +

1

μ 𝐃

then a solution 𝐱 for (𝐏0ϵ) must be close to it

𝐱 − 𝐱 22 ≤

4ϵ2

1 − δ2s≤

4ϵ2

1 − 2s − 1 μ 𝐃

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Graph sparse coding:

Theoretical Guarantees

20

Theorem: If the true representation 𝐗 satisfies

𝐗 0,∞ = s <1

21 +

1 + f β, 𝐋𝐜μ 𝐃

then a solution 𝐗 for (𝐏𝟎,∞ϵ ) must be close to it

𝐗 − 𝐗F

2≤

4ϵ2

1 − δ2s≤

4ϵ2

1 − 2s − 1 μ 𝐃 + f β, 𝐋𝐜

≥ 0

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Back to DGRDL…

21

Initialize Dictionary

Sparse Code Using ADMM

pursuit

Dictionary Update Atom-by-atom + coeffs. (modified update rule)

dictionary atoms are smooth graph signals

similar signals have similar sparse codes

graph is adapted to promote smoothness

Graph Update

Dictionary Learning for Graph Signals By: Yael Yankelevsky

22

Results: Network Data Recovery

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Settings:

N=578 sensors

M=2892 signals

1500 for training

1392 for testing

Graph signal = daily avg. bottleneck (min.) measured at each station

Traffic Dataset

23

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Representation

24

. . .

Original signals

Sparse Coding

. . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Representation

25

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Denoising

26

. . .

Original signals

Sparse Coding +

AWGN

. . .

Noisy signals

. . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Denoising

27

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Inpainting

28

. . .

Original signals

Sparse Coding

Incomplete signals

Discard p% of the samples

randomly

. . . . . .

Reconstructed signals

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Inpainting

29

K-SVD GRDL (fixed L) GRDL (learned L) DGRDL

Dictionary Learning for Graph Signals By: Yael Yankelevsky

30

Results: Image Denoising Revisited

Dictionary Learning for Graph Signals By: Yael Yankelevsky

𝑦𝑗

𝑦𝑖

𝑀

𝑛2

A Glimpse at Image Processing

31

𝑛

𝑛

𝐋 is an 𝑛 × 𝑛 grid (patch structure)

𝐃 is learned from only 1000 patches

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Image Denoising (σ=25)

32

Original Noisy (20.18dB) K-SVD (28.35dB) DGRDL (28.50dB)

Original Noisy (20.18dB) K-SVD (30.56dB) DGRDL (30.71dB)

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Learn the underlying patch structure (pixel dependencies) from the data

Structure Inference

33

input image

learned

L

Dictionary Learning for Graph Signals By: Yael Yankelevsky

Time to Conclude…

34

We developed an efficient algorithm for joint learning of the

dictionary and the graph

We have shown how sparsity-based models

become applicable also for graph structured data

We demonstrated how various applications can benefit from the

new model

Processing data is enabled by an appropriate

modeling that exposes its inner structure

Extensions include supervised

dictionary learning and supporting high dimensions

Dictionary Learning for Graph Signals By: Yael Yankelevsky

35

Thank You