ICML 2020

On Learning Sets of Symmetric Elements

Haggai Maron, Or Litany, Gal Chechik, Ethan Fetaya [1] Nvidia Research [2] Stanford University [3] Bar-Ilan University

[1] [2] [1,3] [3]

Motivation and Overview

Set Symmetry

{ }Input set

Previous work (DeepSets, PointNet) targeted training a deep network over sets

DeepNet

x1x2⋮xm ,

x1x2⋮xm ,

x1x2⋮xm , …

Set+Elements symmetry

{ }Input set

Main challenge: What architecture is optimal when elements of the set have their own symmetries?

DeepNet, , …

Both the set and its elements have symmetries.

Deep Symmetric sets

{ }Input image set Output

Set symmetry: Order invariance/equivariance

{ }=

Set symmetry: Order invariance/equivariance

{ }={ }

Set symmetry: Order invariance/equivariance

{ }={ }

Element symmetry: Translation invariance/equivariance

{ }

{ }{ }

Element symmetry: Translation invariance/equivariance

{ }{ }

Element symmetry: Translation invariance/equivariance

Applications

Modalities1D signals 2D images 3D pointclouds Graph

This paperA principled approach for learning sets of complex elements (graphs, point clouds, images)

Characterize maximally expressive linear layers that respect the symmetries (DSS layers)

Prove universality results

Experimentally demonstrate that DSS networks outperform baselines

Previous work

Deep sets [Zaheer et al. 2017]

CNN

CNN

CNN

Siamese

Deep sets [Zaheer et al. 2017]

CNN

CNN

CNN

FeaturesSiamese

Deep sets [Zaheer et al. 2017]

CNN

CNN

CNN

DeepSets

FeaturesSiamese Deeps sets block

Deep sets [Zaheer et al.]

Previous work: information sharing

Aittala and Durand, ECCV 2018

Sridhar et al., NeuriPS 2019

Liu et al., ICCV 2019

CNN

CNN

CNN

CNN

CNN

CNN

Information sharing layer

Our approach

InvarianceMany Learning tasks are invariant to natural transformations (symmetries)

More formally. Let be a subgroup:

is invariant if , for all

e.g. image classification

H ≤ Sn

f : ℝn → ℝ f(τ ⋅ x) = f(x) τ ∈ Hτ

f f

“Cat”

EquivarianceLet be a subgroup:

Equivariant if ,

e.g. edge detection

H ≤ Sn

f(τ ⋅ x) = τ ⋅ f(x)

τ

τf f

Invariant neural networks

⋯

Equivariant FC Invariant

• Invariant by construction

Deep Symmetric Sets

with symmetry group

Want to be invariant/equivariant to both and the ordering

Formally the symmetry group is

x1, …, xn ∈ ℝd G ≤ Sd

G

H = Sn × G ≤ Snd

G

Main challenges

• What is the space of linear equivariant layers for specific ?H = SN × G

• What is the space of linear equivariant layers for a given ?

• Can we compute these operators efficiently?

H = SN × G

Main challenges

• What is the space of linear equivariant layers for a given ?

• Can we compute these operators efficiently?

• Do we lose expressive power?

H = SN × G

-invariant networksH

-invariant continuous functionsH

Continuous functions

Main challenges

• What is the space of linear equivariant layers for a given ?

• Can we compute these operators efficiently?

• Do we lose expressive power?

H = SN × G

-invariant networksH

-invariant continuous functionsH

Continuous functions

Gap?

Main challenges

Theorem: Any linear −equivariant layer is of the form

where are linear -equivariant functions

We call these layers Deep Sets for Symmetric elements layers (DSS)

SN × G L : ℝn×d → ℝn×d

L(X)i = LG1 (xi) + ∑

j≠i

LG2 (xj)

LG1 , LG

2 G

-equivariant layersH

DSS for images

are images

is the group of circular translations

-equivariant layers are convolutions

x1, …, xn

G 2D

G

Single DSS layer

DSS for images

are images

is the group of circular translations

-equivariant layers are convolutions

x1, …, xn

G 2D

G

CONV

CONV

CONV

Single DSS layer

DSS for images

are images

is the group of circular translations

-equivariant layers are convolutions

x1, …, xn

G 2D

G

CONV

CONV

CONV

+

Single DSS layer

DSS for images

are images

is the group of circular translations

-equivariant layers are convolutions

x1, …, xn

G 2D

G

CONV

CONV

CONV

CONV

+

Single DSS layer

DSS for images

Siamese part

Information sharing part

CONV

CONV

CONV

CONV

+ +

Single DSS layer

Expressive powerTheorem

If G-equivariant networks are universal appoximators for G-equivariant functions, then so are DSS networks for -equivariant functions.

SN × G

Expressive powerTheorem

If G-equivariant networks are universal appoximators for G-equivariant functions, then so are DSS networks for -equivariant functions.

• Main tool:

• Noether’s Theorem (Invariant theory)

• For any finite group , the ring of invariant polynomials is finitely generated.

• Generators can be used to create continuous unique encodings for elements in

SN × G

H ℝ[x1, . . . , xn]H

ℝn×d /H

Results

Signal classification

Image selection

Shape selection

Conclusions

A general framework for learning sets of complex elements

Generalizes many previous works

Expressivity results

Works well in many tasks and data types