deep learning as optimal control problems...learning, campus jussieu, sorbonne université, paris...

Regularisation for Inverse Problems and Machine Learning, Campus Jussieu, Sorbonne Université, Paris 19.11.2019

Deep learning as optimal control problems

Martin Benning, Queen Mary University of London (QMUL)

Models and numerical methods

This is joint work with Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren and Carola-Bibiane Schönlieb

m.benning@qmul.ac.uk

This is joint work with

MB, Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schönlieb. "Deep learning as optimal control problems: models and numerical methods." arXiv preprint arXiv:1904.05657 (2019). To appear in Journal of Computational Dynamics in December 2019

• Introduction

• Runge-Kutta (RK) networks

• Numerical results

• Regularisation properties of RK network

INTRODUCTION

Introduction

In this part we consider supervised machine learning problems, e.g. classification

Goal of classification is to find mapping given input and output samples .

g : ℝn → {c0, c1, …, cK−1}{(xi, ci)}m

Here every class label for indicates that the input belongs to the corresponding class, e.g.

ci ∈ {c0, c1, …, cK−1} i ∈ {1,…, m}xi

for , then belongs to class ci = cl l ∈ {0,1,…, K − 1} xi l

m.benning@qmul.ac.uk 6

Introduction

image source

Training data

xi 2 Rd

Example:

source

Introduction

g : ℝn → {c0, c1, …, cK−1}{(xi, ci)}m

g(x) := 𝒞 (Wh(x, u) + μ)• is the hypothesis function mapping from to • is a weight vector • is a bias factor • is a model function parametrised by parameters

𝒞 ℝ {c0, c1, …, cK−1}W ∈ ℝ1×n

μ ∈ ℝh : ℝn × P → ℝn u ∈ P

A potential model for such a classifier function isg

Introduction

g : ℝn → {c0, c1, …, cK−1}{(xi, ci)}m

g(x) := 𝒞 (Wh(x, u) + μ)A potential model for such a classifier function isg

We can train the model parameters by minimising a cost function of the formm

∑i=1

L (𝒞(W h(xi, u) + μ), ci) + ℛ(u)

with respect to , and u W μ

Introduction

g : ℝn → {c0, c1, …, cK−1}{(xi, ci)}m

We can train the model parameters by minimising a cost function of the form

∑i=1

𝒞 (Wh(xi, u) + μ) − ci2

+ ℛ(u)

Introduction

g : ℝn → {c0, c1, …, cK−1}{(xi, ci)}m

We can train the model parameters by minimising a cost function of the formm

∑i=1

[log (1 + exp(Wh(xi, u) + μ)) − ci (Wh(xi, u) + μ)] + ℛ(u)

Introduction

We can train the model parameters by minimising a cost function of the form

∑i=1

𝒞 (Wh(xi, u) + μ) − ci2

+ ℛ(u)

with respect to , and u W μHow do we choose the model function ?h

For example as a deep neural network such as the residual network (ResNet), i.e.

y[ j+1]i = y[ j]

i + f (y[ j]i , u[ j]), j = 0,…, N − 1, y[0]

i = xi .

h(xi, u) = y[N ]i , u = (u[0], …, u[N−1]) ,

K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning for Image Recognition, in IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770–778.

Introduction

How do we choose the model function ?hFor example as a deep neural network such as the residual network (ResNet), i.e.

y[ j+1]i = y[ j]

i + f (y[ j]i , u[ j]), j = 0,…, N − 1, y[0]

i = xi .

h(xi, u) = y[N ]i , u = (u[0], …, u[N−1]) ,

K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning for Image Recognition, in IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770–778.

Possible choices for and :u[ j] f

u[ j] := (K[ j], β[ j]), j = 0,…, N − 1,

f (y[ j]i , u[ j]) := σ(K[ j]y[ j]

i + β[ j])

K[ j] ∈ ℝn×n β[ j] ∈ ℝn×1

σ(x) = tanh(x)

Introduction

Training the ResNet with (mean) squared error cost function therefore reads as

y[ j+1]i = y[ j]

i + f (y[ j]i , u[ j]), j = 0,…, N − 1, y[0]

i = xi .

miny,u,W,μ

∑i=1

𝒞 (Wy[N ]i + μ) − ci

+ ℛ(u),

subject to

Introduction

y[ j+1]i = y[ j]

i + f (y[ j]i , u[ j]), j = 0,…, N − 1, y[0]

i = xi .

miny,u,W,μ

∑i=1

𝒞 (Wy[N ]i + μ) − ci

+ ℛ(u),

subject to

E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2017), 014004.

By introducing a step-size parameter we can view the constraint as a forward Euler discretisation of the differential equation

·yi = f (yi, u), t ∈ [0,T ], yi(0) = xi .

Training the ResNet with (mean) squared error cost function therefore reads as

Introduction

In the continuum we therefore obtain the following optimal control problem

miny,u,W,μ

∑i=1

𝒞 (Wyi(T ) + μ) − ci2

+ ℛ(u),

subject to

B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert and E. Holtham, Reversible architectures for arbitrarily deep residual neural networks, in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

·yi = f (yi, u), t ∈ [0,T ], yi(0) = xi .

www.qmul.ac.uk m.benning@qmul.ac.uk

A lot of research in this direction

• Yann LeCun. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21–28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.

• Weinan E, A Proposal on Machine Learning via Dynamical Systems, Comm. in Math. and Stat. 2017. • B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert and E. Holtham, Reversible Architectures for Arbitrarily Deep Residual

Neural Networks, arXiv: 1709.03698v1, AAAI (National Conference on Artificial Intelligence). • E. Haber, L. Ruthotto, Stable Architectures for Deep Neural Networks, arXiv: 1705.03341v2 • Qianxiao Li Shuji Hao, An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks • Qianxiao Li, Long Chen, Cheng Tai, Weinan E, Maximum Principle Based Algorithms for Deep Learning, Journal of Machine

Learning Research 18 (2018). • L. Ruthotto and E. Haber, Deep Neural Networks Motivated by Partial Differential Equations, arXiv:1804.04272 • Lu, Y., Zhong, A., Li, Q., Bin Dong. (2017, October 27). Beyond Finite Layer Neural Networks: Bridging Deep Architectures

and Numerical Differential Equations. arXiv.org. • Chen, T. Q., Rubanova, Y., Bettencourt, J., Duvenaud, D. (2018). Neural Ordinary Differential Equations. Presented at

NeurIPS. • Gholami, A., Keutzer, K., Biros, G. (2019, February 26). ANODE: Unconditionally Accurate Memory-Efficient Gradients for

Neural ODEs. • Zhang, T., Yao, Z., Gholami, A., Keutzer, K., Gonzalez, J., Biros, G., Mahoney, M. (2019, June 9). ANODEV2: A Coupled

Neural ODE Evolution Framework.

www.qmul.ac.uk m.benning@qmul.ac.uk

A lot of research in this direction

• Yann LeCun. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21–28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.

• Weinan E, A Proposal on Machine Learning via Dynamical Systems, Comm. in Math. and Stat. 2017. • B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert and E. Holtham, Reversible Architectures for Arbitrarily Deep Residual

Neural Networks, arXiv: 1709.03698v1, AAAI (National Conference on Artificial Intelligence). • E. Haber, L. Ruthotto, Stable Architectures for Deep Neural Networks, arXiv: 1705.03341v2 • Qianxiao Li Shuji Hao, An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks • Qianxiao Li, Long Chen, Cheng Tai, Weinan E, Maximum Principle Based Algorithms for Deep Learning, Journal of Machine

Learning Research 18 (2018). • L. Ruthotto and E. Haber, Deep Neural Networks Motivated by Partial Differential Equations, arXiv:1804.04272 • Lu, Y., Zhong, A., Li, Q., Bin Dong. (2017, October 27). Beyond Finite Layer Neural Networks: Bridging Deep Architectures

and Numerical Differential Equations. arXiv.org. • Chen, T. Q., Rubanova, Y., Bettencourt, J., Duvenaud, D. (2018). Neural Ordinary Differential Equations. Presented at

NeurIPS. • Gholami, A., Keutzer, K., Biros, G. (2019, February 26). ANODE: Unconditionally Accurate Memory-Efficient Gradients for

Neural ODEs. • Zhang, T., Yao, Z., Gholami, A., Keutzer, K., Gonzalez, J., Biros, G., Mahoney, M. (2019, June 9). ANODEV2: A Coupled

Neural ODE Evolution Framework.

DEEP LEARNING AS OPTIMAL CONTROL PROBLEMS

miny,u,W,μ

∑i=1

𝒞 (Wyi(T ) + μ) − ci2

+ ℛ(u),

subject to·yi = f (yi, u), t ∈ [0,T ], yi(0) = xi .

Deep learning optimal control problem

subject to·y = f (y, u), t ∈ [0, T ], y(0) = x .

miny, u

𝒥(y(T ))

Variational calculus: consider variations

y := y + εv , u := w + εw .

How does the system behave for ?·y = f (y, u) ε → 0

·y = f (y, u), t ∈ [0, T ], y(0) = x .

Variational calculus: consider variations

y := y + εv , u := w + εw .

How does the system behave for ?·y = f (y, u) ε → 0

Differential equation constraint

ddε ( d

dty(t))

f (y(t), u(t))ε=0

∂y f (y(t), u(t)) v(t) + ∂u f (y(t), u(t)) w(t)

subject to·y = f (y, u), t ∈ [0, T ], y(0) = x .

miny, u

𝒥(y(T ))

Variational equation

Jacobian of w.r.t. to f uJacobian of w.r.t. to f y

v(t) ∂y f (y(t), u(t)) v(t) + ∂u f (y(t), u(t)) w(t)=

Adjoint equationThe adjoint of the variational equation is a system of differential equations for a variable , obtained assumingp(t)

⟨p(t), v(t)⟩ = ⟨p(0), v(0)⟩, ∀t ∈ [0,T ] .

This implies⟨p(t), ·v(t)⟩ = − ⟨ ·p(t), v(t)⟩,

which yields ddt

p(t) = − (∂y f (y(t), u(t)))⊤

p(t) ,

with constraint (∂u f (y(t), u(t)))⊤ p(t) = 0 .

First-order necessary conditions for optimality

·y = f (y, u), y(0) = x ,Then, the boundary value problem system

miny, u

𝒥(y(T )) ,

·p = − (∂y f (y, u))⊤

p , p(T ) = ∂y𝒥(y)y=y(T )

0 = (∂u f (y, u))⊤ p , t ∈ [0, T ] ,

expresses the first order necessary conditions for optimality of

Sketch of proof: set up a Lagrange functional and compute the optimality conditions.

Associated Hamiltonian systemFor this optimal control problem, there is an associated Hamiltonian system with Hamiltonian

ℋ(y, p, u) := p⊤ f (y, u)

with·y = ∂pℋ, ·p = − ∂yℋ, ∂uℋ = 0.

The constraint implies that this is a differential algebraic equation of index one

0 = (∂u f (y, u))⊤ p

Provided the Hessian is invertible, we can regard this system as an ODE when applying numerical methods.

∂u,uℋ

Numerical discretisation of the optimal control problem

How do we discretise the boundary value problem system

·y = f (y, u), y(0) = x ,

·p = − (∂y f (y, u))⊤

p , p(T ) = ∂y𝒥(y)y=y(T )

0 = (∂u f (y, u))⊤ p , t ∈ [0, T ] ?

Symplectic Runge-Kutta methods are suited to this problem!

J. M. Sanz-Serna, Symplectic Runge-Kutta schemes for adjoint equations automatic differentiation, optimal control and more, SIAM Rev., 58 (2016), 3–33.

y[ j+1] = y[ j] + Δts

∑i=1

bi f [ j]i

f [ j]i = f (y[ j]

i , u[ j]i ), i = 1,…, s,

y[ j]i = y[ j] + Δt

∑l=1

ai,l f [ j]l , i = 1,…, s

p[ j+1] = p[ j] + Δts

∑i=1

bi ℓ[ j]i

ℓ[ j]i = − ∂y f (y[ j]

i , u[ j]i )T p[ j]

i , i = 1,…, s,

p[ j]i = p[ j] + Δt

∑l=1

ai,lℓ[ j]l , i = 1,…, s

Forward pass

Backward pass

y[0] = x

p[N ] := ∂𝒥(y[N ])

(∂u f (y[ j]i , u[ j]

p[ j]i = 0

biai, j + bjai, j − bibj = 0bi = bi

This is the symplectic part!

y[ j+1] = y[ j] + Δt f (y[ j], u[ j]),

p[ j+1] = p[ j] − Δt (∂y f (y[ j], u[ j]))T

p[ j+1],

y[0] = x

p[N ] := ∂y𝒥(y[N ])

0 = (∂u f (y[ j]i , u[ j]

p[ j+1]i

Special case: symplectic Euler

Has the advantage that bilinear forms are preserved after discretisation, i.e. will be preserved after discretisation.⟨p(t), v(t)⟩

Proposition: if the continuous optimal control system is discretised with a symplectic partitioned Runge-Kutta method with , then the first order optimality for the discrete control problem

bi ≠ 0

min{u[ j]

i }N−1j=0 ,

{y[ j]}Nj=1, {y[ j]

i }Nj=1,

𝒥(y[N ]),

subject toy[ j+1] = y[ j] + Δt

∑i=1

bi f [ j]i

f [ j]i = f (y[ j]

i , u[ j]i ), i = 1,…, s,

y[ j]i = y[ j] + Δt

∑m=1

ai,m f [ j]m , i = 1,…, s,

is satisfied.

What does this mean in practice?

If we use symplectic partitioned Runge-Kutta methods, then first-optimise-then-discretise and first-discretise-then-optimise

lead to the same optimal control problems.

·y = f (y, u)

y[ j]i = y[ j] + Δt

∑m=1

ai,m f [ j]m min

{u [ j]i }N−1

j=0 ,{y[ j]}Nj=1,{y[ j]

i }Nj=1,

𝒥(y[N ])

miny, u

𝒥(y(T ))optimise

optimise

discretise discretise

RUNGE-KUTTA NETWORKS

Runge-Kutta networks

The discretisation of the optimal control problem with symplectic partitioned Runge-Kutta methods inspires a range of different network architectures.

Activation function: with parameters f (y, u) = σ(Ky + β) u = (K, β)

y[ j+1] = y[ j] + Δts

∑i=1

bi f [ j]i

f [ j]i = σ(K[ j]

i y[ j]i + β[ j]

i ), i = 1,…, s,

y[ j]i = y[ j] + Δt

∑l=1

ai,l f [ j]l , i = 1,…, s

Runge-Kutta networks:

Runge-Kutta networks: special case , the ResNets = 1

y[ j+1] = y[ j] + Δt σ (K[ j]y[ j] + β[ j])Backpropagation: γ[ j] = σ′ (K[ j]y[ j] + β[ j]) ⊙ p[ j+1]

p[ j+1] = p[ j] − Δt (K[ j])⊤γ[ j]

Runge-Kutta networks: special case , the ResNets = 1

y[ j+1] = y[ j] + Δt σ (K[ j]y[ j] + β[ j])γ[ j] = σ′ (K[ j]y[ j] + β[ j]) ⊙ p[ j+1]

p[ j+1] = p[ j] − Δt (K[ j])⊤γ[ j]

Parameter gradient: ∂K[ j]𝒥(y[N ]) = Δt γ[ j] (y[ j])⊤

∂β[ j]𝒥(y[N ]) = Δt γ[ j]

ODENet

Activation function: with parameters f (y, u) = α σ(Ky + β) u = (K, α, β)

Example: ResNet with varying time steps, i.e.

y[ j+1] = y[ j] + Δt α[ j] σ (K[ j]y[ j] + β[ j]) ;

we refer to this model as the ODENet.

Parameter gradient: ∂α[ j]𝒥(y[N ]) = Δt ⟨p[ j+1], σ (K[ j]y[ j] + β[ j])⟩

ODENet

Example: ResNet with varying time steps, i.e.

y[ j+1] = y[ j] + Δt α[ j] σ (K[ j]y[ j] + β[ j]) ;

we refer to this model as the ODENet.

In some applications it can make sense to assume that the learned time steps have to lie in the simplex

S := α ∈ ℝN α[ j] ≥ 0,N

∑j=1

α[ j] = 1 .

Activation function: with parameters f (y, u) = α σ(Ky + β) u = (K, α, β)

NUMERICAL RESULTS

Binary classification: ci ∈ {0,1}

Hypothesis function: 𝒞(x) =1

1 + exp(−x)

miny,u,W,μ

∑i=1

𝒞 (Wyi(T ) + μ) − ci2

+ ℛ(u),

subject to·yi = f (yi, u), t ∈ [0,T ], yi(0) = xi .

Varying number of layers after discretisation

Activation function: σ(x) = tanh(x)

www.qmul.ac.uk /QMUL @QMUL

Datasets2D data sets

donut1d donut2d

spiral squares

Results for donut1d

Results for squares

Transformation of features

Transformations for fixed classifier

Robustness to initialisations

Optimisation

Adaptive time steps for ODENet

Comparison of Runge-Kutta NetworksWe now train different Runge-Kutta networks for the same classification tasks

Butcher tableau ResNet/Euler Improved Euler

Kutta(3) Kutta(4)

Comparison of Runge-Kutta Networks

ODENet for MNIST

REGULARISATION PROPERTIES

Regularisation properties

Let us consider the continuous formulation of the neural network constraint

y(t) = σ (K(t)y(t) + β(t))

Let us consider the continuous formulation of the neural network constraint

y(t) = σ (K(t)y(t) + β(t))

and modify it as suggested in *

−K(t)⊤

Then we can verify the following stability estimate if is chosen to be monotone: σ

∥y(t) − yδ(t)∥ ≤ C ∥y(0) − yδ(0)∥ , for t ≥ 0 .

Here is a constant and and denote solutions of the flow for initial values and .

C > 0 y(t) yδ(t)y(0) yδ(0)

*Lars Ruthotto and Eldad Haber. "Deep neural networks motivated by partial differential equations." arXiv preprint arXiv:1804.04272 (2018).

∥y(t) − yδ(t)∥ ≤ C ∥y(0) − yδ(0)∥ , for t ≥ 0 .

Proof:ddt ( 1

2∥y(t) − yδ(t)∥)

= − ⟨K(t)⊤(σ (K(t)y(t) + β(t)) − σ (K(t)yδ(t) + β(t))), y(t) − yδ(t)⟩= − ⟨σ (K(t)y(t) + β(t)) − σ (K(t)yδ(t) + β(t)), K(t)y(t) + β(t) − (K(t)yδ(t) + β(t))⟩

= ⟨ ddt

y(t) −ddt

yδ(t), y(t) − yδ(t)⟩

∥y(t) − yδ(t)∥ ≤ C ∥y(0) − yδ(0)∥ , for t ≥ 0 .

Proof:

= − ⟨σ (K(t)y(t) + β(t)) − σ (K(t)yδ(t) + β(t)), K(t)y(t) + β(t) − (K(t)yδ(t) + β(t))⟩= − ⟨σ(z(t)) − σ(zδ(t)), z(t) − zδ(t)⟩

for and .z(t) := K(t)y(t) + β(t) zδ(t) := K(t)yδ(t) + β(t)

ddt ( 1

2∥y(t) − yδ(t)∥)

= ⟨ ddt

y(t) −ddt

yδ(t), y(t) − yδ(t)⟩

≤ 0 ,

∥y(t) − yδ(t)∥ ≤ C ∥y(0) − yδ(0)∥ , for t ≥ 0 .

Proof:ddt ( 1

2∥y(t) − yδ(t)∥)

Integrating from 0 to now yieldst12

∥y(t) − yδ(t)∥2 −12

∥y(0) − yδ(0)∥2 ≤ 0 ,

which concludes the proof. ∎

≤ 0 .

Regularisation propertiesIs stability a desirable property in neural networks?

Explaining and Harnessing Adversarial Examples, Goodfellow et al, ICLR 2015.

Adversarial examples in image classification:

Regularisation propertiesHardening of deep neural networks:

suppose we have inputs and with , then our network produces outputs with

y(0) = x yδ(0) = xδ

∥x − xδ∥ ≤ δ

∥y(t) − yδ(t)∥ ≤ C δ .

limδ→0

∥y(t) − yδ(t)∥ = 0 .Hence, we observe

This continuous dependency is also a feature of regularisation operators;

We should discretise or train such that this stability estimate is preserved.

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

Connection to inverse problems for example with additional reaction term:

−λA⊤ ( Ay(t) − f)

Let and denote solutions of this flow for data and with and identical initial values. For this model we can derive the following stability estimate:

y(t) yδ(t) f f δ ∥f − f δ∥ ≤ δ

ddt ( 1

2∥y(t) − yδ(t)∥2) ≤ − λ ⟨A⊤(Ay(t) − f ) − A⊤(Ayδ(t) − f δ), y(t) − yδ(t)⟩

= − λ ⟨A(y(t) − yδ(t)) − ( f − f δ), A(y(t) − yδ(t))⟩

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)

ddt ( 1

2∥y(t) − yδ(t)∥2) = − λ ⟨A(y(t) − yδ(t)) − ( f − f δ), A(y(t) − yδ(t))⟩

≤ λ ⟨f − f δ, A(y(t) − yδ(t))⟩≤ λ∥f − f δ∥∥A(y(t) − yδ(t))∥

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)

ddt ( 1

2∥y(t) − yδ(t)∥2) ≤ λ∥f − f δ∥∥A(y(t) − yδ(t))∥

≤ λδ∥A∥∥y(t) − yδ(t)∥

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)Let and denote solutions of this flow for data and with and identical initial values. For this model we can derive the following stability estimate:

ddt ( 1

2∥y(t) − yδ(t)∥2) ≤ λδ∥A∥∥y(t) − yδ(t)∥

⟹ ∥y(t) − yδ(t)∥ ≤ ∥y(0) − yδ(0)∥ + λδ∥A∥ t

= λ∥A∥t δ

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)Let and denote solutions of this flow for data and with and identical initial values. For this model we can derive the following stability estimate:

∥y(t) − yδ(t)∥ ≤ λ∥A∥t δ .

Hence, we need to choose a stopping time to guarantee convergence of this stability estimate, i.e. .

t*limδ↓0

∥y(t) − yδ(t)∥ = 0

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)When do we converge to a solution of the inverse problem, i.e. ?y† Ay† = f

ddt ( 1

2∥yδ(t) − y†∥2) = ⟨ d

dtyδ(t), yδ(t) − y†⟩

= − ⟨K(t)⊤σ (K(t)yδ(t) + β(t)) + λA⊤ (Ayδ(t) − f δ), y(t) − y†⟩−⟨K(t)⊤σ (K(t)yδ(t) + β(t)), y(t) − y†⟩

first inner product

−λ ⟨A⊤ (Ayδ(t) − f δ), y(t) − y†⟩second inner product

Regularisation propertiesWhen do we converge to a solution of the inverse problem, i.e. ?y† Ay† = f

−⟨K(t)⊤σ (K(t)yδ(t) + β(t)), y(t) − y†⟩first inner product

= −⟨K(t)⊤σ (K(t)yδ(t) + β(t)) − K(t)⊤σ (K(t)y† + β(t)), y(t) − y†⟩+⟨K(t)⊤σ (K(t)y† + β(t)), y(t) − y†⟩

≤ ⟨K(t)⊤σ (K(t)y† + β(t)), y(t) − y†⟩Monotonic increase of σ

−⟨K(t)⊤σ (K(t)yδ(t) + β(t)), y(t) − y†⟩first inner product

≤ ⟨K(t)⊤σ (K(t)y† + β(t)), y(t) − y†⟩

K(t)⊤σ (K(t)y† + β(t)) y(t) − y†≤

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

ddt ( 1

2∥yδ(t) − y†∥2) = ⟨ d

= − ⟨K(t)⊤σ (K(t)yδ(t) + β(t)) + λA⊤ (Ayδ(t) − f δ), y(t) − y†⟩−λ ⟨A⊤ (Ayδ(t) − f δ), y(t) − y†⟩

second inner product

K(t)⊤σ (K(t)y† + β(t)) y(t) − y†≤

= −λ ⟨Ayδ(t) − f δ, Ay(t) − Ay†⟩

−λ ⟨A⊤ (Ayδ(t) − f δ), y(t) − y†⟩second inner product

= − λ ⟨Ayδ(t) − f δ, Ay(t) − f⟩−λ ⟨Ayδ(t) − f + f − f δ, Ay(t) − f⟩=

= −λ Ayδ(t) − f2

− λ ⟨f − f δ, Ayδ(t) − f⟩≤ λ f − f δ Ayδ(t) − f λδ∥A∥ yδ(t) − y†≤

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

ddt ( 1

2∥yδ(t) − y†∥2) = ⟨ d

= − ⟨K(t)⊤σ (K(t)yδ(t) + β(t)) + λA⊤ (Ayδ(t) − f δ), y(t) − y†⟩K(t)⊤σ (K(t)y† + β(t)) y(t) − y†≤ + λδ∥A∥ yδ(t) − y†

= ( K(t)⊤σ (K(t)y† + β(t)) + λδ∥A∥) yδ(t) − y†

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

ddt ( 1

2∥yδ(t) − y†∥2) ≤ ( K(t)⊤σ (K(t)y† + β(t)) + λδ∥A∥) yδ(t) − y†

Assumption (source condition): is bounded for all .∥K(t)⊤σ (K(t)y† + β(t)) ∥ t ≥ 0

⟹ddt ( 1

2∥yδ(t) − y†∥2) ≤ (C + λδ∥A∥) yδ(t) − y†

y(t) = −K(t)⊤σ (K(t)y(t) + β(t))

−λA⊤ ( Ay(t) − f)Assumption (source condition): is bounded for all .∥K(t)⊤σ (K(t)y† + β(t)) ∥ t ≥ 0

ddt ( 1

2∥yδ(t) − y†∥2) ≤ (C + λδ∥A∥) yδ(t) − y†

∥yδ(t) − y†∥ ≤ ∥yδ(0) − y†∥ + (C + λδ∥A∥) t

Regularisation propertiesAssumption (source condition): is bounded for all .∥K(t)⊤σ (K(t)y† + β(t)) ∥ t ≥ 0

ddt ( 1

2∥yδ(t) − y†∥2) ≤ (C + λδ∥A∥) yδ(t) − y†

∥yδ(t) − y†∥ ≤ ∥yδ(0) − y†∥ + (C + λδ∥A∥) t

Suppose and , then we could estimateA = Id yδ(0) = f δ

∥yδ(t) − y†∥ ≤ ∥f δ − f ∥ +(C + λδ∥A∥) t

Regularisation propertiesAssumption (source condition): is bounded for all .∥K(t)⊤σ (K(t)y† + β(t)) ∥ t ≥ 0

ddt ( 1

2∥yδ(t) − y†∥2) ≤ (C + λδ∥A∥) yδ(t) − y†

∥yδ(t) − y†∥ ≤ ∥yδ(0) − y†∥ + (C + λδ∥A∥) t

Suppose and , then we could estimateA = Id yδ(0) = f δ

∥yδ(t) − y†∥ ≤ δ +(C + λδ∥A∥) t

and conclude for .limδ↓0

∥yδ(t*) − y†∥ = 0 t* ∼ δ

Thank you for your attention!Acknowledgements:

deep learning as optimal control problems...learning, campus jussieu, sorbonne université, paris...

Documents

deep learning for revenue-optimal auctions with...

deep optimal stopping - jmlr

deep learning - oxford...

learning deep learning

deep learning in real world @deep learning tokyo

deep learning: an introduction from the nlp perspective deep...

deep learning – fall 2013 instructor: bhiksha raj paper:...

from reinforcement learning to deep reinforcement...

toward optimal fpga implementation of deep convolutional

optimal image to lidar deep learning regression for height

executing deep learning strategies masterclass preview -...

introduction to deep learning - computer graphics€¦ ·...

optimal real-time landing using deep networks

optimal auctions through deep learning · 1 introduction...

learning deep stochastic optimal control policies...

intro to deep learning - bptt bright minds … ·...

deep learning - south dakota state universitydeep learning...

power-optimal pipelining in deep submicron...

flying high: deep imitation learning of optimal control

ai?! · ai 1.machine learning from a signal processing...