infusing physics + structure into machine learning...physics-infused learning for robotics and...
TRANSCRIPT
![Page 1: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/1.jpg)
Anima Anandkumar
INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING
![Page 2: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/2.jpg)
2
TRINITY OF AI/ML
DATACOMPUTE
ALGORITHMS
![Page 3: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/3.jpg)
PHYSICS-INFUSED LEARNING
Data Priors
Learning = Computational Reasoning over Data & Priors
+Learning =
How to use Physics as a (new) form of Prior?• Learnability
• Generalization
![Page 4: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/4.jpg)
4
What are some of the hardest
challenges for AI?
![Page 5: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/5.jpg)
5
“ Mens sana in corpore sano.”
Juvenal in Satire X.
![Page 6: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/6.jpg)
6
ExplorersPlanetary, Underwater, and Space Explorers
TransformersSwarms of Robots Transforming Shapes and Functions
GuardiansDynamic Event Monitors and First Responders
TransportersRobotic Flying Ambulances and Delivery Drones
PartnersRobots Helping and Entertaining People
ROBOTICSMOONSHOTS
![Page 7: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/7.jpg)
7
MIND & BODYNEXT-GENERATION AI
Fine-grained reactive Control
Instinctive:Making and adapting plans
Deliberative:
Sense and react to human
Behavioral:Acting for the greater good
Multi-Agent:
![Page 8: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/8.jpg)
8
PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL
Data Priors
Learning = Computational Reasoning over Data & Priors
+Learning =
How to use Physics as a (new) form of Prior?• Learnability
• Generalization
![Page 9: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/9.jpg)
9
BASELINE: MODEL-BASED CONTROL (NO LEARNING)
𝑠𝑡+1 = 𝐹 𝑠𝑡 , 𝑢𝑡 + 𝜖
New State
Current State
Current Action (aka control input)
Unmodeled Disturbance / Error
Robust Control (fancy contraction mappings)
• Stability guarantees (e.g., Lyapunov)
• Precision/optimality depends on error
(Value Iteration is also contraction mapping)
![Page 10: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/10.jpg)
LEARNING RESIDUAL DYNAMICS FOR DRONE LANDING
𝑠𝑡+1 = 𝑓 𝑠𝑡 , 𝑎𝑡 + ሚ𝑓 𝑠𝑡 , 𝑎𝑡 + 𝜖
New State
Current State
Current Action (aka control input)
Unmodeled Disturbance
𝑓 = nominal dynamicsሚ𝑓 = learned dynamics
Use existing control methods to generate actions
• Provably robust (even using deep learning)
• Requires ሚ𝑓 Lipschitz & bounded error
![Page 11: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/11.jpg)
11
CONTROL SYSTEM FORMULATION
• Dynamics:
• Control:
• Unknown forces & moments:
Learn the Residual(function of state and control input)
Learn the Residual
![Page 12: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/12.jpg)
12
DATA COLLECTION (MANUAL EXPLORATION)
• Learn ground effect:
• (s,u): height, velocity, attitude and four control inputs
෨𝐹 𝑠, 𝑢 →
Spectral-Normalized4-Layer Feed-Forward
Spectral Normalization: Ensures ෩𝑭 is Lipshitz[Bartlett et al., NeurIPS 2017][Miyato et al., ICLR 2018]
Ongoing Research:
Safe Exploration
![Page 13: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/13.jpg)
13
PREDICTION RESULTS
Height (m)
Gro
un
d E
ffe
ct (N
)
![Page 14: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/14.jpg)
GENERALIZATION PERFORMANCE ON DRONE
Our Model Baseline
Heig
ht
Vertical Velocity
Spectral Regularized NN Conventional NN
Neural Lander: Stable Drone Landing Control using Learned Dynamics, ICRA 2019
Guanya
ShiXichen
ShiMichael
O’Connell
Kamyar
Azizzadenesheli
Rose
YuSoon-Jo
Chung
Yisong
Yue
![Page 15: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/15.jpg)
15
CONTROLLER DESIGN (SIMPLIFIED)
Nonlinear Feedback Linearization:
Cancel out ground effect ෨𝐹(𝑠, 𝑢𝑜𝑙𝑑):
𝑢𝑛𝑜𝑚𝑖𝑛𝑎𝑙 = 𝐾𝑠𝜂
Feedback Linearization (PD control)
𝜂 =𝑝 − 𝑝∗
𝑣 − 𝑣∗Desired Trajectory(tracking error)
𝑢 = 𝑢𝑛𝑜𝑚𝑖𝑛𝑎𝑙 + 𝑢𝑟𝑒𝑠𝑖𝑑𝑢𝑎𝑙Requires Lipschitz & small time delay
![Page 16: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/16.jpg)
16
STABILITY GUARANTEES
Assumptions:
Desired states along position trajectory bounded
Control updates faster than state dynamics
Learning error bounded (new): Bounded Lipschitz (through spectral normalization of layers
Stability Guarantee:(simplified)
𝜂(t) ≤ 𝜂(0) exp𝜆 − ෨𝐿𝜌
𝐶𝑡 +
𝜖
𝜆 − ෨𝐿𝜌
⟹ 𝜂(t) →𝜖
𝜆𝑚𝑖𝑛 𝐾 − ෨𝐿𝜌
Exponentially fast
Unmodeled disturbance
Lipschitz of NN
Time delaycontrol gain
![Page 17: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/17.jpg)
17
CAST @ CALTECHLEARNING TO LAND
![Page 18: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/18.jpg)
18
TESTING TRAJECTORY TRACKING
Move around a circle super close to the ground
![Page 19: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/19.jpg)
19
CAST @ CALTECHDRONE WIND TESTING LAB
![Page 20: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/20.jpg)
20
TAKEAWAYS
Control methods => analytic guarantees
Blend w/ learning => improve precision/flexibility
Preserve side guarantees
Sometimes interpret as functional regularization
(side guarantees)
(possibly relaxed)
(speeds up learning)
![Page 21: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/21.jpg)
21
Blending Data Driven Learning
with Symbolic Reasoning
![Page 22: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/22.jpg)
22
AGE-OLD DEBATE IN AI
Symbolic reasoning:
• Humans have impressive ability at symbolic reasoning
• Compositional: can draw complex inferences from simple axioms.
Representation learning:
• Data driven: Do not need to know the base concepts
• Black box and not compositional: cannot easily combine and create more complex systems.
Symbols vs. Representations
Combining Symbolic Expressions & Black-box Function Evaluations in Neural Programs, ICLR 2018
ForoughArabshahi
SameerSingh
![Page 23: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/23.jpg)
23
SYMBOLIC + NUMERICAL INPUT
Goal: Learn a domain of functions (sin, cos, log…)
Training on numerical input-output does not generalize.
Data Augmentation with Symbolic Expressions
Efficiently encode relationships between functions.
Solution:
Design networks to use both
symbolic + numeric
Leverage the observed structure of the data
Hierarchical expressions
![Page 24: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/24.jpg)
24
TASKS CONSIDERED
◎Mathematical equation verification○ sin2 𝜃 + cos2 𝜃 = 1 ???
◎Mathematical question answering ○ sin2 𝜃 +
2= 1
◎Solving differential equations
![Page 25: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/25.jpg)
25
EXPLOITING HIERARCHICAL REPRESENTATIONS
Symbolic expression Function Evaluation Data Point Number Encoding Data Point
![Page 26: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/26.jpg)
26
REPRESENTING MATHEMATICAL EQUATIONS
◎Grammar rules
![Page 27: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/27.jpg)
27
DOMAIN
![Page 28: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/28.jpg)
28
TREE-LSTM FOR CAPTURING HIERARCHIES
![Page 29: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/29.jpg)
29
DATASET GENERATION
◎Random local changes
Replace Node Shrink Node Expand Node
![Page 30: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/30.jpg)
30
DATASET GENERATION
◎Sub-tree matching
Dictionary key-value pairChoose Node Replace with value’s pattern
![Page 31: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/31.jpg)
Majority Class Sympy LSTM : sym TreeLSTM : sym TreeLSTM:sym+num
Generalization 50.24% 81.74% 81.71% 95.18% 97.20%
Extrapolation 44.78% 71.93% 76.40% 93.27% 96.17%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%A
CC
UR
AC
Y
EQUATION VERIFICATION
![Page 32: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/32.jpg)
32
EQUATION COMPLETION
![Page 33: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/33.jpg)
33
EQUATION COMPLETION
![Page 34: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/34.jpg)
34
TAKE-AWAYS
Vastly Improved numerical evaluation: 90% over function-fitting baseline.
Generalization to verifying symbolic equations of higher depth
Combining symbolic + numerical data helps in better generalization for both tasks: symbolic and numerical evaluation.
LSTM: Symbolic TreeLSTM: Symbolic TreeLSTM: symbolic + numeric
76.40 % 93.27 % 96.17 %
![Page 35: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/35.jpg)
35
TENSORS PLAY A CENTRAL ROLE
DATACOMPUTE
ALGORITHMS
![Page 36: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/36.jpg)
36
TENSOR : EXTENSION OF MATRIX
![Page 37: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/37.jpg)
37
TENSORS FOR DATAENCODE MULTI-DIMENSIONALITY
Image: 3 dimensionsWidth * Height * Channels
Video: 4 dimensionsWidth * Height * Channels * Time
![Page 38: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/38.jpg)
38
TENSORS FOR MODELS
STANDARD CNN USE LINEAR ALGEBRA
![Page 39: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/39.jpg)
39
Jean Kossaifi, Zack Chase Lipton, Aran Khanna, Tommaso Furlanello, A
Jupyters notebook: https://github.com/JeanKossaifi/tensorly-notebooks
TENSORS FOR MODELS
TENSORIZED NEURAL NETWORKS
![Page 40: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/40.jpg)
40
SPACE SAVING IN DEEP TENSORIZED NETWORKS
![Page 41: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/41.jpg)
41
T E N S O R L Y : H I G H - L E V E L A P I F O R T E N S O R A L G E B R A
• Python programming
• User-friendly API
• Multiple backends: flexible + scalable
• Example notebooks
Jean Kossaifi
![Page 42: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/42.jpg)
42© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TENSORLY WITH PYTORCH BACKEND
import tensorly as tl
from tensorly.random import tucker_tensor
tl.set_backend(‘pytorch’)
core, factors = tucker_tensor((5, 5, 5),
rank=(3, 3, 3))
core = Variable(core, requires_grad=True)
factors = [Variable(f, requires_grad=True) for f in factors]
optimiser = torch.optim.Adam([core]+factors, lr=lr)
for i in range(1, n_iter):
optimiser.zero_grad()
rec = tucker_to_tensor(core, factors)
loss = (rec - tensor).pow(2).sum()
for f in factors:
loss = loss + 0.01*f.pow(2).sum()
loss.backward()
optimiser.step()
Set Pytorch backend
Attach gradients
Set optimizer
Tucker Tensor form
![Page 43: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/43.jpg)
43
AIREVOLUTIONIZING MANUFACTURING AND LOGISTICS
![Page 44: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/44.jpg)
44
NVIDIA ISAAC —WHERE ROBOTS GO TO LEARN
![Page 45: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/45.jpg)
46
Cars Pedestrians Path
Lanes Signs Lights
Cars Pedestrians Path
Lanes Signs Lights
1. COLLECT & PROCESS DATA 2. TRAIN MODELS
3. SIMULATE 4. DRIVE
NVIDIA DRIVEFROM TRAINING TO SAFETY
![Page 46: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/46.jpg)
47
TAKEAWAYS
End-to-end learning from scratch is impossible in most settings
Blend DL w/ prior knowledge => improve data efficiency, generalization, model size
Obtain side guarantees like stability + safety,
Outstanding challenge (application dependent):
what is right blend of prior knowledge vs data?
![Page 47: INFUSING PHYSICS + STRUCTURE INTO MACHINE LEARNING...PHYSICS-INFUSED LEARNING FOR ROBOTICS AND CONTROL Data Priors Learning = Computational Reasoning over Data & Priors Learning =](https://reader030.vdocuments.net/reader030/viewer/2022040617/5f1f8b8229340d0d50248fd9/html5/thumbnails/47.jpg)
48
Thank you