A Viewpoint on Perception, Planning, and

Control

Claire Tomlin

Somil Bansal, Varun Tolani, Aleksandra Faust, Jitendra Malik

How to efficiently navigate an autonomous system with a monocular RGB

camera to a goal in an a priori unknown environment?

[Bansal, Tolani, Gupta, Malik, Tomlin, CoRL 2019]

𝒑·

𝒙 = 𝒗𝒄𝒐𝒔(𝜽) + 𝒅𝒙

𝒑·

𝒚 = 𝒗𝒔𝒊𝒏(𝜽) + 𝒅𝒚

𝒗·= 𝒂

𝜽·

= 𝝎 [Bansal, Tolani, Gupta, Malik, Tomlin, CoRL 2019]

Success rate in reaching the goal (%):

Time taken to reach the goal (s):

Average acceleration along

the trajectory (𝑚/𝑠2):

Average jerk along the

trajectory (𝑚/𝑠3):

model-based E2E

Success Rate

Control Profile

model-based E2E

Some lessons learned

• Data representation is important

• Optimal control can be too optimal

• Waypoint representation

vs

vs

More lessons learned

• Building on existing NN architectures

• Image and perspective distortions during training

• RL on supervised learning

Safety Challenges

• Monitor: Is the image data in the training distribution?

• What is the uncertainty around the output of the

perception module?

• How should this uncertainty affect the planning and

control?

• More complex environments?

SURREAL

SD3DIS

HumANav

SURREAL

SD3DIS[𝑥, 𝑦, 𝜃]

Robot State

[𝑥, 𝑦, 𝜃, 𝑣, 𝜔]

Human State & Control

Human Identity- Gender

- Texture (Clothing, Skin Color,

Facial Features)

- Body Shape (Tall, Short, etc.)

HumANav

SURREAL

SD3DIS

HumANav

[𝑥, 𝑦, 𝜃]

Robot State

[𝑥, 𝑦, 𝜃, 𝑣, 𝜔]

Human State & Control

Human Identity- Gender

- Texture (Clothing, Skin Color,

Facial Features)

- Body Shape (Tall, Short, etc.)

Summary

Incorporating perception in the control loop

Supervising learning using optimal control

• a perception-planning-control pipeline

• comparison with a more traditional SLAM pipeline

• applied to a vision-based navigation task

Models of human motion

Challenges for safe control