agile robotics program review ar team mit, lincoln lab, draper lab, bae august 8, 2008

Agile Robotics Agile Robotics Program ReviewProgram Review

AR TeamAR TeamMIT, Lincoln Lab, Draper Lab, BAEMIT, Lincoln Lab, Draper Lab, BAE

August 8, 2008August 8, 2008

Agile Robotics TeamAgile Robotics Team

Albert Huang

Luke Fletcher

Matthew Walter

Lenny ParitskyTroy JonesMitch Leammukda

Steve ProulxRob Truax

Missy Cummings

Jake Crandall

Birsen Donmez

Nicholas RoyRuss Tedrake

Edmund Williams

Ed Corbett

Mike Boulet

Bryt BradleyNira Manokharan

Seth Teller Matt Antone

Emma Brunskill

Geoff Carrigan

Mofe Uku

Jeong hwan JeonEmilio Frazzoli Jon How

Scott CyphersStephanie SeneffTara Sainath Lee HetheringtonAlex Gruenstein

Howard ShrobeAndrew CorreaJim GlassRandy Davis

Platformbuildup

Safety& intent

Situationalawareness

Planning& control

Supervisorinterface

Programsupport

Matthew Walter

Seth TellerMitch LeammukdaLenny Paritsky

Jim Liu

Paul Pepin

Brandon Luders

Daniela RusNicholas Roy

Whole-SSAsimulation

John Leonard

Ryne Barry

USA LIA

DDRE

CASCOM

ARL

AgendaAgenda0900 – 0930: Arrive MIT Kiva room (32G-449)

0930 – 0945: Informal introductions

0945 – 1030: Summary goals and status

1030 – 1045: Break, walk to demonstration venues

1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva)

1200 – 1300: Lunch (Highlights of other MIT robotics)

1300 – 1415: Technical briefings

1415 – 1430: Break

1430 – 1530: Technical briefings

1530 – 1630: Feedback and discussion

1630: Main group adjourns

1630 – 1700: Program management discussion

1700: Program review adjourns

Administrative NotesAdministrative Notes

• Name placards

• Room security

• Kiva conference room

• Bathrooms

• Handouts / briefing charts

AgendaAgenda0900 – 0930: Arrive MIT Kiva conference room (32G-449)0930 – 0945: Informal introductions0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch (Highlights of other MIT robotics)1300 – 1415: Technical briefings1415 – 1430: Break 1430 – 1530: Technical briefings1530 – 1630: Feedback and discussion1630: Main group adjourns1630 – 1700: Program management discussion1700: Program review adjourns

Project GoalsProject Goals

• Build upon foundation of DARPA Challenge – Real-time perception, planning, and control– Avoid brittle aspects (prior map, GPS) of DGC

• Collaborate with LIA, CASCOM– Develop “Agile Robotics” autonomous forklift

• Introduce new mobile manipulation aspects – Whole-pallet manipulation (Year 1)– Sub-pallet manipulation (Years 2-3)

• Demonstrate prototype at end of Year 1

• Migrate capability to Army vehicle Years 2,3

Summary StatusSummary Status• Tackling problem along four fronts, in parallel:

– Requirements analysis: LIA, CASCOM, Ft. Campbell– Simulation studies for sensing, dataflow, throughput– Mockup experiments, drive-by-wire elements– Full-scale prototype development, data collection

• At present, substantially ahead of schedule– Several September, December milestones achieved– Demonstrated components in simulation, on mockup– (Partially) working full-scale prototype by fall 2008

• Anticipated fruitful directions for Years 2, 3– Improved manipulation, reasoning, sensing, scaling

1. Autonomous vehicle (flatbed truck) arrives at reception area with two pallets2. Autonomous forklift meets arriving truck at reception area3. Forklift unloads one pallet, transports it to specified outdoor bulk lot ASL4. Forklift unloads second pallet, moves it to queueing area in pickup zone5. Later, forklift loads queued pallet onto indicated truck in pickup zone6. Autonomous truck departs pickup zone

Bulk lot

PickupReception

1

Scenario: A Commandable ForkliftScenario: A Commandable Forklift(Notional SSA Layout)


Bulk lot

PickupReception

Forklift

Supervisor

2

Demonstration ScenarioDemonstration Scenario


Bulk lot

PickupReception

ASL

3



Bulk lot

PickupReception

4

QA



Bulk lot

PickupReception

5



Bulk lot

PickupReception

6


Design RequirementsDesign Requirements• Develop plausible prototype capability

– Compatible with existing Army platforms– Safe, acceptable to Army personnel– Capable of locating, moving, placing pallets– Affordable (acceptable cost increment)– Rugged (operates outdoors on uneven terrain)

Principal DeliverablesPrincipal Deliverables

• Prototype Demonstration (March 2009)– Single voice-, gesture-commandable forklift– Whole-pallet manipulation: truck/ground to ground/truck

• Documentation of effort results– Detailed drive-by-wire conversion procedure– Study of task requirements, sensor choices– Report on system architecture and algorithms

• Plans for continued capability development– Technology transition plan (Years Two and Three)– Extend capability along several dimensions

• Fine-grain manipulation; higher-level reasoning; multiple forklifts and supervisors; adverse environments; GPS-denial

Working TimelineWorking Timeline

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr

Y2 Year 1

Requirements analysis, system & interface design, safety, user testing

Simulation development

Mockup forklift

Full-scale prototype

Capability migration

Visit to Visit to Visit to CASCOM Incursion Seamless Shouted Testing & LIA CASCOM Ft. Campbell Visit to MIT detection handoff warnings validation

Port DGC Speech Gesture Pallet Situational Whole-SSA codebase integration support detection awareness simulation

Drive-by-wire Sensor Planning Pallet prototyping placement and control engagement

Drive-by-wire Sensor Planning Pallet Pallet Vehicle prototyping placement and control engagement mobility interaction

Lincoln FMTV Laboratory studies

KickoffMeeting

Program Review

Year 1 Demo

(Rented forklift) (Purchased forklift)

Program Review

Today

Preliminaries: Drive-by-Wire Platform Development

Prototype AutonomousCapability Development

Milestone Status Milestone Status ((= completed)= completed)

• Capability 0.0: 30 Jun 08 Analyze and summarize existing SSA operational practices Demonstrate low-fidelity simulation of terrain, pallets, and trucks Develop prototype speech, gesture interface to command forklift

• Capability 0.1: 30 Sep 08 Build a partially-actuated mockup forklift

• Gather data on pallet approach, pallet and slot detection / localization– Develop prototype interface, perception, planning, control algorithms– Demonstrate mockup working in a real-world scenario to identify, localize,

select, engage, lift, transport, and place one pallet at a time as directed• Capability 0.2: 31 Dec 08

– Convert a stock manual forklift to drive-by-wire control Identify model to purchase Lease example of target forklift• Procure suitable sensor suite and compute server• Instrument leased forklift with sensor suite• Transfer drive-by-wire elements to prototype forklift

– Team members complete required OSHA forklift training– Understand degrees of freedom, sensing and control of forklift– Develop functional interface, perception, planning, control algorithms

• Capability 0.3: 31 Mar 09– Prototype demonstration, venue to be determined (tentatively Ft. Belvoir)



Y2 Year 1



Mockup forklift








KickoffMeeting

Program Review

Year 1 Demo


Program Review

Today



Understanding the TaskUnderstanding the Task• MIT team visits LIA, CASCOM, Ft. Campbell• Collect, study relevant Dept. of Army pamphlets• CASCOM forklift operator visits MIT• OSHA forklift operator certification for team members

DGC System ArchitectureDGC System Architecture

Perception

NavigatorMDF

Goal

Trajectory

Steer, gas, brake

Vehicle states

Drivable surface, Lane markings, Obstacles,

Traffic vehicles

Local mapDrivableSurface, Hazards

SituationalPlanner

Vehicle Controller

Landrover LR3

Vehicle State Estimator

SensorsSensorsSensorsSensorsSensorsSensorsSensorsRNDF

AR System ArchitectureAR System Architecture

• Existing elements (from DGC codebase)– Sensor handlers and terrain perception– Ethernet and CANbus networks

• Novel elements– Whole-SSA environment model– Interpretation of supervisor speech and gestures– Detection of trucks, pallets, loads, pallet slots– Forklift mast planning and control– Seamless autonomy handoff, return– Shouted warning detection– Visible and audible apparent intent


Drive-by-Wire Modifications

Mobility and ManipulationPlanning and

Control

Sensing andSituationalAwareness

Supervisor

Speech,Gestures

Failure: Diagnostic Information

Success: Confirming InformationTo bystanders,via annunciators

Standard Forklift Forklift

Operator

SupervisorInterface/

InterpreterTablet


Perception

Navigator

Goal

Trajectory

Steer, gas, brake, transmission, parking brake, mast

Vehicle state

Local mapTerrain,People,Trucks,Pallets

SituationalPlanner

Vehicle, mast Controller

Forklift

Vehicle State Estimator

SensorsSensorsSensorsSensorsSensorsSensorsSensorsSupervisorInterface/

Interpreter

Safety:Incursions,

ShoutedWarnings

Speech &Gesture

Interpretation

SensorsSensorsSensorsSensorsSensorsSensorsMicrophones

Whole-SSAEnvironment

Model

Visible, AudibleApparent Intent

(Novel elements(Novel elements shown in shown in red boxesred boxes))

Sensor data



Y2 Year 1



Mockup forklift








KickoffMeeting

Program Review

Year 1 Demo


Program Review

Today



Simulation StudiesSimulation Studies

• Experiments with:– Sensor placement– Truck, pallet, slot detection– SSA structure and layout– Planning, control algorithms– Supervisor interface & bi-directional dataflow

Whole-SSA SimulationWhole-SSA Simulation

• Simulated whole-SSA operation– Multiple forklifts, supervisors, customers

• Predict task throughput, wait times for a variety of operating parameters– Speed, # trucks, # supervisors, failure rate &c.



Y2 Year 1



Mockup forklift








KickoffMeeting

Program Review

Year 1 Demo


Program Review

Today



Mockup ForkliftMockup Forklift• Enables experiments with:

– Forklift elements to be converted to Drive-by-Wire:• Parking brake and latch release• Pedal brake• Steering wheel and column• Mast raise, lower, tilt• Mast tine spread and side-shift• Rapid transfer to prototype forklift when available

– Sensing, planning and control algorithms for:• Rear-wheel steering• Pallet detection, localization, approach• Pallet engagement (tine insertion), transport, placement• Unloaded, loaded mobility• Incursion detection (operator override)



Y2 Year 1



Mockup forklift








KickoffMeeting

Program Review

Year 1 Demo


Program Review

Today



Rental, Purchase ForkliftRental, Purchase Forklift• Enables experiments with:

– Sensor placement and sensor data under pallet approach

– Electrical interfaces to existing (electrical) DBW elements

– Characterization of suspension dynamics when under load

– Annunciator visibility/audibility • Purchase forklift as base platform

for eventual demonstration prototype– Toyota 8FGU-15 lift truck ordered May 19, 2008 – Toyota arranged expedited delivery: July 23, 2008 – Transfer actuated mechanisms from mockup in August – Begin integrated outdoor experiments August 2008

Principal Research AdvancesPrincipal Research Advances

• Operation in semi-structured environment

• Hierarchical task-level autonomy

• Voice/gesture interface from bot’s-eye-view

• Seamless autonomy handoff & return

• Annunciation of apparent intent

• Detection of shouted warnings

Semi-Structured EnvironmentSemi-Structured Environment

Idea: Give forklift a narrated, guided tour!

Manually-drivenforklift path

Operator utteranceand / or gesture

Hierarchical Task-Level AutonomyHierarchical Task-Level Autonomy

• Not tele-operated; Not fully autonomous

• But: at a useful operating point in between– Summon forklift to working area– Direct it toward one truck (of several)– Direct it toward one aspect (of several)– Direct it toward one pallet (of several)– Help it localize pallet, slots (if necessary)– Direct it to destination (bulk lot, issue area etc.)

• Rich, incremental path to full autonomy

Voice and Gesture InterfaceVoice and Gesture Interface

• PDA interface for commanding forklift

• Interpretation of supervisor’s speech

• Supervisor uses stylus gestures to: – Summon and

direct forklift – Confirm or edit

motion paths – Indicate which

pallet, slots areto be engaged

Autonomy Hand-off and ReturnAutonomy Hand-off and Return

• Fundamental design constraint:– Forklift pauses if human approaches– Relinquishes autonomy if s/he enters cabin– Operates indistinguishably from manual forklift– Returns to autonomous mode after human exit

• Perhaps after an explicit go-ahead from human

• Six distinct, independent safety layers– Described later in briefing

• Challenging implications for planning layer– Described later in briefing

Apparent IntentApparent Intent

• Visible/Audible mode annunciators – Words, symbols, LEDs on cabin exterior– Speakers for audible annunciation– Redundant modes for safety

• Robot announces when:– It is about to move– It is stuck and needs help– It is paused, awaiting

supervisor’s instructions

DemonstrationsDemonstrations0900 – 0930: Arrive MIT Kiva conference room (32G-449)0930 – 0945: Informal introductions0945 – 1030: Summary goals and status 1030 – 1045: Break, walk to demonstration venues1045 – 1145: Demonstrations (Hangar, Holodeck, Kiva) 1200 – 1300: Lunch [Highlights of other MIT robotics]1300 – 1415: Technical briefings1415 – 1430: Break 1430 – 1530: Technical briefings1530 – 1630: Feedback and discussion1630: Main group adjourns1630 – 1700: Program management discussion1700: Program review adjourns

agile robotics program review ar team mit, lincoln lab, draper lab, bae august 8, 2008

Documents

mit robotics

mit kiva room

scaling slide

mit kiva conference

program management discussion

technical briefings

summary goals

forklift unloads