1
Vision Based Following of Locally Linear Structures using an Unmanned Aerial Vehicle
Sivakumar Rathinam, Zu Whan Kim,
Raja Sengupta
Center for Collaborative Control of Unmanned VehiclesUniversity of California, Berkeley
2
Motivation
Aim: Enable UAV use for infrastructure monitoring• Traffic monitoring, aqueduct inspection, pipeline monitoring ….
3
Our Technology
Keep the vehicle over the structure using with vision in the loop
Complement GPS waypoint navigation• Waypoint navigation to get the vehicle over the
structure• Lock it on using vision for accurate imaging
4
Forest Fire Monitoring
Inter-Office Cargo Delivery
Motivation
Unmanned Aerial Vehicles for Traffic Surveillance • The Ohio Department of Transportation, The Florida Department of
Transportation, The Georgia Department of Transportation
Lane changes, Average inter-vehicle distances, Heavy vehicle counts, Accidents, Vehicle trajectories, Type of vehicles etc.
The road should be in view.
Coifman et. al, Surface Transportation Surveillance from Unmanned Aerial Vehicles
“The turning radius of the fixed wing UAV is such that changing directions at waypoints can take some time and space until the vehicle regains its course. When traversing roadway links of lengths less than 400 ft, large portions of the links went unobserved.”
5
Motivation
Hanshin Expressway, Japan 1995 Alaska pipeline
The visual feedback compensates GPS inaccuracies and tracks the structure even it is shifted from the assumed location.
6
Generalization: Vision Based Following of Locally Linear Structures(Closed Loop on the California Aqueduct, June 2005)
7
The average error of the position of the vehicle from the curve was 10 meters over a length of 700 meters of the canal.
Algorithm ran at 5 Hz
Results Tracking the California Aqueduct
8
Current UAV Platform Configuration
Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance
Ground-to-Air UHF Antenna for ground operator interface
GPS Antenna for navigation
802.11b Antenna for A-2-A comm.
Payload Tray for on-board computations and devices
Payload Switch Access Door for enabling / disabling on-board devices
9
Current Payload Configuration
Off-the-shelf PC-104 with custom Vibration Isolation
Orinoco 802.11b Card and Amplifier for A-2-A comm.
Analog Video Transmitter for surveillance purposes
Printed Circuit Board for Power and Signal Distribution among devices.
Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft.
Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground.
10
Problem
Follow a given curved structure based on visual feedback.
Overhead View
11
Following a Structure using Visual Feedback
1. Structure detection
a. Learn the structure from a one example
b. Real time structure detection of the structure
c. Curve fitting
2. Tracking
a. Transformation of image to ground coordinates
b. Control the vehicle to follow the structure
Hardware in the loop setup and evaluation
Experiments
12
Basic Detection Idea
Locally linear: Structure should look approximately linear in each image
13
1a. Learning the Structure from One Example
Rectify image-Finding the vanishing point
14
1a. Learning the Structure from One Example
mean
variance
Road Template
•Mean intensity will show high variation at the boundary•The variance in between the boundary points should be low•Done off-line•Can be automated or manual
15
1b. Real Time Detection in each Image
Road Template
For every 4th horizontal scan line pick several boundary hypotheses-Each hypothesis is a pair of features (high local intensity gradient)-Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line
16
1b. Real Time Detection in each Image
Road Template
For every 4th horizontal scan line pick several boundary hypotheses-Each hypothesis is a pair of features (high local intensity gradient)-Score each hypothesis for match quality with learnt profile -Keep the best three hypotheses for each scan line
Corr(Ih’(p),L)
17
1c. Curve Fitting
Road Template
RANSAC for Curve Fitting
Pick four scan lines at randomand four center hypothesesi.e., one from each line
Fit a cubic splineScore the cubic spline
Pick the spline with the best score
Set of supporting scan linematches
18
Cal Road Detection on MLB Video(No Control)
Generic corridor detection by one-dimensional learning•Roads•Aqueducts•Perimeters•Pipelines•Power Lines
19
2a. Transforming Image to Ground Coordinates
Height is measured by the pressure sensors.
Use accelerometers and the gyros in the avionics package to calculate the transformation
• Roll and pitch
Internal calibration parameters
Z
Y
X
Coordinates
20
2b. Controlling the Vehicle to Follow the Structure
Find a connecting contour that joins the current position to the desired curve and follow that path
• Position and slope at the origin and the look ahead distance (Soatto 2000)
21
Literature review – Vision Based Road Following Systems
VITS (1988) • Tracked roads at 13 miles/hr
Dickmanns (1992)• Tracked roads in autobahn at speeds up to 62 miles/hr
Taylor et.al (1999)• Tracked roads at speeds up to 75 miles/hr
Eric Frew et.al (2003)• Unmanned Aerial Vehicle • Tracked roads at around 44 miles/hr
22
More dangerous stuff……Obstacle Avoidance
Experiment flown on a Sig Rascal airframe with a Piccolo avionics package and vision processing on an onboard PC104.
An 8.5 foot diameter balloon was used as the obstacle (distance currently calculated using GPS).
23
Flight Demonstration
24
Related Work
Vision-based obstacle avoidance has been studied primarily in the context of mobile ground robots.• Lenser ’03, Ohya ’00, Lorigo ‘97,
Vision based navigation of UAVs• Saripalli ’02, Shakernia ’02, Furst ’98 – Landing with known markings• Sinopoli ’01, Doherty ‘00 – Visual landmark navigation (terrain avoidance)
for helicopter• Ettinger ’02, Pipitone ’01, Kim ’03 – Pose estimation for aircraft
Obstacle/Collision Avoidance for UAVs• Mitchell ‘01 – Aircraft avoiding known aircraft• Sigurd ’03 – Aircraft with magnetic sensors• Sastry ‘03 – Helicopters avoiding known helicopters/obstacles• How ’02 – MILP for Obstacle Avoidance
Vision based obstacle avoidance• Barrows ’03 – Biomimetic reactive control
25
Related Research
Ground robots• Fixed baseline stereo – JPL, many others• Monocular map construction – Lenser (CMU), Kim (Berkeley)• Cooperative stereo - CMU
Optical Flow• Helicopter ground following – Srinivasan/Chahl (Australia)• Corridor following - USC helicopter• Micro UAV obstacle avoidance – Centeye
UAV depth map construction • Lidar – CMU Helicopter Project, Sastry (Berkeley Helicopter Project).• Vision + high precision IMU – Bhanu (joint with Honeywell)
Stereo Vision• GT Helicopter
26
Requires DepthTypically use Stereo Vision
Given the image coordinates of a feature in one image• if one can find the image coordinates of the feature in
the other image (feature matching), and• if one knows the rotation and translation of the two
image planes then one knows the world coordinates of the feature (Ego-motion Estimation)
27
Problem with Depth Estimation by Stereo Vision
ZZ+ Z-0
z
Increased accuracy requires increased camera separation
28
Accurate Depth Estimation is a Problem
Range error due to pixel errors is . fB
Z
dp
dZ
2
29
Approach
UAVs flying at low altitudes must autonomously avoid obstacles
Strategy• Segment the image into sky and non-sky
Non-sky in the middle OBSTACLE
• Strategy 1 Aim at the sky
• Strategy 2 If it looms faster than a threshold and is in the middle AVOID
Else do NOTHING
30
Tailored to………………..
For most UAV applications (>50 m), the obstacles of concern will be large objects such as towers, buildings or large trees
For these cases, the problem of obstacle detection is different from that of ground vehicles in environments cluttered with many obstacles.
VS
31
Segmentation at Moffet Airfield
Results for multiple regions found (only largest regions shown, dark blue represents all small regions)
32
Sky Segmentation
33
Flight Demonstration
-500
-450
-400
-350
-300
-250
-200
-150
-100
-50
0
-100 0 100 200 300 400
Balloon
y po
siti
on (
m)
x position (m)
direction of flight
autonomous control started
avoidance with GPS
-500
-450
-400
-350
-300
-250
-200
-150
-100
-50
0
-100 0 100 200 300 400
Balloon
y po
siti
on (
m)
x position (m)
direction of flight
autonomous control started
avoidance with GPS
34
Vision Processing
Classification: balloon/horizon correctly found in ~ 90% of images
Time results: ~2Hz (120ms SVM, 200-600 ms horizon)
35
Flying Low
Helicopter pilots fly low
FAA requires see and avoid
Find the freeway and follow it
36
Used Sectionals to build a Manhattan model at 300 feet (approx.)
Simulation testing of Control
37
Cal UAV: Target CapabilitiesObstacle Avoidance
Simulation testing of Control• Flight through Manhattan model (300+ feet)
38
End 2