computer vision for unmanned surface vehicles · semantic segmentation graphical model (ssm)...
TRANSCRIPT
Computer vision for unmanned surface vehicles
Matej Kristan
Visual Cognitive Systems LaboratoryFaculty of computer and information science
University of Ljubljana, Slovenia
Summer Course 2018 Ljubljana – Autonomous carsLjubljana, Slovenia | 12 Jul. 2018
Matej Kristan, [email protected], SSL-AC 2018
Research interests
1. Online density estimationKristan et al., IEEE SMCB 2013 ;Kristan et al., IVC2009; Kristan et al., PR2011 ; Narbutas et al., VC2017;
2. Deep structured networksTabernik et al., CVPR 2018 ; Tabernik et al., ICPR 2016 ; Tabernik et al., CVIU 2015 ; Kristan et al., SCIA 2013 ; Tabernik et al., ICVS 2013 ; Tabernik ICPR 2012
3. Robotic visionBovcon et al, IROS 218 ;Bovcon et al, RAS 2017; Uršič et al., IJRR 2017; Uršič et al., ICRA 2016 ; Mandeljc et al., ICRA 2016 ;Kristan et al., IEEE TCyb 2016;Uršič et al., IJRAS 2013 ;Kristan et al., IMAVIS 2013;Uršič et al., IROS 2012Skočaj et al., EPIROB 2010
4. Visual trackingLukežič et al., IJCV 2018 ; Čehovin et al., ICCV2017;Lukežič et al., CVPR 2017;Lukežič et al., IEEE TCyb 2017Kristan et al., IEEE TPAMI 2016 ;Čehovin et al., IEEE TIP 2016 ; Čehovin et al., WACV2016 ; Kristan et al., ICCV-W 2015 ; Kristan et al., ECCV-W 2014 ; Čehovin et al., IEEE TPAMI 2013 ; Kristan et al., ICCV-W 2013 ; Kristan et al., IEEE SMCB2010 ; Kristan et al., PR 2009 ; Kristan et al., CVIU 2009 ;
0. Industrial R&D
2/39
Matej Kristan, [email protected], SSL-AC 2018
What are Unmanned surface vehicles (USV)DARPA ACTUV (2016), ~40m
MR Mariner 560 (2013), ~6m Electric USV, ~1-2m
Rolls-Royce concept cargo ship (2016)
3/39
Matej Kristan, [email protected], SSL-AC 2018
Recently involved in two USV-related projects
Autonomous sailboat Autonomous electric boat
Member of steering committee As principal investigator (PI)
4/39
Matej Kristan, [email protected], SSL-AC 2018
Autonomous sailboat (2016 -): uTransat challenge
• Mission: Build an unmanned self-powered sailboat that will cross the Atlantic!
• Challenge established in 2005 (ISAE, Toluz, France)
• Partners: Zavod404 and University of Ljubljana (FE, FMF, FPP, FRI, FS)
• Several local supporters (Ocean tec and Justin Yacht Design)
Small size: ≤ 2.4mExclusively wind-poweredNO human interventionSpecified starting and finish region (>5000km)Finish within 4 monthsHas to periodically report position via GPS
~50 ships have tried and failed
5/39
Matej Kristan, [email protected], SSL-AC 2018
uTransat’s about students!
• Shell/sails design, Actuators, Sensory systems, Communication, Power system
• Climate/weather analysis, AI navigation system, Public relations, Funding
6/39
Matej Kristan, [email protected], SSL-AC 2018
It’s all coming together
• Initially ~100 students, 2 years later ~30 student
• Hardware done 𝛽, Software “done” 𝛽 – now integration starts 𝛽2
• Hopefully soon to see some real sea action!7/39
Matej Kristan, [email protected], SSL-AC 2018
Actually, there’ll be two boats!
• Major difference in hull design and actuation (hydraulics instead of motors)
• The predicted corridor and AI planning kept the same
8/39
Matej Kristan, [email protected], SSL-AC 2018
Unmanned surface vehicles (USV)
Project APSIS 2007:
• Small sized (~2m), “low”-cost electric USVs
• Safer renewable energy harvesting
• Hydroelectric plants maintenance (3D barim.)
• Green coastal communities
• Regular water and natural disaster control
Project ViAMaRo 2017-2020: [2 FTE/year; Partners FRI, FE, Harpha d.o.o., Robotina d.o.o.]
• Robust computer vision methods for autonomous water surface vehicles
• The project is focused on “Perception” for autonomy9/39
Matej Kristan, [email protected], SSL-AC 2018
USV hardware specs
• Size: 2.3 x 0.7m
• Weight: 55 kg
• Drive: electic propulsor (±90°)
• Sensors: GPS (diff.), IMU, compass, US (distance ~7m),
sonar (depth), anemometer (wind), cameras
• Communication: radio(2.4GHz,narr. band)
• Control: NAV (ARM 9), Whitebox (i7 PC, 16GB SSD)
• Batteries: LiPo ; 4h autonomy
[ motor/control + electronics/telemetry + whitebox]
Max. speed 2.5 m/s
10/39
Matej Kristan, [email protected], SSL-AC 2018
USV system scheme
NAV (ARM9)
• Auto-navigation types• Self-supervised PID• Manual control
WhiteboxCore i7, 3GHz
4 cores
UDP
2.4 GHz Narrow band
Control station
position: ….
velocity: ….
attitude: ….
wind: …
battery: …
depth: ….
range: …
video: ….
USB
1280x720(10fps)
11/39
Matej Kristan, [email protected], SSL-AC 2018
• Map + sonar keep USV from running aground
USV partially autonomous
• Finely tuned control systems (Vranac&Mozetič, Harpha Sea d.o.o.)
NAV (ARM9)
~1.5m
12/39
Matej Kristan, [email protected], SSL-AC 2018
• Problem: obstacles not present in maps
• Dynamic obstacles
Challenges: Static vs dynamic obstacles
NAV (ARM9)
~1.5m
13/39
Matej Kristan, [email protected], SSL-AC 2018
Challenges: Types of sensors
• Laser rangers (e.g., LIDARs)
• Single plane detection (boat tilting)
• Potentially use stronger lasers + multiple planes
• Disturbances due to water spraying, wakes, foam
• Problems: weight, energy consumption, partially submerged objects
14/39
Matej Kristan, [email protected], SSL-AC 2018
Challenges: Types of sensors
• Camera as energy-efficient information-rich sensor
• Full filed-of-view detection
• Challenge: what does the obstacle look like?
15/39
Matej Kristan, [email protected], SSL-AC 2018
Which visual property constitutes an obstacle?
• Problem: Obstacles not visually consistent
• Common to all obstacles: Surrounded (or floating on) by water
• Problem: Water not visually consistent across views
Three semantic regionsBottom region is water
16/39
Matej Kristan, [email protected], SSL-AC 2018
Semantic segmentation graphical model (SSM)
Observed measurementat i-th pixel.
Unobserved pixel label.
Prior distribution overpixel labels.
[Diplaros et al., TNN2007]
Efficient optimization derived by Kristan et al., IEEE TCYB 2016
Co
nd
itio
nal
pri
ors
Hyp
er-p
rio
rsdata term local consistency term
17/39
Matej Kristan, [email protected], SSL-AC 2018
Application for obstacle detection
Fitting+labelling(core i7 single thread): ~70fps Matlab
18/39
Matej Kristan, [email protected], SSL-AC 2018
Marine Obstacle Detection Dataset (MODD)
• MODD1 (monocular camera, 12 sequences) 1
• MODD2 (stereo, IMU, 28 sequences – 11675 frames) 2
• Different weather conditions and time of day
Gulf ofKoper
1Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 20162Bovcon, Mandeljc, Perš, Kristan. Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation, RAS 2018
19/39
Matej Kristan, [email protected], SSL-AC 2018
Marine Obstacle Detection Dataset (MODD)
• Obstacles and edge of water manually annotated
1Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 20162Bovcon, Mandeljc, Perš, Kristan. Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation, RAS 2018
20/39
Matej Kristan, [email protected], SSL-AC 2018
SSM segmentation examples
Kristan, Sulić, Kovačič, Perš, Fast image-based obstacle detection from unmanned surface vehicles, IEEE TCyb 2016
21/39
Matej Kristan, [email protected], SSL-AC 2018
Drawbacks of SSM
• Relies solely on visual information
• Over/under estimates water edge
• Fails in presence of visual ambiguities
[Kristan et al., IEEE TCYB 2016]
IMUUse IMUfor horizonestimation!
22/39
Matej Kristan, [email protected], SSL-AC 2018
Estimation of horizon line from IMU measurements
• Three differently oriented coordinate systemsFrom calibrationFrom IMU
. . .? ? ht
23/39
Matej Kristan, [email protected], SSL-AC 2018
ISSM vs SSM
Bovcon, Perš, Kristan, Improving vision-based obstacle detection on USV using inertial sensor, ISPA201725/39
Matej Kristan, [email protected], SSL-AC 2018
Drawbacks of ISSM
• A lot of false positive detections caused by
• sun flares on camera lens,
• sun glitter on the water surface,
• sea foam, …
[Bovcon et al., ISPA 2017]
Use stereofor consistencyverification!
26/39
Matej Kristan, [email protected], SSL-AC 2018
ISSM with stereo extension: segmentation
• Enforce segmentation consistency in both images!
• Modification of the graphical model [Bovcon et. al, ISPA 2017]
• Compute disparity and align L/R image pixels
Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 2018
27/39
Matej Kristan, [email protected], SSL-AC 2018
ISSM with stereo extension: appearance consistency
Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 2018
• Epipolar geometry of rectified stereo images
• NCC template matching with a high threshold
Filtered
Left image Right image
(15 false detections)
28/39
Matej Kristan, [email protected], SSL-AC 2018
ISSM vs IeSSM
IeSSM:
• Reduced false positives
• Increased true positives
29/39
Matej Kristan, [email protected], SSL-AC 2018
IeSSM visualization
Bovcon and Kristan, Obstacle Detection for USVs by Joint Stereo-View Semantic Segmentation, IROS 201830/39
Matej Kristan, [email protected], SSL-AC 2018
Issues remain
• Segmentation fails when driving
towards the sun
• Plan to integrate compass into our
segmentation to predict sun position
• Reflections are a major problem
• Currently exploring “modern” segmentation
approaches – CNNs, e.g., UNet1
1Ronneberger, Fischer, Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015
31/39
Matej Kristan, [email protected], SSL-AC 2018
• Water surface is approximately flat
• Detect obstacles by 3D outliers of a “plane model”
Detection by stereo reconstruction
32/39
Matej Kristan, [email protected], SSL-AC 2018
• Disparity computation – compute 3D point cloud by triangulation
• Robust plane fitting (RANSAC)
Detection by stereo reconstruction
3D point cloud
Disparity map
33/39
Matej Kristan, [email protected], SSL-AC 2018
IMU
Improving plane fitting & detection
• Stereo does not work without texture! – apply IMU constraints
• Apply semantic segmentation (SSM) to identify water pixels
• Verify detections over several time-steps to remove false positives
by 3D “fingerprints”
With SSM
With SSM
34/39
Matej Kristan, [email protected], SSL-AC 2018
Stereo-based detection
• Close-range detection
works fairly well
• Problems:
• Obstacles > 20m away
• Low obstacles
• Ongoing work:
• A new dataset with
LIDAR included
Muhovič, Mandeljc, Bovcon, Kristan, Perš, Depth Fingerprinting for Obstacle Tracking using 3D Point Cloud, journal submission 2018
35/39
Matej Kristan, [email protected], SSL-AC 2018
Other ongoing USV-related work
• Developing fast algorithms for long-term tracking1,2
• Camera motion simulators for testing trackers3
1Lukežič, Čehovin, Vojir, Matas, Kristan, FCLT -- A Fully-Correlational Long-Term Tracker, Arxiv 20182Lukežič, Vojir, Čehovin Zajc, Matas and Kristan, Discriminative Correlation Filter Tracker with Channel and Spatial Reliability, IJCV 20183Čehovin, Lukežič, Leonardis, Kristan, Beyond Standard Benchmarks: Parameterizing Performance Evaluation in Visual Object Tracking, ICCV 2017
36/39
Matej Kristan, [email protected], SSL-AC 2018
Outlook and conclusions
• Two examples of autonomous robotic boats
• Our work on obstacle detection
• Details in papers:
• Several open issues remain:
• Stereo system failures in homogenous regions, reflections
• Driving towards the sun
• Obstacle tracking, re-identification, long-term tracking
1Kristan et al., IEEE TCyb 2016 ; 2Bovcon et al., ISPA2017 ; 3Bovcon et al., RAS 2018 ; 4Bovcon & Kristan , IROS 2018; 5Muhovič et al., prepub 2018 ; 6Lukežič et al., IJCV 2018 ; 7Lukežič et al. arxiv 2018; 8Čehovin et al., ICCV 2017
37/39
Matej Kristan, [email protected], SSL-AC 2018
Online resources
http://www.vicos.si/Projects/Viamaro
MODD1, MODD2 + reference Matlab code
38/39
Matej Kristan, [email protected], SSL-AC 2018
Thanks
• Janez Perš, Borja Bovcon, Rok Mandeljc, Jon Muhovič, Duško Vranac,
Dean Mozetič, Stanislav Kovačič, Vildana Sulić, Aljoša Žerjal
• Harpha Sea d.o.o.
• Robotina d.o.o.
• Rok Capuder, Zavod404
and uTransat teams
ViAMaRo is a basic research project
funded by Slovenian Research Agency
ARRS under code J2-817539/39