Stabilization and Georegistration of Aerial
Video Over Mountain Terrain by Means of LIDAR
Mark Pritt, PhDLockheed Martin
Gaithersburg, [email protected]
Kevin LaTouretteLockheed MartinGoodyear, [email protected]
IGARSS 2011, Vancouver, CanadaJuly 24-29, 2011
2
Problem: Georegistration
· Georegistration is the assignment of 3-D geographic coordinates to the pixels of an image.
· It is required for many geospatial applications: Fusion of imagery with other sensor data Alignment of imagery with GIS and map graphics Accurate 3-D geolocation
· Inaccurate georegistration can be a major problem:
Misaligned GIS
Correctly aligned
3
Solution
· Our solution is image registration to a high-resolution digital elevation model (DEM): A DEM post spacing of 1 or 2 meters yields good results. It also works with 10-meter post spacing.
· Works with terrain data derived from many sources: LIDAR: BuckEye, ALIRT, Commercial Stereo Photogrammetry: Socet Set® DSM SAR: Stereo and Interferometry USGS DEMs
4
· Create predicted images from the DEM, illumination conditions, sensor model estimates and actual images.
· Register the images while refining the sensor model.· Iterate.
Methods
Aerial Video Sensor
Image Plane
Scene
Occlusion
Illumination
Shadow
Predicted Images
5
Methods (cont)
Predicted Image
from DEM
Predicted Image from
Aerial Image
Registration Tie Point
Detections
The algorithm identifies tie points between the
predicted and the actual images by means of NCC
(normalized cross correlation) with RANSAC
outlier removal.
6
· The algorithm uses the refined sensor model as the initial guess for the next video frame:
· The refined sensor model enables georegistration. Exterior orientation: Platform position and rotation angles Interior orientation: Focal length, pixel aspect ratio, principal point
and radial distortion
Methods (cont)
Initial Camera
•Estimate camera model
•Use camera focal length & platform GPS if avail.
Register
•Predict images from DEM and camera
•Register images with NCC
Refine
•Compose registration fcn & camera
•LS fit for better cam estimate
• Iterate
Next Frame
•Register to previous frame
•Compose with cam of prev. frame for init. cam estimate
Iterate
• Iterate for each video frame
Finish
•Trajectory•Propagate geo data from DEM
•Resample images for orthomosaic
7
Example 1: Aerial Motion Imagery
Inputs:
Aerial Motion Imagery over Arizona, U.S.
16 Mpix, 3.3 fps, panchromatic
1/3 Arc-second USGS DEM
Area: 64 km2
Post Spacing: 10 m
8
Example 1 (cont)
Problem: Too shaky to find moving objects
Zoomed to full resolution (1 m)
9
· Outputs: Sensor camera models Images georegistered to DEM Platform trajectory
Example 1: Results
10
Example 1 Results (cont)
ATV Vehicle Human
Pickup Truck
Video is now stabilized, and as a
result, moving objects are easily
detected.
11
Example 2: Oblique Motion Imagery
Inputs:Oblique Motion Imagery Over
Arizona, U.S.
16 Mpix, 3.4 fps, pan
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
12
Example 2: Results
Map coordinates
Stabilized Video Inset
Orthorectified Video
Background LIDAR DEM Aligned
Map Graphics
Target Tracking
Aligned Map
Graphics
13
Example 2 Results (cont)
IMAGE 1 Camera Iteration1 2 3
Num tie points: 319 318 282
RMSE: 17.4 4.8 2.9Mean Δx: 1.4 -0.7 0.1Mean Δy: -3.8 -0.1 0Sigma Δx: 15.8 4 2.5Sigma Δy: 6 2.6 1.5
IMAGE 591 Camera Iteration1 2 3
Num tie points 681 687 681
RMSE 2.7 0.6 0.3Mean Δx 1 0 0Mean Δy 0.9 0 0Sigma Δx 2.1 0.5 0.3Sigma Δy 0.9 0.2 0.1
· How fast does the algorithm converge?
1 2 30
5
10
15
20
Tie Point Residuals
RMSEmeansigma
Camera IterationIm
age
Pixe
ls
1 2 30
0.5
1
1.5
2
2.5
3
Tie Point Residuals
RMSEmeansigma
Camera Iteration
Imag
e Pi
xels
The initial error is high, but it
decreases after only several iterations.
Subsequent frames have better initial
sensor model estimates and require only 2
iterations.
14
Example 3: Aerial Video
Inputs:
Aerial Video Over Arizona, U.S.
720 x 480 Color 30 fps
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
15
Example 3: Results
Map coordinates
Orthorectified Video
Background Image
Draped Over DEM
Aligned Map
Graphics
16
Example 3 Results (cont)
Map Graphics Stay Aligned with Features in Video
17
Example 4: Thermal Infrared Video
Inputs:MWIR Video Over White
Tank Mountains in Arizona
1 Mpix, 3.3 fps
Commercial LIDAR DEM
Post Spacing: 2 m
18
Example 4: Results
BackgroundLIDAR DEM
Video Mosaic
Inset: Original Video
with Map Graphics Overlay
Video Mosaic Georegistered and
Draped Over Mountains in Google Earth
19
Demo
Click picture to play video
20
Conclusion
· We have introduced a new method for aerial video georegistration and stabilization.
· It registers images to high-resolution DEMs by: Generating predicted images from the DEM and sensor model; Registering these predicted images to the actual images; Correcting the sensor model estimates with the registration results.
· Processing speed is 1 sec per 16-Mpix image on a PC.· Absolute geospatial accuracy is about 1-2 meters.
We are developing a rigorous error propagation model to quantify the accuracy.
· Applications: Video stabilization and mosacs Cross-sensor registration Alignment with GIS map graphics