1 dirs meeting, february 8 th, 2008 prudhvi gurram 3d scene reconstruction from aerial video prudhvi...

1

DIRS Meeting, February 8th, 2008Prudhvi Gurram

3D Scene Reconstruction from Aerial Video

Prudhvi Gurram, Eli Saber, Harvey RhodyChester F. Carlson Center for Imaging Science

Rochester Institute of Technology

DIRS Group Meeting

February 8th, 2008

2


Outline

Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

3


Outline


4


Motivation Objective

Extraction of 3D geometry of a scene from multi-modal data sets Possible Approaches

Manual Interpretation of Stereo Imagery (Very intensive and time consuming for large areas in the order of days or even months)

Automated processing of video frames to build stereo mosaics for the extraction of 3D geometry

Combine this with information from LIDAR to improve the accuracy of the 3D Scene.

Combine the 3D coordinates with material properties from Hyperspectral imaging to render a 3D Scene which conforms both geometrically and radiometrically to real world

5


High-Resolution VideoHigh-Resolution VideoLidar DataLidar Data

Spectral ImagerySpectral Imagery

Spectrally-Accurate Spectrally-Accurate Scene ModelScene Model

Motivation

Rapidly construct radiometrically-correct scene models based on

multi-sensor data for use in DIRSIG synthetic scene generation

6


Outline


7


Video Frames

Pre-processing of the video frames

Ray InterpolationObject Extraction

and Modeling 3D Structure

Exterior Orientation (EO) and Interior Orientation (IO)

parameters

Orientation-corrected video frames

Stereo Mosaics

Block Diagram

8


Outline


9


Pre-processing of Video Frames Correct the orientation of the frames so that all the frames have same

orientation as the first frame. Observed motion parallax of objects is due to translational motion of

camera only. Effects of translational motion of camera in Z direction are lost during

digitization process.

)()( TPRTPRRRP worldworld

A world point can be expressed in camera coordinate system with Rotation matrix R and camera center at T as

worldP

10


Pre-processing (Contd…)

)( 11TPKP worldim

Considering the first frame as reference frame (rotation matrix is an identity matrix)

The image coordinates in any frame i are transformed by matrix A, to observe only translational motion in the sensor

)(11'iworldimiimim TPKPKKRAPP

iii

TPKR

k

ky

kx

P worldim

The image coordinates of this point are given by (Interior Orientation parameters are embedded in matrix K

11


Outline


12


Introduction

Why are we using Parallel Ray Interpolation? To convert the view from perspective view to parallel-perspective view To use motion parallax information (while creating mosaics) To make the stereo mosaics seamless

Perspective view Parallel view

13


Introduction - AnimatedPerspective ViewParallel ViewParallel View from Perspective View

Using Fixed Lines

14


Ray InterpolationViewpoint 2Viewpoint 1

InterpolatedViewpoint

Image (Mosaic) Plane

Point in the image planefrom viewpoint 1


Point in the image plane from the interpolated viewpoint

Acknowledgement:

Zhigang Zhu et al.,

City College of New York,

New York City, NY

15


Fast PRISM Algorithm

Block Diagram for building Left or Right Stereo Mosaic

Fixed Line

Fixed Line

Matching CurveControl Points

PRISM

Matching CurveControl Points

Stitching CurveControl Points

on Mosaic

Fixed LineControl Points


Global Transformation

Global Transformation


on Mosaic


On Mosaic

DestinationTriangles

SourceTriangles

SourceTriangles

DestinationTriangles

AffineTransformation

AffineTransformation

Stitched SlicesIn Mosaic

Frame K

Frame K+1

Preparing Input Imagesfor Geometric Transformation

Geometric Transformations

Transformed Image DataInput Video FramesColor Code

16


Geometry

Frame 1 Frame 2

17


Geometry

Fixed Lines

Image Frame

18


Geometry

Frame 1:

Frame 2:

Fixed Line

Fixed Line

Overlapped Region

19


Geometry

Frame 1:

Frame 2:

Source Triangles

20


Interpolating points along the stitching line

Viewpoint 2Viewpoint 1InterpolatedViewpoint

Image (Mosaic) Plane



Point in the image plane from the interpolated viewpoint

21


GeometryDestination Trianglesin the Left Mosaic

22


Motion Parallax

Frame 1 Frame 2Interpolated Frame(before triangulation)

23


Interpolation

Frame 1 Frame 2Overlay of Frames 1 and 2Interpolation

24


Interpolation

Frame 1 Frame 2Interpolation

25


Triangulation Problem

Frame 1 Frame 2Interpolated Frame(after triangulation)

26


Triangulation Artifacts

Before After

27


Modified Triangulation

Make sure that none of the triangles include regions with different motion parallax

Find edges of different regions and align the sides of triangles with the edges

But aerial images are too noisy to obtain continuous edges around different objects

Use segmentation The inner boundary of each segment forms an edge

of a region/object

28


Overlapped region

Frame 1:

Frame 2:

29


Segmented images

Segmented Frame 1:

Segmented Frame 2:

30


Frame k

Segments in Overlapped Region

One of the segments

Significant points using Convex Hull around the segment

Triangulation

31


Frame k+1Matching curve

The other part of the segment between matching curve and fixed line

Triangulation

32


Mosaic

“Orphan” Pixels Orphan pixels filled

• Using a constraint inherent in the Parallel-Perspective view

• Parallel view in dominant motion direction and Perspective view in the other direction

• Do not consider motion parallax along x direction

X

Y

Direction

33


Results – Set 1Motion

Parallax

Frame 1 Frame 2

Fast PRISM Modified PRISM

34


Results – Set 2

Fast PRISM

Modified PRISM

Visual Artifact

Corrected by our method

DIR cm

X 0.1

Y 4

Z 0

X

Y

Camera Motion

Direction

Z

35


Results – Set 3

Visual artifact Corrected by our method

RIT imagery – Collected by WASP Lite at an altitude of 3000ft and Frames captured at 3Hz frequency

1 8 14 39 48 78

Frames

36


Stereo Mosaic – Fast PRISM

37


Stereo Mosaic – Modified PRISM

Publications:1. P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel-Perspective

Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007.2. P. Gurram, E. Saber, and H. Rhody, “Segment-based Mesh Design for Building Parallel-Perspective

Stereo Mosaics”, To be submitted to IEEE Transactions on Image Processing

38


Outline


Stereo Geometry

39


Image Courtesy:Z. Zhu, A. Hanson and E. Riseman, “Generalized Parallel-Perspective Stereo Mosaics from Airborne Video,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, pp. 226 – 237, 2004.

Find disparity between the points

lr yy ,

lr yyy Disparity

)1(yd

yHZ

Distance from the

Sensor to the point

HZ

Height of the point from ground

yd

yHh

– the coordinate of any pixel in the right and left mosaics in the dominant motion direction respectively

– the baseline of the stereo mosaicsyd

41


3D from Passive Video

Left mosaic, right mosaic and Nadir mosaic are built by controlling the angle of the parallel view of the mosaics

1 5 10 15 20 25 30

Image Plane

Scene

Viewpoints

Left mosaic Nadir mosaic Right mosaic

Sensor motion

42


Object extraction and modeling

Nadir mosaic Segmentation

Tree regions

Building surfaces extraction using height information

Each surface

Boundary of each surface

Right mosaic

Left mosaic

Plane fit for each surface using disparity between

mosaics

Corners through Curvature Scale Space

Edges through line fit between corners

CAD model of each building

DTM

Digital Elevation Model (DEM)

Reconstructed scene

100 200 300 400 500 600 700 800 900 1000 1100

50

100

150

200

250

300

350

400

450

500

43


Results

… …Raw Video Frames

1 2 20 29

After Pre-processing

… …

Video collected over Center for Imaging Science,RIT using WASP LT at an altitude of 1000 ft and with an overlap of about 90% to 98% between frames

44


Stereo Mosaics

Nadir

Left Right

Stereo Pair

45


Object extraction

100 200 300 400 500 600 700 800 900 1000 1100

50

100

150

200

250

300

350

400

450

500

Nadir Mosaic Tree regions

Segmented Nadir Mosaic

46


Object modeling

Noise

Solar shadow

Problems in 3D model of a

building due to solar shadow and noise in

images

Hypotheses of symmetry in the building and flat surfaces on the

roof of the building

Reconstructed building from different perspectivesPublications:

1. P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel-Perspective Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007.

2. P. Gurram, S. Lach, E. Saber, H. Rhody, and J. Kerekes, “3D Scene Reconstruction through a Fusion of Passive Video and Lidar Imagery”, Proceedings of 36th Applied Imagery and Pattern Recognition Workshop (AIPR'06), Washington, D. C., 2006

47


Situation 1: Edges not Visible in Video

Solar shadow

Noise

There is no information in these cases as one planar surface merges with a neighboring surface at a different height during segmentation

Video

Lidar

Raw Lidar CAD Model from Lidar

Good Edges and Planes

48


Situation 2: Lidar Data is Undersampled

The reconstruction begins to break down• Edges misrepresented or missed altogether

• Segmentation Fails

Video

Good Edges, Corners, and Planes

Lidar

49


Situation 3: Textureless Non-planar Surfaces

No variation of disparity on the textureless surface

Fit a spherical model to the data using Levenberg-Marquardt algorithm

Video

Lidar

RIT Observatory

50


Outline


51


Conclusions and Future Work

Verification process is only as accurate as camera parameters are.

Model is being overfitted to the data. Optimize the model according to the

uncertainty in data using Bayesian networks.

Evidence from other modalities like lidar data can be used to refine the model.

52


Bayesian or Belief network

Image Understanding

Algorithms(Visual and Lidar)

Control System

Belief Network

53


An example of belief network

Region

Buildings Trees Terrain

Feature 1

Feature 2Feature 3

Feature 4 Feature 5

Feature op.

Feature op. Feature op.

Feature op. Feature op.

Known StructureKnown DataLearning from data

54


Thank you!

1 dirs meeting, february 8 th, 2008 prudhvi gurram 3d scene reconstruction from aerial video prudhvi...

Documents

future workdirs meeting

real worlddirs meeting

d coordinates

aerial videoprudhvi

camera center

correct scene models

d geometrycombine

video framescorrect