isabella toschi, fabio remondino - eurosdr · 2015-10-28 · isabella toschi, fabio remondino 3d...

Isabella Toschi, Fabio Remondino

3D Optical Metrology (3DOM),

Bruno Kessler

Foundation (FBK),

Trento, Italy

EuroSDR/ISPRS Workshop on

Oblique Cameras and Dense Image Matching

19-20 October 2015

Southampton, UK

to demonstrate the behavior of different oblique imaging platforms in real-world

conditions and study the influence of the additional slanted cameras on:

The quality of 3D point triangulation is

evaluated under several imaging

configurations, by assessing

• accuracy/precision of 3D points;

• block deformations in object space.

• The completeness of the

reconstructed 3D scene is studied in

relation to the camera tilt angle and

the image overlap.

• Point cloud generation and filtering

experiences are reported.

• Two main approaches of 3D building

modelling are discussed taking into

account the contribution of oblique

imagery for façade reconstruction.

• 3D reconstruction experiences are

reported.

MIDAS 5 UltraCam Osprey I

Nadir Oblique 45° Nadir Oblique 45°

Sensor size (mm) 36 x 24

(Canon EOS 1Ds Mk3) 70 x 45

23.5 x 36 (L/R)

71.5 x 23.5 (F/B)

Focal length (mm) 80 100 51 80

FOV along/across (°) ~17/25 ~13.5/20.5 ~48/70 ~17/25.5 (L/R)

~51/17 (F/B)

MILAN (MIDAS 5) GRAZ (UltraCam Osprey I)

GSD (*) (m) ~0.10 ~0.12

Y/X Overlap (*) (%) 70/30 75/65

# Images Nadir/Oblique 125/375 20/160

(*) Computed on nadir images

MIDAS 5

Forward and backward oblique views

Side oblique views

along-track overlap

across-track overlap

UltraCam Osprey I



along-track overlap

across-track overlap

Side oblique views

MILAN (3 km by 5 km)

Camera network (0) (1) (2) (3) (4)

GRAZ (3 km by 1.5 km)

Camera network (1) (2) (3) (4)

GCP

CP

σGCP, CP

~ 0.05 m

Rupnik, E., Nex, F. and Remondino, F., 2013. Automatic orientation of large blocks of oblique

images. Int. Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences,

40(1/W1), pp. 299-304.

ID

Constant/

Unknown/

Observ.

Input

images EOR IOR+AP

GCPs,

observed

in nadir

GCPs,

observed

in oblique

Tie

points

1FX

C

Nadir

U

O

2FX

C Nadir +

oblique

U

O

3FX

C Nadir +

oblique

U

O

1FR

C

Nadir

U

O

2FR

C Nadir +

oblique

U

O

3FR

C Nadir +

oblique

U

O

• FR – free camera parameters (IOR + AP, Fraser calibration model)

• FX – fixed camera parameters (IOR + AP)

• 1 to 3 – GCP configurations

• Each test is performed both with and without GNSS information

Rupnik, E., Nex, F., Toschi, I., Remondino, F., 2015. Aerial multi-camera systems: Accuracy and

block triangulation issues. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 101, pp.

233–246

GCPs config. 1FX 2FX 3FX 1FR 2FR 3FR Dataset

0 - 4 0.49 0.47 0.55 0.47 0.45 0.52 Milan

1 - 4 0.38 0.36 0.39 0.31 0.29 0.31 Graz

Internal quality (precision) of the adjustment process:

mean re-projection error [pixel]

Take-away messages:

• From scenario to scenario, σ0 remains near constant due to the applied

robust weighting function (Apero).

• Introducing oblique images into the network, but allowing GCP image

measurements only in nadir images (2FX, 2FR), brings about a slight

improvement in the re-projection error.

• When GCPs are measured both in nadir and oblique views (3FR, 3FX),

the re-projection error grows.

• σ0 decreases in self-calibrating BBA, but the improvement is minor.

Accuracy of the adjustment process:

RMS error on CPs [m]

Take-away message:

• For the Milan dataset, observations in oblique views improve object point

accuracy in height.

MILAN

GCP

config.

RMS

[m]

No.

GCP

No.

CP 1FR 2FR 3FR 1FX 2FX 3FX

0

x 0.12

(0.16)

0.12

(0.14)

0.15

(0.15)

0.10

(0.16)

0.07

(0.13)

0.13

(0.16)

y 7 12 0.12

(0.18)

0.10

(0.12)

0.15

(0.14)

0.18

(0.17)

0.28

(0.13)

0.30

(0.14)

z 0.42

(0.32)

0.30

(0.30)

0.13

(0.20)

0.45

(0.34)

0.55

(0.31)

0.20

(0.20)

1

x 0.23

(0.26)

0.22

(0.21)

0.18

(0.16)

0.36

(0.26)

0.16

(0.20)

0.21

(0.17)

y 4 15 0.22

(0.19)

0.14

(0.13)

0.16

(0.14)

0.27

(0.18)

0.48

(0.14)

0.45

(0.15)

z 6.85

(0.76)

1.39

(0.54)

0.26

(0.22)

10.68

(0.77)

3.68

(0.56)

0.64

(0.20)

Influence of oblique views

GRAZ

GCP

config.

RMS

[m]

No.

GCP

No.


1

x 0.04

(0.04)

0.04

(0.04)

0.05

(0.05)

0.04

(0.05)

0.04

(0.06)

0.03

(0.05)

y 4 3 0.03

(0.03)

0.06

(0.04)

0.04

(0.03)

0.04

(0.03)

0.06

(0.04)

0.05

(0.04)

z 0.08

(0.04)

0.08

(0.10)

0.02

(0.04)

0.17

(0.20)

0.13

(0.18)

0.13

(0.16)

4

x 0.05

(0.04)

0.04

(0.06)

0.04

(0.06)

0.04

(0.06)

0.05

(0.15)

0.04

(0.13)

y 4 3 0.05

(0.04)

0.05

(0.04)

0.06

(0.03)

0.06

(0.03)

0.05

(0.03)

0.06

(0.06)

z 1.07

(0.07)

2.47

(0.13)

2.86

(0.18)

2.22

(0.21)

2.86

(0.28)

3.32

(0.42)



Take-away messages:

• For the Graz dataset the advantage of oblique views is less obvious.

• In the presence of unfavorable point distributions, nadir images (1FR)

deliver better Z-accuracy results if compared to 2FR or 3FR with the

oblique views included.

Influence of oblique views



Take-away message:

• For the Milan dataset, BBA without GNSS generally shows a gain in

accuracy if self-calibration is performed. In the GNSS-assisted solutions

this evidence is less pronounced.

MILAN

GCP

config.

RMS

[m]

No.

GCP

No.


0

x 0.12

(0.16)

0.12

(0.14)

0.15

(0.15)

0.10

(0.16)

0.07

(0.13)

0.13

(0.16)

y 7 12 0.12

(0.18)

0.10

(0.12)

0.15

(0.14)

0.18

(0.17)

0.28

(0.13)

0.30

(0.14)

z 0.42

(0.32)

0.30

(0.30)

0.13

(0.20)

0.45

(0.34)

0.55

(0.31)

0.20

(0.20)

1

x 0.23

(0.26)

0.22

(0.21)

0.18

(0.16)

0.36

(0.26)

0.16

(0.20)

0.21

(0.17)

y 4 15 0.22

(0.19)

0.14

(0.13)

0.16

(0.14)

0.27

(0.18)

0.48

(0.14)

0.45

(0.15)

z 6.85

(0.76)

1.39

(0.54)

0.26

(0.22)

10.68

(0.77)

3.68

(0.56)

0.64

(0.20)

Influence of self-calibration

GRAZ

GCP

config.

RMS

[m]

No.

GCP

No.


1

x 0.04

(0.04)

0.04

(0.04)

0.05

(0.05)

0.04

(0.05)

0.04

(0.06)

0.03

(0.05)

y 4 3 0.03

(0.03)

0.06

(0.04)

0.04

(0.03)

0.04

(0.03)

0.06

(0.04)

0.05

(0.04)

z 0.08

(0.04)

0.08

(0.10)

0.02

(0.04)

0.17

(0.20)

0.13

(0.18)

0.13

(0.16)

4

x 0.05

(0.04)

0.04

(0.06)

0.04

(0.06)

0.04

(0.06)

0.05

(0.15)

0.04

(0.13)

y 4 3 0.05

(0.04)

0.05

(0.04)

0.06

(0.03)

0.06

(0.03)

0.05

(0.03)

0.06

(0.06)

z 1.07

(0.07)

2.47

(0.13)

2.86

(0.18)

2.22

(0.21)

2.86

(0.28)

3.32

(0.42)



Take-away message:

• For the Graz dataset, freeing self-calibration parameters in the bundle

adjustment improves results in the Z coordinates for all GCPs

configurations.

Influence of self-calibration



Take-away messages:

• For the Milan dataset, in the presence of unfavorable point distributions

extrapolation effects emerge and are reflected in larger RMS error

values (the Z coordinate suffers the most).

• The inclusion of GNSS constraints largely mitigates the problem.

MILAN

GCP

config.

RMS

[m]

No.

GCP

No.


0

x 0.12

(0.16)

0.12

(0.14)

0.15

(0.15)

0.10

(0.16)

0.07

(0.13)

0.13

(0.16)

y 7 12 0.12

(0.18)

0.10

(0.12)

0.15

(0.14)

0.18

(0.17)

0.28

(0.13)

0.30

(0.14)

z 0.42

(0.32)

0.30

(0.30)

0.13

(0.20)

0.45

(0.34)

0.55

(0.31)

0.20

(0.20)

4

x 0.28

(0.28)

0.20

(0.24)

0.19

(0.23)

0.63

(0.27)

1.16

(0.22)

1.46

(0.25)

y 3 16 0.21

(0.19)

0.23

(0.09)

0.28

(0.13)

0.72

(0.18)

2.02

(0.10)

2.50

(0.11)

z 4.30

(0.72)

2.24

(0.52)

1.59

(0.22)

7.62

(0.70)

4.75

(0.52)

2.30

(0.24)

Influence of GCP distribution

GRAZ

GCP

config.

RMS

[m]

No.

GCP

No.


1

x 0.04

(0.04)

0.04

(0.04)

0.05

(0.05)

0.04

(0.05)

0.04

(0.06)

0.03

(0.05)

y 4 3 0.03

(0.03)

0.06

(0.04)

0.04

(0.03)

0.04

(0.03)

0.06

(0.04)

0.05

(0.04)

z 0.08

(0.04)

0.08

(0.10)

0.02

(0.04)

0.17

(0.20)

0.13

(0.18)

0.13

(0.16)

4

x 0.05

(0.04)

0.04

(0.06)

0.04

(0.06)

0.04

(0.06)

0.05

(0.15)

0.04

(0.13)

y 4 3 0.05

(0.04)

0.05

(0.04)

0.06

(0.03)

0.06

(0.03)

0.05

(0.03)

0.06

(0.06)

z 1.07

(0.07)

2.47

(0.13)

2.86

(0.18)

2.22

(0.21)

2.86

(0.28)

3.32

(0.42)


RMS error on CPs [m] Influence of GCP distribution

Take-away messages:

• The sensitivity of the Graz network is minor, especially when GNNS

constraints are included.

• The imaging geometry and extent of image overlap are the most

probable factors influencing this sensitivity.

Possible deformations and the level of noise in the derived 3D object

coordinates are evaluated. Two types of comparisons are performed:

• In order to establish the influence of the inclusion of oblique

images, scenarios 1FX and 1FR are compared to scenarios 3FX and

3FR. The influence of self-calibration on the triangulated 3D points

is expressed in the differences 1FX with 1FR and 3FX with 3FR.

Results obtained in GCP configuration 0 (Milan) and 1 (Graz) are

used.

• To determine the influence of the GCP distribution, the oblique

image blocks 3FR and 3FX, in all GCP configurations, are compared

against a reference result. The reference is unique for all comparisons

and was chosen to be the one reporting the optimal RMS error values

at CPs, namely 3FR in GCP configuration 0 (Milan) and 1 (Graz).

Mean differences, standard deviations and color coded maps of the

geometric differences (Euclidean distance) are produced.

Influence of GCP distribution

MILAN

Re

fere

nc

e:

3F

R in

GC

P c

on

fig

. 0

1 2 3 4

no-GNSS

μ = 0.18 m

σ = 0.09 m

μ = 0.50 m

σ = 0.27 m

μ = 1.10 m

σ = 0.51 m

μ = 0.98 m

σ = 0.63 m

GNSS

μ = 0.09 m

σ = 0.04 m

μ = 0.18 m

σ = 0.09 m

μ = 0.28 m

σ = 0.15 m

μ = 0.14 m

σ = 0.07 m

0

6σ [m]

Take-away messages:

• In both datasets, GCPs induce a ‘strain’ on the entire block, i.e. the discrepancies

are kept small within the general region enclosed by GCPs whereas extrapolation

errors increased away from this region.

• When GNSS constraints are introduced into the BBA, the magnitude of these

discrepancies is substantially reduced, as is the extrapolation error.

Influence of the tilt angle

𝛼 < tan−1𝑠

ℎ

Take-away messages:

• When the inequality holds, the point will not be occluded by the neighboring

building.

• The taller the city architecture the lower the camera incidence angle should be (s

and h play a major role in planning a successful urban survey campaign!)

• A compromise should be found between the camera tilt setting, given focal length,

sensor size, overlap and the geometry of the surveyed area.

View angle from the perspective centre to the foot of the building:

tilt angle ± the angular point position in the image

Street width

Building height

Influence of the overlap

Take-away messages:

• Because of the large dissimilarities between O/N views, points should be matched

across views of similar look direction (i) within the strip and (ii) from neighboring

strips.

• Both the along- and across-track fold calculated on the oblique images should

exceed 50% (but larger datasets and higher number of 3D points should be

managed!)

Reconstruction using only forward/backward

images within one strip

Reconstruction using only forward/backward

images within two adjacent strips

Graz dataset

UltraCam Osprey I - GRAZ dataset

SURE -

GENERAL VIEW

ONLY NADIR IMAGES

NADIR AND OBLIQUE IMAGES


MICMAC – GENERAL VIEW (nadir + oblique images)


PRE-FILTERING

MICMAC – CLOSE-UP VIEWS

(nadir + oblique images) POST-FILTERING

Mixed pixels filtering

(considering the direction of

the perspective rays and the

local surface directions).

Rupnik, E., Nex, F. and Remondino, F., 2014. Oblique multi-camera systems – orientation and

dense matching issues. International Archives of Photogrammetry, Remote Sensing and

Spatial Information Sciences, XL-3/W1, pp. 107-114.


PRE-FILTERING

POST-FILTERING

Vegetation filtering

(considering the RGB values

associated to extracted point

clouds)

PRE-FILTERING

MICMAC -

CLOSE-UP VIEWS

(nadir/oblique images)

important for the successive

building reconstruction

From the dense point clouds, there are basically two successive

steps:

Huge and unique mesh/polygonal model generation

3D modeling of each single building

(*) According to the OGC standard CityGML

General pipeline (based on Tridicon BuildingFinder module and filtered

point clouds)

Point cloud splitting into

tiles by user specified size.

Segmentation: the result of the segmentation

process is a partition of the points, where all

points in one segment belong to the same

shape.

Building ground height definition: a terrain

model is generated from the point cloud in for

determining the building ground heights.

Building generation: the detection of

building objects is performed by searching

for selected types of internal- (or user-)

defined roof types.

UltraCam Osprey I system - GRAZ dataset

From filtered

point cloud

To building

models

(LOD2, no roof

overhang)


SINGLE BUILDING MODELS

UNIQUE MESH MODELS

IGI PentaCam camera - DORTMUND dataset (EuroSDR/ISPRS Benchmark)

Microsoft Ultracam camera – TRENTO dataset (only nadir, 10 cm GSD)

demonstrated the behavior of different oblique imaging platforms in real-world

conditions and studied the influence of the additional slanted cameras on:

The inclusion of oblique imagery into the BBA

brings about:

- better accuracy on Z-coordinate;

- improved image block stability when the

overlap is small, or the GCPs distribution is

poor, or there is not GNSS assistance.

-To ensure high quality façade reconstruction

from oblique views, it is desirable that points

are intersected from at least two neighboring

strips.

- A compromise should be found between the

camera tilt setting, given focal length, sensor

size and the overlap within the flight lines.

- New solutions to derive structured

information out of unstructured point clouds

are required.

- The use of oblique imagery to provide

detailed information on building facades

should be integrated with data collection from

terrestrial viewpoints.

Isabella Toschi, Fabio Remondino

3D Optical Metrology (3DOM),

Bruno Kessler

Foundation (FBK),

Trento, Italy

Acknowledgments:

Ewelina Rupnik (IGN, France), Francesco Nex (Univ Twente / ITC, The Netherlands)

isabella toschi, fabio remondino - eurosdr · 2015-10-28 · isabella toschi, fabio remondino 3d...

Documents