mixed reality and tangible interface click to edit master ...anab/teaching/m2r-tui/elective... ·...

Click to edit Master title style

Interaction speciality – Computer Science Master - University Paris-Saclay

Mixed Reality and Tangible interface

3D scene acquisition

Jean-Marc Vezien

[email protected]

2

“Mixed Reality and tangible interfaces” Interaction speciality – Computer Science Master - University Paris-Saclay

Plan of the lecture

1. Intro : 3D vision and AR

2. 3D vision: how

3. 3D vision : applications

3


3D Vision for AR

4


3D reconstruction(here : stereo analysis

+ 3D matching)

Augmentation needs 3D acquisition

5


CG graphics+



+ 3D matching)

6


CG graphics

Compositing masks

(shadows, occlusions…)

+match move



+ 3D matching)

7


CG graphics

Compositing masks

(shadows, occlusions…)

Augmented image

+


match move


+ 3D matching)

8


3D vision: How ?

Sensor and techniques for 3D imaging:

• Active sensors

• Passive sensors

9


3D vision: How ?


• Active sensors

• Passive sensors

10


Active sensors

Generate a signal to analyze 3D

environment, usually light:

o Structured in geometry

o Using a coherent source (laser)

Influence environment: application

domain limited

11


Non optical active sensors

Many sensors use waves to penetrate matter and analyze

3D content

Echography (sounds)

Radiography, tomography, scanner (X rays),

Nuclear Magnetic Resonance (alpha rays),

Etc.

Very expensive, highly specialized (medical)

12


LIDAR

Principle (radar):

Wave: infra-red laser with low amplitude modulated

signal.

Swipping with mirror

Mhzfm 3001

2T

13


LIDAR (Cont’d)

Ex.: LADAR from Perceptron Inc.

http://www.perceptron.com

• Laser diode @ 835 nm

• 50 mW power

• Distance: 3 to 100 m.

• Viewing angle

- horizontal: from 15 to 60 °- vertical: from 3 to 72 °

• Resolution: 1000 x 2000 pixels

14


LIDAR (Cont’d)

15


Triangulation-based sensor

Principle:

Camera

source

Surface

16


Several methods

«natural » light , with several illuminating

patterns

coherent light: laser stripe created with

cylindrical lens

Triangulation-based sensor

17


Structured lighting: results

18


Structured lighting: Kinect

Kinect, 2010

“Augmented Reality Magic Mirror

using the Kinect “

Tobias Blum

IR structured light

19


Structures Lighting : Hololens

Hololens (2016, 3000$)

• Augmented reality « see-through » glasses

• Markerless 3D scanner with :

• Depth camera (Kinect-like) Spatial Mapping

SLAM = Simultaneous

Localization and Mapping

= markerless tracking !

20


Triangulation scanner : cyberware

Laser sweep

Cyberware (1998)

Artec (2014): 12 sec. Scan.

21


Acquisition time: a few seconds (no real-time)

Controlled environment: low ambient light, no scattering (metals), avoid blue colors.

Distance: few centimeters to few meters

Cost (100 k€)

Triangulation sensors: problems

22


Triangulation sensors: problems

As the separation between emitter and receptor

increases:

Better precision

Increasing chance of occlusions

reception

Difficult compromise

emission

23


Triangulation: results

24


Triangulation: results

Integration of multiple scans: several million

polygons

25


Moiré effect

Grid 1Grid 2

Light fringes

Light source

Dark fringes

Camera

26


Problems:

Small depth of field

Controlled environment

Sophisticated image analysis (not real-

time)

No absolute Z: ambiguity.

Moiré effect (Cont’d)

27


Moiré : results

Posture analysis

Historical review and experience with the use of surface topographic

systems in children with idiopathic scoliosis

XC Liu, JG Thometz, JC Tassone, LC Paulsen (2013)

Face scan

28


3D vision: How ?


• Active sensors

• Passive sensors

29


Passive sensors

Receive natural light, no active intervention

Ex: CCD or CMOS arrays

Many techniques can extract 3D information from images

30


Passive sensors (shape from X)

X = base for feature extraction in image

X = motion

X = shading (not shadows !)

X = illumination

X = texture

X = ..., combination of previous

31


Use extrinsic constraints to compute intrinsic

properties

Intrinsic

properties

Extrinsic

properties

The aspect of an object is a function of:

• geometry (form, rugosity …)

• photometry (color, texture)

• Light sources

• Position of camera w.r.t. object

+

Passive sensors (shape from X)

32


X = motion

• matching features: points, segments, regions, between

images.

• known camera motion reconstruction of (X, Y, Z).

33


Stereovision (special case of motion)

Generate dense 3D maps

(data redundancy)

Computing Differential Properties of 3-D Shapes from Stereoscopic Images

without 3-D Models (Devernay, 1994)

34


Z.(Xl – Xr) = F Tx = Cte.

Disparity : ΔX = .(Xl – Xr)

Disparity map = depth map !

Rectified stereo (parallel cameras)

35


Gesture capture device

• IR lighting (3 LEDS)

• Fast camera (150 fps)

• volume is 600 x 600 x 600 mm

• stated 0.01mm depth resolution

• precision < 0.2 mm static

• 1.2 mm for dynamic setups

• Real-time stereo computation for

fingers: all is in the software

• Patents pending !

Infra red stereo: Leap motion

1/10-inch High End VGA, CMOS

Camera 640H x 480V. 16-bit wide data

(2 x 8 bits interleaved images)

Tremor amplitude varies between 0.4 mm ± 0.2 mm

36


Many advantages:

• Rather simple, accessible, many existing software

solutions

• Robust w.r.t. outside conditions

• Dense 3D maps possible

• Real-time computation possible (GPU, FPGA)

Stereovision

37


Limitations, constraints:

need to match primitives (points,

segments, regions)

calibrate stereoscopic cameras…

or buy specific hardware

resulting 3D model is weakly

structured (cloud of 3D points)

Stereovision

38


X = shading

Compute surface orientation from single image, with hypothesis on:

Surface nature (homogeneous + type)

Illumination (Ex: point source at infinity)

Approach introduced at the beginning of XXth century

Opposition Surge: moon is made of Regolith

Full moon is 12x brighter than half crescent

(should be x 2) Hapke parameters

39


INRIAlpes, 2006

Shape from shading : results

40


Many assumptions:

One homogeneous surface

Very sensitive to illumination

Orientation only

Well adapted to platenary or aerial

observation

Shape from shading : results

41


Shape from lighting

The aspect of a surface varies a lot as a function of

illumination

Light sources

Absorbing

coating

Camera

42


Hypotheses: surface is continuous (Z=f(x,y)), with uniform material.

Each light produces one constraint I(x,y) per point of surface Z=f(x,y).

Enough constraints problem solved.

Ex: Lambertian surface:

norientatiosurface

factorreflexion = 1 <A <0

i image =

i sourcelightofposition

)cos( minimize

2

1

n

i

i

M

i

nii

I

AIE

Shape from lighting

43


Georghiades (2001)

Shape from lighting: results

44


Problems:

Too many assumptions

Contrained environment

Qualitative results

Shape from lighting

45


Statistical SFS

2010: “3D face recovery from intensities of general and unknown lighting

using partial least squares”

Ham Rara, Shireen Elhabian, Thomas Starr, Aly Farag

CVIP Laboratory, University of Louisville (USA)

46


X = texture

Surface with regular (repetitive) patterns

pattern deformation = f (depth)

47


Two steps:

Image analysis: Compute a transform capturing

deformation. Ex: Fourier transform, 2D

momentums, etc.

Convert measure into depth with regularity

assumption

Shape from texture

Combined with stereo: stereo from texture

48


Shape from texture: results

49


Limitations:

Single surface, single homogeneous texture.

Sensitive to illumination conditions.

Use with caution

Shape from texture

50


Implicit reconstruction

It is not mandatory to compute 3D to see 3D objets (depend on

application). Ex: movie industry.

Is it possible to generate «intermediary» images between 2

reference frames ?

= Estimation of equivalent motion for all pixels in « virtual » view.

Knowing P1(x,y) et P2(x,y), can one predict P3(x,y), assuming the

position Cam3 relative to Cam1 et Cam2 is known ?

51


Implicit image interpolation

52


Ex: for a plane, projections are linked by homography :

Estimation of H : matching features between

images (points, regions…)

1

y

x

= p

...

...

...

= H andwithpHp LR

pL pR

P

Implicit 3D surface reconstruction

53


Mirror original

painting

Region map (hand drawn) Depth map

3D Interpolation : example

Global 3D Planar Reconstruction with Uncalibrated Cameras and Rectified Stereo Geometry

Jean-Philippe Tarel (1999).

54


Augmented image acquisition: result

55


Augmented image acquisition: stitching

http://www.ptgui.com/Hugin : http://hugin.sourceforge.net/

56


3D Vision: applications

57


3D VISION: applications

• Medical imaging

• Ergonomics / Biometry

• Building, CAD

• Inspection, control

• Exploration of 3D environments

• Assisted driving

• Military applications

• Satellite imagery

• Virtual Reality

• Multimedia content creation

58


Medical imaging

• Computer assisted diagnosis: indication of volumes,

shapes, etc.

• Surgery planning

• Forensic medecine

59


Techniques :

Moiré, stereovison, laser-based imaging

(shape analysis)

Application:

• Detect deformations

• Prosthetic fitting

Medical imaging

Design And Virtual Prototyping Of Rehabilitation Aids (1997)

Venkat Krovi , G. K. Ananthasuresh , Jean-marc Vezien

60


Ergonomics, anthropometric measures

• Goal: determine precise

characteristics for each individual

Jack (U.penn):

graphical system for

human simulation

• Objective: measure, adapt,

recognize

61


Techniques:

• Laser

• Stereo

• direct measurements

Application:

• Design/ergonomics


62



Workplace design

63



3D Motion capture

and analysis

http://www.swri.org

SOUTHWEST RESEARCH INSTITUTE

Cameras

IR

markers

64


Construction, CAD

Help make computer-generated objects:

• Sensors to guide tools precisely

• Design a virtual model (design, simulations).

• Tools: laser, stereo.

65


Construction, CAD

66


Inspection, control

Check that a manufactured object is

within specifications

• Micro-electronics, microchip

production.

• cars (Ex: wheel alignement)

Structured light, excellent precision:

1/1000 mm !

http://www.cognitens.com/

http://www.cyberoptics.com/

67


Exploration

Autonomous robots:

2010 : MDARS(Mobile Detection

Assessment Response System) -

US Army

1988: K3A Navmaster

(Cybermotion, Inc.)

http://www.public.navy.mil/spawar/Pacific/Robotics/Pages/Publications.aspx

68


Binocular heads (stereovision maps)

Semi-autonomous robots

Exploration

69


Techniques: Stereo, Lidar, ultrasounds

images: fast but not precise

Laser (lidar): slower but more precise.

Problems:

fusion of geometric data coming from multiple

sources

calibration of dynamic data

Exploration

70


Partial recontruction of the environment :

(semi-) autonomous driving

Limited interventions

Applications:

Hostile environment (nuclear, chemical, explosive)

Repetitive tasks (plants)

Still costly !

Exploration

71


«intelligent» car :

environnement sensing driver

Techniques:

• Laser, Stereo

• Ultrasound

Proximity sensorOn-board

computing

Augmented windshield

Assisted driving (smart car)

72


Navlab project (univ. de Carnegie Mellon, USA):

Several run-of-the-mill car tested.

Highway testing in real conditions

No vision (sensitivity) : lidar, magnetic beacons


DARPA Challenge : race between automated

vehicules

• Tough competition

• High reward 2 M$ high incentive

• Soon : urban driving 132-mile Mojave Desert course

7 Hours at speed = 19.1 mph

73



Tesla (2015)

Model S assistive driving

camera on top of the windshield

GPS sensor

360 °ultrasonic acoustic location sensors (close range)

forward-looking Lidar (long range)

100 000 $

Cloud-connected learning

Hands-off possible (road following, soon traffic lights…)

July 2016: Tesla driver dies in first fatal crash while using autopilot mode

The autopilot sensors on the Model S failed to distinguish a white tractor-trailer crossing the highway

against a bright sky.

74


Google car: a completely autonomous car !


75


Military applications

• Hostile/unknown environment

NAVLAB 2

Objectives:

Analyze terrain to assist driving

(wounded occupant, bad visibility,

night)

Techniques:

Lidar: obstacle avoidance, route

finding

Video + neural net = driving

76


Terrain reconstruction from a collection of

images from above.

Applications:

- climate (water set)

- ecological surveillance (rainforest, flood areas).

- military

- topography and mapping (google maps)

Aerial and satellite imagery

77



Olympus mons (Mars)

Washington area

Techniques:

• Stereoscopy

• Shape from shading

78


Limitations:

Beware of illumination assumptions !

The « face » on Mars


79


Virtual and Augmented Reality

Analyze Modify reality

Techniques:

• Stereoscopy

• Marker tracking

• Shape from motion

• Laser scan

• 3D object registration

80


Augmented reality

3D scene analysis can be

simple yet effective

Augmented content can be 3D

but not geometry-related

81


Planar AR: homography

)( dZnYnXn

dMnM

zyx

t

MHMd

nTR

d

MnTMR

TMRm

tt

.

~

Z

Y

X

hhh

hhh

hhh

s

sv

su

333231

232221

131211

8 unknowns 4 (m,M) pairs = 4 corners !

82


Homography for AR

MHm .~

Z

Y

X

hhh

hhh

hhh

s

sv

su

333231

232221

131211

X

Y

ʘZ = 0

2D 2D transform !

If we find H-1, we can remap image into the original 2D space

H

83


Homography computation

d

TnRH

t

Z

Y

X

H

s

sv

su

Points are in (X,Y) plane

Eq: Z = Z0

100

0 2322

131211

hh

hhh

H

Make 2D points on surface correspond to pixel

location (automatic or interactive)

write on any

surface

Writing with weemote

84


Augmented reality

Many applications:

• Architecture

• Design

• Medical

• Crisis simulation

• Marketing

• Multimedia/games

• Military …

85


THE END !

mixed reality and tangible interface click to edit master ...anab/teaching/m2r-tui/elective... ·...

Documents