Stockman CSE/MSU Fall 2008 1

Models and Matching

Methods of modeling objects and their environments;

Methods of matching models to sensed data for recogniton

Stockman CSE/MSU Fall 2008 2

Some methods to study Mesh models (surface) Vertex-edge-face models (surface) Functional forms: superquadrics (surface) Generalized cylinders (volume) Voxel sets and octrees (volume) View class models (image-based) Recognition by appearance (image-based) Functional models and the Theory of

affordances (object-oriented)

Stockman CSE/MSU Fall 2008 3

Models are what models do

Stockman CSE/MSU Fall 2008 4

What do models do?

Stockman CSE/MSU Fall 2008 5

Vertex-edge-face models

Polyhedra and extensions;Model the surface of objects

Stockman CSE/MSU Fall 2008 6

Vertex-Edge-Face model

Stockman CSE/MSU Fall 2008 7

Sample object

All surfaces are planar or cylindrical

Stockman CSE/MSU Fall 2008 8

Matching methods Hypothesize point correspondences Filter on distances Compute 3D alignment of model to

data Verify positions of other model points,

edges, or faces. You can now do this! LOTS of work in the literature on this!

Can work for many industrial objects (and human faces perhaps!)

Stockman CSE/MSU Fall 2008 9

Triangular meshes

Very general and used by most CAD systems.

Stockman CSE/MSU Fall 2008 10

Texture-mapped mesh dog

Courtesy of Kari PuliWith each triangle is a mapping of its vertices into pixels [r, c] of a color image. Thus any point of any triangle can be assigned a color [R, G, B]. There may be several images available to create these mappings.

3D SURFACE MODEL

SURFACE PLUS TEXTURE

Stockman CSE/MSU Fall 2008 11

Meshes are very general

They are usually verbose and often are too detailed for many operations, but are often used in CAD. (Volumetric cube models are actually displayed here: made from many views by Kari Pulli.)

Stockman CSE/MSU Fall 2008 12

Modeling the human body for clothing industry and …

Multiple Structured light scanners used: could this be a service industry such as Kinkos?

Actually cross sections of a generalized cylinder model.

Stockman CSE/MSU Fall 2008 13

Mesh characteristics

+ can be easy to generate from scanned data

Stockman CSE/MSU Fall 2008 14

Making mesh models

Stockman CSE/MSU Fall 2008 15

Marching cubeshttp://www.exaflop.org/docs/marchcubes/ (James Sharman)

"Marching Cubes: A High Resolution 3D Surface Construction Algorithm",William E. Lorensen and Harvey E. Cline,Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

Raster scan through image F(r, c).

Look for adjacent pixels, one above threshold and one below threshold.

Interpolate real coordinates for f(x, y) = t in between

Stockman CSE/MSU Fall 2008 16

Marching in 3D space F(s, r, c)

Some voxel corners are above threshold t and some are below.

Stockman CSE/MSU Fall 2008 17

PhD work by Paul Albee 2004

Used Argonne National Labs scanner High energy, high resolution planar

Xrays penetrate object rotating on a turntable

Computer aided tomography synthesizes a 3D volume of densities with voxel size of about 5 microns

Stockman CSE/MSU Fall 2008 18

Segmentation of Scutigera a tiny crablike organism

Slice j of material density

F( sj, r, c )“thresholded” volume

Stockman CSE/MSU Fall 2008 19

Some common 3D problems analyze blood vessel structure in head capture structure and motion of

vertebrae of spine analyze porosity and structure of soil analyze structure of materials automatic segmentation into regions automatic correspondence of 3D points

at two instants of of time 3D volume visualization and virtual tours

Stockman CSE/MSU Fall 2008 20

Scanning technique abstraction

CCD camera (row) material sample X-ray planes

scintillator

Pin head

rotate

X-rays partly absorbed by sample; excite scintillator producing one row in the camera image; rotate sample a few degrees and produce another row; 3D reconstruction using CT

Stockman CSE/MSU Fall 2008 21

Scutigera: a tiny crustacean

• organism is smaller than 1 mm

• scanned at Argonne

• volume segmented and

meshed by Paul Albee

• roughly ten million triangles

to represent the surface

• anaglyph created for 3D

visualization

(view with stereo glasses)

Stockman CSE/MSU Fall 2008 22

Presentation of Results to User

Can explore the 3D data using rotation/translation Can create stereo images from 3D data

Stockman CSE/MSU Fall 2008 23

Physics-based models

Can be used to make meshes;Meshes retain perfect

topology;Can span spots of bad or no

data

Stockman CSE/MSU Fall 2008 24

Physics-based modeling

Stockman CSE/MSU Fall 2008 25

Forces move points on the model; halt at scanned data

Stockman CSE/MSU Fall 2008 26

Fitting an active contour to image data

Stockman CSE/MSU Fall 2008 27

Balloon model for closed object surface

Courtesy of Chen and Medioni

Stockman CSE/MSU Fall 2008 28

Balloon evolution• balloon stops at data points

• mesh forces constrain neighbors

• large triangles split into 4 triangles

• resulting mesh has correct topology

• hard CS part is detecting when balloon should be stopped by data point

Stockman CSE/MSU Fall 2008 29

Physics-based models

Can also model dynamic behavior of solids (Finite Element Methods)

Stockman CSE/MSU Fall 2008 30

Tagged MRI: 3D interest points can be written to body!

The MRI sensor tags living tissue and can sense its movement. Motion of a 3D tetrahedral finite elements model can then be analyzed. FMA model attempts to model the real physics of the heart. Work by Jinah Park and Dimitry Metaxes.

Stockman CSE/MSU Fall 2008 31

Algorithms from computer graphics make mesh models from blobs

Marching squares applied to some connected image region (blob)

Marching cubes applied to some connected set of voxels (blob)

See a CG text for algorithms: see the visualization toolkit for software

Stockman CSE/MSU Fall 2008 32

The octree for compression

Stockman CSE/MSU Fall 2008 33

Generalized cylinders

Stockman CSE/MSU Fall 2008 34

Generalized cylinders

• component parts have axis

• cross section function describes variation along axis

• good for articulated objects, such as animals, tools

• can be extracted from intensity images with difficulty

Stockman CSE/MSU Fall 2008 35

Extracting a model from a segmented image region

Courtesy of Chen and Medioni

Stockman CSE/MSU Fall 2008 36

Interpreting frames from video

Can we match a frame region to a model?

What about a sequence of frames? Can we determine what actions

the body is doing?

Stockman CSE/MSU Fall 2008 37

Generalized cylinders

Stockman CSE/MSU Fall 2008 38

View class models

Objects modeled by the distinct views that they can

produce

Stockman CSE/MSU Fall 2008 39

“aspect model” of a cube

Stockman CSE/MSU Fall 2008 40

Recognition using an aspect model

Stockman CSE/MSU Fall 2008 41

View class model of chair

2D Graph-matching (as in Ch 11) used to evaluate match.

Stockman CSE/MSU Fall 2008 42

Side view classes of Ford Taurus (Chen and Stockman)

These were made in the PRIP Lab from a scale model.

Viewpoints in between can be generated from x and y curvature stored on boundary.

Viewpoints matched to real image boundaries via optimization.

Stockman CSE/MSU Fall 2008 43

Matching image edges to model limbs

Could recognize car model at stoplight or gate or in car wash.

Stockman CSE/MSU Fall 2008 44

Appearance-based models

Using a basis of sub images;Using PCA to compress bases;

Eigenfaces (see older .pdf slides 14C)

Stockman CSE/MSU Fall 2008 45

Function-based modeling

Object-oriented;What parts does the object have;

What behaviors does it have;What can be done with it?

(See plastic slides of Louise Starks’s work.)

Stockman CSE/MSU Fall 2008 46

Louise Stark: chair model Dozens of CAD models of chairs Program analyzes model for * stable pose * seat of right size * height off ground right size * no obstruction to body on seat * program would accept a trash can (which could also pass as a container)

Stockman CSE/MSU Fall 2008 47

Theory of affordances: J.J. Gibson

An object can be “sittable”: a large number of chair types, a box of certain size, a trash can turned over, …

An object can be “walkable”: the floor, ground, thick ice, bridge, ...

An object can be a “container”: a cup, a hat, a barrel, a box, …

An object can be “throwable”: a ball, a book, a coin, an apple, a small chair, …

Stockman CSE/MSU Fall 2008 48

Minski’s theory of frames(Schank’s theory of scripts)

Frames are learned expectations – frame for a room, a car, a party, an argument, …

Frame is evoked by current situation – how? (hard)

Human “fills in” the details of the current frame (easier)

Stockman CSE/MSU Fall 2008 49

Make a frame for my house

Item 1 Item 2 Item 3 Item 4 Item 5 Item 6