cv: perceiving 3d from 2d

MSU CSE 803 Stockman 1

CV: Perceiving 3D from 2D

Many cues from 2D images enable

interpretation of the structure of the 3D

world producing them


Topic roadmap Labeling 3D structure in a 2D image Labeling constraints on edge graphs Huffman-Clowes-Waltz labeling Other cues motion parallax shape from texture or shading stereo from two 2d images


Topic roadmap: mathematical models Shape from shading Depth from stereo Depth from focus Perspective transformation

(review)


Many 3D cues

How can humans and other machines reconstruct the 3D nature of a scene from 2D images?What other world knowledge needs to be added in the process?


Vocabulary for image labeling

Interpret the local structure of the scene in the image space


Some terms for local 3D structure

Crease: (convex + or concave -) formed by an abrupt change to a surface or the joining of two surfaces. Surface on both sides of the crease can be sensed

Blade: ( > ) as in the blade of a knife, where the normal to the occluding surface element. Occluding and occluded surfaces unrelated.

(left) intensity image of 3 blocks (right) result of 5x5 Prewitt operator


Limb: smooth object contour

Limb: (>>) formed by viewing a smooth 3D object, such as an arm or a soup can: when approaching the contour, the surface normal becomes perpendicular to the line of sight. (Right side of arrow is the occluding surface.)

egg

Soup can


Albedo and lighting mark: (M) surface mark or

change of “albedo” (reflectance) and not the 3d surface, creating an intensity contour in the image

shading: (S) illumination change due to a change in lighting or shadow on the surface, creating an intensity contour in the image


Labeling image contours interprets the 3D scene structure

+An egg and a thin cup on a table top lighted from the top right

Logo on cup is a “mark” on the material

“shadow” relates to illumination, not material


“Intrinsic Image” stores 3D info in “pixels” and not intensity.

For each point of the image, we want depth to the 3D surface point, surface normal at that point, albedo of the surface material, and illumination of that surface point.


Practice labeling the contours

(left) an unopened can of Brand X soda is a solid blue can with a bright orange block ‘X’. (right) an empty box with all four flaps open so even the bottom of the box is visible.


3D scene versus 2D image

Creases Corners Faces Occlusions (for

some viewpoint)

Edges Junctions Regions Blades, limbs, T’s


Labeling of simple polyhedra

Labeling of a block floating in space. BJ and KI are convex creases. Blades AB, BC, CD, etc model the occlusion of the background. Junction K is a convex trihedral corner. Junction D is a T-junction modeling the occlusion of blade CD by blade JE.


Trihedral Blocks World Image Junctions: only 16 cases!

Only 16 possible junctions in 2D formed by viewing 3D corners formed by 3 planes and viewed from a general viewpoint! From top to bottom: L-junctions, arrows, forks, and T-junctions.


Challenge !

Create a scene shot of no more than 2 blocks that

creates all 16 junctions in the image


How do we obtain the catalog? think about solid/empty

assignments to the 8 octants about the X-Y-Z-origin

think about non-accidental viewpoints

account for all possible topologies of junctions and edges

then handle T-junction occlusions


Blocks world labeling

Left: block floating in space

Right: block glued to a wall at the back


Try labeling these: interpret the 3D structure, then label parts

What does it mean if we can’t label them? If we can label them?


Waltz filtering discards edge interpretations spanning junctions


1975 researchers very excited very strong constraints on

interpretations several hundred in catalogue when

cracks and shadows allowed (Waltz): algorithm works very well with them

but, world is not made of blocks! later on, curved blocks world work

done but not as interesting


Backtracking or interpretation tree


Interpretation tree search


“Necker cube” has multiple interpretations

A human staring at one of these cubes typically experiences changing interpretations. The interpretation of the two forks (G and H) flip-flops between “front corner” and “back corner”. What is the explanation?

Label the different interpretations


Depth cues in 2D images


“Interposition” cue

Def: Interposition occurs when one object occludes another object, thus indicating that the occluding object is closer to the viewer than the occluded object.


interposition • T-junctions indicate occlusion: top is occluding edge while bar is the occluded edge• Bench occludes lamp post• leg occludes bench• lamp post occludes fence• railing occludes trees• trees occlude steeple


• Perspective scaling: railing looks smaller at the left; bench looks smaller at the right; 2 steeples are far away• Forshortening: the bench is sharply angled relative to the viewpoint; image length is affected accordingly


Texture gradient reveals surface orientation

Texture Gradient: change of image texture along some direction, often corresponding to a change in distance or orientation in the 3D world containing the objects creating the texture.

( In East Lansing, we call it “corn” not “maize’. )

Note also that the rows appear to converge in 2D


3D Cues from Perspective


3D Cues from perspective


More 3D cues

Virtual lines Falsely perceived interposition


More 3D cues

2D alignment usually means 3d alignment

2D image curves create perception of 3D surface


“structured light” can enhance surfaces in industrial vision

Sculpted object Potatoes with light stripes


Shape (normals) from shading

Cylinder with white paper and pen stripes

Intensities plotted as a surface

Clearly intensity encodes shape in this case


Shape (normals) from shading

Plot of intensity of one image row reveals the 3D shape of these diffusely reflecting objects.

cv: perceiving 3d from 2d

Documents

d imagesmsu cse

d surface point

d cues

d world

d object

junction d

d structurecrease

d info