A Camera-Projector System for Real-Time 3D Video

Marcelo Bernardes, Luiz Velho, Asla Sá, Paulo Carvalho

IMPA - VISGRAF Laboratory

Procams 2005

OverviewOverview

• How it works

• (b,s)-BCSL code

• Video + (b,s)-BCSL code

• Reconstruction pipeline

• Snapshots

• Why it works?

• Discussions

How it works - Step 1How it works - Step 1Projecting and capturing color patterns

– Two slides (S1, S2) having vertical color stripes specially coded are projected on the object. Each slide is followed by the projection of its color complement.

– A camera captures the four projected patterns on scene.

S1S1

S1’S1’

S2’S2’

S2S2

t0

t1

t2

t3

Object

Color slides

How it works - Step 2How it works - Step 2Camera/projector correspondence

– Zero crossings and projected color stripes are robustly identified in camera images using complementary slides.

– Projected color sequences are decoded for each zero crossing giving camera/projector correspondence.

-

-

Zero crossings Projected colors

Zero crossings in camera space

Corresponding stripe boundaries in projector space

+

How it works - Step 3How it works - Step 3Photometry and geometry reconstruction

– Geometry is computed using camera/projector correspondence images and calibration matrices.

– Texture image is obtained by a simple combination of each complementary slide pair. For example, the maximum of each channel gives an image that approximates the full white projector light.

Reconstructed texture Reconstructed geometry

+ *

(b,s)-BCSL code(b,s)-BCSL code

• The (b,s)-BCSL method defines a coding/decoding procedure for unambiguously finding the number of a stripe transition using s slides and b colors.

• In our case, we use b=6 colors (R,G,B,Y,M,C) and s=2 slides which gives two codes of length 900 as illustrated below:

• The color transitions R,G in slide 1 and G,C in slide 2 uniquely map to the transition number p in the O(1) decoding procedure.

Slide 2

Slide 1Stripe transition number = p

R G

CG…

…

1…

…

p 899

Slide 2 color sequence:

Slide 1 color sequence:

p+1p-1

Video + (b,s)-BCSL codeVideo + (b,s)-BCSL code

• The key for real time 3D video is the combination of the (b,s)-BCSL code with video stream.

• Our scheme has the following features:– A video signal is generated in such a way that each

frame contains a slide in the even field the and its complement in the odd field. Frames (S1,S1’) and (S2,S2’) are interleaved in time.

– The video signal is sent to both projector and camera. They are synchronized through a genlock signal.

– Synchronized camera output is grabbed and pushed into the reconstruction pipeline.

Reconstruction pipelineReconstruction pipeline

• Our reconstruction pipeline is as simple as possible achieving real time 3D video with high quality geometry and photometry at 30Hz. This is possible because:– Every input frame captured gives a new texture image (by combining

both fields). – New zero crossings and projected color map are computed for every

input frame and correlated to the previous frame zero crossings and projected color map. The (b,s)-BCSL decoded transitions give a new geometry set.

• The following diagram illustrates the reconstruction pipeline. The frame arrived at time ti gives texture pi from its fields and geometry gi by correlation with the frame arrived at time ti-1.

S1S1' S2S2' S1S1' S2S2'Video frames at 30Hz

t1 t2 t3 t4

p1 p2 p3 p4g2 g3 g43D Data

SnapshotsSnapshots

• The bunny-cube 3D video:

geometry

texture

Composed virtual scene

SnapshotsSnapshots

Moving hand (rendered with lines)

Computed surface normals

SnapshotsSnapshots

• Face and mouth movement

SnapshotsSnapshots

• Walking around

Why It works?Why It works?

• Complementary slides projection is suitable for both photometry and geometry detection.

• Projected stripe colors are robustly detected through camera/projector color calibration.

• Stripe transitions are robustly detected by zero-crossings.

• Slides are captured at 60Hz. This is fast enough for capturing “reasonably normal” motion between consecutive frames.

• Transition decoding is performed in O(1).• While objects move the stripes projected over

their surface remain practically stationary!

DiscussionDiscussion

Current System Embodiment uses NTSC video

Pros and Cons• Standard off-the-shelf equipment

– Widely Available and Good Cost-Benefit

• Small resolution– 640x240 per field. (It reduces the maximum number of stripes

around 75.)– Composite video signal has poor color fidelity. (It reduces the

transition detection precision at stripe boundaries).

Solution: High Definition Digital Video.

Alternatives for 3D VideoAlternatives for 3D Video

• Technologies– Range Sensors

(requires additional camera) – Stereo Methods

• Passive Stereo (not robust for real-time)• Active Stereo (needs a projected pattern)

Our system: active stereo with complementary color patterns

– Drawback: projector varying light can be uncomfortable for human subjects

– Solution: switching complementary slides leads to white light perception as the projection/capture frequency reaches the fusion limit.