Robust Calibration of Camera-Projector System for Multi-Planar Displays Mark Ashdown1, Rahul Sukthankar2,3

Cambridge Research Laboratory HP Laboratories Cambridge HPL-2003-24 January 30th , 2003* E-mail: mark@ashdown.name, rahuls@cs.cum.edu camera, projector, computer vision, homography

We present a robust calibration method for aligning a camera-projector system to multiple planar surfaces. Unlike prior work, we do not recover the 3D scene geometry, nor do we assume knowledge of projector or camera position. We recover the mapping between the projector and each surface in three stages. In the first stage, we recover planar homographies between the projector and the camera through each surface using an uncalibrated variant of structured light. In the second stage, we express the homographies from the camera to each display surface as the composition of a metric rectification and a similarity transform. Our metric rectification algorithm uses several images of a rectangular object. In the third stage, we obtain the homographies between the projector and each surface by combining the results of the previous two stages. Inconsistencies appear along the boundaries between adjacent surfaces; we eliminate them through a process of iterative refinement. Standard techniques for recovering homographies from line correspondences and performing metric rectification are very sensitive to image processing outliers. We present robust algorithms for both tasks, and confirm that accuracy is maintained in the presence of outliers, both in simulation and on our interactive application that spans a table and adjacent wall.

Our calibration method enables users to quickly set up multi-planar displays as they are needed, using any available projector and camera. These displays could be applied to visualization tasks in medical imaging, architecture and geographic information systems.

* Internal Accession Date Only Approved for External Publication 1 University of Cambridge Computer Lab, J.J. Thomson Avenue, Cambridge CB3 0FD; UK 2 HP Labs; One Cambridge Center, Cambridge, MA 02142 3 The Robotics Institute; Carnegie Mellon University, Pittsburgh, PA 15213 Copyright Hewlett-Packard Company 2003

Robust Calibration of Camera-Projector System for Multi-Planar Displays

Mark Ashdown1 Rahul Sukthankar2,3

mark@ashdown.name rahuls@cs.cmu.edu

1 University of Cambridge Computer Lab; J. J. Thomson Avenue; Cambridge CB3 0FD; U.K.2 HP Labs (CRL); One Cambridge Center; Cambridge, MA 02142; U.S.A.

3 The Robotics Institute; Carnegie Mellon University; Pittsburgh, PA 15213; U.S.A.http://www.cs.cmu.edu/˜rahuls/Research/Projector/

Abstract

We present a robust calibration method for aligning acamera-projector system to multiple planar surfaces. Un-like prior work, we do not recover the 3D scene geometry,nor do we assume knowledge of projector or camera posi-tion.

We recover the mapping between the projector and eachsurface in three stages. In the first stage, we recover pla-nar homographies between the projector and the camerathrough each surface using an uncalibrated variant of struc-tured light. In the second stage, we express the homogra-phies from the camera to each display surface as the com-position of a metric rectification and a similarity transform.Our metric rectification algorithm uses several images of arectangular object. In the third stage, we obtain the homo-graphies between the projector and each surface by combin-ing the results of the previous two stages. Inconsistenciesappear along the boundaries between adjacent surfaces; weeliminate them through a process of iterative refinement.

Standard techniques for recovering homographies fromline correspondences and performing metric rectificationare very sensitive to image processing outliers. We presentrobust algorithms for both tasks, and confirm that accuracyis maintained in the presence of outliers, both in simulationand on our interactive application that spans a table and ad-jacent wall.

Our calibration method enables users to quickly set upmulti-planar displays as they are needed, using any avail-able projector and camera. These displays could be appliedto visualization tasks in medical imaging, architecture andgeographic information systems.

1. IntroductionRecent advances in commodity high-resolution ultra-portable projectors have stimulated the development of avariety of novel projected displays such as large multi-projector walls [2], steerable projected displays [6], im-

projector

camera

surface 1

surface 2

Figure 1: A two-surface camera-projector system runninga topographic visualization. The camera and projector aremounted in unknown locations.

mersive environments [1, 8], intelligent presentation sys-tems [7, 9] and remote-collaboration tools [10]. The ef-fort involved in manually calibrating multiple projectors toeach other and aligning projected displays to physical sur-faces has motivated research in camera-projector systems,where techniques adopted from multi-view computer visionare applied to collections of projectors and cameras.

This paper describes a calibration method for creatingnovel projected displays where users can visualize data onmultiple planar surfaces (Figure 1). Unlike prior work, wedo not recover the 3D scene geometry, nor do we assumeknowledge of projector or camera position. This enablesusers to quickly set up multi-surface displays as they areneeded, using any available projector and camera.

The remainder of this paper is structured as follows. Sec-tion 2 discusses related work. Sections 3, 4, 5 and 6 describeand evaluate our calibration algorithms. Section 7 summa-rizes our contributions and proposes some promising direc-tions for future research.

2. Related Work

The calibration of multi-surface displays using a camera-projector system is explored in the Office of the Future [8],where a complete 3D model of the display surfaces is ex-tracted using structured light from a projector. That paperassumes that the intrinsic and extrinsic parameters of thecamera and projector are known. We do not need to recover3D scene geometry and the locations of our camera and pro-jector are unknown.

Automatic calibration of a camera-projector system ispresented in [9]. While that paper does not assume knowl-edge of projector and camera position, it assumes that thedisplay surface consists of a single rectangle of known as-pect ratio (whose boundaries are visible to the camera), andthat the mapping between projector and surface can be de-scribed by a single planar homography. Our method ap-plies to multiple surfaces, and we present more general al-gorithms for recovering the mappings.

The self-calibrating projector [7] automatically aligns itsimage to any (single) vertical planar surface using orienta-tion information obtained from a tilt sensor. Although theglobal position of the components is not known, the systemrequires that the projector and camera be rigidly attached toform a calibrated stereo pair.

The use of homographies to describe transforms inmulti-view geometry has become ubiquitous in computervision (Hartley and Zisserman [3] present a good introduc-tion to the subject). There has been much recent activityin the area of recovering 2D planar homographies, partic-ularly from noisy point correspondences [4]. Treating theprojector as a camera enables many results from the field tobe applied directly to our problem. One of our algorithmsfor metric rectification is derived from unstratified rectifica-tion [5].

The idea of projecting displays onto everyday objects isbecoming more popular [6, 8]. An extreme example of sucha system is the CAVE [1], where the user is immersed in-side a multi-surface projected display. A serious drawbackof many multi-surface displays is that they are designed fora single user, whose head is tracked in real-time (since theprojected scene and surface geometry do not match). Whileour calibration methods could be applied to those displays,we focus on displays where the 3D surface geometry agreeswith the projected visualizations. These displays can beused by multiple users since all users agree on the orien-tations of the projected objects.

3. Multi-Planar Calibration

The goal is to produce a projected display on several flatsurfaces, without requiring any knowledge of the 3D layout.A projector and a camera, mounted in unknown locations,

projector

camerasurface 1

surface 2

Figure 2: The 3D geometry of the display surfaces is un-known. The projector and camera are arbitrarily placed sothat their fields of view contain the surfaces. The goal is torecover a mapping from the projector to each surface.

are used to calibrate and create the display (Figure 2) withminimal input from the user.

We assume that camera and projector optics can be mod-eled as projectivities [3], therefore we require a homogra-phy i

PHS from the projector to a Euclidian frame on eachsurfacei. Since these cannot be observed directly, for eachsurface we decompose the mapping into two homographies:from projector to camera (through the surface), and fromcamera to surface. This generalizes an existing method forcalibrating a projector to a single surface [9].

Calibration consists of three stages. The first stage (Sec-tion 4) employs an uncalibrated variant of structured light toidentify the planar surfaces, and to recover mappingsi

PHC

between projector and camera through each surfacei. Theprojector displays a series of horizontal and vertical linesthat are observed by the camera. A line that crosses multi-ple surfaces appears as several line sections (Figure 3). Weexploit this to segment the camera image into regions corre-sponding to the planar surfaces. We fit a line to each section,intersect these lines and connect the intersections1 to deter-mine the precise boundary (in the camera image) betweeneach pair of adjacent surfaces. The same line sections arethen used to recover planar homographies from projector tocamera through the respective surfaces. For the line sec-tions to be distinguishable, the camera and projector mustbe suitably displaced from one another: if they were at thesame location, for instance, projected lines would alwaysappear straight, irrespective of the shape of the surface.

In the second stage (Section 5), we recover homogra-phies,iCHS from the camera to each surfacei. Structuredlight cannot be used for this stage since the relative positionsof the projector and camera are unknown. We decomposethe mapping into two parts: a metric-rectifying homogra-phy that maps the image of each surface into an arbitrary

1Simply connecting the raw intersections would give jagged bound-aries; we model the boundaries as straight lines, fitted with orthogonalregression.

2

surface 1

surface 2

Figure 3: Uncalibrated structured light is used to identifyboundaries between adjacent planar surfaces in the cameraimage.

Euclidean frame, and a similarity transform that aligns thisframe to the surface. We explore two methods for metricrectification: (1) observing a set of line pairs known to beperpendicular on each surface [3, 5]; (2) observing a rect-angle of known aspect ratio on each surface. To recover thesimilarity transform, we constrain the origin of each sur-face’s coordinate frame to a point on a boundary, we con-strain its x-axis to lie along that boundary, and select a scal-ing that is consistent across all surfaces. Thus, the size ofa projected object may be chosen arbitrarily, but should notchange as it is moved between surfaces.

The final stage (Section 6) combines the homographiescalculated in the two previous stages:

iPHS = i

CHSiPHC ,

for each surfacei. Since the homographiesiPHS for the dif-

ferent surfaces were calculated independently, there is noguarantee that they will be consistent along inter-surfaceboundaries. We apply an iterative refinement technique toensure continuity across boundaries.

4. Homography fromLine Correspondences

The standard method of calculating a homography betweentwo planes is to obtain corresponding homogeneous pointsxi andx′i. A homographyH that maps one set to the otherwill satisfy2

Hxi = x′i .

By seeking theH for which x′i × Hxi = 0, a closed-formleast-squares solution is reached. Since the three rows ofthis constraint are not linearly independent, the third row ofeach constraint can safely be discarded. Therefore,n ≥ 4

2In homogeneous coordinates, the 2D point(x, y) is represented by thevectorx = (xw, yw, w)>, ∀w 6= 0. Homogeneous vectors and homo-graphies are therefore invariant to non-zero scaling factors. Throughout thepaper this scaling is implicit, andx = y should be interpreted asx = ky,for somek 6= 0.

point correspondences can be combined into a single null-space problem of size2n×9 and solved using singular valuedecomposition (SVD). This method of recovering homogra-phies minimizes algebraic rather than geometric error, andis commonly known as the Direct Linear Transform (DLT).

Lines and points are duals in 2D projective geometry [3].A line described by the equationax + bx + c = 0 can berepresented by the 3-element vectorl = (a, b, c)>, wherewe can express the line in a canonical form with the con-straint‖l‖ = 1. The line,l′, resulting from a transformationthroughH is given byl′ = H−>l. A variant of the DLTalgorithm can be used to calculate homographies from linecorrespondences. This closed-form solution attempts to sat-isfy l′i×H−>li = 0 in a least-squares sense. However, sinceany element of the line vector can be zero, all three rows ofeach constraint should be included (resulting in a null-spaceproblem of size3n× 9).

We present a new algorithm for calculation of homogra-phies from line correspondences in Section 4.1, and evalu-ate it in Section 4.2.

4.1. AlgorithmFor each surfacei, we calculate a homographyiPHC fromthe projector (through surfacei) to the camera using linesfitted to pixels in the camera image, and the known posi-tions of those lines in the projector. The lines in the cameraare subject to error, and occasionally the image processingtechniques used to find them produce a totally incorrect re-sult. Least-squares minimization is very sensitive to out-liers, so we require an algorithm to calculate a homographyfrom line correspondences that is robust to such outliers.Homography calculation from noisy point correspondenceshas been addressed [3, 4], but here we present an algorithmusing line correspondences that is based on an algorithmfor calculating the fundamental matrix for epipolar geome-try [11].

Our algorithm takes a set ofn line correspondencesL = {(lj , l′j) : 1 ≤ j ≤ n}, and an upper boundε on thefraction of outliers (e.g., 10%), and returns a homography.It consists of two main steps. First, we employ DLT to com-pute homographies using several random minimal subsetsof line correspondences and record the subset that achievesthe lowest error for the remaining correspondences. Sec-ond, we classify each line correspondence as an inlier or anoutlier depending on its error based on the recorded minimalsubset. We then apply DLT to compute the final homogra-phy using only the inliers.

The sizes of each minimal subset is 4 since this manyline correspondences are required to compute a homogra-phy.3 If we require a probabilityP that at least one of the

3It is important to ensure that the subset is not degenerate (i.e., no threelines should be concurrent).

3

subsets does not contain any outliers, the number of subsetsm that must be used is given by

m =⌈

log(1− P )log[1− (1− ε)s]

⌉.

Typical values areP=0.9999 andε=0.1, giving the numberof samplesm=9. The error for each homography based onsline correspondences is the medianM of the errors betweenthe desired and actual locations for the remainingn − scorrespondences not used in the homography calculation:

M = medianL d(l′j , Hlj) ,

where the errord measures the closeness of one line to an-other, andH is the homography derived from the currentrandom minimal subset. We used(m1,m2) = ‖m1×m2‖

‖m1‖ ‖m2‖ .The median error for the recorded minimal subset is usedto calculate an estimate of the standard deviationσ̂ of theerrors [11]:

σ̂ = 1.4826(

1 +5

n− p

)M,

whereM is the value of the median error. The final ho-mographyi

PHC is calculated from the inliers using DLT.We choose the inliers to be those line correspondences forwhichd(l′i, Hli) < 2.5 σ̂, whereH is the homography calcu-lated from the recorded subset.

4.2. EvaluationTo test the algorithm above we synthesized homographiesby randomly positioning a virtual projector and cameraabove a virtual surface to simulate the hardware (Figure 4).The surface is defined to be a 1×1 unit square, and a ran-dom pointX on the surface is selected to be the “look point”for the camera. The cameraC is positioned at a fixed dis-tance fromX, with a random azimuth, elevation4 and rollangle. The imaged lines are expressed in(ρ, θ) form5 andperturbed with zero-mean Gaussian noise,N(0, σ2).

We explored several noise levels in the simulated exper-iments discussed below. Forρ, σρ ranged up to 1% of thescreen width, in increments of 0.2%. Forθ, σθ ranged up to0.01π in increments of 0.002π. Figure 5 shows the results(averaged over 10,000 runs) from two noise levels: level 1(σρ=0.2, σθ=0.002π) and level 4(σρ=0.8, σθ=0.008π).

The error in the homography calculated from the lines ismeasured by transforming ten random points and comput-ing the mean geometric distance between their expected andactual locations (expressed as a proportion of the cameraimage width). Gross outliers were added to the line corre-spondences in incremental proportions from 0 to 5%, by re-placing some lines with others drawn between two random

4To produce realistic images, elevation is greater than 20◦.5This is consistent with the Hough transform used in our physical setup.

20o

X

C(a)

X

C(b)

Figure 4: (a) In simulation the virtual camera is placed onon a hemisphere above a random point on the surface, withan elevation of at least 20 degrees. (b) The camera’s field ofview is created to include the whole surface.

0.001

0.01

0.1

1

10

0 0. 5 1 1. 5 2 2. 5 3 3.5 4 4. 5 5

proba bility of outlier %

po

intt

ran

sfe

re

rro

r

D LT 1

D LT 4

R obus t 1

R obus t 4

Figure 5: Comparison in the point transfer error betweenDLT and our robust line homography algorithm for two noiselevels. The error is plotted on a log scale. DLT degradeseven with a very small fraction of outliers. See text for de-tails.

points in the camera. The results show that while the closed-form solution becomes useless in the presence of outliers,our robust method successfully identifies the outliers andremoves them. Additional experiments (not detailed here)show that the error curve remains flat for fractions of out-liers up to 20%.

5. Metric RectificationAs discussed in Section 3, the mappingi

CHS, from camerato surfacei, can be decomposed into a metric-rectifying ho-mography and a similarity transform. The method for deter-mining the latter was discussed previously. We now presenttwo algorithms for calculating the former.

5.1. Algorithm 1Liebowitz and Zisserman [5] describe a method for metricrectification of the image of a planar surface given five ormore images of line pairs that are orthogonal on the sur-face. For instance, for our display system, the user could

4

simply scatter a few rectangular objects (like postcards) inthe scene.

The conic dual to the circular pointsC∗∞ on a Euclidiansurface is

C∗∞ =

1 0 00 1 00 0 0

.

For perpendicular linesl and m, l>C∗∞m = 0. Whenthese lines are imaged through a perspective transformH,the conic dual to the circular points becomesC∗

′

∞ = HC∗∞H>.The goal is to recoverH (up to a similarity transform),

since iCHS = H−1. To do this, we first findC∗

′

∞ for thegiven camera image. The images of the perpendicular linessatisfy l′>C∗

′

∞m′ = 0, so each such pair imposes a linearconstraint on the elements ofC∗

′

∞. The constraints can beformed into a null-space problem, and five or more suchconstraints enable a closed-form solution that gives us anestimate ofC∗

′

∞ [3]. This formulation is analogous to theDLT algorithm, and minimizes an algebraic rather than a ge-ometric error. The value being minimized,

∑(l′>C∗

′

∞m′)2,is related to the square of the cosine of the angle betweenthe rectified lines.

The next step is to factor our estimate ofC∗′

∞ into the formHC∗∞H>. We decompose our estimate using singular valuedecomposition into the formUDV>, whereD is a diagonalmatrix. If the input data are well-conditionedD33 should besmall compared toD11 andD22, so we assumeD33 = 0 andexpressD in the formD = BC∗∞B> where

B = B> =

±√D11 0 0

0 ±√D22 0

0 0 1

.

This gives us the desired decomposition for our estimate ofC∗

′

∞ as

U(BC∗∞B>)V> = (UB)C∗∞(UB)> = HC∗∞H> ,

whereH = UB. The choices for the signs of the diagonalelements inB might seem to create four possibilities forHbut only two need to be considered: one where the signs areequal and one where they are different. Both versions ofHwill correctly rectify angles and length ratios in the cameraimage, but one will reverse orientation in our region of in-terest. We determine which is the correct one by combiningeach with a similarity to get the full homography from cam-era to surface as described in section 3, then testing thosehomographies with a point that is known to be on a partic-ular side of the boundary line; only one of them will map itto the correct side.

This closed-form algorithm for metric rectification(termed CF) is attractive because it does not require any spe-cialized calibration target: any set of perpendicular lines onthe surface can be used. In practice, we obtain line pairs by

taking a few images of a rectangular object, such as a post-card, on each surface. However, CF shares DLT’s inherentsusceptibility to gross outliers in the line pairs, motivatingthe development of a robust variant.

As with the robust line homography algorithm (Sec-tion 4.1), our algorithm for robust metric rectification clas-sifies each input line pair as an inlier or outlier, then com-putes final rectifying homographies from only the inliers. Ittakes a set ofn line pairsP = {(lj ,mj) : 1 ≤ j ≤ n}in the camera frame that are known to be orthogonal on thesurface, and an upper boundε on the fraction of outliers. Itreturns two homographies, one of which is the correct one.The correct homography is identified as described above.The proportionε and the minimum number of line pairssrequired to compute a rectifying homography (s=5 in thiscase) yield the number of random minimal subsets requiredas for our line homography algorithm. ForP=0.9999 andε=0.1, we requirem=11 random minimal subsets. The ro-bust algorithm proceeds by employing CF on these subsetsof line pairs and records the subset that achieves the low-est error for the remainingn − s line pairs once they havebeen transformed using the computed rectifying homogra-phy. The error metric is

medianP d(Hlj , Hmj) ,

whered gives the cosine of the angle between two lines,andH is the rectifying homography derived from the currentrandom minimal subset.

Using the homography from the recorded minimal sub-set, we classify each line pair inP as an inlier or outlier.As cosines, the error values for the pairs are bounded by therange [0,1], so we simply deem thednεe pairs with the high-est error to be outliers. The final rectifying homography isthen computed using CF with only the inliers.

5.2. EvaluationTo test Algorithm 1 we generated homographies by ran-domly positioning a virtual camera above a virtual surfaceas for the line homography. Random pairs of perpendicularlines on the surface were transformed with the homogra-phy, then the resulting lines in(ρ, θ) were perturbed withzero-mean Gaussian noise,N(0, σ2). As in Section 4.2,we considered a number of noise levels. Forρ, σρ rangedup to 0.1% of the screen width, in increments of 0.02%.For θ, σθ ranged up to 0.001π in increments of 0.0002π.Figure 6 shows results (averaged over 1000 runs) from twonoise levels: level 1(σρ=0.02, σθ=0.0002π) and level 4(σρ=0.08, σθ=0.0008π).

Gross outliers were added to the line pairs in various pro-portions. To evaluate the algorithm, we generated a set ofrandom orthogonal line pairs on the surface, and mappedthem through the known imaging homography. These im-aged line pairs were then transformed using the rectifying

5

0

5

10

15

20

25

30

0 0. 5 1 1. 5 2 2. 5 3 3. 5 4 4. 5 5

proba bility of outlier %

rec

tific

atio

ne

rro

r(d

eg

ree

s)

C F 1

C F 4

R obus t 1

R obus t 4

Figure 6: Comparison in the rectification angle error be-tween CF and our robust metric rectification algorithm fortwo noise levels. CF degrades rapidly as outliers are added.See text for details.

homography. The rectification error is the average deviationof rectified angles from the desired90◦. Our results showthat when no outliers are present our robust algorithm hasan accuracy similar to that of CF, but as outliers are addedCF’s error quickly becomes unacceptable, while that of therobust method remains level. The robust method effectivelyidentifies and removes the outliers. Additional experiments(not detailed here) show that the error curve remains flat forfractions of outliers up to 25%.

5.3. Algorithm 2An alternate approach to metric rectification relies on imag-ing a calibration object, for example a rectangle with aknown aspect ratio,a. Such a rectangle can be definedby the four points:{(0, 0), (a, 0), (a, 1), (0, 1)}. The cor-responding points in the camera frame are given by the im-age coordinates of the corners. Using DLT, we can recovera rectifying mapping, which can be composed with a sim-ilarity transform to obtainiCHS. In practice, we can use anobject with more feature points (e.g., a checkerboard) andreplace the simple DLT with its more robust RANSAC vari-ant [3]. The primary disadvantage of this algorithm is that itrequires the user to supply an artificial target with a precisecalibration pattern.

5.4. DiscussionWe evaluated both Algorithm 1 and 2 in simulation and onour camera-projector display system. We found that thesimpler algorithm performed better (at the expense of userinconvenience). Our robust Algorithm 1 successfully re-moves all of the outliers, but the inherent sensitivity in themethod used to obtainC∗

′

∞ necessitates very accurate imageprocessing estimates of observed lines. The standard devi-ation of the line estimates in our current camera-projector

(a) (b)

Figure 7: (a) Mappings can be inconsistent along inter-surface boundaries. (b) Our iterative refinement algorithm(Section 6.1) addresses this problem.

setup is too high for Algorithm 1. We are exploring exten-sions to Algorithm 1 (such as better normalization) to makethe technique more practical; our display system currentlyuses Algorithm 2.

6. Constraints Between HomographiesAs discussed in Section 3, the homographiesi

PHS from pro-jector to each surfacei are calculated independently, andthus there is no guarantee that they will be consistent alongboundaries between surfaces (Figure 7). We now present analgorithm for iteratively refining these homographies.

6.1. AlgorithmTo simplify the following discussion, let us consider a two-surface setup. Let1SHP = 1

PH−1S and 2

SHP = 2PH

−1S be

the homographies mapping each of the two surfaces to theprojector. Ideally those two homographies will transformpoints on the boundary line identically; ifx is a point onthe boundary, we would like1SHPx = 2

SHPx. This will typ-ically not be the case due to image processing error, so weapply this constraint by refining the homographies with aniterative algorithm (Figure 8).

We start by generating five points on each surface (Fig-ure 8a), and generate initial correspondences in the projec-tor using the current values of1SHP and 2

SHP (Figure 8b).The locations of the points on the two surfaces are fixedthroughout the algorithm, but their corresponding points inthe projector move as1SHP and2

SHP are adjusted. At leastthree common points are needed on the shared boundary;this is to ensure that length ratios for the two homographiesagree along the boundary.

The goal of the iterative algorithm is to refine1SHP and2SHP until each common point on the boundaryx maps tothe same point in the projector (i.e.,1

SHPx = 2SHPx) re-

gardless of which homography is used. Figure 8c illustratesthe iterative process. Each common point on the bound-ary transforms to two distinct points in the projector, andwe create a new point in the projector by taking the mid-point of these. We set these new points as the correspon-

6

(a)

(d)(c)

(b)surface 1

surface 2 commonpoints

surface 1

surface 2

surface 1

surface 2

surface 1

surface 2

boundaryline

Figure 8: (a) Five points are defined on each surface. (b)Corresponding points are created in the projector. (c) Eachboundary point pair is replaced by its midpoint. (d) The erroris the sum of distances between transformed points.

dences for the boundary points on both surfaces. These newpoints, along with the (unchanged) correspondences of thenon-boundary points are used to recompute1

SHP and 2SHP

for the next iteration, using DLT. We terminate the refine-ment process when the improvement in the error falls belowa threshold. The error (Figure 8d) is measured as the sumof the separations of the boundary points plus the sum ofdistances between desired and transformed locations of thenon-boundary points (all measured in the projector frame).

6.2. EvaluationFigure 9 shows a trace of error from a run of our iterative re-finement algorithm. The combined error drops rapidly withthe number of iterations; within 20 iterations, the inconsis-tency between the homographies along the shared boundaryis imperceptible. If the initial estimates for the homogra-phies are sufficiently inaccurate, this refinement techniquecan converge to a bad solution. However, in practice, wefind that our robust algorithms generate sufficiently goodestimates foriPHS that refinement improves the final cali-bration of our display system.

7. ConclusionWe have presented robust calibration algorithms for align-ing a camera-projector system to multiple planar surfaces.Our algorithm for calculating homographies from line cor-respondences works even when a significant fraction of thedata consists of image processing outliers. This enables usto deploy the uncalibrated structured light system in condi-tions where the DLT algorithm fails catastrophically (Fig-

0

5

10

15

20

0

ite ra tion

err

or(p

ixel

s)

20 40

Figure 9: Using our homography refinement algorithm, thesum of errors between transformed points decreases rapidlywith successive iterations. This ensures that homographieson adjacent surfaces become consistent along their com-mon boundary.

ure 10a). For metric rectification, Algorithm 1 successfullyremoves outliers, but is limited by the inherent instabilityof the conic-dual method (Figure 10b). We are pursuingmethods for improving its performance. We currently em-ploy the simpler, but less natural, Algorithm 2 (Figure 10c).Our iterative homography refinement technique convergesin only a few iterations and significantly reduces inconsis-tencies along boundaries of multi-surface displays. Noneof the techniques presented in this paper require significantcomputation, and all can be deployed in practical camera-projector systems (requiring under a second of processingtime). Our aim is to calibrate multi-surface display systemswithout the use of specialized targets, simply by imagingeveryday objects (such as sheets of paper) in the scene.

We have implemented an interactive application for amulti-surface display that enables users to manipulate im-ages on each surface, and move them between surfaces (Fig-ure 10c). Multi-surface displays could also be used to si-multaneously present different visualizations of a dataset.For example, a geologist could manipulate an overhead mapon a horizontal surface while examining a vertical cross-section through the dataset on the other surface (Figure 1).Similarly, an architect or CAD tool user could simultane-ously manipulate plan and elevation views on different sur-faces. A three-surface display projected into a corner couldbe an affordable alternative to expensive immersive volu-metric visualization displays, enabling users to examine 3Dstructure by clipping or projecting the data on to each sur-face. All of these displays can be created using a singlecommodity projector and camera, on any available surfacesin the user’s environment.

7

(a) (b) (c)

Figure 10: Multi-surface display aligned using three calibration techniques: (a) The standard calibration (iPHC using DLT,iCHS using CF) is unusable due to image processing outliers – the projected images are not even close to being rectangular;(b) The robust algorithm (iPHC using robust algorithm, i

CHS using Algorithm 1) removes outliers, but the metric rectificationfrom Algorithm 1 is insufficiently accurate; (c) Our current implementation (iPHC using robust algorithm, i

CHS using Algorithm 2)correctly calibrates the display to both surfaces – the images are now rectangular and aligned to the surfaces.

Acknowledgments

This project was made possible by the HP Labs summerinternship program through which Mark Ashdown was ableto work at CRL. We would like to thank Keith Packard andMatthew Mullin for their help and comments.

References[1] C. Cruz-Neira, D. Sandlin, and T. DeFanti. Surround-screen

projection-based virtual reality: The design and implemen-tation of the CAVE. InProceedings of SIGGRAPH, 1993.

[2] T. Funkhouser and K. Li. Large format displays.ComputerGraphics and Applications, 20(4), 2000. (guest editor intro-duction to special issue).

[3] R. Hartley and A. Zisserman.Multiple View Geometry inComputer Vision. Cambridge University Press, 2000.

[4] K. Kanatani, N. Ohta, and Y. Kanazawa. Optimal homogra-phy computation with a reliability measure.IEICE Transac-tions on Information and Systems, E83-D(7), 2000.

[5] D. Liebowitz and A. Zisserman. Metric rectification for per-spective images of planes. InProceedings of Computer Vi-sion and Pattern Recognition, 1998.

[6] C. Pinhanez. The Everywhere display. InProceedings ofUbiquitous Computing, 2001.

[7] R. Raskar and P. Beardsley. A self-correcting projector. InProceedings of Computer Vision and Pattern Recognition,2001.

[8] R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, andH. Fuchs. The office of the future: A unified approach toimage-based modeling and spatially immersive displays. InProceedings of SIGGRAPH, 1998.

[9] R. Sukthankar, R. Stockton, and M. Mullin. Smarter pre-sentations: Exploiting homography in camera-projector sys-tems. InProceedings of International Conference on Com-puter Vision, 2001.

[10] N. Takao, J. Shi, and S. Baker. Tele-graffiti. Techni-cal Report CMU-RI-TR-02-10, Carnegie Mellon University,March 2002.

[11] G. Xu and Z. Zhang. Epipolar Geometry in Stereo, Mo-tion and Object Recognition. Kluwer Academic Publishers,1996.

8