The Visual Computer (1998) 14:153±165� Springer-Verlag 1998 153

Resolving occlusionin image sequencemade easy

Kiem Ching Ong, Hung Chuan Teh,Tiow Seng Tan

Department of Information,Systems and Computer Science,National University of Singapore,Lower Kent Ridge Road, Singapore 119260E-mail: ong-kc@hotmail.com,{tehhc|tants}@comp.nus.edu.sg

While the task of seamlessly mergingcomputer-generated 3D objects into an im-age sequence can be done manually, sucheffort often lacks consistency across theimages. It is also time consuming andprone to error. This paper proposes aframework that solves the occlusion prob-lem without assuming a priori computermodels from the input scene. It includes anew algorithm to derive approximate 3Dmodels about the real scene based on re-covered geometry information and user-supplied segmentation results. The frame-work has been implemented, and it worksfor amateur home videos. The result is aneasy-to-use system for applications likethe visualization of new architectures in areal environment.

Key words: Augmented reality ± Occlu-sion ± Structure-from-motion ± Segmenta-tion

1 Introduction

The deployment of computer graphics to insert vir-tual objects into real scene has many applicationsin medicine, manufacturing, visualization and en-tertainment (Azuma 1997). To enhance the illusionthat the computer objects are actually present inthe real scene, three problems need to be solved:

1. Common viewing parameters must be foundwhen the virtual camera used to render the in-serted computer objects is to correspond asclosely as possible to the actual camera.

2. The problem of occlusion must be solved whenthe computer objects are to be properly occlud-ed by objects in the real scene in the event thatthey are placed behind them.

3. Illumination must be attended to when the com-puter objects are to be subject to the same lightingconditions as the real objects and must have appro-priate shadows cast to and from the real objects.

This paper focuses on achieving correct occlusionand does not deal with the illumination issue. Theproblem of occlusion occurs when the insertedcomputer objects are placed behind real objectsin the scene. Usually, the region occupied by thereal objects is masked so that no computer objectscan be drawn within. This mask essentially cap-tures the silhouettes of the blocking real objects,and the mask edges dictate how seamless themerging will be. The masking effect can beachieved in two dimensions by designating a 2Dpixel region in the real image manually (Nakamaeet al. 1986; Ertl et al. 1991) or semiautomatically(Berger 1997). This method is simple but tediousbecause each frame requires its own mask. Fur-thermore, flexibility suffers as the 2D mask willin effect occlude all inserted computer objects re-gardless of whether they are in front or behindthe real objects. The masking effect can also beachieved in three dimensions with known comput-er models or simply depth values. When a comput-er model of the blocking real object is registered inthe image, i.e. its 2D projection matches the realobject in the image, correct occlusion is achievedif the inserted computer object is placed behindthe real object (Breen et al. 1996). This approachis general, but unfortunately, computer models ofreal scene are usually unavailable, and it is oftentime consuming to create one by hand. Due to this,some researchers use depth information derivedfrom the real scene for occlusion resolution (Koch1993; Wloka and Anderson 1995). Typically, ste-

Correspondence to: H.C. Teh or T.S. Tan

154

reoscopic image pairs are used for feature corre-spondence to recover the depth value at each pixel.Since this approach has no notion of the geometri-cal aspect of the real objects, silhouettes of the realobjects are not explicitly delineated. Due to errorsin feature correspondence, the recovered depth val-ues often cause an unsatisfactory merging effect,especially at the occluding boundary of the real ob-jects.In view of the flexibility of the 3D mask and theimportance of accurate masking at the edges ofreal objects, we propose the following framework.Some real object silhouettes are first segmented byhand in selected frames called key frames. The re-sult is then used to construct automatically approx-imate 3D computer models, which serve as 3Dmasks to the real objects. The 3D models are ap-proximate because their geometry may not corre-spond exactly to that of the physical real objectsthey represent. However, they are constructed insuch a way that, when projected onto the images,they cover exactly the silhouettes of their respec-tive real objects. In general, we offer a solutionto the following problem: given a monocular im-age sequence with neither a priori knowledgeabout the scene nor the availability of any realscene models, resolve the occlusion problem thatoccurs when computer objects are inserted intothe scene. The framework is not intended forreal-time applications; rather, an image sequenceof the real scene is captured, digitized and thenprocessed. Nonetheless, the whole framework isdesigned to be fully automated with little interven-tion from users, except for object-silhouette extrac-tion, when the user outperforms the computer.Based on the framework, an architecture visualiza-tion application has been implemented. Such a sys-tem assists an architect to evaluate the impact ofhis or her design by merging a computer modelof the architecture in a video of the location whereit is to be built. The augmented video is most use-ful in multimedia presentations since it conveysthe architectural design more realistically. Further-more, the framework requires only a monocularimage sequence that can easily be obtained witha video camera. Virtual objects are merged intoreal images interactively through a GUI.In Sect. 2, we outline the proposed framework.Sections 3±7 zoom into the respective moduleswithin the framework and examine the implemen-tation issues. Results from our implemented sys-

tem are shown in Sect. 8, and we conclude this pa-per in Sect. 9.

2 Proposed framework

Given an input image sequence, the proposedframework (Fig. 1) derives sparse 3D featurepoints from the scene using a structure-from-mo-tion algorithm from computer vision. We use theuser-segmented object silhouettes in the keyframes to cluster the 3D points of the real objectsto which they belong. We then use each of theclustered point sets to define and subsequently en-large 3D bounding boxes of the real objects so thattheir 2D projections onto the key frames will com-pletely cover their respective object silhouettes.These bounding boxes form the initial models,which we call clones to the real objects.Since our purpose in building the 3D clones is todeal with occlusion, we want to construct themso that they project exactly to their respective ob-ject regions in the real images. Therefore, for eachof the initial clones, their 2D object silhouettes inthe key frames are back-projected to the 3D spaceto trim away unwanted 3D regions in the clone.This, however, results in overtrimming due to er-rors in the estimated virtual camera parameters.A set of 3D patch voxels that can dynamically beturned on or off is used to compensate for regionson clones that are to be retained in certain keyframes, but have been trimmed in others. Geomet-rically, the resultant clone is a solid 3D object witha set of dynamic 3D patch voxels that are turned onor off, depending on which key frame is active.With the clones, we can then establish a commonreference system between the real objects in theimages and the inserted virtual objects. The subse-quent merging process is simply depth comparisonwith which proper occlusion is realized.This framework does not set out to recover detail3D models of real objects. We exploit the fact thatthe clone is intended for occlusion and not for vi-sual rendering. As a result, in terms of geometry,only approximate 3D clones are constructed. Thereare cases when exact geometry of the clone is de-sirable. This happens when there is physical con-tact between real and virtual objects; for example,in making a virtual vase sit on top of a real table.Our framework is more concerned with exhibitingcorrect visual occlusion for assessment purposes.

155

Also, since our approach constructs clones fromthe silhouettes, the registration problem associatedwith the model-based method is nonexistent.

3 Recovering structure fromthe image stream

A recent trend in solving structure-from-motionproblem has focused on using a long image stream,which has a twofold advantage. Firstly, the featurecorrespondence problem between successive framescan more easily be solved due to a small change inthe image content. Secondly, data redundancy canbe exploited to counteract noise in images.The structure-from-motion problem with perspec-tive projection is inherently a nonlinear minimiza-tion problem. This class of problems is susceptibleto local minima in the parameter space, and a goodformulation is essential to better manage its com-plexity during minimization. Hu and Ahuja(1993) and Szeliski and Kang (1994) have used

nonlinear optimization techniques to extract both3D structure and camera motion from imagestreams. However, there are researchers who relaxthe requirement of perspective projection to its ap-proximations like orthography, scaled orthographyand paraperspective in order to convert the nonlin-ear problem into a linear one. In particular, the fac-torization method by Tomasi and Kanade (1992),which assumes orthographic projection, has showngood results. In fact, Yu et al. (1996) have man-aged to use the Taylor series expansion to general-ize their factorization method to higher-order ap-proximations to perspective projection. Apart fromusing point features to recover scene structure as inthese papers, Taylor and Kriegman (1995) andVieville and Faugeras (1990) have used straight-line features to solve the structure-from-motionproblem. Quan and Kanade (1997) have recentlyextended the original factorization method to linefeatures.In our implementation, we employ the factoriza-tion method under orthography by Tomasi andKanade. This method encodes an image stream

Fig. 1. Framework for resolving occlusion in image sequence

156

as a 2F�P measurement matrix of the image coor-dinates (u, v) of P points tracked through F frames.Under orthography, the registered 2F�P matrix(W) has rank 3 and can be decomposed into cameraorientation (R) and the object shape (S).

W2F�P

� R2F�3

S3�P

Their method is robust due to the use of singularvalue decomposition, which is a numerically stablefactorization method. Furthermore, shape informa-tion is derived without the use of intermediatecamera-centered depth values, which are knownto be noise-sensitive for distant objects.Even though the orthographic projection assump-tion is tolerable in our intended application (archi-tecture visualization where the camera is far awayfrom the real scene), it does limit the kind of cam-era motion allowed. For example, the camerashould not move towards or away from the scenein order to minimize perspective distortion. Also,when the camera is purely translated across thescene with a constant line-of-sight vector, the fac-torization method will fail due to violation of thebasic principle of Ullman's (1979) results, whichsay that at least three distinct views are requiredto determine structure and motion under orthogra-phy. This is because the images generated by thiskind of camera motion effectively only give a 2Dview of the real objects.The output from Tomasi and Kanade's factoriza-tion algorithm is the matrices R and S. R containsthe relative orientation of the camera in eachframe, whereas S contains the recovered 3D fea-ture point coordinates. The origin of the coordinatesystem is at the centroid of the recovered 3D pointsand its orientation is arbitrarily aligned to the firstframe's camera orientation.

4 Camera parameter derivation

The recovered camera orientation alone is not suf-ficient to specify a virtual camera completely. Wealso require the camera position and zoom factor ineach frame. In our implementation, the camerazoom factor has been assumed to be constantthroughout the image sequence. As a result, thereare only 3F+1 unknown camera parameters to re-cover (F is the number of frames). To this end,

we make use of the recovered 3D feature points,together with their corresponding tracked 2D fea-ture points, to fit for the 3F+1 unknowns usingLevenberg-Marquardt minimization (Press et al.1992). The error function to minimize is

c2 a� � �XF

f�1

XP

p�1

ufp

vfp

� �tracked

ÿ ufp

vfp

� �computed

where:

F: number of framesP: number of feature pointsa: vector of 3F+1 camera parameters

ufp

vfp

� �tracked

: tracked 2D feature point p in frame f

ufp

vfp

� �computed

: computed 2D projection of 3D fea-ture point p in frame f given the es-timate of a.

Experimentally, we find that when orthography isstill used as the underlying projection model, thesolution converges faster (less than ten iterations)and is more accurate. When perspective projectionis used, the fitting process sometimes fails becausethe solution does not converge.

5 Image segmentationand key frame selection

To date, a robust and automatic image segmenta-tion algorithm is still elusive. Snakes (Kass et al.1988), Intelligent Scissors (Mortensen and Barrett1995) and the ubiquitous magic wand availablein most image processing packages, though speedup the segmentation task, still require human guid-ance to achieve correct segmentation. Boundarytracking across multiple frames for image segmen-tation has been explored by Ueda et al. (1992) us-ing Snakes and by Mitsunaga et al. (1995) using al-pha values. Nevertheless, these methods are notperfect at all times, and user correction is neededwhen the segmentation is unsatisfactory.The proposed framework relies on correct object-silhouette segmentation in the key frames to guidethe construction of 3D clones of real objects. Thedefinition of the key frames depends largely onthe geometrical changes to the real objects in the

157

images. A frame should be made a key frame whenthere is large geometrical change, such as the ap-pearance or disappearance of facets on real objects.Generally, the more key frames the user defines,the more accurate the constructed clones will be.In the implementation, we use Intelligent Scissorsas a tool to help users define the object silhouette.We find that using Intelligent Scissors is intuitive,and its interactive snapping live-wire feature ishighly desirable in practice. However, the live wiresometimes snaps to a nearby wrong edge that ex-hibits a stronger edge characteristic. To accommo-date this, a simple point-and-click straight-line tool(which does not automatically snap to edges) hasbeen incorporated into Intelligent Scissors, andwe alternate between the two at our convenience.We have, by default, made the first and the lastframes key frames, and users have to decide whichof the other intermediate ones are to be key frames.

6 Clone construction

The most important function that a 3D clone of areal object serves is to occlude the inserted virtualobjects. When a virtual object occludes a real ob-ject in the image, simply overlaying the virtual ob-ject on top of the image will do the trick. This isnot so straightforward when a real object in the im-age is to occlude the inserted virtual object. Weneed a reference 3D clone of the real object in or-der to determine which part of the virtual object isto be occluded. Since the virtual objects should on-ly be occluded at the boundary of real objects, the3D clone should be constructed in such a way that

its 2D projection onto the image plane covers ex-actly the real object's silhouette in every frame.Due to the use of orthographic projection in thecamera fitting process, we also assume orthogra-phy as the underlying projection model in cloneconstruction. The whole process is accomplishedin five steps.

Step 1. Initial model definition

Based on the user-supplied object silhouettes(Fig. 2b) in the key frames, we cluster the recov-ered 3D feature points from the factorization meth-od (Sect. 3) to their respective real objects. Foreach such object, we then define a rough model us-ing the bounding box of the 3D feature points. Fig-ure 2c shows the world-coordinate-axis-alignedbounding box of the 3D feature points derivedfrom the box.

Step 2. Model orientation determination

Due to the fact that our centroid-based coordinatesystem has an arbitrary orientation, the groundtruth information is unknown. This means thatwe cannot assume that the ground is on the x�zplane, and thus our initial rough model may nothave the correct orientation. Having a correct ori-entation, though not critical, speeds up subsequentprocessing. As a result, we provide an interactivemechanism for users to change the orientation ofthe initial model. Figure 2d shows that the userhas interactively adjusted the orientation of the

a b c d

Fig. 2. Constructing a 3D clone for thewhite box (panel a)

a b c d

e f g h

158

bounding box so that it is closer to that of the whitebox.

Step 3. Model enlargement

Since a good clone is one that projects to its 2D ob-ject silhouette exactly, we need to ascertain thatthe initial model at least covers its object silhou-ettes in all the key frames, albeit overcover. Wesystematically enlarge the model with the follow-ing algorithm. Figure 2e shows that the enlargedmodel projects to completely cover the user-de-fined object silhouette.The algorithm works as follows. Assume that theinitial model is bounded by bottom-left-far point(x1, y1, z1) and top-right-near point (x2, y2, z2)where x1<x2, y1<y2, and z1<z2, the algorithm goesthrough all the key frames in turn to progressivelyenlarge the model.For each key frame, the algorithm computes the2D (axis parallel) bounding box of the object sil-houette and locates two points (possibly one pointin degenerate cases), each on the object silhouette,that are farthest apart on each boundary edge of thebounding box. For each of these eight (or less)points, the algorithm checks whether any enlarge-ment is necessary, as in the next paragraph.For each of the eight (or less) points, the algorithmcasts a ray from the point on the image plane to the3D space, using the computed camera parameters.If the ray intersects the current model, then no en-largement is necessary. Otherwise, the ray is out-side the current model, indicating that enlargementis necessary because the model does not project tocover this point. In this case, the algorithm findsthe corner, say C, of the current model closest tothe ray. Let 3D point (x, y, z) be the point on theray closest to C. Then, the algorithm enlarges thecurrent model by setting: x1=min(x1, x); y1=min(y1, y); z1=min(z1, z); x2=max(x2, x); y2=max(y2, y)and z2=max(z2, z).

Step 4. Model trimming

This step is crucial because it trims away the 3Dregion on the model that does not project to the2D object silhouette. To facilitate the trimmingprocess, we use a voxel representation for the mod-el. Each model is subdivided into voxels of similar

size and all these voxels are initially assigned ºINºto indicate that they belong to the model. The di-mension of the voxel is computed so that each vox-el can only potentially project to cover a singlepixel. We subsequently employ a ray-casting algo-rithm that casts rays from the image plane to the3D space to trim the model. When a ray originat-ing from a pixel outside the object silhouette inter-sects with the model, we assign ºOUTº to voxelsthat fall on the path of the ray. The whole processcan be implemented efficiently by casting raysalong the boundary of the object silhouette and dis-carding the 3D region that does not project to thesilhouette.Another way to look at the whole trimming pro-cess is by solid object intersection. First we definea valid 3D region (model defined after Step 3)where intersection is allowed. The resultant modelis essentially the intersection of the object-frustumback-projected from the object silhouettes in thekey frames to the 3D space. Figure 2f shows theconstructed model, which is represented by voxels.Notice that the model has missed out boundaries atthe top and bottom of the box.

Step 5. Derivation of dynamic patch voxel

Up to this point, we have had 3D clones that wehave trimmed using their respective object silhou-ettes in the key frames. Due to inaccuracy in thecomputed camera parameters (camera position,orientation and zoom factor), the resultant clonemay be overtrimmed at certain places, especiallynear the boundary. This implies that the clone willno longer project to cover exactly the object sil-houette in some key frames, and visual error willoccur at the occluding boundary between realand virtual objects during merging. To remedythis, we make use of a set of dynamic 3D patchvoxels that have the same voxel dimension as theclone. For each key frame, we locate the missingregion on the clone that will make the clone pro-ject perfectly to the object silhouette. We later en-code this missing region using patch voxels.The challenge of this problem is, given that theclone does not project exactly onto the silhouette,how to reinstate the missing 3D region on theclone. We know that the missing region is stillwithin the bounding box of the clone and mustbe geometrically close to the overtrimmed model,

159

so we sort of extrapolate the model with the fol-lowing algorithm.The patch voxels needed for each key frame arecomputed with two passes through all pixels withinthe object silhouette.Let us discuss the first pass. For each pixel, the al-gorithm casts a ray from the pixel on image planeto the 3D space (using camera parameters comput-ed earlier). If the ray does not intersect the clone,then overtrimming of the clone has occurred forthis pixel and patching of voxels is needed for thispixel in the second pass. Otherwise, the algorithmcomputes d1 and d2, the nearest and farthest z-co-ordinates, respectively, of the clone intersectedby the ray.In the second pass, we create patch voxels for pix-els identified by the first pass. For each such pixel,the algorithm estimates d1 (and d2) of the pixel asthe average of the d1 values (and d2 values, respec-tively) of the surrounding eight (or less) pixels.Note that the estimation has to be done in some or-der of pixels closest to pixels with known d1 and d2values and so on. Next, the algorithm casts a rayfrom the pixel so as to encode the ray segment be-tween d1 and d2 as patch voxels.Note that the last statement on casting a ray is re-placed in practice by casting more than one ray perpixel. This is to ensure that any potential patchvoxel that may belong to the missing region isnot overlooked. Additionally, instead of castingrays to the 3D space as in the first pass of the al-gorithm, we can also project the voxels of theclone to the image and then perform the secondpass for pixels that fall outside the projected re-gion, but within the user-defined object silhouette.Now, each key frame has a set of 3D patch voxelsthat will be employed in the following manner.During merging of virtual objects into real images,when the frame is a key frame, we use the over-trimmed model, together with the key frame'spatch voxels, as a clone to the real object. This en-sures that the object silhouette is exactly coveredby the clone. For intermediate frames that are notkey frames, we define the clone to be the over-trimmed model together with the patch voxels ofthe nearest key frame. Thus, when we step throughthe image sequence, the clone, which is the over-trimmed model, is dynamically patched up withpatch voxels belonging to different key frames.Figure 2g, h shows two views of the clone withpatch voxels drawn.

The clone construction process is somewhat simi-lar to Szeliski's (1993) work, which uses the octreeto construct 3D object models from a set of imag-es. However, we wish to point out that his workdoes not cater to the overtrimming case, which of-ten occurs due to errors in the estimated cameraparameters.

7 Merging with virtual objects

To achieve occlusion between real and virtual ob-jects, we make use of the hardware Z-buffer in thegraphics workstation. The Z-buffer is used primar-ily for visible surface determination. It keeps trackof the nearest z-coordinate (in camera coordinate)at each pixel. A pixel colour is only updated ifthe incoming z value is smaller than that storedin the Z-buffer, i.e. is closer to the camera.To merge the virtual objects with the real images,the following rendering pipeline is used. Note thatno specific texture-mapping algorithm is required.

Step 1. Display the image in the frame buffer.Step 2. Render clones (both overtrimmed model

and patch voxels) in the Z-buffer only.Step 3. Render virtual objects in both the frame

buffer and the Z-buffer. A virtual objectappears occluded in the merged image ifit is behind real objects with clones ren-dered in Step 2.

In order to ensure that the clones still conform tothe way they are built, we use orthographic projec-tion to render the clones to the Z buffer. The insert-ed virtual objects are rendered with perspectiveprojection model.

8 Results

Figure 3a shows a sample of eight frames from a99-frame video sequence. Figure 3b shows theaugmented video after we add in a virtual domeand two trees. The dome is made up of polygonsand is rendered with a single, fixed light source.The trees are texture mapped from images ontosome 3D planes. To illustrate the correct occlusionbetween the inserted virtual objects and the realobject in the images, we position the dome andone of the trees behind the front building and letthe other tree block the front building. Notice that

160

frame 1

frame 42

frame 84

frame 56

frame 99

frame 70

frame 14 frame 28

a

frame 1

frame 42

frame 84

frame 56

frame 99

frame 70

frame 14 frame 28

b

Fig. 3a, b. a Original image sequence; b augmented image sequence;

161

Fig. 3c±f. c user-defined object silhouettes; d recovered 3D feature points; e two views of the constructed clone (overtrimmed modeland patch voxels); f two views of the objects represented in the computer: 3D feature points, clones, and virtual objects

c

d

e

f

162

occlusion is properly shown at the occludingboundary in all the frames. Also, the estimatedcamera parameters used to render the computer ob-jects are accurate because the virtual objects movetogether with the images.Altogether, six segmented silhouettes of the frontbuilding (in frames 1, 21, 41, 61, 81 and 99) havebeen provided, and two of such segmentation re-sults are shown in Fig. 3c. The input video se-quence is fed into Tomasi and Kanade's factoriza-tion algorithm, and we manage to recover 204 fea-ture points from the scene (Fig. 3d). Based on theobject silhouette, the program then clusters the fea-ture points belonging to the real building and con-structs an initial model for the building using the3D bounding box of these points. This is followedby model trimming, which eventually yields the

clone (overtrimmed model with patch voxels foreach key frame) as shown in Fig. 3e. For better un-derstanding, we also provide two views of all thecomputer objects used in the merging process(Fig. 3f).Figures 4 and 5 show some augmented framesfrom another two image sequences. In the first se-quence, segmentation results from eight keyframes (1, 10, 20, 30, 40, 50, 60 and 70) are usedto construct the clone for the real temple before weinsert the virtual dome behind the structure. In thesecond sequence, five key frames (1, 10, 20, 30and 43) are used to construct clone for the exot-ic-looking mushroom-shaped real structure. Twosimilar looking 3D virtual structures and a 3D vir-tual statue are then placed behind and in front ofthe real structure, respectively.

frame 1 frame 25

frame 45 frame 70

Fig. 4. Temple sequence

163

We have run the program for these examples on anSGI High Impact machine. A GUI has been pro-vided to enable users to interactively manipulatevirtual object positions and orientations. A PC ver-sion is also currently under development.

9 Conclusion

Exhibiting correct occlusion between virtual andreal objects remains one of the most desirable fea-tures, when we insert virtual objects into a real en-vironment. The proposed framework integrates a

new clone construction algorithm with other best-of-tools from computer vision and image processingto solve the occlusion problem. A structure-from-motion algorithm from computer vision is em-ployed to derive sparse 3D points from the scene.Using these points together with user-segmentedobject silhouettes in the key frames, we adopt aclone construction procedure to build approximateclones that exactly project to their respective realobject silhouettes. The framework is easy to imple-ment, and no expensive equipment is required.In this paper, we have not addressed the issue oflighting, which is as critical as occlusion for a con-

frame 1 frame 15

frame 30 frame 43

Fig. 5. Hotel sequence

164

vincing merge of virtual objects into a real envi-ronment. The 3D clones constructed in our workcan be used as a basis to derive illumination inter-action between virtual and real objects. Also, wesee potential that automatic segmentation on theintermediate frames between two consecutive keyframes can be performed. The dependencybetween the projection model assumed in thestructure-from-motion algorithm and that used inclone construction is another area worth investi-gating.

Acknowledgements. This work is supported by the NationalUniversity of Singapore under grants RP940641 and RP960618.

References

Azuma RT (1997) A survey of augmented reality. Presence:Teleoperators and Virtual Reality 6:355±385

Berger MO (1997) Resolving occlusion in augmented reality: acontour based approach without 3D reconstruction. Confer-ence on Computer Vision and Pattern Recognition, pp 91±96

Breen DE, Whitaker RT, Rose E, Tuceryan M (1996) Interactiveocclusion and automatic object placement for augmented re-ality. Eurographics 15:C11±C22

Ertl G, Mueller-Seelich H, Tabatabai B (1991) Move-x: a sys-tem for combining video films and computer animation.Proceeding of the International Conf. Eurographics '91 pp305±314

Hu X, Ahuja N (1993) Motion and structure estimation usinglong sequence motion models. Image Vision Comput 11:549±570

Kass M, Witikin A, Terzopoulos D (1988) Snakes: active con-tour models. International Journal of Comp Vision, Vol. 1,No. 3, pp 321±331

Koch R (1993) Automatic reconstruction of buildings from ste-reoscopic image sequences. Eurographics 12:C339±C350

Mitsunaga T, Yokoyama T, Totsuka T (1995) AutoKey: humanassisted key extraction. Proceedings of SIGGRAPH 95 (LosAngeles, CA). In Computer Graphics Proceedings, AnnualConference Series, 1995, ACM Press pp 265±272

Mortensen EN, Barrett WA (1995) Intelligent Scissors for imagecomposition. Proceedings of SIGGRAPH 95 (Los Angeles,CA). In Computer Graphics Proceedings, Annual Confer-ence Series, 1995, ACM Press pp 191±198

Nakamae E, Harada K, Ishizaki T, Nishita T (1986) A montagemethod: the overlaying of the computer-generated imagesonto a background photograph. Siggraph, Dallas, ACMPress pp 207±214

Press WH, Teukolsky SA, Vetterling WT, Flannery BP (1992)Numerical recipes in C: the art of scientific computing,2nd edn. Cambridge University Press, Cambridge, pp 683±688

Quan L, Kanade T (1997) Journal Version: Affine structurefrom line correspondences with Uncalibrated Affine Cam-eras. IEEE Transactions on Pattern Analysis & Machine In-telligence vol 19 No 8pp 834±845

Szeliski R (1993) Rapid octree construction from image se-quences. CVGIP: Image Understanding 58:23±32

Szeliski R, Kang SB (1994) Recovering 3D shape and motionfrom image streams using non-linear least squares. J VisualCommun Image Rep 5:10±28

Taylor CJ, Kriegman DJ (1995) Structure and motion from linesegments in multiple images. IEEE Trans Patt Anal MachineIntell 17:1021±1032

Tomasi C, Kanade T (1992) Shape and motion from imagestreams under orthography: a factorization method. Int JComput Vision 9:137±154

Ueda N, Mase K, Suenaga Y (1992) A contour tracking methodusing elastic contour model and energy minimization ap-proach, IEICE Trans J75-D-II:111±120

Ullman S (1979) The interpretation of visual motion. MIT Press,Cambridge, Mass

Vieville T, Faugeras O (1990) Feed-forward recovery of motionand structure from a sequence of 2D-line matches. IEEE In-ternational Conference on Computer Vision, pp 517

Wloka M, Anderson B (1995) Resolving occlusion in augment-ed reality. Symposium on Interactive 3D Graphics Proceed-ings, New York, ACM Press pp 5±12

Yu H, Chen Q, Xu G, Yachida M (1996) 3D shape and motionby SVD under higher-order approximation of perspectiveprojection. Proceedings of 13th International Conferenceon Pattern Recognition, pp 456±460

KIEM-CHING ONG receivedhis BSc (Hons) and MSc in Com-puter Science from the NationalUniversity of Singapore in 1996and 1998, respectively. He isnow working as an R&D Engi-neer in a commercial simulationcompany. His research interestsinclude computer graphics, com-puter animation and their appli-cations.

165

HUNG-CHUAN TEH receivedhis PhD (1972) in Physics fromMcMaster University, Canada.His previous research in physicsincludes studies of the thermal,magnetic and relaxation proper-ties of matters using inelasticthermal neutron scattering andlaser-light scattering. He is cur-rently an Associate Professor atthe Department of InformationSystems and Computer Scienceat the National University of Sin-gapore. His current research in-terests include geometric model-ing, global illumination and im-age-based rendering. He is amember of the ACM and IEEE.

TIOW-SENG TAN received hisBSc, majoring in Computer Sci-ence and Mathematics, from theNational University of Singapore(NUS) in 1984. Having workedfor a few years in the IT industry,he then returned to NUS to com-plete an MSc in 1988, and then aPhD in 1992 at the University ofIllinois at Urbana-Champaign.He is currently a Senior Lecturerat the Department of InformationSystems and Computer Science,NUS. His research interests in-clude computational geometry,computer graphics and imageprocessing.