creating and exploring a large photorealistic virtual space inria / csail / adobe first ieee...
TRANSCRIPT
![Page 1: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/1.jpg)
Creating and Exploring a Large Photorealistic Virtual Space
INRIA / CSAIL / Adobe
First IEEE Workshop on Internet Vision, associated with CVPR 2008.
![Page 2: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/2.jpg)
Outline• Introduction• Constructing the image space• Navigating the virtual 3D space• Limitations and Conclusion
![Page 3: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/3.jpg)
Introduction
• We present a system for exploring large collections of photos in a virtual 3D space.
• Let users navigate within each theme using intuitive 3D controls that include move left/right, zoom and rotate.
![Page 4: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/4.jpg)
• In a similar fashion we can create infinite zoom effects that resemble the ”Droste effect”.
![Page 5: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/5.jpg)
Constructing the image space
•The image database– We have collected ~6 million images from Flickr
based on keyword and group searches• typical image size is 500x375 pixels• 720GB of disk space (jpeg compressed)
![Page 6: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/6.jpg)
Image representation
Color layout
GIST [Oliva and Torralba’01]
Original image
![Page 7: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/7.jpg)
Obtaining semantically coherent themesWe further break-up the collection into themes of semantically coherent scenes:
Train SVM-based classifiers from 1-2k training images [Oliva and Torralba, 2001]
![Page 8: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/8.jpg)
Basic camera motions
Forward motion Camera rotation Camera pan
Starting from a single image, find a sequence of images to simulate a camera motion:
![Page 9: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/9.jpg)
3. Find a match to fill the missing pixels
Scene matching with camera view transformations: Translation
1. Move camera
2. View from the virtual camera
4. Locally align images
5. Find a seam
6. Blend in the gradient domain
![Page 10: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/10.jpg)
4. Stitched rotation
Scene matching with camera view transformations: Camera rotation
1. Rotate camera
2. View from the virtual camera
3. Find a match to fill-in the missing pixels
5. Display on a cylinder
![Page 11: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/11.jpg)
Steps
• Collect images• Classify images into topic themes• Calculate the descriptors:
– GIST– RGB
• Build the graph• Find the path for given query image(s)
– Dijkstra algorithm• Alignment
– Gradient decent alignment• Composition
– Poisson blending
![Page 12: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/12.jpg)
More “infinite” images – camera translation
![Page 13: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/13.jpg)
Forward
Rotate (left/right)
Pan (left/right)
• Nodes represent Images
• Edges represent particular motions:
• Edge cost is given by the cost of the image match under the particular transformation
Image graph
Navigating the virtual 3D space
• Virtual space as an image graph
![Page 14: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/14.jpg)
•Virtual image space laid out in 3D
![Page 15: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/15.jpg)
Limitations and Conclusion
• The larger the database, the better the results.• Compositing two distinct images is always a challenge
and at times, the seam is quite visible.• This system can be used to create photorealistic visual
content for large online virtual 3D worlds like Second Life.
• Create infinite panoramas or use the image taxi to generate tailor-made tours in the virtual 3D space. These applications can find use in games, movies and other media creation processes.
![Page 16: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/16.jpg)
![Page 17: Creating and Exploring a Large Photorealistic Virtual Space INRIA / CSAIL / Adobe First IEEE Workshop on Internet Vision, associated with CVPR 2008](https://reader030.vdocuments.net/reader030/viewer/2022032705/56649d9e5503460f94a8909c/html5/thumbnails/17.jpg)
Thank you!