Photo Tourism

Exploring Photo Collections in 3DPhoto TourismIntroductionThe internet has become a vast, ever-growing repository of visual information about our world.Virtually all of the worlds famous landmarks and cities have been photographed many different times, both from the ground and from the air.Millions of ImagesThere are billions of photographs on the Internet.Representing an extremely large, rich, and nearly comprehensive visual record of virtually every famous place on Earth.

Internet 2012 in numbers7 petabytesHow much photo content Facebookaddedevery month.

300 millionNumber of new photosaddedevery day to Facebook.

5 billionThetotal number of photosuploaded to Instagram since its start, reached in September 2012.

58 Number of photos uploadedevery secondto Instagram.

1 Apple iPhone 4S was the most popularcameraon Flickr.

OpportunityEnormous opportunities, both for research and for practical applications.Mining the collections to create the ultimate virtual experience that could be extremely visually compelling, giving us the ability to walk around in a photorealistic 3-D version of the scene, to let us dial in any time of day or year, and to let us revisit different events.

GoalTo design a novel system for registering large sets of photos and exploring them in a 3D browser, that provides:Accurate 3D reconstruction.Geometric and semantic scene structure.Interactive scene visualization system.Interesting views, and segmenting.Individual labeled objects.Goal

The ConceptThe system is based on the idea of using camera pose (location, orientation, and field of view).First the system computes feature correspondences between images, using descriptors that are robust with respect to variations in pose, scale, and lighting.Then runs an optimization to recover the camera parameters and 3D positions of those features.MethodsSIFTVocabulary Trees.SFM StructureFromMotion.

MethodsSIFTVocabulary Trees.SFM StructureFromMotion.

SIFT

MethodsSIFTVocabulary Trees.SFM StructureFromMotion.

Vocabulary Trees

Building The Tree

Building The TreeBuilding The TreeBuilding The Tree

MethodsSIFTVocabulary Trees.SFM StructureFromMotion.

Structure From MotionThe problem:Given optical flow or point correspondences, compute 3-D motion (translation and rotation) and shape (depth).

Structure From Motion

Structure From MotionAssumptionsThe positions of P points in F frames (F>=3), which are not all coplanar, and have been tracked.The entire sequence has been acquired before starting (batch mode).

Camera calibration not needed, if we accept 3D points up to a scale factor.

Feature Points

Orthographic ProjectionHow to find Translation?Rank TheoremSingular Valued DecompositionNote:Where are we?

Additional ApplicationsImage Morphing.Annotating Objects.Classifying Photos.Additional ApplicationsImage Morphing.Annotating Objects.Classifying Photos.Image Morphing

Camera Transitions

Additional ApplicationsImage Morphing.Annotating Objects.Classifying Photos.Annotating Objects

35Annotating ObjectsSometimes the system also uses simple heuristics to determine if an annotated region is included.

Annotating Objects

Additional ApplicationsImage Morphing.Annotating Objects.Classifying Photos.Classifying PhotosThe system also allows users to classify photos into different categories, such as day and night.The system can then create controls for changing the appearance of a scene by toggling between categories of photos.

References"Photo tourism: Exploring photo collections in3D,"NoahSnavely, Steven M. Seitz, RichardSzeliski,ACM Transactions on Graphics (SIGGRAPH Proceedings), 25(3), 2006, 835-846."Modeling the world from Internet photo collections,"NoahSnavely, Steven M. Seitz, RichardSzeliski,International Journal of Computer Vision (to be published).Scene Reconstruction and Visualization From Community Photo CollectionsNoahSnavely, Ian Simon, MichaelGoesele, RichardSzeliski, and Steven M. Seitz.Building Rome in a DaySameerAgarwal, NoahSnavely, Ian Simon, Steven M. Seitz and RichardSzeliskiInternational Conference on Computer Vision, 2009,Kyoto,Japan. Scalable Recognition with a Vocabulary Tree,David Nister and Henrik Stewenius, Center for Visualization and Virtual EnvironmentsDepartment of Computer Science, University of Kentucky.Multiview Structure from Motion in Trajectory SpaceAamer Zaheer, Ijaz Akhter, Mohammad Haris Baig, Shabbir Marzban, Sohaib Khan, ICCV 2011 Nonrigid Structure From Motion,Yaser Sheikh and Sohaib Khan, ECCV 2010 TUTORIAL.A Vocabulary Tree for Image Classication: Open Source Implementation, Validation and Characterization, Master in Computer Vision and Artificial Intelligence, Report of the research project, Author: Sergi Rubio Manrique, Advisor: Ricardo Toledo.