how graphics, multimedia, vision, visualization, animation...
TRANSCRIPT
![Page 1: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/1.jpg)
Pictures from Piles of DataHow Graphics, Multimedia, Vision, Visualization, Animation and Cartography All Connect
Michael Gleicher
Dept of Computer SciencesUniversity of Wisconsin - Madison
![Page 2: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/2.jpg)
Acknowledgements
• All of this work is done in collaboration with a great group of students.
• This talk is mainly work done with:
Feng Liu – multimedia, video(work supported by NSF, Adobe)
Greg Cipriano– molecules, vis(work supported by NIH)
![Page 3: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/3.jpg)
Pictures from Piles of DataHow Graphics, Multimedia, Vision, Visualization, Animation and Cartography All Connect
Michael Gleicher
Dept of Computer SciencesUniversity of Wisconsin - Madison
![Page 4: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/4.jpg)
Pictures from Piles of DataHow Graphics, Multimedia, Vision, Visualization, Animation and Cartography All Connect
Michael Gleicher
Dept of Computer SciencesUniversity of Wisconsin - Madison
![Page 5: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/5.jpg)
Pictures from Piles of DataHow Graphics, Multimedia, Vision, Visualization, Animation and Cartography All Connect
Michael Gleicher
Dept of Computer SciencesUniversity of Wisconsin - Madison
Stuff I do,and why you might be interested
![Page 6: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/6.jpg)
What do these have in common?
Analysis of Proteins
Scientific DataDisplay
Motion Synthesisfor Characters
Video QualityImprovement
Image and VideoRetargeting
Multimedia DatabaseInformation Extraction
![Page 7: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/7.jpg)
What do these have in common?
• It’s all stuff I’ve done in the past few years
![Page 8: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/8.jpg)
What do these have in common?
• It’s all stuff I’ve done in the past few years
• It involves large amounts of data
• It involves creating effective presentations
• It requires some understanding of the datain order to simplify it
![Page 9: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/9.jpg)
Is there any historical precedent?
Data abundance in the modern age
![Page 10: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/10.jpg)
The world, 1528. Birmingham Public Library Digital Collection. http://bplonline.cdmhost.com/cdm4/results.php?CISOOP1=any&CISOFIELD1=CISOSEARCHALL&CISOROOT=/p4017coll7&CISOBOX1=World1528a.sid
![Page 11: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/11.jpg)
Charte Cosmographique, auec les Noms, Proprietez, Nature & Operations des Vents (1544) Birmingham Public Library
![Page 12: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/12.jpg)
Figura Del Mondo Universale
Figura Del Mondo Universale, 1558
![Page 13: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/13.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 14: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/14.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 15: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/15.jpg)
A Protein Surface
![Page 16: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/16.jpg)
An aside…
How do scientists look at proteins?
Stick and Ball Model (internals)
![Page 17: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/17.jpg)
An aside…
How do scientists look at proteins?
Stick and Ball Model (internals)
Ribbon Diagram (internals)
![Page 18: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/18.jpg)
An aside…
How do scientists look at proteins?
Stick and Ball Model (internals)
Molecular Surface (externals)
![Page 19: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/19.jpg)
A Protein Surface
![Page 20: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/20.jpg)
Molecular Surface Abstraction
![Page 21: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/21.jpg)
What’s Happening?
• Simplification
• Stylized Display
• Surface Indications
![Page 22: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/22.jpg)
Putting Information on Surfaces
Surface Parameterization• Only for local regions• Requires smoothness• Artistic issues?
![Page 23: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/23.jpg)
Patch Smoothing
Before After
![Page 24: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/24.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 25: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/25.jpg)
Put text on a surface?
![Page 26: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/26.jpg)
Put text near the surface?
![Page 27: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/27.jpg)
Put text near the surface?
Text scaffold
![Page 28: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/28.jpg)
Text Scaffolds
![Page 29: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/29.jpg)
Cartographic Labeling
Basic principles:
• Text should be legible
– Smooth, likely straight, path
• Text should be visible
– Always on top of other features
• Text may track important features
– E.g. roads, rivers
• Text should be close to feature
![Page 30: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/30.jpg)
Translate these goals into 3D!
![Page 31: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/31.jpg)
Goals
• Legibility• Text must be readable
• Surface must be smooth
• Visibility• Surface can’t enter occluded regions
• Surface must bound object
• Proximity• Labels must lie as close as possible to
corresponding region
• Shape-conveying• Scaffold should retain as much of the
original shape as possible.
![Page 32: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/32.jpg)
Goals
• Legibility• Text must be readable
• Surface must be smooth
• Visibility• Surface can’t enter occluded regions
• Surface must bound object
• Proximity• Labels must lie as close as possible to
corresponding region
• Shape-conveying• Scaffold should retain as much of the
original shape as possible.
![Page 33: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/33.jpg)
Goals
• Legibility• Text must be readable
• Surface must be smooth
• Visibility• Surface can’t enter occluded regions
• Surface must bound object
• Proximity• Labels must lie as close as possible to
corresponding region
• Shape-conveying• Scaffold should retain as much of the
original shape as possible.
![Page 34: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/34.jpg)
Goals
• Legibility• Text must be readable
• Surface must be smooth
• Visibility• Surface can’t enter occluded regions
• Surface must bound object
• Proximity• Labels must lie as close as possible to
corresponding region
• Shape-conveying• Scaffold should retain as much of the
original shape as possible.
![Page 35: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/35.jpg)
How to meet these goals?
• Create a surface to put labels ona Text Scaffold
• Use 2D cartography methods on the scaffold
![Page 36: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/36.jpg)
Creating Text Scaffolds
• Start with anything
• Convert to Distance Field
• Adapt field to meet goals
• Convert to mesh
![Page 37: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/37.jpg)
Example
![Page 38: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/38.jpg)
![Page 39: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/39.jpg)
Results
![Page 40: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/40.jpg)
Results
![Page 41: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/41.jpg)
Examples
![Page 42: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/42.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 43: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/43.jpg)
From Artistic and Perceptual Principles
To Computational Models
![Page 44: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/44.jpg)
Motivation:More video doesn’t mean better video
Good video takes effort!
![Page 45: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/45.jpg)
Problem: Bad Camera Motion
No planning
No tripod
![Page 46: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/46.jpg)
Problem: Bad Camera Motion Prior Work: Image Stabilization
One part of the problem: jitter
Helped by
Image Stabilization
![Page 47: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/47.jpg)
Problem: Bad Camera MotionSolution: Re-Cinematography
Re-Cinematography:Post-process video clips so that the camera motions better follow the rules of good video.
![Page 48: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/48.jpg)
Rubber duck racesVail, CO, USA, 19 August, 2007
Source Footage Re-Cinematography Result
![Page 49: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/49.jpg)
![Page 50: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/50.jpg)
What the art of cinematography tells us about camera motion
Camera motions should be intentional
– Avoid movement if not necessary
– Move in directed ways
Re-Cinematography:Post-process video clips so that the camera motions appear to better follow the rules.
![Page 51: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/51.jpg)
Re-Cinematography Pipeline
SourceVideo
MotionEstimation
MotionSynthesis
ImageTransform
ResultVideo
![Page 52: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/52.jpg)
Re-Cinematography Pipeline (1)
SourceVideo
MotionEstimation
MotionSynthesis
ImageTransform
ResultVideo
How did the camera move?
![Page 53: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/53.jpg)
Re-Cinematography Pipeline (2)
SourceVideo
MotionEstimation
MotionSynthesis
ImageTransform
ResultVideo
Figure out what motion we want in the result
![Page 54: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/54.jpg)
Re-Cinematography Pipeline (3)
SourceVideo
MotionEstimation
MotionSynthesis
ImageTransform
ResultVideo
Transform the source into the result
![Page 55: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/55.jpg)
Re-Cinematography Pipeline
SceneAnalysis
MotionAnalysis
MotionSynthesis
SourceVideo
MotionEstimation
ImageTransform
ResultVideo
![Page 56: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/56.jpg)
Motion Synthesis Steps
SourceVideo
MotionEstimation
MotionSynthesis
ImageTransform
ResultVideo
CreateMotions
SegmentVideo
OptimizeMotions
![Page 57: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/57.jpg)
3 Key Ideas
• Analyze motion estimates to break video into segments
• Use local mosaics to keyframe new camera motions
• Consider both motion and image quality to automatically keyframe cameras
![Page 58: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/58.jpg)
Local Mosaics
Limit error and motion in each segment
![Page 59: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/59.jpg)
Virtual camera does not have to be where the real camera was
Source frames shown in yellow
Result frames shown in magenta
![Page 60: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/60.jpg)
What paths do we want?
1. Preserve the intent of the source
2. Obey the rule of cinematography:
Camera motion should be intentional
![Page 61: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/61.jpg)
The key insight:Translate cinematography to implementationMotion should be intentional
• Static shots should be static
• Moving shots are goal directed
– Constant velocity with ease in/out
![Page 62: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/62.jpg)
Directed Paths
Interpolate with direct constant* velocity paths
* Possibly with ease-in and out.
![Page 63: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/63.jpg)
Smooth Paths Depart from Original
Source motion Result motion
![Page 64: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/64.jpg)
Changing motion means transforming frames
Source motion Result motion
![Page 65: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/65.jpg)
Transforming frames might cause problems
Source frame Result frame
![Page 66: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/66.jpg)
Penalties for each frane
Offscreen
Uncovered
Distortion
![Page 67: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/67.jpg)
Offscreen
![Page 68: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/68.jpg)
Uncovered
![Page 69: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/69.jpg)
Distorted
![Page 70: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/70.jpg)
A contrived synthetic example to explain key insertion
![Page 71: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/71.jpg)
Try the smooth motion first
![Page 72: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/72.jpg)
Insert a key at the worst point
![Page 73: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/73.jpg)
![Page 74: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/74.jpg)
Examples
• 2X speed to emphasize motion
• Everything is full-frame
• Everything is from real home videos
![Page 75: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/75.jpg)
318 sourceLearning to run
Vail, CO, 19 August 2006
Source Video
318 2X
![Page 76: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/76.jpg)
318 source video
![Page 77: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/77.jpg)
318 resultLearning to run
Vail, CO, 19 August 2006
Re-Cinematography Result
![Page 78: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/78.jpg)
318 result video
![Page 79: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/79.jpg)
318 2X2X speed comparison
Source Footage Re-Cinematography Result
![Page 80: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/80.jpg)
318 2X video2X speed comparison
Source Footage Re-Cinematography Result
![Page 81: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/81.jpg)
A more interesting question:To swing or not to swing
Source Footage Re-Cinematography Result
![Page 82: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/82.jpg)
![Page 83: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/83.jpg)
Artifacts
Where did she come from?
Input: Output:
![Page 84: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/84.jpg)
Sam’s First Steps, July 6th, 2006
Re-CinematographyResult
Skip
![Page 85: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/85.jpg)
First Steps
![Page 86: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/86.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 87: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/87.jpg)
3D experienceswithout 3D models
How do you move the camera to a different place?
![Page 88: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/88.jpg)
Video Stabilization
• Existing approach: 2D stabilization
Track a bunch of points Fit full-frame warps that best smooth point motion
• Limited!
• Doesn’t model parallax
• Can’t reason about camera motions in 3D
![Page 89: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/89.jpg)
3D Video Stabilization• Reconstruct 3D model of world
• Plan new camera path
• Synthesize new images
![Page 90: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/90.jpg)
Novel viewpoint rendering
How to make a video-quality image?
• Build a really high-quality 3D model?Too hardNot enough input data
• Image-Based Rendering using other frames?Violates temporal constraintsComputationally expensive
![Page 91: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/91.jpg)
Novel view from one frame
Impossible?
• Incomplete geometric model (sparse)
• Occlusions / Dis-occlusions
![Page 92: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/92.jpg)
Novel view from one frame
Impossible?
• Incomplete geometric model (sparse)
• Occlusions / Dis-occlusions
Impossible! So Fake it!
• Just need visually plausible, not accuracy
• Viewpoint shifts will be small
• Avoid artifacts
![Page 93: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/93.jpg)
3D Stabilization by Image Warping
• Structure from motion gives sparse points
• 3D camera planning gives motion of points
• Use sparse points to warp image
![Page 95: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/95.jpg)
Talk Roadmap
Molecular Surface Abstraction
Surface Cartography
Warping for 3D VideoOdds and Ends
Re-Cinematography
![Page 96: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/96.jpg)
Too many things to talk about…
But wait… There’s more!
![Page 97: How Graphics, Multimedia, Vision, Visualization, Animation ...gleicher/talks/2009_07_navtec/navtec2.pdf · Odds and Ends Warping for 3D Video Re-Cinematography. Talk Roadmap Molecular](https://reader036.vdocuments.net/reader036/viewer/2022081410/609f1fdf33acb12463646132/html5/thumbnails/97.jpg)
Things we could talk about…
• Creating “abstracted” vector art from data
• Cartoons from data
• Adapting for small displays / interfaces
• Stylized drawing from other data
• Camera control during drives throughs
• Meaningful synthetic drive throughs
• Idealified geometry from real images
• …