virtual cinematography theory and practice for automatic real- time camera control and directing...
TRANSCRIPT
![Page 1: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/1.jpg)
Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-
Time Camera Control and DirectingTime Camera Control and Directing
Liwei He Liwei He Microsoft ResearchMicrosoft Research
http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe
![Page 2: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/2.jpg)
MotivationMotivation
There are 3 elements in computer graphicsThere are 3 elements in computer graphics– lights, scene objects, andlights, scene objects, and
– the camera (focus of this talk)the camera (focus of this talk) Camera control is hard (7 DOFs)Camera control is hard (7 DOFs)
– position (3), direction (3), field of view (1)position (3), direction (3), field of view (1) We may learn from cinematographyWe may learn from cinematography
![Page 3: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/3.jpg)
Roles in cinematographyRoles in cinematography
Low-level responsibilitiesLow-level responsibilities– CameramanCameraman
• Position and move the cameraPosition and move the camera
– Film editorFilm editor• Keep the film to proper lengthKeep the film to proper length• Decide shot transitionsDecide shot transitions
High-level responsibilitiesHigh-level responsibilities– Script writer and directorScript writer and director
• Story-tellingStory-telling
![Page 4: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/4.jpg)
Camera control level 1Camera control level 1
Input: camera position and directionInput: camera position and direction Output: camera transformationOutput: camera transformation Application: low-level graphics library Application: low-level graphics library
routines (Direct3D)routines (Direct3D)
![Page 5: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/5.jpg)
Camera control level 2Camera control level 2
Input: show both A and B, follow C, etc.Input: show both A and B, follow C, etc. Output: camera position and directionOutput: camera position and direction Application: 3D game, computer Application: 3D game, computer
animation packageanimation package
![Page 6: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/6.jpg)
Camera control level 3Camera control level 3
Input: high-level user directionsInput: high-level user directions– show a conversation show a conversation
– show a car chase sceneshow a car chase scene Output: a sequence of level 2 camera specOutput: a sequence of level 2 camera spec Applications: graphical chat, tele-Applications: graphical chat, tele-
conferencing, Virtual Reality gamesconferencing, Virtual Reality games
![Page 7: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/7.jpg)
Camera control level 3 (cont.)Camera control level 3 (cont.)
Encodes cinematographic expertiseEncodes cinematographic expertise– an interesting problem in itselfan interesting problem in itself
Provides an interface that isProvides an interface that is– real-timereal-time
– visually entertainingvisually entertaining
– more informativemore informative
![Page 8: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/8.jpg)
![Page 9: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/9.jpg)
Eyepeep BARG07011 Feldegast
Microsoft V-Chat Lunar Island
Eyepeep says "really"Feldegast says "yup"BARG07011 has joined the conversationEyepeep says "im still older"Feldegast says "lol"DGeste309 does something silly.
BARG07011DGerste309EyepeepFeldegastGrenDayGod
![Page 10: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/10.jpg)
Principles of CinematographyPrinciples of Cinematography
![Page 11: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/11.jpg)
Film structureFilm structure
FilmFilm
......
......
ScenesScenes
ShotsShots
![Page 12: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/12.jpg)
Camera distanceCamera distance
Close upClose up Close shotClose shot Medium shotMedium shot
Full shotFull shot Long shotLong shot
![Page 13: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/13.jpg)
The line of interestThe line of interest
A B
![Page 14: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/14.jpg)
External cameraExternal camera
A B
![Page 15: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/15.jpg)
External cameraExternal camera
A B
![Page 16: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/16.jpg)
Internal cameraInternal camera
A B
![Page 17: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/17.jpg)
Apex cameraApex camera
A B
![Page 18: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/18.jpg)
Moving camerasMoving cameras
A B
pan(A) track(B)
![Page 19: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/19.jpg)
Some rules in film editingSome rules in film editing
Don't cross the line of interestDon't cross the line of interest Avoid jump cuts Let the actor lead Break movement
![Page 20: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/20.jpg)
Don’t cross the line of interestDon’t cross the line of interest
1
2
![Page 21: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/21.jpg)
![Page 22: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/22.jpg)
![Page 23: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/23.jpg)
![Page 24: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/24.jpg)
![Page 25: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/25.jpg)
Some rules in film editingSome rules in film editing
Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement
![Page 26: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/26.jpg)
Avoid jump cutAvoid jump cut
![Page 27: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/27.jpg)
Some rules in film editingSome rules in film editing
Don't cross the line of interest Avoid jump cutsAvoid jump cuts Let the actor lead Break movement
![Page 28: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/28.jpg)
Two-person conversation Two-person conversation
A B
11 22
33 44
![Page 29: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/29.jpg)
2
3
1
A BA B
A
4 B
![Page 30: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/30.jpg)
Three-person conversationThree-person conversation
22
44
33
11
B
A
C
![Page 31: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/31.jpg)
43
2
A B
C
A B
1
A B
C
![Page 32: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/32.jpg)
![Page 33: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/33.jpg)
![Page 34: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/34.jpg)
Bad
![Page 35: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/35.jpg)
Good
![Page 36: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/36.jpg)
x
y
z
U n iv e rse
Three spacesThree spaces
The universe space
![Page 37: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/37.jpg)
Three spacesThree spaces
The eye space (z-axis is the look at direction)
x
y
z
U n iv e rse
zE y e
x y
![Page 38: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/38.jpg)
Three spacesThree spaces
The screen space (screen is ctan(fov/2) away from eye point)
x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
![Page 39: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/39.jpg)
Standard look-at transformationStandard look-at transformation
Given eye position Pfrom, a look at direction
Want a rotation matrix R and a translation to transform a vector from universe space to eye space
x y
TT
x
y
z
U n iv e rse
P f ro m
E y e
![Page 40: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/40.jpg)
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
xy
![Page 41: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/41.jpg)
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
x y
![Page 42: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/42.jpg)
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
But this is not uniqueT
yx
![Page 43: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/43.jpg)
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
We will need to specify an up vector , usually [0,1,0]U
U
T
![Page 44: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/44.jpg)
x
y
z
U n iv e rse
P f ro m
E y e
Standard look-at transformationStandard look-at transformation
In eye space
Z-axis is
X-axis is
Y-axis is
Rotation matrix
Translation is -Pfrom
T
ZUV ˆˆˆ
VTU ˆˆˆ
TUVR ˆˆˆ33
UU V
T
![Page 45: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/45.jpg)
Internal close-up of actor AInternal close-up of actor A
![Page 46: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/46.jpg)
x
y
z
U n iv e rse
P f ro m
E y eP a t
Look at [xLook at [xatat,y,yatat] transformation] transformation
U V
TGiven eye position Pfrom, look at point Pat, and Pat in screen space [xat, yat]
Want
In universe space
T
A
)(ˆfromat PPunitA
![Page 47: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/47.jpg)
Look at [xLook at [xatat,y,yatat] transformation] transformation
In screen space:
H = [xat, yat, ctan(fov/2)]
Solve from the following equations: x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
P f ro m
P a t
H
1ˆˆ
ˆˆ]0,1,0[ˆ
ˆˆ]1,0,0[ˆ
UT
UAH
TAH
T
![Page 48: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/48.jpg)
External of actor A and actor BExternal of actor A and actor B
![Page 49: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/49.jpg)
x
y
z
U n iv e rse
P f ro m
E y eP B
P A
Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]
U V
T
Given PA,PB in universe space and eye space, and distance of eye position Pfrom and Pa.
Want Pfrom and T
A
![Page 50: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/50.jpg)
Use numerical method:
1. Set Pfrom to 0
2. Solve R using the Look at [xat,yat] method
3. Transform E by inverse of R, getting a new Pfrom approximation
4. Goto 2
Look at [xLook at [xAA,y,yAA] and [x] and [xBB,y,yBB]]
x
y
z
E y eS c re e n
-1 + 1
+ 1
-1
P f ro m
P B
H
P A
![Page 51: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/51.jpg)
Medium panning shot of actor AMedium panning shot of actor A
![Page 52: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/52.jpg)
ReferencesReferences
Jim Blinn’s CG&A ’88 articleJim Blinn’s CG&A ’88 article– Where am I? What am I looking at?Where am I? What am I looking at?
Steve Drucker’s Interactive 3D ’92, ’94, Steve Drucker’s Interactive 3D ’92, ’94, ‘95 papers‘95 papers– Cast camera positioning as an Cast camera positioning as an
optimization problemoptimization problem
![Page 53: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/53.jpg)
System DesignSystem Design
![Page 54: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/54.jpg)
Overall system diagramOverall system diagram
Real-timeApplication
Virtual Cinematographer
Renderer
![Page 55: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/55.jpg)
At each time stepAt each time step
1. Application generates events to VC1. Application generates events to VC– ( subject, verb, object )( subject, verb, object )
2. VC determines– camera specifications
– acting hints
3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
![Page 56: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/56.jpg)
At each time stepAt each time step
1. Application generates events to VC– ( subject, verb, object )
2. VC determines2. VC determines– camera specificationscamera specifications
– acting hintsacting hints
3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
![Page 57: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/57.jpg)
At each time stepAt each time step
1. Application generates events to VC– ( subject, verb, object )
2. VC determines– camera specifications
– acting hints
3. Renderer outputs the image3. Renderer outputs the image
Real-timeApplication
Virtual Cinematographer
Renderer
![Page 58: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/58.jpg)
A networked virtual party gameA networked virtual party game
![Page 59: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/59.jpg)
Actors are simulatedActors are simulated
Actor’s mind
LonelinessLoneliness
ThirstThirst
BoredomBoredom
Walk, converse, look around, Walk, converse, look around, drink, etc.drink, etc.
Can be controlled by the usersCan be controlled by the users
![Page 60: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/60.jpg)
Networked virtual party gameNetworked virtual party game
Server
Client
Client
Client
User actionsUser actions
EventsEvents
![Page 61: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/61.jpg)
Inside the Virtual Inside the Virtual CinematographerCinematographer
![Page 62: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/62.jpg)
VC architectureVC architecture
Idioms
Camera modules
![Page 63: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/63.jpg)
VC architectureVC architecture
Idioms
Camera modules
![Page 64: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/64.jpg)
Camera modulesCamera modules
Geometric placement of specific Geometric placement of specific cameras for each shotcameras for each shot
Choose the side of the line of interest Influence acting
![Page 65: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/65.jpg)
Camera module: Camera module: ext(B,A)ext(B,A)
A B
![Page 66: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/66.jpg)
Static camera modulesStatic camera modules
A B
ext(B,A) ext(A,B)
apex(A,B)
int(B)
![Page 67: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/67.jpg)
Moving camera modulesMoving camera modules
A B
pan(A) track(B)
![Page 68: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/68.jpg)
Camera modulesCamera modules
Geometric placement of specific cameras for each shot
Choose the side of the line of interestChoose the side of the line of interest Influence acting
![Page 69: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/69.jpg)
Don’t cross the line of interestDon’t cross the line of interest
1
2
![Page 70: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/70.jpg)
Camera modulesCamera modules
Geometric placement of specific cameras for each shot
Choose the side of the line of interest Influence actingInfluence acting
![Page 71: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/71.jpg)
Acting hintsActing hints
A BA’A’ B’B’
![Page 72: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/72.jpg)
VC architectureVC architecture
Idioms
Camera modules
![Page 73: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/73.jpg)
Previous worksPrevious works
Dave Christianson’s AAAI ’96 paperDave Christianson’s AAAI ’96 paper– Declarative camera control for automatic Declarative camera control for automatic
cinematographycinematography
– An off-line algorithmAn off-line algorithm
– Need to know all events aheadNeed to know all events ahead
– Use plan algorithms in AIUse plan algorithms in AI
![Page 74: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/74.jpg)
Film idiomsFilm idioms
Capture a particular type of sceneCapture a particular type of scene Register relevant eventsRegister relevant events Select shot typesSelect shot types Determine the transitions between shotsDetermine the transitions between shots
![Page 75: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/75.jpg)
Hierarchical idiom structureHierarchical idiom structure
LookaroundLookaround
MasterMaster
ConverseConverse DrinkingDrinking MovingMoving
3Talk3Talk
2Talk2Talk
![Page 76: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/76.jpg)
Hierarchical idiom structureHierarchical idiom structure
Avoids exponential growth of statesAvoids exponential growth of states Allows expertise to be reusedAllows expertise to be reused Provides robustnessProvides robustness
![Page 77: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/77.jpg)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
![Page 78: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/78.jpg)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
T > 8
![Page 79: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/79.jpg)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
T > 8
![Page 80: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/80.jpg)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)D < 10
![Page 81: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/81.jpg)
MovingMoving idiom idiom
track(C)pan(A)
ext(A,B) ext(B,A)
apex(A,B) apex(A,B)
D < 5
![Page 82: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/82.jpg)
MovingMoving idiom idiom
apex(A,B)
D < 5
apex(A,B)
pan(A)
ext(B,A)
T > 8
ext(A,B)
D < 10
T > 8
![Page 83: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/83.jpg)
![Page 84: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/84.jpg)
3Talk3Talk idiom idiom
int(C)
int(C)
ext1to2
2Talk(A,B)
![Page 85: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/85.jpg)
Invoking a sub-idiomInvoking a sub-idiom
int(C)
int(C)
ext1to2
2Talk(A,B)
ext(B,A)ext(B,A)
ext(A,B)
ext(A,B) 2Talk
![Page 86: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/86.jpg)
ContributionsContributions
Real-time camera controlReal-time camera control– lightweightlightweight
– automatic camera placementautomatic camera placement
– automatic shot transitionsautomatic shot transitions Do a reasonable jobDo a reasonable job
– not to replace human in computer animationnot to replace human in computer animation
![Page 87: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/87.jpg)
Work in ProgressWork in Progress
![Page 88: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/88.jpg)
Camera control for chatCamera control for chat
![Page 89: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/89.jpg)
![Page 90: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/90.jpg)
Camera control for live meetingCamera control for live meeting
![Page 91: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/91.jpg)
Image after warpingImage after warping
![Page 92: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/92.jpg)
Camera control for classroomCamera control for classroom
Multiple cameras per classroomMultiple cameras per classroom– Lecturer camerasLecturer cameras
– Audience cameraAudience camera Use audio and vision techniques toUse audio and vision techniques to
– Track the lecturerTrack the lecturer
– Activate the audience cameraActivate the audience camera
![Page 93: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/93.jpg)
Dramatic lightingDramatic lighting
![Page 94: Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research](https://reader036.vdocuments.net/reader036/viewer/2022062620/551b0f4e5503465e7d8b5f6d/html5/thumbnails/94.jpg)
Virtual CinematographyVirtual CinematographyTheory and Practice for Automatic Real-Theory and Practice for Automatic Real-
Time Camera Control and DirectingTime Camera Control and Directing
Liwei He Liwei He Microsoft ResearchMicrosoft Research
http://research.microsoft.com/users/lhehttp://research.microsoft.com/users/lhe