1 anahita: a system for 3d video streaming with depth customization k. calagari, k. templin, t....
TRANSCRIPT
![Page 1: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/1.jpg)
1
Anahita: A System for 3D Video Streaming with Depth Customization
K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda
QCR & MIT & SFU5 November 2014
![Page 2: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/2.jpg)
2
3D content is popular especially in theaters
Motivations
Controlled, homogenous environments
![Page 3: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/3.jpg)
3
… is a lot more challenging because of …
complexity of perceiving and rendering depth
on different displays
&
in dynamic network conditions
Internet 3D Video Streaming …
![Page 4: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/4.jpg)
Sample 3D Displays
4
![Page 5: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/5.jpg)
How do we see 3D?
5
![Page 6: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/6.jpg)
6
Approximate Model for Depth [Holliman 04]
d
e
Left eye
Right eye
p
Left eye
Positive disparity
zd
e
Right eye
p
z
Negative disparity
![Page 7: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/7.jpg)
Comfort zone depends on:
• Presented content• Viewing condition• Screen distance• …
“The zone of comfort: Predicting visual discomfort with stereo displays” by Shibata et al. 2011
Depth & Comfort Zone [Shibata 11]
7
![Page 8: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/8.jpg)
8
Many different display technologies - From anaglyph to auto stereoscopic
Rendering 3D Content
![Page 9: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/9.jpg)
3D Display: Active/Passive Glasses
9
![Page 10: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/10.jpg)
Autostereoscopic
• Parallax Barriers:
• Blocks light in certain direction
3D Display: Glasses-Free
• Lenticular Lens:
• Reflects light to certain direction
10
![Page 11: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/11.jpg)
3D Display: Multi-views
11
![Page 12: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/12.jpg)
12
Dynamic conditions- Bandwidth- Loss rate- Delay
Network Dynamics
![Page 13: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/13.jpg)
13
Depth perception heavily depends on - Display size and technology - User preferences
Current 3D streaming systems are simple add-on to 2D streaming - Poor depth/visual perception and/or discomfort
In Summary …
![Page 14: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/14.jpg)
Customize depth for different displays- Big TV … to phone
Serve different formats- Stereo, V+D, MV + D, …
Adapt to network dynamics - Wired, wireless, …, varying bandwidth
Manage many versions - Compared to 1—3 versions for 2D videos
3D Streaming Systems Need to ..
14
![Page 15: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/15.jpg)
Side By Side
Top Bottom
Frame Sequential
Anaglyph
Row-Interleaved
Column-Interleaved
Video Plus Depth
Display Size
TV
Desktop
Laptop
Tablet
Cell Phone
Display Technology
5 x 7 x 3 = 105 versions for each video
Example
High Quality
Medium Quality
Low Quality
Network Bitrate
15
![Page 16: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/16.jpg)
Enable all devices to render best possible 3D videos
From mobiles to big TVs
Our Challenge
16
![Page 17: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/17.jpg)
Our Solution …
17
![Page 18: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/18.jpg)
18
Show the need for depth optimization - Subjective study on different displays
Anahita: complete system for 3D streaming- Supports most current displays/technologies - Provides efficient management of 3D versions- Enables personalized depth - Is scalable and dynamic (using DASH)
Method for depth expansion and compression- Optimizes depth: from mobile phones to big TVs - Preserves scene structure (important for sports)- Does not introduce visual artifacts- Is computationally inexpensive
Our Contributions
![Page 19: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/19.jpg)
19
Subjective study- 10 subjects- 2 displays (phone & 55” TV)- Series of short 3D clips (from soccer)- Show six different versions
• Original
• Depth compression
• Depth expansion (4 cases)
- Versions are shown in random to subjects- Subjects are asked which version is preferred
The Need for Depth Customization
![Page 20: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/20.jpg)
20
Best version is NOT the original one Depth customization depends on display size
Results
![Page 21: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/21.jpg)
21
SPSS: Structure Preserving Scene Shifting- Expands or compresses depth in stereo video- Simple image processing operations- Preserves lines and planes
- Suitable for field sports: soccer, football, tennis, …
Depth Customization
![Page 22: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/22.jpg)
22
Control disparity of pixels - increase/decrease depth
Two operations- Slant (affects vertical component of disparity) - Stretch (affects horizontal component of disparity)
SPSS: Basic Idea
![Page 23: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/23.jpg)
Depth Gradient
g = (gx, gy)23
![Page 24: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/24.jpg)
Depth Gradient
g = (gx, gy) 24
![Page 25: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/25.jpg)
SPSS: Slant
Left eye image Right eye image
25
![Page 26: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/26.jpg)
gy is changed
SPSS: Slant
26
![Page 27: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/27.jpg)
gx is changed
SPSS: Stretch
27
![Page 28: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/28.jpg)
28
Calculate gradient g = (gx, gy) of scene disparity
Compute slant and stretch factor
Remap left/right views using:
SPSS: Main Steps
![Page 29: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/29.jpg)
Before
After
SPSS: Sample Results
29
![Page 30: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/30.jpg)
Before
After
SPSS: Sample Results
30
![Page 31: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/31.jpg)
31
SPSS applies to scenes with planar depth long shots in field sports Our analysis of multiple full games
- 60—70% of the game
Close-ups and short shots do not benefit from SPSS
Anahita: has automatic shot classifier
SPSS: Coverage
![Page 32: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/32.jpg)
Anahita: System Architecture
32
![Page 33: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/33.jpg)
Anahita: 3D Version Manager
33
![Page 34: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/34.jpg)
34
Based on DASH Versions are organized in segments Most suitable 3D version is chosen for each client
Client can personalize: ask for more/less depth- Served using DASH
Adaptive Streamer
![Page 35: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/35.jpg)
35
Server - Deployed on Amazon- Implements many 3D video processing operations, and
our SPSS depth customization method
Clients- Mobile, tablet (mobile apps)- Stereo TV (web)- Desktop display (web)- Auto stereoscopic 55” display (custom app)
Implementation: Whole System
![Page 36: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/36.jpg)
36
5 displays- mobile, tablet, 15.6” laptop, 27” desktop, 55” TV
3 soccer 3D video clips from YouTube- Man United vs. Wigan (60 sec)- Chelsea vs. Wigan (24 sec)- Chelsea vs. Plymouth (20 sec)
15 subjects- viewed all clips (original & optimized) on all displays in
random- Ranked depth from 1-poor to 5-Excellent
Evaluation: Subjective Study
![Page 37: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/37.jpg)
37
Significant improvements in all cases- up to 35%
Sample Results
![Page 38: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/38.jpg)
38
3D streaming systems are more complex than 2D
Showed the need for depth adaptation
Proposed method to customize depth
Designed 3D streaming system
Evaluation study shows large gains
Conclusions
![Page 39: 1 Anahita: A System for 3D Video Streaming with Depth Customization K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda QCR](https://reader036.vdocuments.net/reader036/viewer/2022062320/56649cfa5503460f949cb991/html5/thumbnails/39.jpg)
Anahita: Meaning
• Ancient Persia: Anahita is source of all water, where warm and clear streams flow through golden channels to all lands and oceans on Earth
• Internet age: Anahita is source of all 3D videos, where high-quality streams of 3D videos flow through network channels to all types of displays