adap%ve(encoding(of(zoomable( videostreamsbasedonuser...
TRANSCRIPT
![Page 1: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/1.jpg)
Adap%ve Encoding of Zoomable Video Streams based on User
Access Pa;ern Ngo Quang Minh Khiem
Guntur Ravindra Wei Tsang Ooi
Na9onal University of Singapore
![Page 2: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/2.jpg)
Zoomable Video
![Page 3: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/3.jpg)
Zoomable Video with Bitstream Switching
![Page 4: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/4.jpg)
Server Client
(x,y,w,h)
GOAL: Minimize bandwidth to transmit RoIs
![Page 5: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/5.jpg)
Dynamic Cropping of ROI
Encode video once Support any RoI cropping
Tiled Streaming (TS)
Monolithic Streaming (MS)
![Page 6: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/6.jpg)
Tiled Streaming
One 9le = k x k macroblocks
Encode each 9le as independantly
decodable video streams
Tiles overlapping with the RoI are transmiPed
![Page 7: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/7.jpg)
Monolithic Streaming
Data outside RoI need for decoding RoI
Single monolithic video
![Page 8: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/8.jpg)
Trade-‐offs with TS and MS
TS
Bigger %le More waste More bits
Smaller %le Less compression More bits
![Page 9: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/9.jpg)
Longer MV More dependency More bits
Shorter MV Less compression More bits
MS
![Page 10: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/10.jpg)
RoI Access PaPern
Reduce bandwidth further, given RoI access sta%s%cs?
![Page 11: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/11.jpg)
Ques9ons in this paper
• Tiled Streaming Different 9le size in the same frame?
• Monolithic Streaming Different mo9on search range?
• How?
![Page 12: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/12.jpg)
Adap9ve Encoding
Given RoI access sta9s9cs, adapt the encoding parameters such that the expected bandwidth
E needed to transmit a RoI is minimized
∑∈
=Rr
rprcE )()(
c(r): compressed size of RoI r p(r): access probability of RoI r
![Page 13: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/13.jpg)
Log user selec?on of RoI
(Online)
Adap9ve Encoded Video
RoI Access PaCern
Encoded Video
![Page 14: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/14.jpg)
Adap%ve Encoding
Adap%ve Tiling (AT)
Monolithic Streaming with RoI-‐aware Coding
(MS-‐PB)
![Page 15: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/15.jpg)
Adap9ve Tiling
Given RoI access paPern, 9le the video such that E is minimized
∑∈
=Tt
tptcE )()(
c(t): compressed size of ?le t p(t): access probability of ?le t
![Page 16: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/16.jpg)
Intui9on
Allowing 9les of different sizes can reduce bandwidth
Regular %ling with 2x2 %les Adap%ve %ling
2
4
1
3
RoI accessed by most users
Merge ?les 1,2,3 and 4
![Page 17: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/17.jpg)
Greedy Heuris9c Tiling
• Start with regular 1x1 9les • Merge a 9le with its neighbors if expected bandwidth is reduced
• Merge newly-‐formed 9le with its neighbors bandwidth is reduced
![Page 18: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/18.jpg)
t1!c(t1) = 9!
p(t1) = 0.8!
t2!c(t2) = 6!
p(t2) = 0.8!
t12!c(t12) = 11!p(t12) = 1!
))c(tp(t ))c(tp(t ))c(tp(t 12122211 ≥+
![Page 19: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/19.jpg)
Resul%ng %le map
RoI Access Pa;ern
![Page 20: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/20.jpg)
Monolithic Streaming with RoI-‐aware Coding
• Referenced MBs form large region outside RoI
• Short mo9on vector: less bandwidth efficient • Probabilis9c boxing mo9on vector (MS-‐PB)
![Page 21: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/21.jpg)
Intui9on
• P(A) – P(AB) > P(B) Increase in size of A when sending R2 is marginal
• P(A) – P(AB) < P(B) Increase in size of A when sending R2 is higher
• [P(A)-‐P(AB)] S(A) > P(B) S(B)
P(A), P(B): sending A, B P(AB) : A and B in same RoI P(A) – P(AB): sending A independent of B
R2
R1
B A
![Page 22: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/22.jpg)
Mo%on Vector Spread aVer MS-‐PB
![Page 23: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/23.jpg)
Evalua9on
• Evaluate AT and MS-‐PB in terms of Bandwidth efficiency Compression efficiency
• Benchmark methods Per-‐RoI Tiled Streaming Monolithic Streaming
![Page 24: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/24.jpg)
Video Sequences
Rush-‐Hour (500 frames)
Bball (200 frames) Rainbow (350 frames)
Tractor (688 frames)
![Page 25: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/25.jpg)
Experiment Setup
• RoI size: 320x192 pel • Video resolu9on 1920x1080 pel • Evalua9on is conducted by a training-‐tes9ng framework Training and test sets have the same distribu9on
• One training and test set for each GoP
![Page 26: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/26.jpg)
0
0.5
1
1.5
2
2.5
3
3.5
4
Bball Rainbow
Expe
cted
Data Ra
te (M
bps)
Test Video
Expected Data Rate for Different Videos without B-‐Frames
PerRoI
MS-‐PB
MS
AT
TS4x4
![Page 27: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/27.jpg)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Bball Rainbow
Expe
cted
Data Ra
te (M
bps)
Test Video
Expected Data Rate for Different Videos with 2 B-‐Frames
PerRoI
MS-‐PB
MS
AT
TS4x4
![Page 28: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/28.jpg)
0
20
40
60
80
100
120
140
160
Bball Rainbow
File Size (M
B)
Test Video
Compressed Video File Size with 2 B-‐Frames
PerRoI
MS-‐PB
MS
AT
TS16x16
0
20
40
60
80
100
120
140
Bball Rainbow
File Size (M
B)
Test Video
Compressed Video File Size without B-‐Frames
PerRoI
MS-‐PB
MS
AT
TS16x16
![Page 29: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/29.jpg)
Presence of B-‐frame
Mo%on Vector Spread without B-‐frame
Mo%on Vector Spread with 2 B-‐frame
Without B-‐frame
MS-‐PB < MS With B-‐frame
MS-‐PB ≈ MS
![Page 30: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/30.jpg)
Conclusion & Future Work
• Propose an adap9ve encoding approach based on user access paPerns
• Reduce bandwidth by 21% (MS-‐PB) and 27% (AT) • Limi9ng mo9on vector is beneficial to zoomable video with wide spread of dependency
• Future work: Computa9onal complexity Diverse user interest of RoI Frequency of Adapta9on
![Page 31: Adap%ve(Encoding(of(Zoomable( VideoStreamsbasedonUser ...web.cs.wpi.edu/~claypool/mmsys-2011/Day3-1_AdaptiveEncoding.pdf · TestVideo(Compressed(Video(File(Size(with(2(BOFrames(PerRoI(MSOPB(MS(AT(TS16x16](https://reader033.vdocuments.net/reader033/viewer/2022042314/5f01aedd7e708231d4008b34/html5/thumbnails/31.jpg)
Thank you
• Ques9ons? • Feedback/Suggese9on?