1

Improved Seam Carving for Videofor Video Resizing

Michael Rubinstein MERL & The Interdisciplinary Center

Ariel ShamirThe Interdisciplinary Center

Shai AvidanAdobe Systems Inc.

Media Size Change

• Water Ski

(c) ariel shamir

2

Previous Work (Size Change)

• Cropping & Scaling:

– [Wang et al. 2004]

– [Fan et al. 2003]

– [Liu and Gleicher 2006]

– [Deselaers et a. 2008]

• Segment & Recombine

– [Tao et al. 2007]

?

(c) ariel shamir

• Non linear scaling (Warping)

– [Wolf et al. 2007]

• Time manipulations:

– [Pritch et al. 2008] – Object based video

– [Chen and Sen 2008] – Graph cut

Overview

1. Extending Seam Carving Operator

to Video

2. Defining a New Energy Measure

(c) ariel shamir

?[Wolf et al. 2007]

3

A Seam

(c) ariel shamir

Seam Carving

(c) ariel shamir

4

Finding the Seam?

(c) ariel shamir

The Optimal Seam

(c) ariel shamir

)(minarg* sEs

s =||||)( III yxE∂∂+

∂∂=

5

Pixel Attribute Dynamic Programming

( ))1,1(),,1(),1,1(min),(),( +−−−−+= jijijijiEji MMMM

5 8 12 3

9 2 3 9

( ))1,1(),,1(),1,1(min),(),( ++ jijijijiEji MMMM

(c) ariel shamir

7 3 4 2

4 5 7 8

Video?

• Naive… frame by frame independently y p y

Time

(c) ariel shamir

6

Jittery Results

(c) ariel shamir

Naïve (2): Global Projection

• Reduction of the video problem to image p gseam carving by using projection of maximum energy through time:

(c) ariel shamir

7

Naïve (2): Global Projection

• Reduction of the video problem to image p gseam carving by using projection of maximum energy through time:

(c) ariel shamir

Results

(c) ariel shamir

Rescale Retarget

8

Results

Rescale

(c) ariel shamir

Retarget

Problems?

• More complex scenes:– Object movement

– Camera movement

(c) ariel shamir

Global Seams

9

More Complex Scenes

• More complex scenes:– Object movement

– Camera movement

(c) ariel shamir

Global Seams

More Complex Scenes

• Seams should adapt and change through time!

(c) ariel shamir

10

Global Solution

(c) ariel shamir

Video Cube

One 3D Seam

(c) ariel shamir

11

What is the challenge?

• Dynamic Programming no longer works in 3D!• Video Cubes:

– Schodl et al. Siggraph 2000, Video Textures

– Kwatra et al. Siggraph 2003, Graph cut textures

– Rav-Acha et al CVPR 2005

(c) ariel shamir

Rav Acha et al. CVPR 2005, Video Mosaics

– Wang et al. Siggraph 2005, Interactive video cutout

– Chen and Sen EG 2008 (short papers), Video Carving

What is the challenge?

• Dynamic Programming no longer works in 3D!• Use Graph Cut:

– Schodl et al. Siggraph 2000, Video Textures

– Kwatra et al. Siggraph 2003, Graph cut textures

– Rav-Acha et al CVPR 2005

(c) ariel shamir

Rav Acha et al. CVPR 2005, Video Mosaics

– Wang et al. Siggraph 2005, Interactive video cutout

– Chen and Sen EG 2008 (short papers), Video Carving

12

What is the challenge?

• How to Define a Seam from a Cut?

(c) ariel shamir

Kwatra et al. Siggraph 2003, Graph cut textures

Constraints

1. Seams should be monotonic!i.e. one pixel in each row

(c) ariel shamir

13

Constraints

1. Seams should be monotonic!2. Seams should be connected!

(c) ariel shamir

ConnectedPiecewise

Piecewise vs. Connected

(c) ariel shamir

14

Standard Construction

pi,j pi,j+1∂y

∂x

∂y

pi,j pi,j+1

∂y ∂y

∂x

S T

(c) ariel shamir

pi+1,j pi+1,j+1

∂xpi+1,j pi+1,j+1∂x

New Construction

• This construction guarantees monotonic and connected seams

• This construction creates seams that are equivalent to the

pij pi,j+1∞

∞

∞

E1

(c) ariel shamir

equivalent to the dynamic programming approach

(Proofs in the paper)

pi+1,j pi+1,j+1∞

∞E1

15

3D Graph Construction

Frame t+2 Time

(c) ariel shamir

Frame t

Frame t+1

Video Cube

3D Graph Cut

(c) ariel shamir

Video Cube

16

Not Enough!

(c) ariel shamir

Also on Images!

(c) ariel shamir

17

Also on Images!

(c) ariel shamir

Changes in Image

(c) ariel shamir

18

Changes in Image

(c) ariel shamir

Changes in Image

(c) ariel shamir

19

Energy Inserted (not only removed)

(c) ariel shamir

Energy Reduced

Energy Increased

Changes in Energy

(c) ariel shamir

20

Tracking Inserted Energy

pi,j-1 pi,j pi,j+1

pi-1,j-1 pi-1,j

pi,j

pi-1,j+1pi-1,j-1

(c) ariel shamir

• Three possibilities When Removing Pixel Pi,j

Pixel Pi,j : Left Seam

pi,j-1 pi,j pi,j+1

pi-1,j-1 pi-1,j

pi,j

pi-1,j+1pi-1,j-1

(c) ariel shamir

21

Pixel Pi,j : Right Seam

pi,j+1pi,jpi,j-1

pi-1,j+1pi-1,j

pi,j

pi-1,j-1 pi-1,j+1

(c) ariel shamir

Pixel Pi,j : Vertical Seam

pi,j-1 pi,j pi,j+1

pi-1,jpi-1,j-1

pi,j

pi-1,j+1pi-1,j

(c) ariel shamir

22

Energy Function

E

(c) ariel shamir

New Forward Looking Energy Function

E

(c) ariel shamir

23

Backward (SIG 07)

(c) ariel shamir

Forward (SIG 08)

(c) ariel shamir

24

Backward

(c) ariel shamir

Forward

(c) ariel shamir

25

Backward

(c) ariel shamir

Forward

(c) ariel shamir

26

Backward Expand

(c) ariel shamir

Forward Expand

(c) ariel shamir

27

Graph Cut Formulation

pij pi,j+1∞

∞

∞-LU -LU+LU +LU

+LR

(c) ariel shamir

pi+1,j pi+1,j+1∞

∞+LR

Results

(c) ariel shamir

28

Results

(c) ariel shamir

Rescale Retarget

Results

Rescale

(c) ariel shamir

Retarget

29

Timing

• Typical video sequences have a resolution of yp q400 x 300 and 400 frames.

• 2 Seconds per seam using multi-resolutions of three levels (details in the paper)

• Total preprocessing time to enable between

(c) ariel shamir

50% to 150% change in aspect ratio takes around 10 minutes.

• Resizing can be performed in real time (multi size videos)

Multi-size Video DEMO?

(c) ariel shamir

30

Multi-size Video

(c) ariel shamir

Limitations

• Nature Scene

(c) ariel shamir

31

…Object Removal

(c) ariel shamir

Summary

1. Graph Cut2. Forward Energy

(c) ariel shamir

?

32

Contributions

• Extend the seam carving operator from images to video while maintaining its simplicity.

• Formulate using graph cut

• Allowing vertical and horizontal content aware size change and define multi-size video

(c) ariel shamir

• A novel forward energy for better content preservation with

Future

• Quality improvements (Energy, Smoothing)y p ( gy, g)

• Faster (recent GPU-based graph-cut optimizations)

• Use of multiple operatorsA t ti

(c) ariel shamir

• Automatic

• Other applications (domains?)…

33

(c) ariel shamir

Thank you!

XT Plane

(c) ariel shamir

Time

34

Monotonic: One Cut in Each Row

pi,j pi,j+1∞

∂y ∂y

∂x

S T

(c) ariel shamir

pi+1,j pi+1,j+1∞∂x

Defining a Seam from a Cut

1. Seams should be monotonic2. Seams should be connected!

(c) ariel shamir

35

Defining a Seam from a Cut

1. Seams should be monotonic2. Seams should be connected!

(c) ariel shamir