Joint edge-directed interpolation and adaptivesharpening filter

Rahul Vanam, Yan Ye, and Serhad DokenInterDigital Communications, Inc., 9710 Scranton Road, San Diego, CA 92121

E-mail: {rahul.vanam, yan.ye, serhad.doken}@interdigital.com

Abstract—In this paper, we present a joint filter that upsamplesand adaptively sharpens a video/image simultaneously. The jointfilter has been designed by combining a low complexity edge-directed interpolator with an adaptive Laplacian of Gaussian(LoG) filter. The joint filter yields sharp upsampled images,and is demonstrated to have lower computational complexityand similar visual quality when compared to using separateupsampling and sharpening filters.

Index Terms—post-processing, blurriness, video resize, sharp-ening filter, Sobel operator.

I. INTRODUCTION

With the advances of wireless networks and the increasingpopularity of video-capable mobile devices such as tablets andsmartphones, consumption of digital video content has beenincreasing rapidly. Video hosting websites such as YouTubeand Dailymotion often stores the same video content in a va-riety of bit rates and resolutions. When a user requests a certainvideo over wireless networks, usually the lower resolution andlower bit rate version is delivered due to bandwidth constraints.A commonly noticed artifact in these videos is blurriness.Further, if the encoded video has a resolution lower than thescreen resolution on the device, the video is usually upsampledbefore being displayed to fit the whole screen. Upsampling canfurther enhance the visibility of the blurriness in the video, andthe blurriness can be reduced using a sharpening filter. Sincemobile devices have power and computation MIPS constraints,low complexity upsampling and sharpening filters are neededto enable real-time video playback with good visual quality.

Much prior work in upsampling and sharpening alreadyexists. For upsampling, it has been shown that edge-directedinterpolation (EDI) reduces blurriness around object edgesin the upsampled image [1-8]. Li et al. performed EDI byusing the geometric duality principle for interpolating highresolution covariance from its low resolution counterpart [2].In [3], EDI is performed by modeling the interpolated imageas a Markov random field. Yang et al. present an explicitEDI scheme, where local gradient magnitudes and angles areused during interpolation [4]. A graphic processor unit friendlyEDI scheme was presented in [5]. In [6], the downsampledimage is first divided into diagonal and non-diagonal directionimages, each of which is interpolated in one of the threespecified directions using 1D polynomial interpolation. Zhangand Wu obtained two directional estimates of the missingpixel and fused them via linear minimum mean squared errorestimation [7]. Cha and Kim used a modified bilinear method

for interpolation, where the error is amended by using theinterpolation error theorem in an edge-adaptive fashion [8].Sharpening filters are used to enhance the edges in an image.Adaptive sharpening filters are often preferred since they donot sharpen noise and flat regions [9-11]. Kim et al. modifiedthe unsharp masking filter to use positive weights for edges,and negative weights for noise-like components [9]. Kotera etal. used a histogram of second-order gradients to determinethe strength of an edge, and then applied a suitable Gaussianderivative filter to sharpen the edges [10]. Kobayashi et al.proposed an approach to classify edges as long/short edgesfor adaptive sharpening [11].

Most of the prior work mentioned above design the up-sampling and sharpening filters separately. In this paper, wepropose a joint filter for simultaneous edge-directed interpo-lation and sharpening. The joint filter preserves the fidelity ofthe edges and enhances them in the upsampled frame, whilereducing the overall computational complexity.

The rest of this paper is organized as follows. Section IIdescribes the design of our joint filter. Results are presentedin Section III, and we conclude in Section IV.

II. SIMULTANEOUS EDGE-DIRECTED INTERPOLATION ANDSHARPENING FILTER

The framework of our joint filter is illustrated in Figure 1. Amodified Sobel operator is applied to the original image pixelsto obtain the local gradient angle and magnitude, which areused to adapt the joint filter for simultaneous EDI and adaptivesharpening.

In this section, we describe the modified Sobel operator.Since the joint filter has been designed by combining EDIand Laplacian of Gaussian (LoG) adaptive sharpening filter,we will first describe the two filters before describing the jointfilter.

Fig. 1. Proposed joint filter framework.

767978-1-4799-0248-4/13/$31.00 ©2013 IEEE GlobalSIP 2013

Fig. 2. Illustration of a pixel grid. Squares and circles represent original andinterpolated pixels, respectively. Labels ‘A’, ‘B’, and ‘C’ correspond to thecenter, row, and column interpolated pixels, respectively. The arrows point tothe original pixels used during interpolation.

A. Modified Sobel operator

The interpolated pixels are categorized into center, row, andcolumn pixels based on their location in the pixel grid, andthey are labeled as ‘A’, ‘B’, and ‘C’, respectively, in Figure2. Our modified Sobel operators operate on rectangular arrayof pixels for computing the horizontal and vertical gradients(∆x and ∆y) for the three pixel categories, and are definedas follows

GcenterX =

[−1 1−1 1

]; GcenterY =

[1 1−1 −1

]

GrowX =

−1 1−2 2−1 1

; GrowY =

1 10 0−1 −1

(1)

GcolX =

[−1 0 1−1 0 1

]; GcolY =

[1 2 1−1 −2 −1

]where suffixes X and Y , represent horizontal and verticaloperators, respectively. The gradient magnitude G, and angleθ are computed as follows

G = |∆x|+ |∆y|,θ = tan−1(∆y/∆x). (2)

B. Edge-directed interpolation

Our EDI has been designed for upsampling a video bytwo in both dimensions. The coefficients and direction ofinterpolation of our EDI are determined by the gradient angleand magnitude computed in Equation (2). The pixel to beinterpolated is classified as an edge pixel, if G ≥ T , where Tis a heuristically determined threshold. If the gradient G < T ,or if either ∆x or ∆y is zero, then interpolation is performedby averaging the nearest neighboring pixels as follows basedon the pixel category:

center pixel A′ = (b2 + b3 + c2 + c3)/4,

row pixels B′ = (b0 + b1)/2, andcolumn pixels C ′ = (a2 + b2)/2, (3)

Fig. 3. Illustration of a pixel grid, where bold entries are original pixels, andthe remaining entries are pixels to be interpolated. The gray region representsa 5× 5 window over which a LoG filter needs to be applied to sharpen C33.

where a2, b0, b1, b2, b3, c2, and c3 correspond to the originalpixels as illustrated in Figure 2. For G ≥ T , that is, what weconsider as edge pixels, we compute the edge angle as

µ = θ + 90o, (4)

since the gradient is perpendicular to the edge. Our interpola-tor, as shown in Equation (5), considers only three ranges ofedge angles for center pixel interpolation, and two ranges forrow and column pixel interpolation, respectively.

A =

12 (c2 + b3), 35

o < µ < 55o12 (b2 + c3), 125

o < µ < 145o

A′, else

B =

{16 (c0 + a1 + 2(b0 + b1)), 0

o < µ < 90o16 (a0 + c1 + 2(b0 + b1)), else

C =

{16 (b1 + a3 + 2(a2 + b2)), 0

o < µ < 90o16 (a1 + b3 + 2(a2 + b2)), else (5)

where A′ is defined in Equation (3). As explained earlier, weuse fewer angle ranges and fewer original pixels during inter-polation since we are primarily interested in a low complexityinterpolator for mobile devices.

C. Modified Laplacian of Gaussian (LoG) filter

The LoG filter of size K × L is defined as

LoGσ[x][y] = h(x, y, σ)− 1

KL

K′∑x=−K′

L′∑y=−L′

h(x, y, σ),

h(x, y, σ) =(x2 + y2 − 2σ2) exp(−x2+y2

2σ2 )

2πσ6∑K′

x=−K′∑L′

y=−L′ exp(−x2+y2

2σ2 ),(6)

where K ′ = K−12 , and L′ = L−1

2 .Figure 3 illustrates a pixel grid, where Cij is a pixel at

location (i,j). The 5 × 5 LoG filter operation for sharpeningpixel C33 is

Ch =

5∑i=1

5∑j=1

Cijhij , (7)

where hij = LoGσ[i− 3][j − 3].The LoG filter is made adaptive by varying σ based on

the gradient magnitude G computed by the modified Sobel

768

operator in Equation (2). The following function is used tomap G to σ:

σ(G) =

∞(no sharpening), G < 1001, 100 ≤ G < 2001.2, 200 ≤ G < 3001.4, G > 300.

(8)

Larger σ is used for larger gradients, to prevent over sharpen-ing which can cause strong edges to look like artifacts.

For brevity, the LoG filter and joint filter design are de-scribed for center pixels only; the design for row and columnpixels is similar. In Figure 3, the entries in bold representoriginal pixels and the remaining entries are pixels to beinterpolated, which we also refer to as “missing pixels” inthis paper. A question arises: how do we sharpen C33 if themissing pixels in Figure 3 have not been interpolated yet? Weanswer this question by using estimated values for the missingpixels when using the LoG filter. The center, row, and columnmissing pixels are estimated using Equation (3). Since the LoGfilter operation is now independent of EDI, we can combineEDI and LoG filter into a joint filter.

We would like to design a modified adaptive sharpening fil-ter g(G) that operates only on original pixels, and yet providesthe same high frequency component Ch in Equation (7). Thatis,

Ch =

3∑i=0

3∑j=0

C2i,2jgij , (9)

and

g(G) =

g00 . . . g03...

. . ....

g30 . . . g33

. (10)

The filter g(G) operates on a 4 × 4 window unlike theLoG filter. To illustrate the design of g(G), we derive thecoefficients g00 and g01. We first express each product term inEquation (7) in terms of the original pixels, using Equation (3)for the missing pixels, as follows:

C11h11 = 0.25(C00 + C02 + C20 + C22)h11

C12h12 = 0.5(C02 + C22)h12

C13h13 = 0.25(C02 + C04 + C22 + C24)h13

... (11)

From Equation (10), the sum of the coefficients correspondingto C00 and C02 are g00 and g01, respectively. Combining theterms in Equation (11), we get

g00 = 0.25h11

g01 = 0.25h11 + 0.5h12 + 0.25h13. (12)

Similarly, we derive the remaining coefficients of g(G).

D. Joint filter

As explained in Section II-C, since our modified LoG filteroperates independent of our EDI, the joint filter is obtainedby combining the two filters as follows

hjoint center(G,µ) = hcenter(G,µ)− g(G), (13)

where hcenter(G,µ) is the EDI for center pixels given inEquation (5), and represented as

hcenter(G,µ) =

0 0 0 00 0 0.5 00 0.5 0 00 0 0 0

,35o < µ < 55o,and G ≥ T

0 0 0 00 0.5 0 00 0 0.5 00 0 0 0

,125o < µ < 145o,and G ≥ T

0 0 0 00 0.25 0.25 00 0.25 0.25 00 0 0 0

, else,

(14)where T is the threshold described in Section II-B. There-fore, the joint filter internally combines the interpolated pixelwith its high frequency component to generate a sharpenedinterpolated pixel. For an original pixel, the joint filter usesthe original pixel itself instead of using an interpolator inEquation (13).

For each pixel category, we can predetermine the originalpixels in the 5× 5 window that would be operated on by thejoint filter. This gives us the joint filter size for different pixelcategories, and they are listed in Table 1. Since the joint filteroperates on original pixels only, its size is smaller than 5× 5.

TABLE IFILTER SIZES.

Pixel type EDI LoG filter Joint filterCenter 2× 2 5× 5 4× 4Row 3× 2 5× 5 3× 4

Column 2× 3 5× 5 4× 3Original – 5× 5 3× 3

III. RESULTS

We compare the complexity and visual quality of threemethods: EDI only, EDI followed by adaptive sharpeningcalled the two-step filter, and the proposed joint filter. Weuse the following test videos downsampled by two in bothdimensions: Legend of Guardians (240 × 102), Harry PotterClip1(240 × 100), Harry Potter Clip2 (240 × 102), Foreman(QCIF), Hall (QCIF), mobile (QCIF), and bus (QCIF), andconsider upsampling these videos by two in both dimensionstogether with adaptive sharpening. Sharpening is disabled forthe chroma components. Since all the filters share the same

769

EDI only Two-step filter Joint filter

Fig. 4. Visual quality examples. Left, middle, and right column images are obtained using EDI, two-step filter, and joint filter, respectively.

modified Sobel operator before filtering, we do not includeit in measuring the computational complexity. For an imageof size N × M , the number of multiplications for EDI =(2 × 2 + 3 × 2 + 2 × 3)NM = 16NM , using Table I.The number of multiplications for a separable LoG filteris 4 × 5 × 2N × 2M = 80NM . The total number ofmultiplications for luma for the two-step filter is 96NM , whilefor the joint filter it is (16 + 12 + 12 + 9)NM = 49NM .Therefore, our joint filter takes about 38.75% fewer multipli-cations compared to the two-step filter. Unlike the two-stepfilter, where original pixels are accessed twice, our joint filteronly accesses them once, thereby reducing the time taken formemory read operation.

We compute the PSNR-Y of the joint filtered output relativeto the two-step filtered output, since our goal is to achievesimilar quality as the two-step filter, while taking fewercomputations. The PSNR-Y of the joint filter averaged overall the test sequences is 41.6 dB, indicating that our joint filteryields filtered images close to the two-step filter. Furthermore,Figure 4 shows that the visual quality of the two filters arevery similar, and that they are both sharper compared to theEDI approach.

IV. CONCLUSION

In this paper, we present a joint filter that performs edge-directed interpolation and adaptive sharpening simultaneously.The joint filter reduces the number of multiplications by38.75% without loss in visual quality over the two-step filterapproach. In our future work, we will investigate designing ajoint filter using other edge-directed interpolation and sharp-ening filters, and support for upsampling factors other thantwo.

REFERENCES

[1] J. Allebach and P. Wong, “Edge-directed interpolation,” in Proc. IEEEICIP, vol. 3, pp. 707-710, Sep. 1996.

[2] X. Li and M. T. Orchard, “New edge directed interpolation,” IEEE Trans.Image Processing, vol. 10, no. 410, pp. 1521-1527, Oct 2001.

[3] M. Li and T. Q. Nguyen, ”Markov random field model-based edge-directed image interpolation,” IEEE Trans. Image Processing, vol. 17,no. 7, pp. 1121-1128, July 2008.

[4] S. Yang et al., “Fine edge-preserving technique for display devices,” IEEETrans. Cons. Electron., vol. 54, no. 4, pp.1761-69.

[5] M. Kraus, M. Eissele, and M. Strengert, “GPU-based edge-directed imageinterpolation,” in Proc. SCIA, pp. 532 - 541, 2007.

[6] D. D. Muresan, “Fast edge directed polynomial interpolation,” in Proc.IEEE ICIP, vol. 2, pp. 990 - 993, 2005.

[7] L.Zhang and X. Wu, “Image interpolation via directional filtering anddata fusion,” IEEE Trans. Image Process., vol. 15, no. 8, pp.2226-2238,Aug. 2006.

[8] Y. Cha and S. Kim, “The error-amended sharp edge (ease) scheme forimage zooming,” IEEE Trans. Image Process., vol. 16, no. 6, pp.1496-1505, Jun. 2007.

[9] S. Kim and J. P. Allebach, “Optimal unsharp mask for image sharpeningand noise removal,” J. Electron. Imag., vol. 15, 2005.

[10] H. Kotera and H. Wang, “Multiscale image sharpening adaptive to edgeprofile,” J. Electronic Imag., vol. 14, no. 1, 2005.

[11] T. Kobayashi and J. Tajima, “Content-Adaptive Automatic Image Sharp-ening,” in Proc. of the 2010 20th ICPR, pp. 2214-2217, 2010.

770