Hindawi Publishing CorporationEURASIP Journal on Advances in Signal ProcessingVolume 2007, Article ID 85963, 9 pagesdoi:10.1155/2007/85963

Research ArticleOrdinal Regression Based Subpixel Shift Estimation for VideoSuper-Resolution

Mithun Das Gupta,1 Shyamsundar Rajaram,1 Thomas S. Huang,1 and Nemanja Petrovic2

1Department of Electrical and Computer Engineering, University of Illinois, Urbana Champaign, IL 61801-2918, USA2Google Inc., 1440 Broadway, New York, NY 10018, USA

Received 2 October 2006; Accepted 3 May 2007

Recommended by Richard R. Schultz

We present a supervised learning-based approach for subpixel motion estimation which is then used to perform video super-resolution. The novelty of this work is the formulation of the problem of subpixel motion estimation in a ranking framework.The ranking formulation is a variant of classification and regression formulation, in which the ordering present in class labelsnamely, the shift between patches is explicitly taken into account. Finally, we demonstrate the applicability of our approach onsuperresolving synthetically generated images with global subpixel shifts and enhancing real video frames by accounting for bothlocal integer and subpixel shifts.

Copyright © 2007 Mithun Das Gupta et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

1. INTRODUCTION

Shift estimation between two or more frames from a videohas been of constant interest to researchers in computer vi-sion. Need for accurate shift estimation arises from manypractical situations. Applications, such as video frame reg-istration, resolution enhancement, super-resolution, andoptical-flow-based tracking, depend on reliable techniquesfor shift estimation for accuracy. Consequently, the accuracyof shift estimationmethods is of utmost importance for theseapplications. Since the Lucas-Kanade [1] algorithm was pro-posed in 1981, image alignment has become one of the mostsignificant contributions of computer vision. Applications ofLucas-Kanade image-alignment technique range from opti-cal flow, tracking, layered motion to mosaic construction,medical image registration, and face coding. The princi-pal idea of their technique was the introduction of image-gradients to infer the location of the target image patch in thesubsequent frames which were best matches based on somearbitrary similarity metric. Many researchers have come upwith refinements of their technique, to compute gradients insmarter ways, or to select the search region in smarter ways,but the principal idea has remained the same. Detailed re-views on motion estimation has been done by Aggarwal andNandhakumar [2], Mitiche and Bouthemy [3], and Nagel[4]. Three main approaches to motion estimation can be

identified as estimation based on spatial-gradient, image cor-relation, and regularization of spatiotemporal energy.

A closely related problem which has not yet receivedmuch focus in the literature is the problem of subpixel shiftestimation which is harder than estimating shifts with pixelaccuracy. The standard approach is to infer such shifts by in-terpolating to a higher resolution and then trying to estimatethe shifts. These methods work relatively well when the sub-pixel shifts are global or are similar for large portions of theimage. But if the shifts are varying drastically across small re-gions in the frames, then these techniques do not performwell. Patch-based techniques have an advantage, since thepatch size can be adjusted based on variance of pixel intensi-ties in a patch which can be a measure of the information ina patch. Most patch-based methods try to estimate the pixelshifts as well as subpixel shifts together, by using pyramidstructures. One inherent drawback of such methods is thatthe neighborhood continuity constraints need to be satisfiedat all levels of the pyramid. We try to answer a few of theseissues in this work. We use Lucas-Kanade shift estimators [1]to estimate the pixel shifts and align the frames up to a pixelaccuracy. We adopt a patch-based approach for subpixel shiftestimation and estimate the subpixel shifts using a learningbased framework upto a quarter pixel accuracy. Hence, eachpatch can be realigned up to a quarter pixel accuracy withoutaffecting the continuity with its neighbors.

2 EURASIP Journal on Advances in Signal Processing

The main contribution of this work is a learning-basedmethod for subpixel estimation which falls under the cat-egory of supervised learning problems where the attributesare given by novel regression coefficient features represent-ing two patches, and their corresponding label is the subpixelshift between them. Traditionally, the standard approach tosolve such supervised learning problems which correspondsto learning the function mapping between the attributes andthe label (subpixel shift) is to pose it as a multiclass classifica-tion problem or a regression problem. However, in our prob-lem setting, there is a certain ordering present in the class la-bels, namely, the fractional shifts, which will not be capturedby a classification or a regression approach. In this work,we exploit efficient learning algorithms that we proposed inour earlier work on ranking/ordinal regression [5] to per-form subpixel shift estimation. The contribution of our ear-lier work was a set of efficient algorithms which learn rankingfunctions efficiently while explicitly capturing the inherentordering present in the class labels. An elaborate descriptionof the ranking algorithms is provided later.

The area of video super-resolution has gained steady in-terest from researchers over the past few years. The prin-cipal idea of super-resolving a video is to use informationfrom temporal neighbors of a frame to help in generatingthe extra information needed for super-resolution. A cer-tain number of neighbors from the past as well as the fu-ture are warped relative to the current frame so that they arealigned with the current frame. The warped images are nowfused with the current frame to generate the super-resolvedframe. This method is repeated for all the frames of the video,to get a super-resolved video. The warping of neighboringframes needs accurate shift estimation. Once the temporalneighbors are warped and aligned with the current frame,then the frames need to be combined for resolution enhance-ment. Most widely used combining methods are simple av-eraging and median operations, due to their simplicity andspeed of implementation. Other sophisticated methods arementioned in [6–10]. Accurate estimation of image motionhas always been one of the most important bottlenecks ofthese techniques. In this work, we address the problem ofaccurate shift estimation by performing accurate estimationup to subpixel accuracy using the ranking/ordinal regressionframework.

Our proposed approach is based on using learning-basedmethods for subpixel motion estimation. The basic idea be-hind our approach is that, once two image patches have beenregistered with respect to each other while accounting for in-teger shifts, the problem that remains to be solved is the esti-mation of subpixel shifts in the x and y directions. The sub-pixel shifts can then be used to align the patches in a higherresolution [11, 12]. The subpixel estimation problem can beposed as a supervised learning problem in which the trainingdata consists of pairs of image patches which are fraction-ally shifted with respect to each other by a known amount.The objective of the learning algorithm is to learn a functionwhich learns the mapping from the features/attributes de-scribing pairs of image patches to the corresponding subpixelshift between them while minimizing a certain loss function.

During the testing phase, given an unseen patch pair, thelearned function is used to estimate the shift. The standardapproach for solving the above supervised learning problemis to learn a multiclass classifier for learning the mappingfrom features to subpixel shifts. However, as shown in ourearlier work [5], supervised learning problems in which thelabels have an ordinal characteristic have to be treated differ-ently by accounting for the ordering information present inthe labels. Such an approach leads to an interesting rankingformulation which is termed ordinal regression in classicalstatistics literature. In the next few sections, we formalize theabove ideas in a more general setting and develop algorithmsfor solving ranking problems.

The rest of the paper is organized as follows. Section 2introduces notation and provides a formal description of therankingmodel. In Section 3, we introduce the rankingmodelused in this work to pose the fractional shift estimation prob-lem into an ordinal regression problem. We provide a de-tailed analysis of the complexity of the ranking model com-pared to the classification model. Next, we describe efficientschemes for performing ranking using standard classifica-tion algorithms. Section 4 reviews the use of motion estima-tion for performing super-resolution and Section 5 describesour super-resolution approach by accounting for subpixelshifts estimated using the ranking framework. In Section 6,we present experimental results of our subpixel shift estima-tion approach for performing super-resolution.

2. NOTATIONS AND PROBLEMDEFINITION

Consider a training sample of size m, say S = {(x1, y1),(x2, y2), . . . , (xm, ym)}, xi ∈ X, yi ∈ Y, where X is the domainrepresenting the space of training examples andY is the spacefrom which labels are assigned to each example. We assumethat X is the n-dimensional space of reals Rn. Under this, forany xi, xj ∈ X we have xi − xj ∈ X.

For the ranking problem, Y = {1, . . . ,K} where K is themaximum rank that can be taken by any example. This issimilar to the multiclass classification problem. However, thespirit of the ranking problem is very different. The ranks re-late to the preference associated with an instance. Given anexample with label k, all the examples with rank less than kare ordered lesser and all the examples with rankmore than kare ordered higher. Such a relationship/viewpoint is not cap-tured in the case of multiclass classification framework. Ingeneral, we will assume that K (the maximum rank) is fixedfor a given problem.

3. THE RANKINGMODEL

In this work, we adopt a functional approach to solve theranking problem. Given a set of data points S, we learn aranker f : X → Y. We assume that there exists an axis insome space, such that when data is projected onto this axis, therelative position of the data points captures the model of userpreferences. In the ranking problem, we will treat f to be a lin-ear function f (xi) = hTxi, whose value is the signed distancefrom some arbitrary hyperplane given by h. The information

Mithun Das Gupta et al. 3

about the relative order of the data points will be capturedby the distance from the hyperplane. In addition to learn-ing h, we also learn (K − 1) thresholds corresponding to thedifferent ranks that are assigned to the data. The learned clas-sifier in this case is expressed as (h; θ1, θ2, . . . , θK−1) with thethresholds satisfying θ1 < θ2 < · · · < θK−1.

The ranking rule in this case is

f(xi) =

⎧⎪⎪⎨

⎪⎪⎩

1 if hTxi < θ1,

κ if θκ−1 < hTxi < θκ,K if θK−1 < hTxi.

(1)

Although, this model of ranking may seem too simplis-tic, as we show in the next few sections, it is quiet pow-erful and we give an analysis relating Vapnik-Chervonenkis(VC) dimension of the learned classifier to what we call rank-dimension of the data. We also show how one can extend theabove framework to the case where learning needs to be donein a space different from the original space. In such a case,learning is done in some high-dimensional space by makinguse of kernels for the mapping.

3.1. Complexity of ranking versus classification

It has been argued [13] that the ranking problem is muchharder than the classification problem. Although this is truein the particular view adopted by [13], in this paper wepresent an alternate viewpoint. We analyze the complexity ofthe ranking problem from the view of the VC dimension. Wedefine the variant of the VC dimension, called rank dimen-sion, for the ranking problem as follows: if the data points areranked with respect to the value of the functional evaluatedon a particular data point, then we say that the rank dimen-sion of the functional is the maximum number of points thatcan be ranked in any arbitrary way using this functional.

Theorem 1. The rank dimension of a linear functional is sameas its VC dimension. Following the notation given in Section 2,it holds for all xi, xj ∈ X , where xi − xj ∈ X and xj − xi ∈ X.

Proof. Let us consider the case of linear classifier h ∈ Rn. Sayone observes a set of m points S = {x1, x2, . . . , xm}, with thecorresponding ranks y1, y2, . . . , ym.

Clearly if we can rank a set of m points in any arbitraryway using a functional, then we can always shatter them (atthe cost of one additional dimension corresponding to thethreshold). Consider a subset S0 ⊂ S such that we want tolabel all the points that belong to S0 as negative and all thepoints that belong to S but not to S0 as positive (i.e., S \ S0).Now if we rank all the points in such a way so that the rankof all the points in S0 is less than the rank of all the pointsin S \ S0, then we can do the classification by just threshold-ing based on the rank. This shows that the rank dimension ofany functional cannot be more than the VC dimension of thesame functional. We know that the VC dimension of a linearclassifier in n-dimensional space is n + 1. That is any set ofn + 1 points (assuming general positions) in n-dimensionalspace can be shattered by an n-dimensional linear classifier.Now we show that any set of n+1 points can be ranked in any

arbitrary way using a linear classifier in n dimensional space.Given any arbitrary ranking of the points, let’s relabel thepoints such that rank(x1) < rank(x2) < · · · < rank(xn+1).Define a new set of points S0 = {0, x2−x1, x3−x2, . . . , xn+1−xn}. Now, if we label the points as {−1, 1, 1, . . . , 1}, the cardi-nality of set S0 is n+1 (n-difference vectors and one 0 vector.)Also it is easy to see that all points in S0 lie in Rn. Now fromthe VC dimension theory, we know that there exists a linearclassifier in n-dimensional space that can shatter S0 accord-ing to the labeling given above. Let this linear classifier be h,with classification as sign(hTx). Then for correct classifica-tion hT(xi − xi−1) > 0 ⇒ hTxi > hTxi−1. That indicates thatthe distance of the original points from the hyperplane doescorresponds to the specified ranking. Hence, we have shownthat any pair of n + 1 points can be ranked in any arbitraryfashion by an n-dimensional classifier and at the same timewe have also shown that the rank dimension cannot be morethan the VC dimension. This shows that the rank dimensionof any classifier is the same as its VC dimension.

This is a very interesting result as it shows that the com-plexity of the hypothesis space for the two problems is thesame. However, as of now, we are not clear about the re-lation between the growth function for the two problems.Further, the relation between the computational complexityof the two problems has to be studied. We present two ap-proaches to solve this problem. The first is referred to as thedifference space approach while the second is referred to asthe embedded space approach

3.2. Difference space approach

Given a training set S, define a new set Sd of difference vec-tors xdi j = xi − xj ; for all i, j : yi �= yj and their correspond-ing labels ydi j = sign(yi − yj). This leads to a dataset of sizeO(m2). Learning a linear classifier for this problem wouldbe the same as learning a ranker h. Once such a ranker islearned, the thresholds for the ranking problem can easily becomputed. This formulation is the same as the one proposedby [14]. Computational complexity of most learning algo-rithms, for example, naı̈ve Bayes, depend linearly on the sizeof the training data and a quadratic increase in the size ofthe data will certainly make most of the existing algorithmsimpractical. Hence, we propose to generate difference vectorsonly among the adjacent rank classes. Formally, given a train-ing set S, obtain a new set Sd made up of difference vectorsxdi j = xi − xj ; for all i, j : yi = yj + 1 and their correspond-ing labels ydi j = +1. This would result in a data-set with onlypositive examples. Again, most standard classification algo-rithms behave well if the number of positive examples is closeto the number of negative examples. To get around this prob-lem, once such a data set is obtained, multiply each examplexdi j and the corresponding label y

di j by qi j where qi j is a ran-

dom variable taking values {−1, 1} with equal probabilities.Clearly, learning a linear classifier over this data set will givea ranker h which will be the same as the one obtained in theprevious case. The size of the data set in this case isO(m2/K).For small K (which is the case in most K-ranking problems)

4 EURASIP Journal on Advances in Signal Processing

α1 α2 αK−2

θ1 θ2 θ3 θK−2 θK−1

Figure 1: Ranking projection on the real line.

this is still too large to handle. Next, we present an efficientapproach that specifically handles the ranking problem with-out exploding the size of the training data set.

3.3. Embedded space approach

In this section, we present a novel formulation that allowsone to map a ranking problem to a standard classificationproblem without increasing the size of the data set. The em-bedded space approach presented in this section is similarin spirit to the model presented in [15], however as we willsee shortly in our model, the dimension of the new spacedoes not grow linearly as the one presented in their paper.Figure 1 graphically depicts the ranking framework. The dis-tance from the hyperplane h of a data point xi is mapped toa one-dimensional space. In this space, θ1, . . . , θK−1 are thedifferent thresholds against which the distance is compared.Note that hTxi; for all xi having rank κ results in a range rep-resented by its left end point θκ−1 and its right end point θκ.Define ακ = θκ+1 − θκ; 1 ≤ κ ≤ K − 1. For the data itemsbelonging to rank 1, there is no lower bound and for all thedata items belonging to rankK there is no upper bound. Byconstruction, it is easy to see that ακ > 0; for all κ. Note thatdata point xi having rank κ > 1 will satisfy (assuming α0 = 0)

hTxi > θκ−1,

hTxi + ακ−1 > θκ,

hTxi +K−2∑

r=κ−1αr > θK−1.

(2)

Similarly, for an example xj with rank κ < K (assuming θK =inf),

hTxj < θκ,

hTxj + ακ < θκ+1,

hTxj +K−2∑

r=καr < θK−1.

(3)

Based on this observation, define h = [h,α1,α2, . . . ,αK−2]and for an example xj with rank 1 < κ < K , define x+j , x

−j

as n + K − 2 dimensional vectors with

x+j [l] =

⎧⎪⎪⎨

⎪⎪⎩

xj[l], 1 ≤ l ≤ n,0, n < l < n + κ− 1,1, n + κ− 1 ≤ l ≤ n + K − 2,

x−j [l] =

⎧⎪⎪⎨

⎪⎪⎩

xj[l], 1 ≤ l ≤ n,0, n < l < n + κ,

1, n + κ ≤ l ≤ n + K − 2.

(4)

For an example xj with rank κ = 1, we define only x−j asabove and for an example with rank κ = K , we define onlyx+j again as above. This formulation assumes that θK−1 = 0.It is easy to see that one can assume this without loss ofgenerality (by increasing the dimension of x by 1 one canget around this). Once we have defined x+j , x

−j , the ranking

problem simply reduces to learning a classifier h in n+K − 2dimensional space such that h

Tx+j > 0 and h

Tx−j < 0. This

is a standard classification problem with at most 2m trainingexamples, half of which have label +1 (examples x+j ) and therest have label −1 (examples x−j ). Even though, the overalldimension of the data points and the weight vector h is in-creased by K − 2, this representation limits the number oftraining data points to be O(m). Note that although we haveslightly increased the dimension by K −2, the number of pa-rameters that need to be learned is still the same (the classi-fier and the thresholds). Interestingly, any linear classificationmethod can now be used to solve this problem. It is easy toprove that if there exists a classifier that learns the above rulewith no error on the training data, then all the ακs are alwayspositive which is a requirement for the classification prob-lem to be the same as the ranking problem. Next, we showhow one can use kernel classifiers (SVM) to solve the rank-ing problem for data sets for which a linear ranker might notexist.

3.4. Kernel classifiers: SVM

In many real world problems, it may not be possible to comeup with a linear function that would be powerful enough tolearn the ranking of the data. In such a scenario, standardpractice is tomake use of kernels which allow nonlinearmap-ping of data. We will denote a kernel asK(·, ·) = φT(·)φ(·)which corresponds to using the nonlinear mapping φ(·) overthe original feature vector.

Solving ranking problems. For solving the ranking prob-lem, we have proposed the mapping given in Section 3.3, onehas to be careful in using kernel classifiers with this mapping.

To see this, note that if xj has rank κ, then hTx+j > 0⇒ hTxj >

θκ−1; hTx−j < 0 ⇒ hTxj > θκ but K(h, x+j ) = φT(h)φ(x+j ) >

0 � φT(h)φ(xj) > θκ−1. This is again because of the non-linearity of the mapping φ(·). However, one can again getaround this problem by defining a new kernel function. Fora kernel function K and the corresponding mapping φ(·),let us define a new kernel function K and with the corre-sponding mapping φ(·) as

φ(x) = [φ(x), x[n + 1 : n + K − 2]],φ(h) = [φ(h),h[n + 1 : n + K − 2]]. (5)

Note that, only the first n dimensions of x corresponding to xare projected to a higher dimensional space. The new kernelfunction can hence be decomposed into sum of two kernelfunctions where the first term is obtained by evaluation ofkernel over the first n dimensions of the vector and second

Mithun Das Gupta et al. 5

term is obtained by evaluating a linear kernel over the re-maining dimensions,

K(xi, x j

) =K(xi, xj)

+ xi[n + 1 : n + K − 2]Tx j[n + 1 : n + K − 2].(6)

However, when using the SVM algorithm with kernels, onehas to be careful while working in the embedded space.Learning algorithms typically minimize the norm ‖h‖ andnot ‖h‖ as should have been the case. In the next section, weintroduce the problem of ordinal regression and show howone can get around this problem.

3.5. Reduction to ordinal regression

In this section, we show how one can actually get around theproblem of minimizing ‖h‖ as against minimizing ‖h‖. Wewant to solve the following problem

min12‖h‖2, subject to y+/−j

(hTx+/−j + b

)> 0. (7)

The inequality in the above formulation for xj with rank yjcan be written as,

−b−n+k−2∑

l=n+1h(l)x−j (l) = θyj > hTxj

> −b −n+k−2∑

l=n+1h(l)x+j (l) = θyj−1.

(8)

In this analysis, we will assume that with respect to thresholdθκ’s, there is a margin of at least � such that for any data pointxj with corresponding rank yj , we have

θyj−1 + � < hTxj , 1 < yj ≤ K. (9)

Now, the problem given in (7) can be reframed as

min12‖h‖2, subject to hTxj < θyj , ∀1 ≤ yj < K

hTxj > θyj−1 + �; 1 < y j ≤ K.(10)

This leads to the following Lagrange formulation,

LP = 12‖h‖2 +

m−mK∑

j=1γ+j(hTxj − θyj

)

+m∑

j=m1+1γ−j(θyj−1 + � − hTxj

)−∑

j

γ+j −∑

j

γ−j ,

(11)

where mκ refers to number of elements having rank κ. Theranker h is obtained by minimizing the above cost functionunder the positivity constraints for γ+j and γ

−j . Dual formu-

lation Ld of the above problem can be obtained by followingthe steps as in [16],

Ld=−12n∑

i=1

n∑

j=1

(γ−i −γ+i

)(γ−j −γ+j

)K(xi, xj

)−∑

j

γ+j −∑

j

γ−j

(12)

with constraintsmκ∑

p=mκ−1+1γ+p =

mκ+1∑

p=mκ+1γ−p , ∀κ ∈ [2,K − 1]. (13)

We have introduced γ+l , γ−m ∀l ∈ [nK−1 + 1,nK ] and m ∈

[1,n] for simplicity of notation. It is interesting to note that(12) has the same form as a regression problem. The valueof θκ’s is obtained using Karush-Kuhn-Tucker (KKT) condi-tions.

4. SUPER-RESOLUTION USINGMOTION ESTIMATION

In this section, we elaborate on the standard technique [17]that is used for video super-resolution by accounting for mo-tion estimation. The outline of the super-resolution tech-nique is as follows.

(1) Bilinearly interpolate the frames to double their origi-nal size.

(2) Warp (t − n/2) to (t + n/2) frames onto the referenceframe t. This is one of the most important steps sinceit generates the extra information needed for super-resolving the frames. The quality of the super-resolvedframes depends on the accuracy of the image align-ment techniques. A review of the different methods ofperforming this step is given in [18, 19].

(3) Obtain a robust estimate for the tth frame. The wayin which the extra information is combined to pro-duce the frames plays an important role in determin-ing their quality. Simple techniques like averaging ormedian operations to more complex techniques likecovariance intersection can all be employed in thisstep. A few of the techniques for information fusionare [20–22].

(4) Iterate over all the frames (excluding the boundaryframes).

(5) Repeat steps 2–4 until the estimates converge.(6) Perform an optional deblurring operation on all the

frames.

In this work, we follow the guidelines introducing our ownalgorithmic modules at appropriate places. For image regis-tration and warping we use the hierarchical Lucas-Kanademethod [1] as used in the original work by Baker and Kanade[17].We use image patches of size 4×4 for all the experimentsreported in this work. This particular size was found to bea good tradeoff between the gain attained by encoding in-formation more than that contained by individual pixels andthe smoothness introduced due to the patch size. As shown insome of our experiments, the subpixel shifts are still presentafter integer alignment and it is at this juncture that we per-form our learning-based subpixel shift estimation algorithm.The details of our algorithm are provided in Section 5. Forobtaining the robust estimate of the tth frame, we note thatthere is a tradeoff between the simplicity of the technique andthe time taken for estimation. We employ simple techniqueslike mean or median and find that the mean works quitewell for the results reported in this work. As pointed in theoriginal work by Baker and Kanade [17], simple mean works

6 EURASIP Journal on Advances in Signal Processing

comparably against other complicated methods, and hencekeeping the huge amount of video data in view, we adopt thesimple mean as a method to combine the multiple frames.One point to note is that the 5th step in the algorithm whichis essentially iterating over the whole algorithm is avoided forspeed-up issues. All the results reported in this work are runjust for one iteration. Also the last step (6) has been avoidedin all our experiments. We take the liberty of omitting the laststep from all our results since this step is independent of thealgorithm used for the previous steps, and this work is prin-cipally focussed on performing the shift estimation and noton spatial domain deblurring techniques.

5. SUBPIXEL SHIFT ESTIMATION

Subpixel shift estimation involves identifying the shift in thex and y directions between two patches wherein, we assumethat they have already been aligned in such a way that thereare no integer shifts between them. Traditionally, this prob-lem has been solved using resolution pyramids in which thesubpixel shift problem is posed as an integer shift problemin higher resolution. However, such a technique is limitedby the interpolation algorithm used for increasing the res-olution. In this work, we adopt a learning strategy, namely,the ranking framework discussed earlier for estimating thesubpixel shifts without increasing resolution. We note thatthe notion of preference modeled by the ranking framework,corresponds to the subpixel shift between two patches, sayin the x direction. Consider three patches denoted by p1, p2,and p3 and let the shift between p2 and p1 be a quarter pixelshift and the shift between p3 and p1 be a half a pixel shiftin the x direction. The ranking framework accounts for theordering information, that is, p1 is closer to p2 than p3. Suchordering information is not captured if we use a multiclassclassifier. The estimation problem becomes unrealistic whenposed as a regression problem because it imposes a metric onthe ranker output.

5.1. Polar coordinates

The subpixel shift estimation problem involves estimatingtwo different rankers which capture the shifts in the x andy directions. However, we note that the two ranking prob-lems are interrelated and treating them independently resultsin bad empirical performance. Hence, we decouple the rela-tion present in the Euclidean setting to an extent by posingthe shift estimation problem in the polar domain which cor-responds to estimating the shift in the radial direction andangular direction. In the radial direction, the learning prob-lem falls under the category of the ranking formulation elab-orated in Section 1. However, estimating shifts in the angulardirection leads to a different formulation that we term cir-cular ordinality. Such a behavior arises naturally because ofthe equivalence of an angular shift of 0 and 2π. Modelingsuch behavior explicitly needs defining the notion of an an-gular margin introducing nonlinearities in the cost functionwhich is hard to optimize. We overcome the above problem,by a two step algorithm. The first step involves a classifier that

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Figure 2: Image super-resolution. For each class, clockwise fromtop left: (a), (e) original image, (b), (f) quarter pixel accuracy, (d),(h) integer-pixel accuracy, (c), (g) half-pixel accuracy.

identifies the angular shift between two patches as either be-ing in the upper half space (which corresponds to an angularshift of 0− π) or in the lower half space (which correspondsto an angular shift of π− 2π). The second step involves iden-tifying the angular shift in the relevant half space which canbe solved using the ranking framework. In this work, we dis-cretize the angular space uniformly into eight segments.

5.2. Regression coefficients as features

An important component of modeling is to identify/construct features/attributes that efficiently represent the in-put space in an informative way such that they aid in solv-ing the learning problem well. Another novelty of this workis a set of novel features for representing patch pairs. Con-sider two patches pi and p j , where we denote the pixels inthe patches as pik, and let there be P pixels in each patch.

Mithun Das Gupta et al. 7

(a) (b) (c) (d)

1401201008060402000

20

40

60

80

100

120

140

(e)

140120100806040200−20020

40

60

80

100

120

140

(f)

1401201008060402000

20

40

60

80

100

120

140

(g)

1401201008060402000

20

40

60

80

100

120

140

(h)

Figure 3: Top row: frames from the video. Bottom row: scaled flow vectors. Green: Lucas-Kanade. Red: our method.

We denote the set of adjacent neighbors of pixel location k asN (k). We exploit the nature of subpixel shifts to model everynonboundary pixel in the patch p jk, as a linear combinationof the pixels pil in the patch pi where l ∈ {N (k) ∪ k}. Theabove step results in a set of linear equations given by

p jk = wTi jpil, l ∈{N (k)∪ k}, (14)

where wij represents the weight vector and k indicates oneof the nonboundary pixels in the patch p j . Further, we notethat the weight vector is invariant of k. We solve the abovelinear regression problem in a least mean square error senseto obtain the regression coefficients vector wij . These regres-sion coefficients are used to represent the patch pair pi and p j .Higher order models can be used to model the patch depen-dencies. Potential candidates to replace the linear predictorscan be median filter-based predictors [23] or hybrid filters[24].

6. EXPERIMENTAL RESULTS

The experiments that we performed to demonstrate the ap-plicability of our approach can be broadly classified into twocategories. The first experiment involves estimating globalmotion for static images where the motion is generated ex-clusively by camera motion and the scene is assumed to befixed. We estimate the amount of subpixel shift within con-secutive frames and project the frames onto a higher resolu-tion grid by accounting for the subpixel shifts. The unknownpixels on the grid are then interpolated to generate high-resolution images. In the second experiment, we demon-strate the applicability of our approach on video frames

which have a moving foreground object and varying localmotion across different parts of the frame. We use subpixelalignment techniques to generate the super-resolved video aselaborated in Section 4.

6.1. Global subpixel motion

In this subsection, we investigate image super-resolutionwith global subpixel shift estimation in which we syntheti-cally generate the images by simulating camera shift againsta static background. We test our method for 3 different cat-egories of images, namely, cars, license plates, and humansubjects. The training data used to learn the radial direc-tion ranker and the classifier and ranker for angular direc-tion estimation includes a mix of images belonging to differ-ent categories from the Corel data set. The technique of es-timating subpixel shifts is then used to perform static imagesuper-resolution. We used the traditional spatial alignmentmethod, wherein multiple low-resolution images are alignedon a higher resolution grid. The unknown pixels remain-ing on the grid are generated by bicubic interpolation. Sincethe resolution improvement quality depends on the num-ber of grid locations that can be filled accurately, withoutinterpolation, subpixel accuracies perform significantly bet-ter than accounting for pure integer shifts. Also, the higherthe resolution of the subpixel estimation is the better theresults would be. We perform these experiments for the 3classes of images mentioned above. The results are shownin (Figure 2). Clearly, our method (quarter pixel accuracy)and half-pixel accuracy are better than integer pixel accuracy.Note the edges of the car bonnet or the edges of the digits inthe plate.

8 EURASIP Journal on Advances in Signal Processing

(a) (b) (c)

Figure 4: Left to right: (a) original scaled frame, (b) bicubic interpolation, (c) our method.

6.2. Local subpixel motion

In this subsection we report video super-resolution results byestimating subpixel shifts of face videos. All the video are shotwith a Canon SD 450 camera. The frames are downsampledto create the low-resolution inputs to our system.

In the first set of experiments, we verify the claim thatsubpixel shifts still remain intact even after the frames havebeen stabilized with respect to their temporal neighbors. Forthis experiment, we took a video of a walking person and sta-bilized the video using the algorithm proposed in [25, 26].We use feature tracking to obtain the flow vectors and thenuse our method to obtain the subpixel shifts in addition tothe flow vectors. We use Lucas-Kanade [1] to obtain the op-tical flow vectors. The results are shown in Figure 3.

The final set of results show sequence of frames froma face video. Each frame has been super-resolved usingthe approach elaborated in Section 4. We compare our re-sults against bicubic interpolation of the frames as shownin Figure 4. Note that we do not perform the addition de-blurring step which is commonly performed in other videosuper-resolution algorithms. The results clearly indicate theenhancement gain while performing motion and subpixelshift estimation.

7. CONCLUSION AND FUTUREWORK

We have presented a learning-based algorithm for estimatingsubpixel shifts in a patch based setting. The learning basedalgorithm falls under the class of ranking problems in whichthe ordering of the class labels is explicitly accounted forwhich results in better performance over standard classifica-tion and regression approaches. The ranking approach forsubpixel shift estimation is used to perform super-resolutionof images which have undergone a global subpixel shift andenhancement of video frames which have undergone bothinteger and subpixel shifts.

As mentioned earlier, higher order nonlinear models forpatch dependencies can be used to generate the features forthe determination of the azimuth angle. In the future, weplan to use the subpixel shift estimation approach for otherapplications, namely, motion tracking, layered motion, mo-

saic construction, medical image registration, and face cod-ing. In the local subpixel shift estimation scenario, furthersuper-resolution can be performed by aligning pixels in ahigher-dimensional grid.

ACKNOWLEDGMENT

This work was supported in part by Disruptive TechnologyOffice (DTO) under Contract NBCHC060160.

REFERENCES

[1] B. Lucas and T. Kanade, “An iterative image registration tech-nique with an application to stereo vision,” in Proceedingsof the 7th International Joint Conference on Artificial Intelli-gence (IJCAI ’81), pp. 674–679, Vancouver, BC, Canada, Au-gust 1981.

[2] J. K. Aggarwal and N. Nandhakumar, “On the computation ofmotion from sequences of images—a review,” Proceedings ofthe IEEE, vol. 76, no. 8, pp. 917–935, 1988.

[3] A. Mitiche and P. Bouthemy, “Computation and analysis ofimage motion: a synopsis of current problems and methods,”International Journal of Computer Vision, vol. 19, no. 1, pp. 29–55, 1996.

[4] H.-H. Nagel, “Image sequence evaluation: 30 years and stillgoing strong,” in Proceedings of the 15th International Confer-ence on Pattern Recognition (ICPR ’00), vol. 1, pp. 149–158,Barcelona, Spain, September 2000.

[5] S. Rajaram, A. Garg, X. S. Zhou, and T. S. Huang, “Clas-sification approach towards ranking and sorting problems,”in Proceedings of the 14th European Conference on MachineLearning (ECML ’03), pp. 301–312, Cavtat-Dubrovnik, Croa-tia, September 2003.

[6] M.-C. Chiang and T. Boult, “Efficient image warping andsuper-resolution,” in Proceedings of the 3rd Workshop on Appli-cations of Computer Vision (WACV ’96), pp. 56–61, Sarasota,Fla, USA, December 1996.

[7] F. Dellaert, S. Thrun, and C. Thorpe, “Jacobian images ofsuper-resolved texture maps for model-based motion estima-tion and tracking,” in Proceedings of the 4th IEEE Workshop onApplications of Computer Vision (WACV ’98), pp. 2–7, Prince-ton, NJ, USA, October 1998.

Mithun Das Gupta et al. 9

[8] M. Elad and A. Feuer, “Super-resolution restoration of an im-age sequence: adaptive filtering approach,” IEEE Transactionson Image Processing, vol. 8, no. 3, pp. 387–395, 1999.

[9] R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAPregistration and high-resolution image estimation using a se-quence of undersampled images,” IEEE Transactions on ImageProcessing, vol. 6, no. 12, pp. 1621–1633, 1997.

[10] A. J. Patti, M. I. Sezan, and A. M. Tekalp, “Super-resolutionvideo reconstruction with arbitrary sampling lattices andnonzero aperture time,” IEEE Transactions on Image Process-ing, vol. 6, no. 8, pp. 1064–1076, 1997.

[11] T. S. Huang and R. Tsai, “Multi-frame image restoration andregistration,” in Advances in Computer Vision and Image Pro-cessing, T. S. Huang, Ed., vol. 1, pp. 317–339, JAI Press, Green-wich, Conn, USA, 1984.

[12] S. P. Kim, N. K. Bose, and H. M. Valenzuela, “Recursive recon-struction of high resolution image from noisy undersampledmultiframes,” IEEE Transactions on Acoustics, Speech, and Sig-nal Processing, vol. 38, no. 6, pp. 1013–1027, 1990.

[13] W.W. Cohen, R. E. Schapire, and Y. Singer, “Learning to orderthings,” Journal of Artificial Intelligence Research, vol. 10, pp.243–270, 1999.

[14] R. Herbrich, T. Graepel, and K. Obermayer, “Large marginrank boundaries for ordinal regression,” in Advances in LargeMargin Classifiers, pp. 115–132, MIT Press, Cambridge, Mass,USA, 2000.

[15] D. Roth, S. Har-Paled, and D. Zimak, “Constraint classifica-tion: a new approach to multiclass classification,” in Proceed-ings of the 13th Interntional Conference on Algorithmic LearningTheory (ALT ’02), pp. 365–379, Lübeck, Germany, November2002.

[16] A. Smola and B. Schlkopf, “A tutorial on support vector re-gression,” Tech. Rep. NC2-TR-1998-030, Neural and Compu-tational Learning 2 (NeuroCOLT2), London, UK, 1998.

[17] S. Baker and T. Kanade, “Super-resolution optical flow,” Tech.Rep. CMU-RI-TR-99-36, Carnegie Mellon University, Pitts-burgh, Pa, USA, 1999.

[18] S. Baker and I. Matthews, “Lucas-kanade 20 years on: a uni-fying framework,” International Journal of Computer Vision,vol. 56, no. 3, pp. 221–255, 2004.

[19] G. Wolberg, Digital Image Warping, IEEE Computer SocietyPress, Los Alamitos, Calif, USA, 1992.

[20] X. Li, Y. Zhu, and C. Han, “Unified optimal linear estimationfusion—I: unified models and fusion rules,” in Proceedings ofthe 3rd International Conference on Information Fusion (FU-SION ’00), vol. 1, pp. 10–17, Paris, France, July 2000.

[21] S. Julier and J. Uhlmann, “A non-divergent estimation algo-rithm in the presence of unknown correlations,” in Proceedingsof the IEEE American Control Conference (ACC ’97), vol. 4, pp.2369–2373, Alberquerque, NM, USA, June 1997.

[22] D. Comaniciu, “Nonparametric information fusion for mo-tion estimation,” in Proceedings of the IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPR’03), vol. 1, pp. 59–66, Madison, Wis, USA, June 2003.

[23] T. C. Aysal and K. E. Barner, “Quadratic weighted median fil-ters for edge enhancement of noisy images,” IEEE Transactionson Image Processing, vol. 15, no. 11, pp. 3294–3310, 2006.

[24] T. C. Aysal and K. E. Barner, “Hybrid polynomial filters forGaussian and non-Gaussian noise environments,” IEEE Trans-actions on Signal Processing, vol. 54, no. 12, pp. 4644–4661,2006.

[25] N. Petrovic, N. Jojic, and T. S. Huang, “Hierarchical videoclustering,” in Proceedings of the 6th IEEE Workshop on Multi-

media Signal Processing (MMSP ’04), pp. 423–426, Siena, Italy,September 2004.

[26] N. Jojic, N. Petrovic, B. J. Frey, and T. S. Huang, “Transformedhidden Markov models: estimating mixture models of imagesand inferring spatial transformations in video sequences,” inProceedings of the IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR ’00), vol. 2, pp.26–33, Hilton Head, SC, USA, June 2000.

Mithun Das Gupta received his B.S. de-gree in instrumentation engineering fromIndian Institute of Technology, Kharag-pur in 2001, and his M.S. degree in elec-trical engineering from University of Illi-nois at Urbana Champaign in 2003. Heis currently pursuing his Ph.D. degree un-der the guidance of Professor Thomas S.Huang at the University of Illinois at Ur-bana Champaign. His research interests in-clude learning-based methods for image and video understandingand enhancement.

Shyamsundar Rajaram received the B.S.degree in electrical engineering from theUniversity of Madras, India, in 2000, andthe M.S. degree in electrical engineeringfrom the University of Illinois at Chicagoin 2002. He is currently working on hisPh.D. degree at University of Illinois at Ur-bana, Champaign under Professor ThomasS. Huang. He has published several papersin the field of machine learning and its ap-plications in signal processing, computer vision, information re-trieval, and other domains.

Thomas S. Huang received his B.S. degreein electrical engineering from National Tai-wan University, Taipei, Taiwan, China, andhis M.S. and Sc.D. degrees in electrical en-gineering from the Massachusetts Instituteof Technology, Cambridge, Mass. He was onthe faculty of the Department of ElectricalEngineering at MIT from 1963 to 1973, andon the faculty of the School of Electrical En-gineering and Director of its Laboratory forInformation and Signal Processing at Purdue University from 1973to 1980. In 1980, he joined the University of Illinois at Urbana-Champaign, where he is nowWilliam L. Everitt Distinguished Pro-fessor of Electrical and Computer Engineering, and Research Pro-fessor at the Coordinated Science Laboratory, and Head of the Im-age Formation and Processing Group at the Beckman Institute forAdvanced Science and Technology and Cochair of the Institute’smajor research theme of human-computer intelligent interaction.

Nemanja Petrovic received his Ph.D. de-gree from University of Illinois in 2004. Heis currently a member of research staff atGoogle Inc., New York. His professional in-terests are computer vision and machinelearning. Dr. Petrovic has published morethan 20 papers in the fields of graphi-cal models, video understanding, and dataclustering and image enhancement.

IntroductionNotations and problem definitionThe ranking modelComplexity of ranking versus classificationDifference space approachEmbedded space approachKernel classifiers: SVMReduction to ordinal regression

Super-resolution using Motion EstimationSubpixel Shift EstimationPolar coordinatesRegression coefficients as features

Experimental ResultsGlobal subpixel motionLocal subpixel motion

Conclusion and Future WorkAcknowledgmentREFERENCES