joint feature and texture coding: towards smart video representation via front-end...

1051-8215 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCSVT.2018.2873102, IEEETransactions on Circuits and Systems for Video Technology

1

Joint Feature and Texture Coding: Towards SmartVideo Representation via Front-end IntelligenceSiwei Ma, Member, IEEE, Xiang Zhang, Shiqi Wang, Member, IEEE, Xinfeng Zhang, Member, IEEE,

Chuanmin Jia and Shanshe Wang

Abstract—In this paper, we provide a systematical overviewand analysis on the joint feature and texture representationframework, which aims to smartly and coherently represent thevisual information with the front-end intelligence in the scenarioof video big data applications. In particular, we first demonstratethe advantages of the joint compression framework in terms ofboth reconstruction quality and analysis accuracy. Subsequently,the interactions between visual feature and texture in the jointcompression process are further illustrated. Finally, the futurejoint coding scheme by incorporating the deep learning featuresis envisioned, and future challenges towards seamless and u-nified joint compression are discussed. The joint compressionframework, which bridges the gap between visual analysis andsignal-level representation, is expected to contribute to a series ofapplications such as video surveillance and autonomous driving.

Index Terms—Video compression, feature compression, front-end intelligence

I. INTRODUCTION

RECENT years have witnessed an explosion of video bigdata, especially for surveillance videos [1] which are

becoming the biggest big data in the digital universe. In partic-ular, according to the prediction from NVIDIA [2], there willbe one billion cameras deployed in 2020 producing extremehigh-volume data in real-time. The large-scale visual data areof paramount significance for social security, smart city andintelligent manufacturing. However, it is usually impractical torely on manpower only for the utilization of the video big data,especially in the scenario of safeguarding which requires real-time monitoring and rapid response. Recently, the advancesof computer vision techniques have substantially promotedthe performance of visual analysis and enabled them to beapplied in practical application scenarios. As such, in thesecircumstances, efficient management of the large scale visualdata is highly desired.

This work was supported in part by the National Natural Science Foundationof China (61571017, 61632001), National Basic Research Program of China(973 Program, 2015CB351800), Top-Notch Young Talents Program of China,High-performance Computing Platform of Peking University, Hong KongRGC Early Career Scheme under Grant 9048122 (CityU 21211018), CityUniversity of Hong Kong under Grant 7200539/CS, China Scholarship Coun-cil No. 201706010248, which are gratefully acknowledged. (Correspondingauthor: Shanshe Wang)

S. Ma, X. Zhang, C. Jia and S. Wang are with the Institute of Digital Media,School of Electronics Engineering and Computer Science, Peking University,Beijing 100871, China.

S. Wang is with Department of Computer Science, City University of HongKong, Hong Kong, China.

X. Zhang is with Viterbi School of Engineering, University of SouthernCalifornia.

Along with the advances of hardware processing capabili-ties, the intelligence has been gradually enabled in front-endcameras. The emergence of intelligent front-end has broughtvideo big data management and representation tremendousopportunities. More specifically, not only the video com-pression performance can be improved from the increasedcomputational capabilities, but also the visual analysis taskscan be supported at the front-end. Under the constraint oflimited bandwidth, the lossy compression of the video da-ta leads to inevitable analysis performance degradation. Assuch, the exploration of the structural description with higher-level features from the pristine videos can adequately addressthese issues and enhance the availability of video big data.This motivates the feature compression, which has emergedas an active topic of great significance to both academiaand industry. In view of the importance of feature coding,in contrast to the traditional “compress-then-analyze” (CTA)framework as illustrated in Fig. 1, an alternative paradigm“analyze-then-compress” (ATC) was proposed in [3], [4]. Inparticular, ATC aims at extracting visual features at the front-end, which are subsequently compressed and delivered to theserver side for analysis purpose. Due to the fact that therequired bitrate in representing the feature is far less thanthe texture (here in this work, we use the “texture” as anindication of images or video frames), feature compressioncan greatly facilitate the simultaneous transmission of multiplevideo streams, especially in the scenarios of surveillance videocommunication.

Handcrafted features, which can be typically categorizedinto global and local feature descriptors, have been intensivelyinvestigated in the literatures. Global features usually serve fora quick search from large-scale datasets, but they may fail inapplications such as object matching and localization due tothe lack of local information. Local features, on the other hand,serve as useful tools for refining the search candidates chosenby global features, as local features have good invariance prop-erties to the camera motion, illumination variations and view-point altering. Local feature descriptors such as Scale-InvariantFeature Transform (SIFT) [5] and Speeded Up Robust Features(SURF) [6], were widely adopted in visual analysis tasks dueto the robustness and generalization capability in invariantrepresentations. To simplify them, algorithms were developedto generate the binarized local feature with high adaptability,high quality and low computational cost. Hence, the generationprocess of keypoints and feature descriptors, as well as thematching algorithms for binary descriptors were investigatedsuch as BRIEF [7], ORB [8], BRISK [9] and USB [10]. The



2

binarization of descriptors offers more flexibilities in the senseof computational complexity, such that real-time processingcan be feasibility achieved. Recently, numerous algorithmshave been proposed to compactly represent these handcraftedfeatures. For example, several algorithms have been proposedto compress SIFT feature using transform coding [11] as wellas vector quantization [12]. On the other hand, the globalimage descriptors statistically summarize the high level imageproperties, and the most prominent ones include Bag-of-Words(BoW) [13], Fisher Vector (FV) [14] and VLAD [15]. Forcompact global descriptor representation, efforts have alsobeen devoted to reducing the bits of image features, such astree-structure quantizer [16] for BoW histogram. RegardingVLAD, Chen et al. [17] specifically introduced ResidualEnhanced Visual Vector (REVV) by reducing the VLADdimensions with Linear Discriminative Analysis (LDA), whichwas performed followed by sign binarization. In summary,the major characteristics of the most popular image featuredescriptors are provided in Table I.

For video feature descriptors, the redundancies in tempo-ral domain can be further removed by exploiting statisticaldependencies. In [18] [19], both intra- and inter-feature cod-ing modes are designed to compress SIFT- and BRIEF-like[7] descriptors, and the best mode is selected based on anadvanced mode decision strategy. Towards mobile augmentedreality application, Markar et al. [20] introduced a temporallycoherent keypoint detector to perform inter-frame coding ofcanonical patches. Moreover, for global descriptors, Chen etal. [21], [22] proposed the inter-feature compression schemefor scalable residual based global signatures.

In view of the great importance of the compact featurerepresentation, to further enable the inter-operability with thestandardized bitstream syntax, MPEG has standardized the“Compact Descriptors for Visual Search” (CDVS) [23]–[26]for low bitrate representation of image feature descriptors. Inparticular, the CDVS standard adopts the block-based frequen-cy domain Laplacian of Gaussian interest point detector [27].For local descriptors, a compact SIFT compression schemewith transform followed by ternary scalar quantization isadopted [28]. The location coordinates of the local featuresare also compressed by representing them with a histogramconsisting of a binary histogram map and a histogram countsarray [29]. For global descriptors, CDVS adopts the scal-able compressed Fisher Vector (SCFV) [30] representationfor image retrieval tasks. In particular, the selected SIFTdescriptors are aggregated to the FV, and to compress thehigh dimensional FVs, a subset of Gaussian centroids fromthe Gaussian Mixture Models (GMM) are selected. For thetask of compact descriptors representation for videos, MPEGhas also started the standardization of Compact Descriptorsfor Video Analysis (CDVA) [31]–[33], aiming to provide theinter-operable design and standardize the bitstream to meet thegrowing demand of video analysis. It is also worth mentioningthat the deep learning based features have been adopted inCDVA, due to their superior performance in video analysistasks.

In this work, we systematically review and analyze the jointfeature and texture coding framework, including the design

Front-end Back-end

Network

Texture Stream

Texture

Encoder

Texture

Decoder

Remote monitoring

Analysis/Search

...

Feature

Extraction

(a) Compress-then-analyze

Front-end Back-end

NetworkFeature

Decoder

Feature

Encoder

Feature Stream

...Analysis/Search

Feature

Extraction

(b) Analyze-then-compress

Fig. 1. Two prominent paradigms for visual data management.

philosophy, advantages and future challenges. In particular, wewill start by a brief overview of the joint coding frameworkand demonstrate its superiority quantitatively and qualitatively.Subsequently, we show that the feature and texture are notnecessarily needed to be independently compressed, and theinteractions between them can further boost the joint com-pression performance. In particular, the compressed featuresare able to provide more accurate inter-frame predictions forvideo compression, thus the texture compression performancecan be significantly improved. Moreover, the retrieved similarimage contents from cloud by matching the decoded featurescan further be utilized in high quality frame restoration.Furthermore, due to the considerable number of deep learningalgorithms which have been repeatedly proved to achievesuperior performance in visual analysis tasks, the future workof joint deep learning feature and texture are also thoroughlydiscussed. It is also worth mentioning that in this paper, allthe experiments for handcrafted features are conducted withCPUs only, and for deep learning based features GPUs arealso adopted.

The remainder of this paper is organized as follows. InSection II, we present the joint feature and texture codingframework, and the advantages as well as superiority of thisframework are demonstrated. In Section III, the interactionsbetween feature and texture compression especially for theleverage of the features in improving the texture compressionperformance are discussed. The joint texture and deep learningbased feature compression is envisioned in Section IV and thispaper is concluded in Section V.

II. JOINT COMPRESSION OF FEATURE AND TEXTURE

In this section, we introduce the joint compression archi-tecture of feature and texture, which is a hybrid frameworkthat incorporates the advantages of both the CTA and ATCschemes. As shown in Fig. 1, CTA attempts to compress thetextures acquired at the front-end, the bitstream of which issubsequently transmitted to the server-end for feature extrac-tion, analysis and viewing [34], [35]. By contrast, ATC firstanalyzes the captured image/video by extracting visual features



3

TABLE ISUMMARY OF POPULAR IMAGE FEATURE DESCRIPTORS, WHERE THEIR MAJOR CHARACTERISTICS INCLUDING THE DATA TYPE AND FEATURE

DIMENSION ARE COMPARED.

Feature Descriptors Data Type Dimension Characteristics

Local features

SIFT [5] Float 128 Gradient based

SURF [6] Float 64/128 Integral-image based

BRIEF [7] Binary 128/256/512 Intensity difference tests

ORB [8] Binary 256 Based on BRIEF

BRISK [9] Binary 512 Configurable circular sampling pattern

USB [10] Binary 64 Highly discriminative & fast

Global features

BoW [13] Float

N*

SIFT feature aggregation

FV [14] Float PCA & binary features

VLAD [15] Float Intensity difference tests

REVV [17] Float Sign binarization & LDA

*It should be noted that the dimensions of global features are application-dependent.

at the front-end, which are conveyed to the server side enablinghighly light-weight and efficient visual analysis.

Therefore, both CTA and ATC have their own advantagesin specific application domains. For example, in the scenarioof visual analysis, ATC is superior to CTA in several as-pects. First, features are usually much more compact thantextures, making the transmission more efficient. As such, inthe scenario of low bitrate transmission, the ATC approachis the preferable solution as there exists a lower bound forthe CTA paradigm. For example, as analyzed in [4], thesource rate is required to be higher than 17.3 kbyte/queryfor Oxford data set [36] and 2.4 kbyte/query for ZuBuddata set [37]. Second, performing analysis with the featuresfrom the quality-degraded texture would significantly degradethe analysis performance [38], [39], and ATC overcomesthis by extracting the features from the pristine texture. Weconducted quantitative comparison between CTA and ATCparadigms in terms of the rate and analysis accuracy. As ademonstration, the experiments are based on the task of faceverification. For the CTA paradigm, the state-of-the-art videocoding standard High-Efficiency Video Coding (HEVC) [40]is adopted to compress the facial images with the all intra(AI) configuration, where the quantization parameters (QP )are set to be {22, 27, 32, 37, 42, 47, 51} for a wide range ofbitrate. Then the two deep network models in [41], [42] areutilized to extract the features for face verification. As for theATC paradigm, the deep network model firstly applies on theuncompressed images to obtain raw deep features. Then, theextracted deep features are scaled and quantized into integersand then entropy coded. The quantization can be formulatedas follows,

Qstep =2

QP−46

s= 2

QP−46 −10, C = floor

(C

Qstep

), (1)

where s indicates the scaling factor (equaling to 210), C andC denote feature coefficients before and after quantization,respectively. All the quantized deep feature coefficients com-pose of the feature vector for the verification task. In Fig. 2(a),we use Facenet model [41] as feature extractor to obtainthe 128-dimension face feature vector. The performance of

(a) (b)

Fig. 2. Face verification performance comparison between ATC and CTA.(a) Facenet [41] on LFW dataset [43]; (b) Deepid [42] on Youtube FacesDatabase [44].

two paradigms with face verification task on LFW datasetis illustrated, and obviously ATC outperforms CTA with aclear margin especially under low bitrate circumstances. Inaddition, similar phenomena can also be observed in Fig. 2(b)when evaluating on the Youtube Faces Database with Deepidmodel [42] to extract the 160-dimension face feature vector.The CTA paradigm shows significant performance drop whenthe bitrate is relatively low. However, the ATC paradigm couldachieve comparable performance using much less bandwidth.Finally, ATC does not require the decoding and feature ex-traction of texture at the sever end, which enables the energy-efficient multimedia systems. However, ATC also has its ownlimitations, especially when the video texture is required tobe further viewed or monitored at the server end, or multipletypes of features are needed to be extracted simultaneously.

Therefore, the joint feature and texture representation(JFTR) has been proposed to address these problems, and thearchitecture is illustrated in Fig. 3. In a nutshell, JFTR is asuperset of ATC and CTA that compresses and transmits bothtextures and features [45]–[48]. In analogous to ATC, JFTRextracts and compresses the features at the front-end, suchthat the features extracted from the pristine image/video canbe conveyed and utilized. Moreover, instead of transmittingthe compact features only, the compressed texture is alsorequired to be transmitted such that the receiver receives thehybrid texture-plus-feature bitstream. As a hybrid scheme,JFTR overcomes the weaknesses of ATC and CTA by taking



4

Intelligent Front-end Back-end

Network

TextureEncoder

FeatureDecoder

TextureDecoder

FeatureEncoder

Interactive StreamTexture + Feature

Remote monitoring

Tracking/Re-ID

...

FeatureExtractor

Recognition/Detection

Search/Analysis

Fig. 3. JFTR paradigm for visual search/analysis applications.

advantages of their strengths. In general, ATC and CTA canbe regarded as two extreme cases of JFTR. Along withthe development of front-end intelligence, in the applicationscenarios that both analysis and monitoring are required at theserver end, the superiority of JFTR makes it to be a desiredand promising strategy in the image/video data management.

First, JFTR tackles one major problem of CTA that thevisual analysis performance will be degraded when featuresare extracted from the distorted textures with strong compres-sion artifacts. To demonstrate how the texture quality wouldinfluence the analysis performance, we conduct extensiveexperiments on various analysis-based applications includingobject matching, object localization and visual retrieval, andthe experimental results are shown in Figs. 4&5. To generatetextures with varying compression artifacts, the HEVC refer-ence software is utilized for encoding the videos in datasetswith different quantization parameters (i.e., QP ). Generallyspeaking, a larger value of QP yields stronger compressionartifacts [49]–[51]. Here, we utilize the mobile augmentedreality (MAR) dataset [52] for experiments of object matchingand localization, where two data categories Moving and Videoare used. The Moving category contains videos with cameramotion, glare, blur, zoom, rotation and perspective changes,while the Video category contains videos with both movingcamera and moving objects. For the task of object matching,the performance is evaluated by the number of matched featurepairs between the query (compressed video frame containingthe object of interest) and the target (ground-truth imageof the object of interest). The object localization accuracyis assessed according to the Jaccard index [53]. As for thevisual retrieval task, the Rome landmark dataset (RLD) [54] isutilized, containing 10 Rome landmark videos as queries, andeach query has a number of target images while the remaining10K images serve as distracters. The retrieval performance isevaluated by the precision at rank k and the mean average pre-cision (mAP) indices. In Figs. 4&5, the variations of analysisperformance in terms of the quantization parameter (QP ) areshown. It can be clearly observed that the analysis performanceis significantly degraded at lower bitrate conditions, indicatingthat under bandwidth-limited circumstances CTA approachwhich transmits the low quality texture is inadequate andunreliable for the purpose of intelligent analysis. For JFTR,

20 25 30 35 40 45 50

QP

10

15

20

25

30

35

40

45

# M

atch

ed P

airs

MovingVideo

(a) # matching pairs v.s. QP

20 25 30 35 40 45 50

QP

0.75

0.8

0.85

0.9

0.95

1

Loc

aliz

atio

n A

ccur

acy

MovingVideo

(b) Localization accuracy v.s. QP

Fig. 4. Object matching and localization performances as a function of QP .Two video categories Moving and Video in MAR dataset [52] are tested.

the features are extracted from pristine textures, such thatthe representations of textures can be well preserved andthe corresponding high quality features can be extracted foranalysis-based applications, leading to significantly improvedaccuracy.

Second, comparing with CTA, JFTR is more efficient andenergy-friendly at the server-end for visual analysis. In [4],the computation and energy complexities of ATC and CTAhave been thoroughly compared and analyzed, and it hasbeen shown that CTA is more suitable for deploying at front-end with tight energy constraint, since performing featureextraction could consume 2-5 times of energy than performingimage encoding. By contrast, at the server end, ATC paradigmis more energy-saving than CTA, as CTA has shifted thecomplexity of feature extraction from front-end to the serverend. With the rapid development regarding the computationalcapability of the front-end cameras, performing feature extrac-tion and compact representation at the front-end is promisingand practical. In this sense, JFTR can dramatically reducethe burden of the server and thus improve the quality ofexperience (QoE) with an instant response. This could becrucial for real-time applications at a resource-constrainedplatform where a server may receive and process millions ofrequests simultaneously.

Third, comparing with ATC, JFTR provides extra textureinformation for human-involved applications such as videomonitoring and viewing. ATC attempts to convey compactrepresentation in terms of features, such that significant bitrate



5

20 25 30 35 40 45 50

QP

0.75

0.8

0.85

0.9

0.95

1P

reci

sion

at r

ank

1

N=50N=100N=150N=200

(a) Precision at rank 1 v.s. QP

20 25 30 35 40 45 50

QP

0.5

0.6

0.7

0.8

0.9

Pre

cisi

on a

t ran

k 5

N=50N=100N=150N=200

(b) Precision at rank 5 v.s. QP

20 25 30 35 40 45 50

QP

0.4

0.45

0.5

0.55

0.6

Pre

cisi

on a

t ran

k 10

N=50N=100N=150N=200

(c) Precision at rank 10 v.s. QP

20 25 30 35 40 45 50

QP

0.75

0.8

0.85

0.9

0.95

1M

ean

Ave

rage

Pre

cisi

on

N=50N=100N=150N=200

(d) mAP v.s. QP

Fig. 5. Retrieval performance as a function of QP . The RLD dataset [54]is used for evaluation. The performance is evaluated with varying maximumnumber of extracted feature descriptors (i.e., N ).

reductions can be achieved with the analysis performance be-ing maintained compared with CTA. Nevertheless, the textureinformation is discarded and ignored. For certain utility thatthe pixel level information is further required, the texture isof vital importance and should be stored at the server side.As such, the JFTR scheme which takes advantages of bothATC and CTA, could be optimized via joint optimization tosimultaneously preserve the texture quality and achieve highefficiency analysis. In other words, the JFTR can achievesimilar analysis performance with ATC while maintainingcomparable visual quality with CTA, at the expense of con-veying both feature and texture bitstream.

In summary, JFTR benefits from both texture and featurecompression and enriches smart video representation via front-end intelligence. Though both feature and texture bitstreamsare required to be transmitted, comparing with textures, thefeature data are usually much more compact. We compare thebitrate of texture and feature under four different operationpoints (OP), where the four OPs range from high bitrate (OP1)to low bitrate (OP4). We report the results on videos andimages in Table II and Table III, respectively. For videos, twocategories in the MAR dataset [52] are used for evaluation,and the SIFT features are extracted from each video frameand compressed by the method in [48]. Regarding images, theLFW dataset [43] is used and the Facenet [41] is utilized forextracting features. Different QP values are applied to textureand features for different OPs, and for videos the maximumnumber of extracted features also varies for different OPs.From Table II, one can observe that the feature bitstream onlyoccupies a minority part in the total bitstream, and the ratio offeature bitrate to texture bitrate ranges from 1% to 7% fromhigh bitrate to low bitrate. The ratio in image tests as shown inTable III is relatively higher but still less than 15%. Moreover,

TABLE IIIBITRATE COMPARISON BETWEEN TEXTURE AND FEATURE FOR IMAGES IN

LFW DATASET [43].

Texture Bitrate (bpp) Feature Bitrate (bpp) F/T Ratio

OP1 0.4998 0.0229 4.60%

OP2 0.1999 0.0150 7.50%

OP3 0.0807 0.0094 11.60%

OP4 0.0367 0.0054 14.70%

the encoded features can further facilitate the texture codingby investigating their interactions [47], [48], [55]. It is alsoworth noting that the bitrate consumption is highly relevant tothe transmission, such that the costs for feature transmissionwould be relatively lower than texture.

III. FEATURE BASED TEXTURE COMPRESSION

In general, the feature and texture can be compressedindividually, or the interactions between them can be fur-ther exploited to improve the overall coding performance. Inthis section, we introduce several techniques that enable theefficient compression of video texture based on the featurerepresentation. The principle behind these techniques is thatthe feature and texture are redundant to a certain degree, inthe sense that features aim to provide informative descriptionsof the distinctive characteristics of textures. Moreover, suchframework also establishes an interactive work flow for thedecoding and utilization of the features and textures. Morespecifically, the features which are highly compact and fre-quently visited for retrieval and analysis, can be convenientlyaccessed without the reference of the texture. By contrast, thetexture which occupies a high proportion of the total bitrate,requires the support of the feature information for decoding,such that better coding efficiency can also be ensured.

A. Feature based Motion Information Derivation

Local feature descriptors, which are required to be highlydistinctive and invariant towards camera motion, illuminationvariations and viewpoint altering, can provide more accuratemotion predictions among video frames than traditional block-based motion compensation [56]. Previous works [57], [58]have been proposed to utilize local features to derive globalmotion matrix, which is transmitted to the decoder side forimproving inter-frame prediction efficiency. With the JFTRscheme which incorporates features with textures, the globalmotion information is not required to be conveyed since theyare already implied in the compact feature representation.More importantly, the transmitted features can describe moreflexible local motions such that the coding efficiency can befurther improved.

As presented in [48], the local features extracted from thepristine video sequences are used to facilitate video compres-sion. At the decoder side, the distinctive visual features arefirst obtained before video decoding, such that affine motioncompensation in inter coding can be feasibly supported. Inparticular, each coding block is assumed to be affine trans-formed from a patch in a reference frame, and the affine



6

TABLE IIBITRATE COMPARISON BETWEEN TEXTURE AND FEATURE FOR VIDEOS IN MAR DATASET [52].

Moving Category Video Category

Texture Bitrate (Kbps) Feature Bitrate (Kbps) F/T Ratio Texture Bitrate (Kbps) Feature Bitrate (Kbps) F/T Ratio

OP1 1870.68 21.64 1.16% 3213.96 22.02 0.69%

OP2 572.89 17.15 2.99% 1057.29 17.63 1.67%

OP3 226.21 12.22 5.40% 427.32 12.75 2.98%

OP4 102.05 6.81 6.67% 194.43 7.21 3.71%

transform can describe the motion with translation, rotationand scaling. Instead of transmitting the affine transform matrix,the matrix can be calculated simultaneously at encoder anddecoder sides if there exists matching feature pairs betweenthe current coding block and a reference frame. As shown inFig. 6, the feature matching is performed between the currentcoding block and a corresponding reference frame by ratio testgiven the decoded features. Then the initial transform matrixT0 ∈ R3×3 can be obtained with RANSAC algorithm [59]by excluding the outlier matchings. For more accurate motionprediction, T0 is further refined by minimizing the predictionerrors as follows,

T = argminT

{∑i

[I ′(x′i, y′i)− I(xi, yi)]

}

s.t.

x′iy′i1

= T

xiyi1

, (2)

where I(xi, yi) and I ′(x′i, y′i) are pixels in the current coding

block and the corresponding reference frame after affine trans-formation, respectively. This can be solved by gradient descentmethod, and the differences between initial affine matrix andthe refined one (T − T0) can be transmitted.

Accordingly, the transformed prediction block can be ob-tained by applying the derived affine matrix to each pixelof current block, and the residual block by subtracting thepredicted block from the original block can be representedin the Discrete Cosine Transform (DCT) domain with lessnon-zero coefficients, leading to better coding efficiency. Theexperimental results in [48] have shown significant improve-ments on video coding efficiency by utilizing the featurematching based inter-frame predictions. The improvements areeven more promising especially for video sequences with com-plex motions, e.g., the videos captured by a moving mobilecamera, which contain extensive rotation and scaling motionsthat conventional translation based block motion cannot effi-ciently handle. For particular scenario such as mobile capturedvideos, it has been demonstrated that over 18% bitrate can besaved when applying texture plus feature joint compressioncompared to the texture compression only. Another benefitshown in [48] is that efficient and accurate visual retrieval canbe achieved simultaneously.

An alternative idea is utilizing the derived motion infor-mation from features as additional motion vector predictors[60]. In a similar way, the transmitted features can be usedfor inter-frame prediction by feature matching. Specifically,

T

PredictionBlock

Current Block

Residual Block

Reference Frame

Fig. 6. Affine motion compensation based on feature matching.

for each inter-prediction block, one motion vector can beattained by feature matching between the current block andreference frame. The derived motion vector might be a goodpredictor of motion vector for current block, therefore it isthen added into the motion vector prediction candidates andmerge candidates. Spatial and temporal extensions are appliedif there is no feature point in current block, where the motionvector can be inherited from larger blocks in the current frameor co-located blocks in neighboring frames. It has been shownthat this scheme can bring over 1% bitrate savings, indicatingthe potential of features in providing more accurate motioninformation for video coding.

B. Cloud based Image and Video Compression

One distinctive property of feature descriptors is that thevisually similar patterns and textures can be effectively re-trieved based on feature matching. As such, in the scenarioof cloud computing that images and videos are stored in thecloud side for further processing and utilizing, the retrievedsimilar images and videos can play an important role in theredundancy removal to further improve the texture codingperformance. In the literature, there are a series of worksaiming to improve the quality of reconstructed texture andremove the redundancy within the texture based on hand-crafted features [61]–[68], which validate that the featurebased image and video compression is a promising directionfor further investigation.



7

Recently, cloud based image compression has receivedsufficient interests [64], [69]. In [64], the cloud based imagecompression framework was proposed. In particular, the imageis described as the corresponding down-sampled version andlocal feature descriptors such as SIFT. The down-sampled im-age is then compressed with the traditional image/video codec.Moreover, as the down-sampled version can already providethe rough feature descriptor information, only the residualsbetween the original SIFT and predicted SIFT from the down-sampled decompressed image are required to be transmitted.As the SIFT residuals are generally sparse and compact, a lightversion of the image can be obtained and transmitted to thecloud side, achieving much higher compression ratio comparedto the traditional image compression standards. In the cloudside, the reconstructed SIFT descriptors can be applied toobtain the correlated and visually similar images in the cloud.In particular, the correlated information can be obtained at thepatch level, such that image stitching can be performed withthe high quality retrieved patch. The patch stitching is guidedbased on the distortion between the up-sampled patch fromthe down-sampled version transmitted to the cloud and thetransformed version from the retrieved high quality patch. Dueto the external reference information provided, much bettercompression performance can be achieved when the retrievedcorrelated image can be obtained. However, one major obstacleof the proposed scheme is that it is difficult to ensure that thecorrelated images always exist in the decoder side.

The inherent philosophy behind [64] is the external informa-tion provided by features is very valuable in image restoration,especially image super-resolution. Inspired by this, a series ofworks have been proposed for cloud based landmark imagesuper-resolution [63] and image denoising [62]. In particular,for landmark image super-resolution, the difficulty lies in thatonly the feature descriptors from the up-sampled image areavailable, such that the Bundled SIFT descriptors [70] are usedto obtained the retrieved images. These retrieved images arefurther aligned based on global registration, and the structure-aware matching criterion was adopted to achieve reliablereconstruction. For cloud based image denoising, the externaland internal data cubes are used to filter the image in the 3Dtransform domain, consisting of a 2D wavelet transform and a1D Hadamard transform. In order to obtain the similar patchesgiven the noisy version, a graph based matching strategy isadopted to achieve robust patch retrieval.

Inspired by the fact that the feature descriptors exhibitpowerful capabilities in finding the highly correlated refer-ences, the image set compression can be further facilitatedby such feature assistant texture coding framework. In [66],the local feature based photo album compression scheme wasproposed, which shows much superior performance comparedto the HEVC inter/intra coding. In this scheme, the extractedfeatures from each image in an image set are used for twomain purposes. First, the similarity of every image pair isestimated by measuring the distance between the matchedfeatures of two images. Thus, an efficient prediction structurecan be achieved by solving a classic graph problem, whereevery image in the image set represents a vertex, and theedge between two vertexes represents the similarity of two

images. To maximize the correlations of the image set, theminimum spanning tree (MST) of the graph can be estab-lished, leading to more efficient inter-image prediction andhigher compression ratio. Second, the features can also beused for guiding the inter-image predictions by a feature-based multi-model geometry transform approach. Traditionalblock matching based motion search fails to find effectivepredictions for image set coding, since images in an imageset may have very different luminance strengths and viewingperspectives. To tackle this, a three-step prediction strategyincluding geometric deformation, photometric transformationand block-based motion compensation was proposed. Geom-etry deformation aims to compensate global motions causedby the camera perspective difference, where the features areutilized to calculate the deformation matrix by RANSACalgorithm [59]. To increase the prediction accuracy, multiplegeometry transformation matrices are obtained by solving agraph-cut problem. After geometric deformation, the photo-metric transformation is performed in order to reduce theluminance differences between two similar images, where alinear transform model is estimated by minimizing the pixeldifferences between all matched feature points. After that ablock motion compensation is further introduced to eliminatethe impact of local shifts. The prediction structure, geometrydeformation and photometric transformation parameters aretransmitted to the decoder side as additional overhead. In [61],the performance of image set compression is further improvedbased on the novel inter-image redundancy measure, whichwas developed for optimizing the prediction structure. In par-ticular, the SIFT feature matching is performed between twosimilar images, and the covered area of matched SIFT pointsand the distance of each pair of matched SIFT descriptors areutilized for calculating the image correlation.

Considering the large-volume near-duplicate videos (NDVs)existing in the cloud, the near-duplicate video compressionwas proposed in [65]. In this scheme, the GPU based MapRe-duce framework was employed to NDV retrieval, and thehand-crafted features such as Hessian-Affine detector andSIFT are employed to calculate the similarity. The similarvideos are subsequently grouped by establishing the standardminimum-cut problem in graph theory, such that the intra-group similarities can be maximized. After partitioning thevideos into several groups among which the intra-group sim-ilarity is maximized and inter-group similarity is minimized,the effective compression strategy was subsequently proposed.In particular, one basic video is selected for each group ofvideo based on intra-group similarity, which is encoded inde-pendently while the remaining videos are encoded by referenc-ing the corresponding basic video for inter-video prediction.Since the resolutions, luminance and geometric location ofvideos within a group may be diverse, and the temporallyco-located frames in videos with the same group cannot beguaranteed to represent the same visual content, the Inter-Video Reference Index (IVRI) was investigated by analyzingthe frame similarity between the basic video and dependentvideos to achieve more efficient prediction. Moreover, theHomographic Transformer (HT) and Light Regulator (LR) aredeployed for basic videos to produce more accurate reference



8

for inter-video prediction. During the NDV joint coding fordependent videos, there is an additional reference frame to beemployed for motion estimation and mode decision besides thetraditional temporal intra-video reference frames, and all theother encoding processes remain the same as HEVC encoder.The bitstream of dependently coded video is concatenated withthat of the basic video to form the NDV bitstream. To achieverandom access capability for joint NDV compression, at thedecoder side, the basic video is decoded first, after whichthe dependency videos can be decoded. The NDV bitstreamcan also be transcoded into HEVC compliant bitstream byfirst decoding basic video for inter-video reference framegeneration, and subsequently the cascaded transcoding of eachdependent video is performed.

IV. ENVISIONING THE FUTURE

Despite significant progress on the joint coding of featureand texture with compact hand-crafted feature representation,the recent advances of deep learning features are also drivingthe development of many visual analysis tasks. In particular,there is a rather rich literature of deep learning modelsand features, which are consistently outperforming the hand-crafted features in many vision tasks. Moreover, recent devel-opments on the optimization and acceleration of deep neuralnetworks [71]–[73] have greatly reduced the computationalcosts, making the deployment of deep neural networks infront-end devices more feasible. With these schemes, morethan 10x acceleration and real-time applications of deep neuralnetworks could be achieved with little accuracy degradation.Furthermore, efficient models have been investigated [74] formobile and embedded vision applications. Energy-efficientpipelines [75] have been proposed to reduce the memorybandwidth of deep neural networks. With these approaches,applying the joint compression schemes in front-end becomespromising in the near future. Here, we will discuss andenvision the future joint compression of deep learning featuresand texture, from the perspectives of deep feature compres-sion, interactions between deep features and video texturesin compact representation, as well as the joint bit allocationbetween video textures and deep features for optimal codingperformance.

In contrast to traditional hand-crafted feature descriptors,the compression of deep feature descriptors raises manychallenging issues. In particular, deep neural networks suchas convolutional neural networks are composed of multiplehierarchical layers (e.g., convolutional and fully connectedlayers). As such, the identification of the layer to be com-pressed is meaningful to balance the compact feature size andthe representation capability. Moreover, the rapid evolutionof deep neural network and the lack of the versatile deepneural network model for various visual analysis tasks alsomotivate the utilization of the layer with high generalizationability in the joint compression framework. In addition, thereexists high redundancy in the representation layers, suchthat advanced inner- and inter-layer prediction methods arealso desired to remove the redundancy. In [76], [77], theparadigm of collaborative intelligence was proposed to bridge

the gap between mobile-only and cloud-only schemes. Morespecifically, for collaborative intelligence the feature extractionprocess with deep neural network is split into the front andback ends. For front-end such as the mobile device, the limitedcomputational resources only support the feature extraction ata certain layer, such that these middle features are uploadedto the cloud for further processing and analyses. In [78], thequantization and compression of the features are studied, andthe HEVC Range extension with 4:0:0 sampling format [79]is used to compress the quantized features. It is shown that thefeature size can be reduced by up to 70% without sacrificingthe accuracy. Moreover, the powerful deep learning feature hasalso been recognized by MPEG when developing the standardof CDVA, and the combinations of deep learning and hand-crafted features are adopted to achieve better video analysisperformance. These pioneer studies lay the groundwork forthe future investigation of joint deep feature and texturecompression. However, the most challenging issue of learningbased feature compression and further standardization is howto learn a general feature for various intelligence tasks basedon visual contents from different domains. This is still an openquestion because most of the success we see today is specifictask oriented.

Generally speaking, the deep features also imply the low-level texture information, and the recent developments of thedeep generative models such as the generative adversarialnetwork (GAN) [80] and variational auto-encoder (VAE) [81]have also demonstrated the powerful ability in generating thetexture information given the deep learning features. In [82],[83], the end-to-end deep learning based image coding frame-work is comprehensively studied, where the deep learningfeatures are compressed to recover the texture information.However, most of these works are developed based on the factthat the compressed features targeting at the reconstruction ofthe textures only, and the visual analysis tasks have not beenfurther taken into account. In [84], based on the observationthat the deep learning models used for compression have highsimilarities with the ones used for inference, the decodingand reconstruction of the texture content are bypassed andtwo analysis tasks image classification and semantic segmen-tation are directly performed on the compressed deep learningfeatures. In [85], it is shown that the facial images can bereconstructed from the deep features which are extracted forface recognition, based on which the vulnerabilities of the facerecognition based on facial template reconstruction attack arestudied.

Given the video texture and features to be transmitted, anatural question is how to allocate the given bit rate budget toeach individual such that the final utility can be ensured andoptimized. Here, the utility is defined based on the ultimate uti-lization of the bitstream, including visual viewing/monitoringand analysis. This corresponds to the well established rate-distortion theory in traditional video compression. However,in the joint compression scenario, the distortion term, i.e., theutility involves multiple factors, making the problem rathercomplicated. In particular, the joint bit allocation process can



9

be formulated as follows,

maxQPt,QPf

U(QPt, QPf ), s.t. Rt +Rf ≤ Rc, (3)

where U denotes the utility function and QPt and QPf are thequantization parameters for texture and feature. Rt and Rf arethe corresponding bit rate and Rc is the bit rate constraint. Theconstrained problem can be converted into an unconstrainedproblem with a Lagrange multiplier λ:

minQPt,QPf

J = −U(QPt, QPf ) + λ · (Rt +Rf ) (4)

Here, J denotes the joint rate-utility cost. In principle, the u-tility U can be measured as the combination of the signal levelvisual quality and visual analysis accuracy, which can also beexpressed as functions of their corresponding bit-rates. In [86],the utility function is defined as the Texture-Feature-Quality-Index based on linear combination of the texture quality interms of MSE and analysis accuracy such as the error rateof face recognition. As such, the optimal bit allocation canbe performed by computing the derivation of J to Rt andRf , respectively. The most challenging issue in the joint bitallocation is the definition of a trusted utility measure as designcriteria for optimization. In particular, in video surveillancewe may encounter complex application scenarios where themulti-task applications (e.g., face recognition, personal re-identification, car re-identification) are involved. As such, howto reasonably combine these analysis criteria with the texturequality should be further investigated.

Overall, from our perspectives, we believe that the jointcompression strategy is an important direction that deservesfurther studies and investigations along with the advances ofdeep learning. First, we believe the learning based especiallythe deep learning based features will play an important role inthe JFTR for intelligence application scenarios. Subsequently,from our point of view, the inverse problem of generatingtexture from features is also challenging, and it is the key issuein the joint compression loop. This leads to the third directionof joint compression, i.e., how to achieve a good tradeoffbetween the feature compression and texture compression.

V. CONCLUSIONS

In this paper, we have reviewed the advantages of thejoint compression of texture and feature, which is an alter-native strategy in representing the visual information withthe increasing proliferation of intelligent front-end camerasand processing units. We have also reviewed the emerginginteraction methods between the texture and feature in thecompression process, aiming to further improve the codingefficiency of texture based on the feature information. Thecore advantages of the joint representation lie in three aspects,including ensured feature quality, energy efficient featureutilization and enhanced coding performance. The potentialapplications of the joint compression scheme are also dis-cussed and the future development of this joint coding schemewith deep learning based feature extraction is envisioned.As such, here the message we are trying to convey is notthat one should abandon the use of video compression norfeature compression. Rather, in some scenarios when both

visual monitoring/viewing and analysis are required, the jointscheme is a more powerful alternative that can be practical-ly deployed. It is also expected that further motivations toconsider the joint feature and texture coding can be providedtowards smart video representation, and great benefits willbe brought forward along with the advances of the front-end intelligence. While the video coding and feature codingstandardizations will continue to be developed to achieve betterrepresentation efficiency within their own application scopes,the standardization of joint feature and texture compressionis still in its infancy age. To further enable the interactionsand inter-operations, it is also our greater desire to see suchstandard to be developed and used, especially in applicationswhere both analysis and monitoring might be required to meetthe application requirement.

REFERENCES

[1] T. Huang, “Surveillance video: The biggest big data,” Computing Now,vol. 7, no. 2, pp. 82–91, 2014.

[2] “Nvidia ai city challenge,” https://www.aicitychallenge.org/.[3] B. Girod, V. Chandrasekhar, D. M. Chen, N.-M. Cheung, R. Grzeszczuk,

Y. Reznik, G. Takacs, S. S. Tsai, and R. Vedantham, “Mobile visualsearch,” Signal Processing Magazine, IEEE, vol. 28, no. 4, pp. 61–76,2011.

[4] A. Redondi, L. Baroffio, L. Bianchi, M. Cesana, and M. Tagliasacchi,“Compress-then-analyze versus analyze-then-compress: What is bestin visual sensor networks?” IEEE Trans. Mobile Computing, vol. 15,no. 12, pp. 3000–3013, 2016.

[5] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”International journal of computer vision, vol. 60, no. 2, pp. 91–110,2004.

[6] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robustfeatures,” in Computer Vision–ECCV. Springer, 2006, pp. 404–417.

[7] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, and P. Fua,“BRIEF: Computing a local binary descriptor very fast,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1281–1298, 2012.

[8] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: an efficientalternative to sift or surf,” in IEEE Int. Conf. Computer Vision (ICCV).IEEE, 2011, pp. 2564–2571.

[9] S. Leutenegger, M. Chli, and R. Y. Siegwart, “BRISK: Binary robustinvariant scalable keypoints,” in IEEE Int. Conf. Computer Vision(ICCV). IEEE, 2011, pp. 2548–2555.

[10] S. Zhang, Q. Tian, Q. Huang, W. Gao, and Y. Rui, “Usb: ultrashortbinary descriptor for fast visual matching and retrieval,” IEEE Trans.Image Process., vol. 23, no. 8, pp. 3671–3683, 2014.

[11] V. Chandrasekhar, G. Takacs, D. Chen, S. S. Tsai, J. Singh, and B. Girod,“Transform coding of image feature descriptors,” in IS&T/SPIE Elec-tronic Imaging. International Society for Optics and Photonics, 2009,pp. 725 710–725 710.

[12] H. Jegou, M. Douze, and C. Schmid, “Product quantization for nearestneighbor search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 1,pp. 117–128, 2011.

[13] J. Sivic and A. Zisserman, “Video google: A text retrieval approach toobject matching in videos,” in IEEE Int. Conf. Computer Vision. IEEE,2003, pp. 1470–1477.

[14] F. Perronnin, Y. Liu, J. Sanchez, and H. Poirier, “Large-scale imageretrieval with compressed fisher vectors,” in IEEE Conf. Computer Visionand Pattern Recognition. IEEE, 2010, pp. 3384–3391.

[15] H. Jegou, M. Douze, C. Schmid, and P. Perez, “Aggregating localdescriptors into a compact image representation,” in Computer Visionand Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE,2010, pp. 3304–3311.

[16] D. M. Chen, S. S. Tsai, V. Chandrasekhar, G. Takacs, J. Singh, andB. Girod, “Tree histogram coding for mobile image matching,” in DataCompression Conference. IEEE, 2009, pp. 143–152.

[17] D. Chen, S. Tsai, V. Chandrasekhar, G. Takacs, R. Vedantham,R. Grzeszczuk, and B. Girod, “Residual enhanced visual vector as acompact signature for mobile visual search,” Signal Processing, vol. 93,no. 8, pp. 2316–2327, 2013.



10

[18] L. Baroffio, M. Cesana, A. Redondi, M. Tagliasacchi, and S. Tubaro,“Coding visual features extracted from video sequences,” IEEE Trans.Image Process., 2013.

[19] L. Baroffio, J. Ascenso, M. Cesana, A. Redondi, and M. Tagliasacchi,“Coding binary local features extracted from video sequences,” in IEEEInt. Conf. Image Processing, 2014.

[20] M. Makar, V. Chandrasekhar, S. Tsai, D. Chen, and B. Girod, “Inter-frame coding of feature descriptors for mobile augmented reality,” IEEETrans. Image Process., 2014.

[21] D. M. Chen, M. Makar, A. F. Araujo, and B. Girod, “Interframe codingof global image signatures for mobile augmented reality,” in DataCompression Conference (DCC). IEEE, 2014, pp. 33–42.

[22] D. M. Chen and B. Girod, “A hybrid mobile visual search system withcompact global signatures,” IEEE Trans. Multimedia, vol. 17, no. 7, pp.1019–1030, 2015.

[23] B. Girod, V. Chandrasekhar, R. Grzeszczuk, and Y. A. Reznik, “Mobilevisual search: Architectures, technologies, and the emerging mpegstandard,” IEEE Trans. Multimedia, vol. 18, no. 3, pp. 86–94, 2011.

[24] L.-Y. Duan, W. Sun, X. Zhang, S. Wang, J. Chen, J. Yin, S. See,T. Huang, A. C. Kot, and W. Gao, “Fast MPEG-CDVS Encoder WithGPU-CPU Hybrid Computing,” IEEE Trans. on Image Processing,vol. 27, no. 5, pp. 2201–2216, 2018.

[25] L.-Y. Duan, T. Huang, and W. Gao, “Overview of the MPEG CDVSstandard,” in Data Compression Conference (DCC), 2015.

[26] L.-Y. Duan, V. Chandrasekhar, J. Chen, J. Lin, Z. Wang, T. Huang,B. Girod, and W. Gao, “Overview of the MPEG-CDVS standard,” IEEETrans. Image Process., vol. 25, no. 1, pp. 179–194, 2016.

[27] J. Chen, L.-Y. Duan, F. Gao, J. Cai, A. C. Kot, and T. Huang, “Alow complexity interest point detector,” Signal Processing Letters, IEEE,vol. 22, no. 2, pp. 172–176, 2015.

[28] S. Paschalakis, K. Wnukowicz, M. Bober, A. Mosca, and M. Mattel-liano, “CDVS CE2: Local descriptor compression proposal,” ISO/IECJTC1/SC29/WG11/M25929, 2012.

[29] I. JTC1/SC29/WG11/M25883, “Cdvs core experiment 3: Stan-ford/peking/huawei contribution,” 2012.

[30] J. Lin, L.-Y. Duan, Y. Huang, S. Luo, T. Huang, and W. Gao, “Rate-adaptive compact fisher codes for mobile visual search,” Signal Process-ing Letters, IEEE, vol. 21, no. 2, pp. 195–198, 2014.

[31] I. JTC1/SC29/WG11/N15339, “Call for proposals for compact descrip-tors for video analysis (CDVA) - search and retrieval,” 2015.

[32] “Compact descriptors for video analysis: Objectives, applications anduse cases,” ISO/IEC JTC1/SC29/WG11/N14507, Valencia, Mar, 2014.

[33] L.-Y. Duan, V. Chandrasekhar, S. Wang, Y. Lou, J. Lin, Y. Bai, T. Huang,A. C. Kot, and W. Gao, “Compact descriptors for video analysis: theemerging mpeg standard,” arXiv preprint arXiv:1704.08141, 2017.

[34] J. Chao and E. Steinbach, “Preserving SIFT features in JPEG-encodedimages,” in IEEE Int. Conf. Image Processing (ICIP), Sept 2011, pp.301–304.

[35] J. Chao, R. Huitl, E. Steinbach, and D. Schroeder, “A novel rate controlframework for SIFT/SURF feature preservation in H.264/AVC videocompression,” IEEE Trans. Circuits Syst. Video Technol., vol. PP, no. 99,pp. 1–1, 2014.

[36] Oxbuildings, http://www.robots.ox.ac.uk/∼vgg/data/oxbuildings/.[37] Zubud, http://www.vision.ee.ethz.ch/showroom/zubud/index.en.html.[38] A. Zabala and X. Pons, “Effects of lossy compression on remote sensing

image classification of forest areas,” Int. Journal of Applied EarthObservation and Geoinformation, vol. 13, no. 1, pp. 43–51, 2011.

[39] ——, “Impact of lossy compression on mapping crop areas from remotesensing,” Int. Journal of Remote Sensing, vol. 34, no. 8, pp. 2796–2813,2013.

[40] G. J. Sullivan, J. Ohm, W.-J. Han, and T. Wiegand, “Overview of thehigh efficiency video coding (HEVC) standard,” IEEE Trans. CircuitsSyst. Video Technol., vol. 22, no. 12, pp. 1649–1668, 2012.

[41] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified em-bedding for face recognition and clustering,” in IEEE conference oncomputer vision and pattern recognition, 2015, pp. 815–823.

[42] W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang,Z. Wang, C.-C. Loy et al., “Deepid-net: Deformable deep convolutionalneural networks for object detection,” in IEEE Conf. Computer Visionand Pattern Recognition, 2015, pp. 2403–2412.

[43] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeledfaces in the wild: A database for studying face recognition in uncon-strained environments,” University of Massachusetts, Amherst, Tech.Rep. 07-49, October 2007.

[44] L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrainedvideos with matched background similarity,” in Computer Vision and

Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011,pp. 529–534.

[45] L. Baroffio, M. Cesana, A. Redondi, M. Tagliasacchi, and S. Tubaro,“Hybrid coding of visual content and local image features,” in IEEE Int.Conf. Image Processing (ICIP). IEEE, 2015, pp. 2530–2534.

[46] J. Chao and E. Steinbach, “Keypoint encoding for improved featureextraction from compressed video at low bitrates,” IEEE Trans. Multi-media, vol. 18, no. 1, pp. 25–39, 2016.

[47] X. Zhang, S. Ma, S. Wang, S. Wang, X. Zhang, and W. Gao, “From vi-sual search to video compression: A compact representation frameworkfor video feature descriptors,” in Data Compression Conference, 2016,pp. 407–416.

[48] X. Zhang, S. Ma, S. Wang, X. Zhang, H. Sun, and W. Gao, “A jointcompression scheme of video feature descriptors and visual content,”IEEE Trans. Image Process., vol. 26, no. 2, pp. 633–647, 2017.

[49] X. Zhang, R. Xiong, W. Lin, S. Ma, J. Liu, and W. Gao, “Video compres-sion artifact reduction via spatio-temporal multi-hypothesis prediction,”IEEE Trans. on Image Processing, vol. 24, no. 12, pp. 6048–6061, 2015.

[50] X. Zhang, W. Lin, R. Xiong, X. Liu, S. Ma, and W. Gao, “Low-rank decomposition-based restoration of compressed images via adaptivenoise estimation,” IEEE Trans. on Image Processing, vol. 25, no. 9, pp.4158–4171, 2016.

[51] X. Zhang, R. Xiong, W. Lin, J. Zhang, S. Wang, S. Ma, and W. Gao,“Low-rank-based nonlocal adaptive loop filter for high-efficiency videocompression,” IEEE Trans. on Circuits and Systems for Video Technol-ogy, vol. 27, no. 10, pp. 2177–2188, 2017.

[52] M. Makar, S. S. Tsai, V. Chandrasekhar, D. Chen, and B. Girod, “In-terframe coding of canonical patches for low bit-rate mobile augmentedreality,” Int. Journal of Semantic Computing, vol. 7, no. 1, pp. 5–24,2013.

[53] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas,F. Schaffalitzky, T. Kadir, and L. Van Gool, “A comparison of affineregion detectors,” International journal of computer vision, vol. 65, no.1-2, pp. 43–72, 2005.

[54] GreenEyes, “Rome landmark dataset [online],” https://sites.google.com/site/greeneyesprojectpolimi/downloads/datasets/rome-landmark-dataset.

[55] H. Yue, X. Sun, J. Yang, and F. Wu, “Cloud-based image coding formobile devices–toward thousands to one compression,” IEEE Trans.Multimedia, vol. 15, no. 4, pp. 845–857, June 2013.

[56] Y.-H. Fok and O. C. Au, “An improved fast feature-based block motionestimation,” in Proceedings of 1st Int. Conf. Image Processing, vol. 3,1994, pp. 741–745.

[57] S. K. Kim, S. J. Kang, T. S. Wang, and S. J. Ko, “Feature pointclassification based global motion estimation for video stabilization,”IEEE Trans. Consum. Electron., vol. 59, no. 1, pp. 267–272, Feb. 2013.

[58] M. Tok, A. Glantz, A. Krutz, and T. Sikora, “Feature-based globalmotion estimation using the helmholtz principle,” in IEEE Int. Conf.Acoustics, Speech and Signal Processing (ICASSP), May 2011, pp.1561–1564.

[59] M. A. Fischler and R. C. Bolles, “Random sample consensus: Aparadigm for model fitting with applications to image analysis andautomated cartography,” Commun. ACM, vol. 24, no. 6, pp. 381–395,Jun. 1981.

[60] X. Zhang, S. Wang, S. Wang, S. Ma, and W. Gao, “Feature-matchingbased motion prediction for high efficiency video coding in cloud,” inIEEE Int. Conf. Multimedia Expo Workshops (ICMEW), 2015, pp. 1–6.

[61] X. Zhang, Y. Zhang, W. Lin, S. Ma, and W. Gao, “An inter-image re-dundancy measure for image set compression,” in IEEE Int. Symposiumon Circuits and Systems (ISCAS). IEEE, 2015, pp. 1274–1277.

[62] H. Yue, X. Sun, J. Yang, and F. Wu, “Image denoising by exploringexternal and internal correlations,” IEEE Trans. Image Process., vol. 24,no. 6, pp. 1967–1982, 2015.

[63] ——, “Landmark image super-resolution by retrieving web images,”IEEE Trans. Image Process., vol. 22, no. 12, pp. 4865–4878, 2013.

[64] ——, “Cloud-based image coding for mobile devices toward thousandsto one compression,” IEEE Trans. Multimedia, vol. 15, no. 4, pp. 845–857, 2013.

[65] H. Wang, T. Tian, M. Ma, and J. Wu, “Joint compression of near-duplicate videos,” IEEE Trans. Multimedia, vol. 19, no. 5, pp. 908–920,2017.

[66] Z. Shi, X. Sun, and F. Wu, “Photo album compression for cloud storageusing local features,” IEEE Journal on emerging and selected topics incircuits and systems, vol. 4, no. 1, pp. 17–28, 2014.

[67] X. Zhang, W. Lin, Y. Zhang, S. Wang, S. Ma, L. Duan, and W. Gao,“Rate-distortion optimized sparse coding with ordered dictionary forimage set compression,” IEEE Transactions on Circuits and Systems forVideo Technology, 2017.



11

[68] X. Zhang, W. Lin, S. Ma, S. Wang, and W. Gao, “Rate-distortion basedsparse coding for image set compression.” in VCIP, 2015, pp. 1–4.

[69] X. Liu, G. Cheung, C.-W. Lin, D. Zhao, and W. Gao, “Prior-basedquantization bin matching for cloud storage of jpeg images,” IEEETransactions on Image Processing, vol. 27, no. 7, pp. 3222–3235, 2018.

[70] L. Dai, X. Sun, F. Wu, and N. Yu, “Large scale image retrieval withvisual groups,” in IEEE Int. Conf. Image Processing (ICIP). IEEE,2013, pp. 2582–2586.

[71] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressingdeep neural networks with pruning, trained quantization and huffmancoding,” Fiber, vol. 56, no. 4, pp. 3–7, 2015.

[72] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, andW. J. Dally, “EIE: Efficient inference engine on compressed deepneural network,” in ACM/IEEE International Symposium on ComputerArchitecture, 2016, pp. 243–254.

[73] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net:Imagenet classification using binary convolutional neural networks,” inEuropean Conference on Computer Vision, 2016, pp. 525–542.

[74] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand,M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neuralnetworks for mobile vision applications,” 2017.

[75] Y. H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural net-works,” IEEE Journal of Solid-State Circuits, vol. 52, no. 1, pp. 127–138, 2017.

[76] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, andL. Tang, “Neurosurgeon: Collaborative intelligence between the cloudand mobile edge,” in Int. Conf. Architectural Support for ProgrammingLanguages and Operating Systems. ACM, 2017, pp. 615–629.

[77] A. E. Eshratifar, M. S. Abrishami, and M. Pedram, “Jointdnn: an efficienttraining and inference engine for intelligent mobile cloud computingservices,” arXiv preprint arXiv:1801.08618, 2018.

[78] H. Choi and I. V. Bajic, “Deep feature compression for collaborativeobject detection,” arXiv preprint arXiv:1802.03931, 2018.

[79] D. Flynn, D. Marpe, M. Naccari, T. Nguyen, C. Rosewarne, K. Sharman,J. Sole, and J. Xu, “Overview of the range extensions for the HEVCstandard: Tools, profiles, and performance,” IEEE Trans. Circuits Syst.Video Technol., vol. 26, no. 1, pp. 4–19, 2016.

[80] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” inAdvances in neural information processing systems, 2014, pp. 2672–2680.

[81] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin,“Variational autoencoder for deep learning of images, labels and cap-tions,” in Advances in neural information processing systems, 2016, pp.2352–2360.

[82] J. Balle, V. Laparra, and E. P. Simoncelli, “End-to-end optimized imagecompression,” arXiv preprint arXiv:1611.01704, 2016.

[83] O. Rippel and L. Bourdev, “Real-time adaptive image compression,”arXiv preprint arXiv:1705.05823, 2017.

[84] R. Torfason, F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, andL. Van Gool, “Towards image understanding from deep compressionwithout decoding,” arXiv preprint arXiv:1803.06131, 2018.

[85] G. Mai, K. Cao, P. C. Yuen, and A. K. Jain, “Face image reconstructionfrom deep templates,” arXiv preprint arXiv:1703.00832, 2017.

[86] Y. Li, C. Jia, S. Wang, X. Zhang, S. Wang, S. Ma, and W. Gao, “Largescale image retrieval with visual groups,” in IEEE Int. Conf. MultimediaBig Data. IEEE, 2018.

Siwei Ma received the B.S. degree from ShandongNormal University, Jinan, China, in 1999, and thePh.D. degree in computer science from the Instituteof Computing Technology, Chinese Academy ofSciences, Beijing, China, in 2005. He held a post-doctoral position with the University of SouthernCalifornia, Los Angeles, CA, USA, from 2005 to2007. He joined the Institute of Digital Media,School of Electronics Engineering and ComputerScience, Peking University, Beijing, where he iscurrently a Professor. He has authored over 100

technical articles in refereed journals and proceedings in the areas of imageand video coding, video processing, video streaming, and transmission.

Xiang Zhang received the B.S. degree in computerscience from the Harbin Institute of Technology,Harbin, China, in 2013. He is currently workingtoward the Ph.D. degree at Peking University. Hisresearch interests include image/video quality as-sessment, video compression, and visual retrieval.

Shiqi Wang received the B.S. degree in computerscience from the Harbin Institute of Technology in2008 and the Ph.D. degree in computer applicationtechnology from the Peking University in 2014.From 2014 to 2016, he was a Post-Doctoral Fellowwith the Department of Electrical and Comput-er Engineering, University of Waterloo, Waterloo,Canada. From 2016 to 2017, he was a Research Fel-low with the Rapid-Rich Object Search Laboratory,Nanyang Technological University, Singapore. He iscurrently an Assistant Professor with the Department

of Computer Science, City University of Hong Kong. He has proposed over40 technical proposals to ISO/MPEG, ITU-T and AVS standards. His researchinterests include image/video compression, analysis and quality assessment.

Xinfeng Zhang (M’16) received the B.S. degree incomputer science from Hebei University of Technol-ogy, Tianjin, China, in 2007, and the Ph.D. degreein computer science from the Institute of ComputingTechnology, Chinese Academy of Sciences, Beijing,China, in 2014. From, Jul. 2014 to Oct. 2017, hewas a research fellow in Rapid-Rich Object Search(ROSE) Lab in Nanyang Technological University,Singapore. He is currently a Postdoc Fellow withthe School of Electrical Engineering System, Uni-versity of Southern California, Los Angeles, USA.

His research interests include image and video processing, image and videocompression.

Chuanmin Jia received B.Sc. in Computer Scienceand Technology from Beijing University of Postsand Telecommunications (BUPT), China, in 2015.He is currently pursuing the Ph.D. degree with thedepartment of Computer Science in Peking Univer-sity, Beijing, China. From 2017 to 2018, he wasa visiting student with the Video Lab, New YorkUniversity, NY, USA. His research interests includevideo coding, machine learning and light field imagecompression.

Shanshe Wang received the B.S. degree from theDepartment of Mathematics, Heilongjiang Universi-ty, Harbin, China, in 2004, the M.S. degree in com-puter software and theory from Northeast PetroleumUniversity, Daqing, China, in 2010, and the Ph.D.degree in computer science from the Harbin Instituteof Technology, Harbin, in 2014. He currently holdsa postdoctorate position in Peking University. Hiscurrent research interests include video compression,image and video quality assessment.

joint feature and texture coding: towards smart video representation via front-end...

Documents