low bit rate mobile visual search

8/4/2019 Low Bit Rate Mobile Visual Search

1/59

InstituteofDigitalMedia,TheSchoolofCS&EE,PekingUniversity,ChinaAug.18,2011,NTU,Singapore


2/59

2


3/59

3


4/59

ProductPackaging

LandmarksLogos

Faces

Cars Pets

UnpackagedProductsigidTexturedArticulateContourDefined

Momentoftruthforcomputervision Instancematching,notclassrecognition

4


5/59

PervasiveMobileVisualSearch

DevelopedSystems

Mobot

NevenVision

Ideixis

Snaptell

PointandFound

GoogleGoggle


6/59

SideInformationisuseful

formobilevisualsearch

VisualSearch

SideInformation

Mobile

VisualSearch

Highefficient

visualvocabularycoding

Desirablesearch

precision&fastspeedover

extremelylargescaledatasets

Goodscalabilityinterms

ofupstreamquery

coding

complexity


7/59

PervasiveSide(Context)Information

GPS AccessPoints RFIDTags

VisuallyEasilyRecognizedBarcode,

LabelsorLogos


8/59

Amobilephotoqueryhastobetransmittedfromamobile

devicetoaserver

Oftenoverarelativelyslowwirelesslink3G

Thequalityofuserexperienceheavilydependsonhowmuchinformationhastobetransmitted,especiallyinthe

scenarioofrealityaugmentation

8


9/59

Compactlocalfeaturedescriptors e.g.CHoG Stanford,DAISYEFPL,PCASIFTDescriptorsCMU

Compactimagesignatures e.g.miniBoW,AggregateLocalDescriptorsINTRIA

Previousworksstilldemand2KBto4KBvisualdescriptorsperqueryimageunsuitableforrealityaugmentation

Withinanunstablewirelessnetworke.g.3Gnetwork,the

deliverycoststilloccasionallydelaysthequery9


10/59

TransformCodingofSIFT/SURFdescriptors

Chandrasekharetal.,VCIP09 Directcompressionoforientedimagepatch

M.Makaretal.,ICASSP09

Descriptordesignedforcompressibility:CHoGChandrasekharetal.,CVPR09

TreeHistogramCoding

Chenetal.,DCC09 CompressionofSpatialLayoutofLocalFeatures

Tsaietal.,Mobimedia10

10


11/59

CompactDescriptorforVisualSearch AnongoingMPEGADHOCGroup

Standardschedule 2010,AprilInitializetheCDVSADHOCGroup

2011,July,FinalCFP

Evaluationset Mobilequery,especiallylandmarkqueries

MajorRequirementofCompactDescriptors

Compactness

Discriminability11


12/59

12


13/59

Allstateof theartcompactdescriptorsaredesigned

onlybasedonvisualcontentstatistics

Incorporatethecheaplyavailablemobilecontexte.g.GPS

orbasestationtagstosupervise thedescriptordesign13


14/59

Effectiveandefficient,lowbitratemobilevisualsearch

sayhundredsofbitspervisualquery

Coding/Compressionofanoriginallyhighdimensional

imagesignaturese.g.BoW

14

M islearntfromthemobilecontext


15/59


16/59

Userentersagivenregion

Themobileenddownloadsorprestoretheregion

specificdescriptionfunction

Themobileusertakesaquery

ThemobileendextractstheinitialBoW,compressesit

intoaregionspecificcompactdescriptor 16


17/59

Geographicalregiondivision

Learningcompactdescriptorwithineachregion


18/59

18


19/59

CrawledfromFlickrandPanoramio websites

Beijing

NewYorkCity

Barcelona

Singapore

Florence

ThegeographicaldistributionofFlickerandPanoramioPhotosinBeijing 19


20/59

20

Notations:intotalmregionscurrentgeotaggedphotoxregionassignmentyi withim


21/59

21

ExpectationStep

MaximizationStep


22/59

22

ThegeographicaldistributionofFlickerand

PanoramioPhotosinBeijingCity

ThevisualawarepartitionofBeijing

intogeographicalregions.Different

colorsdenotedifferentclusters.


23/59

23


24/59

24

Forthequerybelongingregion,compactcodewordsarelearnttotransmit

Thelearntcompactcodebookandtheextracteddescriptorsinexemplarqueries.

Left:thecompactcodebookinthequerysassignedregion;Middle:thequery,

wherecolorhighlightsdenotethedetecteddescriptorsonthequery;Right:the

transmittedwords.Weonlytransmittheiroccurrenceindexinpractice.


25/59

Learningacompressionmatrix

MMK

MK

ThecompactcodebookU isgeneratedviaUMTV UsescalablevocabularytreeSVTtobuildtheinitialhigh

dimvocabulary Hierarchicalquantizationoflocalfeatures

AgoodcodebookU shouldminimizetherankingloss

25

where istherankingpositionweightofIxwithrespecttoIq

,suchthatahigherrankcorrespondstoalargeweight.


26/59

26


27/59

withsampleloss,Boosting

ErrorWeighting Ranking DecodedSignature

27

Given aregion,aimtominimizetheoverallcost:


28/59

Findtheithbestcodeword

Errorweightingupdate

Compressionfunctionupdate

28


29/59

Formoredetails,pls refertoLocationDiscriminativeVocabularyCodingfor

MobileLandmarkSearch,Int.JournalofComputerVision,InPress


30/59

30


31/59

Achickenandeggproblem

Solvedviaaniterativecooptimization

31


32/59

32

Codinglength ofregioni Themaximalcodinglength ofallregions


33/59

CompactnessLearningProcessoverBeijingCity

33ThegeographicalvisualizationofthedescriptorcompactnessinBeijingcitythroughiterativecooptimizationT1to20,whichwouldbetalkedlater

Wenormalizetheminvs.max

ratioofdescriptorlengthsand

maptheratiotothesaturationofredcolor.Thegreenpoints

denotethedistributionofgeo

taggedphotos.Ingeneral,less

saturatedmapcorrespondsto

moreoptimaldescriptors.


34/59

34


35/59

Over1MgeographicaltaggedphotosfromFlickrandPanoramio websites

Beijing,NewYorkCity,Barcelona,Singapore,etc..

Fromthegeographicalmapofeachcity,wechoose30

mostdenseregionsand30randomregions Askvolunteerstomanuallyidentifyoneormoredominantviews.

Allnearduplicatedlandmarkphotostoagivenviewarelabeledin

itsbelongingandnearbyregions

35


36/59

Nister etal.2006

Jegou etal.,2010

Chandrasekharetal.,2009a

Chenetal.,2009

Ouralternativeapproach36


37/59


38/59


39/59

39

mAP withrespecttodifferentregions

mAP variancesindifferentregions,wedrawtwodimensionallatticesto

dividedregionswithrespecttodifferentimagevolumesanddescriptor

bits,thenaveragethemAP forregionsfallingintoeachlattice.


40/59

40


41/59

41


42/59


43/59

43


44/59

ensureinteroperabilityof

visualsearchapplications

anddatabases,

enablehighlevelof

performanceof

implementations

conformant

tothestandard,

simplifydesignofdescriptor

extractionandmatchingfor

visualsearchapplications,

enablehardwaresupportfordescriptorextractionand

matchinginmobiledevices,

reduceloadonwireless

networkscarryingvisual

searchrelatedinformation.


45/59

Meeting# Date Action97 July1822,2011 FinalCfPissued

98 Nov.28Dec.02,2011 Initialevaluationofproposalsandassignmentofcrosschecks

99 Feb.0610,2012 WD1

101 July1620,2012 CD

103 Jan.2125,2013 DIS

105 July,2013 FDIS


46/59

Requirement Description Fulfillmentinformation

Sufficiency Descriptorsshallbeselfcontained,inthesensethatno

otherdataarenecessaryformatching

Y/N

Formatindependence

Descriptorsshallbeindependentoftheimageformat Y/N

Robustness Highmatchingaccuracyshallbeachievedatleastfor

imagesoftexturedrigidobjects,landmarks,andprinted

documents.Thematchingaccuracyshallberobustto

changesinvantagepoint,cameraparameters,lighting

conditions,aswellasinthepresenceofpartial

occlusions

*Retrievalandpair

wisematching

accuracyresults

obtainedfordifferent

descriptorlengths.

Compactness Shallminimizelengths/sizeofimagedescriptors

Scalability 1. Shall allow adaptation of descriptor lengths to

support the required performance level and

database size.2. Shall enable design of webscale visual search

applications and databases.

Rangeofdescriptor

lengthsanddatabase

sizes

supported.

Extractioncomplexity

Shallallowdescriptorextractionwithlowcomplexity(in

termsofmemoryandcomputation)

*Timinginformation

fordescriptor

extractionopertations.

Matchingcomplexity

1. Shall allow matching of descriptors with low

complexity (in terms of memory and

computation).

2. If decoding of descriptors is required formatching, such decoding shall also be possible

with low complexity.

*Timinginformation

forretreivalandpair

wisematching

opertations.

Localization 1. Shall support visual search algorithms that

identify and localize matching regions of the

query image and the database image

2. Shall support visual search algorithms that

provide an estimate of a geometric

transformation between matching regions of the

query image and the database image

*Localizationaccuracy

results.

(*)resultsobtainedusingdatabasesandtestproceduresspecifiedintheevaluationframework[1].


47/59

The Framework of Multichannel


48/59

TheFrameworkofMulti channel

CompactVisualDescriptor(MCVD)

Theencodedcompactdescriptorincludestwoparts:

ChannelIdentification +ChannelDependentVisualDescriptorNote:

A

channel

refers

to

subdivide

(partition)

the

reference

database.

Within

each

channel,

wemay generateachanneldependent(extremely)compactdescriptor.

The Framework of Multichannel


49/59

Thechannellearningcanbebasedoncontextualtags(e.g.GPSorRFID)

orcanbesolelyonthevisualstatistics.

TheFrameworkofMulti channel

CompactVisualDescriptor(MCVD)

Wh d di i k l


50/59

GPS

Access

Points

RFIDTags

StandardizationEfforts

VisuallyRecognized

Barcode,Logos,etc.

Wherestandardizationtakesplace

J ifi i f Ch l i MCVD


51/59

ChannelNum=1

Channel1Model Channel2Model ChannelKModel

ChannelDivision

JustificationofChannelinMCVD

J tifi ti f Ch l i MCVD


52/59

ChannelNum=1

Channel1Model Channel2Model ChannelKModel

ChannelDivision BoostingCompactCodebook

Channel1

Compact

Codebook

Channel2

Compact

Codebook

JustificationofChannelinMCVD


53/59

53

TwotypicalMCVDcaseswithorwithoutcontextualinformation,with

comparisontothestateoftheart,overmillionscalelandmarkdataset.


54/59

54

ComparisonofmAP withrespecttotheupstreamquerybitratefor

product/CD/bookcoverBenchmarkDatabases(publiclyavailable).


55/59

55

Search

Battery

Bandwidth


56/59

56

Phone Battery:

Voltage: 4.0 V, Capability:

1400 mAH (or 20.2K Joules)

Sending Images

20.2K Joules/52.4 Joules

~ 385 queries in total

Sending MCVD:

20.2K Joules/8.1Joules

~ 2494 queries in total

Averageenergyconsumptioncomparisonthroughthe3Gwireless

link,betweentransmittingtheentirequeryimageandtheextracting

andtransmittingofMCVDandothercompactdescriptors.


57/59

57

O l t


58/59

Ourrelevantpapers

SelectedPapers LocationDiscriminativeVocabularyCodingforMobileLandmarkSearch,Int.JournalofComputerVision,InPress LearningCompactVisualDescriptorforLowBitRateMobileLandmark

Search,IJCAI2011,Barcelona,Spain,Jul.2011 TowardsLowBitRateMobileVisualSearchwithMultipleChannel

Coding,ACMMM2011,Arizona,USA,Nov.2011 MPEGInputContributions

CompactDescriptorsforVisualSearch,m18542,MPEG94th PKUBench:AContextualRichBenchmarkforMobileVisualSearch,

m19188,MPEG95th MultipleChannelCompactVisualDescriptorwithAdaptiveChannel

Learning,m19985,MPEG96th TopiclevelSamplingTowardsOptimizedLocalitySensitiveVocabulary

Coding,m21199,MPEG97th

Bandwidth


59/59

Thanks!

Q & A

Visual

Search

Side

Information

MobileVisual

Search

Search

Battery

low bit rate mobile visual search

Documents