image and video upscaling from local self examples
DESCRIPTION
Image and Video Upscaling from Local Self Examples. Gilad Freedman Raanan Fattal Hebrew University of Jerusalem. Background and o verview Algorithm description L ocal self similarity Non-dyadic filter bank Filter design Results. Single i mage upscaling. 1. l arge 2. realistic - PowerPoint PPT PresentationTRANSCRIPT
Image and Video Upscaling from Local Self Examples
Gilad FreedmanRaanan Fattal
Hebrew University of Jerusalem
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
Single image upscaling
1. large2. realistic3. faithful4. fast
Previous workparametric image model example based
Freeman et al. 2002
generic looking edges
Sun et al. 2008
Shan et al 2008
Fattal 2007
Glasner et al. 2009
noisyresult
New approach: locale example based
corner step edge
line step edge
non smooth shading
local self similarity small upscaling ratios
1/2
4/5
new non-dyadic filter bank
local self similarity
local search speed
locality too few examples
Increase exemplar quality and size
maintain search locality
novel components:
1. local self similarity2. non-dyadic filter bank
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
Local self-examples upscaling
low pass original image high pass
interpolated image
frequency content
low pass high pass
interpolated image
For each patch:
Search a local area for best example
Take corresponding
patch
Add to interpolated
image
frequency content
Local self-examples upscaling
Local self-examples upscaling
low pass high pass
interpolated image
Repeat for all patches, to fill the high frequenciesfrequency content
OverviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
Local self similarity
cropped downscaled
Local self similarity
Patches in original image can matched locally with ones in downscaled version
Local examples are enough
full imageimage database
query db image local
4.0 2.9 3.55
1.6 1.05 1.05
2.7 2.05 2.05
3.3 2.96 3.06
6.5 5.61 5.61
best matches
Visual assessment – external, exact NN, local
Large externalexample database
Searching theentire image
Searching localregions in image
external database
global search
local search
Comparison of example search methods
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
Need for non-dyadic scalingslarge ratios mixed ratiossmall ratios
Dyadic filters
1:2
full frequency
content
higher half
lower half
dyadic filter bank
Non-dyadic filter bank
1:2
4:5
small scaling ratios
better examples
full frequency
content
higher part
lower part
non-dyadic filter bank
Non-dyadic filters: downscaling
1. convolve with 2 filters2. subsample each by 3
dyadic case:
example for the 2:3 ratio:
1. convolve with one filter2. subsample by 2
Non-dyadic filters: upscaling
1. zero upsample by 22. convolve with 2 filters3. sum
dyadic case:
example for the 2:3 ratio:
1. zero upsample by 12. convolve with 1 filter
Use of the filters in upscalingUpscaling using inverse scaling
filters
Smoothing by downscaling
and upscaling
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
When interpolating, smooth areas come from inputUniformly spaced grids should remain uniform
1. Uniform stretch
0
255
brig
htne
ss
grid coordinates
2. Consistency
upsample downsample
The interpolated image, if downscaled should be equal to the input. Formally,
Previous methods achieve consistency by solving large linear systems to achieve this property
3. PSF modeling
Large image - small camera point spread
function
Small image - large camera point spread
function
Difference between point spread functions
4. Low frequency spanfr
eque
ncy
When upsampling don’t add new frequencies
Upsampling filter should be low-pass
original interpolated
5. Singularities preservation
≈blurred Image interpolated image
similar amount of blur
Real time video upsampling on GPUmain GPU memory
GPU cores
NTSC to full HD @ 24 fps
Search and filter-banksare both local operations
Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults
Bicubic x3(zoomed in)
Ours X3(zoomed in)
Bicubic x3 Ours X3
Genuine Fractals™ x4 Ours X4
Glasner et al. 2009 x4 Ours X4
Thank you!
Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn