spatio-temporal quincunx sub-sampling.. and how we get there david lyon
TRANSCRIPT
![Page 1: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/1.jpg)
Spatio-Temporal Quincunx Sub-Sampling
. . and how we get there
David Lyon
![Page 2: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/2.jpg)
Overview
Sampling in Television and Film The problems of aliasing Filtering requirements Conversion between differing formats Problems that can occur How we can mitigate some of the problems and maintain
or improve quality
![Page 3: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/3.jpg)
Sampling Theory
Harry Nyquist – 1889 to 1976 “The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth of the channel”
![Page 4: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/4.jpg)
Sampling Theory
Harry Nyquist – 1889 to 1976 “The number of independent pulses that can be put through a
telegraph channel per unit time is limited to twice the bandwidth of the channel”
Later Nyquist-Shannon “Exact reconstruction of a continuous-time baseband signal from
its samples is possible if the signal is bandlimited and the sampling frequency is greater than twice the signal bandwidth”
![Page 5: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/5.jpg)
Sampling Theory
Frequency
Amplitude
Fs
![Page 6: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/6.jpg)
Sampling Theory
Audio:20kHz bandwidth, Fs = 44.1kHz, 48kHz
Frequency
Amplitude
Fs
![Page 7: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/7.jpg)
Sampling Theory
Audio:20kHz bandwidth, Fs = 44.1kHz, 48kHz
Video:5.75MHz bandwidth, Fs = 13.5MHz30MHz bandwidth, Fs = 74.25MHz
Frequency
Amplitude
Fs
![Page 8: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/8.jpg)
Aliasing
Frequency
Amplitude
Fs
Nyquist
Frequency
![Page 9: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/9.jpg)
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
Frequency
Amplitude
Fs
Nyquist
Frequency
![Page 10: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/10.jpg)
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
These signals CANNOT be removed afterwards
Frequency
Amplitude
Fs
Nyquist
Frequency
![Page 11: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/11.jpg)
Aliasing
Frequencies above Fs/2 are “reflected” into the lower portion of the spectrum and become entangled with the low-frequency signals
These signals CANNOT be removed afterwards Filtering BEFORE sampling is needed
Frequency
Amplitude
Fs
Nyquist
Frequency
![Page 12: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/12.jpg)
Image Sampling
Horizontal - pixels
Vertical - lines
Temporal – frames
![Page 13: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/13.jpg)
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
![Page 14: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/14.jpg)
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Vertical resolutionSampling rate of 480, 576, 720, 1080 samples/picture height
• Resulting resolution of 240, 288, 360 or 540 cycles/ph
![Page 15: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/15.jpg)
Image Sampling
Horizontal resolutionSampling rate of 720, 1280, 1920 or 2048 samples/picture width
• Resulting resolution of 360, 640, 960 or 1024 cycles/pw
Vertical resolutionSampling rate of 480, 576, 720, 1080 samples/picture height
• Resulting resolution of 240, 288, 360 or 540 cycles/ph
Temporal resolutionSampling rate of 24, 25, 30, 50, 60 . . . samples/second
• Resulting resolution of 12, 15, 25, 30 cycles/sec
![Page 16: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/16.jpg)
Re-sampling
Image size changes are common
![Page 17: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/17.jpg)
Re-sampling
Image size changes are commonSimple example of interpolating a 1080 picture to 480:
• Input resolution is 540 cycles/ph
• Output resolution is 240 cycles/ph (division by 2.25)
Vertical Frequency
Am
plitu
de 1080
Vertical Frequency
Am
plitu
de 480
FilterPotential Alias
![Page 18: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/18.jpg)
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
![Page 19: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/19.jpg)
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
Alias terms are “folded” about the Nyquist point Inverted in frequency, inverted “movement”Highly noticeable to the human eye, which references its own
internal 3D model
![Page 20: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/20.jpg)
Re-sampling
Interpolation is only one part of the problemFiltering is needed to control the signal spectrum and avoid the
introduction of aliasesSimple interpolators are generally poor filters
Alias terms are “folded” about the Nyquist point Inverted in frequency, inverted “movement”Highly noticeable to the human eye, which references its own
internal 3D model
Alias terms left in the image will be shifted again in any subsequent operationsPotentially cumulative problems
![Page 21: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/21.jpg)
3D Sampling
Horizontal - pixels
Vertical - lines
Temporal – frames Restricted by
practical limitations
Linked by aspect ratio
and pixel shape
![Page 22: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/22.jpg)
Spatio-Temporal Sampling
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
![Page 23: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/23.jpg)
Spatio-Temporal Sampling
Filtering:Spatial – optical LPF and lens MTF
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
![Page 24: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/24.jpg)
Spatio-Temporal Sampling
Filtering:Spatial – optical LPF and lens MTFTemporal – integration time of sensor system
Spatial - lines
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Potential alias
Potential alias
![Page 25: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/25.jpg)
Spatio-Temporal Sub-Sampling
Temporal – frames
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Spectrum
Spatial - lines
Where is the filter?
Potential alias
Potential alias
![Page 26: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/26.jpg)
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
?
Spectrum
Temporal
Ver
tical
Horizontal
![Page 27: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/27.jpg)
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Adaptive filtering
?
Spectrum
Temporal
Ver
tical
Horizontal
![Page 28: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/28.jpg)
Up-conversion
Temporal Frequency
Spatial Frequency
Frame Rate
No of Lines
Adaptive filtering Motion compensation
?
Spectrum
Temporal
Ver
tical
Horizontal
![Page 29: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/29.jpg)
Format Interchange
Temporal Frequency
Spatial Frequency
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
![Page 30: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/30.jpg)
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
![Page 31: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/31.jpg)
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
Mixing formats such as film and video is to be avoided
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
![Page 32: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/32.jpg)
Format Interchange
Temporal Frequency
Spatial Frequency
Conversion between formats requires care
Mixing formats such as film and video is to be avoided
1080p down-conversion might raise new challenges
1080p
720p
Film
1080i480i
1080p (24)
15c/s 30c/s0c/s
0c/ph
250c/ph
500c/ph
![Page 33: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/33.jpg)
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
FrequencyAm
plitu
de 96
Filter
FrequencyAm
plitu
de 48
![Page 34: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/34.jpg)
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
![Page 35: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/35.jpg)
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
1080p allows similar gains for outputs of 720p and 1080iGood temporal filtering must introduce delay
![Page 36: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/36.jpg)
Over-sampling
Commonly applied to audio – eg 96kHz down to 48kHzAllows the use of a high performance digital filter:
1080p allows similar gains for outputs of 720p and 1080iGood temporal filtering must introduce delay
Film sampling at >1080 lines/ph also allows controlled down-sampling
![Page 37: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/37.jpg)
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
![Page 38: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/38.jpg)
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken
![Page 39: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/39.jpg)
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
![Page 40: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/40.jpg)
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
Poor anti-alias filtering leads to additional compression concatenation artefacts
![Page 41: Spatio-Temporal Quincunx Sub-Sampling.. and how we get there David Lyon](https://reader035.vdocuments.net/reader035/viewer/2022081519/56649dbf5503460f94ab3bf6/html5/thumbnails/41.jpg)
Conclusion
Spatio-temporal quincunx sub-sampling (aka interlace) is likely to be with us for some time
Modern cameras and processing can stress the format unless care is taken Imprinted alias is difficult (or impossible) to removeCamera integration is an important filter for interlace
Poor anti-alias filtering leads to additional compression concatenation artefacts
1080p down-conversion could make the stress worse