Video Digitization and Format ConversionVideo Digitization and Format ConversionVideo Digitization and Format ConversionVideo Digitization and Format Conversion
Yao WangPolytechnic University, Brooklyn, NY11201
© Yao Wang, 2003 Video digitization and format conversion
Outline
• How to capture digital video– Direct acquisition: digital video camcorder– Digitizing analog raster video– Digitizing analog film video (frame-based)
• Chrominance subsampling• Color coordinate • Digital video formats and applications• Video format conversion
– Deinterlacing– NTSC <-> PAL/SECAM– SDTV <-> HDTV
© Yao Wang, 2003 Video digitization and format conversion 3
A2D Conversion
• Digitization = Sampling + Quantization
xc(t) x[n] = xc(nT) �[ ]x n
Quanti-zation
Sampling
SamplingPeriod
T
QuantizationInterval
Q
© Yao Wang, 2003 Video digitization and format conversion 4
Illustration of Sampling and Quantization
0 0.2 0.4 0.6 0.8 1
-1
-0.5
0
0.5
1T=0.1Q=0.25
© Yao Wang, 2003 Video digitization and format conversion 5
Video Raster Revisited
Field 1 Field 2
Progressive Frame Interlaced Frame
How to convert a raster video to digital signal?
Horizontal retrace
Vertical retrace
© Yao Wang, 2003 Video digitization and format conversion 6
Digitizing A Raster Video
• Digitization = Sampling + Quantization• Sample the raster waveform = Sample along the horizontal
direction• Apply the above sampling on Y,I,Q rasters separately• Quantize Y,I,Q samples to 8 bits each• How should we select the sampling rate?
– Must be faster than the Nyquist rate (twice the highest frequency)– For the samples to be aligned vertically, the sampling rate should
be multiples of the line rate– Horizontal sampling interval = vertical sampling interval (square
pixel)– Total sampling rate equal among different systems (525/30 vs
625/25MHz 5.13)(PAL/SECAM864(NTSC)858 === lls fff
© Yao Wang, 2003 Video digitization and format conversion 7
BT.601* Video Format
480
lines
525
lines
122pel
16pel
858 pels720 pels
ActiveArea
525/60: 60 field/s57
6 lin
es
625
lines
864 pels
132pel
12pel
720 pels
ActiveArea
625/50: 50 field/s
* BT.601 is formerly known as CCIR601Pixels in non-shaded areas correspond to samples taken during horizontal/vertical retrace
© Yao Wang, 2003 Video digitization and format conversion 8
Color Coordinate: YCbCr
• YCbCr = Digital equivalent of YUV– Cb = U = B-Y, Cr = V = R- Y– Each represented in 8 bits
© Yao Wang, 2003 Video digitization and format conversion 9
RGB <--> YCbCr
Y_d = 0.257 R_d + 0.504 G_d + 0.098 B_d + 16,C_b = -0.148 R_d - 0.291 G_d + 0.439 B_d + 128,C_r = 0.439 R_d -0.368 G_d - 0.071 B_d + 128,
R_d = 1.164 Y_d’ + 0.0 C_b’+ 1.596 C_r’,G_d = 1.164 Y_d’ - 0.392 C_b’ -0.813 C_r’,B_d = 1.164 Y_d’ + 2.017 C_b’ + 0.0 C_r’,
Y_d’=Y_d -16, C_b’=C_b-128, C_r’=C_r-128
© Yao Wang, 2003 Video digitization and format conversion 10
Sampling of Chrominance Components
• Human eye is less sensitive to chrominance• Chrominance signal has lower bandwidth
– Can be sampled at lower rate than Y
© Yao Wang, 2003 Video digitization and format conversion 11
Chrominance Subsampling Formats
4:2:0For every 2x2 Y Pixels
1 Cb & 1 Cr Pixel(Subsampling by 2:1 bothhorizontally and vertically)
4:2:2 For every 2x2 Y Pixels
2 Cb & 2 Cr Pixel(Subsampling by 2:1
horizontally only)
4:4:4 For every 2x2 Y Pixels
4 Cb & 4 Cr Pixel(No subsampling)
Y Pixel Cb and Cr Pixel
4:1:1For every 4x1 Y Pixels
1 Cb & 1 Cr Pixel(Subsampling by 4:1
horizontally only)
I,Q raster are sampled at ½rate
I,Q raster are sampled at ¼rate
I,Q raster are sampled at the same rate as Y
Vertical down-sampling from 4:2:2
© Yao Wang, 2003 Video digitization and format conversion 12
Digital Video Formats
Video Format Y Size Color Sampling
Frame Rate (Hz)
Raw Data Rate (Mbps)
HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps SMPTE296M 1280x720 4:2:0 24P/30P/60P 265/332/664 SMPTE295M 1920x1080 4:2:0 24P/30P/60I 597/746/746 Video production, MPEG2, 15-50 Mbps BT.601 720x480/576 4:4:4 60I/50I 249 BT.601 720x480/576 4:2:2 60I/50I 166 High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps BT.601 720x480/576 4:2:0 60I/50I 124 Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps SIF 352x240/288 4:2:0 30P/25P 30 Video conferencing over ISDN/Internet, H.261/H.263/MPEG4, 128-384 Kbps CIF 352x288 4:2:0 30P 37 Video telephony over wired/wireless modem, H.263/MPEG4, 20-64 Kbps QCIF 176x144 4:2:0 30P 9.1
© Yao Wang, 2003 Video digitization and format conversion 13
Video Format Conversion
• Needed for various applications– Deinterlacing (interlaced -> progressive)– NTSC (525/50 ) <-> PAL (625/60)– SDTV <-> HDTV
• Using digital interpolation filters!– Filtering within one frame (spatial interpolation)– Filtering across frames for the same pixel (temporal
interpolation)– Spatial-temporal filtering
© Yao Wang, 2003 Video digitization and format conversion 14
Why Deinterlacing
• Needed for – Display interlaced sequence in progressive monitor with
twice frame rate– Convert old TV material to SDTV/HDTV with good quality– Convert between NTSC and PAL with good quality– …
© Yao Wang, 2003 Video digitization and format conversion 15
Deinterlacing Problem
From [Wang02]
© Yao Wang, 2003 Video digitization and format conversion 16
Deinterlacing by Temporal Interpolation
• Field merging:– Merge two fields into one frame, repeats the frame twice– Equivalent to temporal interpolating using replication filter
• Field averaging– Averaging corresponding lines in preceding and following fields– Equivalent to temporal interpolation using linear averaging filter – Require two frame store !
• Both methods have problem when there are significant motion between two fields, especially moving vertical lines
• But can recover spatial details on stationary objects very well
© Yao Wang, 2003 Video digitization and format conversion 17
100 100 50
100 100 100
100 100 100
100 100 100
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 100 50
100 50 50
100 50 50
100 50 50
100 100 50
100 100 100
100 100 50
100 100 100
100 100 100
Frame 1 Frame 2
Field 1 Field 2 Field 1 Field 2
An Example for Illustration
© Yao Wang, 2003 Video digitization and format conversion 18
100 100 50
100 100 100
100 100 100
100 100 100
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 50 50
100 75 50
100 100 50
100 75 75
100 100 50
100 75 50
100 50 50
100 50 50
100 50 50
100 100 50
100 100 100
100 100 50
100 100 100
100 100 100
Frame 1 Frame 2
Field 1 Field 2 Field 1 Field 2
Deinterlacing by field averaging
Missing line3 = (field1 of frame1+field1 of frame2)/2
© Yao Wang, 2003 Video digitization and format conversion 19
Deinterlacing by Spatial Interpolation
• Line averaging within same field– D=(C+E)/2– Equivalent to vertical interpolation using linear averaging
filter– Will not be able to recover fine details (e.g. alternating
horizontal lines)– Can recover moving vertical lines very well
• Higher order vertical filter can be used to improve quality
• Line/field averaging– D=(C+E+K+R)/4– The motion and spatial blur are both introduced but not as
obvious, on average may be better
© Yao Wang, 2003 Video digitization and format conversion 20
100 100 50
100 100 100
100 100 100
100 100 100
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 100 50
100 100 50
100 100 50
100 100 50
100 50 50
100 50 50
100 50 50
100 100 50
100 100 100
100 100 50
100 100 100
100 100 100
Frame 1 Frame 2
Field 1 Field 2 Field 1 Field 2
Deinterlacing by line averaging
Missing line 3= (line2+line4)/2
© Yao Wang, 2003 Video digitization and format conversion 21
100 100 50
100 100 100
100 100 100
100 100 100
100 100 50
100 50 50
100 100 50
100 50 50
100 100 50
100 50 50
100 88 50
100 100 50
100 88 62.5
100 100 50
100 88 50
100 50 50
100 50 50
100 50 50
100 100 50
100 100 100
100 100 50
100 100 100
100 100 100
Frame 1 Frame 2
Field 1 Field 2 Field 1 Field 2
Deinterlacing by field and line averaging
Missing line 3= (line2+line4+field1,frame1+field1,frame2)/4
© Yao Wang, 2003 Video digitization and format conversion 22
Field 1 of Frame1
Field 2 of Frame1
© Yao Wang, 2003 Video digitization and format conversion 23
Deinterlaced Field 2 by Merging Field 1 and Field 2
© Yao Wang, 2003 Video digitization and format conversion 24
Deinterlaced Field Using Field Averaging
Deinterlaced Field Using Line Averaging
© Yao Wang, 2003 Video digitization and format conversion 26
Deinterlaced Field Using Line and Field Averaging
© Yao Wang, 2003 Video digitization and format conversion 27
Matlab Implementation
• Go through matlab script at class– deinterlacing.m
© Yao Wang, 2003 Video digitization and format conversion 28
More Advanced Deinterlacing Methods
• Motion adaptive:– Switch between temporal/spatial interpolation depending on
motion detection results
• Edge adaptive– For spatial interpolation: interpolate along edge direction
© Yao Wang, 2003 Video digitization and format conversion 29
PAL -> NTSC
13125=625*21=525*25 300=50*6=60*5
From [Wang02]
© Yao Wang, 2003 Video digitization and format conversion 30
625 -> 525 lines
From [Wang02]
© Yao Wang, 2003 Video digitization and format conversion 31
50 -> 60 fields
From [Wang02]
© Yao Wang, 2003 Video digitization and format conversion 32
24 frames -> 60 fields
From [Wang02]
© Yao Wang, 2003 Video digitization and format conversion 33
What you should know
• What are the factors considered when choosing the sampling rate for digitizing a raster video?
• Why should we sample chrominance components at lower sampling rate than the luminance component?
• What are the difference between 4:4:4, 4:2:2, 4:1:1, 4:2:0 colorformats?
• What are some of the common video formats and their intended applications?
• What is deinterlacing? What are some simple deinterlacingmethod (should be able to do calculation on example images and understand matlab scripts)
• What are some other format conversion problems? How are they accomplished?
© Yao Wang, 2003 Video digitization and format conversion 34
References
• Y. Wang, J. Ostermann, Y. Q. Zhang, Video Processing and Communications, Prentice Hall, 2002. chapter 1, 3, 4. Mainly Sec. 1.5.