system architecture for isp hardware system architecture for isp hardware fpga based isp simulator...

3. ISP Hardware Design & Verification

• System Architecture for ISP Hardware• FPGA Based ISP Simulator• FPGA Based ISP Simulator GUI Com-

ponets• Design & Verification for ISP Functional

Blocks

System Architecture for ISP Hardware

- Total 12 functional blocks

- Sytem clock : pixel clock

- Input : 8-bit bayer

- Output : 16-bit 4:2:2 YCbCr

- Functional block control through

I2C

- Function selector generates

“enable” singal to each

fucnctional block of ISP

- Operational frequency for blocks : More than 100 MHz

Hardware architectue for ISP sytem

Synthesis Results of ISP Hareware Design

- Op. freq. : 105.85MHz

- Critical path : AWB

Setup time of the ISP functional block measured in slow model.

Resource usage

Item Value

Total logic ele-ments 10,338

Total registers 7,431

Total memory bits 316,896

FPGA Based Real-time ISP Simulator

FPGA Based ISP Simulator

① Test Enveronment : ╶ 8M CIS ╶ PC╶ DE2-70 FPGA board╶ HyImage V4P board

② HyImage interface board :╶ Communication bet. PC

& FPGA

③ Operation speed :╶ In case of full resolution : 7~8 Frames per sec.╶ In case of down scaling : ╶ 10 Frame per sec.

FPGA Based ISP Simulator

PCB Board Design for FPGA-PC Con-nection

gpio0, gpio1 ╶ FPGA board GPIO

Required of PCB de-sign for FPGA board I/O with a fixed power pins(3.3V)

PCB board for FPGA-PC conncetion

Connection information for PCB mode

FPGA Based ISP Simulator GUI

ISP Simulator GUI

①

②

③

④

⑤

⑥

No.

Main function Details

① Down scaling - Full/down scaling selection

② Color interpo-lation

- 5x5 adaptive interpolation- 3x3 bilinear interpolation

③ Option- Selection for Gamma value- Setting weight for Edge en-hancement

④ Auto focus - Selection for operation

⑤ AE_Option - Setting for in-door/out-door

⑥ ISP Function - Activate each functional block

Edge enhancement weight value setting window

Gamma value setting window

FPGA Based ISP Simulator GUI

Gamma value setting window- possible from 0.1 through 0.9

Edge enhancement set-ting window- Value setting: 1 ~ 5 - Increase by 0.5

ISP Functional Block Design & Verification – Color Interpolation

Applied Algorithm╶ 3ⅹ3 bilinear color interpolation

Process① Line buffer : 3 line data

store② 3x3 data generator : 3x3

data matrix generation③ Bilinear operator : bilinear

interpolation ⇒ Applying total 4-level pipe-

line

ISP Functional Block Design & Verification – Color Interpolation

Program Source Analysis╶ Hardware file configuration

⇒ inter_3_3.v⇒ buffer_3_3_8bit.v : 3x3 buffer⇒ dpsram3200x8_2.v : 3x3 buffer memory simulation file gen-

eration (Modelsim file)⇒ inter_3_3.v⇒ buffer_3_3_8bit.v⇒ dpsram3200x8_2.v⇒ image_gen.v : 8bits bayer data generation fil⇒ inter_tb.v : Test-bench top file

╶ Simulation File Generation(Matlab file)⇒ rgbtogbrg.m : bayer pattern file generation from image-file in-

put⇒ interpolation_3_3.m : Result image generation after perform-

ing 3x3 interpolation⇒ image_out.m : Converting hardware-based 3x3 interpolation

results to a corresponding image file

ISP Functional Block Design & Verification- Color interpolation

5x5 adaptive color interpolation

Hardware file composition╶ color_inter_new.v : color interpolation top file╶ buffer_5_5.v : 5 line buffer╶ dpsram3200x16.v : memory for 5-line buffer╶ pattern_gen.v : generation pattern value according to

bayer pattern╶ param.h : Store 8 masks

Simulation file composition╶ 4 higher level files╶ inter_tb.v : Top test-bench file╶ image_gen.v : bayer pattern input generation

ISP Functional Block Design & Verification-Color interpolation

Simulation file configuration╶ Matlab file╶ rgbtogbrg.m : generate bayer pattern file from image file input╶ interpolation_5_5.m : Result image generation after 5x5 inter-

polation ╶ image_out.m : Converting hardware based 5x5 interpolation

results to image file

ISP Functional Block Design & Verification– Edge enhancement

Applied algorithm╶ Unsharp mask filter

Performing process① Buffer 3x3

╶ Store 3-line data

② 3x3 data generator ╶ Generate 3x3 data matrix

③ Sum╶ 3-line summation

④ Divider╶ Get average data

⑤ Add multiplication result of weight and subtracted value from original pixel data to the original pixel

⑥ Total 19-stage pipeline

Buffer3x3(2 clk cycles)

PCLK

PRESETn

PI_VS

PI_EN

YCbCr_in[15:0]

Y_in1

Y_in2

Y_in3Sum

3x3 array Divide

by 9

1 clk cycles 1 clk cycles 12 clk cycles

sum mean Calculate output pixel

Delay 13 clk cyclesP_11

CbCr_delay Delay 3 clk cycles

Y_reg

CbCr_out

2 clk cycles

Delay 17 clk cycles

PO_VS

PO_EN

vsync_mem

hsync_mem

Y

Litmit0_255

Y_out

1 clk cycles

Y_delay

ISP Functional Block Design & Verification- Auto White Balance

Gray World Algorithm

Hardware file composition╶ AWB.v : AWB top file╶ divider.vhd : division calculation

Simulation file composition╶ awb_lee3.v : AWB top file╶ awb_lee3_tb.v : Test bench top file╶ divider.vhd: 21bit / 11bit╶ dpsram3200x16.v : memory for 5-line buffer╶ im_64x64.txt : Text image file╶ RGB_image_gen.v : RGB pattern input generation╶ RGB_image_write.v

ISP Functional Block Design & Verification - Auto White Balance

Simulation file composition╶ Matlab file * im2text.m : convert an image file to a text file * text2im.m : convert a text file to an image file * compare_result.m : result comparison

GWA AWB 하드웨어 구조

ISP Functional Block Design & Verification-Auto White Balance

Source analysis╶ See reference files

- Applying 1-stage pipeline

- Consists of 5 sub-blocks

- Signal generator : generate a frame com-pletion signal

- FIFO : Perform delay for Divider block

- Accumulator : perform total summation R, G, B pixels

- Shifter : perform average of R, G ,B

- Divider : divide R, G, B average total by each R, G, B average

Signal generator Shifter (22 right)

Divider

Accumulator

FIFO

PI_

HS

PI_

VS

PI_R

PI_G

PI_B

End_of_frame

R_g

ain

G_g

ain

B_g

ain

x x xPI_R

PI_G

PI_B

Weighted_R Weighted_G Weighted_B

ISP Functional Block Design & Verification– Edge enhancement

Program source analysis╶ Hardware file composition

⇒ Edge_enhance.v : edge_enhance.top file⇒ divider.vhd : division calculation⇒ dpsram3200x32.v : Line buffer memory⇒ FIFO.v : FIFO process

╶ Simulation file composition(Modelsim file)⇒ buffer3_3.v : 3 line buffer ⇒ edge_en_tb.v : edge enhance top file⇒ FIFO.v dpsram3200x32.v edge_enhance.v ⇒ divider.vhd YCbCr_image_gen.v : YCbCr image generation⇒ Reg_Nbits.v YCbCr_image_write.v

╶ Simulation file composition(Matlab file)⇒ im2textYCbCr.m : convert into test file after converting image file

to YCbCr⇒ textYCbCr2im.m : convert text file to YCbCr image file⇒ compare_result.m : Result comparison

그림 틀

ISP Functional Block Design & Verification– Auto Focus

Applied algorithm╶ Tenengrad (calculating focusing value)╶ Global search(Movement of focusing position)

Performing process① 3 line Buffer : store 3-line

Y data ② 3x3 data generator :

generate 3x3 data matrix③ Focus value calculating :

calculate focus value per frame

④ Main operator : setting the moving range for ac-tuator step

⑤ I2C Controller : activate actuator step movement

ISP Funcional Block Design & Verification– Auto Focus

Program source analysis╶ Hardware file composition

⇒ global_se.v : autofocus.top file⇒ focus_val.v : calculate focus value ⇒ I2C_AF.v : Moving Motor⇒ dpsram3200x16: Line buffer memory⇒ af_mem.v : memory access

╶ Simulation file composition(Modelsim file)⇒ Imposible to derive program by Modelsim (register control)

ISP Functional Block Design & Verification 기– Gamma correction

Applied algorithm╶ Piece-wise linear Gamma-correction

Performing process

Top block for gamma correction

ISP Functional Block Design & Verification– Gamma correction

Performing process① Step decision

╶ Gamma correction performing by variousLinear partitions, not by gamma curve

+()=()()()

② Calculating 8-step output pixel using Gamma table

Piece-wise linear gamma correction

4. Embedded System De-sign & Verification

• Software ISP Simulator• Software ISP Simulator GUI• Hardware ISP 설계 및 검증

ISP Embedded System

Block diagram of an ISP Em-bedded System

1. SoCBase 1.0 Based Design

2. Process : ARM926EJS

3. FPGA : Virtex4LV80

4. Design block :

- ISP top

·I2C controller

·ISP functional block control

·ISP function blocks

- Dualport SRAM controller

- VGA controller

Design Details

System spec.

Design Tool

SoCBase 1.0

SoC base 1.0

1. Single-processor based SoC de-

velopment application platform

2. Provided with total 26 IPs

· AMBA bus

· Memory controller

· External unit controller

Hardware Design & Verification

Dual port SRAM controller

Dual port SRAM controller Stuc-ture

① dual port SRAM con-troller design for high-speed store/output of image data

② Use dual port FIFO ╶ - 8192 x 32 bits

③ Generate SRAM con-trol data from Write/read data controller

④ Required in case for output of high-resolu-tion image

Hardware Design & Verification

VGA controller

╶ Clock generation through control of ICS 307 chip input sig-nal

╶ Clock generation of 25MHz for 640x480 resolution output╶ VGA controller usage example : Refer to HBE-SoC-IPD

Users Guide p.250 ╶ ICS 307 chip manual : HBE-SoC-IPD Users Guide p.265 참조

Ics_307block

Vga dataCtroller

Top_VGA

iCLK_25Ics_sclk

ics_data

ics_strobe

oVGA_R[7:0]oVGA_G[7:0]

oVGA_B[7:0]oVGA_HS

oVGA_VSoVGA_BLANK

oVGA_SYNK

oVGA_CLOCK

iRST

Ics_307

chipClk_108

SRAM-VGA Simulation

╶ Source file location : ⇒ sram_vga_test folder

╶ Input file : ycbcr_in.txt╶ Output file : image_out.txt╶ SRAM model generation╶ Confirm image through Matlab

RGB

Text In-put

Image generator

Dual port memory controller

VGA con-troller

SRAM model Text

Output

24bits RGB out-

put

YCbCr YCbCr YCbCr

YCbCr

Color con-version

RGB

SRAM-VGA Simulation

Simulation result

Original image(3200x2408)

Resulting image(800x600)

Down scaling

Pin Configuration

Simulator

VGA

ARMFPGAUAR

TCIS

ISP embedded system verification environ-ment

① Access slave address through Multi-ICE② Control ISP functional blocks through embedded soft-

ware

Future Study

PCB Connection bet. CIS-IPD board╶ Need to remove noise of output images

SDRAM memory based dualport memory controller design╶ Need to increase output resolution

ISP functional block software/hardware partition╶ SystemC based

RT level simulation╶ Use Seamless CVE based virtual core