mipi devcon 2016: image sensor and display connectivity disruption

Image Sensor and Display Connectivity Disruption

Use Case Trends from a Continually Evolving

Mobile Ecosystem

Grant Jennings Lattice Semiconductor

Agenda •  Image Sensor and Display Evolution

•  Influence of Mobile on Horizontal Markets •  Permutations of Interface Standards

• Enabling Innovation Through Bridging •  Image Sensor and Display Use Case Challenges •  Bridging Use Case Examples

• Enabling Innovation Through Programmability • Future Considerations

2

3

2-5 Cameras 1-2 Displays

Cost/Power/Size Critical

1-4 Display Interfaces 0-16 Camera Interfaces

Varying Requirements

Difficult to find the right logic devices to accommodate all applications and ideas

Image Sensor and Display Evolution Influence of Mobile on Horizontal Markets

4

Smartphone Pre-Smartphone

Source Synchronous

CMOS

CSI-2 MIPI DSI

D-P

HY MIPI DPI

RG

B PROPRIETARY

MIPI DBI

GPIF SubLVDS

HiSPi

SLVS PROPRIETARY

Larger Systems

Embedded Clock

SLVS-EC

FPD-LINK III

eDP

PROPRIETARY

SDI ETHERNET

M-P

HY

Difficult to find build IP/interface knowledge base

Image Sensor and Display Evolution Permutations of Interface Standards

C-PHY LVDS





5

6

Resolving unforeseen issues Enabling backward compatibility

Solving new problems

Bridge IC DSLR PROCESSOR

Sub-LVDS IMAGE SENSOR

D-PHY

Incompatible inputs

MCU SPI D-PHY

DISPLAY

Incompatible outputs

Bridge IC

APPLICATION PROCESSOR

D-PHY DUAL D-PHY

DISPLAY

Insufficient outputs

D-PHY

D-PHY

MOBILE PROCESSOR

D-PHY

Insufficient inputs

Rig

ht E

ye

Left

Eye

Bridge IC Bridge IC

Enabling Innovation Through Bridging

7

Camera #2

Camera #N

.

.

.

Processor

Reference Clock

MIPI CSI-2

MIPI CSI-2

MIPI CSI-2

MIPI CSI-2

I2C

I2C

I2C

I2C

Sync

Sync

Sync

APPLICATIONS

Enabling Innovation Through Bridging Multiple Cameras

BandwidthAggrega.on,FrameSynchroniza.on,VirtualChannels,I2CAddressing

Bridge

Camera #1


8

Bridge

Processor

Camera

Proprietary Interface MIPI CSI-2

I2C

SPI

Readout Control

APPLICATIONS

Proprietary/Custom Bridging

FrameReadoutControl,I2CtoSPI,BandwidthAggrega.on,Pixelvs.ByteSerializa.on

9

DSI to SPI/CMOS

SPI to DSI

CSI-2 to SPI/CMOS Sensor

BridgeuPObject

Detection

SPI CSI-2

Display Bridge

Frame Data

Display Controller

DSI SPI uP

Specialty Display

Bridge CMOS DSI AP/uP

APPLICATIONS

Enabling Innovation Through Bridging Low Speed Peripheral Interfacing

BandwidthReduc.onThroughCo-Processing,Master/Slave,CustomizedInterfacing

10

AP DSI

DSI

DSI

Display

Display

X Gbps

X/2 Gbps

X/2 Gbps

SpecialtyImageProcessor

DSI

DSI

CSI-2

Display

AP,Bridge,etc.

X Gbps

X Gbps

X Gbps

Display DSI

CSI-2

AP

CSI-2 Camera

APPLICATIONS

Bridge

Bridge

Bridge

Enabling Innovation Through Bridging Splitters or Bandwidth Reducers

SpliIng,ConfigurableInterfaceIP,Cropping/Scaling,ClockDomainCrossing


11

Applica:onProcessor

Bridge

DCS Commands for New Display

DSI DSI

BridgeRAW to YUV CSI-2 CSI-2

Cropping

Crop/Scale

Packet Repair

Continuous Clock

APPLICATIONS

Display

Applica:onProcessor

Camera

Configura.on/Control,DataTypeConversion,ShortPacketRepair,EnablingTermina.on

Repeaters





12

COST

POWER

PERFORMANCE

SIZE

13

CSI-2

DSI RAW

YUV

YCbCr I2C SPI I3C

D-PHY

C-PHY SLVS

HiSPi

8bpp 10bpp

12bpp 14bpp

18bpp 24bpp HS signaling

DPCM Virtual Channels

Embedded Data

Encoding

1 lane 3 lane 2 PHYs

RAW

1 Logic Device

Clocking scheme

Developing a bridge IC for a singular use case often not cost effective

Hard to find silicon for non-conforming cases Hard to develop IP and architectural knowledge

1000s of permutations

IMAGE SENSOR

CAMERA INTERFACE BRIDGE DISPLAY INTERFACE BRIDGE

IMAGE SENSOR BRIDGING EXAMPLES DISPLAY SENSOR BRIDGING EXAMPLES


1:2 Display Splitter Bridge


1:2 High-Resolution Display Bridge High-Resolution Camera Bridge


Programmable Bridge

Programmable Bridge

Programmable Bridge

Programmable Bridge

Programmable Bridge

Programmable Bridge



Display

Display

Display

Display

Single Programmable Video Bridge Handles Many Use Cases With Single Device

14

FPGAs ASIC/ASSPs

Time-to-MarketCostFlexibility

Performance

Power/Size

The pASSP Continuum The Best of Both Worlds

15

16

Programmable IO

RX: D-PHY / SubLVDS / LVDS / SLVS200 / CMOS

TX: LVDS / CMOS

1.2 Gbps/Lane 14 IO / 7 Pairs

MIPI D-PHY

6 Gbps RX & TX

4 data lanes 1 clock lane

MIPI D-PHY

6 Gbps RX & TX

4 data lanes 1 clock lane

GPIOs I2C/SPI

Programmable FPGA Fabric

5,936 LUTs 180 kbits block RAM

47 kbits distributed RAM

Enough FPGA resources to handle video: Muxing Merging

Demuxing Arbitration Splitting

Data Conversion Custom Protocol Design

Programmable IO

RX: D-PHY / SubLVDS / LVDS / SLVS200 /

CMOS

TX: LVDS / CMOS

1.2 Gbps/Lane 16 IO / 8 Pairs

Power Management Unit

CrossLink

Must have right amount and types of interfaces Must have right amount of programmability

Example – Lattice CrossLink™ Block Diagram Problem 1 – Having the right device

17

Display IPs Camera IPs DSI to DSI Dual CSI-2 to CSI-2

DSI to Dual DSI CSI-2 to Parallel CMOS

Dual DSI to Dual DSI Parallel CMOS to CSI-2

DSI to LVDS SubLVDS to CSI-2

DSI to Dual LVDS

Dual DSI to Dual LVDS

DSI to DPI

Single LVDS to DSI

Dual LVDS to DSI

Dual LVDS to Dual DSI

DPI to DSI

Must have end-to-end solutions for common use cases

IPs and Starting Points Problem 2 – Having the Right IP

Configurable Data Types (RAW, YUV, RGB, YCbCr, Custom, Bits per Pixel) Configurable Number of Data Lanes and PHYs Adjustable Input & Output Frequencies DCS Command ROM IP Customization

18

36 WLCSP 64 ucfBGA 81 csfBGA 80 ctfBGA

0.4 mm pitch 0.4 mm pitch 0.5 mm pitch 0.65 mm pitch

2.46 x 2.46 mm 3.5 x 3.5 mm 4.5 x 4.5 mm 6.5 x 6.5 mm

36 WLCSP 80 ctfBGA

Can’t pay a penalty for using programmable logic

Small Form Factor Problem 3 – Power, Cost, Size

Today’s Bridging Devices Need to be Better than Stand-Alone ASSPs

1X

Quicker Time to Market Custom solutions developed in real time

Same Low Power Optimized use cases

Same Behavior Firmware retention in the device

Smaller Form Factor WLCS and flip chip packages

Same Low Cost Pad limited not gate limited

ASSP RAW, YUV

8, 10, 12 bpp 1-lane, 2-lane, 4-lane

Continuous, Non-Continuous

pASSP

Firmware #1 YUV, 8 bpp, 1-lane

Firmware #2 RAW, 16 bpp, 4-lane

Firmware #2 JPG, 8 bpp, 2-lane

CFG vs.

19

Example of ASSP Versus pASSP Paradigms:





20

Future Considerations

21

•  Ecosystem standards enable innovation •  Cost, power, size •  Brings commonality to invoke new uses

•  FPGA programmability enables standards adoption •  Reduces risks with backward compatibility

•  FPGA programmability enables innovation •  New and unforeseen use cases made possible

•  FPGA programmability is inexpensive •  With the right architecture, same as ASSP

•  Help FPGA programmability help you •  Working together though MIPI to enable the next wave of use cases

Thank You Connecting Everything to Anything….

22