Embedded Software for Video

Wayne WolfPrinceton University and MediaWorks Technology

Outline

Initial observations.Sample application: smart cameras.Levels of abstraction in software:

process; data; instructions.

Lessons.

Characteristics of embedded computing

Functional and non-functional requirements: performance, power, cost.

Some aspects of design methodology are similar to hardware design.

Embedded software delivers more complex functions than HW models.

Will often be delivered in chips---that’s where the money is.

Ozer et al: human activity recognition

JPEG IMAGE/MPEG VIDEO

UncompressedIMAGE/VIDEO

EIGENSPACE MATCHING

PRINCIPALCOMPONENTS

RELATIONALGRAPH MATCHING

Motion Vectors

DCTExtracted Area

Extracted Area

Pixel Values

OBJECT DETECTIONOBJECT DETECTION

OBJECT DETECTIONOBJECT DETECTION

Previous work

Watlington et al.: Dataflow architecture for real-time video processing.

Wandell et al.: Programmable sensor array.

Foote et al.: Panaromic camera system.

Nicolescu et al.: Algorithms to pan, tilt, zoom array of cameras.

MIT Media Lab.: Smart rooms for several applications.

Keck Lab.: Multiple cameras on networked PCs.

Our hardware platform

TriMedia BoardTM32 (VLIW)

PC

I Bu

s Host PC

CameraNTSC Shared

Memory

TriMedia BoardTM32 (VLIW)

SharedMemory

CameraNTSC

Background elimination

Skin color detection

contour following

Superellipse fitting

graph matching

Background elimination

Skin color detection

Contour following

Superellipse fitting

Graph matching

Skin color detection Contour following Superellipse fitting

time

Skin detection CPU time

Contour following CPU time

Superellipse fitting CPU time

Graph matching CPU time

Characteristics of current video computing

Streaming data (with control).

A wide range of types of algorithms.

Data volum

eA

lgorithm com

plexity

Challenges in embedded video software

Video is more than pushing pixels.Programs run a long time---harder to

verify.Must derive programs through many

stages from functional to real-time.Must effectively use a deep memory

hierarchy: registers through off-chip.

Characteristics of next-generation video systems

Distributed computation and networking: Distributed analysis of video. Data fusion.

Design methodology for this system

Algorithm design: Matlab.C implementation on existing

platform.Software optimization.Design of custom platform and

software.

Memory hierarchy management

On-ChipMemory

Off-ChipMemory

Cache

Cache

Buffers

Registers

Levels of abstraction and optimization

Instruction: opcode selection, register allocation.

Data: loop transformations, data layout, buffer optimization.

Module: inlining, outlining.Process: decomposition,

communication, etc.

Questions from smart camera

Instructions: Which loops are predictable and which

are not?Data:

How does data layout change at each stage?

How do we buffer and lay out data?Process:

How do we manage multiple processes?

View from a start-up

Standards body software sucks.Configurable CPUs: offer some

advantages, but harder to characterize.Embedded OS verification: WinCE, Linux.Must manage multiple versions of

software.Can’t buy many useful components.Verification, verification, verification.

ACM TECS

Look for the TECS Web site soon atwww.ee.princeton.edu/~tecs