global edge ian carvalho architect 14 october 2010 copyright © 2010, global edge software ltd.,...

Global Edge

Ian Carvalho

Architect

14 October 2010

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Version 1.0

ID 730L: Getting Started with Multimedia Programming on Linux on SH7724

Copyright © 2010, Global Edge Software Ltd., Bangalore, India

Mr. Ian Carvalho

System Architect, Global Edge Software Ltd.

Responsible for Technology Strategy

Currently Driving Development Programs for:

Video Telephony

Video Surveillance

Wi-Fi Media Streaming

Previous Experience

Embedded System Development

Wi-Fi Software Development

VoIP Stack & STB Application Development

2


Renesas Technology and Solution Portfolio

Microcontrollers& Microprocessors

#1 Market shareworldwide *

Analog andPower Devices#1 Market share

in low-voltageMOSFET**

Solutionsfor

Innovation

Solutionsfor

InnovationASIC, ASSP& Memory

Advanced and proven technologies

* MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010

** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis).

Copyright © 2010, Global Edge Software Ltd., Bangalore, India4

Renesas Technology and Solution Portfolio

Microcontrollers& Microprocessors

#1 Market shareworldwide *

Analog andPower Devices#1 Market share

in low-voltageMOSFET**

ASIC, ASSP& Memory

Advanced and proven technologies

* MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010

** Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis).

Solutionsfor

Innovation

Solutionsfor

Innovation

5 © 2010 Renesas Electronics America Inc. All rights reserved.

5

Microcontroller and Microprocessor Line-up

Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive

Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 uA standby Medical, Automotive & Industrial

Legacy Cores Next-generation migration to RX

High Performance CPU, FPU, DSC

Embedded Security

Up to 10 DMIPS, 130nm process350 uA/MHz, 1uA standbyCapacitive touch

Up to 25 DMIPS, 150nm process190 uA/MHz, 0.3uA standbyApplication-specific integration

Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security

Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 uA standby Ethernet, CAN, USB, Motor Control, TFT Display

High Performance CPU, Low Power

Ultra Low PowerGeneral Purpose

6 © 2010 Renesas Electronics America Inc. All rights reserved.

6

Microcontroller and Microprocessor Line-up

Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive

Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 uA standby Medical, Automotive & Industrial

Legacy Cores Next-generation migration to RX

High Performance CPU, FPU, DSC

Embedded Security

Up to 10 DMIPS, 130nm process350 uA/MHz, 1uA standbyCapacitive touch

Up to 25 DMIPS, 150nm process190 uA/MHz, 0.3uA standbyApplication-specific integration

Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security

Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 uA standby Ethernet, CAN, USB, Motor Control, TFT Display

High Performance CPU, Low Power

Ultra Low PowerGeneral Purpose

SuperH

Innovation

7


Agenda

SH7724 Platform Features SH7724 (ECOVEC) Kit Contents Included Software Multimedia on Linux Frameworks & Libraries LAB Session

Setting up the Hardware Platform Setting up the Development Environment Linux Kernel Configuration and Compilation Writing a Media Application

– Capture, Scale and Display Demonstration using SH Codecs

Video Capture, Encode, Decode, Scale and Display Conclusion

Optimizations Q & A

8


Key Take Aways

9


SH7724 Block Diagram

10


Platform Features

SH7724 SoC Features 32-bit Superscalar RISC CPU with 7-

Stage Pipeline 900 DMIPS @ 500 MHz 3.5 GFLOP single/double precision

FPU with 10 stage pipeline 32KB I-Cache & 32KB D-Cache 256KB L2 Cache MPEG4, H.264 and WMV Accelerators

with 720p HD quality JPEG Accelerator 2D Graphics Accelerator LCD Controller Two camera interface, up to 5M pixel

sensors Sound I/O Module Ethernet MAC (10/100Mbps) ATAPI Interface 2 x USB 2.0 High Speed Host /

Function Controller eMMC 4.2 NAND Interface SD/MMC and SDIO Interface

Development Platform Features Memory

NOR Flash: 64MB 16-bits DDR2 SDRAM: 256MB (optional

512MB) 10/100 Ethernet Interface Audio IN & OUT NTSC / PAL Video Input and Output 2 x Digital Camera Interface DVI Interface LCD Interface with 7” WVGA LCD 6 push-button switches 2 x USB 2.0 Interfaces 3-axis Linear Accelerometer 2 x SD/MMC Slots RTC JTAG Interface

11


Kit Contents

Renesas SH7724 Board (Revision: R0P7724LC) 7” WVGA LCD (800x480) Camera Board SH7724 Linux BSP Bootable USB Memory Stick containing

Linux Kernel Image Root File System

Interface Cables Power Adapter

12


Pictures of Kit Contents

13

Camera Module

Processor Board

Display Module & Touch Screen


Included Software

GNU SH4 Compiler Tool Chain Linux BSP 2.0.0

Linux 2.6.33 Kernel DSP Libraries

Shcodecs Library– Libshcodecs : wrapper library that provides API for

applications to make use of the VPU Middleware Shjpeg Library

– Libshjpeg : JPEG Encoding and Decoding Library Supporting Libraries

Libuiomux : System Resource Manager including UIO Devices Libshveu : Library for controlling VEU (image format conversion,

scaling and rotation)

14


Additional Libraries

VPU Middleware libavcbe : H264 Encoder Core libm4vse : MPEG4 Encoder Core libm4iph : VPU4 Driver

The VPU Middleware Library is provided separately by Renesas on purchasing an Evaluation Kit

15


Framework & Libraries

Linux BSP – Full Featured *nix compatible OS with integrated USB Stack, Networking Stack and a host of other stacks

ALSA – Audio recording & playback framework Video4Linux – Video Capture and Playback Framework DirectFB – Graphics Library with Hardware Acceleration, Input

Device Handling and Abstraction

16


Software Framework Diagram

17

Linux OS, Device Drivers and Sub-Systems (Video4Linux, Networking, ALSA, etc.)

ECOVEC SH 7724 Platform

Camera Display Ethernet USB SDIO Keypad

VPU MiddlewareSH Codecs

Applications

SH VEU

UIOMUX SH JPEG Linux LibrariesVideo4Linux, ALSA, DirectFB

User Space

Kernel Space

Hardware


18

Start of Lab Session

Keep your dice turned to the section of the lab you are on. (Instructionsare provided in the lab handout)

Please refer to the Lab Handout and let’s get started


Checking Progress

We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections.

When done with the lab, your die will have the 6 pointing up as shown here.

19


Lab Description

The lab sessions covers a walk through of the installation procedure for setting up the SH7724 based ECOVEC Platform, its development environment and the development of Multimedia Applications on the Renesas SH7724 Platform using the Tools, Linux Operating System and Libraries provided.

20


Lab Objectives

Setting up the Evaluation Platform Setting Up the Development Environment

Installation of the BSP, Development Tools and Libraries Running the Pre-built Images Preparing and Booting with a USB Drive Development using NFS

SH Linux Kernel Configuration and Compilation Renesas Specific Configuration Items Platform Execution Environment (Bootloader, etc.)

Writing a Simple Media Application Capture, Scale and Display

21


Lab Materials

Lab Workstation Setup Ubuntu Linux Workstation Download Directory with

– Renesas SH4 Linux BSP 2.0– VPU Middleware– Debian Packages for

– DHCP Server, TFTP Server, NFS Server– minicom – Terminal Emulator

Renesas SH7724 Target Setup SH7724 (Ecovec) Platform Display & Camera Modules USB Cable (for Console) Ethernet Cable, Power Adapter

USB Drive with Linux Boot Image and Sample Code Lab Instruction Booklet

22


Setting up the Evaluation Platform

23


Start Lab Session 1

24


Lab Session 1 Complete

Accomplished Setup the development platform and able to run the pre-built

Linux image and the demo applications Next

Setting up the Development Environment

25


Setting Up The Development Environment

Ubuntu Linux as the Development Host platform Any IDE that the User is Comfortable with can be used

For the purpose of the LAB we will be using Anjuta DevStudio which is the GNOME Integrated Development Environment

For convenience and saving time all the required software has been downloaded and installed on the Lab machines

Procedure to Setup the Development Environment Download the BSP & Tool Chain

– Available from https://oss.renesas.com/ Install the BSP & Tool Chain Setting up for Network Booting Booting with a Pre-built Image over the Network

26


Start Lab Session 2

27



Accomplished Setting up the Development Environment by installing the SH4

Linux BSP for the SH7724 Installed and Configured the Networking Utilities that are required

to boot of the Network Are able to boot of the Network using pre-built Linux images

Next Session SH Linux Kernel Configuration and Compilation

28


SH Linux Kernel Configuration and Compilation

In this Session we will: Configure the Linux Kernel Compile the Linux Kernel Install and execute the newly compiled Linux Kernel image

29


Start Lab Session 3

30



Accomplished Configured the Linux Kernel for SH7724 Platform Built the Kernel Image Are able to boot of the our new SH Linux Kernel Image

Next Session Writing a Simple Media Application

31


Writing a Simple Media Application

Objective Capture 720p Frames from the Camera @ 30 fps Scale for WVGA (800x480) Display Output the frames on the Display

Lab Session Code Walk Through Compilation Execution

32


Start Lab Session 4

33



Accomplished Went through the code required for a simple media application

that captures video from a Camera, Scales the captured image and displays the scaled image on the LCD

Compiled the application on the Host Development Environment that was setup earlier

Executed the program successfully

34


Optimizations Necessary to be Real Time

Using buffers that are provided by the UIOMUX Library to avoid coping of data

Threading is required in order achieve 720p Capture and Encode @ 30 fps

35


Conclusion

Summarize the capabilities of the platform Standard Linux System

Linux Programming on all platforms is the same Ease of use Porting to target is easy Many frameworks, libraries and application readily available

Where to get additional information Renesas web site Renesas Open Source Web Site

– https://oss.renesas.com/ Video4Linux

– http://linux.bytesex.org/v4l2/ Ubuntu Linux Distribution

– http://www.ubuntu.com/

36

https://oss.renesas.com/

http://linux.bytesex.org/v4l2/

http://www.ubuntu.com/


37

Questions?


38

Question

1. What clock speed does SH7724 Processor run at? 500 MHz2. What is the DMIPS performance of the SH7724 SoC? 900 DMIPS or 1.8 MIPS / MHz3. How many camera’s are supported on the SH7724 SoC? 24. What is VEU? What is it used for? Video Engine Unit – It is used for video pre and post processing

operations like scaling, color space conversion, etc.5. What is VPU? What is it used for? Video Processing Unit. It is used for Encoding and Decoding

video in various formats like H.264, MPEG4, WMV, etc.6. What is the camera resolution set to in the media application

presented? 720p or 1280 x 7207. What is the CPU utilization when running the CSD application? ~4%

39

Thank You!

40

Appendix


Using Hardware Accelerated Codecs

Demonstration of Using SH Codecs to

Uses the H.264 Hardware Accelerated Codec

Builds on Previous Sessions Application

Capture 640x480 Frames from the Camera

Encode the Video using the H.264 Encoder

Display Encoded Images Properties

Decode the Encoded Video

Output the frames on the Display

Code Walk Through of the Video Encode and Decode Process

41


Useful Libraries

X Toolkits

GTK+ for GUI Development

Qt for GUI Development

GStreamer for recording and playback of audio and video

Libavcodec for encoding and decoding audio and video

SDL – Direct Audio / Video Playback Library

DirectFB – Hardware Accelerated Display Library

42


Abbreviations

VPU – Video Processing Unit

VEU – Video Engine Unit

JPU – JPEG Processing Unit

43

global edge ian carvalho architect 14 october 2010 copyright © 2010, global edge software ltd.,...

Documents

nm process video

ua standby ethernet

global edge software

power mosfet

unit basis

annual market share

power devices

innovation slide