1

打造智能电视的应用生态系统

Leon Zhang – 章立

Strategic Software AlliancesARM

<<li.zhang@arm.com>>SinaWeibo: Leon章立

2

Everywhere, ARM® Powered SmartTV

3

Experience, ARM Powered SmartTV

4

The Challenge is about ApplicationThe existing flourishing application ecosystem Google Play : ~ 500,000+ Apple App Store : ~ 590,000+ Windows Phone App : ~ 100,000+

However, can it benefit SmartTV user directly? When you install an application from mobile/tablet application market

to your SmartTV, what do you expect to happen? Often happen: exception,and can’t run; display issue; layout

issue; user interactive issue; not full features workable Rarely happen: run and play well Even it runs and plays well, do you think it is a real application

you want to play on the TV screen?

5

Application Ecosystem for SmartTV

Applications

OS

Hardware

Video Streamer AppManager

Security

Open Middleware

CPU & GPU

OS & S/W

Tools

6

ARCH & TREND OVERVIEWEvolution of CPU & GPU Architecture

7

Home Consumer Entertainment TrendsConnected home drivers Convergence Intuitive, 3D video and

graphics UI Connectivity and

social networking Energy efficiency Content Security Reducing cost

Enabling with ARM 2GHz Cortex™-A9MP and

Cortex-A15 TrustZone ® Virtualization

Mali ™ -3D graphics /GPGPU Physical IP, system

interconnect, compute systems Software ecosystem OS, browser

optimizations for core components Plug-ins and CODECs Development toolsMobile investments to be re-used in connected home

8

Drivers for A15 and Mali T6xx big-LITTLE Fully positive feedback by the industry

Ultra high performance Esp for single thread app

Virtualization SW flexibility Convergence

GPGPU OpenCL/Renderscript Video enhancement Gesture/voice/face recognition Augmented Reality

TV OS ( Guest )

App2App1

Tablet OS ( Guest )

App4App3

Virtual Machine Monitor (VMM) orHypervisor

Cortex-A15

9

SOFTWARE OVERVIEWEvolution of TV & STB Middleware

10

Evolution of TV & STB Middleware

11

Open Platform Scenarios

Cross Platform

Cloud Service

Local Resource Sharing Internet Resource Sharing Value-Added Services

Open PlatformWeb Application(HTML5/CSS/JavaScript/Web GL)

SDM DCM yWidgetLibrary

ExtendAPI g

DataMgmt …

Application Framework

12

The Open Platform and the FoundationsOpen Platform

Web Application(HTML5/CSS/JavaScript/Web GL)

SDM DCM WidgetLibrary

ExtendAPI

DataMgmt …

Application Framework •Android Davlik•QT Framework•Adoble AIR Framework•iOS Framework•Bada Framework•Android•MeeGo/Tizen/……•RIM OS•WebOS•Bada•iOS

•Linux + Open or Proprietary M/W stacks

•Proprietary OS + Open or Proprietary M/W stacks

• HTML5/CSS/JavaScript/WebGL

13

Tomorrow’s Converged Home Network

In-Home Coax Network

Gateway

MOCA->Ethernet

QAM+CAS

DOCSIS

EoC(e.g MoCA)

ETHERNET

Gateway is suited to act as hub for all services in the home

Gateway can talk to CE devices

Operator Managed Device

Power Efficient & Compliant

CE Devices all use standard Middleware

Fiber

EoC(e.g MoCA)

e.g. North America Cable Market

14

SOFTWARE OPTIMIZATIONOptimizing software platform for the application development

15

ARM Enablement Focus for the Foundation SW

Linux Kernel

LinuxMiddleware

WebBrowser

Webkit

JavaScriptCore

RIM OS

AIRFlash

AIRRuntime

Webkit

AIR for Mobile

JavaScriptCore

Flash

ProprietaryGraphics

QtRuntime

Webkit

Widget Toolkit

V8 JavaScriptEngine

Linux Kernel

AndroidMiddleware

MobileBrowser

Webkit

V8 JavaScriptEngine

Mobile

AIRFlash

Dalvik

LLV

M

Linux Kernel

LinuxMiddleware

SKIA 2DGraphics

ChromeBrowser

Webkit

V8 JavaScriptEngine

Chrome OS

X Windows

Flash

VP

8

Linux Kernel

AndroidMiddleware

TabletBrowser

Webkit

V8 JavaScriptEngine

Tablet

Flash

Dalvik

Linux Kernel

AndroidMiddleware

ChromeBrowser

Webkit

V8 JavaScriptEngine

Google TV

Flash

Dalvik

LLV

M

LLV

MLL

VM

LLV

M

LLV

M

VP

8

VP

8V

P8

VP

8

VP

8

LLV

M

VP

8

SKIA 2D SKIA 2D SKIA 2D

16

What’s inside your browser

JavaScript

17

JavaScript Acceleration on ARM JavaScript accelerated by compiling to native code using JIT ARM performance increased 5X in 1 year ARM works on Multiple JITs Google V8 TraceMonkey JaegerMonkey IonMonkey Squirrel Fish Nitro Tamarin

ARM contributions to upstream projects results in performance optimizations across the ecosystem Example: ARM and Google continually optimizing V8 together Optimizations released into public: http://code.google.com/p/v8 More info at: http://bit.ly/v8ARMTurbo

Google V8

Android Chrome Chrome OS webOS

18

Optimization in HTML5 Optimizing DOM constructs for HTML5 tags <canvas> 2D bitmap graphics NEON optimized- libpixman, SKIA

<svg> 2D vector graphics NEON optimized- libpixman,SKIA feFilters optimized using SMP and NEON

Example: SVG 1.1 defines filters now sped up by ParallelJobs Turbulance 60-80% ConvolveMatrix 20-30% Morphonlogy 20-25% Lighting 10-20% [NEONized version takes this to 3X] Gaussian Blur 15% [NEONized version takes this to 4X]

<video><audio> NEON optimized : TheorARM, WebM VP8 and ffmpeg

CSS Parsing Redesigned CSS parser giving 5% speedup+

Added new RGB color parsing giving 30% speedup

19

Improving HTML5 with

CSS3 Animation (Accelerated Compositing) HTML5 2D Canvas (Accelerated 2D Canvas)

Evidence of improvement in Webkit using a GPU

20

NEON in open source Google WebM – 11,000 lines NEON assembler! Bluez – official Linux Bluetooth protocol stack Pixman (part of cairo 2D graphics library) ffmpeg (libav) – libavcodec

LGPL media player used in many Linux distros and products Extensive NEON optimizations

x264 – Google Summer Of Code 2009 GPL H.264 encoder – e.g. for video conferencing

Android – NEON optimizations Skia library, S32A_D565_Opaque 5x faster using NEON Available in Google Skia tree from 03-Aug-2009

LLVM – code generation backend used by Android Renderscript Eigen2 – C++ vector math / linear algebra template library Theorarm – libtheora NEON version (optimized by Google) libjpeg / libjpeg-turbo – optimized JPEG decode libpng – optimized PNG decode FFTW – NEON enabled FFT library Liboil / liborc – runtime compiler for SIMD processing DirectFB – Neon enabled s/w fallback

21

TOOLSDeliver the powerful develop Tools for the application developers

22

Anatomy of an Android Application

Marmalade System Libraries

Marmalade System Libraries

Native Activity App

Native Activity App

Custom Native Library

Custom Native Library Dalvik

EEDalvik

EE

Android System LibrariesAndroid System Libraries

Dalvik Native LibrariesDalvik Native Libraries

Flash+AIREE

Flash+AIREE

Gaya3D EE

Gaya3D EE

Mono EE

Mono EE

C/C++/AsmC/C++/Asm JavaJava ActionScript, C#, GayaScriptActionScript, C#, GayaScript

Dalvik Native LibrariesDalvik Native Libraries

Languages

Execution Environments

Native code Intermediary Byte CodeIntermediary Byte Code

Android Application – Run as a Dalvik process

Android Platform

ARM

23

ARM Native Code Advantage

Up to 4x faster performance when using NEON

Fast reuse of existing and cross platform software IP, including C/C++/Asm code

ARM native code example: Significant impact on a range of

applications Games Media Players Browsers VoIP clients

All 3D OpenGL ES content leverages ARM code

24

DS-5™CE for Android App Developers

Android ADT

NDK

SDK

adb

CDT(C/C++)

JDT(Java)

Streamline DS-5 Debugger

Native code debug,Performance analysis

App package generation,Debug bridge,

Android Virtual DeviceC/C++ and Java editors,

Java debug GUI

Ethernet, WiFi*

USB

*Available in Q2 2012

25

Timeline: The Big Picture Find hotspots, system glitches, critical conditions at a glance

Select from 40+ CPU counters, OS level and custom metrics

eAccumulate counters, measure timeand find instant hotspots

Select one or more processes tovisualize their instant load on CPU

Combined task switch trace andsampled profile for all threads

26

Mali GPU Graphics AnalysisOpenGL® API eventsCPU, and GPU fragment and

vertex processing activity

Frame buffer filmstrip

Hardware and Software counters

Visualize application activity per processoror processor activity per application

27

Enabling Developers on OEM Devices

ARM DS-5CommunityEdition

OEMs SDK

Develop

Debug

Optimize

Google AndroidSDK + NDK

Enables… Better development

platform Faster development

cycle Better quality content

28

Android NDK

projectne10.orggithub.com/projectNe10

29

Ne10 Introduction Ne10 is designed to provide a set of common, useful

functions which have been optimised for the ARM architecture and NEON provide consistent well tested behaviour and that can be easily incorporated into applications

C and JNI interfaces are provided (JNI Not publicly available yet)

The library is modular; therefore the functionality that is notrequired within an App can be discarded

30

Ne10 - Routines

Images and Video•Image processing•Color conversion•JPEG/JPEG2000•Video Coding•Ray-Tracing/Rendering

Communications& Signal Processing

•DSP•Audio Coding•Speech Coding

Data Processing•Data Compression•Cryptography•String Processing•Matrix/Vector Algebra•Data Integrity

Targeting three main categories of functions

31

Ne10Droid – The App in actionNe10Droid is a benchmarking Android App that uses Ne10Routines are written using VFP in C, VFP in Assembly and NEON.Example routines:arm_result_t normalize_vec2f(arm_vec2f_t * dst, arm_vec2f_t * src, unsigned int count);arm_result_t normalize_vec3f(arm_vec3f_t * dst, arm_vec3f_t * src, unsigned int count);arm_result_t normalize_vec4f(arm_vec4f_t * dst, arm_vec4f_t * src, unsigned int count);

32

Ne10 - Future plans

Dual licensing under both Apache 2.0 and MIT as to be compatible with GPLv2.

Ne10 will include DSP functions (Think Zephyr and OpenMAX)

working on using Ne10 to optimize Box2D and the popular game engine

We want your input on future functionality and participation!

33

Summary

Application is an increasingly critical differentiator for TV The application ecosystem is being powered by ARM CPU/GPU ARM is enabling the optimized software platform and powerful

tools for the application developers Enjoy the TV experience on ARM