0018-9162/05/$20.00 © 2005 IEEE68 Computer

C O V E R F E A T U R E

P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y

Evaluating DigitalEntertainment System Performance

D igital entertainment systems—which en-compass a broad range of applications,including smart phones, set-top boxes,MP3 players, DVD players and recorders,game consoles, and digital cameras—have

become the driving force behind the expansion ofthe semiconductor market, outstripping even PCs.Consider, for example, that in 2003, smartphonesrepresented about 3 percent of the 500 millionmobile phones sold worldwide, with analystsexpecting them to grow at triple-digit year-over-year rates.

Moving beyond the voice-centric model, morethan half of the 600 million mobile phones sold in2004 included a color display and digital camera.The implementation of more advanced featuressuch as accelerated 2D and 3D graphics, videocon-ferencing, mobile multimedia, and games has raisedthe performance requirements of these models. Thesame holds true for any of the other digital enter-tainment devices.

Rapid advances in semiconductor technology,microarchitectures, and embedded systems havemade the adoption of these features possible. As aresult, software complexity will continue toincrease to keep pace with the overall system com-plexities.

Manufacturers of devices such as semiconductorsand entire systems race to create products with thelatest technology and bring them to market beforetheir competitors. Hence, the need to evaluate per-formance goes well beyond the consumer’s demandsfor more entertainment and functionality value.Benchmarks and performance evaluation play an

integral role in processor, algorithm, and systemdesign.

CATEGORIZING DIGITAL ENTERTAINMENT BENCHMARKS

The best digital entertainment benchmark is theexact application that will be run on the device. Forexample, if the device will be running video playback,the appropriate test is to run an MPEG-4 decode algo-rithm. The more realistic the benchmark, the morevaluable it will be for evaluation purposes. However,many modern benchmarks are very long, and per-forming simulations on them can become prohibi-tively time-consuming, as would be the case whenevaluating IP cores or simulated platforms.

Research has shown that short synthetic streamsof instructions can be created to approximatelymatch the behavior of the instruction stream fromthe full execution.1,2 However, these streams may notbe sufficient to take system-level features such ascaches and memory into account, and they certainlyare not sufficient to test hardware accelerators andother CPU offload engines.

The first step in creating a suite of digital enter-tainment benchmarks is to define the target, sincethe benchmarks will differ depending on whetherthey are used to evaluate a microprocessor’smicroarchitecture, its system-level functionality, oran algorithm’s behavior. Many proprietary bench-marks, such as BDTI and FutureMark, will fulfillthese goals, but the approaches taken by universityresearchers and industry consortiums, such asMediaBench3 and EEMBC (www.eembc.org), pro-vide a more open and democratic alternative.

The Embedded Microprocessor Benchmark Consortium’s DENBench suiteof digital media benchmarks provides a spectrum of tools for assessing andrefining the video and audio performance of digital devices.

Markus LevyEEMBC

July 2005 69

MediaBench: An academic projectThe MediaBench benchmark suite consists of

several applications in the image processing, com-munications, and DSP categories. MediaBenchincludes applications such as JPEG (image com-pression), MPEG (video transmission encoding anddecoding), GSM (full-rate speech transcoding),G.721 (voice compression), Ghostscript (PostScriptlanguage interpretation), and adaptive differentialpulse code modulation (ADPCM).

An academic effort to assemble several media-pro-cessing-related benchmarks, MediaBench focuses oncomplete applications, using publicly available code,for multimedia and communications systems. Thebenchmark suite also requires the use of high-levellanguage to stress compilation technology.

EEMBC: The consortium approachThe Embedded Microprocessor Benchmark

Consortium’s members formed this nonprofit orga-nization in April 1997 to develop performancebenchmarks for processors in embedded applica-tions. Developers designed the original EEMBC(pronounced “embassy”) benchmark suite to reflectreal-world applications and some synthetic bench-marks. These benchmarks target the automotiveand industrial, consumer, networking, office auto-mation, and telecommunications markets.

EEMBC dissected applications from these do-mains and derived 37 individual algorithms that con-stitute its version 1.0 suite of benchmarks. In addi-tion to developing the benchmark code, EEMBCseeks to provide certified benchmark scores to inter-ested public sectors such as the embedded systemdesign community.

EEMBC has evolved in parallel with the indus-try. The organization recently released a new suiteof digital media benchmarks that include MP3decode, MPEG-4 video encode and decode, MPEG-2 video encode and decode, image filtering, JPEGcompression and decompression, and a variety ofcryptography algorithms, as the “DENBench Suite”sidebar describes. Referred to as the DENBenchsuite for digital entertainment, this collection pro-vides representative benchmarks for the audio,video, and still-image processing found in consumerelectronics and home entertainment systems, video-conferencing systems, multimedia-enabled cellulartelephones, and digital cameras.

Compared with previous EEMBC benchmarkkernels that measure processor performance in con-sumer applications, the DENBench suite makesmore intensive use of processor computational abil-ity, caches, and system memory, thereby providing

a detailed analysis of processor strengths and weak-nesses. Going beyond evaluation of the processorcore, the benchmarks also evaluate complex archi-tectures and complex memory hierarchies. Somefind it useful to partition the workload for thesebenchmarks, putting the control tasks on a general-purpose processor and the computationally inten-sive tasks on a hardware accelerator, coprocessor,or DSP engine.

DENBench Suite

Derived from industry sources following a mandate from membersthat they be written in relatively strict ANSI C, developers can use theDENBench suite for most processing platforms.

The suite consists of 14 benchmarks. Each benchmark must runthrough five to seven different data sets, yielding a total of 69 tests.EEMBC provides benchmark scores for the individual benchmarks anddivides the suite into four minisuites, each with its own mathematicallyderived benchmark scores. The DENmark suite provides a geometricmean of all four minisuites.

MPEG EncodeMark—Measures motion-video encoding and has a geo-metric mean of 10 tests × 1,000; consists of two benchmarks:

• MPEG-2 video encoding• MPEG-4 video encoding

MPEG DecodeMark—Measures motion-video and digital-audio decod-ing and has a geometric mean of 15 tests × 1,000; consists of the fol-lowing benchmarks:

• MPEG-2 video decoding• MPEG-4 video decoding• MPEG-2 Layer 3 MP3 player audio decoding

CryptoMark—Measures data encryption and decryption and has a geo-metric mean of 4 tests × 10; consists of the following benchmarks:

• AES, the Advanced Encryption Standard;• DES, the Data Encryption Standard;• RSA, Rivest-Shamir-Adleman public-key cryptography; and• Huffman decoding for data decompression

ImageMark—Measures still-image compression, decompression, andcolor-space conversion and has a geometric mean of 35 tests × 10; con-sists of the following benchmarks:

• JPEG Compress—performs still-image data compression using theJPEG standard;

• JPEG Decomp—performs still-image data decompression using theJPEG standard;

• RGB to CMYK—converts red-green-blue color space to cyan-magenta-yellow-black color space;

• RGB to YIQ—converts red-green-blue color space to a luminance-chrominance color space; and

• RGB to HPG—converts red-green-blue color space to Hewlett-Packard graphics

70 Computer

System memory

Digital entertainmentbenchmark code

Media stream

Test harnesscontrol code

Memorycontroller

Timers

Processor

Icache

Dcache

Target hardware

Figure 1. A common framework, or test harness, implements the EEMBC benchmarks. A host computer can use the framework to communicate to a device under test through most types of interfaces, including serial, parallel, and PCI ports.

DEFINING THE BENCHMARKSAs expected with a consortium, defining a bench-

mark involves a lengthy and arduous process.Indeed, the process for these newly released mediabenchmarks began nearly three years ago. At theearliest stages of development, selecting a repre-sentative set of applications to sufficiently test theprocessors and systems under consideration pro-vided the first challenge. The next challengeinvolved determining how to make these bench-marks portable enough to run on the wide varietyof processors and configurations that constitute thetargeted digital consumer applications.

Many popular benchmarks perform a fixedworkload. Throughput benchmarks, on the otherhand, have no concept of finishing a fixed amountof work. Developers use throughput benchmarksto measure the rate at which work gets done.EEMBC’s approach has always been based on afixed workload.

Using the MPEG-x benchmark as an example, afixed workload approach would process a videowith a specific number of frames, measuring howlong it took to process the entire video. Alterna-tively, running the benchmark for a fixed amountof time would measure the number of framesprocessed.

Although EEMBC implements the formerapproach, a potential issue arises because fixed-work benchmarks become outdated when com-puting capability or cache and memory capacityincrease—which occurred with some of EEMBC’sfirst-generation benchmarks. The tricky balanceinvolves making the input data set large enough tothwart the most robust processors and systems butalso small enough for use with lower-end platforms

or to run on simulated environments.When developing video codec benchmarks, the

encoder and decoder portions must be measuredseparately. These two elements stress differentparts of the CPU platform and are used in differ-ent numbers. For example, decoders are more com-mon than encoders, and most devices can onlydecode.

CREATING PORTABLE BENCHMARKSCreating portable benchmarks, whether using

MediaBench or the DENBench suite, requires usinga high-level language to stress compilation tech-nology. Derived from industry sources, all bench-marks are written in relatively strict ANSI C.Theoretically, researchers can use ANSI C to com-pile benchmarks for most processing platforms,although hand coding will be required to makebenchmark optimizations that run on an offloadengine.

The DENBench suite runs natively, directly onthe processor hardware and without an operatingsystem. Although this represents a deviation frommost real-world implementations, it supportsportability because it eliminates any operating sys-tem dependencies or application-programminginterface issues.

EEMBC implements all of its benchmarks usinga common framework, or test harness, to ensurethat everyone runs the benchmarks similarly. Thistest harness, which runs on a host computer, com-municates with the target platform and extracts ahardware adaptation layer for easy porting, asFigure 1 shows.

A truly portable benchmark ensures that usersrun it consistently, compile the same code, andprocess the same workload. A disadvantage of suchbenchmarks is that few, if any, users would everimplement this type of code in a real system, espe-cially a performance-, power-, and memory-con-strained system.

To accommodate real-world implementations,EEMBC deviates from other benchmark approachesby supporting both out-of-the-box portable codeand optimized implementations. Researchers canmodify the benchmark code of an optimized imple-mentation to take advantage of hardware accelera-tors or coprocessors or special instructions, or evento utilize intrinsics that will extend the compiler’sfunctionality. Out-of-the-box benchmarks cannottake advantage of a multiprocessing or multi-threading system’s resources, however, which is espe-cially disadvantageous with media benchmarks thatsupport a significant amount of parallelism.

July 2005 71

DERIVING BENCHMARKS FOR MULTIPROCESSING

Developers refer to the two fundamental multi-processing methods as symmetric multiprocessing(SMP) and asymmetric multiprocessing (AMP).Also known as hetero- geneous multiprocessing,with AMP the system designer can use the proces-sor best suited for a specific task. For example, themajority of mobile phones use the Texas Instru-ments AMP, the open multimedia appli- cationsplatform. OMAP consists of an ARM core and aDSP. The platform uses the ARM processor to runthe application code and user interface, and it usesthe DSP for modem functions and accelerating mul-timedia algorithms such as MPEG-4 decode.

Theoretically, developers can rewrite the DEN-Bench code to accommodate either SMP or AMP,but this requires a relatively manual process.

Although realizing any of these multiprocessingmethods in hardware is relatively straightforward,the true challenge is to make multiprocessing trans-parent to the system designer or benchmark porter.Ultimately, system designers will be able to imple-ment multiprocessing applications with minimaleffort, which motivates developers to place the par-titioning burden on the operating system, compiler,and other software tool vendors.

A simple, albeit ineffective, solution to bench-marking multiprocessor systems would be to mea-sure the total throughput available from runningmultiple instances of either the same or differentapplications, eliminating any interapplicationdependencies. However, creating realistic usage sce-narios for running multiple independent applica-tions requires extra effort. Specifically, anybenchmark should run different applications tostress the platform’s ability to support the multiplecache contexts associated with multiple applica-tions, as opposed to highlighting the ability to cachea single application across multiple processors.

Developers can derive an efficient multiprocess-ing benchmark by parallelizing a single task to scaleacross multiple instruction contexts. This type ofbenchmark must stress the system in terms of fine-grained synchronization access to shared resources.But there’s more to it than that. A specific SMPbenchmark suite must be on hand, rather than justan SMP optimization of existing suites.

Parallelism has three potential characterizationsin this context:

• task decomposition parallelizes a single algo-rithm to share its workload over multipleprocessors;

• multiple algorithms look more at howthe bigger system, including the OS, han-dles the workload from multiple con-current algorithms; and

• multiple streams look at the bigger sys-tem, but concentrate more on datathroughput and how a system can han-dle multiple data channels.

If we look solely at task decomposition, wecould consider SMP simply as an optimiza-tion of an existing suite’s algorithms. For example,we could parallelize each of the kernels in theDENBench to use multiple processors. But for theother two forms of parallelism, this does not reallyfit.

Existing SMP benchmarks such as SPECint-Rate(www.at.nwu.edu/rtg/sp/hardware_performance.ssi) consider some aspect of multiple streams bylooking at the throughput of an MP systemthrough a simple duplication of a smaller test runin multiple instances simultaneously. This does notmeet EEMBC’s real-world criteria because it failsto consider the decomposition of multiple streamsor interstream synchronization as part of the testresult.

Instead, in the key SMP markets, a device mustperform more than one simultaneous function.This is evident today, for example, when someoneuses a mobile phone to place a video call. In thiscase, the system processes encode-decode algo-rithms along with the user interface and, possibly,networking protocols. The SMP benchmark suitemust evaluate the stress points on the hardware andOS, while supporting these multiple code and dataworking sets.

DEFINING CONFORMANCE AND QUALITYDevelopers can optimize the EEMBC bench-

marks to take better advantage of each platform’sfeature set. Beyond performance testing, however,we must also consider the matter of defining con-formance and quality testing if we focus on any-thing other than a standard encoding algorithm.Some would favor avoiding anything except a pre-cisely defined encoding algorithm, while mostwould like to implement the encoder solution thatbest fits their architecture.

An encoder’s output cannot be verified easilybecause, for example, an MPEG-2-compliant out-put with varying degrees of quality can be createdin several ways. The output can require drasticallydifferent CPU times depending on the motionsearch algorithm, range, and so on.

The true challenge is to make

multiprocessingtransparent to thesystem designer orbenchmark porter.

72 Computer

Other than heuristics such as the peak signal-to-noise ratio and perceptual quality adaptation, thereis no standard for checking the output’s quality.Industry standards define decoders and the com-pliance rules for their outputs more tightly.Developers can also apply the peak signal-to-noiseratio test to these outputs.

P erformance and quality provide good startingpoints for evaluating the capabilities of a dig-ital entertainment system, but energy con-

sumption is an equally important metric. Currently, EEMBC is in the final stages of devel-

oping a standardized method for simultaneouslymeasuring power and performance on processor-based systems. Many questions must be answeredin developing a standard for measuring a proces-sor’s power and energy consumption. For example,at the most basic level, every vendor has a slightlydifferent definition of what constitutes a processor:Is it comprised of the core alone or does it includecache and peripherals? Further, power consumptioncan vary significantly with different cache configu-rations, system bus loading, and especially the typeof application code running. Thus, it is important tocharacterize the target system’s power consumptionrather than relying on device specifications.

Another challenge arises because every vendorruns its processors using a different workload, and,typically, these workloads are artificial constructscreated to simulate typical and worst-case scenar-ios. The EEMBC benchmarks guarantee a common

set of workloads, but there is also enough varietyin the types of benchmarks to demonstrate thepower a processor’s different architectural featuresconsume.

This standardization will let processor vendorsand system designers test out the exquisite power-saving features. This additional metric will helpclassify processors for the appropriate target mar-kets, such as portable versus line-powered, and willdemonstrate that having the fastest processor is not necessarily best in the digital entertainmentmarket. �

References1. L. Eeckhout et al., “Control Flow Modeling in Sta-

tistical Simulation for Accurate and Efficient Proces-sor Design Studies,” Proc. 31st Ann. Int’l Symp.Computer Architecture, IEEE Press, 2004, p. 350.

2. R.H. Bell Jr. and L.K. John, “Experiments in Auto-matic Benchmark Synthesis,” Tech. Report TR-040817-01, Laboratory for Computer Architecture,Univ. of Texas at Austin, 2004.

3. C. Lee, M. Potkonjak, and W.H.M. Smith, “Media-Bench: A Tool for Evaluating and Synthesizing Mul-timedia and Communication Systems,” Proc. 30thInt’l Symp. Microarchitecture, ACM Press, 1997, pp.330-335.

Markus Levy is founder and president of theEmbedded Microprocessor Benchmark Consor-tium. Contact him at markus@eembc.org.

Join a community that targets your discipline.

In our Technical Committees, you’re in good company.

www.computer.org/TCsignup/

Looking for a community targeted to your area of expertise? IEEE Computer SocietyTechnical Committees explore a variety of computing niches and provide forums fordialogue among peers. These groups influence our standards development and offerleading conferences in their fields.

JOIN A THINK TANK