95MARCH 2009

GREEN IT

Published by the IEEE Computer Society0018-9162/09/$25.00 © 2009 IEEE

Identifying Shades of Green: The SPECpower Benchmarks

Editor’s note: Under pressure from management, IT professionals expand-ing or replacing their infrastructure struggle to identify servers and PCs that sip rather than gulp electrical power from the grid. Manufacturers tout their wares as green and energy efficient, using metrics of their own creation that are difficult to compre-hend, let alone verify or compare. A need exists for standardized testing that accurately captures computer energy efficiency. Fair benchmarking enables consumers, researchers, and businesses to make informed purchase decisions. In this installment of the Green IT column, guest columnist Klaus-Dieter Lange, chair of the SPECpower committee, discusses the creation and future of the first widely accepted, industry-standard benchmark for energy efficiency.

–Kirk W. Cameron, Green IT editor

Twenty years ago, the computer indu s t r y formed the nonprofit Standard Performance

Evaluation Corporation to establish

industry standards for measuring compute performance. SPEC has since become the largest and most influential benchmark consortium. Its mission is to ensure that the mar-ketplace has a fair and useful set of metrics to differentiate the newest generations of a range of platforms from symmetric multiprocessing and Non-Uniform Memory Architecture to clustered multiprocessing systems.

The SPEC community has devel-oped more than 30 industry-standard benchmarks for system performance evaluation in a variety of application areas and provided thousands of benchmark licenses to companies, resource centers, and educational institutions globally. Organiza-tions using these benchmarks have published more than 10,000 peer-reviewed performance reports.

SPEC reviews all submitted results from its member organizations and other benchmark licensees and allows publication of benchmark results only if they comply with the detailed run and reporting rules defined for each benchmark. Members closely

monitor benchmark results and press releases to ensure compliance with SPEC’s General Availability and Fair Use rules.

SPEC membership is open to any interested company or entity and cur-rently includes computer hardware and software companies, educa-tional institutions, and government agencies.

SPEC bEnChmark dEvEloPmEnt ProCESS

Two factors drive benchmark development: industry demand and interest of SPEC members who pro-vide the engineering resources to design and implement the new bench-mark or workload.

In general, the development pro-cess starts by chartering a working group to research the new area of interest and prepare a benchmark development proposal. After SPEC approves this proposal and commits its resources, the implementation process begins. The group prepares detailed design documents and then codes an OS-independent work-

Klaus-Dieter Lange, HP

To drive energy efficiency initiatives, SPEC established SPECpower_ssj2008, the first industry-standard benchmark for measuring power and performance characteristics of computer systems.

load and user-friendly framework. Next, they finalize the benchmark scope and the run and reporting rules. Finally, they create support-ing documents such as user guides. Rigorous testing during this process ensures that the group accomplishes all benchmark goals. Afterward, SPEC’s general membership reviews the newly designed benchmark, and feedback from an external beta test program is integrated into the final benchmark release package.

Once launched, the new bench-mark enters the normal submission, review, and publication cycle. Due to the rapidly changing technology field, the typical life span of a benchmark is three to five years, after which the benchmark is retired and replaced by its successor.

SPECPowEr_SSj2008The increasing demand for energy-

efficient servers has resulted in the need for power and performance benchmarks. Therefore, the SPEC community established SPECpower, an initiative to augment existing SPEC benchmarks with a power/energy measurement. AMD, Dell, Fujitsu Sie-mens Computers, HP, IBM, Intel, Sun Microsystems, SPEC Associates, and Lawrence Berkeley National Labora-tory committed leading engineers and scientists in the fields of benchmark development and energy efficiency to tackle this task.

On 26 Ja nu a r y 20 06, t he SPECpower committee started the development of SPECpower_ssj2008, the first industry-standard benchmark that measures the power and perfor-mance characteristics of server-class compute equipment. SPEC worked closely with the Environmental Pro-tection Agency to define additional requirements to enhance the bench-

mark suite and satisfy EPA needs where it was feasible. “SPEC Power and Performance Methodology” documents the conclusions of SPEC’s research and the experience SPEC gained from SPECpower_ssj2008’s two-year development. This docu-ment explains how to define power components within benchmarks and how to integrate power measure-ments into benchmark routines. It also addresses the variety of server types and the impact of environ-mental conditions like temperature, humidity, air pressure, and airflow. The latest revision is available at www.spec.org/power_ssj2008/docs/SPECpower-Methodology.pdf.

SPEC believes that a user-friendly benchmark leads to greater avail-ability of results in the marketplace. Therefore, the measurement setup should be relatively straightforward, and the duration of the benchmark should be practical.

For SPECpower_ssj2008, SPEC decided that AC power measure-ments would be made at the AC input to the server. In addition, the sup-ported power analyzer must have an interface that allows automatic data transfer. This significantly increases the ease of use since exact synchro-nization of power and performance data is possible and automation fur-ther reduces the chance of human error.

SPEC developed the Power and Temperature Daemon (PTDaemon), which automatically transfers mea-surements from the power analyzer and temperature sensor to the bench-mark harness. PTDaemon simplifies system interaction with different device interfaces, which allows easily using it with other SPEC benchmarks. Furthermore, developers can imple-ment interfaces for new devices on demand, without revising the entire benchmark.

SPEC concluded that the cost of procurement, maintenance, and space allocation for a climate cham-ber and the potential decrease of ease

0100200300400500600700800

050,000

100,000150,000200,000250,000300,000350,000

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Active idleTarget load levels

Perfo

rman

ce (s

sj_op

s)

Powe

r (wa

tts)

Performance [ssj_ops]Power [watts]

Figure 1. Power and performance at multiple target load levels—100 percent down to 0 percent of the maximum throughput at 10 percent increments.

CoMputeR 96

GREEN IT

0

100

Measurement interval

80% target load

0

100

Measurement interval

20% target load

Utiliz

ation

(%)

Utiliz

ation

(%)

Figure 2. The left figure shows a measurement interval with an 80 percent target load, while the right figure shows a measurement interval with a 20 percent target load.

editor: Kirk W. Cameron, Dept. of Computer Science, Virginia tech; greenit@computer.org

Be on the Cutting Edge of Artifi cial Intelligence!

IEEE Intelligent Systems seeks papers on all aspects of artifi cial intelligence, focusing

on the development of the latest research into practical, fi elded applications.

For guidelines, see www.computer.org/mc/intelligent/author.htm.

The #1 AI Magazinewww.computer.org/intelligent

IEEE

CallArticlesfor

97MARCH 2009

of use outweighed its usefulness for a benchmark. Nevertheless, after sur-veying data centers around the world and consulting with American Society of Heating, Refrigerating, and Air-Conditioning Engineers guidelines, SPEC requires a minimum environ-mental temperature of 20ºC (68º F). A required external temperature sensor connected to the benchmark harness measures the server’s air inlet and records thermal data every 15 seconds.

SPECPowEr_SSj2008 mEtrIC

SPEC benchmarks encourage server design improvements and optimizations at the component and system levels while discourag-ing misrepresentation and “creative marketing” across the industry. To improve server energy efficiency, users must capture the impact of emergent energy-saving technologies, for example, by dynamically clocking the frequency of components includ-ing CPU cores and memory. Therefore, one of the early design decisions was to capture performance and power measurements for the benchmark under various utilization levels.

SPECpower_ssj2008’s workload is a Java application that generates and completes a mix of transactions; the reported throughput is the number of transactions completed per second over a fixed period. The benchmark begins by running a calibration phase designed to determine the sys-tem’s maximum throughput, which is used to calculate the throughput values that correspond to each target load—decreasing in 10 percent increments.

As Figure 1 shows, the benchmark then enters the measurement interval during which the workload iterates through 11 target loads. At each load, the benchmark inserts exponential random delays into the workload stream that statistically converge to a target load level during a test interval. Figure 2 shows a measurement inter-

val with an 80 percent target load and a measurement interval with a 20 percent target load.

SPECpower_ssj2008 reports the energy efficiency in terms of overall ssj_ops/watt. This metric represents the sum of the performance mea-sured at each target load level (in ssj_ops) divided by the sum of the average power (in watts) at each target load including active idle. The following example derives the SPECpower_ssj2008 rating from the data in Figure 2.

SPECpower_ssj2008 =∑

∑Power

Performance

=6,510 watt

1,790,250 ssj_ops

wattssj_ops

= 275 overall

Performance, cost, and avail-ability have been the top priorities of data centers for

many years. However, now energy efficiency is becoming the number one priority for many data centers. Since SPECpower_ssj2008’s initial release in December 2007, the SPEC community has submitted a constant

stream of fully documented results, which are available at www.spec.org/power_ssj2008/results/power_ssj2008.html. Now, more than a year after the initial release, it is clear that the first SPECpower benchmark was a leap forward for the power and performance analysis of enterprise computer servers.

SPECpower_ssj2008 covers one application space. In an effort to rep-resent a wider range of applications, SPEC is working to standardize power efficiency metrics across all applica-ble industry benchmarks including I/O-intensive workloads. Going for-ward, SPEC power efficiency metrics will help data center managers to make informed purchase decisions regarding energy-efficient servers and also will accelerate the develop-ment of energy-efficient servers and components.

Klaus-Dieter Lange is a senior sys-tems performance engineer at HP. Contact him at klaus.lange@hp.com.