ibm systems & technology group 2008 spec® benchmark workshop ibm, intel, 2008 specweb2005 in...

IBM Systems & Technology Group

2008 SPEC® Benchmark Workshop

IBM, INTEL, 2008

SPECweb2005™ in the Real World: Using Internet Information Server (IIS) and PHP

01/27/2008

Sam Warner Steve Worley

IBM, INTEL, 2008

Intel Corporation and IBM Systems & Technology Group 2008 SPEC® Benchmark Workshop

Analysis of various systems Using IIS as web server Using PHP to server dynamic content

SPECweb2005/PHP more representative to “real world”

Results comparison PHP to published JSP™ results Dual-socket results and Multi-socket results

Abstract

IBM, INTEL, 2008


Introduction

SPECweb® benchmark has brought ability to show the relative performance of various systems manufactured in the Web Serving market

SPECweb® can be used for Research efforts at universities Within computer equipment manufactures to evaluate alternative technologies In the purchase process by Information Technology (IT) professionals

SPECweb® has two alternatives to enable performance evaluation Java Server Pages (JSP™)

All current published results Top results are unrealistic at approximately 40000 users

Hypertext Preprocessor (PHP) This paper provides results comparison not found in published results Help insure the benchmark remains useful in IT customer purchase process

IBM, INTEL, 2008


Overview of the SPECweb2005™ Benchmark

Standard Performance Evaluation Corporation benchmark for evaluating the performance of World Wide Web Servers

Supersedes SPECweb99® and SPECweb99_SSL®

Three workloads representing market segment that are part of the benchmark

SPECweb2005_Banking SPECweb2005_Ecommerce SPECweb2005_Support

Each workload variant enables measuring the maximum number of simultaneous user sessions

Still achieving specific QOS metrics Still achieving error rate requirements

IBM, INTEL, 2008


Overview of the SPECweb2005™ Benchmark

Benchmark score

Individual sub-metric scores indicate the total number of simultaneous user sessions the server can support

Overall score is the geometric mean of the three sub-metrics normalized to a reference platform score

A score of 100 represents the same overall performance of the reference platform A score of 20,000 represents a score 200 times that of the reference platform

The benchmark does not require a specific choice of web server software

Only requirement is that the web server support HTTP 1.1 and SSL (HTTPS) Implementation of the web server will lead to differences in observed performance on

the same system under test Current performance using the JSP™ kit achieves performance which is roughly 370x

the reference platform

IBM, INTEL, 2008


Top 15 SPECweb2005™/JSP™ Scores

Rank Web Server Chips Cores Processor Result1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 37,0012 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 37,0013 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 4 Intel Xeon X7350 Processor 30,2614 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 4 Intel Xeon E7330 Processor 26,1195 Rock Web Server v1.4.6 (x86_64), Rock JSP/Servlet Container v1.3.1 (x86_64)2 4 Intel Xeon Processor X5365 26,0776 Rock Web Server v1.4.3 (x86_64), Rock JSP/Servlet Container v1.3.0 (x86_64)2 4 Intel XEON 5365 processor 22,3327 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8222SE 22,2548 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8220 21,4709 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon Processor X5355 20,387

10 Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8220 20,23511 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5355 Processor 19,93112 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon Processor X5355 19,66113 Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 Intel Xeon 7140M Processor 18,98114 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5365 Processor 18,93115 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5355 Processor 18,917

IBM, INTEL, 2008


Overview of Top Fifteen Results

All fifteen submitted with one of two web severs Sun™ Java™ System Web Server 7.0 Accoria Rock Web Server V1.4

13 of the top 15 results

Rock Web Server is undeniably fast No measurable market share from market analysis firms

All results available at SPEC website http://www.spec.org

IBM, INTEL, 2008


SPECweb2005™ and the Real World

IT customers use of performance benchmarks

Seen as way to evaluate the complex combination of Number of users supported Content development language Web server software Web server hardware

Don’t have much time to understand the subtleties of a benchmark

Forces OEMs to be constrained by competitive pressures to produce leading results

Causes the publication of SPECweb2005™ to diverge from real world IT performance

IBM, INTEL, 2008


SPECweb2005™ and the Real World

Trend of web server connections supported Range of 20-2000 sustained customer connections As processing capacities have improved, so has

Concerns and processing for security Complexity in formulating the responses for web server requests

SPECweb99™ Results ranged from 400 to 25000 for the five years of publication Meanwhile, web server connection rates remained in the 1000s

SPECweb2005™/JSP™ Two and four socket systems have begun to achieve levels approaching 50,000

simultaneous users for a duration of ½ hour. In contrast to research papers where published connection loads tend to range

from 2000 to 3000 users SPECweb2005™/PHP

By comparing to SPECweb2005™/JSP™ attempt is to bridge gap between real world loads and published results

IT industry may be better served by comparisons to connection loads seen in their environments

IBM, INTEL, 2008


SPECweb2005™/JSP™ Scores since InceptionSPECweb2005/JSP Scores since inception

50x change in Connections sustained in two years

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

Sep-05 Mar-06 Aug-06 Jun-07 May-07 Jun-07

SP

EC

web

2005

/JS

P S

core

s

0

10

20

30

40

50

60

70

Mem

ory

(GB

), N

um

ber

Net

wo

krs,

Nu

mb

er o

f S

ock

ets

SPECweb2005/JSP Score # sockets Memory (GB) # of Nets

IBM, INTEL, 2008


Analysis of Currently Published SPECweb2005™ Results

Request for Information and Request for Proposal processes have led many OEMs to develop publications for SPECweb2005™.

Forces OEMs to use software combinations that enable leading performance results

Competition Great way to insure that the final purchased product meets the needs of the customer If benchmark closely matches the end-user environment, benchmark produces an

informed purchase

Top 15 results show connection rates that are 10-20x higher than connection loads that are normally seen by web servers.

IBM, INTEL, 2008


Comparison of Web Servers

Of top fifteen SPECweb2005™ results, thirteen use the Accoria Rock web Server v1.4

ProprietaryNon-open sourceNo measurable market share

Web Servers discussed in the paper

Web Server Creator Windows Linux Solaris CGI FastCGI Servlet ASP.NET

IIS Microsoft Yes No No Yes Yes No Yes

Rock Web Server Accoria Networks Yes Yes Yes Yes Yes Yes No

Sun Java System Sun Microsystems Yes Yes Yes Yes Yes Yes No

IBM, INTEL, 2008


Web Servers Currently Used in Submissions

Choice of Web Server up to OEM

Results prepared as market collateral Market conditions dictate the decision of web server Best possible combination of hardware and software If less than ideal component used then consumer education required

Results response to RFI Choice of highly relevant results to customer or industry leading results Lower cost to educate customer by simply achieving highest possible result

Net result is that neither Apache or IIS, both recognized in their widespread use, have been used in any submissions in the last two years

IBM, INTEL, 2008


Comparison of JSP™ to PHP performance

Rank Web Server Chips Cores Processor JSP Results PHP Results1 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 na2 Sun Java[TM] System Web Server 7.0 Update 2 1 8 Sun UltraSPARC T2 na3 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 4 Intel Xeon X7350 Processor na4 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 4 Intel Xeon E7330 Processor na5 Rock Web Server v1.4.6 (x86_64), Rock JSP/Servlet Container v1.3.1 (x86_64)2 4 Intel Xeon Processor X5365 na6 Rock Web Server v1.4.3 (x86_64), Rock JSP/Servlet Container v1.3.0 (x86_64)2 4 Intel XEON 5365 processor na7 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8222SE na8 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8220 na9 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon Processor X5355 20387 1083

10 Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 AMD Opteron 8220 na11 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5355 Processor 19931 108312 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon Processor X5355 19661 108313 Rock Web Server v1.4.0 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)4 2 Intel Xeon 7140M Processor 18981 114314 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5365 Processor na15 Rock Web Server v1.4.1 (x86_64), Rock JSP/Servlet Container v1.2.0 (x86_64)2 4 Intel Xeon X5355 Processor 18917 1083

IBM, INTEL, 2008


JSP™ use in submissions

Presently all of the results submitted have used the JSP™ kit

While JSP™ is widely used, it is not the primary content development language used

To achieve leading industry results an OEM needs to use a software stack that has the lowest overhead to generate a response.

The current implementations of SPECweb2005™ in JSP™ has lower overhead than PHP

IBM, INTEL, 2008


Analysis of Real World Workloads

Industry results on SPECweb2005™/JSP™ have improved by ~50x since inception

Publications from universities show that true connection loads trend at levels 1/10th of those observed in publications.

SPECweb99™ this same disconnect

Using the SPECweb2005™/PHP kit on equivalent systems we find that the change in connection loads observed is not as high.

Intel® Xeon® X5355 JSP™ shows 18,000-20,000 connections PHP shows 1000 connections

IBM, INTEL, 2008


Web Server Statistics

Aversa and Bestavros Paper TCP connection load of between 496 and 663 requests served Response time between 0.92s and 0.26s

Grottke paper Connection rates of 390 connections/second (c/s)

Cao paper Connection rates of between 0.18 c/s and 34 c/s Connection rates of between 2.41 c/s and 230 c/s

Seems reasonable to look at a scenario where the top connection rates are in the 2000-3000 connections/second range

IBM, INTEL, 2008


Discussion of use of IIS

According to various web server market analysis companies approximately 35% of web servers on the internet use IIS

Tends to be used in websites with higher traffic volumes

No web server market analysis companies are able to measure the market share represented by Rock Web Server

One is left to wonder how the IT industry correlates the performance reported from SPECweb2005™ submissions to their own environment

IBM, INTEL, 2008


Web Server Market Share% Internet Market Share

0

20

40

60

80

100

Alexa 1000 MediaMetrix 500

100 Hot NetCraftNon-active

NetCraftActive

Market Research Firm

Apache IIS Netscape Zues

Developer 7-Oct Percent 7-Nov Percent ChangeApache 68,155,320 47.73% 76,028,287 50.76% 3.03Microsoft 53,017,735 37.13% 53,679,916 35.84% -1.29Google 7,763,516 5.44% 7,910,879 5.28% -0.15lighttpd 1,541,779 1.08% 1,505,122 1.00% -0.07Sun 2,262,019 1.58% 619,262 0.41% Yes

IBM, INTEL, 2008


Discussion of use of PHP

Variety of market analysis companies have analyzed the percentage of the market share for the various content development languages.

These research firms show levels from 35% to 75% for PHP Depends on whether the analysis includes all known websites or newer websites

being actively developed

The trend is that PHP development has increased from ~25% to ~50% of content development from 2001 to 2007

IBM, INTEL, 2008


SPECweb2005™/PHP Results and Analysis

The key value in a benchmark is its ability to accurately predict for the market segment intended the value of the new hardware or software component.

With the SPECweb2005™/PHP kit there appears to be more similarity

Hardware requirements to achieve peak results are decreased dramatically.

With the SPECweb2005™/PHP results are closer to the default OEM configurations

IBM, INTEL, 2008


Dual-Socket Results

System chosen was Intel® Xeon® 5355 processor dual socket platform Support had the highest connection load Banking produced the lowest connection load

Processor utilization 95% processor utilizations for Support and Ecommerce 80% processor utilization for Banking

Banking workload using PHP has software serialization due in part to the session state having higher processing and IO requirements that limit the processor utilization achievable

SPECweb2005 Score Support Bank Ecommerce Simultaneous Connections GeomeanX5350 964 1900 750 1400 1259X5355 1082 2200 800 1600 1412

IBM, INTEL, 2008


Dual-Socket Response Times

Response times for Ecommerce where nearly equivalent to SPECweb2005™/JSP™ results

Response times for Support were approximately 0.83x that of JSP™

Response times for Banking were lower at 0.47x that of JSP™

Response times are nearly the same across the three workloads.

PHP JSP

Avg Resp Avg Resp

Time (sec)

Time (sec)

Support 5.133 6.334Ecommerce 1.432 1.511Banking 0.652 1.366

SPECweb2005

IBM, INTEL, 2008


Multi-Socket Results

System chosen was Intel® Xeon® 7140M processor

Equivalent characteristics observed to two-socket system

Connection rates improved slightly for Ecommerce and Banking

Slight degradation on the Support workload

Processor UtilizationSupport and Ecommerce higher than 90%Banking slightly higher than 70% (Due to high software serialization)

SPECweb2005 Score Support Bank Ecommerce Simultaneous Connections Geomean7140M 1141 2100 900 1750 1490

IBM, INTEL, 2008


Comparison of Configurations to Published Results

Major determining factor for choice of JSP™ kit is the path length for JSP™ to server the dynamic content is much shorter than for PHP to perform the same task

Significant contrast in hardware requirements required for a top performing JSP™ result and a top performing PHP result

NetworkJSP™ requires 5-18 subnets of 1GbPHP drops to two 1Gb subnets

DiskJSP™ requires 10 times the storage

kit variant Memory Network physical disks SPECweb2005 Score4Socket PHP 32 3 4 1083

JSP 64 17 58 18981

2Socket PHP 16 2 4 1083JSP 32 9 58 20387

IBM, INTEL, 2008


Conclusions

By using the SPECweb2005™/PHP kit a new system can be evaluated at a level of performance with respect to connection loads and response times that more closely matches a real world scenario.

The amount of hardware required to analyze a new system with the SPECweb2005™/PHP kit is much smaller than using the SPECweb2005™/JSP™ kit

IBM, INTEL, 2008


www.cs.ualberta.ca/~amaral/SPEC08

ibm systems & technology group 2008 spec® benchmark workshop ibm, intel, 2008 specweb2005 in...

Documents

spec benchmark workshop

benchmark specweb2005

intel corporation

ibm systems technology

java system web server

web server support http

information technology

overall performance