internet influence on server evolution cmg 2000 conference orlando, florida dec 11, 2000 john...
TRANSCRIPT
Internet Influence on
Server Evolution
CMG 2000 ConferenceOrlando, FloridaDec 11, 2000
John BaudrexlIntel Corporation
Technology & Research LabsLabsLabs
IntelIntel
© 2000 Intel Corporation
2
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
Note: The following material represents the views of the author only, and is not intended to contract, promise, or represent any current or future Intel products.
© 2000 Intel Corporation
3
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
net/www are growing rapidly 1.5B webpages, 1M added daily 400M users, 95M host machines 13M domain names
china: 534K domains in 1 week
bandwidth demand driven by: new user access technologies new applications
larger, latency-sensitive data immense rise in IP traffic
IP BW doubles every 100 days mobile access on the rise
burgeoning e-biz adds fuel forcing buildout of high- bandwidth optical networks
Net Impacts
‘99
# URLs on the WWW
Data Size (KB)
0
100
10000
E-mail Image Hi ResImage
Movie
1000
10
File
100000
© 2000 Intel Corporation
4
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
internet growth giving rise to specialized service providers (SPs) NAPs, ISPs, ASPs, search engines, cacheing, etc colocation services
web/E-biz hosting, storage outsourcing, electronic brokers
gears of internet economy now rely on SPs to be agile, scalable, highly available
SPs deploying powerful, rack-mounted gear mix of networking, storage devices many single function servers
3 traditional tiers: front-end, mid-tier, back-end
SPs beginning to influence server design data centers are becoming a crossroads...
requirements: security, all the ‘bilities’
Net Impacts
© 2000 Intel Corporation
5
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Net Impacts many SPs are locating near NAPs for better performance (response)
17 router hops on ave often tens of ms apart caching technologies are proliferating
more data center floor space needed difficult to build big ones expensive: $500-$700/sf not uncommon retrofits not much cheaper managed services hold key to improved ROI global situation
in the background, Moore’s Law keeps time... so what does all this mean?
© 2000 Intel Corporation
6
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
incentive/capability to make servers result: new “higher density” servers
powerful, memory-rich more units in less volume, floor space easier to deploy and manage rapidly/remotely configurable reasonable cost more modular, available, reliable the server is the FRU - service it offline
prediction: high density storage also coming 10,000 petabytes of storage anticipated online
worldwide by 2005 much of it stored in data centers
smaller!
Net Impacts
“1U is too big.”
© 2000 Intel Corporation
7
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
© 2000 Intel Corporation
8
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
The I/O Squeeze shrinking form factor colliding with:
increasing Hz, shorter parallel busses CPU and memory power dissipation (heat) size of power supply/heat removal components size of on-board I/O infrastructure
very careful signal/layer layout required thermal challenges abound
in opposition with reliability
not much room for fans, PS, VRMs, etc.
liquid-based cooling solutions being pondered
can’t pack enough I/O close enough anyway without violating PCI standards specs...
© 2000 Intel Corporation
9
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
PCI bus technology: 1994 - industry readily embraced spec
32-bit/33MHz propagated rapidly first generation chipsets had limited scalability
1996 – first 64/33 bus available 1998 – first 64/66 bus available 2000 – first PCI-X bus (100MHz) available trends -
hot plug busses tuning/awareness has led to better performance
getting good feeds and speeds nowadays seeing 80-90% efficiency on modern busses BW exceeds most I/O interface technology req’s multi-channel cards emerging
The I/O Squeeze
© 2000 Intel Corporation
10
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
PCI bus technology issues -
memory mapped “load-store” ops stall faster CPUs no standard driver card behavior scalability - out of slots, out of I/O reliability - bus parity err causes system reset stretching a parallel PCI bus is impractical
cost (more pins, cables, connectors, etc.) skew/signal integrity issues increased power needed
The I/O Squeeze
© 2000 Intel Corporation
11
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
The I/O Squeeze
lower speeds = more slots, longer busses multiple cards/chls can lead to chaotic bus behavior
higher speeds = fewer slots, shorter bus lengths 133MHz pushing limit of modern parallel busses one slot looks like internal bus extension
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
64/6664/66
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
||||
64/1064/1000
||||
||||
||||
||||
||||
||||
||||
64/1364/1333
MemCtrl
I/OHub
Memory
I/OBridgeI/O
BridgeI/OBridgeCPU
™
CPU
™
high density system boundary
PCI Slot
s
traditional system boundary
© 2000 Intel Corporation
12
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
© 2000 Intel Corporation
13
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
InfiniBand* Technology...a new interconnect
Rev 1.0 spec at: www.infinibandta.org
© 2000 Intel Corporation
14
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
InfiniBand* Technology new industry specification endorsed by majority server OEM, I/O people many new concepts - products avail end ’01
allows separation of CPU/memory from I/O reliable, scalable, switched, packet-based, hardware-routed, serial message-passing fabric architecture designed to endure at least 10 years
defines protocol, pgm model, wire, interop, err handling expected to replace PCI in servers
new type of I/O bridge no PCI slots anymore, plug in hoses instead
based on VI Architecture: QP/CQ concept, rDMAs high speed serial pt-to-pt links (2.5Gb/sec)
widths: 1x = 500MB/sec, 4x = 2GB/sec, 12x = 6GB/sec bi-dir range of service classes, flow control, QoS “verbs” software interface for IPC, I/O
many other features – see website
© 2000 Intel Corporation
15
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
InfiniBand* Technology
RouterRouter
CACA
NetworkNetwork
Lin
kL
ink
SysSysMemMem
CPUCPU
CPUCPU
MemMemCntlrCntlr HCAHCA
LinkLinkSwitchSwitch
LinkLink
Lin
kL
ink
TCATCA TargetTarget
TCATCA
TargetTargetH
ost
In
terc
on
nec
tH
ost
In
terc
on
nec
t
RouterRouterLinkLinkSource: InfiniBand* Trade Association
one interconnect for storage, net, IPC
© 2000 Intel Corporation
16
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
© 2000 Intel Corporation
17
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
A Clustering Vision skinny powerful servers
1-2 IA processors, maybe even Itanium™ gobs of memory (>4GB if you want) all-InfiniBand* I/O – no PCI slots completely remote management external 48V power supply
legacy free no floppy, no keyboard, mouse, video, CD-rom,
serial, parallel - all gone! no hard drive – boot from anywhere reduces volume, minimizes human interaction
only 2 kinds of hoses attached power cord InfiniBand* cables
© 2000 Intel Corporation
18
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
A Clustering Vision cluster together as needed
deploy lightweight IPC protocol such as VIA out-of-band management and configuration
attach external I/O devices (gateways) storage (IBA-FC/SCSI), network (IBA-LAN/WAN) etc.
add intelligent, scalable network services distributed, smart packet processing at line rates
build in large distributed memory use lightweight IPC transport over IBA remote memory access << local/ext disk access fast response time – replaces some cache tech meets e-commerce scalability requirement
new price/perf potential for e-Biz IA platforms usually best price/performance IBA* chipsets low cost/port over time
© 2000 Intel Corporation
19
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
A Clustering Vision ten nodes
20 Itanium™ (“McKinley”) processors gobs of MIPS, flops
80GB distributed RAM >1TB internal HDD 40GB/s bi-directional redundant I/O BW
two 4x IBA* links per node
6kW peak power 3kW average
total 10U - 18” high!
© 2000 Intel Corporation
20
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
© 2000 Intel Corporation
21
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
thermal challenges some compelling solutions emerging rack may also now require thermal engineering
products implementing InfiniBand* links still a year away
partners will gain early access true legacy free systems still a pipe dream
product quality BIOS, backward compatibility work underway in this direction
clusters are hard to implement and manage reliable failover hard enough performance clustering harder still linux helps some lots of momentum in the industry to change this
Some Issues
© 2000 Intel Corporation
22
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
increased density impacts the data center power delivery
W/sf2 req’s increasing faster than industry supply cooling
new construction: raised floor retrofit: ceiling supply only
cost effectiveness site selection demands proximity to power/net grid reality is multi-dimensioned, complex
provision for greater backup capability more batteries, generators, conversion/delivery
environmental zoning prevents storage of sufficient diesel fuel! heat plume an EPA concern
despite apparent economies of scale,“super” data centers will be rare
Some Issues
© 2000 Intel Corporation
23
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Product Heat-Density Trends
Some Issues
© 2000 Intel Corporation
24
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
electrical power consumption in U.S. today: 8% internet, 5% computing projection: by 2010, 30% of U.S. consumption will go to IT
data center power density trend mainframe days: 20-40 W/ft2
“standard” today: ~60-100 W/ft2
(way) high end today: ~200 W/ft2
some folks looking for 300-500 W/ft2
cooling physics 1 ton of AC, 500cfm airflow at 20oF ΔT dissipates ~4kW generating 1 ton of AC takes ~1kW of input power harder to remove heat than to supply power
equipment heat generation (commonly operate at ~50% of nameplate listed below) SMP servers today: 7U = 700W; 4U = 400W; 2U = 200W; 1U = 150W 3U Clariion 5000-series 10 drive FC JBOD disk array = 350W 11U Cisco Catalyst 6000-series 32Gb LAN/WAN switch = 1.3kW
a 42U rack: occupies 15-20 ft2, accounting for walkways and access 100 W/U * 42U * 50% nameplate ÷ 20 ft2/rack
power density requirement ~105 W/ft2
Issues: Density FactoidsSource: Intel ASL research
© 2000 Intel Corporation
25
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary
© 2000 Intel Corporation
26
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Summary internet growth creating crossroads at
service providers, data centers Moore’s Law, SPs are driving server
evolution to higher densities pushing I/O out of the box data center infrastructure not ready for higher
density equipment impacts need to develop mitigation strategy
IBA* Architecture fits data center req’s convergence creates compelling sol’n space
high density large memory InfiniBand* Architecture
© 2000 Intel Corporation
27
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
Backup Material
© 2000 Intel Corporation
28
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
0
50
100
150
200
250
300
350
400
450
1997 1998 1999 2000 2001
Year
Max B
an
dw
idth
(M
Bp
s)
ATMEthernetFC1394SCSI
Source: In-Stat, Electronic Trend
Chart based on bi-directional bandwidths, and onestimated availability of new standards/technologies
I/O Interface Bandwidth Trend: Increasing!
© 2000 Intel Corporation
29
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
HOW LONG TO REACH 30% PENETRATION? (U.S. Internet Council, Apr. 1999).
Internet -- 7 years Television – 17 years Telephone – 38 years Electricity – 46 years.
# ONLINE WORLDWIDE 1998 – 95.43 million people. (eMarketer eStats 1999). 1998, Dec. – 144 million (IDC, Dec. 1999). 1999, Dec. – 240 million (IDC, Dec. 1999). 2000, Sept. – 377 million (Nua Internet Surveys (www.nua.ie)2002 – over 490 million (Computer Industry Almanac, Nov. 1999). 2005 – over 765 million (Computer Industry Almanac, Nov. 1999).
Net Explosion
Source : An IDC Telebriefing, March 4, 1999
0100200300400500600700800
1997 1998 1999 2000 2001 2002 2003
Devices
Users
© 2000 Intel Corporation
30
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
HOST COMPUTERS: In July 1999 there were 56.2 million "host" computers supporting web pages. In July 1997 there were 19.5 million host computers, with 3.2 million hosts in July 1994, and a mere 80,000 in July 1989. (Internet Software Consortium – Internet Domain Survey).
Monthly Average Hosts in Millions (Nov. 2000, Telecordia Technologies www.netsizer.com)Month 1998 1999 2000 Jan 30.3529 44.2292 70.2238Feb 31.3906 48.0616 72.9501Mar 32.4267 50.4808 75.1010Apr 33.5716 53.4689 77.1504May 34.5158 55.8092 80.1905Jun 35.4504 57.3499 82.8749Jul 36.4862 59.2922 85.8418Aug 37.2169 61.3617 88.6271Sep 38.5637 63.3344 91.5614Oct 39.9339 65.6407 94.2659Nov 41.5190 67.3174 -Dec 43.0331 68.8020 -
Net Explosion
© 2000 Intel Corporation
31
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
BACKBONE CAPACITY: The capacity of the Internet backbone to carry information is doubling every 100 days. (U.S. Internet Council, Apr. 1999).
DATA TRAFFIC SURPASSING VOICE: Voice traffic is growing at 10% per year or less, while data traffic is conservatively estimated to be growing at 125% per year, meaning voice will be less than 1% of the total traffic by 2007. (Technology Futures, Inc March 2000).
EMAIL VOLUME: Average U.S. consumer will receive 1,600 commercial email messages in 2005, up from 40 in 1999, while non-marketing and personal correspondence will more than double from approximately 1,750 emails per year in 1999 to almost 4,000 in 2005 (Jupiter Communications, May 2000).
DOMAIN NAMES: There are 12,844,877 unique domain names (e.g. Cisco.com) registered worldwide, with 428,023 new domain names registered each week. (NetNames Statistics 12/28/1999).
Net Explosion
© 2000 Intel Corporation
32
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
TOTAL AMOUNT OF DATA: 1,570,000,000 pages, 29,400,000,000,000 bytes of text, 353,000,000 images, and 5,880,000,000,000 bytes of image data. (The Censorware Project, Jan. 26, 1999).
NEW DATA EACH DAY: In just the last 24 hours, the web has added 3,180,000 new pages, 59,700,000,000 new bytes of text, 716,000 new images, and 11,900,000,000 new bytes of image data. (The Censorware Project, Jan. 26, 1999).
WEB PAGE LIFE SPAN: The average life span of a web page is about 44 days. (The Censorware Project, Jan. 26, 1999).
TRACKING WEB CONTENT CHANGES: To keep up with the changes to web content, you'd need to download about 873,000,000,000 bytes of information per day, which would mean you'd need a connection capable of downloading 10,100,000 bytes per second. (The Censorware Project, Jan. 26, 1999).
Net Explosion
© 2000 Intel Corporation
33
LabsLabsIntelIntel
*All trademarks and brands are the property of their respective owners
DOCUMENTS ON WEB: The web now contains over one billion unique, indexable documents. (Emarketer/ Inktomi Corp., Jan. 2000).
CORPORATE EMAILS: In 2004, companies will send more than 200 billion e-mails. (Forrester Research, February 2000).
DOWNLOAD SPEEDS IMPROVING: Average homepage took 4.73 seconds to download by the end of 1999, 27% faster than at the beginning of the year (6.49 seconds). (Keynote Systems, April 2000).
NEW WEB PAGES: On average, more than 3 million Web pages were created every day in 1999. (IDC, Apr. 2000).
WEB HITS/DAY 1999: U.S. web pages averaged one billion hits per day (aggregate) in October 1999. (eMarketer/Media Metrix, Nov. 1999).
Net Explosion