IDE disk servers at CERN

Helge Meinhard / CERN-IT

CERN OpenLab workshop

17 March 2003

Introduction

HEP computing in the past: mostly reading from, processing, and writing to (tape) files sequentially

Mainframe era (until ~ 1995): single machine, CPUs, tape drives, little disk space

In response to scaling problem, development of SHIFT architecture (early 1990s) Scalable farm out of ‘commodity’ components

RISC CPUs (PowerPC, MIPS, Alpha, PA-RISC, Sparc) SCSI disks

SHIFT architecture

Network- FDDI- Hippi- Myrinet- Ethernet

Tapeserver

Diskserver

Batchserver

Interactiveserver

BatchanddiskSMPNetwork

- Ethernet

PC batch nodes

1995: First studies at CERN of PCs as batch nodes (Windows NT)

1997 onwards: Rapidly growing interest in Linux (on IA32 only)

1998/99: First production farms for interactive and batch services running Linux on PC hardware at CERN

PC disk servers

1997/98: Prototypes with SCSI disks 1998/99: Prototypes with EIDE disks

Different IDE adapters Not RAIDed

1999/2000: First Jumbo servers (20 x 75 GB) put into production

2001: First rack-mounted systems 2002: 97 new servers (54 TB usable) 2003: 1.3 TB usable in one server at 13 kCHF Total usable capacity today: ~ 200 TB

1997: ~700 GB SCSI/Sparc

2000/2001: 750 GB PC/EIDE (1)

2000/2001: 750 GB PC/EIDE (2)

2002: 670 GB PC/EIDE

2 systems

Assumed price per GB usable

0

50

100

150

200

250

300

350

400

450

1994 1995 1996 1997 1998 1999 2000 2001 2002

Time

CH

F /

GB

Disks only

Complete systems

SCSI/RISC

EIDE/PC

Real price per GB incl. infrastructure

0

5

10

15

20

25

30

35

40

45

50

Oct-

00

Nov-0

0

Dec-0

0

Jan-

01

Feb-0

1

Mar

-01

Apr-0

1

May

-01

Jun-

01

Jul-0

1

Aug-0

1

Sep-0

1

Oct-

01

Nov-0

1

Dec-0

1

Jan-

02

Feb-0

2

Mar

-02

Apr-0

2

May

-02

Jun-

02

Jul-0

2

Aug-0

2

Sep-0

2

Oct-

02

Nov-0

2

Dec-0

2

Jan-

03

Feb-0

3

Date

CH

F /

GB

Capacity per server

0

500

1000

1500

2000

2500

3000

3500

Oct-

00

Nov-0

0

Dec-0

0

Jan-

01

Feb-0

1

Mar

-01

Apr-0

1

May

-01

Jun-

01

Jul-0

1

Aug-0

1

Sep-0

1

Oct-

01

Nov-0

1

Dec-0

1

Jan-

02

Feb-0

2

Mar

-02

Apr-0

2

May

-02

Jun-

02

Jul-0

2

Aug-0

2

Sep-0

2

Oct-

02

Nov-0

2

Dec-0

2

Jan-

03

Feb-0

3

GB

Gross

Usable

Today’s servers: Specifications

19” rackmount, IA32 processor(s), 1 GB, 2x80 GB system disks, GigE (1000BaseT), redundant power supplies

>500 GB usable space on data disks Hardware RAID offering redundancy Hot-swap disk trays

Performance requirements network – disk: 50 MB/s reading from server @ 500 GB 40 MB/s writing to server @ 500 GB

5 years on-site warranty

Lessons learnt

Capacity is not everything, for good performance need CPU, memory, RAID cards Good OS and application software Network connectivity Large number of spindles

Firmware of RAID controllers and disks critical Redundancy (RAID) is a must, required

performance possible only with mirroring (RAID 1) so far

Outlook

Good price/performance has risen interest in other application domains at CERN AFS and MS DFS servers Web servers, mail servers Software servers (Linux installation) Data base servers (Oracle, Objectivity/DB)

Access pattern of physics analysis likely to change Investigating different file systems (XFS), RAID 5 (in

software), … Architecture constantly being reviewed

Alternatives investigated: data disks scattered over large number of batch nodes; SAN

Conclusion

Architecture of early 1990s still valid May even carry us into LHC era…

Important improvements made Price/performance Reliability (RAID)

Will review architecture soon (2003) New application areas New access patterns for physics analysis