besiii computing environment

BesIII Computing Environment ------Computer Centre , IHEP, Beijing.

[email protected]

BESIII Computing Environment


[email protected]

1. Computing in IHEP

2. BESIII Computing

Pre-Grid

1. Computing in IHEP


[email protected]

Event size (kB)

Number of Event Data volume （ GB

）

Raw data 2 ~1.5×108 300Rec. data 6 ~1.5×108 900DST data 0.6 ~1.5×108 90

Current BESII ComputingCurrent BESII Computing

The total real and Monte Carlo data produced by The total real and Monte Carlo data produced by

BES and BESII are about BES and BESII are about 13 TB13 TB. .

The BESII computing environment is The BESII computing environment is built on:built on:

•PC/Linux/Cluster

•NFS/AFS

•openPBS/GangLia

•CERN/CASTOR


Schematic view of BESII computing environment


The BESII Computing Environment:The BESII Computing Environment:

• 6 file servers (2 GB memory and extra gigabit network adapter for

each).

• 7.3 TB disk space.

• 64 worker nodes (Pentium IV 2.6 and 2.8 GHz CPUs and 1 GB memory for each).

• CISCO and Cabletron Switches.


2.BesIII Computing

BesIII Computing Environment------Computer Centre , IHEP, Beijing.

[email protected]

Event size (kB)

Number of Event Data volume （ TB

）

Raw data 12 ~1.×1010 120Rec. data 24 ~1.×1010 240DST data 2 ~1.×1010 20

• The BESIII experiment is planed to begin to collect data in 2007.

• The peak luminosity of the BEPCII at the J/ Ψ resonance will be about 1033cm-2s-1, with the peak event rate at 3000 Hz.

• The J/ Ψ production in the first year will collect about 1×1010 J/Ψ events.

• In the following years the production for Ψ’, D and Ds studies one will also totally collect about another 1×1010 events .

BesIII Computing Environment

Estimated data size of BESIII experiment in the first production year


Data volume （ TB ） Storage mediumJ/ Ψ event Ψ’, D, Ds event

Raw data 120 120 Tape

Rec. data 240×2 240×2 Tape

DST data 20×2 20×2 Disk

MC 240 240 Tape

MC-Rec. 240×2 240×2 Tape

MC-DST 20×2 20×2 Disk

Sum 2,800 （ TB ）

• The number of events of the MC generated data will be 2 times as the real data.

• 2 versions of the reconstructed real data will be stored , while the simulated data will store only one copy

Estimated total amount of data of BESIII in it’s whole production life time

1. It is very complicated and difficult to estimate the CPU number needed for the whole data production works, MC simulation works, and physics analysis, since there’s no direct experience of the data processing from BESIII it self at present, and the IT technology develops so fast.

2. The total estimated CPU power needed for data process and data analysis for BESIII experiment will be accumulated up to

1,000 to 2,000 the fastest CPUs at present.

3. This number is a very rough estimation, and will vary with the knowledge of the BESIII data processing, and with the IT trend in the following years.



Data flow of BESIII experiment

and other services: HTTP service Mail service AFS Home directory service Application service …


The BesIII Computing Environment will provide 3 main services:

1. Public interactive Login service

2. Batch job service

3. Storage service

BESIII Analysis

Public Logon Service Platform

Physicist user

Batch cluster

Mass storage

Uniform environment variables

Uniform file accessUniform tool accessUniform library access

Load Balancing System

Data access

Computing

•MC

•Reconstruction

•Analysis PB

GB/MB

Computing Mode

Pre-Grid

The operating system and tools software of the BESIII computing environment will mainly use free software.


One should note the risk when using free software: performance and stability.

------Computer Centre , IHEP, Beijing.

[email protected]

OpenPBS vs. PBS Pro


PBS Pro vs. LSF


[email protected]

BQS?

CONDOR?

……?


Physical part: Logical part:

1. Computing Element

2. Storage Element

3. Network Element

4. Security Element

I. Security Policy

II. Administration Policy

III. User Policy

Construction/Upgrading

Maintenance/Supporting


The BESIII computing environment is planed to be constructed step by step during 2005 –2010.

1. Jan., 2005—Dec., 2005: Study on each of the computing components, CPU, disk, Tape, SLC3 linux system, batch job management software, etc. Design a computing prototype, test the hardware and software components in an integrated environment.

2. Jan., 2006—Dec., 2006: Build a test-bed according to the prototype with full functions, with the 1/10 scale of the complete BESIII computing environment. Do the first Data Challenge and detailed system tests. Extend the test-bed to a 1/2 scale and then do the second Data Challenge.

3. Jan., 2007— : Extend the system to the full scale and provide full computing services needed by the first year data production of BESIII experiment. Extend the system along with the increasingly requirements of the experiment.


BESIII Prototyp

e


•Test the hardware

•Test the software

•Data Challenge Test

•Train people

•…

On the prototype, we need:

BES On line system

Offline PC Farm YBJ

CERN (CMS&Atlas) Computing Center

pku

CASTOR

Physics data cache

Tier 1

Tier 2

Tier 3This diagram only

Shows HEPDataGrid in China

10Gbps

E-science


1. The BESIII computing environment will be constructed in an open architecture, which is robust, secure, scalable and extendable, and is open friendly to be integrated with other new technologies, such as Grid.

2. To share the computing resources with other BESIII collaborators, and other HEP experiments.

Grid?

Thanks


[email protected]

Suppose:10 physicists access the 1/100 of the whole data and do analysis, ， want to get results in 24 小时：

55 SI95

J/Ψ

J/Ψ + Ψ ’ : 2.6*104 *2 --> 6 *104 * 2 ？） peak requirement

J/Ψ:

Estimation


BesIII Computing Technical Proposal ------Computer Centre , IHEP, Beijin

g.

[email protected]

3. IT Technology Trend


g.

[email protected]

PIV 2.8G =:1000 SI2K, <1000 USD

( 包括主板、内存、 CPU 、硬盘、机箱 )

PASTA, the Technology Tracking Team for Processors, Memory, Storage and Architectures, LCB, CERN.

1 SI95 ～ 10 SI2K~10 CU


g.

[email protected]

1 SI95 ～ 10 SI2K~10 CU


g.

[email protected]

2000 SI2K

1 SI95 ～ 10 SI2K~10 CU


g.

[email protected]


g.

[email protected]

0

200

400

600

800

1000

1200

1400

2. 4G 2. 6G 2. 8G 3. 0G 3. 2G

I ntel P4 CPU系列

SI20

00

00. 20. 40. 60. 811. 21. 41. 61. 8

/SI2

000

圆

SI 2000/SI 2000圆

0100200300400500600700800900

1000

2000+ 2400+ 2600+ 2800+ 3000+AMD XP CPU系列

SI20

000

0. 2

0. 4

0. 6

0. 8

1

1. 2

1. 4

/SI2

000

圆

SI 2000

/ SI 2000圆

AMD:Advanced Micro Devices, Epox 8KHA+ Motherboard, AMD Athlon (TM) XP

Intel:Dell Precision Workstation 360 (P4) 系列

Data from Chinese market, Oct. 2004


g.

[email protected]

Storage


g.

[email protected]不同容量硬盘性价比：2004年10月，中国北京。

012345678

80G 120G 160G 200G 250G

GB硬盘容量（）

/GB

圆

数据来源：媒体公告价格。

Storage


g.

[email protected]

Comparison of current different Tape Technology

Storage


g.

[email protected]

Comparison of different Tape Medias.

LTO2?


g.

[email protected]

Market Response


g.

[email protected]

4.Cost Estimate


g.

[email protected]

Electrical Power supply:1. Computing box: 300w * 500 (box) = 150 KW

2. Storage system: 10 KW

3. Cooling system: 10 KW

170 KW

Room Space:1. Computing box: 150cm*50cm*500(box)=375 m2/(3 layer) =: 125m2

2. Storage system: 50m2

3. Cooling system: 10m2

~200 m2

PC Farms @ CERN

http://tjs.web.cern.ch/tjs/


g.

[email protected]

Man Power:1. R&D group 5(soft)

+5(hard)

2. Maintenance&Technical Support 5(soft)+10(hard)

30??


g.

[email protected]

5.Load Balancing & Cluster

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxp

lus0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

lxb

atc

h0

01

load balancing

AFS/LSF?

dis

k0

01

rfio

tap

e0

01

rfio

dis

k0

01 tap

e0

01

Batch Servers(FARM)

Interactive Servers (LxPlus)

Disk Servers

http://tjs.web.cern.ch/tjs/

Data Storage(CASTOR)

Tape Servers

IHEP future computing environment Junior

CERN?

IHEP Public Linux Logon Platform(LxPlus?)

LVS

besiii computing environment

Documents

data of besiii

besiii data processing

data analysis

simulated data

data production works

reconstructed real data

mc generated data

monte carlo data