grid efforts in belle
DESCRIPTION
Grid Efforts in Belle. Hideyuki Nakazawa (National Central University, Taiwan), Belle Collaboration, KEK. Out Line. Belle experiment Computing system in Belle LCG at KEK and Belle VO status Introduction of SRB Summary. Mt. Tsukuba. Belle. KEKB. 3km. Linac. Belle Experiment. - PowerPoint PPT PresentationTRANSCRIPT
3/27/2007 Grid Efforts in Belle 1
Grid Efforts in Belle
Hideyuki Nakazawa(National Central University, Taiwan),
Belle Collaboration, KEK
3/27/2007 Grid Efforts in Belle 2
Out Line
Belle experiment Computing system in Belle LCG at KEK and Belle VO status Introduction of SRB Summary
3/27/2007 Grid Efforts in Belle 3
KEKB Accelerator•Asymmetric e+e- collider•3.5 GeV on 8 GeV
•3 km circumference•22mrad Crossing Angle•Continuous InjectionBelle Detector•Generic purpose•7 sub-detectors
“B factory” experiment at KEK (Japan).
BelleKEKB
Linac
3km
Mt. Tsukuba
Belle Experiment
BBSee )4(
3/27/2007 Grid Efforts in Belle 4
Belle Collaboration
13 countries, 57 institutes, ~400 collaborators
IHEP, ViennaITEPKanagawa U.KEKKorea U.Krakow Inst. of Nucl. Phys.Kyoto U. Kyungpook Nat’l U. EPF Lausanne Jozef Stefan Inst. / U. of Ljubljana / U. of MariborU. of Melbourne
Aomori U.BINPChiba U.Chonnam Nat’l U.U. of CincinnatiEwha Womans U.Frankfurt U.Gyeongsang Nat’l U.U. of HawaiiHiroshima Tech.IHEP, BeijingIHEP, Moscow
Nagoya U.Nara Women’s U.National Central U.Nat’l Kaoshiung Normal U.National Taiwan U.National United U.Nihon Dental CollegeNiigata U.Osaka U.Osaka City U.Panjab U.Peking U.U. of PittsburghPrinceton U.RikenSaga U.USTC
Seoul National U.Shinshu U.Sungkyunkwan U.U. of SydneyTata InstituteToho U.Tohoku U.Tohuku Gakuin U.U. of TokyoTokyo Inst. of Tech.Tokyo Metropolitan U.Tokyo U. of Agri. and Tech.Toyama Nat’l CollegeU. of TsukubaUtkal U.VPIYonsei U.
Lots of contribution from TaiwanLots of contribution from Taiwan
3/27/2007 Grid Efforts in Belle 5
LuminosityProduce large amount of B mesons!!
peak luminosity1.7118 × 1034 cm-2s-1
710 fb-1
1 fb-1~106 BB
Inte
grat
ed L
umin
osit
y (f
b-1)
●Crab CavityCrab Cavity installed,installed, being tuned now.being tuned now. Luminosity doubled?Luminosity doubled?
Integrated Luminosity
1 fb-1 ~ 1TB / day
3/27/2007 Grid Efforts in Belle 6
History of Belle computing system
Performance 1997-4 years
2001-5 years
2006-6 years
Computing Server[SPECint2000 rate]
~100(WS)
~1,250(WS+PC)
~42,500(PC)
Disk Capacity [TB]
~4 ~9 1000
Tape Library Capacity[TB]
160 620 3,500
Work Group Server[# of hosts]
3+(9) 11 80+16FS
User Workstation[# of hosts]
25WS+68X
23WS+100PC
128PC
3/27/2007 Grid Efforts in Belle 7
Overview of the B Computer
Storage
ComputingServers
WorkgroupServers
reservedfor Grid
On-lineReconstructionFarm
3/27/2007 Grid Efforts in Belle 8
Belle SystemBelle System
Computing Server: ~42,500 SPECint2KComputing Server: ~42,500 SPECint2KStorage System (DISK): 1PBStorage System (DISK): 1PB
Storage System (HSM): 3.5PBStorage System (HSM): 3.5PB
3/27/2007 Grid Efforts in Belle 9
Data Production at Belle
onlinereconstructionfarm
““MDST” dataMDST” data (four vector, PID info etc.)(four vector, PID info etc.)
rawdata +rawdata +““DST” dataDST” data
production
Users' analyes
hadron 120TB+ others
~ 1PB
MCMC
Generation and
DetectorSimulation
2.5THz(to finish in6 months)
2THz(to finish in2 months)
HSM
non-HSM
Loose Loose SelectionSelection Criteria Criteria
@500/fb@500/fb
3/27/2007 Grid Efforts in Belle 10
Why Grid in Belle? No urgent requirement No urgent requirement Belle shifts to precise and exotic measurementBelle shifts to precise and exotic measurement
More MC statistics necessary for precise More MC statistics necessary for precise measurementmeasurement
New skim for exotic processNew skim for exotic process Lesson in de facto standardLesson in de facto standard
Maybe we should Maybe we should start considering start considering
about Gridabout Grid
Just my feeling
Just my feeling
3/27/2007 Grid Efforts in Belle 11
Grid Introduction Strategy Strong support from KEK CRCStrong support from KEK CRC Starting with MC production and Starting with MC production and
accumulating experiences, gradually accumulating experiences, gradually shift to handle experimental data shift to handle experimental data
RecruitmentRecruitment Some collaborators who have running Some collaborators who have running
LCG are preparing to join the Belle VOLCG are preparing to join the Belle VO Experiencing Grid potential may Experiencing Grid potential may
changechangeBelle’s recognitionBelle’s recognition??
3/27/2007 Grid Efforts in Belle 12
LCG Deployment at KEKLCG Deployment at KEK
JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 Since Nov. 2005.Since Nov. 2005. Registered to GOC, in operatiRegistered to GOC, in operati
on as WLCGon as WLCG Site Role:Site Role:
practice for production systepractice for production system JP-KEK-CRC-02.m JP-KEK-CRC-02.
test use among university groups in Japtest use among university groups in Japan.an.
Resource and Component:Resource and Component: SL-3.0.5 w/ gLite-3.0 laterSL-3.0.5 w/ gLite-3.0 later CPU: 14, Storage: ~1.5TBCPU: 14, Storage: ~1.5TB FTS, FTA, RB, MON, BDII, LFC, CE, SEFTS, FTA, RB, MON, BDII, LFC, CE, SE
Supporting VOs:Supporting VOs: bellebelle, apdg, g4med, ppj, dteam, ops an, apdg, g4med, ppj, dteam, ops an
d aild ail
JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02 Since early 2006.Since early 2006. Registered to GOC, in operation Registered to GOC, in operation
as WLCGas WLCG Site Role:Site Role:
More stable services based on KEKMore stable services based on KEK-1 experiences. -1 experiences.
Resource and Component:Resource and Component: SL or SLC w/ gLite-3.0 laterSL or SLC w/ gLite-3.0 later CPU: 48, Storage: ~1TB (w/o HPSCPU: 48, Storage: ~1TB (w/o HPS
S)S) Full componentsFull components
Supporting VOs:Supporting VOs: bellebelle, apdg, g4med, atlasj, ppj, ilc, , apdg, g4med, atlasj, ppj, ilc,
dteam, ops and aildteam, ops and ail
Operation is supported by great efforts by APOperation is supported by great efforts by APROC members in ASGC.ROC members in ASGC.Operation is supported by great efforts by APOperation is supported by great efforts by APROC members in ASGC.ROC members in ASGC.
3/27/2007 Grid Efforts in Belle 13
Belle VOBelle VO 9 sites Belle software are installed to 3 sites (KEK x2, ASGC)
~60 CPUs 2TB storage MC production ongoing
Installation manual ready GFAL with Belle software
3/27/2007 Grid Efforts in Belle 14
Total Number of Jobs at KEK in 2006Total Number of Jobs at KEK in 2006
JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02
200200
700700
400400
1,0001,000
1,4001,400
BelleBelleBelleBelleBelleBelleBelleBelle
3/27/2007 Grid Efforts in Belle 15
Total CPU Time at KEK in 2006Total CPU Time at KEK in 2006(Normalized by 1kSI2K)(Normalized by 1kSI2K)
JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01JP-KEK-CRC-01 JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02JP-KEK-CRC-02
4,0004,000
3,0003,000
1,000 [hrs kSI2K]1,000 [hrs kSI2K]
12,00012,000
10,00010,000
4,0004,000
BelleBelleBelleBelleBelleBelleBelleBelle
3/27/2007 Grid Efforts in Belle 16
Logical Site OverviewLogical Site Overview
KEK FirewallKEK Firewall SuperSINETSuperSINETSuperSINETSuperSINET
HSMHSMHSMHSM
Grid LANGrid LAN
KEK-2KEK-2202.13.197.0/24202.13.197.0/24
KEK-2KEK-2202.13.197.0/24202.13.197.0/24
KEK-DMZKEK-DMZ
MCATMCAT172.22.28.0/24172.22.28.0/24
MCATMCAT172.22.28.0/24172.22.28.0/24
130.87.224.0/21130.87.224.0/21
SRBSRB172.22.28.0/24172.22.28.0/24
130.87.224.0/21130.87.224.0/21
SRBSRB172.22.28.0/24172.22.28.0/24
SRB-DSISRB-DSI130.87.104.0/22130.87.104.0/22
SRB-DSISRB-DSI130.87.104.0/22130.87.104.0/22
KEK-CCKEK-CC
KEK-1KEK-1130.87.208.0/22130.87.208.0/22
KEK-1KEK-1130.87.208.0/22130.87.208.0/22
$ scp output Belle:$ scp output Belle:
$ scp input Grid:$ scp input Grid:
Local files CPUsCPUs
WSWS
3/27/2007 Grid Efforts in Belle 17
SRB Introduction ScheduleSRB Introduction Schedule
Construction PlanningConstruction Planning GridGrid Belle OperationBelle Operation NetworkingNetworking KEKCC/IBMKEKCC/IBM
Construction PlanningConstruction Planning GridGrid Belle OperationBelle Operation NetworkingNetworking KEKCC/IBMKEKCC/IBM
MCATMCATMCATMCAT
SRBSRBSRBSRB
FWFWFWFW
SRB-DSISRB-DSISRB-DSISRB-DSI
TestTestTestTest
ConnectionConnectionConnectionConnection
Start OperationStart OperationStart OperationStart Operation
PreparationPreparation
3/27/2007 Grid Efforts in Belle 18
Belle Grid Deployment Future PlanBelle Grid Deployment Future Plan Federate with Japanese universities.Federate with Japanese universities.
KEK hosts the Belle experiment and behaves as Tier-0.KEK hosts the Belle experiment and behaves as Tier-0. Univ. with reasonable resources: full LCG (Tier-1)Univ. with reasonable resources: full LCG (Tier-1) Univ. without resources: UIUniv. without resources: UI
The central services such The central services such as VOMS, LFC and FTS as VOMS, LFC and FTS are provided by KEK. are provided by KEK.
KEK also covers web KEK also covers web Information and support Information and support service.service.
Grid operation is co-Grid operation is co-operated with 1~2 staffs operated with 1~2 staffs in each full LCG site.in each full LCG site.
JP-KEK-CRC-02JP-KEK-CRC-02 JP-KEK-CRC-03JP-KEK-CRC-03
UniversityUniversityUIUI
UniversityUniversityUIUI
UniversityUniversityUIUI
UniversityUniversity
UIUIUniversityUniversity
UIUIUniversityUniversity
UIUIUniversityUniversity
UIUIUniversityUniversity
UIUIUniversityUniversity
UIUI
Tier-0Tier-0
Tier-1Tier-1
deploy in the futuredeploy in the future
preliminary designpreliminary design
3/27/2007 Grid Efforts in Belle 19
Summary
Belle VO launchedBelle software are installed to 3
sites KEK sites are mainly used by Belle
MC production ongoingSRB is being introduced
3/27/2007 Grid Efforts in Belle 20
Additonal (Belle's) Resources
We now obtain high-performance computer system;but we didn't suddenly switch to the “less expensive” system.
350TB disks1.5PB tapes
934 CPUs
20units/20TB
We have been testing suchsystem for several years.
●Linux based PC clusters●S-ATA disk based RAIDdrives
●S-AIT tape drives
1000TB disks3.5PB tapes
2280 CPUs B computerfor comparison
These resources have been essentialfor Belle (production/analysis)
3/27/2007 Grid Efforts in Belle 21
Belle Grid Deployment PlanBelle Grid Deployment Plan
We are planning a 2-phased deployment for BELLE experimWe are planning a 2-phased deployment for BELLE experiments.ents. Phase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with oPhase-1: BELLE user uses VO in JP-KEK-CRC-02 sharing with o
ther VOs.ther VOs. JP-KEK-CRC-02 consists of “JP-KEK-CRC-02 consists of “Central Computing SystemCentral Computing System” maintaine” maintaine
d by IBM corporation.d by IBM corporation. Available resources:Available resources:
CPU: 72 processors (opteron), SE: 200TB (with HPSS)CPU: 72 processors (opteron), SE: 200TB (with HPSS) Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production Phase-2: Deployment of JP-KEK-CRC-03 as BELLE Production
SystemSystem JP-KEK-CRC-03 uses a part of “JP-KEK-CRC-03 uses a part of “B Factory Computer SystemB Factory Computer System” resour” resour
ces.ces. Available resources (maximum estimation)Available resources (maximum estimation)
CPU: 2200 CPU,CPU: 2200 CPU, SE: 1PB (disk), 3.5 PB (HSM)SE: 1PB (disk), 3.5 PB (HSM) This system will be maintained by CRC and NetOne corporation.This system will be maintained by CRC and NetOne corporation.
3/27/2007 Grid Efforts in Belle 22
Computing Servers
●DELL Power Edge 1855Xeon 3.6GHz x2Memory 1GB
●Made in Taiwan [Quanta]●WG: 80 servers (for login)Linux (RHEL)
●CS: 1128 serversLinux (CentOS)
●total: 45662 SPEC CINT2000 Rate.equivalent to 8.7THz
CPU will be increased by x2.5 (i.e. to 110000 SPEC CINT2000 Rate) in 2009.
1 enclosure = 10 nodes / 7U space1 rack = 50 nodes
3/27/2007 Grid Efforts in Belle 23
Storage System (Disk)●Total 1PBwith 42 file servers(1.5PB in 2009)
●SATAII 500GB diskx ~2000(~1.8 failure/day ?)
●3 types of RAID(to avoid problems)
●HSM = 370 TBnon-HSM = 630 TB
ADTX ArrayMasStor LP15drive/3U/7.5TB
Nexan SATA Beast42drive/4U/21TB
SystemWorksMASTER RAID B123016drive/3U/8TB(made in Taiwan)
3/27/2007 Grid Efforts in Belle 24
Storage System (Tape)
●Backup●90TB + 12drv + 3srv●LTO3 400GB/volume●NetVault
●HSM: PetaSite (SONY)●3.5PB + 60drv + 13srv●SAIT 500GB/volume ●30MB/s drive●Petaserve