atlas computing status in ihep erming pei, cc-ihep yangzhou, may 15’ th 2009
TRANSCRIPT
ATLAS computing status in IHEP
Erming Pei, CC-IHEP Yangzhou, May 15’th 2009
Agenda
• Farm• Grid• Issues
• File System
Farm
Resource
• Old farm– Slc3: atlas02 + 8 Cores – Slc4: autilas + 16 Cores • will integrated to new farm
• New farm– atlasui02 + 128 Cores– New Server, New release (in testing)
File System Size Used Avail % Mounted on
Storage
• HOME– AFS 8.6G 0 8.6G 0% /afs– 202.122.33.48:/home/atlas 932G 88G 845G 10% /ihepbatch/home-
atlas
• Software– bjlcg2.ihep.ac.cn:/data/exp_soft 1.1T 327G 791G 30% /ihepbatch/exp_soft– autilas.ihep.ac.cn:/opt/atlassw 29G 16G 13G 57% /opt/atlassw
• Data– 192.168.50.30:/atlas/data0 2.8T 543M 2.8T 1% /ihepbatch/atlasdata0– 192.168.50.30:/atlas/data1 3.7T 1.4T 2.3T 38% /ihepbatch/atlasdata1– 192.168.50.30:/atlas/data2 2.8T 1.3T 1.5T 47% /ihepbatch/atlasdata2– 192.168.50.30:/atlas/data3 3.1T 512K 3.1T 1% /ihepbatch/atlasdata3– 192.168.50.30:/atlas/data4 3.1T 512K 3.1T 1% /ihepbatch/atlasdata4– 192.168.50.30:/atlas/data5 3.1T 512K 3.1T 1% /ihepbatch/atlasdata5
Storage
Grid software repository
AFS
/hom
e/at
las
SE(DPM)
ATLAS Disk
Server
HOME
Local Data
ATLAS software Grid Data
atlasui02autilas
Torque/Maui
Software
• DQ2 enduser tools: /opt/atlassw/DQ2/endusers• Ganga: 5.1.10 (updated by Lianyou)
Job management• Server: Torque • Scheduler: Maui • Both are optimized
atlasui02autilas
Torque Server Maui
Job monitor
Local DPM Access
• T3 T2, DPM accessing failed – “rfio:/…”
• Reason:– Both Castor and DPM have rf* tools– use the same library: libshift.so
• Solution:– Link DPM library (libdpm.so) to Castor library
(libshift.so)
Tests with Athena 14.2.23
• Jobs: – Simulation jobs– Reconstruction jobs
• Tests:– Old farm– New farm– Front end – Back end– Interactive (directly on computing nodes)
Grid
GangaRobot
Stress tests (GangaRobot)
Panda Jobs
Grid (Tier-2)
Disk Usage
Issues
• Many job failures in testing, a few succeeded • Conclusion: – I/O issue• Standardize job submitting operations• move data from HOME space to Data disks
– Most probably something wrong with the new batch system(the latest version, torque 2.4.1) • will change to other versions and test again.
– Next step• Separate Local software environment from Grid
Issues
AFS
/hom
e/at
las
SE(DPM)
ATLAS Disk
Server
HOME
Local Data
atlasui02autilas
Torque/Maui
Local Software
Grid software repository
NFS
Comments• Standardize your operations
– Put your input data to /atlas/datax1 or from DPM. – Submit jobs from /home/atlas/xxx
• afs space not support for batch jobs currently
– Put your output data to /atlas/datax2 – Please don't mix Home and Data space. – Add some debug sentences to your script
• e.g., Add 'hostname’ to your job script so that can know which node your job was running.
• Insert intervals when submit bulk jobs • Data space
– Public/Private– Public dataset classified by dataset name rather than by user name
File System
NFS
Luster MDS Server
Disk Server
LUSTRE 压力测试(一)• 采用 600 个 BES 分析作业,运行 8 个小时,没有
出现问题,读性能稳定在 800MB/s
LUSTRE 压力测试(二)• 采用 256 个 dd 写作业,同时运行一天,没有出
现问题,性能稳定在 350MB/s
实际应用测试
测试方法
• 在集群上设置两个测试专用队列 btq1,btq2 ,每个队列 300个 CPU ;每个队列中均有 2CPU , 4CPU , 8CPU 的计算结点
• 分别在两个队列上提交, 300 个, 250 个, 200 个, 150个, 100 个, 50 个分析作业
• 队列的分析作业分别对 LUSTRE 、 GPFS 文件系统中的数据文件进行分析计算(主要是读操作和少量写操作)
• 查看作业运行期间,计算结点的运行效率网络流量,以及文件服务器的网络流量
• 计算结点的运行效率取值参考 CPU USER 利用率
测试结果- cpu 利用率8cpu CPU计算结点- 利用率
0
10
20
30
40
50
60
70
80
300-作业 250-作业 200-作业 150-作业 100-作业 50-作业
同时运行作业数
CPU利
用率
(%
)
l ust re- 8cpu
gpf s- 8cpu
★
测试结果-网络流量8CPU计算结点网络流量统计
0
10
20
30
40
50
60
300-作业 250-作业 200-作业 150-作业 100-作业 50-作业
同时运行作业数
MByte/Sec
l ust re- 8cpu
gpf s- 8cpu
★
结论• 在当前情况下, 150个分析作业同时运行效果较好-- CPU的利用率达到 60%以上。
• 推测:要满足 1500个分析作业同时高效运行,需要 30个左右文件服务器支持的并行文件系统
Questions?