NAOC HPC Environment and Applications

Changhua Li

Chinese Virtual Observatory (China-VO)National Astronomical Observatory of China

VO & HPC

一 . 数据与计算是 VO 资源的两大组成部分，两者缺一不可

二 . 高性能计算是进行大规模数值模拟与可视化的必备条件

高效能低成本多尺度离散模拟超级计算应用系统（ GPU 超级计算系统） (2009)

------GPU 机群在数值天体物理学研究中的应用

LAOHU CLUSTER

LAOHU SOFT.

1. 配备有专业的集群作业管理系统 platform LSF2. 支持多种 MPI(openmpi)3. CUDA 4.04. FFT, GSL,IDL,gnuplot 等许多开源工具基础库软件

5. 自主研发的 GPU 资源管理中间件，真正实现了 CPU/GPU

异构集群环境下的作业全自动管理提供 GPU 设备分配接口 GetFreeGpuDevID()

共享内存技术实现进程间通信 对 SAPPORO 开源库进行修改

LAOHU Users

55

20

1510

国台海外北大其它

LAOHU Util.

Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sep. Oct.0%

10%20%30%40%50%60%70%80%90%

100%

LAOHU Application List1. NBODY Simulations (NBODY6++, phiGPU, Galactic Nuclei,

Star Clusters)2. NBODY Simulations (Gadget2, galactic dynamics)3. Correlator4. Gravitational Microlensing5. Local spirals formation through major merger6. Dark energy survey7. TREND, the Mento carlo simulation for the extreme-high energy Extensive AirShower(EAS)8. Parallelization of Herschel Interactive Processing Environment9. The HII region and PDR modeling based on CLOUDY code10. Reconstructing primordial power spectrum and dark energy

equation of state……

LAOHU publication

文章标题 作者 时间 下载链接

Astrophysical Supercomputing with “Green” GPU Clusters in Jülich and Beijing

Rainer,peter 03/2012 http://inside.hlrs.de/pdfs/inSiDE_spring2012.pdf

Loops formed by tidal tails as fossil records of a major merger

Wang, J.; Hammer, F.; Athanassoula, E.; Puech, M.; Yang, Y.; Flores, H.

02/2012 http://adsabs.harvard.edu/abs/2012A%26A...538A.121W

Made-to-measure galaxy models - III Modelling with Milky Way observations

Long,R.J.; Mao,Shude; Shen,Juntai; Wang,Yougang

09/2012http://adsabs.harvard.edu/abs/2012arXiv1209.0145L

Made-to-measure galaxy models - II. Elliptical and lenticular galaxies

Long, R. J.; Mao, Shude

04/2012http://adsabs.harvard.edu/abs/2012MNRAS.421.2580L

A New Model for the Milky Way Bar

Wang,Yougang; Zhao,Hongsheng; Mao,Shude; Rich, R.M.

09/2012http://adsabs.harvard.edu/abs/2012arXiv1209.0963W

On the Survivability and Metamorphism of Tidally Disrupted Giant Planets: the Role of Dense Cores

Liu, Shang-Fei; Guillochon, James; Lin, Douglas N. C.; Ramirez-Ruiz, Enrico

11/2012http://adsabs.harvard.edu/abs/2012arXiv1211.1971L

Interaction of Recoiling Supermassive Black Holes with Stars in Galactic Nuclei

Li, Shuo; Liu, F. K.; Berczik, Peter; Chen, Xian; Spurzem, Rainer

03/2012 http://adsabs.harvard.edu/abs/2012ApJ...748...65L

Berczik, P., Nitadori, K., Zhong S., Spurzem, R., Hamada, T, Wang, X.W., Berentzen, I., Veles, A., Ge, W., Proceedings of the

International conference on High Performance Computing

High Performance massively parallel direct N-body simulations on large GPU clusters 

Amaro-Seoane, P., Miller, M. C., Kennedy, G. F., Monthly Notices of the Royal Astronomical Society

Tidal disruptions of separated binaries in galactic nuclei 

Just, A., Yurin, D., Makukov, M., Berczik, P., Omarov, C., Spurzem, R., Vilkoviskij, E. Y., The Astrophysical Journal

Enhanced Accretion Rates of Stars on Supermassive Black Holes by Star-Disk Interactions in Galactic Nuclei 

Taani, A., Naso, L., Wei, Y., Zhang, C., Zhao, Y., Astrophysics and Space Science

Modeling the spatial distribution of neutron stars in the Galaxy 

Olczak, C., Spurzem, R., Henning, T., Kaczmarek, T., Pfalzner, S., Harfst, S., Portegies Zwart, S., Advances in Computational

Astrophysics: Methods, Tools, and Outcome

Dynamics in Young Star Clusters: From Planets to Massive Stars 

Spurzem, R., Berczik, P., Zhong, S., Nitadori, K., Hamada, T., Berentzen, I., Veles, A., Advances in Computational Astrophysics:

Methods, Tools, and Outcome

Supermassive Black Hole Binaries in High Performance Massively Parallel Direct N-body Simulations on Large GPU Clusters 

Khan, F. M., Preto, M., Berczik, P., Berentzen, I., Just, A., Spurzem, R., The Astrophysical Journal

Mergers of Unequal-mass Galaxies: Supermassive Black Hole Binary Evolution and Structure of Merger Remnants 

Li, S., Liu, F. K., Berczik, P., Chen, X., Spurzem, R., The Astrophysical Journal

Interaction of Recoiling Supermassive Black Holes with Stars in Galactic Nuclei

..... http://silkroad.bao.ac.cn/web/index.php/research/publications

LAOHU & HPC-GRID

PORTAL(Gridway, Dirac)GLOBUS Toolkits

Internet

AstroGrid-D

LAOHU in the future

• 1. Upgrade the GPU Cards to K20

• 2. Add a Intel phi 5110 Card

• other requirement.

Correlator

1. 射电望远镜

2. 综合孔径成像

相关器理论基础

T

T jiT

ji dttvtvT

tvtv )()(2

1lim)()(

sdesIsBuski

ij

)(

)()(

1 1 2 2 1 2 1 2 1 2 1 2( )*( ) ( ) ( )n n n n n n n n n n n na b i a b i a a b b a b b a i

数据相关处理流程数据采集 数据采集数据采集

数据转换byte-short)（

数据转换byte-short)（

数据转换byte-short)（

FFT FFTFFT

11 12 13 21 22 23 31 32 33

13 23 33 12 22 32 11 21 31

数据相关 数据相关数据相关

数据保存 数据保存数据保存

4进程

1进程

5进程 6进程

数据发送

数据分块 数据分块数据分块

发

送

进

程

接

收

进

程

1进程 2进程 3进程

并行化处理 ( 一 )

• Openmpi 调度所有的进程，使得各进程能够根据 RANK 来执行不同的操作

• 采用双缓冲机制来实现采集与发送的并行 ,MPI 接收与执行的并行

• 采用 OPENMP 来增加进程内部的并行性 ,使得单进程在不同的处理阶段形成多线程的流水线结构。同时，通过 OPENMP也可以优化循环代码，使得循环操作也能够尽可能的实现并行。

并行化处理 ( 二 )

T1 T5T4T3T2

C1. .n Cn+1..2n C2n+1..3n C3n+1. .4n C4n+1. .5n

FFT1. .n

Tran1. .n

FFTn+1..2n

Trann+1..2n

ISend1. .n ISendn+1..2n

IRecvn+1..2nIRecv1. .n

Shuf(1. .n)Cor1. .n

. . .

. . .

. . .

. . .

IRecv2n+1. .3n

. . .

ISend2n+1. .3n

C5n+1. .6n

FFT n+1. . n２ ３

Tran n+1. . n２ ３

FFT3n+1. .4n

Tran3n+1. .4n

FFT4n+1. .5n

Tran4n+1. .5n

Shuf(n+1..2n)

Corn+1..2n

Shuf(2n+1..3n)

Cor2n+1. .3n

IRecv3n+1. .4n

ISend3n+1. .4n

T6时间

Rank 发送 接收 FFT 发送 接收 CMAC4(A) 793(A) 792(F) 22 40 22

1（ R） 118 31

0（ R） 117 31

4(A) 1122　 　 　 　 　3(A) 1121　 　 　 　 　2(F) 　 1088 41 22　 　1（ R） 　 　 　 53 31

0（ R） 　 　 　 54 31

2(A) 241(F) 33 83 110( R) 86 12(A) 32　 　 　 　 　1(F) 　 11 32 11　 　0( R) 　 　 　 32 12(A) 241(F) 11 28 110( R) 27 1

并行化处理 ( 三 )

THANKS!