setting up small grid testbed & using globus, mpich-g2
DESCRIPTION
Computational Fluid Dynamics Lab. Setting up Small Grid Testbed & Using Globus, MPICH-G2 Korea Advanced Institute of Science and Technology Div. of Aerospace Engineering Dehee Kim 2002. 9. 25. Contents. Introduction to GT 2.0 and MPICH-G2 How to install Globus CFD Lab. Grid Testbed - PowerPoint PPT PresentationTRANSCRIPT
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Setting up Small Grid Testbed& Using Globus, MPICH-G2
Korea Advanced Institute of Science and TechnologyDiv. of Aerospace Engineering
Dehee Kim2002. 9. 25
Computational Fluid Dynamics LabComputational Fluid Dynamics Lab..
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Contents
Introduction to GT 2.0 and MPICH-G2
How to install Globus
CFD Lab. Grid Testbed
Numerical Test on Testbed
About Network Bandwidth
Concluding Remarks
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
GT 2.0
Globus Toolkit 2.0
Major improvements over the Globus Toolkit 1.1.3 and 1.1.4 releases
Data Grid Components MDS Components GRAM Components Packaging Technology Security Components Various changes for supporting MPICH-G2
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
MPICH-G2
● What is MPICH-G2?
grid-enabled implementation of the MPI v1.1 standard converts data in messages sent between machines of different architectures supports multiprotocol communication
● How does MPICH-G2 differ from MPICH-G?
Increased bandwidth Reduced latency for intra-machine messaging Increased latency for inter-machine (TCP) messaging
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Construction of Grid Testbed
Installation Procedure
1. Set up small PC cluster system - rsh, NFS, automount, ntp, … - back end nodes with hard disk
2. Install F77, F90 compiler - Absoft F90, pgf90, etc. - Set environment variables and path
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Construction of Grid Testbed
If you does not install a Fortran compiler before the installation of GT 2.0, you will see following message …..Checking for minix/config.hChecking for volatile… yesRunning device-specific setup program*#Globus device overrode C compiler setting*#F90 compiler not present; disabling F90 supportDisabling long double(not supported by Globus data Conversion library)Checking whether cross-compiling……..
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Construction of Grid Testbed
3. Install Job Queuing System - PBS, CONDOR, LSF, etc. - 2 rpm files(for PBS) - Front end : /usr/spool/pbs/server_priv/nodes
- Back end : /usr/spool/pbs/mom_priv/config $clienthost cluster.hpcnet.ne.kr
/usr/spool/pbs/default_server
4. Install GT 2.0 - Using simple CA5. Install MPICH-G2 ./configure –device=globus2 \ -fc=/opt/absoft/bin/f77 \ -f90=/opt/absoft/bin/f90 \ : -flavor=gcc32dbg \ --prefix=/usr/local/mpich-1.2.4-g2
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Construction of Grid Testbed
A trial and error
/etc/xinetd.d/globus-gatekeeper
Service globus-gatekeeper{
socket_type = stream …..}
Service globus-gatekeeper{
socket_type=stream …..}
O.K.
Parsing Error!
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Construction of Grid Testbed
6. Modify some scripts if necessary - If various environment variables related with
jobmanager were not set, set the variables in some files
$GLOBUS_LOCATION/libexec/globus_sh.tools.sh $GLOBUS_LOCATION/libexec/globus_gram-job-manager-tools.sh
…..GLOBUS_GRAM_JOB_MANAGER_MPIRUN=/usr/local/mpich-1.2.4-g2/bin/mpirunGLOBUS_GRAM_JOB_MANAGER_QDEL=/usr/local/bin/qdelGLOBUS_GRAM_JOB_MANAGER_QSTAT=/usr/local/bin/qstatGLOBUS_GRAM_JOB_MANAGER_QSUB=/usr/local/bin/qsubGLOBUS_GRAM_JOB_MANAGER_QSELECT=/usr/local/bin/qselect……
7. Configure the firewall policy for Globus
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
OOS-Linux 2.4.x, 2.2.xS-Linux 2.4.x, 2.2.x
KKAIST CFD Lab. AIST CFD Lab.
– – 1 Front-end, 4-execution nodes(1.8GHz, 512M RAM)1 Front-end, 4-execution nodes(1.8GHz, 512M RAM)
KISTI supercomputing centerKISTI supercomputing center
– – 1 Front-end, 4-execution nodes(450MHz, 256M RAM)1 Front-end, 4-execution nodes(450MHz, 256M RAM)
Globus Toolkit 2.0, MPICH-G2, ABSOFT F90Globus Toolkit 2.0, MPICH-G2, ABSOFT F90
Job Scheduler – Portable Batch SystemJob Scheduler – Portable Batch System
CFD Lab. Grid Testbed
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
CFD Lab. Grid Testbed
duy.kaist.ac.kr/jobmanager-pbs cluster.hpcnet.ne.kr/jobmanager-pbs
MPICH-G2
scheduler-pbs
scheduler-pbs
Execution nodes Execution nodes
duy.kaist.ac.kr/jobmanager-pbs cluster.hpcnet.ne.kr/jobmanager-pbs
MPICH-G2
scheduler-pbs
scheduler-pbs
Execution nodes Execution nodes
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Numerical Test on Testbed
Design Optimization : 2-D designDesign Optimization : 2-D design
2-2-D adjoint sensitivity analysisD adjoint sensitivity analysis 2-2-D airfoil designD airfoil design Design for drag minimization of RAE 2822 airfoilDesign for drag minimization of RAE 2822 airfoil Grid system Grid system : 383 : 383 x 65 C type x 65 C type Flow conditions Flow conditions : : M=0.729, AoA=2.31M=0.729, AoA=2.31oo, Re = 6.5 x 10, Re = 6.5 x 1066
10 Hicks-Henne functions10 Hicks-Henne functions
x
-Cp
0 0.25 0.5 0.75 1
-1
-0.5
0
0.5
1
1.5
RAE2822Designed
InviscidViscousRAE 2822
Pressure distributionPressure distribution airfoil before and after designairfoil before and after design
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Design Optimization : 3-D designDesign Optimization : 3-D design
3-3-D adjoint sensitivity analysisD adjoint sensitivity analysis 3-3-D wing designD wing design Design for drag minimization of ONERA M6 wingDesign for drag minimization of ONERA M6 wing Grid system Grid system : 193 : 193 x 49 x 33 C-O type x 49 x 33 C-O type Flow conditions Flow conditions : : M=0.84, AoA=3.06M=0.84, AoA=3.06oo, Re = 11.7 x 10, Re = 11.7 x 1066
50 Hicks-Henne functions50 Hicks-Henne functions
X
Y
Z
X
Y
Z
ONERA M6ONERA M6 Designed wingDesigned wing
Numerical Test on Testbed
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Computation Time
Design Optimization : Computation TimeDesign Optimization : Computation Time Flow analysis around 2-D airfoil and design optimizationFlow analysis around 2-D airfoil and design optimization
deviceresource
Flow analysis (ch_p4)
Design(ch_p4)
Flow analysis(globus2)
Design(globus2)
I 158.0 467.7 158.7 478.9
II 388.8 1166.7 392.9 1170.4
III 410.9 1432.1
Flow analysis around 3-D wing and design optimizationFlow analysis around 3-D wing and design optimization((case IIIcase III)) Flow analysis : Flow analysis : 2047.72047.7 Design : Design : 15674.0 15674.0
I : Pentium 4 1.7 GHz CPU, 4 nodes, 512M RAMI : Pentium 4 1.7 GHz CPU, 4 nodes, 512M RAMII : Pentium 2 450 MHz CPU, 4 nodes, 256M RAMII : Pentium 2 450 MHz CPU, 4 nodes, 256M RAMIII : I &III : I & IIII
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
DFVLR Axial Fan – Dr. J. S. Yoon
3-D Compressible Navier-Stokes Solver
k-ω Turbulent Modeling 3 Stage Runge-Kutta Time
Marching & Central Scheme 28 Blade(45*19*19) MPICH-G
Surface Pressure Contours
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Computation Time at various Network Bandwidth
Com. Env.
Result1CPU
Parallel(2Node)
Computation based on Globus(Remote 2 Node, Mbps)
2 5 10 15 25 45 155
Time(Sec) 3773 1998 3367 2488 2235 2147 2078 2047 2038
T/Tp 0.94 1 1.69 1.25 1.12 1.08 1.04 1.02 1.02
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Varying efficiency on time of day
Variation of computation time
• PC Cluster front ↔ IBM SP2
• 1:1 CPU, 200 iterations
• Seriously varying efficiency on time of day
• Need for proper QoS and CPU Reservation
KA
IST C
om
puta
tional Flu
id D
ynam
ics Lab
.
Concluding Remarks
Setup of small testbed
Test for design applications
Need for obtaining vast computing resources
Implementation to diskless cluster - public IP, private IP(e.g. pacx-mpi)