chapter 10 cloud-enabling dust storm forecasting

Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.

Chapter 10 Cloud-Enabling Dust Storm Forecasting

Qunying Huang, Jizhe Xia, Manzhu Yu, Karl Benedict and Myra Bambacus


Learning Objectives

1. General computing challenges for computing intensive applications

2. How cloud computing can help address those issues

3. Configure HPC cluster on the cloud4. Deploy dust storm model onto the cloud5. Run the dust storm on the cloud6. Performance and cost analysis


Learning Materials

•Videos: o Chapter_10_Video.mp4

•Scripts, Files and others:o mirror.tar.gz

3


Learning Modules1. Dust storm modeling and challenges

2. Dust storm model cloud deployment• General steps• Special considerations

3. Use case: Arizona Phoenix 2011 July 05• Nested modeling

• Cloud performance and efficiency analysis

4. Conclusion and discussions


Dust Storm Hazards

Desertification

Illness & Diseases

Traffic & Car accidences

Air Pollution

Ecological System

Global/regional Climate

Phoenix Dust Storm a "100-Year Event“, 2011, July 5th


Dust storm models

• Eta-4bin (Kallos et al., 1997; Nickovic et al., 1997)• Low resolution (30 KM)• Large area

• Eta-8bin (Nickovic et al., 1997; Nickovic et al., 2011)• 8 categories of dust particles• Low resolution (30 KM)• Large area

• NMM-dust (Janjic et al., 2001; Janjic, 2003)• High resolution (3 KM)• Small area


Dust storm modeling • Dust storm modeling (Purohit et al., 1999)

• Divide the domain into three-dimensional grid cells• Solve a series of numerical equations on each cell• Numerical calculations are repeated on each cell


Dust Storm Forecasting Challenges

Computing Intensity

Big Data

Large Geographic Scope

Time Critical Task

Dust Storm Forecasting

O(n^4)

Input/output

Finish One-day forecasting in 2 hours

Southwest U.S.


Learning Modules

1. Dust storm modeling and challenges2. Dust storm model cloud deployment

• General steps• Special considerations

3. Use case: Arizona Phoenix 2011 July 05• Nested modeling• Cloud performance and efficiency analysis



Dust Storm Model Deployment onto the Cloud

2. Launch one cluster instance as the head node

3. SSH to the instance

1. Authorize network access

10. Export the NFS directory to the computing node

9. Deploy the model on the NFS exporting directory

8. Mount the volume to the NFS exporting directory

4. Install the software dependency andmiddleware, e.g., NFS, and MPICH2

5. Create a new AMI from the head nodeand start an instance from the new AMI

11. Configure and test the model

12. Create two new AMIs from the running instances

6. Configure the middleware on both nodes to enable communication 7. Create an EBS volume

Video: Chapter_10_Video.mp4


Step 1. Authorize network Access


• Configure firewall rule for the firewall group “hpc” • Open port 22 for SSH• Open port 9000 -10000 for MPICH2

Video: Chapter_10_Video.mp40:00 – 1:35


Step 2. Lunch one cluster instance as the head node


• Use Amazon cluster compute AMI• Create a ssh key pair “hpc”, and save the public key pair file to local

storage as “hpc.pem”• Use the security group “hpc”

Step 3. SSH to the instance

• Use the public ssh key file “hpc.pem”• Change the user permission for “hpc.pem” as 600 (user root read permission

only) Video: Chapter_10_Video.mp4

1:35 – 4:10


Step 4. Install software dependencies

[root@domU-head~] yum install gcc gcc-c++ autoconf automake[root@domU-head~] yum –y install nfs-utilsnfs-utils-lib sytem-config-nfs #install NFS[root@domU-head~] wget http://www.mcs.anl.gov/research/projects/mpich2staging/goodell/downloads/tarballs/1.5/mpich2-1.5.tar.gz # download MPICH2 package[root@domU-head~] tar –zvxf mpich2-1.5.tar.gz # Unzip[root@domU-head~] mkdir /home/clouduser/mpich2-install # create an installation directory[root@domU-head~] cd mpich2-1.5 [root@domU-head~] ./configure -prefix=/home/clouduser/mpich2-install --enable-g=all --enable-fc --enable-shared --enable-sharedlibs=gcc --enable-debuginfo [root@domU-head~] make # Build MPICH2[root@domU-head~] make install # Install MPICH2

Install NFS and MPICH2:


http://www.mcs.anl.gov/research/projects/mpich2staging/

http://www.mcs.anl.gov/research/projects/mpich2staging/


Step 5. Create a new AMI from the head node


• Create a computing node from the new AMI

[root@domU-head ~] mkidr /headMnt # create a NFS export directory[root@domU-head ~] echo "/headMnt *rw " >> /etc/exports [root@domU-head ~] exportfs -ra[root@domU-head ~] service nfs start #start up NFS

Configure and start NFS:

Step 4. Install software dependencies



Step 6. Configure the head node and computing node Keyless access from the head node to the computing nodes

[root@domU-head ~] vi /etc/hosts #access to the hosts list of the head node[root@domU-head ~] ssh-keygen -t rsa #create a public key at the head node

[root@domU-computing~] mkdir –p /root/.ssh/[root@domU-computing ~] scp root@domU-head: /root/.ssh/id_ras.pub /root/.ssh/ #copy the public key from the head node to the computing node[root@domU-computing ~] cat /root/.ssh/id_ras.pub >> /root/.ssh/authorized_keys


Video: Chapter_10_Video.mp413:02- 17:55


Step 7. Create an EBS volume


• Attach to the head node

Step 8. Mount the volume to the NFS exporting directory

• Make a file system for the EBS volume• Mount the volume to head node directory /headMnt



Step 9. Deploy the model


• Download the model under NFS directory (/headMnt)• Export the NFS directory (/headMnt) to the computing node

[root@domU-computing ~] mkdir /computingMnt # create the directory [root@domU-computing ~] mount -t nfs -o rw domU-head:/headMnt /headMnt #Mount the volume to the NFS export directory

Step 10. Export the NFS directory to the computing node



Step 11. Configure and Test the model


• cd /headMnt/mirror/performancetest/scripts• ./run_test.sh ec2 >& ec2.log &

Step 12. Create two new AMIs from the running instances




2. Dust storm model cloud deployment

• General steps

• Special considerations





Special considerations

1. Configuring a virtual cluster environment

• Create placement group

2. Loosely coupled nested modeling and cloud

computing

3. Auto-scaling

• write scripts with the EC2 APIs


Create placement group

12




• General steps






Nested model

Tightly Nesting

AOI: NMM-dust(3KM)

AOI: NMM-dust(3KM)

ETA-8bin (30KM)

Loosely Nesting

Nested Subdomain #2(3KM)

Nested Subdomain #3(3KM)

Domain #1(30KM)

A model run with multiple resolutionsModifications of models (Michalakes et al.,

1998)Knowledge of placement for high-resolution

nested subdomains

ETA-8bin identifies AOIs(Area of Interesting)<Low resolution (30 KM) , Large area>NMM-dust performs forecasting over AOIs<High resolution (3 KM ), Small area>

Nested model: provide high resolution results for one or several area of interests over a large area.


Loosely Nested Model

Low-resolution model domain area and sub-regions (Area Of Interests, AOIs) identified for high-resolution model execution

Low-resolution model Results

Figure a. 18 AOIs Distribution


Learning Modules

1. Dust storm modeling and challenges


• General steps






Use 2 hours for one-day forecasting over Southwest of U.S. (37 X 20 degree)

18 Subregions run on 18 Amazon EC2 virtual cluster

Run Under Cloud >> Performance analysis


Run Under Cloud >> Cost analysis cont

The yearly cost of a local cluster is around 12.7 times higher than that of the EC2 cloud service if 28 EC2 instances (with 400 CPU cores) are leveraged to handle the high resolution and concurrent computing requirements for a duration of 48 hours.


Run Under Cloud >> Cost analysisItems Local cluster Amazon EC2 Options for cloud environment

Procure cluster ~4 weeks None N/A

Configure cluster operating

system (OS)~1 weeks

None

Use a public AMI with OS installed

~1 week Harden image from scratch

Configure dust storm model ~1 days~2 hours

Use a public AMI with most

required software dependencies

installed

~1 days Harden image from scratch

Start cluster 120s 45s

N/AStop cluster 60s 57s

Resume cluster N/A 45s

Total time needed for the first

time deployment~ 5 weeks

~2 Hours Use a public AMI

~ 1 week Harden image from scratch


Large-scale forecasting is Computable (2 hours) Loosely coupled nested model Cloud computing

Being capable of provisioning a large amount of

computing power in a few minutes Economically sustaining low access rates and low

resolution models

Conclusion


Discussion questions1. What are computing challenges for dust storm forecasting?2. What are the general steps to deploy dust storm model on the cloud?3. Which instance type is better for dust storm forecasting, regular instance or HPC

instance? Why?4. How to configure a virtual high performance computing (HPC) cluster to support

computing-intensive applications?5. How Elastic Block Storage (ebs) service is used in supporting the dust storm model

deployment to the cloud?6. How to create a placement group for HPC instances using both Amazon web console

management and command line tools? Why we need this step?7. Compared to Chapter 5 deployment for general applications onto the cloud, what are

the special considerations for dust storm model?8. Why can cloud computing achieve cost-efficiency?9. Why cloud computing provides a good solution to support disruptive event (e.g., dust

storm) simulation?10. What are the remaining issues while using cloud computing to support dust storm

simulation?


1. Huang Q., Yang C., Benedict K., Chen, S., Rezgui A., Xie J., 2013. Enabling Dust storm Forecasting Using Cloud Computing, International Journal of Digital Earth. DOI:10.1080/17538947.2012.749949.

2. Huang Q., Yang C., Benedict K., Rezgui A., Xie J., Xia J., Chen, S., 2012. Using Adaptively Coupled Models and High-performance Computing for Enabling the Computability of Dust Storm Forecasting, International Journal of Geographic Information Science. DOI:10.1080/13658816.2012.715650.

3. Xie J., Yang C., Zhou B., Huang Q., 2010. High Performance Computing for the Simulation of Dust Storms. Computers, Environment, and Urban Systems, 34(4): 278-290

Reference

chapter 10 cloud-enabling dust storm forecasting

Documents

dust storm modelseta

spatial cloud computing

clouddeploy dust storm

practical approach

crc press

day forecasting

kmlarge area nmmdust

manzhu yu