chapter 5 cloud-enabling geospatial applications kai liu, qunying huang, and jizhe xia
DESCRIPTION
Chapter 5 Cloud-Enabling Geospatial Applications Kai Liu, Qunying Huang, and Jizhe Xia. Learning Objectives. Summarize the common components for geospatial applications Summarize the general steps to deploy cloud-enabled - PowerPoint PPT PresentationTRANSCRIPT
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Chapter 5 Cloud-Enabling Geospatial Applications
Kai Liu, Qunying Huang, and Jizhe Xia
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Objectives
•Summarize the common components for geospatial applications•Summarize the general steps to deploy cloud-enabled geospatial applications, and study the detailed steps using two use cases
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Materials
•Videos:o Chapter_5-Video_1.movo Chapter_5_Video_2.mp4
•Scripts, Files and others:o hpc.zip
3
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Modules
1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC
2. General Steps to Deploy Cloud-Enable Geospatial Applications
3. Use Cases• Database-driven web applications• Typical HPC Applications
4. Conclusion and discussions
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Common Components for Geospatial Applications:Server-side scripting
• Server-side scripting
• Executed on the server
• Popular Server-side scripting languages
• ASP
• PHP
• JSP
• Perl and Ruby
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
• Spatial database
Database with spatial plug-ins or data types to enable spatial data
analysis, e.g.
• PostGIS plug-in of PostgreSQL
• Oracle Spatial
• Spatial engine of Microsoft SQL server
Common Components for Geospatial Applications:Spatial database
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
• HPC
• High Performance Computing (HPC)
• Parallel Computing is a popular way for HPC. There are two task
decomposition methods:
• Domain decomposition
• Functional decomposition
• HPC open source solutions, e.g., • Condor
• MPICH2
• Hadoop Mapreduce
Common Components for Geospatial Applications:HPC
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Modules
1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC
2. General Steps to Deploy Cloud-Enable Geospatial Applications
3. Use Cases• Database-driven web applications• Typical HPC Applications
4. Conclusion and discussions
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
General Steps to Deploy Cloud-enabled Geospatial Applications
• Set up environments• Software and Libraries (e.g., HTTP server, database server and JRE)• Environment variables (e.g., JAVA_PATH)
• Deploy application• Customization of the application (e.g., database, storage services,
email services and log services)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Modules
1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC
2. General Steps to Deploy Cloud-Enable Geospatial Applications
3. Use Cases• Database-driven web applications• Typical HPC Applications
4. Conclusion and discussions
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Database-driven web application
The procedure of deploying Drupal site onto EC2
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 1, 2 & 3
• Port 22 for SSH and Port 80 for HTTP should be opened for login and
HTTP access
• Drupal is supported by most Linux versions but Ubuntu is highly
recommended by the Drupal community
• Linux and Mac users could use the SSH command in a terminal to
login to the instance, while Windows users could use Putty
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov0:00-4:56
Play Video(1-3)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 4Set up Environments
• Install and configure the Apache HTTP server and the MySQL DBMS. The following commands can be used in the console:
$: sudo apt-get update$: sudo apt-get install apache2$: sudo apt-get install mysql-server mysql-client$: sudo tasksel install lamp-server
$: sudo mysql_secure_installation
• Secure MySQL Installation
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov4:56-10:20
Play Video(4)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
• Enable Apache Rewrite Module
$: sudo a2enmod rewrite
$: sudo mysqladmin -u root -p create ec2drupal$: mysql -u root -pmysql> create database drupal;mysql> grant all privileges on drupal.* to ec2drupal@localhost identified by
'your_password';mysql> flush privileges;mysql> \q
• Create a MySQL Account
Step 4 Set up Environments(cont’d)
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 5 Transfer files onto instance
• wget is supported by most Linux systems to download files from the web.
$: wget http://ftp.drupal.org/files/projects/drupal-7.19.zip
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov10:20-11:06
Play Video(5)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 6 Deploy the application
• Install unzip utility and extract the Drupal files• Deploy the application
$: sudo apt-get install unzip$: sudo unzip drupal-7.19.zip$: sudo mv drupal-7.19 /var/www/drupal$: sudo chown www-data:www-data /var/www/drupal –R$: sudo service apache2 restart
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov11:06-12:23
Play Video(6)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 7Customization
Customize Drupal’s database and administration management through its interface: http://INSTANCEIP/Drupal/install.php
INSTANCEIP should be replaced with the targeted IP address.
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov12:23-20:40
Play Video(7)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 8Create a new AMI
• The last step is to create a new AMI based on the running instance. Chapter 4, Section 4.3.1 describes the creation of an AMI. The image can be reused to create multiple virtual instances.
2. Launch an instance with Ubuntu 12.04
3. Login to the instance
4. Set up environments, e.g., Apache HTTP, DBMSs.
7. Customization of the application
8. Create a new AMI from the running instance
5.Transfer Drupal files onto instance
6. Deploy Drupal site
1. Authorize network access
Video: Chapter_5-Video_1.mov20:40-end
Play Video(8)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Modules
1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC
2. General Steps to Deploy Cloud-Enable Geospatial Applications
3. Use Cases• Database-driven web applications• Typical HPC Applications
4. Conclusion and discussions
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
The process of configuring an HPC system to run DEM interpolation on EC2 (Blue color indicates the additional steps for
configuring a virtual HPC environment)
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
2. Launch a cluster instance as the head node
1. Authorize network access
6. Configure the middleware on both nodes to enable communication
5. Start another instance from the new AMI as a computing node
3.Install the middleware packages, e.g. Condor
4. Create a new AMI from the running instance
7. Transfer the DEM data and interpolation code to the head node
8. Run the DEM interpolation
Video: Chapter_5-Video_2.mp4
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
• Port 22 for SSH and the ports for the communication managed by middleware between master node and computing nodes (e.g., 9000-9999 in this case) should be opened.
• EC2 provides cluster instances for running HPC applications. Users can also select High-CPU Instances or High Memory Cluster Instances depending on whether the geospatial application is data- or computing-intensive
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
Step 1 : Authorize network access
Video: Chapter_5-Video_2.mp40:00 – 1: 10
Play Video(1)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Select instance type based on the CPU, memory, and networking
requirements of the geospatial HPC application
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
Step 2 : Lunch an instance
Step 3. Install the middleware packagesIn this example, Condor is used as the middleware solution
$ rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm ## install additional packages
$ yum install yum-plugin-priorities $ rpm -Uvh http://repo.grid.iu.edu/osg-el6-release-latest.rpm
$ yum install condor$ touch /etc/condor/condor_config.local ## Create Condor configuration file
Video: Chapter_5-Video_2.mp41:10 – 8:46
Play Video(2-3)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Add the following content to the configuration file
“/etc/condor/config.d/local.conf”## OSG cluster configuration# List of daemons on the node (Condor central manager requires collector and
negotiator, # schedd required to submit jobs, startd to run jobs)DAEMON_LIST = MASTER, COLLECTOR, NEGOTIATOR
Start the condor service$ service condor start ## Start the service$ service condor stop ## Stop the service
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Condor Commands
Command Description Usage Example
condor_submit Submit a jobcondor_submit [submit file] $ condor_submit submit.file
condor_qShow status of jobs
condor_q [cluster]
$ condor_q 1170
condor_rmRemove jobs from the queue
condor_rm [cluster]
$ condor_rm 1170
condor_statusShow the status of resources
condor_status $ condor_status -all
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Condor needs Java Development Kit (JDK)
[root@domU-12-31-39-13-DD-FF ~]# chmod a+x jdk-6u14-linux-x64.bin [root@domU-12-31-39-13-DD-FF ~]# ./jdk-6u14-linux-x64.bin
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
This step is to store the state of configuring the HPC environment in case the master node crashes. In addition, new computing node instances can be launched directly from this AMI, and all software dependencies will already have been installed and configured.
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
Step 4. Create a new AMI from the running instance
This step is to launch other instances from the new AMI as a computing node as indicated in step 2 (except with a different AMI)
Step 5. Start another instance
Video: Chapter_5-Video_2.mp48:46 – 10:56 Play Video(4-5)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 6. Configure the middleware on all nodes to enable communication
On the computing node, the configuration file
“/etc/condor/config.d/local.conf” should be changed to:DAEMON_LIST = MASTER, SCHEDD, STARTD
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
Step 7. Transfer the DEM data and interpolation code Use scp command Video: Chapter_5-Video_2.mp4
10:56 – 18:00Play Video(6-7)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 8. Run the DEM interpolationAn example of the submission file:Universe = javaExecutable = interpolate.classArguments = interpolate DEMfile.txt ## interpolate is the java main program, and DEMfile.txt is the inputinitialdir = dir.$(Process) ## input directoryoutput =../interpolate.output.$(Process) ## output fileerror = interpolate.error.$(Process)log = ../interpolate.logrequirements = (Memory > 1024) # Select machine with memory size bigger than 1024Mbtransfer_input_files = MyPoint.class, PngWriter.class,interpolate.class, cutfile.txtshould_transfer_files = ALWAYSwhen_to_transfer_output = ON_EXITqueue 12 ## concurrent process numbers
MyPoint.class, PngWriter.class, and interpolate.class are the Java
programs, and DEMfile.txt is the input for each task.
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Step 8. Run the DEM interpolation
HPC Application Deployment onto the Cloud >> Use DEM interpolation as an example
The command “condor_submit” can be used to submit the tasks to the
cluster using a submission file named “interpolate_submit” as follows:
[root@domU-12-31-39-13-DD-FF ~]# su condor # use the condor account[root@domU-12-31-39-13-DD-FF ~]# condor_submit interpolate_submit
Video: Chapter_5-Video_2.mp4
18:00 – end Play Video(8)
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Learning Modules
1. Common Components for Geospatial Applications• Server-side scripting• Database• HPC
2. General Steps to Deploy Cloud-Enable Geospatial Applications
3. Use Cases• Database-driven web applications• Typical HPC Applications
4. Conclusion and discussions
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
Conclusions and Discussions
1. How to select the optimal cloud services and configurations? 2. Enumerate some other common components for geospatial applications
besides server-side scripting, database and HPC.3. What are the general steps to deploy a geospatial application onto cloud
services?4. What is the database service provided by Amazon AWS? How to use it?5. What is the database service provided by Windows Azure? How to use it? 6. Can you list some other geospatial applications?
K. Liu, Q. Huang, J. Xia, 2013. Chapter 5 Cloud-enabling geoscience applications, 2013. In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 73-89.
References
• Clarke, K. C. 2003. Geocomputation’s future at the extremes: High performance computing and nanoclients. Parallel Computing29, no. 10: 1281–1295.
• Huang, Q. and C. Yang. 2011. Optimizing grid computing configuration and scheduling for geoscience analysis—An example with interpolating DEM. Computers & Geosciences37, no. 2:165–176.
• Mower, J. E. 1996. Developing parallel procedures for line simplification. International Journal of Geographical Information Systems10, no. 6: 699–712.
• Xie, J., C. Yang, B. Zhou and Q. Huang. 2010. High performance computing for the simulation of dust storms. Computers, Environment and Urban Systems34, no. 4: 278–290.