cloud computing lec7 - courses.cs.ut.ee file25/03/2014 satish srirama 11/38. aws cloudformation •...

Basics of Cloud Computing – Lecture 7

More AWS and

Research at Mobile & Cloud LabResearch at Mobile & Cloud Lab

Satish Srirama

Outline

• More Amazon Web Services

• Cloud based Research @ Mobile & Cloud Lab

25/03/2014 2/38Satish Srirama

Cloud Providers and Services – we

already discussed• Amazon Web Services

– Amazon EC2

– Amazon S3

– Amazon EBS

– Amazon Elastic Load Balancing

– Amazon Auto Scale

– Amazon CloudWatch

• Eucalyptus• Eucalyptus

• OpenStack

• SciCloud

• Management providers– ElasticFox

– RightScale

• PaaS– Google AppEngine

– Windows Azure


MORE AWS

25/03/2014 4Satish Srirama

AWS we discuss

• AWS Management Console

• AWS Identity and Access Management

• AWS Elastic Beanstalk

• AWS CloudFormation• AWS CloudFormation

• Amazon Simple Workflow Service

• Amazon Elastic MapReduce


AWS Management Console

• Hope some of you have started using Amazon accounts

• You can manage your complete Amazon account with management console (Similar to Hybridfox)– AMI Management– AMI Management

– Instance Management

– Security Group Management

– Elastic IP Management

– Elastic Block Store

– Key Pair management etc.

• Have different panes for different services


AWS Management Console - screenshot

https://console.aws.amazon.com/


AWS Identity and Access Management

(IAM)

• How can an enterprise or group of people use a single credit card?

• Manage IAM users

– Create new users and manage them– Create new users and manage them

– Create groups

• Manage permissions

– Creating policies

• Manage credentials

– Create and assign temporary security credentials


IAM policy

• Example policy giving access to complete EC2

http://aws.amazon.com/iam/


AWS Elastic Beanstalk

• Enables to easily deploy and manage applications in the

AWS cloud

– Simply upload a bundle of the applications built using .NET, PHP

and Java technologies

• Automatically handles the deployment details of capacity • Automatically handles the deployment details of capacity

provisioning, load balancing, auto-scaling, and

application health monitoring

• Something similar to PaaS

• One retains full control over the AWS resources powering

the application

– You can access the underlying resources at any time


AWS Elastic Beanstalk

• AWS EB is built using familiar software stacks such as

the Apache HTTP Server for PHP, IIS 7.5 for .NET, and

Apache Tomcat for Java

• There is no additional charge for Elastic Beanstalk

– Only the underlying AWS resources (e.g. Amazon EC2,

Amazon S3) are charged

• Leverages AWS services such as Amazon EC2, S3, SNS,

ELB, and Auto Scaling to deliver the same highly

reliable, scalable, and cost-effective infrastructure

http://aws.amazon.com/elasticbeanstalk


AWS CloudFormation

• Provides an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion

• It is based on templates model– Templates describe the AWS resources, the associated

dependencies, and runtime parameters to run an app. dependencies, and runtime parameters to run an app.

– The templates describe stacks, which are set of software and hardware resources.

– Something similar to CloudML and RightScale server templates

• Hides several details– How the AWS services need to be provisioned

– Subtleties of how to make those dependencies work.


AWS CloudFormation

• Amazon provides several pre-built templates to start common apps as:

– WordPress (blog)

– LAMP stack

– Gollum (wiki used by GitHub)– Gollum (wiki used by GitHub)

– …

• There is no additional charge for AWS CloudFormation. You pay for AWS resources (e.g. EC2 instances, Elastic Load Balancers, etc.) http://aws.amazon.com/cloudformation/


Amazon Simple Workflow Service

• A workflow service for building scalable, resilient

applications

• Reliably coordinates all of the processing steps within

applications

– such as business processes, sophisticated data analytics – such as business processes, sophisticated data analytics

applications, or managing cloud infrastructure services

• Manages task execution dependencies, scheduling, and

concurrency

• Provides simple API calls from code written in any language

• Capable to run on EC2 instances, or any of the customer’s

machines located anywhere in the world


Amazon Simple Workflow Service

• Maintains application state

• Tracks workflow executions and logs their

progress

• Holds and dispatches tasks• Holds and dispatches tasks

• Controls which tasks each of the application

hosts will be assigned to execute

• http://aws.amazon.com/swf/


Amazon Elastic MapReduce

• Web interface and command-line tools for running Hadoop jobs on EC2

• Data stored in Amazon S3

• Monitors job and shuts machines after use• Monitors job and shuts machines after use

• Running a job

– Upload job jar & input data to S3

– Create the cluster

– Create a Job Flow as steps

– Wait for the completion and examine the results


http://aws.amazon.com/elasticmapreduce/

Other interesting AWS

• Amazon Relational Database Service

– Provides access to the capabilities of familiar

database engines

– MySQL, Oracle or Microsoft SQL Server– MySQL, Oracle or Microsoft SQL Server

• NoSQL databases

– Simple DB

– DynamoDB


CLOUD BASED RESEARCH @

MOBILE & CLOUD LAB


Scientific Computing on the Cloud

• Public clouds provide very convenient access to computing resources

– On-demand and in real-time

– As long as you can afford them– As long as you can afford them

• High performance computing (HPC) on cloud

– Virtualization and communication latencies are major hindrances [Srirama et al, SPJ 2011; Batrashev et al, HPCS 2011]

• Things have improved significantly over the years

– Research at scale


Adapting Computing Problems to

Cloud• Reducing the algorithms to cloud computing frameworks

like MapReduce [Srirama et al, FGCS 2012]

• Designed a classification on how the algorithms can be adapted to MR– Algorithm � single MapReduce job

• Monte Carlo, RSA breaking• Monte Carlo, RSA breaking

– Algorithm � n MapReduce jobs• CLARA (Clustering), Matrix Multiplication

– Each iteration in algorithm � single MapReduce job• PAM (Clustering)

– Each iteration in algorithm � n MapReduce jobs • Conjugate Gradient

• Applicable especially for Hadoop MapReduce


Issues with Hadoop MapReduce

• It is designed and suitable for:

– Data processing tasks

– Embarrassingly parallel tasks

• Has serious issues with iterative algorithms

– Long „start up“ and „clean up“ times ~17 seconds

– No way to keep important data in memory between MapReduce job executions

– At each iteration, all data is read again from HDFS and written back there at the end

– Results in a significant overhead in every iteration


Alternative Approaches

• Restructuring algorithms into non-iterative versions – CLARA instead of PAM [Jakovits & Srirama, Nordicloud 2013]

• Alternative MapReduce implementations that are designed to handle iterative algorithms designed to handle iterative algorithms – E.g. Twister [Jakovits et al, ParCo 2011], HaLoop, Spark

• Alternative distributed computing models– Bulk Synchronous Parallel model [Valiant, 1990] [Jakovits et al,

HPCS 2013]

– Building a fault-tolerant BSP framework (NEWT) [Kromonov et al, UCC 2013]


Remodeling Enterprise Applications for

the Cloud

• Remodeling workflow based applications for the cloud

– To reduce communication latencies among the components

Intuition: Reduce inter-node communication and – Intuition: Reduce inter-node communication and to increase the intra-node communication

• LP based mathematical models to find ideal deployment configuration [Paniagua et al, iiWAS 2011]

– Based on the loads and regions


[Tomi T Ahonen][Tomi T Ahonen]


Mobile Applications

• One can do interesting things on mobiles directly

– Today’s mobiles are far more capable

– Location-based services (LBSs), mobile social networking, mobile commerce, context-aware services etc.

• It is also possible to make the mobile a service provider• It is also possible to make the mobile a service provider

– Mobile web service provisioning [Srirama et al, ICIW 2006; Srirama

and Paniagua, MS 2013]

– Challenges in security, scalability, discovery and middleware are studied [Srirama, PhD 2008]

– Mobile Social Network in Proximity [Chang et al, ICSOC 2012; PMC

2013]


Mobile Cloud Applications

• Bring the cloud infrastructure to the proximity

of the mobile user

• Mobile has significant advantage by going

cloud-awarecloud-aware

– Increased data storage capacity

– Availability of unlimited processing power

– PC-like functionality for mobile applications

– Extended battery life


Mobile Cloud Binding Models

Task Delegation Code Offloading

[Flores & Srirama, JSS 2013]25/03/2014 27/38Satish Srirama

Mobile Cloud Middleware

[Srirama and Paniagua, MS 2013]

[Flores et al, MoMM 2011; Flores and Srirama, JSS 2013]

[Warren et al, IEEE PC 2014]


CroudSTag – Scenario

• CroudSTag takes the pictures/videos from the cloud and tries to recognize people

– Pictures/Videos are actually taken by the phone

– Processes the videos

– Recognizes people using facial recognition technologies– Recognizes people using facial recognition technologies

• Reports the user a list of people recognized in the pictures

• The user decides whether to add them or not to the social group

• The people selected by the user receive a message in facebook inviting them to join the social group


CroudSTag [Srirama et al, PCS 2011;

SOCA 2012]

• Cloud services used

– Media storage on

Amazon S3

– Processing videos on Facial Recognition

Process

Taking picture/video

using the camera

Selecting CloudMain Menu

Send Asynchronous

Notification and Results

Storage Services

1.

2.

3.

Facebook

Login

– Processing videos on

Elastic MapReduce

– face.com to recognize

people on facebook

– Starting social group

on facebookSend invitation to the

social group

Selecting people

Selecting Cloud

Facebook

Authentication

Start Process

4.

5.

6.

7.

8.

9.

Send next

invitation

Back to Menu


Code Offloading

• Studied extensively by community [MAUI, Cloudlets etc.]

• Is Mobile Cloud taking full advantage of Cloud Computing?– Parallelization and elasticity are not exploited

• Offloading from a different perspective• Offloading from a different perspective– “Offloading is a global learning process rather than just a

local decision process“ [Flores and Srirama, MCS 2013]

• How it can learn?– Analysis of code offloading traces which are generated by

the massive amount of devices that connect to cloud

“EMCO: Evidence-based mobile code offloading“


Evidence-based Mobile Code

Offloading

[Flores and Srirama, MCS 2013]25/03/2014 32/38Satish Srirama

Process-intensive Tasks on Cloud

• Media processing

– CroudSTag demonstrates image and video

processing

• Sensor data analysis• Sensor data analysis

– Human activity recognition [Srirama et al, NGMAST 2011]

– Context aware gaming

– MapReduce based sensor data analysis [Paniagua et al,

MobiWIS 2012]


Data Analytics on the Cloud

• Cloud scale data storage solutions

• Cloud scale data analytics

– Pig & Hive

• NoSQL• NoSQL

• Implementing graph algorithms on graph

databases

Large-scale Data Processing on the Cloud -

MTAT.08.036 (Fall 2014)


WE ALWAYS WELCOME NEW IDEAS!

email: [email protected]


This week in lab

• Advanced Google AppEngine

– You will try accessing DB


Next Week

• Summarize what we have learnt

• How to prepare for the examination


References

• Check Amazon videos and webinars at http://aws.amazon.com/resources/webinars/

• List of Publications - Satish Narayana Srirama - http://math.ut.ee/~srirama/publications.html

• [Warren et al, IEEE PC 2014] I. Warren, A. Meads, S. N. Srirama, T. Weerasinghe, C. Paniagua: Push Notification Mechanisms for Pervasive Smartphone Applications, IEEE Pervasive Computing, ISSN: 1536-1268, PC-2012-12-0133.R1 (Accepted for publication).

• [Flores and Srirama, JSS 2013] H. Flores, S. N. Srirama: Mobile Cloud Middleware, Journal of Systems and Software, ISSN: 0164-1212. Elsevier. DOI: 10.1016/j.jss.2013.09.012 (In print).

• [Chang et al, PMC 2013] C. Chang, S. N. Srirama, S. Ling: Towards an Adaptive Mediation Framework for Mobile Social Network in Proximity, Pervasive and Mobile Computing Journal, MUCS Fast track, ISSN: 1574-1192. Elsevier. DOI: 10.1016/j.pmcj.2013.02.004 (In print)

• [Kromonov et al, UCC 2013] I. Kromonov, P. Jakovits, S. N. Srirama: NEWT - A fault tolerant BSP framework on Hadoop YARN, 6th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2013), December 9-12, 2013, pp. 309-310. IEEE.

• [Jakovits and Srirama, Nordicloud 2013] P. Jakovits, S. N. Srirama: Clustering on the Cloud: Reducing CLARA to MapReduce, 2nd Nordic Symposium on Cloud Computing & Internet Technologies (NordiCloud 2013), September 02-03, 2013, pp. 64-71. ACM.

• [Jakovits et al, HPCS 2013] P. Jakovits, S. N. Srirama, I. Kromonov: Viability of the Bulk Synchronous Parallel Model for Science on Cloud, The 2013 (11th) International Conference on High Performance Computing & Simulation (HPCS 2013), July 01-05, 2013, pp. 41-48. IEEE.

• [Jakovits et al, HPCS 2013] P. Jakovits, S. N. Srirama, I. Kromonov: Viability of the Bulk Synchronous Parallel Model for Science on Cloud, The 2013 (11th) International Conference on High Performance Computing & Simulation (HPCS 2013), July 01-05, 2013, pp. 41-48. IEEE.

• [Srirama and Paniagua, MS 2013] S. N. Srirama, C. Paniagua: Mobile Web Service Provisioning and Discovery in Android Days, The 2013 IEEE International Conference on Mobile Services (MS 2013), June 27 - July 02, 2013, pp. 15-22. IEEE.

• [Flores and Srirama, MCS 2013] H. Flores, S. N. Srirama: Adaptive Code Offloading for Mobile Cloud Applications: Exploiting Fuzzy Sets and Evidence-based Learning, The Fourth ACM Workshop on Mobile Cloud Computing and Services (MCS 2013) @ The 11th International Conference on Mobile Systems, Applications and Services (MobiSys 2013), June 25-28, 2013, pp. 9-16. ACM.

• [Srirama et al, SOCA 2012] S. N. Srirama, C. Paniagua, H. Flores: Social Group Formation with Mobile Cloud Services, Service Oriented Computing and Applications Journal, ISSN: 1863-2386, 6(4):351-362, 2012. Springer. DOI: 10.1007/s11761-012-0111-5.

• [Srirama et al, FGCS 2012] S. N. Srirama, P. Jakovits, E. Vainikko: Adapting Scientific Computing Problems to Clouds using MapReduce, Future Generation Computer Systems Journal, 28(1):184-192, 2012. Elsevier press. DOI 10.1016/j.future.2011.05.025.

• [Chang et al, ICSOC 2012] C. Chang, S. N. Srirama, S. Ling: An Adaptive Mediation Framework for Mobile P2P Social Content Sharing, 10th International Conference on Service Oriented Computing (ICSOC 2012), November 12-16, 2012, pp. 374-388. Springer LNCS.

• [Paniagua et al, MobiWIS 2012] C. Paniagua, H. Flores, S. N. Srirama: Mobile Sensor Data Classification for Human Activity Recognition using MapReduce on Cloud, The 9th International Conference on Mobile Web Information Systems (MobiWIS 2012), August 27-29, 2012, v. 10 of ProcediaComputer Science, pp. 585-592. Elsevier.

• [Srirama et al, SPJ 2011] S. N. Srirama, O. Batrashev, P. Jakovits, E. Vainikko: Scalability of Parallel Scientific Applications on the Cloud, Scientific Programming Journal, Special Issue on Science-driven Cloud Computing, 19(2-3):91-105, 2011. IOS Press. DOI 10.3233/SPR-2011-0320.

25/03/2014 Satish Srirama 38/38

References - continued

• [Flores et al, MoMM 2011] H. Flores, S. N. Srirama, C. Paniagua: A Generic Middleware Framework for Handling Process Intensive Hybrid Cloud Services from Mobiles, The 9th International Conference on Advances in Mobile Computing & Multimedia (MoMM-2011), December 5-7, 2011, pp. 87-95. ACM.

• [Paniagua et al, iiWAS 2011] C. Paniagua, S. N. Srirama, H. Flores: Bakabs: Managing Load of Cloud-based Web Applications from Mobiles, The 13th International Conference on Information Integration and Web-based Applications & Services (iiWAS-2011), December 5-7, 2011, pp. 489-495. ACM.

• [Srirama et al, PCS 2011] S. N. Srirama, C. Paniagua, H. Flores: CroudSTag: Social Group Formation with Facial Recognition and Mobile Cloud Services, The 8th International Conference on Mobile Web Information Systems (MobiWIS 2011), September 19-21, 2011, v. 5 of Procedia Computer Science, pp. 633-640. Elsevier. doi: 10.1016/j.procs.2011.07.082.

• [Srirama et al, NGMAST 2011] S. N. Srirama, H. Flores, C. Paniagua: Zompopo: Mobile Calendar Prediction based on Human Activities Recognition using the Accelerometer and Cloud Services, 5th International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST 2011), September 14-16, 2011, pp. 63-69. IEEE.

• [Jakovits et al, ParCo 2011] P. Jakovits, S. N. Srirama, E. Vainikko: MapReduce for Scientific Computing - Viability for non-embarrassingly parallel algorithms, The 14th International Parallel Computing conference (ParCo 2011), August 30-September 2, 2011.

• [Batrashev et al, HPCS 2011] O. Batrashev, S. N. Srirama, E. Vainikko: Benchmarking DOUG on the Cloud, The 2011 International Conference on High • [Batrashev et al, HPCS 2011] O. Batrashev, S. N. Srirama, E. Vainikko: Benchmarking DOUG on the Cloud, The 2011 International Conference on High Performance Computing & Simulation (HPCS 2011), July 4-8, 2011, pp. 677-685. IEEE.

• [MAUI] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl: MAUI: making smartphones last longer with code offload, in Proceedings of the 8th international conference on Mobile systems, applications, and services. ACM, 2010, pp. 49–62.

• [Cloudlets] M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies: The case for vm-based cloudlets in mobile computing, Pervasive Computing, IEEE, vol. 8, no. 4, pp. 14–23, 2009.

• [Srirama, PhD 2008] S. N. Srirama: Mobile Hosts in Enterprise Service Integration, PhD thesis, RWTH Aachen University, September, 2008.

• [Srirama et al, ICIW 2006] S. N. Srirama, M. Jarke, W. Prinz: Mobile Web Service Provisioning, Proceedings of the Advanced International Conference on Telecommunications and International Conference on Internet and Web Applications and Services (AICT-ICIW 2006), February 23-25, 2006, pp. 120-125. IEEE Computer Society Press.

• [Valiant, 1990] L. G. Valiant: A bridging model for parallel computation, Commun. ACM, vol. 33, no. 8, pp. 103–111, Aug. 1990.

25/03/2014 Satish Srirama 39/38

cloud computing lec7 - courses.cs.ut.ee file25/03/2014 satish srirama 11/38. aws cloudformation •...

Documents