introduction to cloud computing and big data
TRANSCRIPT
Introduc)on to Cloud Compu)ng and Big Data
Waheed Iqbal, Ph.D h<p://www.waheediqbal.info
12th May, 2015
Google Trends: Cloud Compu)ng, Big Data, and Distributed Systems
Cloud Compu)ng Defini)on
Cloud compu)ng is a model to provide scalable resources (network, storage, applica)ons, services, compu)ng power etc.) over the Internet with minimal management efforts.
Cloud Compu)ng Defini)on (Cont.) Na)onal Ins)tute of Standards and Technology (NIST) has published 16th draS of Cloud Compu)ng defini)on. Cloud Compu)ng model is composed of following five essen)al characteris)cs: 1. On-‐demand self service (get resources/services without human
interven)on) 2. Broad network access (accessible using mobile, laptop, tablets, and
worksta)ons) 3. Resource pooling (different physical and virtual resources dynamically
assigned and reassigned according to consumer demand)
4. Rapid elas)city (shrink and grow capabili)es) 5. Measured services (resource usage monitor, control, and report
transparently)
Cloud Compu)ng Ecosystem De
ploymen
t Mod
els
Service Mod
els
Cloud Ecosystem (Cont.)
Source: h<p://blog.ascens-‐ist.eu/2011/03/dreaming-‐of-‐fluffy-‐clouds/
Dynamic Provisioning
Lets discuss more about the most important characteris)c (Rapid Elas)city/Dynamic Provisioning) of Cloud Compu)ng!
Dynamic Provisioning (Cont.)
• In tradi)onal compu)ng model, two common problems : 1. Underes)mate system u)liza)on which result in under provision
Resources
Demand
Capacity
1 2 3
Resources
Demand
Capacity
1 2 3
Resources
Demand
Capacity
Time (days) 1 2 3
Loss Users
Loss Revenue
Dynamic Provisioning (Cont.)
2. Overes)mate system u)liza)on which result in low u)liza)on #
• How to solve this problem ?? – Dynamically provision resources
Unused resources
Demand
Capacity
Time
Resources
Dynamic Provisioning (Cont.)
• Cloud resources should be provisioned dynamically – Meet seasonal demand varia)ons – Meet demand varia)ons between different industries – Meet burst demand for some extraordinary events
Demand
Capacity
Time
Resources
Demand Capacity
Time
Resources
Mul)-‐)er Web Applica)on
Lets discuss a case using dynamic provisioning in mul)-‐)er web applica)ons!
Mul)-‐)er Web Applica)on (Cont.)
• Single-‐)er web applica)on: consists only web server mostly to serve sta)c pages and dynamic pages without database interac)on
• Mul)-‐)er web applica)on: consists on Web server, DB server, Applica)on server, Batch job processors etc
• A single )er resource management is easy comparing to mul)-‐)er applica)on!
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server
0 100 200 300 400 500 600 700 800 900
0 20 40 60 80
Respon
se Tim
e (m
s)
Number of Users/Request
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server
0 100 200 300 400 500 600 700 800 900
0 20 40 60 80
Respon
se Tim
e (m
s)
Number of Users/Request
Mul)-‐)er Web Applica)on (Cont.)
Load balancing helps to maintain performance!
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server Load Balancer
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server
Mul)-‐)er Web Applica)on (Cont.)
Network
Web Server Database Server Load Balancer
Cloud Datacenter
Internet
Cloud Compu)ng: Take Home Message
Source: Introduc)on to Amazon Web Services by Jeff Barr, Senior Web Services Evangelist
Data Growth
• Google (as of around 2009) processes around 24 petabytes of data every day
• This is quite a lot, how much? Lets try to visualize the scale of data!
Let's imagine that a single byte is represented by a single grain of rice
1K or 1024 bytes would a bowl of rice
1 MB 1 GB 2 containers
1 TB 2048 Shipping Containers
The Model Has Changed…
The Model of Genera)ng/Consuming Data has Changed
Old Model: Few companies are genera)ng data, all others are consuming data
New Model: all of us are genera)ng data, and all of us are consuming data
Big Data Defini)on
No single standard defini)on! “Big Data is high volume, high velocity, and/or high variety informa7on assets that require new forms of processing to enable enhanced decision making, insight discovery and process op7miza7on.” (Gartner)
“Big Data is a data that is difficult to store and process using tradi7onal techniques on commodity hardware to analyse and extract knowledge.” (Waheed)
Who’s Genera)ng Big Data
Social media and networks (all of us are genera)ng data)
ScienJfic instruments (collec)ng all sorts of data)
Mobile devices (tracking all objects all the )me)
Sensor technology and networks (measuring all kinds of data)
Big Data: 3V’s (Model for Describing Big Data)
Type of Data • Rela)onal Data (Tables/Transac)on/Legacy Data)
• Unstructured Data / Text Data (Web, Applica)on/Server Logs)
• Semi-‐structured Data (XML) • Graph Data
– Social Network
• Streaming Data – You can only scan the data once
Big Data and Cloud Compu)ng Technologies
Real-‐)me Tweeter Analy)cs: An Example
Ques)ons and Discussion
Acknowledgment
• Some of the material used are copied from: – Lecture Notes on Introduc)on to Cloud Compu)ng – Introductory slides of course CS525 Large-‐Scale Data Management by Dr. Mohamed Eltabakh
– Big-‐Data Tutotrial by Marko Grobelnik – Big-‐Data Lecture Slides by Ruoming Jin's – What is cloud compu7ng by Read Maloney, Product Manger, Amazon Web Services
– Most of the images used in this presenta)on are taken from the Internet