february epd webinar: how do i...use picloud for cloud computing?
DESCRIPTION
In this Enthought Python Distribution Webinar, Ken Elkabany, co-founder of PiCloud, shows us how to run scientific and numeric Python code remotely on Amazon EC2. Through a partnership with Enthought, PiCloud now hosts EPD on it's cloud servers, allowing all EPD users to run their code remotely with ease. Several demonstrations are provided. For example code, visit http://enthought.com/training/webinars.php.TRANSCRIPT
How do I...use PiCloud for cloud computing?
Ken Elkabany • February 5, 2010
• Python 2.6
• 75+ libraries SciPy, NumPy, matplotlib, IPython, Mayavi...
• NumPy 1.4 faster (20-30%) import times, better handling of
special floating point types
• 64-bit available on all platforms in addition to 32-bit
• PiCloud’s cloud library for cloud computing
January 5, 2010 release
Ken ElkabanyCo-Founder & CEO, PiCloud
• UC Berkeley
• Founded PiCloud in 2008
© Copyright 2010 by PiCloud, Inc.
Who are we?
PiCloud, Inc. Founded in 2009, and based in Sunnyvale, CA Our mission: Simplify cloud computing The PiCloud Cloud-Computing Platform
Integrated into the Python programming language Partnership with Enthought, Inc. to include our cloud
library in EPD (Enthought Python Distribution) Beta-release
2
© Copyright 2010 by PiCloud, Inc.
What is cloud computing?
Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet.
3
© Copyright 2010 by PiCloud, Inc.
What does a cloud look like?
Low upfront costs Servers are not purchased, only rented (utility
computing)
Multi-tenant Centralization of infrastructure geographically High-levels of utilization and efficiency
Scalable and Agile On-demand provisioning of resources
Secure, Reliable, Sustainable
4
© Copyright 2010 by PiCloud, Inc.
Cloud Computing Ecosystem
Cloud infrastructures Allow for the provisioning of
virtualized resources Compute: Boot up and log into
a Linux or Windows server Storage: Specialized services
like S3
The PiCloud platform sits on top of infrastructure, abstracting its details away
5
© Copyright 2010 by PiCloud, Inc.
The PiCloud Abstraction
OutputInputInput Output
6
© Copyright 2010 by PiCloud, Inc.
AutoTagger Facebook photo-tagging assistant PiCloud’s First Customer
Basic Functionality Detect and extract faces from
a single photo Repeatedly apply face detection
to all photos in an album
7
Our motivation
© Copyright 2010 by PiCloud, Inc.
Functional view: I need to detect faces in these photos for the user
Server view: I need 20 high performance virtual servers to detect faces in these photos in parallel
PiCloud
Design Objectives:•Load balancing•Auto-Scaling•Performance Monitoring, introspection•Error detection, reporting, and handling.
8
Our motivation
© Copyright 2010 by PiCloud, Inc.
Roadmap
Brief tour of the website Example 1: First steps with PiCloud
Using the web and console interface
Example 2: Monte Carlo PI Example 3: Basic twitter analyzer Conclusion
9
© Copyright 2010 by PiCloud, Inc.
Example 1: First steps
Run a simple function on the cloud
>>> def func(): # define the function>>> return 3*3>>>>>> import cloud # import our library>>> cloud.setkey(API_KEY, API_SECRET_KEY) # set your key (can do in cfgs)>>> jid = cloud.call(func) # returns job id>>> cloud.status(jid) # returns job status ‘processing’>>> cloud.status(jid) # returns job status ‘done’>>> jid = cloud.result(jid) # returns the result9
10
© Copyright 2010 by PiCloud, Inc.
Example 1: First steps
3 Simple Steps Import our library Set your api key (this is how we identify you) Pass your function into cloud.call
That’s all you need to run any function on the cloud! Your function can do anything that Python allows
you to do. (except open listening sockets)
11
© Copyright 2010 by PiCloud, Inc.
If you’re trying to following along…12
But you don’t have an API Key yet, try this:import cloudcloud.start_simulator()# do not call cloud.setkey
© Copyright 2010 by PiCloud, Inc.
Web interface13
http://www.picloud.com/accounts/ Manage API Keys View functions/jobs
Status (queued, processing, done) Resource usage (profiling)
Track compute usage with analytics Python C-extensions support Documentation
© Copyright 2010 by PiCloud, Inc.
Example 1: First steps14
What if there’s an exception?
>>> def func(): # define the function>>> return ‘a’ + 3>>>>>> import cloud # import our library>>> cloud.setkey(API_KEY, API_SECRET_KEY) # set your key (can do in cfgs)>>> jid = cloud.call(func) # returns job id>>> cloud.status(jid) # returns job status ‘error’>>> cloud.result(jid) # returns job status cloud.cloud.CloudException: Job 7199: Traceback (most recent call last):File "<stdin>", line 2, in func
TypeError: cannot concatenate 'str' and 'int' objects
© Copyright 2010 by PiCloud, Inc.
Under the hood
Distributed File System,
DHT
Distributed Database
Load balancers, schedulers
Workers (compute servers)
PiCloudInterface
You
ApplicationUsing PiCloud
Cloud LibraryIdentifies and Transmits data
and source dependencies
MonitoringAnalyze workload
15
© Copyright 2010 by PiCloud, Inc.
Under the hood
How does our advanced system translate over to tangible benefits for you? Robust Scalable Secure Multi-tenant
Higher utilization -> cheaper
Optimized Easy maintenance
16
© Copyright 2010 by PiCloud, Inc.
Using the simulator
Motivation Copying the state of your Python interpreter to PiCloud
can be costly, and should be optimized to a minimum We’ll warn you automatically if you are sending over 1MB
of information
In many cases, faster to test things locally (and there are no charges!)
How do I use it?cloud.start_simulator()
17
© Copyright 2010 by PiCloud, Inc.
Example 2: Monte Carlo π
Estimate PI using the Monte Carlo method Randomly throw darts at the board below
throwndartstotalcircleinlandingdartsnumPI
__)____(*4
18
4)2()__( 2
2
rrcircleindartP
Derived from
© Copyright 2010 by PiCloud, Inc.
Example 2: Monte Carlo PI
Map a function across a series of data points
19
def monteCarlo(num_test):"""Throw num_test darts at a 1x1 rectangleReturn how many appear within the quarter circle""" numInCircle = 0y = random.random()
for _ in xrange(num_test):x = random.random()y = random.random()if x*x + y*y < 1.0:numInCircle += 1
return numInCircle
© Copyright 2010 by PiCloud, Inc.
20
Example 2: Monte Carlo PI
1 line modification to run this on the cloud
20
import randomimport cloud
numMaps = 6numTests = 10000000
def calcPi():
"""Send off numMaps monteCarlo simulations in parallel"""jids = cloud.map(monteCarlo, [numTests for _ in range(numMaps)], _high_cpu=True)numInCircleResults = cloud.result(jids)numInCircle = sum(numInCircleResults)
pi = (4 * numInCircle) / float(numTests* numMaps)return pi
if __name__ == '__main__':pi = calcPi()
print 'Pi determined to be %s' % pi
© Copyright 2010 by PiCloud, Inc.
Example 2: Monte Carlo PI
How much did this cost? Use cloud.info to get cpu time used (runtime) Cost = (cpu time) * rate Cost = 60 seconds * 0.0013 per minute = $0.0013
21
© Copyright 2010 by PiCloud, Inc.
Example 2: Monte Carlo PI22
What was the _high_cpu attribute for?
Automatically run your code with more CPU power Default is 1 CPU unit (1-1.2ghz Xeon processor) _high_cpu is 2.5 CPU units
jids = cloud.map(monteCarlo, [numTests for _ in range(numMaps)], _high_cpu=True)
© Copyright 2010 by PiCloud, Inc.
Example 3: Twitter Analyzer
Determine relationships between followers, # following, and statuses using linear regression
We use cloud.map to scrape twitter for user data Run linear regressions in parallel using
cloud.call Graph statuses/followers with pylab NOTE: To use this example, you will need to modify
the posted file to use your own Twitter account information.
23
© Copyright 2010 by PiCloud, Inc.
Conclusion
You can now leverage the compute power of a cluster of servers with only a couple lines of code!
We’ve given you a taste of the features PiCloud has to offer, but there are many more!
If you have not already, register at www.picloud.com Questions? [email protected]
24