neon
DESCRIPTION
NEON. TNC2010, May 31, Vilnius Maarten Koopmans for UNINETT Sigma [email protected]. Who. ING Group -2002 SURFnet 2002-6 ICTU (govt) 2006-8 vrijheid.net 2008- qtask.com ibeamsystems.com Uninett Sigma. Different mindsets. NEON Goals. state-of-the art of cloud computing; - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/1.jpg)
NEON
TNC2010, May 31, VilniusMaarten Koopmans for UNINETT Sigma
![Page 2: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/2.jpg)
22
WhoWho•ING Group -2002•SURFnet 2002-6•ICTU (govt) 2006-8•vrijheid.net 2008-
•qtask.com•ibeamsystems.com•Uninett Sigma
•ING Group -2002•SURFnet 2002-6•ICTU (govt) 2006-8•vrijheid.net 2008-
•qtask.com•ibeamsystems.com•Uninett Sigma
![Page 3: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/3.jpg)
Different mindsets
![Page 4: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/4.jpg)
NEON Goals
state-of-the art of cloud computing;
cost of moving and running non-HPC jobs on a cloud computing environment;
how to do this in practice;
a list of identified risks/benefits on a short/long perspective.
![Page 5: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/5.jpg)
Areas
![Page 6: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/6.jpg)
Shortlists
![Page 7: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/7.jpg)
![Page 8: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/8.jpg)
#8: Why deliver?
Resource
Cost inMedium DC
≈ 1000 servers
Cost inVery Large DC
≈ 50,000 serversRatio
Network $95 / Mbps / month $13 / Mbps / month 7.1x
Storage $2.20 / GB / month $0.40 / GB / month 5.7x
Administration ≈140 servers/admin >1000 servers/admin 7.1x
![Page 9: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/9.jpg)
#7 Design to fail
![Page 10: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/10.jpg)
#6 USP: “root” - really?
![Page 11: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/11.jpg)
#5 Can you trust it?
“Cloud computing is about gracefully losing control while maintaining accountability even if the operational responsibility falls upon one or more third parties.”
![Page 12: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/12.jpg)
#4.5 “Core infra”
ComputingComputingComputingComputing
STORAGESTORAGE
Queues
![Page 13: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/13.jpg)
#4 Public clouds lead
• Spot instances
• Elastic load balancing
• Virtual Private Cloud
• Elastic map-reduce
• Cloudfront
• .....
• SQS
• SimpleDB
• Cloudwatch
• Autoscaling
• RDS
![Page 14: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/14.jpg)
#3: Management
![Page 15: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/15.jpg)
#2 Keep an eye on Apache
ZooKeeper
![Page 16: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/16.jpg)
#1 The USERS are key
![Page 18: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/18.jpg)
Why again?
![Page 19: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/19.jpg)
Let’s zoom in on storage
![Page 20: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/20.jpg)
Storage: requirements
• No client to access the data.
•Transparent versioning.
•Transparent encryption - both transport and storage
•AAI integration
• Allow sharing of resources
![Page 21: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/21.jpg)
Current Cloud storage
API based - complex for end users
![Page 22: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/22.jpg)
No AAI integration at all
![Page 23: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/23.jpg)
So...
![Page 24: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/24.jpg)
AAI: enrollment
AAI
Cloudbacked storage
initial request
access grantedtoken returned (rotating?)
authenticate user user authenticated
![Page 25: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/25.jpg)
“Just WebDAV”
Webdav daemon
Encryption
Resource naming
Versioning
Storage cloud
Continue
Map resource to hash code version = 1
Stream data metered through encryption ...........into the cloud
Metering
![Page 26: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/26.jpg)
...Locking...
Webdav daemon Locking
Resource naming
Versioning
Return lock
Map resource to hash code Get reference to most recent version
Acquire lock
![Page 27: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/27.jpg)
...downloading...
Webdav daemon Decryption
Resource naming
Versioning
Storage cloud
Continue
Get most recent version = 1
Stream data metered via decryption ...........from the cloud
Metering
![Page 28: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/28.jpg)
Stand on the shoulders of giants• WebDAV widely deployed, lots of 3rd party clients.
• Service on top of Java VM
• Scala (integration language)
• Cloud access libraries (often Java based)
• Apache ZooKeeper (configuration management, locking)
• Apache Cassandra or HBase (metering)
• AAI integration components
• ...
![Page 29: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/29.jpg)
Lessons learnt so far:Lessons learnt so far:• WebDAV is a nice start for client-less access to file based resources.
• CPU intensive due to the encryption per "stream". A language (model) with concurrency support is a big plus. This breaks the trend of asynchronous I/O based network services.
• Stand on the shoulders of giants: Apache Zookeeper, Bookkeeper, the JVM, Scala language, libraries for cloud access
• OS X requires DAV level 2 and does a lot of locking. But: from 10.5.x onwards it also does HTTP 1.1 chunked encoding; that broke a lot of servers.
• Windows works best with digest authentication.
• Linux seems to be most forgiving and least demanding.
• all clients support SSL.
• WebDAV's XML is relatively simple but the usage may differ per client type.
![Page 31: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/31.jpg)
Cloud management
![Page 32: NEON](https://reader035.vdocuments.net/reader035/viewer/2022062423/568148f0550346895db60d52/html5/thumbnails/32.jpg)
Computing• Torque/PBS on AWS via RightScale• OSGi on Eucalyptus• MPI• R
•Challenge: Matlab, BLAST etc. - how to deal with licensing?