university of massachusetts amherst november 2 nd , 2011
DESCRIPTION
GENI DiCloud. University of Massachusetts Amherst November 2 nd , 2011. Project Summary. DiCloud enables GENI experiments to incorporate cloud computing resources Focuses on Amazon web services EC2 (VMs), EBS (block storage), S3 (object storage) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/1.jpg)
Sponsored by the National Science Foundation
University of Massachusetts AmherstNovember 2nd, 2011
GENI DiCloud
![Page 2: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/2.jpg)
Sponsored by the National Science Foundation 2
Project Summary
• DiCloud enables GENI experiments to incorporate cloud computing resources– Focuses on Amazon web services– EC2 (VMs), EBS (block storage), S3 (object storage)
• How are cloud resources different from other GENI resources?– They cost money!
• Requires active Amazon account and credit card– Require authentication with Amazon
![Page 3: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/3.jpg)
Sponsored by the National Science Foundation 3
Project Summary
• DiCloud tracks per GENI-user Amazon costs– Amazon charges can be complicated
• Pay per VM time used• Pay for data transfer in and out• Pay for IOs to block devices
– No current facilities for tracking per user costs
• Enables controlled access to cloud resources– Stand-alone: setup your own DiCloud instance to track
your own costs– GENI CF-mode: place “underneath” GENI CF, where
AM pays the bills, but gives GENI users access
![Page 4: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/4.jpg)
Sponsored by the National Science Foundation 4
Elastic Compute Cloud (EC2)
• Rent VMs from Amazon “on-demand”– http://wwww.amazon.com/ec2
![Page 5: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/5.jpg)
Sponsored by the National Science Foundation 5
Elastic Compute Cloud (EC2)
• Rent VMs from Amazon “on-demand”– Pay per hour based on VM performance– Pay for data transfer in/out
![Page 6: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/6.jpg)
Sponsored by the National Science Foundation 6
Elastic Block Store (EBS)
• Rent block storage from Amazon– Pay per GB-month (currently $0.10)– Also pay per 1 million IO requests (currently $0.10)
![Page 7: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/7.jpg)
Sponsored by the National Science Foundation 7
Simple Storage Service (S3)
• Rent object storage from Amazon– Pay per GB-month – Pay for data transfer in/out
![Page 8: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/8.jpg)
Sponsored by the National Science Foundation 8
Web Portal
![Page 9: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/9.jpg)
Sponsored by the National Science Foundation 9
Web Portal
![Page 10: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/10.jpg)
Sponsored by the National Science Foundation 10
Web Portal
![Page 11: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/11.jpg)
Sponsored by the National Science Foundation 11
Tutorial Step 5: Web Portal
![Page 12: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/12.jpg)
Sponsored by the National Science Foundation 12
Tutorial Step 5: Web Portal• Now logout and log back in as user “geni”
– Then click on “Request EC2 resources”– Use ami-013d6c44 as the AMI id
![Page 13: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/13.jpg)
Sponsored by the National Science Foundation 13
Tutorial Step 5: Web Portal
• Now go back to your user page– Click on status next to the VM you just created
![Page 14: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/14.jpg)
Sponsored by the National Science Foundation 14
Tutorial Step 5: Web Portal
• Now lets create an S3 bucket– Click on “Request new S3 buckets”
![Page 15: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/15.jpg)
Sponsored by the National Science Foundation 15
Tutorial Step 5: Web Portal
• Once back at the user page, put something in your bucket– Select the s3_test file in ~/Tutorials/DiCloud/s3_test
![Page 16: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/16.jpg)
Sponsored by the National Science Foundation 16
Tutorial Step 5: Web Portal
• Now get that same thing back from your bucket
![Page 17: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/17.jpg)
Sponsored by the National Science Foundation 17
Tutorial Step 5: Web Portal
• Now lets create an EBS volume and attach it to our VM– Click request new EBS volumes– Make it 1GB, in us-west-1, in us-west-1<availability num>
![Page 18: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/18.jpg)
Sponsored by the National Science Foundation 18
Tutorial Step 5: Web Portal
• Attach the volume to our VM– Click on attach, enter instance id, and sdg for device
![Page 19: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/19.jpg)
Sponsored by the National Science Foundation 19
Tutorial Step 5: Web Portal
• If we ssh into our VM we can see that the volume is now attached
![Page 20: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/20.jpg)
Sponsored by the National Science Foundation 20
Demo Data Flow
• Dynamic end-to-end Nowcasting – Mapping Nowcast Workflows onto GENI
Archival Storage
Radar Nodes
“raw” live data
Upstream LDM feed
archived netcdf data
Nowcast Processing
aggregatedmulti-radar data
Post to Web
Nowcast images for display
![Page 21: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/21.jpg)
Sponsored by the National Science Foundation 21
Multi-radar NetCDF Data
Nowcast Processing
1. DiCloud Archival Service (S3)2. LDM Data Feed (EC2)
“raw” live data
Generate “raw” live dataViSE/CASA radar nodes
http://stb.ece.uprm.edu/current.jsp
Use proxy to track usage-based spending on Amazon and enforce quotas and limits
http://geni.cs.umass.edu/vise/dicloud.php
1. Ingest mulit-radar data feeds2. Merge and grid multi-radar data2. Generate 1min, 5min, and 10min Nowcasts3. Send results over NLR to Umass4. Repeat
ViSE views steerable radars as shared, virtualized resources
http://geni.cs.umass.edu/vise
Nowcast images for display
![Page 22: University of Massachusetts Amherst November 2 nd , 2011](https://reader036.vdocuments.net/reader036/viewer/2022062410/56815dc0550346895dcbeb25/html5/thumbnails/22.jpg)
Sponsored by the National Science Foundation 22
Demo Overview
• Dynamic end-to-end Nowcasting on GENI – Slice of sensing, networking, computing, and storage
Archival Storage
Radar Nodes
Upstream LDM feed
Nowcast Processing
Post to Web
Generate “raw” dataViSE/CASA radars
Archive radar dataAmazon S3
“raw” live data
Archived data availableto downstream nodes
archived netcdf data
aggregatedmulti-radar data
Nowcast images for display
GenerateNowcasts