8 mattwoodaws-intro-pdf-110411093115-phpapp01
DESCRIPTION
Matt Wood of AWS "Cloud Research"Europe April 2011 @ the Eagle Genomics SymposiumTRANSCRIPT
![Page 1: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/1.jpg)
Cloud Research
Matt WoodT E C H N O L O G Y E V A N G E L I S T
![Page 2: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/2.jpg)
Hello.
![Page 3: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/3.jpg)
Text
![Page 4: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/4.jpg)
Thank you.
![Page 5: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/5.jpg)
The Cloud by Example
![Page 6: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/6.jpg)
The Cloud by Example
![Page 7: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/7.jpg)
Infrastructureservices
![Page 8: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/8.jpg)
?
![Page 9: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/9.jpg)
On demand
![Page 10: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/10.jpg)
Pay as you go
![Page 11: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/11.jpg)
Pay for what you use
![Page 12: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/12.jpg)
Elastic capacity
![Page 13: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/13.jpg)
Capacity
Time
Estimated demand
![Page 14: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/14.jpg)
Capacity
Time
Estimated demand
Investment
Infrastructure
![Page 15: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/15.jpg)
Capacity
Time
Realdemand
Infrastructure
![Page 16: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/16.jpg)
Capacity
Time
Realdemand
Elasticcapacity
![Page 17: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/17.jpg)
Agility
![Page 18: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/18.jpg)
Faster to prototype
![Page 19: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/19.jpg)
Faster to production
![Page 20: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/20.jpg)
Undifferentiated heavy lifting
![Page 21: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/21.jpg)
Tools for accelerating
research
![Page 22: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/22.jpg)
0
75
150
225
300
Q4 2006Q4 2007
Q4 2008Q4 2009
Q4 2010
![Page 23: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/23.jpg)
The Cloud by Example
![Page 24: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/24.jpg)
![Page 25: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/25.jpg)
Data management
![Page 26: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/26.jpg)
Biomarker Warehousepre-clinical, clinical, 3rd party data and publications
!"#$%"&&'
!#%&$(%&&&'
!)*(%"&&'
+,'-./01'
23,3415'61789:1'
;<./5'=>?6@'
6178170' 6A.7341' B817-135'
Estimated cost: 10 TB warehouse over 3 years
![Page 27: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/27.jpg)
Data processing
![Page 28: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/28.jpg)
![Page 30: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/30.jpg)
http://web.mit.edu/stardev/cluster/
![Page 31: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/31.jpg)
http://cyclecomputing.comhttp://wiki.github.com/documentcloud/cloud-crowd
sudo gem install cloud-crowd
![Page 33: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/33.jpg)
Input S3 bucket
Output S3 bucket
Amazon S3
Hadoop
Amazon EC2 Instances
Input dataset
outputresults
Deploy Application
Web Console, Command line tools
End
Notify
Get ResultsInput Data
Amazon Elastic MapReduce
Hadoop Hadoop
Hadoop
Hadoop
Hadoop
Elastic MapReduce
Elastic MapReduce
![Page 34: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/34.jpg)
Preprocessed reads
Map: Bowtie
Sort: Bin and partition
Reduce: SoapSNP
Crossbow: Rapid whole genome SNP analysis
Langmead B, Schatz MC, Lin, J, Pop M, Salzberg SL. Genome Biol 10(11): R134.
![Page 35: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/35.jpg)
CloudBurst
Catalog k-mers Collect seeds End-to-end alignment
http://cloudburst-bio.sourceforge.net; Bioinformatics 2009 25: 1363-1369
![Page 36: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/36.jpg)
ASSEMBLING GENOMES
140 million 454 reads
Image: Ma) Wood
![Page 37: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/37.jpg)
Map 100 million, 100 base paired end readsQuad core with 5 GB of RAM would take 16 days
30 high-memory instances; 32 hours; $195
BLAT @ U. PENN
![Page 38: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/38.jpg)
HEAVY-ION COLLISIONS @ RHIC
Problem: Quark physics conference imminent but no compute resources handy
Solution: NIMBUS context broker allowed researchers to provision 300 nodes and get the simulations done
![Page 39: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/39.jpg)
Collaboration
![Page 40: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/40.jpg)
http://aws.amazon.com/publicdatasets/
![Page 41: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/41.jpg)
![Page 44: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/44.jpg)
![Page 45: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/45.jpg)
Applications and platforms
![Page 48: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/48.jpg)
Security
![Page 49: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/49.jpg)
Shared responsibility
![Page 50: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/50.jpg)
Requirement based access
![Page 51: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/51.jpg)
Certification
![Page 52: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/52.jpg)
ISO 27001+
SAS 70 Type II
![Page 53: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/53.jpg)
PCI DSSLevel 1
![Page 54: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/54.jpg)
Security organisation Employee lifecycle
Logical security Secure data handling
Physical security Environmental safeguards
Change management Incident handling
Data integrity Availability and redundancy
Control objectives
![Page 55: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/55.jpg)
Data access control
![Page 56: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/56.jpg)
Identity and access
![Page 57: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/57.jpg)
Independent buildings Separate flood zonesGeographically
separated
Redundantpower
Redundant connectivity Highly monitored
![Page 58: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/58.jpg)
Default deny firewall
![Page 59: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/59.jpg)
Security groups
![Page 60: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/60.jpg)
DDOSMan in the Middle
IP spoofing
![Page 61: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/61.jpg)
Resource isolationVirtual Private Cloud
![Page 62: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/62.jpg)
Amazon Web Services infrastructure
Secure VPN connection over the internet
VPN Gateway Router
Customer’s isolated AWS resources
Subnet 1 Subnet 2
Subnet 4Subnet 3
Customer’s network
![Page 63: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/63.jpg)
Dedicated instancesVirtual Private Cloud
![Page 64: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/64.jpg)
aws.amazon.com/security
![Page 65: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/65.jpg)
Data stays local
![Page 66: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/66.jpg)
aws.amazon.com
![Page 67: 8 mattwoodaws-intro-pdf-110411093115-phpapp01](https://reader033.vdocuments.net/reader033/viewer/2022060119/558c8a06d8b42a0c3d8b46c3/html5/thumbnails/67.jpg)
Thank you!