drilett aws vpc_presentation_shared
TRANSCRIPT
Teaching Network Infrastructure, Database Concepts, and Web-Based
Application Development using Cloud Services: An Introduction
David RilettJohn Russo
“The most radical and transformative of inventions are those that empower others to unleash their creativity - to pursue their dreams.” Jeff Bezos, Letter to Shareholders, 2012
Forbes #12
VPC Introductory
Amazon EC2 AMI Elastic IP
Amazon Elastic Compute Cloud
Amazon Simple Storage Service
Amazon S3
Amazon SimpleDB
Amazon SimpleDB
What’s in AWS Virtual Private Cloud anyway?(Mouse over screentips for answers)
VPC IntermediateAmazon Elastic Block Store
Amazon Glacier AWS Elastic Beanstalk
Amazon ElastiCache
AWS Identity and Access Management
VPC AdvancedAmazon Route 53 Elastic Load
Balancing
Amazon Route 53 route table
AWS CloudFormation Amazon CloudWatch
Amazon DynamoDB
Amazon Relational Database Service
Amazon Simple Queue Service
VPC ExpertAmazon Virtual Private Cloud
Amazon VPC router Internet gateway
customer gateway
virtual private
gateway
VPN connection
Auto Scaling Amazon Redshift
AWS OpsWorks
Use Case: "VPC with a Single Public Subnet Only"
Host a simple, public-facing website
Host a basic web application, such as a blog or simple website Secure the website by creating security group rules which allow the webserver to respond to inbound HTTP and SSL requests from the Internet while simultaneously prohibiting the webserver from initiating outbound connections to the Internet.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario1.html
Use Case: "VPC with Public and Private Subnets" • Multi-tier web applications that strictly
enforce access and security restrictions between your webservers, application servers, and databases.
• Launch webservers in a publicly accessible subnet
• Launch application servers and databases in non-publically accessible subnets.
• Application servers and databases can’t be directly accessed from the Internet, but access the Internet via a NAT instance to download patches, for example.
• Control access between the servers and subnets using inbound and outbound packet filtering by network access control lists and security groups.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
"VPC with Public and Private Subnets" Extended for Security: Bastion Server
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo-vpc-bh.html
• In this case there are multiple instances in the private subnet.
• The NAT blocks SSH to private subnet.
• A Bastion or jump server can SSH to private network.
• Bastion Server access is limited by ACL.
Use Case: "VPC with Public and Private Subnets and Hardware VPN Access"
• Host scalable web applications in the AWS cloud that are connected to your local datacenter.
• Create a VPC where instances in subnet one are webservers to communicate with the Internet.
• Subnet two has application servers to communicate with databases on your corporate network with IPsec VPN connection between VPC and your local network to secure communication between cloud application servers and local.
• Webservers and application servers in VPC implement Amazon EC2 elasticity and Auto Scaling.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario3.html
Use Case: "VPC with a Private Subnet Only and Hardware VPN Access"
• Move local applications to the cloud, launch additional webservers, or add more compute capacity to your network by connecting your VPC to your local network.
• Because your VPC can be hosted behind a local firewall, IT resources can move into the cloud without changing how users access them.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario3.html
Use Case: Disaster Recovery• Backup mission-critical data from your datacenter to a
small number of Amazon EC2 instances with Amazon Elastic Block Store (EBS) volumes
• Import your virtual machine images to Amazon EC2. • In the event of a disaster in your own datacenter, quickly
launch replacement compute capacity in AWS to ensure business continuity.
• When the disaster is over, send mission-critical data back to your datacenter and terminate the Amazon EC2 instances.
• All the benefits of a disaster recovery site at a fraction of the normal cost with no tapes etc.
http://aws.amazon.com/disaster-recovery/
Project CollaborationGitHub and CloudFormation are tremendous innovations for collaboration. As engineering schools trend more towards interdisciplinary projects and external collaborations, tools such as these must play a central role. For networking, the CloudFormation JSON scripts become a project deliverable, particularly for external projects. Once a custom template is created, nearly anyone can instantiate the network and its components using their own AWS credentials and account. These scripts can be delivered and updated on Git.
The skills needed for creating and administrating these projects are Git (strong learning curve for developers), JSON (easy for those familiar with Javascript), Ruby, networking fundamentals (TCP stack, CIDR notation, NAT, firewalls, port filtering, access control lists), web security, application configuration, operating systems (Ubuntu, CentOS, Windows), databases (Oracle, NoSQL, MySQL etc.),bash shell and powershell, AWS, VPC, and CloudFormation. There are also tools for big data analytics such as Redshift and Map/Reduce but these may be more relevant to large, commercial projects, unless big data is the project.
There are other systems such as Microsoft Azure, EMC, Oracle and OpenStack.
Visual Studio 2013AWS Plugin and Git
AWS Explorer Git Team Explorer
VS2013
AWS Solution Calculator (AWS Plugin)
AWS For Eclipse
Git on Eclipse
Introductory Lesson Plans
• Use free private GitHub accounts to teach programming courses.• Create EC2 instances for each student to learn any OS.• Create individual EC2 instances for LAMP stack (MySQL dbase-backed web site).• Add NoMachine or Remote Desktop above to work in GUI.• Use Oracle for database concepts on individual EC2.• Configure course instance for programming specific subject (i.e.BioPerl for
bioinformatics).• Configure course instance with AMI for specific subject (i.e. CloudBioLinux).• Scripts traded from GitHub.• Use micro-instances and free tier to limit costs.
Intermediate Lesson Plans• Combine Github and AWS instance for group project.• Use CloudFormation to configure “VPC with a Single Public Subnet Only”
manually on VPC.• Use Cisco Cloud Services Router AMI to teach IOS XE command line etc.• Build "VPC with Public and Private Subnets" from template, make changes to
NAT and security settings and test. Add and configure subdomain and submit CloudFormation script and test report.
• Build any network and create a template from it to submit with Cloudformer.• Scripts traded from GitHub.• Use AWS educator grants to defray cost. AWS grant.
Advanced Lesson Plans• Build "VPC with Public and Private Subnets" and add and Bastion server and
submit CloudFormation script and test report.• Send any CloudFormation script by GitHub have students build and test and
submit revised script and test report.• Students build any VPC project, reverse engineer to script and submit on Git. • Pull any student script or branch into your Git repo on Visual Studio 2013 or
Eclipse, build to test and deploy to test.• Create advanced database project and test (i.e. web site with stored procedures
or NoSQL).• Scripts traded from GitHub.• Use AWS educator grants to defray cost. AWS grant.
Expert Lesson Plans
• Build "VPC with Public and Private Subnets" and add load balancing to front end. Submit reverse engineered script and report. (HA)
• Build "VPC with Public and Private Subnets" and add cache to backend end. Submit reverse engineered script and test report.
• Build "VPC with Public and Private Subnets" and add VPN to local network and block SSH on NAT. Submit reverse engineered script and report.
• Replace CF “user data” configuration with CHEF scripting on above.• Use CF “user data” to install Puppet and configure and test Puppet Master
reconfiguration of servers.• Build "VPC with Public and Private Subnets" and add load balancing to front end.
Submit reverse engineered script and report. • Build a multi-tiered web site configured by CHEF and Puppet. Submit reverse
engineered script and report. • Build "VPC with Public and Private Subnets" and add auto scaling. Submit reverse
engineered script and test report. • Scripts traded from GitHub.• Use AWS educator grants to defray cost.
Git and GitHub• “Git is a distributed revision control and source code management (SCM)
system with an emphasis on speed. Git was initially designed and developed by Linus Torvalds for Linux kernel development in 2005.” Wikipedia
• Git keeps copies of repositories locally and keeps history with pointers to a tree-like structure.
• The GitHub site combines social networking with Git. Free access is public, paid access can be private and GitHub provides free private accounts for educators for lab work. Repos can be cloned, branched, pushed, pulled and merged and GitHub provides the communication backbone for pull requests and general collaboration.
“Throughout my entire college career, all programming assignments have had to be submitted through BlackBoard. To be as diplomatic as possible, BlackBoard does a lot of things to help students & teachers. Unfortunately it does not do a single one of them well. The normal way to submit coding assignments is to have homework assigned through BlackBoard and to have to code up the solutions and then zip them up and submit them using the web interface. The best part about using GitHub for homework submissions is that it is incredibly easy to give feedback and help the students through learning the material.” Josh Davis, TA
Git and GitHub“There are two options for how to structure the repositories for your class:ForksAnyone who can see the base repository can see all forks, i.e. all students can see one another's work.Different solutions can be cross-referenced, encouraging peer learning.Pull requests can be used for submission and code review.Ideal for collaborative assignments, which can act like open source projects.SandboxingIndividuals can only see repositories when explicitly given access, i.e. students are blocked from seeing one another's work.Requires manual repository and permission setup.”source: https://education.github.com/guide
From GitHub: Hey drilett, we have awesome news...We've upgraded @WIT-CSN to a Gold plan with free private repositories. If you need help getting started with GitHub in your class see: https://education.github.com/guide
Git and GitHub
Git includes “diff” for comparing versions on GitHub, in local GUI and in bash shell.
Cloud Formation Template
CloudFormation scripts and automates cloud deployment of servers, databases, firewalls, routers, applications and networks generally. Existing AWS VPC’s can be captured as scripts (reverse engineered) The process uses JSON templates.
• There are six top level JSON objects: 1. AWS Template Format Version 2. Description3. Parameters 4. Mappings5. Resources6. Outputs• A top-level object for the Resources object is required. • Must declare at least one resource.
http://aws.amazon.com/cloudformation/aws-cloudformation-templates/
CloudFormation (JSON)
JSON(Used much like XML, perhaps faster)
• JSON is built on the following two structures:• A collection of name/value pairs. In various languages, this is realized as an object, record, struct,
dictionary, hash table, keyed list, or associative array.• An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.• These are universal data structures. Virtually all modern programming languages support them in
one form or another. • In JSON, they take on these forms:• An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends
with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).
• An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
• A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
• A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. A character is represented as a single character string. A string is very much like a C or Java string.
• A number is very much like a C or Java number, except no octal and hexadecimal formats.• JSON is a lightweight, plain-text alternative to XML used for client-server communication in web
apps. source: json.org
CloudFormation + Chef• Repeatable builds• Known configuration• Source control of infrastructure
"files" : { "/etc/chef/solo.rb" : { "content" : { "Fn::Join" : ["\n", [ "log_level :info", "log_location STDOUT", "file_cache_path \"/var/chef-solo\"", "cookbook_path \"/var/chef-solo/cookbooks\"", "json_attribs \"/etc/chef/node.json\"", "recipe_url \"https://s3.amazonaws.com/cloudformation-examples/wordpress.tar.gz\"" ]] },
In “UserData” section of the JSON template after yum and pull on Ruby gems etc.”
Ruby
Poignant Guide to Ruby
• Interpretive programming: No compilation ; you edit and send your program to the interpreter.• Dynamic programming: Almost everything done at runtime. Types and class and method definitions
are determined at runtime.• Write programs within programs and execute them.• Familiar syntax: Java, Perl, Python, C/C++ "ish"Simple factorial function:def factorial(n) if n == 0 return 1 else return n * factorial(n-1) endend• Iterators: loop abstraction is built in, blocks can be attached to method call which can call back the
block. Array with each method example:ary = [1,2,3,4,5]ary.each do |i|puts i*2end # prints 2,4,6,8,10 for each line• More: ...
LearnToProgram
Puppet
Puppet can automate the entire infrastructure lifecycle: discovery, provisioning, operating system and application configuration, management and reporting.
Puppet is an open source, configuration management framework that knows how to configure all machines to a predetermined state automatically once provided the required “manifest” set.
• Use to configure multiple machines similarly.• Use to set up infrastructure that requires dynamic scaling-up and scaling-
down of machines to a pre-built configuration.• Use to control all configured machines so a centralized change gets
propagated to all automatically according to type etc.
Puppet Labs Learn
Database
• Amazon Web Services provides several different database products.– MySQL, Oracle and MS SQL Instances• Deployed as pre-built Amazon Machine Images• Need to include Elastic Block Storage for persistance• Prices varies by size and type of DB
– Amazon Relational Database Service • Fault-tolerant, scalable database management system
– NoSQL databases
AWS Simple Icons: Database
new!
Typical Database Project
EBS Volume
S3
Database Server
Web Server
Web Serverr
ELB
Database Project
• Essentially, an entire Web-based database application hosted on AWS
• Web servers, elastic load balancers and database servers
• Students placed all static code on S3 storage for quick deployment of new servers
• Could have been improved by using RDS instead of MySQL instance
Database Project
• What students learned from this project– Reinforced database design learned in previous
courses– Some networking concepts– Internet programming using HTML, JavaScript,
Jquery, CSS and PHP– How to deploy an application in the cloud
Foundational Level
Getting Started with AWS Session – TechnicalAudience Level: Foundational• Security• Credential management• Provisioning• Managing• Amazon EC2• Amazon EBS• Amazon S3• Elastic Load Balancing• Auto Scaling• Amazon CloudWatch• Amazon RDS.• Virtual Private Network (VPN) connection between your datacenter and your VPC
https://aws.amazon.com/aws-summit-2014/amsterdam/
AWS Summits 2014
June 10,2014
Intermediate: HA
Architecting Highly Available Applications on AWSAudience Level: Intermediate• High availability options for workloads• Architectural design patterns for automating fault tolerance• Architectural design patterns for high availability• Application security
Intermediate: Big Data
Store, Manage and Analyze Big Data in the CloudAudience Level: Intermediate
• Big Data scenarios that merge structured and unstructured data into a data warehouse for analysis
• Gather and store large quantities of data• Amazon Elastic Map Reduce to process raw data using
the massive scale of Amazon EC2 • Amazon Redshift to analyze data and make it accessible
to business users. AWS Data Pipeline and Amazon Kinesis.
NYC AWS Session “Levels”
Introductory – “I’m new to this.” Gives an overview of the a service or feature with the assumption that attendees are new to the topic. These sessions highlight basic use cases, features, functions, and benefits.
Advanced – “I get it and want to learn more.”Advanced sessions will dive deeper into the topic. Code may be shared but will not be the primary focus of the session.
Expert – “I live and breathe this stuff.”Much of these sessions are code-based, covering best practices, advanced tricks, and future developments.
July 10, 2014
https://aws.amazon.com/aws-summit-2014/new-york/
Sources:Cisco Cloud Services Router (IOS XE command line etc.)
Josh Davis on GitHub
ComS 342: Principles of Programming languages at Iowa State during Spring 2014.
GitHub Educator Discount
Github for Windows
JSON
Ruby Intro
Road to the Whitehouse
Bitly = TMuOpa
• A virtual private cloud (VPC) of size /16 (example CIDR: 10.0.0.0/16). This provides 65,536 private IP addresses.
• A subnet of size /24 (example CIDR: 10.0.0.0/24). This provides 256 private IP addresses.
• An Internet gateway. This connects the VPC to the Internet and to other AWS products, such as Amazon Simple Storage Service (Amazon S3).
• An instance with a private IP address in the subnet range (example: 10.0.0.6), which enables the instance to communicate with other instances in the VPC, and an Elastic IP address (example: 198.51.100.2), which enables the instance to be reached from the Internet.
• A route table entry that enables instances in the subnet to communicate with other instances in the VPC, and a route table entry that enables instances in the subnet to communicate directly over the Internet.
• A virtual private cloud (VPC) of size /16 (example CIDR: 10.0.0.0/16). This provides 65,536 private IP addresses.
• A public subnet of size /24 (example CIDR: 10.0.0.0/24). This provides 256 private IP addresses.• A private subnet of size /24 (example CIDR: 10.0.1.0/24). This provides 256 private IP addresses.• An Internet gateway. This connects the VPC to the Internet and to other AWS products, such as Amazon
Simple Storage Service (Amazon S3).• Instances with private IP addresses in the subnet range (examples: 10.0.0.5, 10.0.1.5), which enables them
to communicate with each other and other instances in the VPC. Instances in the public subnet also have Elastic IP addresses (example: 198.51.100.1), which enable them to be reached from the Internet. Instances in the private subnet are back-end servers that don't need to accept incoming traffic from the Internet; however, they can send requests to the Internet using the NAT instance (see the next bullet).
• A network address translation (NAT) instance with its own Elastic IP address. This enables instances in the private subnet to send requests to the Internet (for example, for software updates).
• A custom route table associated with the public subnet. This route table contains an entry that enables instances in the subnet to communicate with other instances in the VPC, and an entry that enables instances in the subnet to communicate directly with the Internet.
• The main route table associated with the private subnet. The route table contains an entry that enables instances in the subnet to communicate with other instances in the VPC, and an entry that enables instances in the subnet to communicate with the Internet through the NAT instance.
• A virtual private cloud (VPC) of size /16 (example CIDR: 10.0.0.0/16). This provides 65,536 private IP addresses.
• A public subnet of size /24 (example CIDR: 10.0.0.0/24). This provides 256 private IP addresses.• A VPN-only subnet of size /24 (example CIDR: 10.0.1.0/24). This provides 256 private IP addresses.• An Internet gateway. This connects the VPC to the Internet and to other AWS products, such as Amazon
Simple Storage Service (Amazon S3).• A VPN connection between your VPC and your network. The VPN connection consists of a virtual private
gateway located on the Amazon side of the VPN connection and a customer gateway located on your side of the VPN connection.
• Instances with private IP addresses in the subnet range (examples: 10.0.0.5 and 10.0.1.5), which enables the instances to communicate with each other and other instances in the VPC. Instances in the public subnet also have Elastic IP addresses (example: 198.51.100.1), which enables them to be reached from the Internet. Instances in the VPN-only subnet are back-end servers that don't need to accept incoming traffic from the Internet, but can send and receive traffic from your network.
• A custom route table associated with the public subnet. This route table contains an entry that enables instances in the subnet to communicate with other instances in the VPC, and an entry that enables instances in the subnet to communicate directly with the Internet.
• The main route table associated with the VPN-only subnet. The route table contains an entry that enables instances in the subnet to communicate with other instances in the VPC, and an entry that enables instances in the subnet to communicate directly with your network.
CloudFormer supports:Amazon VPC (VPC, Subnets, RouteTables, Routes, etc.)Amazon CloudWatch AlarmsAmazon DynamoDBAmazon ElastiCacheAmazon SNSAmazon S3 Bucket PolicyAmazon SQS Queue PolicyAmazon SNS Topic PolicyThe CloudFormer tool allows you to select any of the AWS resources running in your account for inclusion in the template. If you select resources that have dependent resources (for example, an Amazon EC2 instance is associated with an EC2 security group), the tool will automatically select any dependent resources. You can override any of the pre-selected resources or add other resources as necessary . you have full control of the resources to be included in your template. Logical names are chosen based on the existing resource names, however, you can edit the names that will be included in the template and you can add output parameters based on the attributes available for any of the resources. The template can be uploaded to your S3 bucket and launched directly via the AWS CloudFormation stack creation wizard.The CloudFormer tool is intended to create a starting point for your template. Once created, you can customize, for example:
Add Parameters to enable stacks to be configured at launch time.Add Mappings to allow the template to be customized to the specific environment.Replace static values with "Ref" and "Fn::GetAtt" functions to flow property data between resources where the value of one property is dependent on the value of a property from a different resource.Fill in your Amazon EC2 instance userdata to pass parameters to your EC2 instances at launch time.Customize your Amazon RDS DB instance database names and master passwords.
Class Leadingbits
Size of netwo
rknumber bit field
Size of rest
bit field
Numberof
networks
Addresses
per network
Start address
End address
Class A 0 8 24 128 (27)
16,777,216 (224) 0.0.0.0 127.255.
255.255
Class B 10 16 16 16,384 (214)
65,536 (216)
128.0.0.0
191.255.255.255
Class C 110 24 8 2,097,152 (221)
256 (28)
192.0.0.0
223.255.255.255
IP Addressing Reference
IPv4 CIDR ReferenceIP/CIDR Δ to last IP addr MaskHosts (*) Size Notesa.b.c.d/32 +0.0.0.0 255.255.255.255 1 1/256 Ca.b.c.d/31 +0.0.0.1 255.255.255.254 2 1/128 C d = 0 ... (2n) ... 254a.b.c.d/30 +0.0.0.3 255.255.255.252 4 1/64 C d = 0 ... (4n) ... 252a.b.c.d/29 +0.0.0.7 255.255.255.248 8 1/32 C d = 0 ... (8n) ... 248a.b.c.d/28 +0.0.0.15 255.255.255.240 16 1/16 C d = 0 ... (16n) ... 240a.b.c.d/27 +0.0.0.31 255.255.255.224 32 ⅛ C d = 0 ... (32n) ... 224a.b.c.d/26 +0.0.0.63 255.255.255.192 64 ¼ C d = 0, 64, 128, 192a.b.c.d/25 +0.0.0.127 255.255.255.128 128 ½ C d = 0, 128a.b.c.0/24 +0.0.0.255 255.255.255.000 256 1 Ca.b.c.0/23 +0.0.1.255 255.255.254.000 512 2 C c = 0 ... (2n) ... 254a.b.c.0/22 +0.0.3.255 255.255.252.000 1,024 4 C c = 0 ... (4n) ... 252a.b.c.0/21 +0.0.7.255 255.255.248.000 2,048 8 C c = 0 ... (8n) ... 248a.b.c.0/20 +0.0.15.255 255.255.240.000 4,096 16 C c = 0 ... (16n) ... 240a.b.c.0/19 +0.0.31.255 255.255.224.000 8,192 32 C c = 0 ... (32n) ... 224a.b.c.0/18 +0.0.63.255 255.255.192.000 16,384 64 C c = 0, 64, 128, 192a.b.c.0/17 +0.0.127.255 255.255.128.000 32,768 128 C c = 0, 128a.b.0.0/16 +0.0.255.255 255.255.000.000 65,536 256 C = 1 Ba.b.0.0/15 +0.1.255.255 255.254.000.000 131,072 2 B b = 0 ... (2n) ... 254a.b.0.0/14 +0.3.255.255 255.252.000.000 262,144 4 B b = 0 ... (4n) ... 252a.b.0.0/13 +0.7.255.255 255.248.000.000 524,288 8 B b = 0 ... (8n) ... 248a.b.0.0/12 +0.15.255.255 255.240.000.000 1,048,576 16 B b = 0 ... (16n) ... 240a.b.0.0/11 +0.31.255.255 255.224.000.000 2,097,152 32 B b = 0 ... (32n) ... 224a.b.0.0/10 +0.63.255.255 255.192.000.000 4,194,304 64 B b = 0, 64, 128, 192a.b.0.0/9 +0.127.255.255 255.128.000.000 8,388,608 128 B b = 0, 128a.0.0.0/8 +0.255.255.255 255.000.000.000 16,777,216 256 B = 1 Aa.0.0.0/7 +1.255.255.255 254.000.000.000 33,554,432 2 A a = 0 ... (2n) ... 254a.0.0.0/6 +3.255.255.255 252.000.000.000 67,108,864 4 A a = 0 ... (4n) ... 252a.0.0.0/5 +7.255.255.255 248.000.000.000 134,217,728 8 A a = 0 ... (8n) ... 248a.0.0.0/4 +15.255.255.255 240.000.000.000 268,435,456 16 A a = 0 ... (16n) ... 240a.0.0.0/3 +31.255.255.255 224.000.000.000 536,870,912 32 A a = 0 ... (32n) ... 224a.0.0.0/2 +63.255.255.255 192.000.000.000 1,073,741,824 64 A a = 0, 64, 128, 192a.0.0.0/1 +127.255.255.255 128.000.000.000 2,147,483,648 128 A a = 0, 1280.0.0.0/0 +255.255.255.255 000.000.000.000 4,294,967,296 256 A