aws intro for knight news fellows
DESCRIPTION
I gave a brief intro about using AWS to our Mozilla Knight News Fellows.TRANSCRIPT
![Page 1: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/1.jpg)
AWS Fun
![Page 2: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/2.jpg)
A short bit of history
● Not that long ago, a "server" was :○ One piece of hardware○ One operating system○ Physically racked, powered, networked in a
managed datacenter
● People started playing with "virtualization"○ One piece of hardware○ Multiple operating systems running independently○ Physically racked, powered, networked in a
managed datacenter
![Page 3: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/3.jpg)
As virtualization was taking off...
● Mid-2000's, Amazon.com buys a TON of hardware.
● The mantra for the folks building Amazon.com infrastructure:Provide service style endpoint access to infrastructure management for internal use.
EVERYTHING IS AN API
![Page 4: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/4.jpg)
At the same time....
Marketing depts everywhere goto town, asmarketing does...
VIRTUALIZE IT ALL!!!!
![Page 5: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/5.jpg)
Amazon Realizes...
If we run the virtual server hosts...
And we just open up our internal infrastructure API's to end users...
![Page 6: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/6.jpg)
$$$$$$$$$$$$$$$$$$$$$$$$$$
![Page 7: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/7.jpg)
Marketing took over, now everything is Cloud.
![Page 8: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/8.jpg)
By Cloud, I mean....
● Must be distributed.
● Must be programatically accessible
● Is multi-tenanted (you are not the only user of the hardware)
![Page 9: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/9.jpg)
In general, what is AWS?
● A collection of commonly used pieces of software, made easily accessible in:
○ Distributed environment: Multiple Availability zones per region, multiple regions
○ Programatically accessible infrastructure
For example: Mysql, MS SQL, Memcached, Linux, Windows, CDN, DNS Management, User/Admin management, Firewalls, Load balancers...
![Page 10: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/10.jpg)
Common components of infrastructurein your old datacenter
![Page 11: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/11.jpg)
Common components of infrastructurein AWS
![Page 12: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/12.jpg)
Some of what this buys us
● We can spin up replica environments● Easier functional testing...in STAGING● Load test against prod without touching prod● Build in automated deployments and testing,
making pushing to prod a breeze for all devs.
● This makes the feedback loop tighter, faster, and keeps changes and their inevitable bugs more in context
● This all wraps up to make you, the devs, more confident to try new things.
![Page 13: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/13.jpg)
Controlling all of that infrastructure
![Page 14: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/14.jpg)
Lots of configuration management options....
● Chef (Opscode)
● Puppet (What I use)
● AMI's (Server images)
● Cloudformation (AWS Service)
![Page 15: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/15.jpg)
![Page 16: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/16.jpg)
But wait...isn't the cloud dangerous?
● Yes! Just as dangerous as your datacenter
● Secrets stores in S3, managed by puppet
● Each app has its own key, security groups
● Managing security via security groups, ssh keys
![Page 17: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/17.jpg)
![Page 18: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/18.jpg)
General scaling on AWS
● Use autoscale groups (even if you never have to autoscale)
● You can trigger autoscaling on any metric● Use EBS and instance store autoscale
groups○ 30 seconds to "traffic ready" prebuilt EBS instance
vs. 2-10 min for a instance store○ Keep a baseline # of instance store nodes, for when
EBS has issues.○ You can have multiple autoscale groups load into
one ELB (so, app-ebs-fastscale-group and app-instancestore-noscale-group)
![Page 19: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/19.jpg)
General scaling on AWS
● For high IO data (RDS or self-managed EC2), use provisioned IOPS.
● On EC2, EBS volumes can be RAID10'd...need a 50k IOPS volume? :D Great way to vertically scale.
●
![Page 20: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/20.jpg)
General scaling on AWS
● Adhere to 12factor.net rules so you can horiziontally scale○ CNAME all resources, such as mysql servers. If you
can easily move a resource, you can easily vertically scale it elsewhere and move to it.
○ Store dependent content away from web tier nodes, ie media, user uploads. If a web node dies and you lost anything, you did it wrong.
○ All pieces of app modular, independently scalable and revvable without retooling
![Page 21: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/21.jpg)
![Page 22: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/22.jpg)
General High Availability on AWS● Multi-Region (Each region has multiple AZ's)
● Multi-Availability Zone for ○ RDS (built in) (takes ~3 min to failover)○ Load balancing○ Autoscaling groups (3 AZ's recommended)
● Dynamic DNS
● Health Checks on apps
![Page 23: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/23.jpg)
General High Availability on AWS
● Mix in instance store baseline with EBS for fast scaling for when EBS has issues.
● Health Checks on apps
● Status updates to S3 file, updates app to point to failover resources... No db? Write to a SQS queue!
● Oh yeah, use a lot of SQS!
![Page 24: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/24.jpg)
![Page 25: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/25.jpg)
![Page 26: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/26.jpg)
![Page 27: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/27.jpg)
CNAME for all the resources (12-factor friendly)
![Page 28: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/28.jpg)
Easier to move, failover, rebuild
![Page 29: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/29.jpg)
RDS Tricks● Multi-AZ, takes ~3 min to failover
● EBS volumes of greater storage get better performance, always use 300gb for prod, even for small instances.
● Read slaves have a lot of challenges with schema changes. It is usually faster to just rebuild slaves
● For monitoring, grant repl client to user
![Page 30: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/30.jpg)
Some other tricks● ELB's are EBS-backed EC2 Instances...when EBS
alerts go out, be careful!● Setup ifttt alerts for AWS RSS status updates● Use New Relic. Please!● IAM Roles allow for interaction with AWS
infrastructure...think, a monitoring server that tells an autoscale group to respond to a problem by launching new nodes
● Route53 is awesome. Alias A records, super reliable, you can keep low ttl's
![Page 31: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/31.jpg)
Pay Amazon Less● Reserved instances can save a lot of money
● Spot instances are great for batch and processing, EMR, Cluster Compute
● S3 static hosting is ridiculously inexpensive. Go that route for anything static.
● For dev work, Heroku is great, no cost for apps that do not scale
![Page 32: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/32.jpg)
Other random advice...
![Page 33: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/33.jpg)
Good stuff● http://www.12factor.net/● http://paulstamatiou.com/how-to-getting-started-with-
amazon-ec2● http://loggly.com/blog/2011/05/send-custom-metrics-to-
cloudwatchs-api/● https://github.com/toolness/fleeting● AWS Marketplace has a lot of good stuff● My example repos: https://github.
com/mozilla/sys_config_examples and https://github.com/mozilla/sys_scripts_examples
● https://help.ubuntu.com/community/CloudInit● http://awsofa.info/ ●
![Page 34: AWS Intro for Knight News Fellows](https://reader033.vdocuments.net/reader033/viewer/2022042813/545b596baf79590b088b5e79/html5/thumbnails/34.jpg)
Demo time (if there is time)
-Building a new autoscale group/app?-Managing infrastructure via fabric, jenkins, puppet-Show off the puppet systems config setup?