Scaling DevOps

Download Scaling DevOps

Post on 15-Jan-2015

389 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

Scaling DevOps from the Seattle Scalability Meetup, December 2013

TRANSCRIPT

<ul><li> 1. Scaling DevOps Jeffrey Hulten Whitepages, Inc. Wednesday, December 4, 13</li></ul> <p> 2. Who Am I? Jeffrey Hulten - @jhulten Software Engineer, Architecture Improvement at WhitePages, Inc. Organizer of Seattle DevOps MeetUp Too Ops for Dev, Too Dev for OpsWednesday, December 4, 13 3. Scaling DevOpsYour Tools Arent Enough Your First 100 Machines Watching the HorizonWednesday, December 4, 13 4. Your Tools Arent Enough Culture Deploy a Package Manage the Metal Minimize the Surface Area Growing PainsWednesday, December 4, 13 5. Culture Not Just for Bacteria Anymore Starts with one person Infect your friends Try changing things Mutation must allow for failure Requirement for survivalWednesday, December 4, 13 6. Deploy a Package TGZ OS Package (.deb / .rpm) Machine Image (AMI) LXC Image (Docker)Wednesday, December 4, 13 7. OS Packages Made EasierFPM Package Cong Separately Package RepositoryWednesday, December 4, 13 8. FPM: Efng Package Management https://github.com/jordansissel/fpm $ gem install fpm Make RPM, DEB, and Solaris packages and Puppet module Create from directories, RPM, DEB, Gem, PyPi, Tarball and moreWednesday, December 4, 13 9. Example: Jenkins NAME=jenkins VERSION=1.396 .PHONY: package package: rm -f jenkins.war wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war fpm -s dir -t deb -n $(NAME) -v $(VERSION)--prefix /opt/jenkins jenkins.warWednesday, December 4, 13 10. Machine Images &amp; Containers Machine Image Entire OS All Services Enabled Containers Minimal OS Image Run One Process (Replace `init`)Wednesday, December 4, 13 11. Manage the Metal Application as Inventory...Wednesday, December 4, 13 12. Minimize the Surface Area What is Surface Area? Libraries? Versions? Components? Data Stores?Wednesday, December 4, 13 13. Minimize the Surface Area What is Surface Area? Libraries? Versions? Components? Data Stores? Deployables...Wednesday, December 4, 13 14. Growing Pains Dont bottle knowledge There are no specialists Think about the smallest solution Delay your decisionsWednesday, December 4, 13 15. Your First 20 Machines DEMO TIME! Source at:https://github.com/whitepages/scaledemo-20-machinesAWS CloudFormation, cloud-init...Wednesday, December 4, 13 16. Application: Random Numbers as a ServiceWednesday, December 4, 13 17. CloudFormationDeploy Stacks Templates in JSON Parameters for ReuseWednesday, December 4, 13 18. Template: Parameters "Parameters": { "OperatorEmail": { "Type": "String", "Description": "Email address to notify if there are any scaling operations" }, "InstanceType": { "Description": "WebServer EC2 instance type", "Type": "String", "AllowedValues": [ "t1.micro", "m1.small", ...], "Default": "m1.small", "ConstraintDescription": "must be a valid EC2 instance type." }, ... "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(d{1,3}).(d{1,3}).(d{1,3}).(d{1,3})/(d{1,2})", "Description": "The IP address range that can be used to SSH to the EC2 instances", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },Wednesday, December 4, 13 19. Template: Parameters "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(d{1,3}).(d{1,3}).../(d{1,2})", "Description": "...", "ConstraintDescription": "..." }Wednesday, December 4, 13 20. Template: Mappings "Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-955b79fc" }, "us-west-1": { "64": "ami-6ca89929" }, "us-west-2": { "64": "ami-a8395c98" } }, ... },Wednesday, December 4, 13 21. Template: Resources "Resources": { "NotificationTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [ { "Protocol": "email", "Endpoint": { "Ref": "OperatorEmail" } } ] } }, ... }Wednesday, December 4, 13 22. AutoScaling Group "RandServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "NotificationConfiguration": { "NotificationTypes": [ "autoscaling:EC2_INSTANCE_LAUNCH", "autoscaling:EC2_INSTANCE_LAUNCH_ERROR", "autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" ], "TopicARN": { "Ref": "NotificationTopic" } }, "MinSize": "1", "MaxSize": "100", "LaunchConfigurationName": { "Ref": ..." } }} Wednesday, December 4, 13 23. Launch Cong "RandLaunchConfig": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "SecurityGroups": [{ "Ref": InstanceSecurityGroup" }], "InstanceType": { "Ref": "InstanceType" }, "KeyName": { "Ref": "KeyName" }, "UserData": { "Fn::Base64": { "Fn::Join" : ["",[ "#includen", "https://.../cloud-config.txtn" ]]}}, "ImageId": {"Fn::FindInMap": [...] } } },Wednesday, December 4, 13 24. Cloud Init #cloud-config packages: - ruby1.9.3 - git - build-essential runcmd: - /usr/bin/gem install bundler - /usr/bin/git clone "https://github.com/whitepages/ scaledemo-20-machines.git" /opt/rand - cd /opt/rand &amp;&amp; /usr/local/bin/bundle install - cd /opt/rand &amp;&amp;/usr/local/bin/bundle exec randserver &amp;Wednesday, December 4, 13 25. Watching the Horizon Docker / LXC SmartStack Akka Cluster / Riak CoreWednesday, December 4, 13 26. Questions? jhulten@whitepages.com Twitter: @jhulten Github: jhulten whitepages whitepages.com/careersWednesday, December 4, 13 </p>