HOW I LEARNED TO STOP WORRYING AND LOVE THE CLOUD
Using Amazon's Elastic Computing Cloud ( and a few too many Star Wars analogies) to create a scalable server infrastructure for School of Everything.
David O'Dwyer (SupportFour) Peter Brownell (Code Positive / School of Everything)
Dr Clonelove:
WHAT'S AHEADHow we built a server cluster on Amazon EC2 for a Drupal based website.It's all about how to add and remove servers on the fly.
The set upWhy use the cloud ?An overview of our server infrastructure
How we got started with the cloudSome sysadmin religious mumbo-jumboGeneral best-practice for any cluster
TechnologyThe key tools to make it happen
Image by - http://www.flickr.com/photos/powerpig/
THE SILVER LININGWhy choose the cloud ?
Startups are poorPay for servers by the hourAdd a new server in a few minutesTurn them off when not needed
http://flickr.com/photos/balakov/2468552226
SCALE DOWNBe able to grow and shrinkThe key word really is “Elastic”Know how you will add and remove new servers
Photo credit: http://www.flickr.com/photos/34558813@N07/
BASIC STRUCTURECode built on DrupalMachine images run Debian LennyMySQL database with Replication & MySQL proxyMany custom scripts
WHAT WE BUILTBasic configuration – grow as needed
2 web-servers (small instance) 1 primary database (small instance) 2 slave databases 2 off-cloud machines for monitoring and backups
THE ENDWork Backwards
Non-persistence – all changes lost on power downMust be able to rebuild everything - automaticallyishStart with your plan for complete failureMake sure you can test your backup system
ULTIMATE FAILURE
Be able to rebuild from backupsDocument and blogFire Drills – practice makes perfect
Assume that everything may just vanish
CLOUD CITYNever have faith in just one allySet up a base off the cloud Keep a backup on the groundMonitor from outsideMail smarthost to avoid auto greylisting
IT'S A TRAP!What does it look like – working or not.
Don't trust anything! Monitor as much as possible - Clues for where problems arise - Capacity planning
We use Nagios and Munin.
NOAH'S ARK PRINCIPLE:Expect the worst and have two of everythingTraditionally expensiveAdding new servers takes timeYou may just end up with a lot of hardware doing nothing
http://www.thebricktestament.com/genesis/the_flood/gn07_07_10.html
IT'S MUCH EASIER WITH CLONES
Configure one original, and make many copies, in multiple zonesOnly pay for the time you useNoah says: “It's not having two of everything running, it's being able to”
http://www.flickr.com/photos/jeremymates/2362399109/
Virtual machines are cheap – but you still need to plan
STORAGE YOU CAN RELY ONHow safe is your data Local disk – EC2 Ephemeral Storage (/mnt)EBS - Elastic Block Store Point-in-time snapshots
CLONE 2 CLONE FILESYSTEM
Avoid dependencies
Local copy of uploaded files GlusterFS syncs files instantlyPrimary webserver uses persistent storage
Image by - http://www.flickr.com/photos/powerpig/
SMALLER IS BETTER
Database replication – easy and reliableMultiple small images as database serversMySql proxy makes cluster transparent
MySQL(tm) is MyFriend
http://www.flickr.com/photos/powerpig/MySQL is a registered trademark of MySQL inc, no it's Sun Microsystems, no, woops looks like it's Oracle Inc. now. How weird is that ? Anyway, we like it, and are happy to endorse them – however owns them.
Best practice guide: http://www.mysql.com/why-mysql/white-papers/mysql_wp_best_practices_mysql_ec2.php
THE BALANCING ACTRound Robin DNS is the cheap solution for load balancingEach machine configures dynamic DNS on first startupInvestigating new amazon load balancing
Image by - http://www.flickr.com/photos/powerpig/
WHAT WE SAIDCloud infrastructure is not alien, it's just a little less solid – it can grow or shrink.The cloud forces best practice – no time for slacker sysadmins!
Better safe than sorryRemember Noah, avoid dependenciesWork backwards – rebuild, restore, backup, operateKeep a base in the real worldMonitor everything
Managing growthLoads of local storage, persistent elastic blocks, instant snapshots and S3Synchronise user uploads with GlusterFSScale MySQL with replication and MySQL ProxyGrowing number of load balancing options
MailMail smarthost off-cloud to avoid greylisting
Image by - http://www.flickr.com/photos/powerpig/
http://supportfour.com
http://schoolofeverything.com/
Want to learn more about Amazon Web Services ? Sign up on School of Everything and tag yourself with AWS.
http://schoolofeverything.com/subject/AWS