deployment with fabric
TRANSCRIPT
![Page 1: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/1.jpg)
Prodution Architecture and
Deploymentwith Fabric
- Andy McCurdy -@andymccurdy
![Page 2: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/2.jpg)
Whiskey Media
![Page 3: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/3.jpg)
Whiskey Sites
![Page 4: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/4.jpg)
Your First Django App
![Page 5: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/5.jpg)
Basic Config (web)•Apache•mod_wsgi•use daemon mode•threads more efficient•processes if you're unsure of thread safety
WSGIDaemonProcess my-site python-path=/home/code/ processes=2 threads=150 maximum-requests=5000WSGIProcessGroup my-siteWSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi
![Page 6: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/6.jpg)
Basic Config (media)•Nginx•Use Nginx to proxy traffic to Apache•Meanwhile Nginx serves media
upstream my-site { server 127.0.0.1:8000;}server { listen 80; location ~ ^/media/ { root /home/code/my-site; expires 30d; } location / { proxy_pass http://my-site; proxy_set_header X-Real-IP $remote_addr; }}
![Page 7: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/7.jpg)
Basic Config (db)•Databases•PostgreSQL
•Frank Wiles @ www.revsys.com
•MySQL•Percona @ www.mysqlperformanceblog.com
![Page 8: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/8.jpg)
Basic Config (cache)•Use Memcached! Even 16mb will significantly help against a Digg or being Slashdot'ed•It's incredibly easy...
# settings.pyMIDDLEWARE_CLASSES = ( 'django.middleware.cache.UpdateCacheMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.cache.FetchFromCacheMiddleware')
CACHE_BACKEND = 'memcached://127.0.0.1:11211/'CACHE_MIDDLEWARE_SECONDS = 60*5 # 5 minutesCACHE_MIDDLEWARE_ANONYMOUS_ONLY = True
![Page 9: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/9.jpg)
Basic Deployment•Copy Code & Media (rsync or scp)•Run DB migrations / syncdb•Bounce wsgi daemons
![Page 10: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/10.jpg)
Managing Growth (db)•Move DB to its own server
•As much RAM as possible•Write-heavy (>10%)? Get fast disks•Tune your config file
![Page 11: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/11.jpg)
Managing Growth (web)•Add more web servers
•Use a resource monitoring tool like Munin to understand if your app is CPU or memory bound
![Page 12: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/12.jpg)
Even More Growth•Replicated or sharded Databases•Multiple load balancers for redundancy•Message queues•Crons•Search daemons (Solr, Sphinx)•etc...
![Page 13: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/13.jpg)
(not so) Basic Deployment
image credit:Brad
Fitzpatrick
![Page 14: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/14.jpg)
Deployment Reqs•Copy media to CDN•Maintenance splash page•Run DB migrations•Install/Upgrade Python dependencies•Add a new web server to the cluster•Execute arbritrary commands for sysadmin maintenance tasks
![Page 15: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/15.jpg)
Deployment OptionsCapistrano
+ Out of box support for common use cases+ Hooks to customize tasks+ Source control integration+ Threaded deployment to multiple hosts
- Ruby :(
![Page 16: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/16.jpg)
Deployment OptionsFabric
+ Very simple, tasks are just Python functions+ Easy to chain together tasks to create complex scripts out of bite size pieces
- No source control integration- No out of box support- Some bugs, although fairly easy to work around, and new maintainer is working on fixes
![Page 17: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/17.jpg)
Fabric Basics•sudo easy_install fabric•need a fabfile.py•from fabric.api import *
•be mindful of tasks that may fail•each remote command starts fresh•changing directories
![Page 18: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/18.jpg)
Core Functionality•local() - Run a command locally
•run() - Run a command remotely
•sudo() - Run a command remotely as another user
•put() - Copy a file from local to remote
•get() - Copy a file from remote to local
•many more helper-ish commands
![Page 19: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/19.jpg)
Authentication•Relies on SSH model•Use SSH keys•Control access to root user via sudoers•When you have to revoke access, you just turn off their SSH account
![Page 20: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/20.jpg)
Configuration•Fabric environment (env) -- it's just a dictionary•Hosts and Roles•Code Repositories•Whatever you need
•~/fabricrc•Global settings or all Fabric deployments•SSH username
![Page 21: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/21.jpg)
Example Config# fabfile.pyfrom fabric.api import *
env.roledefs = { 'web' : ['10.1.1.1', '10.1.1.2'], 'db' : ['10.1.1.3'], 'lb' : ['10.1.1.4'], }
env.repositories = {...}
![Page 22: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/22.jpg)
Tasks# fabfile.pydef uptime(): run('uptime')
$> fab uptime -H 10.1.1.3[10.1.1.3] run: uptime[10.1.1.3] out: 05:20:39 up 88 days, 12:00, 0 users, load average: 0.03, 0.03, 0.00
![Page 23: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/23.jpg)
Mapping Roles to Tasks# fabfile.py
@roles('web')def uptime(): run('uptime')
$> fab uptime[10.1.1.1] run: uptime[10.1.1.1] out: 05:20:39 up 88 days... [10.1.1.2] run: uptime[10.1.1.2] out: 05:20:39 up 88 days...
![Page 24: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/24.jpg)
Decorator Problems•Some problems with Fabric's role management•Can't override decorated tasks at command line as docs suggest
def default_roles(*role_list): def selectively_attach(func): if not env.roles and not env.hosts: return roles(*role_list)(func) else: if env.hosts: func = hosts(*env.hosts)(func) if env.roles: func = roles(*env.roles)(func) return func return selectively_attach
![Page 25: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/25.jpg)
All better now#fabfile.py
@default_roles('web', 'db')def uptime(): run('uptime')
$> fab uptime# runs on all hosts in the 'web' and 'db' roles
$> fab uptime --roles lb# runs only on hosts in the 'lb' role
![Page 26: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/26.jpg)
Dealing with Failures•By default Fabric dies if a task fails•Use a context manager when failures are anticipated
# fabfile.pyfrom __future__ import with_statement # py2.5
def symlink_me(): with settings(warn_only=True): run('rm /path/to/symlink') run('ln -s /home/andy /path/to/symlink')
![Page 27: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/27.jpg)
Easy sys-admin•Make an "invoke" command•Great for sys-admin and one-off tasks
# fabfile.py
@default_roles('all')def invoke(command): "Invoke an arbritrary command" sudo(command)
# install new packages on all hosts in one command$> fab invoke:"apt-get install git-core"
![Page 28: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/28.jpg)
Real World Tasks$> fab --list
Available commands:
bounce_wsgi_procs Bounce the WSGI procs by touching the filesdeploy Full deploymentdeploy_media Push media to S3invoke Invoke an arbritrary commandmigrate Run any migrations via Southreload_nginx Update Nginx's running configsplash_off Configure Nginx to serve the sitesplash_on Configure Nginx to serve a downed-site pageupdate_repositories Push code to serversupdate_dependencies Update dependencies to third party libs
![Page 29: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/29.jpg)
Whiskey's Deploymentdef deploy(splash='no'):
"Full deployment" deploy_media() update_cached_repositories() update_dependencies() generate_releases() if splash == 'yes': splash_on() _symlink_code() migrate() bounce_wsgi_procs() if splash == 'yes': splash_off()
$> fab deploy:splash=yes
![Page 30: Deployment with Fabric](https://reader035.vdocuments.net/reader035/viewer/2022062514/558e9d1f1a28ab53528b4606/html5/thumbnails/30.jpg)
Questions?