Dev Ops:the next generation

Warning!meme and star trek heavy

Who is this TREKKIE

Barney Hanlon

Technical Team Leader for Inviqa

@shrikeh

We’re hiring!

The story so far

(Yes I know that’s Star Wars)

Dev ops

DevOps

DevelopersSysadmins

“Get the code on the server”

FTP CVS

Subversion Git

What about all this OTHER stuff?

• Setting up Virtual hosts

• SSH access

• Log rotation

• Staging servers

• Build pipelines

• SSL certificates

• Minification

• User management

• File permissions

• Firewall rules

• Patching

• Concurrency

• Failover

• Alerting

Developers need to know more than how to push to

Github.

FIVE STAR DevOps

The Five Stars of DevOps

The Five Stars of DevOps

• Monitoring

The Five Stars of DevOps

• Monitoring

• Security

The Five Stars of DevOps

• Monitoring

• Security

• Performance

The Five Stars of DevOps

• Monitoring

• Security

• Performance

• Automation

The Five Stars of DevOps

• Monitoring

• Security

• Performance

• Automation

• Scaleability

Am I doing something in my application that is done better by the infrastructure or an external

service?

Probably.

Got root?

Common reasons to “just log onto the server quickly”

• server logs in /var/log require privileges to tail

• setting permissions on directories

• Processes require restarting

Monitoring

Monitoring

• Log EVERYTHING

• Drupal default visitor logging is heavy

• Should you be writing to the database to log visits?

NO.

Monitoring

• Logging is only one part of monitoring

• Send your Web logs to a remote service

• Set error_log to syslog in php.ini

Logging Services

• SplunkStorm

• Loggly

• Logentries

• Papertrailapp

HOSTING YOUR OWN LOGGING

• Splunk

• GrayLog2

• Sensu

• Munin

• Raven

Other monitoring

• Nagios

• Pingdom

• New Relic

• Piwik/Google Analytics

Profiling

• Don’t be afraid to turn XHProf on in live occasionally

• Regularly check your browser HAR

• Check APC and other caches for smells

Security

Where is the risk?

• Application security

• Infrastructure security

• End user security

Repelling Unwelcome Guests

Tools to help

• JumpCloud

• DuoSecurity

• Ubuntu ACL

Capturing Morpheus…

Not so bad.

Hardening SSL

Don’t bother hardening SSL

SSL is Dead, Long Live TLS

• No one should be using SSL any more.

• Transport Layer Security (TLS)

• Latest version 1.2

Vulnerabilities

• BEAST Attack

• CRIME Attack

• Lucky Thirteen

HTTPS without proper ciphers gives the illusion of security while providing none

ssl_protocols SSLv3 TLSv1; ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;

SSLProtocol all SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5

Nginx

Apache 2Supporting SSLv3 is only required for IE6

Open to Lucky Thirteen attack

Default SSL implementations

“Today, only TLS 1.2 with GCM suites offer fully robust security. All other suites suffer from one problem or another (e.g, RC4, Lucky 13, BEAST), but most are difficult to exploit in practice…”

–Ivan Ristic, Qualys

“…Because GCM suites are not yet widely supported, most communication today is carried out using one of the slightly flawed cipher suites. It is not possible to do better if you're running a public web site.”

–Ivan Ristic, Qualys

Diffie-Hellman Key Exchange

• Diffie-Hellman (DH) and Elliptic Curve Diffie-Hellman (ECDH)

• Allows Perfect Forward Secrecy

• Slow :(

server { … ! add_header Strict-Transport-Security “max-age=31536000; includeSubDomains"; ssl_session_cache shared:SSL:10m; ssl_prefer_server_ciphers on; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS; ssl_ecdh_curve secp521r1; }

https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/

https://www.ssllabs.com/ssltest/

Test your strength!

–No user ever

“I don’t mind that the site is slow, at least my data is safe.”

Performance

SPDY

• Draft HTTP 2.0

• Allows multiplexing a single connection

• Requires HTTPS

• Do you need HTTP at all?

SPDY

PageSpeed

• Library for Apache and Nginx

• Automatic minification of JavaScript, CSS and HTML

• On-the-fly optimisations based on chosen filters

The “SPDY Sandwich”

Nginx Varnish Nginx PHP-FPM

The “SPDY Sandwich”

Nginx Varnish Nginx PHP-FPM

Responsible for: • SPDY / SSL Termination • Serving static assets • Gzipping • Pagespeed is user agent-aware

The “SPDY Sandwich”

Nginx Varnish Nginx PHP-FPM

Responsible for: • Caching dynamic pages • Cookie normalisation

The “SPDY Sandwich”

Nginx Varnish Nginx PHP-FPM

Responsible for: • Serving dynamic pages • Generic Pagespeed optimisations

The “SPDY Sandwich”

Nginx Varnish Nginx PHP-FPM

Responsible for: • PHP interpreter

Cookies

Cross Site Request Forgery (CSRF)

• OWASP recommendation

• Requires a token in the form and a session token

• Breaks most reverse proxies without configuration

Am I doing something in my application that is done better by the infrastructure or an external

service?

OpenResty

• Nginx bundle

• Has modules for connecting to Redis, Drizzle, memcached and many more

• Has Lua to allow pre and post processing on requests and responses

Time for a Simplified Example

<?php !namespace Inviqa\DrupalCamp\Access; !class OpenRestyTokenGenerator implements CsrfTokenGeneratorInterface { private $token; ! public function __construct($csrfToken) { $this->token = $csrfToken; } ! public function get($value = '') { return $this->token; } }

<?php namespace Inviqa\DrupalCamp\Form; !use \Drupal\Core\Form\FormBuilder as CoreFormBuilder; !class FormBuilder extends CoreFormBuilder { public function __construct( ModuleHandlerInterface $module_handler, KeyValueExpirableFactoryInterface$key_value_expirable_factory, EventDispatcherInterface $event_dispatcher, UrlGeneratorInterface $url_generator, TranslationInterface $translation_manager, CsrfTokenGeneratorInterface $csrf_token = NULL, HttpKernel $http_kernel = NULL ) { …

OpenResty Varnish Nginx PHP-FPM

X-CSRF-Tokenize: “[[CSRF Here]]”

Redis

FIXING CSRF WITH LUA

OpenResty Varnish Nginx PHP-FPM

Header cached in Varnish

Redis

FIXING CSRF WITH LUA

OpenResty Varnish Nginx PHP-FPM

Redis

• Parses response (regex)!• Finds token placeholder!• Replaces with real token

FIXING CSRF WITH LUA

! ! header_filter_by_lua!! ! '!! ! if ngx.var.upstream_http_x_csrf_tokenize then!! ! ! -- the backend requested a CSRF token be set!! ! ! local csrf_cookie_token = nil!! ! ! if ngx.var.cookie_csrf then!! ! ! ! -- they have a cookie, just re-use it!! ! ! ! local csrf_cookie_token = ngx.var.cookie_csrf!! ! ! end!!! ! ! local resty_random = require "resty.random"!! ! ! local str = require "resty.string"!!! ! ! if not csrf_cookie_token then!! ! ! ! -- no valid csrf cookie found, let us make one!! ! ! ! !! ! ! ! local cookie_random = resty_random.bytes(16,true)!!! ! ! ! while cookie_random == nil do!! ! ! ! ! -- attempt to generate 16 bytes of!! !! ! -- cryptographically strong (enough) random data!! !! ! cookie_random = resty_random.bytes(16,true)!! ! ! ! end!!! ! ! ! ngx.var.csrf_cookie_token = str.to_hex(cookie_random)!

OpenResty Varnish Nginx PHP-FPM

Redis

• Stores HMAC as value in Redis with random key “csrf_”

• Generates cookie with Redis key as value

FIXING CSRF WITH LUA

!! ! -- we are about to mess around with the content of the page!! ! -- so we need to clear this as it will be wrong!! ! ngx.header.Content_Length = ""! ! ! !! ! -- set the Cookie for the CSRF token!! ! ngx.header.Set_Cookie = "csrf=" .. ngx.var.csrf_cookie_token!! ! ngx.header.tokenize = ngx.var.upstream_http_x_csrf_tokenize!!! ! -- now generate one for the form token!! ! while form_random == nil do!! !! form_random = resty_random.bytes(16,true)!! ! end!!! ! ngx.var.csrf_form_token = str.to_hex(form_random)!!! ! local redis = require "redis"!! ! local client = redis.connect("127.0.0.1", 6379)! !! ! client:set("csrf_" .. ngx.var.csrf_cookie_token, ngx.var.csrf_form_token)!! ! end!

OpenResty Varnish Nginx PHP-FPM

Redis

FIXING CSRF WITH LUA

Check token on way back in

location @backend {!! ! # You can't set variables in nginx dynamically, ! # so set this up as empty first!! ! set $csrf_validate "";!! ! access_by_lua !! ! '! if ngx.req.get_method() == "POST" then! !-- set up forbidden as default! !ngx.var.csrf_validate = ngx.HTTP_FORBIDDEN! !if ngx.var.cookie_csrf then!! ! ! ! local res = ngx.location.capture("/validate-csrf")!! ! ! ! if ngx.HTTP_OK == res.status then!! ! ! ! ngx.req.read_body()!! !! ! local args = ngx.req.get_post_args()!! !! ! local posted_token = tostring(args["csrf"])!! !! ! if posted_token == res.body then!! !! ! ! ngx.var.csrf_validate = ngx.HTTP_OK!! !! ! end!! !! end!! !end! !end! !';

Full gist:https://gist.github.com/shrikeh/4722427

Automation

Tooling

• Bash scripts (!)

• Chef/Puppet (retro)

• Ansible!

Ansible

• Requires no agent!

• Pure SSH

• Modules

• YAML-based configuration

• Playbooks (and playbooks of playbooks)

Ansible PLAYBOOK!- name: ensure SSH key exists digital_ocean: > state=present command=ssh name=case - name: ensure droplet exists digital_ocean: > state=present ssh_key_ids=57705 name={{ inventory_hostname }} size_id=66 region_id=4 image_id=1505699 wait_timeout=500 private_networking=yes virtio=yes wait=yes unique_name=yes wait_timeout=500 register: launched - debug: msg="IP is {{ launched.droplet.id }}" - debug: msg="IP is {{ launched.droplet.ip_address }}"

Ansible

ansible-playbook base.yml -vvv -i "hosts/production"

Configuring Your Application

<?php # /sites/default/settings.php ... $databases['default']['default'] = array( 'driver' => 'mysql', 'database' => 'drupal', 'username' => 'testuser', 'password' => '123secure', 'host' => 'localhost', 'prefix' => '', );

–The Twelve Factor App

“A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open

source at any moment, without compromising any credentials”

“The twelve-factor app stores config in environment variables (often shortened to

env vars or env). Env vars are easy to change between deploys without changing

any code” –The Twelve Factor App

env[db_name] = drupal_live env[db_host] = 192.168.0.2 env[db_user] = liveuser env[db_pass] = verysecurepass env[db_prefix] = drupalcamp_

Put your variables in PHP-FPM

/etc/php/fpm/pools/live.conf

<?php # /sites/default/settings.php ... $databases['default']['default'] = array( 'driver' => 'mysql', 'database' => getenv('db_name'), 'username' => getenv('db_user'), 'password' => getenv('db_pass'), 'host' => getenv('db_host'), 'prefix' => getenv('db_prefix'), );

Provisioning

• Idempotent deployments

• Provision every environment the same way

• Resist the urge to do something manually

• Get into a workflow of automation

Docker

Docker - AN OVERVIEW

• Lightweight Linux Container

• Portable environment

• Install all your PECL dependencies into a container

• Ship it

Problems

Problems

!

• Still heavily in development, no “right way” yet

• Hard to set up syslog inside a container

• Runs as root on the box

That’s being fixed though

We’re Done!

With thanks to Paramount Pictures

and startrek.wikia.com for not suing me

Questions

Thank You!

My first official talk!• Special thanks to Lorna Mitchell, Ian

Barber and Rowan Merewood for all the coaching

• All feedback welcome!