enabling microservices @orbitz - dockercon 2015
TRANSCRIPT
Enabling Microservices @OrbitzSteve Hoffman @bacoboyRick Fast @tortiepointSenior Principal Engineers
Agenda
• Brief Architecture Overview/History
• From Monolithic to Services to Microservices with Docker
• Automated Pipelines
• Questions
#dockercon
#dockercon
2000
#dockercon
2000
Yes, we are STILL hiring!
#dockercon
2000
Yes, we are STILL hiring!
@OrbitzTalent
#dockercon
2000
Web Layer
Business Layer
#dockercon
2000
#dockercon
2000
#dockercon
2000
Business Layer
#dockercon
2000
Business Layer
#dockercon
Business Layer
2000
#dockercon
Business Layer
2003
#dockercon
Business Layer
2003
#dockercon
2003
Business Layer
#dockercon
2003
#dockercon
2004
#dockercon
2004
#dockercon
2004
#dockercon
2012
#dockercon
2012
#dockercon
2012
• Multiple Brands
• Websites
• Webservices
• Multiple Backends
• 500+ apps / thousands of instances
• Deployments Daily (sometimes more)
#dockercon
• Multiple Brands
• Websites
• Webservices
• Multiple Backends
• 500+ apps / thousands of instances
• Deployments Daily (sometimes more)
2015
#dockercon
Process Overkill
#dockercon
Different Provisioning Tools
DEV OPS
#dockercon
Different Provisioning Tools
DEV OPS
Application Platform!=
#dockercon
A New Experiment
• Microservices?
• Decompose single “service” into the actual 40+ sub-services - Any change to sub-service was a deployment of many
• Could it be a simple Spring Boot App in Docker (12 factor?)
• Backward compatible with existing service infrastructure.
• Code to Production w/o help from other Humans
#dockercon
The Docker Slave aka “The Rickbot”
#dockercon
The Docker Slave aka “The Rickbot”
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App Register & Lookup
#dockercon
10.10.10.1010.10.10.10:31002
App
App
App Register & Lookup
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
#dockercon
#dockercon
#dockercon
#dockercon
#dockercon
http://consul.io
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
10.10.10.10
App
App
App
#dockercon
10.10.10.10
10.10.10.10:51515
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
10.10.10.10
App
App
App
#dockercon
10.10.10.1010.10.10.10:1337
App
App
App
#dockercon
Almost done…
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
App
App
App
VIP
#dockercon
App
App
App
VIP
#dockercon
App
App
App
VIP
#dockercon
App
App
App
VIP
https://github.com/QubitProducts/bamboo
#dockercon
App
App
App
#dockercon
App
App
App
#dockercon
AppApp
App
#dockercon
AppApp
App
A Little More Background
#dockercon
#dockercon
Module
Module
Module Module
#dockercon
Editorial Module
• A Continuously Deployed Microservice
• www.orbitz.com - (scroll down)
#dockercon
OrbitzContent
Orchestration Service
Content (Solr)
Editorial Module
Search Module
Hotel Module
#dockercon
OrbitzContent
Orchestration Service
Content (Solr)
Editorial Module
Search Module
Hotel Module
Hotel Team Search Team Content Team
#dockercon
OrbitzContent
Orchestration Service
Content (Solr)
Editorial Module
Search Module
Hotel Module
#dockercon
OrbitzContent
Orchestration Service
Content (Solr)
Editorial Module
Search Module
Hotel Module
Hotel Team
Search Team
Content Team
#dockercon
Pre-Continuous Delivery
1.2-‐BETA-‐20150401-‐113002
Test Discard
Test Discard1.2-‐BETA-‐20150402-‐093002
Build
Build
And so on…
#dockercon
Pre-Continuous Delivery
1.2-‐BETA-‐20150401-‐113002
Test Discard
Test Discard1.2-‐BETA-‐20150402-‐093002
Build
Build
Test Deploy1.2
Build
Continuous Delivery with Jenkins, Docker, and Ansible (And then Marathon)
#dockercon
Yorfast-mbp:git rfast$ yo microservice
_-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___\ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y `
? Enter the name of your service. (E.G. "my-service")
#dockercon
Committer
Pull Request
#dockercon
Committer
Reviewer
Pull Request
#dockercon
Committer
Reviewer
Pull Request
#dockercon
Committer
Reviewer
Pull Request
Jenkins Pipeline
#dockercon
if(…) { x ++; }
Merge
#dockercon
Pipeline (Simplified)
build unit test publish
deploy dev
acceptance tests
deploy qa
deploy staging open RFC deploy
prod close RFC
#dockercon
Build
if(…) { x ++; }
1.2
Merge
#dockercon
Build
if(…) { x ++; }
1.2
Merge
1.2.
editorial-module.jar
./gradlew build
{BUILD_NUMBER}
#dockercon
Build
1.2
Merge
editorial-module.jar
if(…) { x ++; }
171.2.
#dockercon
Build
1.2
Merge
editorial-module.jar
FROM orbitz/java-8 ADD build/editorial-module.jar /opt/orbitz CWD /opt/orbitz CMD java -jar editorial-module.jar
if(…) { x ++; }
171.2.
#dockercon
Build
1.2
Merge
editorial-module.jar orbitz/editorial-module:1.2.17
if(…) { x ++; }
171.2.
#dockercon
Build
orbitz/editorial-module:1.2.17
#dockercon
Deploy
- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}}
git pull
playbook.yml (abridged)
#dockercon
host-001 host-002 host-003 host-004 host-005
- hosts: dev
#dockercon
Deploy
host-001
1.2.16
#dockercon
Deploy
1.2.16 1.2.17
host-001
#dockercon
1.2.16 1.2.17
host-001
Deploy
DATABASE_URL=jdbc://whatever/db/stuff CONSUL_HOST={{ansible_fqdn}} LOGSTASH_HOST={{ansible_fqdn}} GRAPHITE_HOST={{ansible_fqdn}}
#dockercon
Deploy
1.2.16 1.2.17
/health
host-001
#dockercon
Deploy
1.2.16 1.2.17
/health 200 OK
host-001
#dockercon
Deploy
1.2.16 1.2.17
host-001
#dockercon
Deploy
1.2.17
host-001
#dockercon
Deploy
host-002
1.2.16
And so on…host-001
1.2.17
#dockercon
What’s next?
#dockercon
How to handle failure?
#dockercon
What happens when a VM is moved?
#dockercon
What if I need to add capacity?
#dockercon
Deploy (Second Attempt)
- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}}
git pull
playbook.yml (abridged)
#dockercon
Deploy (Second Attempt)- hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image
git pull
playbook.yml (abridged)
- hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
#dockercon
Deploy (Second Attempt)
git pull
playbook.yml (abridged)
- hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
#dockercon
Deploy (Second Attempt)
New host setup
#dockercon
Deploy (Second Attempt)
Mesos Agent
New host setup
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
playbook.yml
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
- tasks: marathon: …
1.2.16 1.2.16
1.2.16
app = GET /v2/apps/editorial-module
if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if
while GET /v2/deployments contains deploy_id // still deploying end
// deploy complete
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
1.2.16 1.2.16
1.2.16
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
1.2.16 1.2.16
1.2.16
1.2.17 1.2.17
1.2.17
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
/health
1.2.16 1.2.16
1.2.16
1.2.17 1.2.17
1.2.17
#dockercon
Deploy (Second Attempt)
- tasks: marathon: …
/health
200 OK 200 OK
200 OK
1.2.16 1.2.16
1.2.16
1.2.17 1.2.17
1.2.17
#dockercon
- tasks: marathon: …
1.2.17 1.2.17
1.2.17
#dockercon
And off to the next environment…
1.2.17 1.2.17
1.2.17
#dockercon
1.2.17 1.2.17
1.2.17
What if?
#dockercon
1.2.17 1.2.17
#dockercon
1.2.17 1.2.17
#dockercon
1.2.17 1.2.17 1.2.17
#dockercon
/health
200 OK
1.2.17 1.2.17 1.2.17
#dockercon
1.2.17 1.2.17 1.2.17
#dockercon
Smoke/Acceptance Testing
1.2.17
1.2.17
1.2.17
#dockercon
Paper Trail
create
tickets
commit(s)
#dockercon
Paper Trail
fail!
close/fail
#dockercon
Paper Trail
ok
close
Build Unit Test
Deploy Dev
Deploy Prod
Deploy Staging
Acceptance TestCode
Review & Push
Production
Pre-Production
Open RFC
Close RFC
#dockercon
What’s inside the containers?
• Spring Boot
• Dropwizard Metrics (formerly Coda Hale, Yammer)
• Consul Registration/Discovery
• Logstash + Logback
• Swagger
• Hystrix
• Retrofit + Consul
OrbitzWorldwide/consul-client
#dockercon
–You
“Why didn’t you use {{.x}}?”
#dockercon
Keeping an eye on…
Kubernetes Amazon Elastic Container
Service
Docker Swarm
#dockercon
Summary
• Create a shared platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it.
• Take people out of the release process - Docker - repeatable applications
- Chef - repeatable infrastructure — Environment aware
- Jenkins - repeatable releases
• Delineate configuration concerns: - Known at Compile time — Bake into Docker image
- Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run
- Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
Thanks!Steve Hoffman
@bacoboy
Rick Fast
@tortiepoint