devops: building by feature with immutable infrastructure at serv.sg
TRANSCRIPT
How we build features at Serv.sg
in a Pandan Leaf...
November 25th 2015
Nicolas [email protected]
https://www.linkedin.com/in/nicolasmashttps://github.com/nicolasmas
Acknowledgment
“Alone we can do so little; together we can do so much”
Sandeep Isac Marco Yang Alex Hayden
Alexander Bulat Andrei Ping Yang
What this talk won’t do
⬗ Entertain you with funny
pictures (sorry). This will be
the only one.
⬗ Describe the tools being
used, unless there is a
specific request (let’s keep
the focus).
⬗ Guarantee that Murphy’s law
won’t impact the demo.
So what is it going to be?
⬗ What do we do at Serv.sg?
⬗ Our (current) technology stack
⬗ Feature build (+ DEMO)
⬗ What is the problem we solved?
⬗ Limits
⬗ Enhancements
⬗ B Tracks: if we really have time
Disclaimer: I am a DevOps padawan, what is described tonight is probably not the best solution ever. Any suggestion is welcome~
What we do at Serv.sg
✓ Real time Mobile workforce location tracking✓ Automated scheduling of service appointments✓ Automated email / SMS reminders of upcoming appointments and contracts.✓ Manage customer data and all history of service appointments✓ Manage Services offered, including recurring contracts and packages, etc.✓ Advanced reports on employee utilization as well as business
Technologies (Stack… for now)
Key backend technology
Data storage
Key frontend technology
Technologies (DevOps)
For now… (bis)
Overview
JIRA-456
JIRA-456.serv.sgJIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
On “New Branch”
JOB Monitor Branches
Overview
On “Branch Deleted”
JIRA-456
JIRA-456.serv.sgJIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
JOB Monitor Branches
It all starts in Jira
⬗ The developer takes ownership of, let’s say the item “JIRA-456” during a
sprint.
⬘ Actually, he does not do it in JIRA, but from github by branching out the
codebase.
⬘ The Branch HAS TO BE named after the jira item, e.g. JIRA-456
It continues on Slack
⬗ The moment the branch is created, we can actually build the
infrastructure:⬘ In slack: “/serv-branch-build”
⬗ Slack is asking a Jenkins job to monitor the github repository for
new/deleted branches
What happens in the jenkins monitoring job
⬗ The Jenkins job will do a few things:⬘ 1. It will connect to github and get all the branches
⬘ 2. It will detect our new branch
⬘ 3. A custom jenkins job will be created for that branch
⬘ Terraform will be invocated
⬗ It will also keep you updated of the progress in Slack
The Infra is built...
...But not yet provisioned !!
Note: we use a pre-baked OS image for the instances
Building the branch
⬗ In Jenkins
Provisioning the branch
⬗ Ansible playbook⬘ Site.yml dynamically set
⬗ Roles⬘ Hostname set
⬘ Supervisord
⬘ Yum additional packages
⬘ Web conf
⬘ DB conf
⬘ DB Load
⬘ (...)
End Result
Destroying the build
⬗ The features has been validated or ditched, the branch is either merged
or destroyed on github
Note: If we need a few back and forth between the product and the dev for the feature, it’s just a matter of running the jenkins branch job to deploy again
Back in Slack
⬗ “/serv-branch-build”
⬗ AWS quick check.. the instances are terminated
BTW, What problem are we solving?
⬗ Deploying by feature is not new. Immutable infra seems to be less
common. The two combined? Anyone doing it?
⬗ From a business/product perspective⬘ More responsive to bug fixing, features deliveries
⬘ No confusion on what has been deployed
⬘ The branch can be hotfixed => one step closer to CD
⬗ From a technical perspective⬘ Leveraging the cloud infrastructure, programmatically
⬘ No environment recycling or reconfiguring with the related uncertainty
⬘ As close to production as we can
⬘ A good way to craft baked OS with the latest update and test it
Limits/Challenges
⬗ Hard to keep track of the full cycle, from the OS baking to the feature
deployed:
⬘ Packer: need to manually describe the target provisioned OS. Hard to test,
need to manage the produced artifacts
⬘ Ansible: need to manually write the roles, hard to test and a pain to
maintain
⬘ Terraform: need to manually describe the infra, hard to test, painful to
maintain and evolve. Some bugs with route 53
⬘ Dependency on Jenkins
⬘ A fair amount of templating (jinja2 style) with a risk induced by the dynamic
aspect of the build (hostnames, EIP...)
⬗ Many pieces in movement, dependency
⬗ Need to be in sync with the product team to check and act upon the
feature as soon as it is available. We don’t want a stack running for ages..
Enhancement
⬗ Use a tool to orchestrate the branch + infra build aspect (Spinnaker.io?
StackStorm.io?) or build a simple micro service to do it
⬗ Choose the branch to build in slack instead of all the branches with the
pattern “JIRA-XYZ”
⬗ Improve the provisioning and better manage the ansible playbook
(Tower.io?)
⬗ How is this compatible with our next architecture steps? (micro services)
Bonus Track - Did you know ?
“The mother of all demos” http://en.wikipedia.
org/wiki/The_Mother_of_All_Demos
⬗ Douglas Engelbart's December 9, 1968
⬗ 90-minute live public demonstration of the online system, NLS, they had
been working on since 1962
⬗ 1000 attendees and a demo of almost all features we use today⬘ Mouse
⬘ hypertext, object addressing and dynamic file linking
⬘ shared-screen collaboration involving two persons at different sites
communicating over a network with audio and video interface.
http://web.stanford.
edu/dept/SUL/library/extra4/sloan/mousesite/1968Demo.html
https://www.youtube.com/playlist?list=PL76DBC8D6718B8FD3
Watch it !!
Bye Bye !