kanban as code: the continuous delivery at lesfurets.com by dimitri baeli

Kanban as CodeCode level impact of a Lean mindset

LesFurets.com

http://LesFurets.com

@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli

Insurance aggregator

5 Products

2,5M quotes per year

Paid by contracts initiated

Our product


The engine

• Complex forms

• 40-200 Fields (Expedia is around 10 fields)

• Risk pricing engine

• 50+ external webservices called in real time

• Collect answers within seconds

• 2B+ fields mapping per year

• Embedded online subscription moduleLesFuretsWebSite

Panel Partners

Data Partners


Competitive Advantage(s)

Product Catalogue :Number of Panel members

Speed of getting new Panel membersAcquisition price compared to google, …

Quality of the offers presented to the usersQuality of the clients presented to the partners

UX of the WebSite :Speed of getting a proposal, clarity of the proposalAvoid mistakes along the form filling


The IT Team

• 23 Developers & 2 Ops

• 2 Functional Teams & 1 Infrastructure & Team

• 1 Single code base of 500.000 lines

• 40.000 unit tests (low level automated tests)

• 200 selenium tests (browser based automated tests)

•1-3 release per day of 1-10 changes

• 1 branch per feature : released when ready

• Soon known as “Kanban as code”

• 20-30 branches in parallel, 1-10 released per day

@beatiefurets

github.com/lesfurets

http://github.com/lesfurets


500.000 Lines of Code (LoC) that’s :

7 dictionaries of 500 pages (140 lines / page)

2 times the 7 Harry Poter volumes (3400 pages)

A new version daily5-10 changes by 25 devs

Intermezzo : 500.000 line of code ?

x 7x 2

Why daily releases ?What read and understood


Manifeste agile

Principe #1

« Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. »

http://agilemanifesto.org/principles.html

http://agilemanifesto.org/principles.html


Food for thought : back to the future


Reading survivor kit : save your life


Minimal Viable : what you can’t ignore


Release early, release often

http://paulhammant.com/2013/03/13/facebook-tbd-take-2/

http://paulhammant.com/2013/03/13/facebook-tbd-take-2/


Learning #1 : Release early, release often

• To reduce the Time to Market :• Release small batches of features

• Release often

• Releasing often• Change the way source code is managed

• Change in the QA strategy (Dev QA their own changes)

• Test tooling is provisioned for the des

• Fast Feedbacks as a gift


Continuous Delivery : 5 Golden Rules

1. Fast build

2. Unbreakable build

3. Easy and automated deployments

4. Runtime Monitoring and alerting

5. Root cause analysis of incidents


Git / Git Flow / Github Flow

Powerful branching model

http://nvie.com/posts/a-successful-git-branching-model/

Master

Branch

Pull Request

Github

http://nvie.com/posts/a-successful-git-branching-model/

Code management in 2017The minimum to know, no excuse !

@beastiefurets@dbaeli

Original code:

master


Change 1 : Lean IT


Change 2 : Beastie Furets


Change 3 : Hello Beastie


A commit


master

Commit 1 : Lean IT

commit 1


A branch (copy)


Change 2 = commit 2

commit 2

master commit 1


Let’s merge the branch


master

Change 1 & Change 2 : combined (merged)

commit 1 merged

commit 2


A conflict of 2 branches


Commit 3 : Hello Beastie

master

commit 3


master

Commit 2 : Hello Lean IT

commit 2


???????

Change 3 : Hello Beastie

master

conflictcommit 3

commit 2


Minimum viable vocabulary (1/2)

•Commit: • is a change within the code base

• described as a “diff” from original (or patch)

• History :• Sequence of Commits (changes)

• Branch:• A full copy of the code base at a given time

• Can be seen as an alternate history

• Merge:• Replay commits of a branch on another

branch 1

branch

commit 1

commit 2

commit 4

commit

merge

commit 3


Minimum viable vocabulary (2/2)

• Conflict: • when 2 commits aren’t compatible

• from 2 branches

• Resolution :• give a solution to the conflict

• Rebase :• change the start point of a branch

• time travel (get into the future)

branch 1

branch 2 commit 2

commit a

conflict

commit 3

commit 2

commit a

commit 3

rebase

Release more often - Step 1Monthly releases (2012)


Plan / estimate / code / test / release monthly

Sprints style 12 releases in a year

2012 : Monthly release


branch (prod)

trunk trunk

3 - 4 semaines 1 - 4 jours5 jours

Release

Code Freeze

branch


Storie 1

branch (prod)

trunk trunk

Release

Code Freeze

branch



Storie 1

branch (prod)

trunk trunk

Storie 2

Release

Code Freeze

branch



Storie 1

branch (prod)

trunk trunk

Storie 2

Release

Code Freeze

branch


Risk of conflicts


Agile Team of 12, SCRUM by the book

•Monthly sprints, planning, coding, testing, demo, release

• 1 Week of testing (code review, manual testing, bug fixes)

• Build Time : 15 minutes• Regression Testing : 1-2 hour

•Blockers : • Build time• Regression Testing• A Week of manual testing

2012 : Monthly releases

Release more often - Step 2Weekly release (2013)


2013 : Weekly release

Planifier / estimer / coder / tester Release weekly

50 releases per year


branch (prod)

trunk trunk


Release

Code Freeze

branch


branch (prod)

trunk trunk

Release

Code Freeze

branch


Storie 1


Storie 1

branch (prod)

trunk trunk

Storie 2

Release

Code Freeze

branch



Storie 1

branch (prod)

trunk trunk

Storie 2

Release

Code Freeze

branch

3 - 4 weeks 1 - 4 days5 days

1 week


• Keep Scrum monthly planning with Weekly releases

•Changes:• Use the patch process to deliver features (cherry pick)

• Build down to 3 minutes (vs 15 minutes)

•Blockers:• Regression Testing + Recette• Code level management (release branch)• Painful weekly patch release

2013 : Getting Weekly


Learning #2 : Patch process as release process

• When something goes wrongs on the live system

•Do you have a strong process to release a fix ?

• NO => you’re living dangerously => get one• YES => why don’t you use it for usual releases ?

•Anyway :• Get a robust and fast delivery process is not an option !

Release more often - Step 3Daily release (2014)


2014: Daily Releases

Ship what is ready … every day !

Marathon style

250 releases per year (already 50 in 2017 !)


master

trunk

1 day 1 day

Model 3: Kanban as code

Storie 1

Storie 2Storie 3


master

trunk

1 day 1 day

Storie 1



master

trunk

1 day 1 day

Storie 1


Is this branch ready ?


master

trunk

1 day 1 day

Storie 1


merge & test

Merge the branch and test


master

trunk

1 day 1 day

Storie 1

merge & test & drop


Merge the branches and test


master

trunk

1 day 1 day

Storie 1

Storie 2

merge & test




master

trunk

1 day 1 day

Storie 1

Storie 2Storie 3

merge & test




Keep the branches independentAvoid the conflict


git-octopus by lesfurets (on GitHub)


master

trunk

1 day 1 day

Storie 1

Storie 2Storie 3



ticket1

ticket2

ticket3

ticket4

ticket5

features releaseslocal

ticket3

master

ticket3

ticket1

master

octopus-features

octopus-releases

few minutes 1 jour à 1 mois 1 - 2 jours

Validation

0 - 2 jours

release


Learning #4 : Don’t block the others

• We keep the features separated until the last moment

• Other teams are merging by default !!!

• Continuous Merge (as test tooling):

• Detect conflicts with others (without requiring you to get their code)

• Allow you to avoid conflits

• Allow you to drop your code temporarely

•Bonus:

• Avoid unfinished code to be used by the others


Break release branches, release from trunk

• Improvments:

• Automated release from finished development

• Per feature code review & demo

• 15 minutes regression testing (testing architecture)

• Release by 1 dev alone

• Monitoring & Root cause analysis of issues

2014: Daily Releases


Learnings : Kanban as Code

#1 : Release early, release often #2 : Patch process as main release process #3 : Avoid merge & conflicts #4 : Don’t block the others


Take away : Kanban as Code

The Goals:

• # A Flow of Features at code level (Branches)

• # Release what is ready when it’s ready (Start by finishing)

• # Branches are your Kanban card (Step 1 of a pull system)

Why does it work :

• # Enforce the features to be independent (Really Independent !)

• # Low risk of conflict on 500.000 Line of Code (daily)

• # No branch can block another one (or it the same)


Beastie Furets (LesFurets IT Team)

@beastiefurets

beastie.lesfurets.com

lesfurets

http://beastie.lesfurets.com

@dbaeli@beastiefurets

MERCI !

69

LesFurets.com LeanKanban.fr29, 30 Novembre 2017 www.leankanban.fr

http://lesfurets.com

http://leankanban.fr

http://www.leankanban.fr


Avoid the conflict


Change 3 : Original (conflict with Change 2)

master

commit 3


Change 3bis : New version (avoid conflict)

master

commit 3

commit 2


Commit 1, 2: Easy merge

master

commit 3

commit 2

resolved

commit 1


Change 1, 2 & 3bis : Merged

master merged

commit 3

commit 2commit 1


Learning #3: Avoid merge & conflicts

• For 2 Features developed separately:

• Merge = enforce to release together

• But Merging new code is the default for many teams

• Avoid merges (or release immediately):

• Trunk Based Development

• Any alternatives ?

kanban as code: the continuous delivery at lesfurets.com by dimitri baeli

Technology