Implementation of an Agile Processfor Multiple Teams Using SVN Dr. Alexander Schwartz

June 11, 2010

2Alex Schwartz

About ...

Alexander Schwartz20+ years in IT industry

„used to be developer“

Software DevelopmentDeveloper (C, C++, Java)Tech Lead, Project LeadConsultant, TrainerCertified Scrum Master

Release ManagerBuild ManagerAgile Configuration ManagerAgile Test Automation

mobile.international mobile.international belongs to eBay Main platform: mobile.de

•Biggest market place for automobiles in germany.

•One of the top-10 websites in germany; 49 million visitor per moth (IVW 02/2009)

Other marketplaces• RO: mobile.ro• PL: pl.mobile.eu• FR: ebay-automobile.fr• IT: ebay-automobile.it

3Alex Schwartz

Agile Configuration Mgmt

Outline

Lean ReleaseApproaches

Agile DevelopmentScrum, Kanban, Lean

Our experiences

with SVN

Agile Dev w/ Branches?

NO ! Our branching pattern

YES!

ReleaseMgmt Dev

Process

Dreilinden, April 30, 2010

4Oliver Lorenz

Agile Development & Configuration Management

5Alex Schwartz

Agile Process & “DONE”

potentially shippable product increment

It’s “DONE”.

6Alex Schwartz

Definition of Done (DoD)

Source: Gupta, Definition of Done: A Reference

7Alex Schwartz

Agile & Branching?

Agile Branching Approach #1Avoid the use of branches.

(Branches are not necessary since all features are done.)

DONE =

releasable

8Alex Schwartz

The Usual Quality Gap

Real World:

done means

„still a lot to do“

Ideal World:

DONE = releasable

?

Releasable

...

Feature tested

...

Unit tests green

Compiles

TestDevelopment

Mainline

Release Codeline

Releasable

...

Feature tested

...

Unit tests green

Compiles

MainlineRelease Codeline

TDevelopment

Test

►Progress of Quality ►Progress of Quality

►Process: Integrated Test Activities►Process: Waterfall

9Alex Schwartz

How to improve the common sence of DONE?

Strategy - Blue or the red pill?Many Aspects Process Tools Infrastructure Skills Mind set

• „QA as Quality Police“

• Developer: „I am not responsible for quality.“

As member of an agile team, I am responisble to deliver value to the customer in time and quality.

Strategy B:

Step by step

Strategy A:

Brute force

Action #1

Evangalize! (firm + carefully)

• Explain the need for an improved „done“ to all stakeholders. Convince first developers.

• Make tiny first steps for improvement + get commitment

• Clarify requirements for Release Management, etc. (Often, we overdo!)

10Alex Schwartz

Our Code & Architecture

Architecture:

Ebay like „pillar architecture“

= separate major use cases

• Java + Spring• MySQL + replication• Build with Maven• All code is built, packaged,

deployed + tested all together

• In numbers:– NLOC: ~ 1 million– #modules: ~ 120– #apps: ~ 50– #servers: ~ 500

11Alex Schwartz

Phase 1 of Our Scrum Transition

TestDevelopment

Mainline

Release Codeline

►CM: Mainline development

►Process: SCRUM with waterfall QA

►Progress of Quality (Done ...)

Part-time assignement

Test system

Experiences

We delivered. (Almost in time.)

Reduced external and internal quality + increased technical debt

Reduced test coverage.

Main impediment: low quality of trunk blocks many teams.

No one feels responsible for bugs.

Situation Q2 2008• parallel development with multiple

distributed teams• 50+ participants (25+ new/external)• 10+ teams of different size• New marketplaces + apps• One big codebase

12Alex Schwartz

Use branches for agile development?

Henrik Kniberg:

http://www.infoq.com/articles/agile-version-control

Yes, it can make sense to use branches in the context of agile development with multiple teams.

13Alex Schwartz

How we do it..

Quality Gate: releasableQuality Gate:

ready for trunk

(< releasable)

Action #2

New Policies

• Every project teams develops in a project branch.

• No commits in the mainline.

• Every contribution for the mainline (and the next release) has to satisfy the conditions of a Quality Gate.

Test TestDev

14Alex Schwartz

Phase 2 of Our Scrum Transition: Project Branches

Test

MainlineRelease Codeline

►CM: Project branches

►Process: SCRUM with small waterfall + big waterfall

►Progress of Quality (Done ...)

Experiences

Increased quality of TRUNK.

Increased release flexibility

Increased release reliability

Increased release „throuput“

Increased responsibility

Increased understanding of DONE

Increased integration of testers (some teams)

Small waterfall (some teams)

Merge overhead

Long R-test phase

Merge risks re-test in R-test phase

Still a QA bottleneck

Problem: completeness of test environments re-test in R-test phase

Part-time assignement

TestDev

15Alex Schwartz

Test

Phase 3 of Our Scrum Transition:

Mainline

Release Codeline

►CM: Project branches

►Process: SCRUM with small waterfall + big waterfall

►Progress of Quality (Done ...)

Experiences

More conditions of quality gates satisfied.

Decreased r-test phase length.

Increased understanding of DONE

Increased integration of testers

(most teams)

Too big contributions

Too low external quality

Merge bugs

TestDev

Action #3

• Reduced regression test phase lenght + team size.

• Testers are assigned to their project team, rather than to the R-test team.

• Be more strict: Verify the conditions of the Quality Gate.

16Alex Schwartz

What about Continuous Integration?

(a) Continuous publish of small changes for every active codeline.

Continuous Integration = 2 concepts

(b) Continuous build

... unit testing

... packaging

... verification

... deployment in test system

... deployment in production

Rule:

Use Cont. Build+Test etc.

for every active codeline.

Rules:• Integrate changes from

TRUNK often• Publish small chunks

17Alex Schwartz

What about Continuous Integration and Testing?

Standard: Use...• Metrics: test coverage + more

Advanced: Try to automate ...• Verification criteria for Quality Gates• Merges• CM tasks (creation of branches + CI build

configurations)• Deployment• DB Setup

Monitoring Code Metrics (sonar.codehaus.org, ...) Branches + Merges Test system status

Rule:

Use Cont. Build+Test etc.

for every active codeline. Challenges- Create + maintain necessary infrastructure

- Who is maintaining it?central team vs. project teams

18Alex Schwartz

Our Experiences with SVN

Overall: Possible?

Known tool Documentation Complexity of basic cmds Tool support around Browsing: FishEye

Constant time branch creation

Absence of bugs

Merging • Speed • Reliability • Correctness • Easy of use

Yes, but we almost failed

19Alex Schwartz

Our Problems (1)

• General– Broken working copies– Broken revisions on server– Server too slow– Incomplete updates

• SVN replication– Replication failures– Dublicated revisions

• FishEye– Too slow or too less content– Major changes– Not constant time bran– Indexing takes veryching– Hard to manipulate

20Alex Schwartz

Our Problems (2) -- Merging

• Too many!

• SVN 1.5..... was a catastrophy– Merge tracking was too buggy– Q: Why merge tracking was

implemented for non-root elements?

– No single merge was working out of the box!

• Path replacements / evil twins

• Reintegrate merges (diff merge) – often not correct (not a copy!)– Option „--reintegrate“ does not work

• Catch-up merges (change set based merge)– Often not correct– Convinience invocation does not work

21Alex Schwartz

Alternative Release Models – Our Release Cadence

Question:Can we do better?

22Alex Schwartz

The Lean Approach

• Focus on value stream

• Limit WIP (work in progress) ... since low WIP reduces cycle time• Continous Deployment: Release often (several times a day !!!)• Who is doing it?

• What about branching?

23Alex Schwartz

The Lean Alternative to Branching: Feature Flags

if (feature_C_enabled) {

if (feature_B_enabled) ..

• No branches at all!

24Alex Schwartz

References

Articles / Papers / Books Henrik Kniberg:

Version Control for Multiple Agile Teams Steve Berczuk. Robert Cowham, Brad Appleton:

An Agile Approach to Release Management Laura Wingerd, Christopher Seiwald:

High-level Best Practices in Software Configuration Management Mayank Gupta:

Definition of Done: A Reference, 2008 Brad Appleton, Steve Berczuk, Ralph Cabrera, Robert Orenstein:

Steamed Lines: Branching Patterns for Parallel Software Development Henrik Kniberg:

Scrum And XP from the Trenches Lisa Crispin and Janet Gregory:

Agile Testing Laura Wingerd:

The Flow of Change

Other Resources CM Crossroads

www.cmcrossroads.com

...