�

AW12 Concurrent�Session�11/13/2013�3:45�PM�

�����

"Test (and More) Patterns for Continuous Software Delivery"

���

Presented by:

Andy Singleton Assembla

���������

Brought�to�you�by:��

��

340�Corporate�Way,�Suite�300,�Orange�Park,�FL�32073�888Ͳ268Ͳ8770�ͼ�904Ͳ278Ͳ0524�ͼ�sqeinfo@sqe.com�ͼ�www.sqe.com

Andy Singleton Assembla

Andy Singleton is the founder of Assembla, a SaaS company providing tools for distributed agile development teams. Andy moved Assembla from bi-weekly releases to multiple daily releases. His forthcoming book Unblock! A Guide to the New Continuous Agile describes workflows for product management, task management, coding, and testing. Previously, Andy founded PowerSteering Software, a provider of enterprise project management software, and Cambridge Interactive, an eBusiness consultancy. He developed financial information systems for Reuters, Bloomberg, SNL Kagan, and Thomson. During his career, Andy has produced more than twenty major software and financial information products.

Be#er%So(ware%East%From%Andy%Singleton,%h#p://andysingleton.com%

www.assembla.com%%

In%this%session%!  IntroducAon%to%CD,%benefits%of%CD,%and%ways%to%pitch%it%! MulAple%code%and%test%pa#erns%to%choose%from%! How%to%get%the%automated%tests%you%need%!  Changes%in%the%role%of%QA%%

Program% Launch%

Releases%

Plan% Program% Test% Doc% Deploy%

Skip%Automate%&%Blend% Lag% Automate%

Pull%

CI%&%CD%

End%up%with%

Waterfall%to%ConAnuous%

Measure%

Assembla%in%2011%!  “Scrumban”%with%iteraAve%releases,%but%conAnuous%planning%to%accommodate%a%distributed%team.%

!  Releases%took%longer%as%system%got%bigger%and%there%was%more%to%test.%2%weeks%V>%3%weeks%

!  Bugs%in%producAon.%%2%days%for%fixes.%%Stressful)!  CompeAtors%achieved%faster%velocity%with%conAnuous%delivery%

! Made%a%study%of%conAnuous%methods%with%our%own%team,%customers,%and%tools.%

!  Now%–%releasing%about%250%Ames%per%month.%%Fewer%bugs.%%Much%less%stress.%

You%can%do%it%incrementally%1.  Release%more%frequently%2.  Improve%%%%%%%

When%to%use%ConAnuous%Delivery?%!  You%provide%an%online%service.%%CompeAAve%pressure%will%force%you%to%conAnuous:%Office%365%vs%Google%Docs.%

!  You%provide%any%service%with%so(ware%inside%!  Your%release%Ames%are%gedng%longer,%or%the%release%process%is%stressful%

!  You%are%developing%a%new%product%with%lean%startup%and%MVP%techniques%

!  You%have%a%big%project%with%a%lot%of%contributors%!  This%applies%to%hardware,%also%

Benefits%for%the%test%process%!  Eliminate%stress%from%the%release%process.%%There%is%never%a%big%batch%of%stuff%to%test%under%pressure.%%

! Add%or%remove%test%layers%to%increase%quality%or%increase%speed%

!  ConAnuous%integraAon%removes%the%stress%of%acquiring%and%fidng%components%from%many%teams%and%suppliers%

!  Your%CEO%wonders%why%the%compeAAon%is%releasing%faster%and%moving%faster%

%

ConAnuous%Delivery%Dial%

Dev

Business

Ops

DEV TEST UAT PROD

CONTINUOUS DELIVERY

RELEASE TRAIN

Self-Service

Controlled

INTEG STAGING

From%Steve%Brodie%and%Rohit%Jainendran%

The%big%quesAon:%How%to%test?%! We%release%so(ware%in%batches%so%that%we%can%test%it.%%That%is%the%whole%reason%for%doing%it.%%We%test%so(ware%%“release%candidates”%to%make%sure%everything%works%together.%

!  In%conAnuous%delivery,%we%might%get%as%li#le%at%10%minutes%to%test%a%release%candidate%

!  If%we%can’t%test%the%complete%version,%how%do%we%guarantee%quality?%%Is%it%magic?%

Test%Layering%–%A%Golden%Ticket%Monitor%your%released%so(ware:%Errors,%Usage%volume,%usage%pa#erns,%user%feedback%

QA%System%with%Human%test%consultants%

Code%review:%Both%a%manual%test,%and%a%place%to%ask%for%test%scripts.%

ConAnuous%integraAon:%Run%automated%tests%before%using%human%review%Ame%

Unit%tests%in%the%development%environment%

Switch%new%features%and%architecture%

Go%Both%Ways%

Veloc i ty )

Quality)

Increase)Quality)

Increase)Velocity)

Switch%and%Unveil%

Hidden%Programmer%sees%a%change%locally.%Change%is%tested%in%the%%main%version%but%not%seen.%

Test% Story%Owner%and%testers%see%the%change%on%test%systems.%

Beta%Insiders%see%it%and%use%it.%%Story%Owner%can%show%it%to%users%for%feedback%or%A/B%tesAng.%

UNVEIL!% The%big%event.%%Communicate%with%all%users.%%Measure%reacAon.%

One%code%version%

No%special%test%builds%

No%longVrunning%branches%

Code%and%Test%Pa#erns%!  Centralized%

!  TradiAonal%process%with%no%branches%finds%and%fixes%problems%as%early%as%possible%

! Distributed%!  Release%every%change%with%its%own%branch%and%test%

!  Temporary%branches%!  Efficient%review,%combines%benefits%of%centralized%and%distributed%

! MAXOS%!  Use%centralized%conAnuous%integraAon%to%manage%a%massively%scalable%IT%system%

Centralized%CI/CD%Contributor%Commits%–%“as%early%as%possible”%to%find%problems%

ConAnuous%IntegraAon%tests%

Fail%V%alarm%

Release%Candidate%Test%System% Release%

QA%TesAng%

Pass%

Distributed%ConAnuous%Delivery%Contributor%Commits%

Branch%or%Fork%

Deployed%version%

Peer%review%merge%requests%

Other%contribuAons%merged%and%released%“as%late%as%possible”%

QA%Consults%Pass%Final%Auto%Test?%

Merge%back%Current%

Deploy%

Distributed:%MulAple%Test%Systems%Contributor%1%

Contributor%2%

ProducAon%Revision%

Release%AnyAme%

CI%System%QA%Team%

Test%System%1%

Test%System%2%

Assembla%spins%up%test%servers%

Test%and%Review%on%Temp%Branches%

Contributor%Commits%

Auto%tests%

Mainline%

Review% Auto%tests% Review%

•  Github%style%–%Pull%requests%or%merge%requests%going%to%CI%system%•  Gerrit%style%–%Temporary%branches%for%each%contribuAon%•  Assembla%git%supports%both%–%merge%requests,%and%temp%branches%%created%%

by%a%push%to%a%protected%branch.%

Test%before%spending%human%Ame%on%review%

The%Services%Megatrend%Desktop Web App Cloud Services

App

DB

Service

Service

Service

Matrix%of%Services%V%MAXOS%

PrioriAzed%Backlog%

Current%Work%

Each%team%releases%when%ready%

Service%team%IntegraAon%test%env%

ProducAon%service%

Service%team%IntegraAon%test%env%

ProducAon%service%

Service%team%IntegraAon%test%env%

ProducAon%service%

Feedback%on%speed,%errors,%usage,%and%requests%

Organize%by%tesAng%as%one%system%

Coordinate%without%big%meeAngs%

PrioriAzed%Backlog%

Current%Work%

Service%team%IntegraAon%test%env%

Service%team%IntegraAon%test%env%

Service%team%IntegraAon%test%env%

ConAnuous%IntegraAon%between%%latest%dev%version%of%each%service%

•  ConAnuous%integraAon%helps%teams%coordinate.%

•  See%dependencies%between%“producers”%and%“consumers”%

•  Errors%and%conflicts%show%related%team%contact%info%

•  MeeAngs%and%changes%negoAated%between%two%teams,%not%many%

Role:%Developer%! Developers%have%more%power%and%responsibility.%! Developers%have%more%responsibility%for%tesAng.%! Developers%(not%QA%or%PM)%decide%when%to%release.%This)is)a)strong)finding.%!  IncenAves%are%correct.%%Developer%might%have%to%come%back%from%Friday%night%beers%to%fix%a%problem.%%This%provides%a%moAvaAon%to%make%good%decisions%and%automate%tesAng.%

!  Features%can%be%released%but%hidden.%%Product%Managers%and%Marketers%will%unveil%when%they%are%ready.%Unblock!%

Role:%QA%! QA%is%a%consultant.%! QA%gets%more%respect.%%Developers%have%to%ASK%for%service.%

! Developers%do%more%of%the%tesAng%work.%%They%should%organize%reviews%and%automated%tests%so%that%bugs%don’t%go%through%into%the%manual%test%process.%

! QA%has%more%Ame%to%invesAgate%usability%! QA%monitors%producAvity%and%quality%metrics%

IncenAves%for%ConAnuous%Flow%You%don’t%need%culture%change.%%You%only%need%to%release%more%frequently%1.  If%you%do%code)review,%you%can%get%automated.%No%

browbeaAng%and%cajoling%is%required.%%Developers%will%ask%for%tests%when%they%review.%

2.  If%developers%decide%to%release,%they%will%take%more%responsibility%for%tesAng%and%automated%tests.%

3.  If%you%release%more%frequently,%developers%will%quickly%learn%not%to%break%the%build%

4.  If%PM’s%take%unveil)and)measurement%responsibility,%they%will%make%be#er%features%

Our%Master%Plan%1.  Release%more%frequently%2.  Improve%%%%%%%

www.conAnuousagile.com/unblock%