1

How Automation Reveal Technical Debt

2

Eric’s Bio

I’m a Lead Consultant at Urbancode where I helps customers get the most out of their build, deploy and release processes. I have 9 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer. I’ve been at the forefront of CI & CD for 7+ years

Eric Minickeric@urbancode.com@EricMinick

3

Technical Debt

4

Why do we accumulate technical debt?

• We leverage technical debt to deliver more faster.

• De-leveraging is rarely accounted for in project planning.

• Green-Shifting*

Scope Time

Resources

* http://www.drdobbs.com/191600661

5

Why should we care?

Baggage that slows the team• Lack of automated tests

lengthen QA cycles

• Fear of merging work

• Unrefactored code slow to work with

• Slow build / deploy processes delay learning and release pace

Quality issues• Lack of tests results in

buggier code

• Releases are error prone and lead to unnecessary outages

6

The limits of what we know

• Known Knowns: Bugs confirmed and tracked

• Known Unknowns: Undiscovered bugs

• Unknown unknowns: One of our project teams is using a GPL’d library making their product impossible to ship

7

Where Automation Helps

• “Testing” for debt: Automated tests, code scans and reports can help identify (and quantify) problems.

• Automation as a learning experience: The act of automating brings surprises.

8

Testing for Debt

• Continuous Integration (multi-component)

• Code Inspection

• Watching Trends

9

Testing for Debt: Continuous Integration

• On code commit, build and test the software

• Roll up changes to build-time or runtime dependencies and test those too to identify API incompatibilities

10

Testing for Debt: Code Inspection

Code Reviews – Rapidly detect issues

Static Code Analysis – Tool based checks for bugs, code duplication, code theft, & non-compliance with dev standards.

11

Testing for Debt: Code Inspection

Code Reviews – Rapidly detect issues

Static Code Analysis – Tool based checks for bugs, code duplication, code theft, & non-compliance with dev standards.

12

Testing for Debt: Watching Trends

13

More visualizations: Sonar

http://nemo.sonarsource.org/dashboard/index/327690?did=6

14

An example safety net

• Continuous build & unit test

• Nightly slow tests / code scans

• Emails identifying new issues – ideally tied against source code changes

• Regular inspection of trends

• Bugs / Stories entered around issues

15

Automation as a learning experience

Implementing the safety net helps us discover unacknowledged debt

16

Tests? What tests?

17

Automation Examples: The Build

• Build time dependencies not understood

• Build scripts missing or incomplete

• “Magic build server” anti-pattern

http://www.urbancode.com/html/resources/webinars/Role_of_Binary_repositories_in_Software_Configuration_Management.html

18

Automation Examples: Deployment

• Deployment scripts scarce

• “Special Instructions” with most deployments indicate non-repeatable process

• Environmental differences handled poorly

• Separation of duties less than real

19

Automation Example: Test Automation

• Requirements less well understood

• Existing tests are few, stale, un-optimized

• Application not architected for testability

20

Expect the unexpected when automating

• At scale, Green-Shifting, has hidden issues

• Include these discoveries in ROI estimations for automation (positively and negatively)

• This is a happy side effect

21

Direct and Indirect Automation Benefits

• Direct: We tested for problems and found them.

• Indirect: In attempting to be more efficient, we automated, and accidently discovered problems.

22

Automating for the team

• Provide a “safety net” to detect and recognize issues.

• Diagnose and repair lack of repeatability in build-deploy-test

• Quantify accumulating debt in support of fighting scope creep

• Team level tooling is fine

23

Automating the Enterprise

• Benefits extend beyond aggregate team level benefits

• Central Automation and Reporting gets us:– Identify who can use shared configuration– Who has tests, who doesn’t– Who is using what tools– Build / deploy failure rates

24

Favorite Examples: Deployment Failures

• Problem: High rate of deployment failures a problem

• Goal: Reduce failure rate from 40% to 5%

• Approach: Avoid manually fixing a deployment. Fix the automation and redeploy.

• Enforcement: Monthly / weekly spreadsheet of success to CIO with a six month plan.

25

Favorite Examples: “End of Day” Commits

• Problem: Developers commit breaking changes at the end of the day

• Goal: Avoid other devs staying late to clean up

• Approach: Report on failures, and react to negative patterns as a team.

26

27

Favorite Examples: Low numbers of tests

• Problem: Unit testing discipline breaking down over time

• Goal: Maintain high or improving coverage

• Approach: Standard coverage tools and an emphasis on upward trends.

• Enforcement: Trending report on monitor over CIO’s door

28

Summary

• We accumulate technical debt as we race to deliver more, faster.

• This causes us to eventually release less, slower, with worse quality

• Automation directly and indirectly helps us find issues.

• There are benefits at both team and enterprise levels.

29

More References

http://urbancode.com/resources

• Enterprise CD Maturity Model• Lean Build & Deployment Automation• Managing Release Risks with Metrics

Blogs.urbancode.comTwitter.com/UrbanCodeSoftFacbebook.com/UrbanCodeSoftSlideshare.net/Urbancode

30

Yes, we sell products for this

• AnthillPro– Build automation and build promotion

• uDeploy– Deployment and release management

31

Questions?

eric@urbancode.com@EricMinick

Slideshare.net/Urbancode