continuous testing

Continuous Testing

J a r e d R i c h a r d s o n A g i l e A r t i s a n s

@ J a r e d R i c h a r d s o n

1

Who am I?

2

Jared Richardson

CoAuthored Ship It! and Career 2.0

Screencast editor for PragProg.com

Started AgileRTP in 2007

2nd public signatory of the Agile Manifesto

Agile coach

AgileArtisans.com

3

Who are you?

4

What would you say you do here?

Roles?

Responsibilities?

Why are you here?

What are your tests for this session?

5

Objectives

Understand the Continuous Paradigm

See how it applies to your teams

Understand the basics of Jenkins

Understand an end-to-end Continuous workflow

Collect practical tips and tricks

6

Outline

Continuous paradigm

Continuous Integration (benefits & workflow)

Introduction to Jenkins

Continuous Testing (benefits & workflow)

Jenkins as a continuous deployment/testing tool

Integration testing tool

7

Breaks

Half-way (90 minutes)?

Each hour ?

8

ContinuousParadigm

9

What Is Software?

10

Feedback Loops

11

What Is Agile?

12

AgileManifesto.org

13

Feedback Loops

14

flickr.com/photos/bobowen/4501875907

flickr.com/photos/30691679@N07/3808395213

flickr.com/photos/kevino/1438502858

15

Feedback Loops are Fractal

flickr.com/photos/dancentury/4579906149

16

Pairing

Test First

Daily Meetings

Demos

???

Feedback Loops Inside Feedback Loops

17

Agile Is...

Tighter Feedback Loops

18

Minimize Dark Times

Code Change

Validation http://flickr.com/photos/bogenfreund/556656621/

19

Fast feedback leads to fast fixes

20

Different Levels

Good at the bottom

Great at the top

Ignore the middle

21

Continuous Feedback

Continuous Integration

Continuous Delivery

Continuous Testing

22

ExerciseHow long is your SDLC?

What are your SDLC feedback bottlenecks?

How long do you have to wait for feedback?

Write down 3 bottlenecks

Gather into groups of 5 to 7

Share and discuss (solved problems?)

Turn in your top 5 items (for the group)

Take ten minutes

23


24


Watch Code

Build product

Run tests

Publish results

25

Pros

Fast feedback Known state Easily consumed history Shines a light

26

26

ConsSlows you down Nags you after every code change Exposes problems Shares issues Forces you to deal with problems Smoke alarm

27flickr.com/photos/117168856@N06/12486837493

27

28

Work

Time

Perception of Problem Accumulation

28

29

Work

Time

Actual Problem Accumulation

29

30

Work

Time

Continuous Problem Resolution

30

GPSAtlas Maps Printed maps! Navigation software What’s better?

31

31

Feedback Loops

32

StepsUse source code management (SCM) Automate the build (script) Acquire a build box (isolated) Install a build environment (dev tools) !

33

33

Java Tooling

Publish results

34

Continuous Integration Servers

Many Jenkins Cruise Control Bamboo Pulse TFS

35

35

JenkinsOpen source Established Split from Hudson Java application WAR file Runs on any application server Self contained XML & text

36

36

Example

Publish results

javac (Oracle & IBM) message broker etc

ws ejb deploy message broker tools sshexec (to scripts)

JUnit HP's UFT Selenium

Web page RSS Email

37

Dependent Builds

Data Layer

Services

Client/GUI

38

Dependent Builds

Data Layer

Data Services

Rich Client

Common Libs

Web Portal

Biz Services

39

Chained Continuous Integration

Team 1 Team 2 Team 3

Integration build

40

Platform Testing

Team 1 Team 3Team 2 Team 4 Team 5

Integration Build

Platform1 Plat 3Plat 2 Plat 4 Plat 5

41

DemoSample project Build script Check into SCM Add to Jenkins Edit/Commit/Repeat

42

42

Continuous Testing

43


Watch Code

Build product

Run tests

Publish results

44

Continuous Testing

Watch CI

Deploy product

Integration tests

Publish results

45

Continuous Testing

Build in CI (compile & unit tests)

Deploy to dev (staging)

Run integration tests

46

What does this tell us?

If it works

(quickly)

47

Continuous deployment is controversial.

Most people ... think I’m advocating low-quality code

or an undisciplined cowboy-coding

development process.

ERIC RIES radar.oreilly.com/2009/03/continuous-deployment-5-eas.html

48

Language is a Tool

Use it

49

What's this look like?

50

Continuous Testing

CI CT

Compile & unit test

Deploy & integration

test

51

Multiplatform Deploys

CICD Win7

CD Linux

CD AIX

52

Websphere Deploys

53

DemoAdd a new Jenkins job Monitor the compile job Deploy the artifact Run integration tests

54

54

ExerciseSame as before... What breaks (in your experience)? Top 3 bug types? Groups of 5 to 7 Gather the top 5 for your group

55

55

As We Explore a Continuous World...

56

56

What Concepts Emerge?

57

57

Nuke and PaveReset (completely) Start over Completely Scripted

58

58

BinariesNuke & Pave Delete Recompile Deploy Use

59flickr.com/photos/x-ray_delta_one/4152356464

59

DataNuke & Pave Delete schema Drop tables Recreate Restock Rails Liquibase


60

Rails ActiveRecord

CREATE TABLE products ( id int(11) NOT NULL auto_increment, name varchar(255), PRIMARY KEY (id) );

61

61

Rails ActiveRecord

change_table :card_deck do |table| tablet.remove :description, :name tablet.string :first_name tablet.index :part_number tablet.rename :list_name, :last_name end

62

62

Rails ActiveRecordrake db:migrate rake db:migrate VERSION=0 rake db:reset

63

63

ActiveRecord Demorake db:reset Drops schema Recreates schema

64

64

Operating System

Nuke & Pave Virtualize Recreate


65

Vagrantvagrant init lnx http://lnx/lnx.boxvagrant upvagrant sshvagrant reload --provision!

Command line operating system provisioning

66

66

Vagrant

config.vm.network "forwarded_port", guest: 80, host: 8080config.vm.synced_folder "../data", “/vagrant_data"

67

67

Steps for a New OS

Install VirtualBoxInstall Vagrantvagrant init hashicorp/precise32vagrant up

68

68

Output

Downloading: https://vagrantcloud.com/hashicorp/boxes/precise32/versions/1/providers/virtualbox.box!

69

69

Demo

Start a Vagrant virtual machine

70

70

Benefits

Scriptable

Reproducible environment

Solid results

Fast results

Repeatable results

71

When?

Day one

Easiest

72

“If you don’t do it on Day One, you probably won’t”

!

Andy Hunt

73andy.pragprog.com

73

Easy?

No

Difficult

Necessary

Code

74

Costs Move to Zero

Once scripted…

What changes?

75

What Other Ideas?

76

Developers

Devs write code normally

SCM

Code is shared via a source code management

tool

Continuous Testing

Providing immediate

feedback and a shared

environment for a QA “First Look”

Triggered Deploy to QA

QA team members trigger

this deploy

Manual Testing

A stable environment for

manual and exploratory testing

77

SCMDevelopers

Watch

Build

Unit Tests

Deploy to Dev

Integration Tests

First Look

Continuous Testing

On Demand Deploy to QA

Manual Testing

Rapid feedbackPair Dev & QA

78

First Look• QA can see what’s been coded immediately

• Differences in opinions can be discussed quickly

• Fewer problems during the larger testing cycles

• Keeps QA & Dev working together as partners

• Faster feedback for QA

79

Continuous Testing

• Includes QA in the continuous feedback cycle

• Retains a traditional QA stable testing environment

• Dev & QA should both review all tests (unit & int.)

• QA requests trigger the QA env. deployment

80

Write Code

Requirements

The Traditional Approach

DemoDiscuss

Rewrite

Typically a confrontation &

argument

The demo forces a

discussion of the requirements

An isolated event, primarily worked by the customer proxy

81

Demo

Write Code

Requirements

The Agile Approach

Rewrite

Discuss before significant time is

invested

Include the Golden Triad

Discuss

82

Alternatives

83

flickr.com/photos/lightsinthedark/5810724330/

Work in a vacuum

84

Ruthlessly Automate!

Builds

Deploys

Tests

flickr.com/photos/pmillera4/9317924474

85

Test Automation

Unit tests

Acceptance Tests

86

Unit Tests

One class

One method

Fast

Illuminating

87

Integration Tests

High level

Lots of coverage

No detail

88

Continuous Feedback

Large amounts of code

Small amounts of change

Tests run after every change

Requires frequent code checkins

89

Continuous Testing

CI CT

Compile & unit test

Deploy & integration

test

90

What's Better?

Unit or Integration?

91

91

Best Is...

Automated Reliable

Frequently executed

92

92

Test LifecycleSet up state Use product Verify results (Clean up)

93

93

DemoExplore Unit tests Explore Selenium

94

94

Set Your Teams Up

Builds work

Functionality works

Biggest risk?

Requirements

95

Developers Testers

Customers

The Team

96

Grid Driven Development

Remove technical barriers

Everyone speaks “Excel”

97

Demo

Write Code

Requirements

The Agile Approach

Rewrite

Discuss before significant time is

invested

Include the Golden Triad

Discuss

98

Testing Strategies

Test First is best

Defect Driven Testing

Blitzkrieg Testing

99

IN Summary...

100

Culture Change

Fundamental change in development

101

StepsAutomate builds


Add unit tests

Automate deployments

Continuous Delivery

Add integration testsflickr.com/photos/untitlism/2547423465

102

Your move

103

Exercise

Two 3x5 cards

3 reasons for Continuous Testing

3 barriers to adoption

Groups of 5 to 7

Brainstorm on solutions

flickr.com/photos/curtisperry/6119618990

104

Tips

Open standards...

Junit or Junit compatible

Avoid vendor lock in

Glue tools

Time matters....

Fast servers

Incremental change

105

Script Tips

Save scripts in source code management

Modular

Basic encapsulation

One target => One function

106

Script Targetsall depends_on clean, compile, test-unit

clean # deletes build artifacts

compile # builds the product (Ruby?)

test-unit # run your tests

deploy_locally # to your dev environment

deploy_dev (?) # to a shared dev environment

test-integration # run your integration tests

107

Resources

JUnit JUnit.org

Jenkins Jenkins-ci.org

BDD Dannorth.net/introducing-bdd

Cucumber Cukes.info

Continuous Deployment en.wikipedia.org/wiki/Continuous_delivery

108

Devops Resources

VagrantUp.com

GetChef.com/chef

PuppetLabs.com

Ansible.com

VirtualBox.org

109

Data Resources

Guides.RubyOnRails.org/active_record_basics.html

Liquibase.org

FlywayDB.org

110

ExerciseUseful or not...

Will you pursue or not...


Continuous Testing

Test automation

How will these ideas help or hurt your org?

Write down your answers and pass them up

111

Retrospective

You're the customers

Were your requirements met?

What else needs to be added to this class?

What can we cut out?

112