Clint SprauveMalcolm Isaacs

The Evolution of Test Automation for Dev/Ops

Quality driving value through DevOps

The “why”--How DevOps presents a great opportunity for quality professionals to expand their organizational impact The “what”--Specific new processes and skills quality professionals bring to DevOps and Shift Left models

The “how”-- How quality professionals and teams are adopting DevOps practices and capabilities

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3

Quality matters to DevOps

Why?

Evolution of Functional Testing

Agile and DevOps challenge testing processes

Current challenges…

Open Source and Continuous Integration Tools

User Experience changesthe definition of quality

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

93%of IT leaders use Agile

for software development projects

$60B

annual savings by businesses using open source

Global population estimated to be connected by 2019*

autonomous things estimated to be attached to the Internet by 2020*

57%

30B

Database activity

Security validation

Inventory query

Database activity

3rd party ship

service

Logistical services

Acc’t info

3rd party database

3rd party reporting

Security validation

3rd party payment services

Account authorization

Shopping app(web or mobile)

Purchase

receivedAdd to cart

Shipping detail

Delivery tracking

Customer login Checkou

t

A Retailer Real-Life ScenarioA Modern, Complex Architecture

Geographies of users?

Flows?

User types?(mobile, web, other)

Services availability and cost?

Single vs. Multiple?

Type of network connections?

Environment availability?

Is it secure?

Does it function?

Functional

Development

USE CASE:

Performance

DevOps

Security

Mobile

QUIT SCENARIO

DevOps is a transformationIt’s a sustained effort to achieve speed with Quality

• With DevOps there is no one true authority

• It’s not a product, spec, job title

• There is no one “right way”

• But one thing is certain- Quality matters

The role and importance of Quality in today’s AD teams is moving from finding

defects to protecting the corporate image World Quality Report– 2015-16

People

Process Technology

The good news– Quality is centralDispel the myths that Quality goes away with DevOps

Dev Ops

Quality

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8

Quality in the age of DevOps

What

Quality Within DevOpsDefining The What

1. Starts with Culture Change2. Quality Integrated Throughout3. Test Constantly4. Relentlessly Automate5. Create Visibility

Starts with Culture ChangeQuality within DevOps

1. Starts with Culture Change. Teams unite around making the org as a whole faster, more competitive.

“Coming together is a beginning; keeping together is progress; working together is success” Henry Ford

Quality Integrated ThroughoutQuality within DevOps

2. Quality Integrated Throughout. Move everything to happen earlier. Testing happens earlier, get comfortable with IDEs, leverage development assets, but add your testing acumen.

Test ConstantlyQuality within DevOps

3. Test Constantly. If you’ve followed point 2 above, you’re testing earlier. You’re also going to test much more often. And consider the Quality Trifecta—ensure Functionality, Performance, Security.

Relentlessly AutomateQuality within DevOps

4. Relentlessly Automate. Automated testing shortens time to market and lowers risk of human error in testing, while increasing the cycles in same amount of time, and increasing coverage. Automate testing, provisioning and virtualize. Then tie it to build life and automated execution integration.

Create VisibilityQuality within DevOps

5. Create Visibility. Seamless communication between people, tools and processes. Removing constraints that slow you down and connecting tools through a framework.  

FrameworkQuality within DevOps

AutomateOrchestrate TransformApplication

DeliveryManagement

IT OperationsManagement

Application testing for desktop, web

and mobile

Infrastructure, domain and compliance automation

From silos to integrated lifecycle management

Processes to improve agility

Customer experience by simulating real world

scenarios

Customer experience by delivering products and

services

DevOpsShared Dev & Ops

methodology across Dev/Test/Stage/Production

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16

In Practice

How

How we have delivered at HPE SoftwareQuality within DevOps

Starts with Culture ChangeQuality within DevOps

• Why we thought to change = Productivity

• Development environment based on shared infrastructure • Unstable• Can be easily damaged• Common for everyone• Operational cost is very high• Unexpected behavior

• Work from anywhere• Latency is killing your connections

Quality Integrated ThroughoutQuality within DevOps

Plan Define Develop Build Test Deploy Release Operate

AGM AGM AGM

Sonar

OO

SM/SAW

Puppet

UCMDB

BSMFortify

Jenkins

Wiki

OpenGrok

GIT/Stash

IntelliJ Maven

ALM OO

Puppet

SM/SAW

Nexus

LeanFT

LR SiS

CSA

Nexus Nexus

Maven Site

Vagrant

Sahara

3RD P

artie

sPr

oces

sH P

CSA

Docker

JUnit

JBehave

Jasmine

Selenium

SharePoint

NV and SV

Organization Example - Evolve the Team StructureChange based on product complexity and team skill set

Dev/Test Organization Additional Responsibilities QA & DEV – one team Support Tickets

Automation from day 1 Forums & social networsk

Manual Testing is Essential Build CI/CD infrastructure

No more deliveries to QA Train internal teams on dev/ops practices

Structure and responsibilities will vary from team to team, project to project

Do not implement a “cookie cutter” approach to quality in Dev/Ops

Test ConstantlyQuality within DevOps

Tests run on each push13,000 automated functional tests run daily

Service Anywhere 3.3 Service Anywhere 3.4336 application tests 624 application tests

40 platform tests 70 platform tests

Manual coverage 100% sanity coverage

Can run only on CI deployment Runs on real deployment too

Relentlessly AutomateQuality within DevOps

• End2End automated flow for continuous testing

• Jenkins acts as the orchestrator

• Flexible & customizable to accommodate specific customer workflows

DEV

QA

STAGE

PROD

Build

Deploy

Test

Jenkins

HP CodarALM

Test case1

Test case2Test case

3....

Test case n

Example – Dev/Ops Automation Test Coverage DistributionTEST TYPE Client Backend Protocols/Regions

Coverage 60% 80% 100%

# of Active Tests

133 306 All Protocols, All regions

Create VisibilityQuality within DevOps

• Accurately Predict if a Code Commit Will Break a Build

• Accurately Predict and Plan for Defect Introduction

Key Lessons LearnedCase Study Results

• DevOps is possible. Huge potential for speed and quality

• The culture changes when you incorporate DevOps

• Discipline – DevOps requires focus and dedication

• Automation and tools are important, but not everything

• Communication, collaboration and trust

More info

– Trials and free editions of tools you can use– Trials

–NV Trials: http://www8.hp.com/us/en/software-solutions/network-virtualization/try-now.html–LeanFT: http://saas.hp.com/signup/try/leanFT

– Community Editions–LR 12.50 with NV: http://saas.hp.com/signup/try/LoadRunner

– Freemium(s)–NV Freemium: http://www.hp.com/go/NVFreemium

Thank you

27