planning, architecting, implementing, and measuring automation

TH AM Tutorial

10/14/2014 8:30:00 AM

"Planning, Architecting, Implementing,

and Measuring Automation"

Presented by:

Michael Sowers

Software Quality Engineering

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Mike Sowers

Software Quality Engineering Mike Sowers has more than twenty-five years of practical experience as a global quality and test leader of internationally distributed test teams across multiple industries. Mike is a senior consultant, skilled in working with both large and small organizations to improve their software development, testing, and delivery approaches. He has worked with companies including Fidelity Investments, PepsiCo, FedEx, Southwest Airlines, Wells Fargo, ADP, and Lockheed to improve software quality, reduce time to market, and decrease costs. With his passion for helping teams deliver software faster, better, and cheaper, Mike has mentored and coached senior software leaders, small teams, and direct contributors worldwide.

Me and My Contemporaries 1

Michael D. Sowers, CIOSoftware Quality Engineering

Planning, Architecting, Implementing, and Measuring

Automation

Copyright M.D. Sowers 2

Administrivia

Messages Meals

Tutorial timing Tutorial materials

Breaks

Pagers andCell phones Facilities

Smoking


Copyright M.D. Sowers

A Heap of Tools


Tutorial Goals

At the end of this tutorial you will be able to:

• Define your integrated automation team

• Document your current state

• Layout a future state and roadmap

• Create your pilot and implementation plan

• Identify initial key measures



Outline

� Introduction� The Compelling Business Need(s)

� Automation Challenges

� Automation Requirements

� Life Cycle Automation

� Overarching Approach� Define the Team, Assess Current State

� Develop the Architecture

� Plan, Pilot & Implement

� Measure and Report Benefits

� Lessons Learned


Outline









� Lessons Learned



Compelling Business Need(s)

• Faster…..........– Development

– Testing

– Deployment

– And more…

• Better………………… – Features

– Performance

– Usability

– And more…

• Cheaper……………………– Elaboration

– Implementation

– Delivery

– And more…


Compelling Business Need(s)

• Faster………………………………… Speed to Market– Development

– Testing

– Deployment

– And more…

• Better……………………………….. Quality Product(s)– Features

– Performance

– Usability

– And more…

• Cheaper……………………………. Competitive Innovation– Elaboration

– Implementation

– Delivery

– And more…



Automation Challenges

• An abundant array of disconnected tools…

with assembly usually required…

focused on execution vs. design…

demanding a high-level of technical expertise..

and significant maintenance…

• Culminating in an investment not always understood and an array of tools not used…


Automation Key Requirements

• A strategy, a vision and a plan

• An architecture and integration (across the lifecycle and between technology/vendors)

• Early lifecycle centric

• Roles based with associated training

• Aligned and integrated with processes

• SUT Focused

• With business oriented measures and reporting



Life Cycle Automation


alm Generalized (video)



Outline









� Lessons Learned


Define the Team

• Cross Functional with representation from:

– Product Office

– Technical Architecture (and standards)

– Software Development (and maintenance team)

– Quality Assurance and Testing

– IT

– Business Units

• Global representation

• Key Opinion Leaders



Define the Team – Exercise

• Draft answers to the questions in the Appendix


Assess the Current State

• Inventory current tools & processes

– Tool Category

– Purpose (single function/multi-function)

– Value

– At what lifecycle stage does the tool apply

– What process(es) does the tool support?

– Existing Linkage to other tools

– License and Maintenance expenses

– How many users (departments, BU’s, teams)

– Current issues/deficiencies

– Platforms Supported

– Others?



Assess – Current State


Assess – Current State



Assess the Current State – Exercise

• Fill in a draft of the spreadsheet in the Appendix


Plan - Vision, Strategy, Approach

• Vision: End in mind/automation dream state?

– 1-3 year time horizon

– Aligned with critical business needs

– A quantified “end in mind”

• Strategy: The “what” of achieving your automation vision

– Engage the key opinion leaders

– Address existing pain points

– Link to business objectives

• Approach: The “how” of achieving your automation strategy

– A blueprint (of current and future state)

– Aligned with the Technical Architecture

– An agile plan – responsive & flexible to change



Establish an End in Mind (Vision-1)







Architectural Considerations

Operations

Access & Presentation

Business

Security

Infrastructure

Application

Data

Integration



Process Considerations


Process Considerations



Establish a Vision – Exercise

• Draft (draw) a vision, an end state of your tools architecture (big boxes are fine to start)


Establish a Plan

• Partial Blueprint

• Integrated Tools Architecture

• Tools Roadmap

• Tools Mapped to the lifecycle

• Roles Mapped to the lifecycle



It’s like building an Airplane in the Air!


Partial Blueprint

Technology Area 2008 2009 2010 2011

Requirements Management Borland -- CaliberRM 2006--

(U)

IBM -- Rational

RequisitePro 8.X-- (P)

IBM -- Rational


IBM -- Rational


HP/Mercury -- Test Director

(Quality Center) for

Requirements Management

9.x-- (E)

HP/Mercury -- Test

Director (Quality

Center) for

Requirements

Management 9.x-- (E)

HP/Mercury -- Test

Director (Quality Center)

for Requirements

Management 9.x-- (P)

HP/Mercury -- Test

Director (Quality Center)

for Requirements

Management 10.x-- (P)

Developer Test CxxTest (R) Parasoft – C++ Test

(P)

Parasoft – C++ Test (P) Parasoft – C++ Test (P)

Test Management IBM -- Rational Test Manager--

(R)

IBM -- Rational Test

Manager-- (R)

HP/Mercury -- TestDirector:

Quality Center 10.X-- (P)

HP/Mercury --

TestDirector: Quality

Center 10.X-- (P)

HP/Mercury --


Center 10.X-- (P)

HP/Mercury --


Center 10.X-- (P)

Functional Automation IBM -- Rational Functional

Tester-- (R)

IBM -- Rational

Functional Tester-- (R)

IBM -- Rational

Functional Tester-- (R)

IBM -- Rational Robot 7.x-- (R) IBM -- Rational Robot

7.x-- (R)

Borland -- Silk Test-- (R) Borland -- Silk Test--

(R)

HP/Mercury -- WinRunner 9.x-

- (R)

HP/Mercury --

WinRunner 9.x– (R)

HP/Mercury – QTP (P) HP/Mercury – QTP (P)



Integrated Tools Architecture


Tools Roadmap (1)



Tools Roadmap (2)


Tools Mapped to Lifecycle



Methods, Tools and Measures Engineers

Functional Test Engineers

Test Automators

Test Coordinators

Test Architects

Performance Test Engineers

Test Designers

Roles Based & Life Cycle Integrated

Production Excellence Manager

SIT PRODUATDITTSDSDSSRAPPIAIP Reviews Function Test Performance Test Operational

Readiness Test

Unit Test

VerificationVerification

ValidationValidation

Production

Excellence

Production

Excellence

Systems Test Engineers


Establish a Plan – Exercise

• Using the examples presented, begin to think about (draft) the planning questions in the appendix



Select, Pilot & Implement

Tool selection/purchase

Review of tools

Evaluate/pilot a select set of tools

Demonstrate tools

Market analysis

Tool requirements specification


Select, Pilot & Implement – Exercise

• Begin to think about (draft) the select, pilot & implement questions in the appendix



Measure & Report Benefits

• Tool Measures in Technical Terms

– Comprehensiveness of Tool Functionality

– Number of issues/defects in or with the tool

– Satisfaction/response time with Tool Support

– Ease of Use

– Ease of Integration (api)

– And more…

•Measures in Business Terms

–The MOST important!


Business Oriented Measures

2008 2009 2010

Expansion

• Applications covered by QA&T Team 90 160 230

• Number of Applications Automated 22 46 78

Effectiveness

• Pre-Release defects 162 200 274

• Post-Release defects 28 22 18

• Requirements coverage (%) 53 71 87

Efficiency

• QA & Test staff to developer ratio 1/5 1/5.5 1/6.5

• Time for regression test runs (days) 8 5 4

Business Results

• QA & Test Total Cycle Time (days) 40 36 31

• Cost per Total Test Cycle (K dollars) 320 288 248

• Post release defect containment (%) 85.3 90.1 94



Measure/Metrics – Exercise

• Begin to think about (draft) the measures/metrics questions in the appendix


Outline









� Lessons Learned



Key Learning's

• Do:

– Engage Sr. Technical Staff

– Secure Stakeholder Alignment

– Have a larger vision, but focus on one area at a time

– Allow for flexibility and customization

– Monitor the “scope” of standardization

• Don’t:

– Permit too much separation between roles

– Ignore existing processes & dependencies

– Limit your thinking to the current technology


(A Few) Keys to Success

Automation as a

Technology Issue

Automation as a Business

Imperative

Technical Measures Business Measures

No Vision, Strategy,

Architecture

Complete Automation

Blueprint

Buckets of Automation Integrated Workflow

Everyone does

everything

Roles Based

FROM TO



A Less Than Functional Home


Remember - We build Airplanes in the Air

• Doing so with isolated tools

• Without LEADERSHIP and a PLAN

• Results in an interesting – but not so useful automation mansion

• Wasting the lives of many skilled crafts men and women (our developers & testers)

• And no value contributed to the business



Tutorial Goals Achieved

You are now able to:

• Define your integrated automation team

• Document your current state

• Layout a future state and roadmap

• Create your pilot and implementation plan

• Identify initial key measures


Wrap Up

• Current Challenges?

• ?

• ?

• ?

• ?



How did we do?

* Great * Wonderful

* Fantastic * Terrific

* Superior * Excellent

Tutorial Evaluation

Tutorial Evaluations


Thank You

• On behalf of Software Quality Engineering, thank you for attending this tutorial.

• If you have further needs for training or consulting, please think first of SQE.

• If I can be of further assistance, please let me know. My email is [email protected]



Appendix

Appendix A

Copyright M. D. Sowers, Software Quality Engineering 1

Integrated Tool Architecture

Team Selection

1. Who are the key stakeholders for software lifecycle automation? Why are they key

stakeholders?

2. Of the key stakeholders, which are critical to include? Is there balanced participation globally?

3. What's the approach for engaging and securing a commitment to participate from the critical

stakeholders?

Appendix A


Tool(s) Inventory (Example)

Appendix A


Tool(s) Inventory (blank)

Appendix A


Tool(s) Implementation Plan

1. What’s the first area of focus? (requirements, dev, test, deployment)

2. For the initial area of focus, map those tools to the life cycle steps/stages

3. Tools to retire, tools to add?

4. Where would tool connections be beneficial?

Appendix A


Select, Pilot & Implement

1. For the specific area of focus or tool selected what are the requirements?

2. What will be the focus of your market research? (Commercial, Open Source, Custom

Development)

3. Who are the key stakeholders that need to be engaged in the demo's?

4. What (1 or 2) teams might be good candidates to pilot the tool?

5. What changes to existing methodology or standards may be needed to adopt this tool into our

existing environment?

Appendix A


Measures/Metrics

1. What specific technical measures will be used to assess the success and contribution of this

tool?

2. What specific business measures will be used to assess the success and contribution of this tool?

3. What information do I need to plan to collect to track these measures?

planning, architecting, implementing, and measuring automation

Technology

software development

implementation plan

architecture plan

global quality

quality products

current state layout

future state

small teams