the testing maturity model in the real world...our tmm assessment model our approach involved: •...

© Marks and Spencer plc 2004

The Testing Maturity Modelin the Real World

W15

Brian Wells & Andrew Goslin1 December 2004


Objectives

The objectives today are to:• Tell you why we needed to assess test process

capabilities• How we used the Test Maturity Model (TMM) method

to undertake a pilot• What was achieved

and• Our views on taking TMM Forward


Agenda

• The Motivators!• CMM/CMMi/TMM Sister Act!• What is TMM and our TMM Assessment

Method• The Maiden Flight of (our) TMM!• What happened in-flight and after landing!• Scheduled Flights for TMM?• Conclusions


How do you assess the quality of the testing process that demonstrates quality?


Motivators

• What were the testing capabilities of our partners & internally?

• Needed an objective robust assessment method, clearly understood by all

• Wanted to identify improvement opportunities in an action plan with realistic timescales

• Need to develop a structured test process improvement framework

• Extend institutionalisation of good practices• Needed a seamless project lifecycle…

but geographically challenged!


CMM/CMMi/TMM Sister Act!

Considered:• Test Process Improvement (TPI) – continuous model

and flexible to organisation• Test Improvement Model (TIM) – Based on CMM but

not as extensive in KPAs nor as detailed• SPICE – Doesn’t address testing in sufficient detail• Also looked at others, but…• Test Maturity Model

• Clear mapping to CMM/CMMi already.• Contains required depth of most aspects of test

and test-related processes


CMM/CMMi/TMM Sister Act - 2

We decided on TMM. Why?• Used experience and structure of CMM assessments

and trained CMM Assessors• TMM can be used as a staged or continuous

assessment model• TMM could integrate into CMMi• TMM can be used in isolation…• In reality supports CMMi process maturity, but

concentrates on testing!


The TMM Structure

5 Levels of maturity, each containing:• Maturity Goals (KPA)• Purpose of Maturity Goal• Maturity sub-goals (Generic & Specific Goals)• Questions (Key Practices)• Critical Views

(Sub practices based on 3 roles: Manager, Developer/Tester & User/Client)


TMM Structure Diagram

LevelsContainIndicate

Testing Capability Maturity GoalsSupported by

Maturity Sub-GoalsAchieved by

Activitiers / Tasks / Responsibilities

Critical ViewsImplementation and

Organisation Adaption

AddressOrganised by

Manager Developer / Tester User / Client


The TMM Structure


TMM Text Book Approach

• Service Mark of Illinois Institute of Technology• Text Book indicates that TMM questionnaires be sent

out to organisations, supported by follow-up interviews if required

• No requirement mandated for supporting artefacts• Assessment based on questions and responses;

evaluation of responses against Critical Views• Critical views based on Senior manager,

Developer/Tester and User/Client• Has basic ranking system: Satisfied / Not Satisfied


Our TMM Assessment Model

Our approach involved:• Organising TMM into CMM/CMMi structure• Assessment through structured interviews of project /

organisational roles• Trained CMM Assessors• Requires for submission of supporting artefacts• Review and consensus of information by assessment

team• Consolidated assessment workbook• Follow CMM/CMMi ratings process and procedures• Structured feedback of findings


TMM - Learning to fly…

• Must obtain clear set of objectives from sponsor before undertaking TMM assessment

• All information is confidential and de-personalised at all levels before presentation

• Format is...• Briefing Session• Interviews• Review of artefacts• Scoring by consensus• Structured feedback including action plan


TMM – Our first take-off

• M&S are working towards CMMi Level 3• All preferred suppliers were CMMi Level 5

certified• We assessed to TMM Levels 2 & 3 only• Everyone had some exposure to CMM/CMMi,

this greatly assisted buy-in of unknown• Some scepticism at beginning, but this

evaporated quickly• All found the experience very useful and results

focused everyone


TMM – Our maiden flight

• All indicated some failures at Level 2 and 3

• Some activities are done well by all…

• No-one does some of the required activities…

• About 50% of the activities were done well by most

• Best practices were highlighted for sharing


TMM – Our first landing

• We established our capability baseline which was positively accepted

• Identified…• Common strengths & weaknesses• Best practices

• Identified other improvement areas

• Easily defined process improvement road map

• Road map was readily accepted and implementation is in progress


How was the flight?

• What went well• Preparation• Structured• Assessment method / process• Credibility, understanding & acceptance• Knowledge of organisations’ at many levels

• What did not go well…


How was the flight?

• Learning's & Improvements• Need to streamline summarisation of results• Supporting tool

• Pitfalls• Time to complete formal assessments• Relative expense• Marketing longer term benefits

• Issues• Needs to be supported by a robust review / QA

process


Our Conclusions

• CMMi level 5 is not necessarily a good indication of Test Process Maturity!

• Practical, clearly structured framework for test process improvements resulted

• TMM is the best structured and most robust method for test process assessment

• Time consuming, but comprehensive


Our Conclusions

• Unexpected findings• Two-way insight into organsations’ culture!• Partners – high turn-over of test resources• Day to day on the ground issues• Immediate ‘micro-process’ improvements

• Intend a follow-up with formal assessment in 12-18 months

• While it was hard work - It was fun!


Scheduled Flights for TMM?

• We intend scheduled flights for TMM and flying in formation with CMMi assessments

• Value and benefits readily accepted by stakeholders

• Investigating a TMM Forum:• Collaboration of practitioners• Common assessment approach and tools• International acceptance as a standard for Test

Process assessments?


Final thoughts…


How to contact us?

• Brian Wells• Mobile +44 (0)7768 855817• Email [email protected]• Andrew Goslin• Mobile +44 (0)7795 266290• Email [email protected]


References

• “Practical Software Testing” by Ilene Burnstein: ISBN 0387951318

• “A Comparison of Test Process Assessment Models” by Ron Swinkel, Frits Phillips Institute dated 17 November 2000

• CMM/CMMi, SEI

the testing maturity model in the real world...our tmm assessment model our approach involved: •...

Documents