softtest irelandcopyright 2008 electromind ltd1 people challenges for test teams stephen k. allott

SoftTest Ireland Copyright 2008 ElectroMind Ltd 1

People Challenges for Test Teams

Stephen K. Allott


Introductions

My background Aims of this workshop

Explain some common testing challenges based on recent experience

Ask for your experiences Agree a list of challenges

we all face as testers Try to come up with some

quick wins & longer term approaches to solutions


My recent clients

Investment bank Airline web site Travel company web site Mobile phone operator Retail web site Travel company back office Recruitment web site

Includes assignments done in association with TestHouse (London) and Neueda Technologies (Antrim & Dublin)


Their technologies

Microsoft Dot (.) NET 1/2/3 Microsoft VSTS / TFS Oracle MS SQL Server Windows XP Citrix Metaframe Quality Centre / QTP


Number of testers

Typically 12 to 14 in the test team A couple of clients had only 1, some have none!


Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Key Area Controlled Efficient Optimising

Test strategy A B C D

Life-cycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communication A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaluation A B

Low-level testing A B C

Composite TPI Test Maturity Matrix from several clients


Summary of TPI results

No documented test process No formal test policy, strategy or plan Many tests, no formal design No metrics No proper test environment

Some tested in “live”

No documented test basis Not involved early enough


Some common observations

Constantly changing business priorities Increasing (frantic) demand for new functionality Weekly, sometimes daily release cycles Attempt to use automation to solve all the problems No numbers to explain testing’s added value Small important teams yet with little influence


People challenges – behaviour

Motivation A few are resigned to how it is, no hope? Some expect spoon feeding Some adopt a negative attitude Many are willing to make improvements, if there was time

Turnover – very high offshore Culture

Some managers get too involved in detail Co-dependency – testers must push back It hasn’t broken yet so . . .


People challenges – skills

Testing SkillsProcessesStandardsTechniquesTools

IT SkillsProgrammingDatabasesInfrastructureWeb

Domain KnowledgeIndustry specificApplication knowledgeE-Business

Soft SkillsCommunicationNegotiationManaging technical teams


People challenges – perception

In general testers are not valued Testers are often blamed

Project delays Poor quality Managing expectations with RBT

Lack of awareness of what the test team does


People challenges – managers

Demand for certified staff Demand for skills & knowledge Pressure to cut costs and use offshore resources Developing good test leads Role of testers in an agile project Selling the value of testing to the organisation


Themes

Facilitated discussion bearing in mind the following themes:

1. Relentless business pressure for new functionality2. Constant and last minute changes to requirements3. Introduction of new development / testing methodologies e.g. agile 4. Very frequent releases, often weekly, sometimes daily!5. Increasing application complexity over time6. Knowledge gap (e.g. what lies beyond the browser?)7. Steep learning curve (business as well as technical)8. Use of contract and offshore resources (loss of in-house expertise)9. Little or no documentation against which to test10. Lack of attention to all testing requirements (e.g. middleware, back

end database, non-functional quality characteristics)


Discussion – how do we respond?

Business Pressures

New functionalityFrequent changeMaturity

TechnologyComplexRapid changeInterfaces

OffshoreCheaperPlentifulExpertise

BehaviourMotivationAttitudeCulture

SkillsAnalysisIT / TechnicalBusinessTestingSoft skills

ManagersCostsTime to marketAgile


Conclusions

In my opinion testers of the future must . . . Measure / appreciate application complexity Be able to accept and manage last minute changes Be flexible but push back with estimates Increase their technical knowledge Learn more about testing, especially techniques Apply the tried and tested techniques Capture metrics to help prove the business case Get involved in the projects much earlier

Recruit / train / certify / develop / coach = Retain Share your views with the group


Final Thoughts / Questions?



[email protected]

+44(0) 773 476 1363

www.electromind.com

mailto:[email protected]

http://www.electromind.com/



Appendix – Bonus Slides


A typical project in trouble

CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3

1427

58 5565

40

116

279

229213

0 0

254

124

263

0

50

100

150

200

250

300

June July August September October

Months

CR

co

un

t

0

50

100

150

200

250

300

Ch

ang

es t

o C

Rs

CRs Churn Bugs


Bug snapshot by priority

Sev 1 Sev 2 Sev 3 Sev 4 Total

Pri 1 57 81 14 152

Pri 2 19 52 12 1 84

Pri 3 6 14 10 30

Pri 4 2 1 3

None 68 288 114 10 480

Total 146 428 154 21 749


Bug snapshot by resolution

Sev 1 Sev 2 Sev 3 Sev 4 Total

By design 10 37 29 4 80

Duplicate 6 33 7 1 47

External 2 16 4 0 22

Fixed 86 254 95 12 447

Not repro 29 56 4 0 89

Postponed 12 17 6 4 39

Will not fix 1 15 9 0 25

Total 146 428 154 21 749


Impact of the improvements

CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3

1427

58 5565

30

61

17 15 10 5

40

116

279

229213

6686

18 15 10 50 0

254

124

263

13

90

12 9 5 10

50

100

150

200

250

300

June Ju

ly

Augus

t

Septe

mbe

r

Octobe

r

Novem

ber

Decem

ber

Janu

ary

Febru

ary

Mar

chApr

il

Months

CR

co

un

t

0

50

100

150

200

250

300

Ch

ang

es t

o C

Rs

CRs Churn Bugs

Events Systems1. Vendor2. User Reg.3. User Shop

Sales and Marketing Systems1. Consulting Practice Mgt.2. Vendor Account Mgt.3. Event Mgt.4. Leads and Referral

Databases1. Profile2. Common3. Event OLTP4. Transaction OLTP5. Contact History OLTP6. Domain Reference

World-wide Marketing DB1. Profile OLAP2. Event OLAP3. Analytical Reporting4. Contract History OLAP5. Domain

Corporate Sales and Marketing Systems

System Interfaces1. Event Orders2. Billing & Financials3. Credit Card (real-time)4. Event Communication / FAX5. Direct email service6. Direct mail Postal interface7. Leads & Referrals8. OLTP Staging Interface9. OLAP Staging Interface10. SAP11. Licensing12. Call Center

Source: Reality Test Inc.


Basic complexity calculation

Characteristic Value Factor Total

Windows 37 8 296

Objects 23 1 23

Tables 47 1 47

Apps Impacted 3 10 30

Factors are weighted from a testing perspective


Comparison across applications

Application UI Back End

Env Test Code

Middle Tier

Total

Sales Mgr. 1360 2250 152 0 0 3762 Call Center 960 824 90 129 0 2003 Web Sales 389 183 109 0 2 681

Middleware component

0 146 23 27 232 428


0

1000

2000

3000

4000

Application Name

Co

mp

lex

ity

Va

lue

Complexity comparison


Actual staffing comparison

0

5

10

15

20

Applications

Nu

mb

er

of

Te

ste

rs

Estimated

Actual


Organisation evaluation

Organisation within the company Process definition and execution Teamwork Ability to do the job


Organisation within the company

Section Item being evaluated Max Score

1.1 Reporting structure 50

1.2 Professionalism 170

1.3 Organisational churn 135

1.4a Staffing – permanence 30

1.4b Staffing – stability 35

1.4c Staffing – morale 50

1.4d Staffing – number of testers 40

1.4e Staffing – organisation type 40

1.5 Budget control 30


Process definition and execution


2.1 Software development life-cycle 60

2.2 Design reviews and documentation 70

2.3 Early participation in development 50

2.4 Ability to influence schedule deadlines 60

2.5 Development standards 60

2.6 Defined testing approach & procedures 190

2.7 Metrics reporting 60

2.8 Churn 90

2.9 Patch releases 40

2.10 Post-project reviews 60


Teamwork


3.1 Support from production support 60

3.2 Support from management 60

3.3 Correct level of quality 90

3.4 Resource allocation 80


Ability to do the job


4.1 Adequate equipment / lab 200

4.2 Automated testing 70

4.3 Programming ability 70

4.4 Non-testing activities 50


Total evaluation score

900

1000

1100

1200

1300

Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01

Date

Eva

lua

tion

Total Evaluation Score


Organisational Evaluation

100

200

300

400

500

Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01

Date

Eva

lua

tion

Organization

Process

Teaming

Ability


Components of a centre of excellence

Resource pool Test lab

Test Managers SIG

Consultants

Knowledge Base

R&D Centre

Tool Experts

Project 1

Project 2

Project 3

Project 4


Benefits of a centre of excellence

Independent testing Breadth & depth Fewer product recalls Fewer production failures Central reference point Knowledgeable staff Expertise across org. Structured career path

Variety of work Identity, pride Morale, professionalism Extra revenue stream Consistent approach Multi-methodologies Earlier involvement

softtest irelandcopyright 2008 electromind ltd1 people challenges for test teams stephen k. allott

Documents

softtest ireland copyright

b c test environment

b c test functions

b c communication

b c d test automation

b c reporting

b c evaluation

b metrics