franck mignet test planning and cost optimization in an ... · as test manager for a typically...
Post on 12-Oct-2020
1 Views
Preview:
TRANSCRIPT
Titel, samenvatting en biografie _________________________________________________________________________________________________________
Franck Mignet
Test planning and cost optimization in an uncontrolled project
Voorjaarsevent Testnet: 22 juni 2009
Summary In their everyday practice, testers and test managers work with real projects, that is: chaotic, lacking process and control. What can a test manager do to accurately and optimally plan the test activities and at the same time reduce the costs? As Test Manager for a typically uncontrolled project last year, Franck Mignet computed a first estimate using structured preparation and test design which both suffered from a high margin of error and was exceeding by 50% the required delivery date. The dilemma was solved by using an innovative mix of agile and structured testing. This presentation will present the solution which was used to bring the project under control, reduce the error margin on the planning and optimize the cost. The key idea was to start with a phase of Exploratory Testing supported by test sheets, which served as an analysis phase. It simultaneously allowed the test team to rapidly find a large number of major defects. This phase represented about 45% of the total test effort. As a result of the work of the first phase, the costs of the second phase, consisting of scripted testing, was divided by two. Using this method, team members were able to re-gain control of the project, delivering with a delay equivalent to 10% of the test time, which we could announce at mid project. This method was later reused during a second, related, project. Looking back at the two projects, the presentation will also present the lessons learned from this experience and indicate some difficult points requiring attention. Biography Entering the world of software testing in 2000, Franck Mignet built up experience in the telecommunication industry as a Test Manager and Test Process Improvement specialist. Since 2005, he has worked for Cimsolutions (www.cimsolutions.nl) carrying out assignments as a Senior Tester and Test Manager for clients. Convinced of the importance of methodologies and test
strategies, he approaches this field without dogmatic ideology, adapting to the context of the project.
CIMSOLUTIONSAutomation for Industry & Business
Case study: test planning and cost optimization in an uncontrolled project
Franck Mignet
Learn, createand make
it work
TestNet, June 22nd, 2009 [1] CIMSOLUTIONSAutomation for Industry & Business
Who Am I
Franck Mignet, 34, FrenchClimbing, AI, jazz, kayak10 years of software testing Industry Test Manager, Test Process consultant.
[2]
CIMSOLUTIONSAutomation for Industry & Business
Who Am I
Franck Mignet, 34, French.Climbing, AI, jazz, kayak.10 years of software testing. Industry. Test Manager, Test Process consultant.
I believe in adaptation, not in ideology
[2] CIMSOLUTIONSAutomation for Industry & Business
Overview
� System - Project – Problem� Chosen solution� Planning and estimate� Results� Adaptation, Challenges� The Next project… making progress?� When can you use this method?� Conclusion
[3]
CIMSOLUTIONSAutomation for Industry & Business
The system
Internal customer
WEB GUI
[4] CIMSOLUTIONSAutomation for Industry & Business
The system
Internal customer
SERVER
MS2003 Server
DB
WEB GUI
[4]
CIMSOLUTIONSAutomation for Industry & Business
The system
Internal customer
SERVER
MS2003 Server
C++ App
DB
WEB GUI
[4] CIMSOLUTIONSAutomation for Industry & Business
The system
Internal customer
SERVER
MS2003 Server
Output Files
(critical)
C++ App
DB
WEB GUI
[4]
CIMSOLUTIONSAutomation for Industry & Business
The system
Internal customer
SERVER
MS2003 Server
Output Files
(critical)
C++ App
DB
SCOPE
WEB GUI
[4] CIMSOLUTIONSAutomation for Industry & Business
Status of the project at the beginning
[5]
CIMSOLUTIONSAutomation for Industry & Business
Status of the project at the beginning
“Code is already complete”
[5] CIMSOLUTIONSAutomation for Industry & Business
Status of the project at the beginning
“Code is already complete”Requirements: “oral tradition”Components Design = 2 x A4
+
[5]
CIMSOLUTIONSAutomation for Industry & Business
Status of the project at the beginning
“Code is already complete”Requirement: “oral tradition”Components Design = 2 x A423 Functionalities in the Project Plan
…… 51 implemented
+ + =
[5] CIMSOLUTIONSAutomation for Industry & Business
Status of the project at the beginning
“Code is already complete”Requirement: “oral tradition”Components Design = 2 x A423 Functionalities in the Project Plan
…… 51 implemented
+ + =
[5]
CIMSOLUTIONSAutomation for Industry & Business
Estimating…
Trying to use the structured approach
RegressionExecutionRamp up + analysis Specification
45 man.weeks
[6] CIMSOLUTIONSAutomation for Industry & Business
Estimating…
Trying to use the structured approachExpected Delivery
RegressionExecutionRamp up + analysis Specification
30 man.weeks
[7]
CIMSOLUTIONSAutomation for Industry & Business
Estimating…
Trying to use the structured approach
And the management’s reaction
Expected Delivery
RegressionExecutionRamp up + analysis Specification
30 man.weeks
[7] CIMSOLUTIONSAutomation for Industry & Business
Estimating…
Trying to use the structured approach
And the management’s reaction
Expected Delivery
RegressionExecutionRamp up + analysis Specification
30 man.weeks
[7]
CIMSOLUTIONSAutomation for Industry & Business
Back to the “planning” board
We have to “discover” the requirementsAnalysis/Test design/ramp up cost too highTest execution starts too late => too risky
[8] CIMSOLUTIONSAutomation for Industry & Business
Back to the “planning” board
We have to “discover” the requirementAnalysis/Test design/ramp up cost too highTest execution starts too late => too risky
[8]
CIMSOLUTIONSAutomation for Industry & Business
Exploratory Testing:
One of the Agile techniquessimultaneous learning, test design and test execution
[9] CIMSOLUTIONSAutomation for Industry & Business
Exploratory Testing:
One of the Agile techniquessimultaneous learning, test design and test execution
Discover, learn, use attacksTester’s thought process and creativityNo scripted tests
[9]
CIMSOLUTIONSAutomation for Industry & Business
Exploratory Testing:
One of the Agile techniquessimultaneous learning, test design and test execution
Discover, learn, use attacksTester’s thought process and creativityNo scripted tests
Strong/Weak points
[9] CIMSOLUTIONSAutomation for Industry & Business
Supporting processes:
Test sheets“Done” , “Test Ideas”
[10]
CIMSOLUTIONSAutomation for Industry & Business
Supporting processes:
Test sheets“Done” , “Test Ideas”
Agile planning (SCRUM like)Burn down, self-organizing
[10] CIMSOLUTIONSAutomation for Industry & Business
Supporting processes:
Test sheets“Done” , “Test Ideas”
Agile planning (SCRUM like)Burn down, self-organizing
Whiteboard/ “Kanban”for tasks & communication
[10]
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
Full ExecutionEstimate
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
Full ExecutionEstimate
TPA
Structuredestimate
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
Full ExecutionEstimate
Experiment
Walkthrough
TPA
Structuredestimate
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
Full ExecutionEstimate
Experiment
Walkthrough
TPA
Structuredestimate
Sampling
CIMSOLUTIONSAutomation for Industry & Business
Input for Estimation:
[11]
ExploratoryEstimate
Full ExecutionEstimate
Additional inputs:
Product Knowledge
Product History
Tools needed
Experience
Experiment
Walkthrough
TPA
Structuredestimate
Sampling
CIMSOLUTIONSAutomation for Industry & Business
Estimating Exploratory Phase :
[12]
ExploratoryEstimate
Full ExecutionEstimate
x
Sampling (~ 30%?)
ExplorationRatio
CIMSOLUTIONSAutomation for Industry & Business
Estimating Exploratory Phase :
[12]
ExploratoryEstimate
Full ExecutionEstimate
~x
1-2vs2-1
Sampling (~ 30%?)
ExplorationRatio
Re-Balancing(Sequence)
CIMSOLUTIONSAutomation for Industry & Business
Estimating Exploratory Phase :
[12]
ExploratoryEstimate
Full ExecutionEstimate
~x +
1-2vs2-1
20% + Tools
Sampling (~ 30%?)
ExplorationRatio
DiscoveryFactor
Re-Balancing(Sequence)
CIMSOLUTIONSAutomation for Industry & Business
Estimating Exploratory Phase :
[12]
ExploratoryEstimate
Full ExecutionEstimate
~x + +
1-2vs2-1 30%
20% + Tools
Sampling (~ 30%?)
ExplorationRatio
DiscoveryFactor
Re-Balancing(Sequence) Debug
Factor
CIMSOLUTIONSAutomation for Industry & Business
Estimating Exploratory Phase :
[12]
ExploratoryEstimate
Full ExecutionEstimate
~x + + x
1-2vs2-1 30%
20% + Tools
Sampling (~ 30%?)
Day/6h
ExplorationRatio
DiscoveryFactor
Re-Balancing(Sequence) Debug
Factor
Efficiency
All-inclusive estimates!!!CIMSOLUTIONSAutomation for Industry & Business
The solution:
Defects
Exploratory Testing
[13]
CIMSOLUTIONSAutomation for Industry & Business
The solution:
Test Sheets
Defects
Exploratory Testing
[13] CIMSOLUTIONSAutomation for Industry & Business
The solution:
Test Sheets
Defects
Structured testingExploratory Testing
Defects
Test Scripts
[13]
CIMSOLUTIONSAutomation for Industry & Business
The solution:
Test Sheets
Defects
RegressionStructured testingExploratory Testing
Defects
30 man.weeks
Test Scripts
[13] CIMSOLUTIONSAutomation for Industry & Business
Did it work? Yes it worked!
RegressionStructured testingExploratory Testing Delay
Delay announced
33 man.weeks
10%
[14]
CIMSOLUTIONSAutomation for Industry & Business
Did it work? Yes it worked!
RegressionStructured testingExploratory Testing Delay
Delay announced
33 man.weeks
10%
[14]
~250 defects found, ~170 fixed4 defects found during acceptance, 1 critical1 new defect found in 6 month
Variability of estimates per functionality
CIMSOLUTIONSAutomation for Industry & Business
Adaptation for the team:
TestersLooking inside the productTrusting the methodologyTrusting the team
[15]
CIMSOLUTIONSAutomation for Industry & Business
Adaptation for the team:
TestersLooking inside the productTrusting the methodologyTrusting the team
Project ManagerUnderstanding the approachDelegating controlAvoiding over-reactions
[15] CIMSOLUTIONSAutomation for Industry & Business
Challenges for the Test Manager:
[16]
CIMSOLUTIONSAutomation for Industry & Business
Challenges for the Test Manager:
Keeping an eye on everythingVision, driving development…
[16] CIMSOLUTIONSAutomation for Industry & Business
Challenges for the Test Manager:
Keeping an eye on everythingVision, driving development…
Knowing what not to testConvergence
[16]
CIMSOLUTIONSAutomation for Industry & Business
Challenges for the Test Manager:
Keeping an eye on everythingVision, driving development…
Knowing what not to testConvergence
Risk assessmentProduct knowledgeIntuition
[16] CIMSOLUTIONSAutomation for Industry & Business
The next project:Subset of previous system:
Migration + Top priority functionalities
Similar Situation
Same MethodExploratory phase, structured phase, regression
Similar results2 man x weeks late (on 22 total) 55% of costs spent in exploratory phase
[17]
CIMSOLUTIONSAutomation for Industry & Business
Lessons learned after 2 projects:
Learning curve complete Reusing tools
[18] CIMSOLUTIONSAutomation for Industry & Business
Lessons learned after 2 projects:
Learning curve complete Reusing tools
Reliability of estimatesHigh variabilityDevelopment quality
[18]
CIMSOLUTIONSAutomation for Industry & Business
Lessons learned after 2 projects:
Learning curve complete Reusing tools
Reliability of estimatesHigh variabilityDevelopment quality
Long term issuesProduct and Process Improvement
[18] CIMSOLUTIONSAutomation for Industry & Business
When can you use this approach?Time of involvement Req. /Scope
definition
Domain Knowledge
Quality
Lead time
[19]
CIMSOLUTIONSAutomation for Industry & Business
When can you use this approach?Time of involvement Req. /Scope
definition
Domain Knowledge
Quality
Lead time
Structured.
[19] CIMSOLUTIONSAutomation for Industry & Business
When can you use this approach?Time of involvement Req. /Scope
definition
Domain Knowledge
Quality
Lead time
Structured.
Our case.
[19]
CIMSOLUTIONSAutomation for Industry & Business
When can you use this approach?Time of involvement Req. /Scope
definition
Domain Knowledge
Quality
Lead time
Structured.
Our case.
[19] CIMSOLUTIONSAutomation for Industry & Business
When can you use this approach?Time of involvement Req. /Scope
definition
Domain Knowledge
Quality
Lead time
Structured.
Our case.
With reviews.
[19]
CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring project back on track and reduce costs and lead time
[20] CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring project back on track and reduce costs and lead timeAgile and Structured testing can work together
[20]
CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring project back on track and reduce costs and lead timeAgile and Structured testing can work togetherAgile can be the way to structure
[20] CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring a project back on track and reduce costs and lead timeAgile and Structured testing can work togetherAgile can be the way to structure
[20]
CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring a project back on track and reduce costs and lead timeAgile and Structured testing can work togetherAgile can be the way to structure
[20] CIMSOLUTIONSAutomation for Industry & Business
Conclusion:
This is the solution to bring a project back on track and reduce costs and lead timeAgile and Structured testing can work togetherAgile can be the way to structure
[20]
CIMSOLUTIONSAutomation for Industry & Business
Learn, createand make
it work
[21]
top related