future of qa
TRANSCRIPT
The Future of Testing / QAThe Future of Testing / QA
Richard BornetRichard [email protected]@estsys.com
416-895-7176416-895-7176www.scenariotester.comwww.scenariotester.com
Can we predict the future?Can we predict the future?
• Most people say no.Most people say no.
• We can predict patterns?We can predict patterns?
• Future is not just passive, we can Future is not just passive, we can affect the future!affect the future!
History of World in One History of World in One SlideSlide• You can predict the future. But you have You can predict the future. But you have
to look at the long term big picture.to look at the long term big picture.
• Stone Age → Agriculture Age → Industrial Stone Age → Agriculture Age → Industrial Age → Information AgeAge → Information Age
• The one common element across the ages The one common element across the ages is CHANGE and change is speeding-up.is CHANGE and change is speeding-up.
• The reason for change is innovation.The reason for change is innovation.
How Innovation Changes How Innovation Changes WorkWork• Home workers Home workers
– Manual Manual – Cottage IndustryCottage Industry
• Central WorkplaceCentral Workplace– Manual WorkManual Work– ToolsTools– Assembly line for component assemblyAssembly line for component assembly– Robots / AutomationRobots / Automation
• Unit Cost Decreased. Unit Cost Decreased. • Fewer individuals involved in creating the product.Fewer individuals involved in creating the product.
So Where Are We in QASo Where Are We in QA• Manual with toolsManual with tools
– WordWord– ExcelExcel– Defect-tracking systemsDefect-tracking systems– Requirements managersRequirements managers– Test automationTest automation
• Change in QA will follow traditional growth Change in QA will follow traditional growth patterns.patterns.
• Future GoalFuture Goal– Much higher productivity and efficiency.Much higher productivity and efficiency.– Significantly improved ROISignificantly improved ROI
Return on InvestmentReturn on Investment
• ROI will be the central driving force ROI will be the central driving force behind IT budgets in next decade.behind IT budgets in next decade.
• QA will have to show a +ve ROI.QA will have to show a +ve ROI.
““Business units will control virtually all Business units will control virtually all discretionary IT spending.discretionary IT spending.””
GartnerGartner
Return on InvestmentReturn on Investment
““Software defects are costing the U.S. Software defects are costing the U.S. economy 60 billion dollars.”economy 60 billion dollars.” National National Institute of Standards and Technology Institute of Standards and Technology
““It is much more cost effective to catch a It is much more cost effective to catch a bug in test than it is in production. About bug in test than it is in production. About half the cost.”half the cost.”
““Quality must be raised without incurring Quality must be raised without incurring additional costs.”additional costs.”
Moving to an improved ROIMoving to an improved ROI
• Organizations are employing Organizations are employing strategies to increase ROI.strategies to increase ROI.
• If catching the defect in QA is more If catching the defect in QA is more cost-effective than finding it in cost-effective than finding it in production then:production then:– Finding it before QA will be even more Finding it before QA will be even more
cost effectivecost effective– Not creating it in the 1Not creating it in the 1stst place will be place will be
preferred.preferred.
So how is QA going to So how is QA going to change?change?• We PredictWe Predict
– More testing by End user 5%More testing by End user 5%
– More testing by Developers 5%More testing by Developers 5%– Significantly shorter development cycles 7%Significantly shorter development cycles 7%– More knowledgeable QA Resources 7%More knowledgeable QA Resources 7%– Increased appreciation for QA Increased appreciation for QA 8% 8%– Improved Processes 14%Improved Processes 14%– More Test Automation 31%More Test Automation 31%
Move to Packaged Move to Packaged SoftwareSoftware• Less in-house development.Less in-house development.
• Buy packages which will execute the Buy packages which will execute the functionsfunctions– SAP, BPCS , PeopleSoft.SAP, BPCS , PeopleSoft.
• Fewer and larger vendors.Fewer and larger vendors.
• Customize for specific business.Customize for specific business.
What this means for QAWhat this means for QA
• Vendors provides a stable application. Vendors provides a stable application. • Vendors thoroughly test application.Vendors thoroughly test application.• QA will need to do less testing.QA will need to do less testing.• TestTest
– A good regressionA good regression– Custom adaptationsCustom adaptations– Concentrate on whether business needs Concentrate on whether business needs
are being met. are being met.
Pre-built componentsPre-built components
• Common model in hardware.Common model in hardware.• Not just objects.Not just objects.• Whole functional modules.Whole functional modules.• Storage facility for On Demand videosStorage facility for On Demand videos
– Database may come for IBMDatabase may come for IBM– Web Services through .NetWeb Services through .Net– Logging and monitoring system through Logging and monitoring system through
another company.another company.
What this means for QAWhat this means for QA
• Vendors provides stable modules.Vendors provides stable modules.• Less testing.Less testing.• Test Test
– A good regressionA good regression– Code that is built in-houseCode that is built in-house– Concentrate on whether business needs are Concentrate on whether business needs are
being met.being met.
• Don’t test the video management system, Don’t test the video management system, it should work.it should work.
OutsourcingOutsourcing
• Bring in a third party to develop and Bring in a third party to develop and maintain your systems.maintain your systems.
• Outsourcers bring:Outsourcers bring:– Stable codeStable code– Libraries of re-usable codeLibraries of re-usable code– Applications tested in various companiesApplications tested in various companies– Costs shared between clientsCosts shared between clients– Faster turnaround.Faster turnaround.
What this means for QAWhat this means for QA
• Outsourcer provides stable code.Outsourcer provides stable code.
• Outsourcers can perform the QA Outsourcers can perform the QA functions.functions.
• If testing in house test less testing. If testing in house test less testing. – Outsourcer responsible for providing Outsourcer responsible for providing
solid codesolid code– Concentrate on whether business needs Concentrate on whether business needs
are being met. are being met.
Offshore developmentOffshore development
• Same as outsourcing except Same as outsourcing except theoretically cheaper.theoretically cheaper.
• Again, you can demand stable and Again, you can demand stable and defect free code.defect free code.
• What this means for QAWhat this means for QA– Less testingLess testing
Move testing to the Move testing to the developersdevelopers
• Some companies have moved testing Some companies have moved testing responsibilities onto their developersresponsibilities onto their developers
• QA departments shut down or QA departments shut down or downsizeddownsized
Move testing to the Move testing to the developersdevelopers
• ProblemsProblems– Programmers over worked.Programmers over worked.– Programmers may not know the business.Programmers may not know the business.– Programmers hate to test.Programmers hate to test.– Programmers have time pressures.Programmers have time pressures.– Person who build should not have sole Person who build should not have sole
responsibility for testing.responsibility for testing.– Programmers may not know how to create and Programmers may not know how to create and
execute thorough testing.execute thorough testing.
Move testing to the Move testing to the developersdevelopers
• AdvantagesAdvantages– Moves testing earlier in development Moves testing earlier in development
cycle.cycle.– Cost effective to catch bugs earlier.Cost effective to catch bugs earlier.– Catch bugs at development stage even Catch bugs at development stage even
more cost effective.more cost effective.– Programmers like to use tools.Programmers like to use tools.
Move testing to the Move testing to the developersdevelopers
• Extreme programmingExtreme programming– Programmers design and build in testsProgrammers design and build in tests
• Large use of Unit Testing toolsLarge use of Unit Testing tools– Programmers build their own toolsProgrammers build their own tools– Many tools are available often for free.Many tools are available often for free.
Test AutomationTest Automation
• Many organizations try to automate.Many organizations try to automate.
• It has huge potential.It has huge potential.
• Many fail (78%).Many fail (78%).
• Almost never is it complete testing.Almost never is it complete testing.
• To often it is used for Smoke Tests.To often it is used for Smoke Tests.
Traditional Development Cycle Traditional Development Cycle Courtesy of BorlandCourtesy of Borland
New ModelNew Model
Efficiency Improvements -Efficiency Improvements -Define / Requirements StageDefine / Requirements Stage
• Huge cost savings by catching problems earlyHuge cost savings by catching problems early– $65 per defect at Requirements stage$65 per defect at Requirements stage– $1000 to $3000 at System test$1000 to $3000 at System test– $10,000 or more in Production$10,000 or more in Production
• Test the requirementsTest the requirements– Proof read the requirementsProof read the requirements
• There are technologies available now that will:There are technologies available now that will:– Give you 100 % test coverageGive you 100 % test coverage– Automatically generate test casesAutomatically generate test cases– Put testers and developers on same pagePut testers and developers on same page
• Better tools for management of requirements.Better tools for management of requirements.• In the future In the future
– Process will be enhanced, simplified and automated more.Process will be enhanced, simplified and automated more.
Efficiency Improvements -Efficiency Improvements -Design StageDesign Stage
• Designers and Developers will build Designers and Developers will build software mindful of how we are software mindful of how we are going to test it. going to test it. – Build tests into the application.Build tests into the application.– Build hooks for testing into application.Build hooks for testing into application.
Efficiency ImprovementsEfficiency Improvements
• Development StageDevelopment Stage– Unit test automationUnit test automation– AutomationAutomation
• Deployment StageDeployment Stage– AutomationAutomation
• ProductionProduction– AutomationAutomation
TEST AUTOMATIONTEST AUTOMATION
• Single biggest change in QA, will be Single biggest change in QA, will be the amount and type of test the amount and type of test automationautomation
• This will include:This will include:– New types of testingNew types of testing– New toolsNew tools– New strategiesNew strategies
Test Automation – Current Test Automation – Current StateState
• Stage 1: Record and PlaybackStage 1: Record and Playback
• Stage 2: Write CodeStage 2: Write Code
• Stage 3: ModularityStage 3: Modularity
• Stage 4: Data Driven TestingStage 4: Data Driven Testing
Test Automation – Who Does Test Automation – Who Does WhatWhat
• Stage 5: Teach the testers how to program.Stage 5: Teach the testers how to program.
• Stage 6: Make the automation tools simplerStage 6: Make the automation tools simpler• Tree view diagrams.Tree view diagrams.
• ““WinRunner Lite”.WinRunner Lite”.
• Stage 7: Teach programmers the business.Stage 7: Teach programmers the business.
• Stage 8 : Bring in Automation experts. Stage 8 : Bring in Automation experts. – Hire a programmer.Hire a programmer.– Bring in an external consultants.Bring in an external consultants.– Create a test automation team.Create a test automation team.
Test Automation – Current Test Automation – Current StateState
• None of these stages fit the way None of these stages fit the way testers work.testers work.– Having to code or handing off your work Having to code or handing off your work
and responsibilities to a 3and responsibilities to a 3rdrd party is not party is not acceptable.acceptable.
• Almost never is it complete testing.Almost never is it complete testing.
• No wonder many automation efforts No wonder many automation efforts fail. (78%).fail. (78%).
Stage 9: Empower the Stage 9: Empower the TESTERS TESTERS
• Give testers an interface to enter test Give testers an interface to enter test cases / test scripts.cases / test scripts.
• Press a button to execute them.Press a button to execute them.
• Needs to be simple and powerful.Needs to be simple and powerful.
• Create test scripts before the Create test scripts before the application is even built.application is even built.
Web ServicesWeb Services
• Endorsed by Microsoft, Borland, Sun and Endorsed by Microsoft, Borland, Sun and IBM.IBM.
• It is the ability to provide hooks to an It is the ability to provide hooks to an application which can be utilized by application which can be utilized by external applications.external applications.
• CIO like it because they can keep their CIO like it because they can keep their legacy systems.legacy systems.
• Users like it because they can get new and Users like it because they can get new and useful interfaces.useful interfaces.
Web Services – Testing Web Services – Testing
• Multiple InputsMultiple Inputs– GUI application, Palm Pilot, your phone, GUI application, Palm Pilot, your phone, – your TV, Voice, special devicesyour TV, Voice, special devices
• Testing will be more interesting as Testing will be more interesting as new forms of accessibility are new forms of accessibility are developed.developed.
• Component model of software.Component model of software.
Web Services – TestingWeb Services – Testing
• Web Services creates a stable Web Services creates a stable environment.environment.
• Testing components.Testing components.• Testing is split between Testing is split between
– Testing the web serviceTesting the web service– Testing the GUI / applicationTesting the GUI / application– Testing the business rules.Testing the business rules.
• New simple tools.New simple tools.
New ModelNew Model
FUTURE: Building the interface FUTURE: Building the interface between the tests and the app.between the tests and the app.
• Currently you Point and ClickCurrently you Point and Click
• Building the scripts automaticallyBuilding the scripts automatically– Screen scrapersScreen scrapers– Web services test tools will pick up the Web services test tools will pick up the
WSDL file.WSDL file.
FUTURE: Building the interface FUTURE: Building the interface between the tests and the app.between the tests and the app.
• Design StageDesign Stage– Developers when they design Developers when they design
application, design hooks into the application, design hooks into the application to allow for data to be application to allow for data to be passed to and from the app.passed to and from the app.
– Design to expose the application to an Design to expose the application to an outside test tool.outside test tool.
– Generally straightforward.Generally straightforward.
FUTURE: Building hooks to test FUTURE: Building hooks to test automation into the application automation into the application
• Application arrives automatically Application arrives automatically automated.automated.
• Test tool automatically connects test Test tool automatically connects test cases and app.cases and app.
• Build extra scripts in VB and Java and Build extra scripts in VB and Java and compile code.compile code.
• Same cost no matter how many Same cost no matter how many testers.testers.
• Cheaper /more control.Cheaper /more control.
FUTURE: Building testing right FUTURE: Building testing right into the application. into the application.
• Hardware manufacturers have been Hardware manufacturers have been doing this for ages.doing this for ages.
• In Production we call it monitoring.In Production we call it monitoring.
• eBay.eBay.
Future of QAFuture of QA
• Test in ProductionTest in Production– Automatic monitors for functionality and Automatic monitors for functionality and
performance.performance.– Programs, which can execute tests both Programs, which can execute tests both
automatically and on request.automatically and on request.
• Same set-up can be used at Same set-up can be used at Deployment and Development stageDeployment and Development stage
Future of QAFuture of QA
• Get requirements right by thoroughly testing them.Get requirements right by thoroughly testing them.• Automating the writing of the test cases and test Automating the writing of the test cases and test
scripts.scripts.• Automated environment for testing Automated environment for testing
– Automatic monitors for functionality and performance.Automatic monitors for functionality and performance.– Programs, which can execute tests both automatically and Programs, which can execute tests both automatically and
on request.on request.
• Deployment testing is automated.Deployment testing is automated.• Production is automatically monitored and testedProduction is automatically monitored and tested• Automated system allows for Troubleshooting and Automated system allows for Troubleshooting and
Diagnostics. Diagnostics.
Future QA – Solves Future QA – Solves problemsproblems
• Increased ROI.Increased ROI.– Decreased costs.Decreased costs.– Complete test coverage.Complete test coverage.– Faster turnaround.Faster turnaround.
• We need to reinvent ourselves or we We need to reinvent ourselves or we will become obsolete.will become obsolete.
• Huge opportunities for QA.Huge opportunities for QA.
New ModelNew Model
Contact InformationContact Information
Richard BornetRichard [email protected]@estsys.com
Web siteWeb sitewww.scenariotester.comwww.scenariotester.com
PresentationPresentationwww.scenariotester.comwww.scenariotester.com and go to Articles and go to Articles