©2009 Ness Technologies – Proprietary and Confidential

How testing and testers evolved within the time and benefitsof testing tools during mature testing lifecycle.

Testwarez 2009, Krakow

Testing Lifecycle: Don’t be a fool, use a proper tool!

Zdeněk Grössl & Lucie Riedlová

20. 10. 2009

www.ness.com

Our aim: to use the mature Testing Lifecycle

Testing is strategic for us

Over 2.500 testers (18 countries worldwide) sharing their corporate knowledge

Different modes – test labs, consulting, custom development, SAAS

Proof? Satisfied customers around the world• Telco• IT• Public and Military• In CZ customers from TOP 100

Let’s see that “Lifecycle”

Testing LifecycleTesting Lifecycle

www.ness.com

What do we mean by “Lifecycle” :-)What do we mean by “Lifecycle” :-)

www.ness.com

From random clicks to Testing LifecycleFrom random clicks to Testing Lifecycle

test coordinadionprogress of testingconnection to management

test casesRQ coveragetest scripts and test datamanagementautomatization

managing test teamreporting about status

test functionality and requirements before

release bug report

inefficient testingdifficult to reproduce tests

uncoordinatefd action

feedback on test processimputs for next iteration

Tools can help a lot

!How to handle this

?

www.ness.com

Test ManagementTest Management

www.ness.com

Testing tools – Test managementTesting tools – Test managementDynamic changes on smaller projectDynamic changes on smaller project

Project:Financial sectorTesting implemented from scratch, about 12 team membersDynamic development processTesting documentation in Excel sheets

Sudden change: Increase of RQs40% new people hiredIssues with versions of Excel sheetsInformation about changes was not reliableA need to formalize testing process

Solution: Tool for test ManagementEasy to setupOnline accessSuitable for smaller teamsKeeps history of records

SPIRA TEAM was selected as pilot

Benefits:Testing was formalizedHistory and more detailed information about testing processSPIRA extended to other projects

www.ness.com

Testing tools – Test management – screenshotsTesting tools – Test management – screenshots

Example: SPIRA TEAM

Project Home

My Page

www.ness.com

Testing tools – Test managementTesting tools – Test managementToo much extra work at world’s HW leaderToo much extra work at world’s HW leader

Project:UAT testing of world-wide application for Sales (12.000 users)Two successive releases were tested, about 1 year between releasesRole: Regional Test Coordinator (40 testers), reporting about progressComplete Worldwide team: 140 members

First release: Shared Excel filesToo much extra workThree different locations to login into before testing each day Reporting for management: combine excel sheets and reports from BugzillaTake care about testers (motivate, coach, inform about bug fixes…)Extremely challenging!

Second release: HP Mercury QCA pro tool: all in one; no multiple loginsSimple administration of all modulesUser-friendly testing Easy overview of testing progress,Bug tracking and traceabilitySophisticated reports for management.

Benefits:More effective testing on such a large project. Clear overview for managementFocus on testing, not on extra activities!

www.ness.com

Testing tools – Test management – screenshotsTesting tools – Test management – screenshots

Example: Evolution from Excel to Mercury QC

www.ness.com

Testing tools – Test managementTesting tools – Test managementSummarySummary

www.ness.com

Test AnalysisTest Analysis

www.ness.com

Testing tools – Test Analysis and DesignTesting tools – Test Analysis and DesignWe got lost in Test CasesWe got lost in Test Cases

Project:Financial sector (saving bank)Deliver complete QA for new module of client’s internal systemOne customer, three dislocated suppliersTotally about 70 new RQs (about ¼ of the whole system)

All in Excel sheets (RQ, TC, test data)

Excel sheet too complexDifficult to review test coveragePoor reporting optionsIssues with access to filesSeveral errors occurred

Requirements on the toolAchieve and review RQ coverage easilySupports evaluation of TCsUser-friendly environmentQuick and easy setupAccessible worldwideSPIRA TEAM selected

Benefits:Administrative / human errors avoidedGreat overview for test analystModule delivered with remaining 6 minor bugs (totally over 140)Transparent for client and increase of our credit

www.ness.com

Testing tools – Test Analysis – screenshotsTesting tools – Test Analysis – screenshots

Example: SPIRA TEAM

Map TC to RQ

RQ coverage overview

www.ness.com

Testing tools – Test Analysis & DesignTesting tools – Test Analysis & DesignSummarySummary

www.ness.com

Test DesignTest Design

15

www.ness.com

Testing tools – Test Design and Analysis Testing tools – Test Design and Analysis Regression testing under stressRegression testing under stress

Project:SPIRA TEAM as testing tool on the project (Finance)Releases 8x per year, hot-fixes need to be released within 8 hoursMuch time spent on regression testing

Need of change:Regression testing not 100% effective under stressAn error was delivered to the customerMore formalized regression tests were not good solution

We need tool for automated testingOptions: Test Complete, Win RunnerSituation on project did not enable it. Good luck: our SW-coder recommended Selenium – add-on for Firefox

Benefits:Future errors avoidedConnection with SPIRA to automatically record test runsIt is useful to discuss testing issues with developers as well

www.ness.com

Testing tools – Test Testing tools – Test Design Design – – screenshotsscreenshots

Example: SPIRA TEAM together with Selenium► Selenium IDE – write and run tests in Firefox► Selenium Remote Control – runs tests in multiple browsers and platforms

(IE7, Safari, Opera, Firefox, Windows, Linux, Solaris, others)

www.ness.com

Testing tools – Test Design and Analysis Testing tools – Test Design and Analysis More than was requested (by Silvia More than was requested (by Silvia ValientovValientováá))

Project:Biggest insurance company in CZ33 methods, 6 web services were tested separately

What was requested?Test the methods like a processCustomer: Buy us a new testing tool!SoapUI was known tool for customer

Why SoapUIThe old tool has requested functionalityProperty Transfer and Xpath Assertion WizardLink responses with requests Support of E2E testing

Benefits:Client can now easily perform this testingNo extra costs required to purchase licensesEasy upgrade from Integration testing to E2E testingWe found what customer really NEEDS, not what he just REQUESTS

www.ness.com

Testing tools – Test Testing tools – Test Design Design – – screenshotsscreenshots

Example: The SOAP UI Story► Used for Web Service Testing

www.ness.com

Testing tools – Test Design and AnalysisTesting tools – Test Design and AnalysisSometimes just look around for a tool!Sometimes just look around for a tool!

Project:Internetbanking system (Lead Czech saving bank)Load testing was newly requiredTested system was developed by NESS Czech

Requirements on the toolSimulate simultaneous access of 300 users per 10 minutesMinimum time and costs to set up a toolRestriction on client’s side

Why Visual Studio 2005 (for Testers)?Tested application was programmed in this environmentDevelopers were familiar with the toolQuick implementation and results in one week

Benefits:Developers can be very good test designersScope of our services for the client broadened

www.ness.com

Testing tools – Test Design & AnalysisTesting tools – Test Design & AnalysisSummarySummary

automated testing

language & environment independent

connection to other tools

www.ness.com

Test ExecutionTest Execution

22

www.ness.com

Testing tools – Test ExecutionTesting tools – Test ExecutionTesting scope broadened (by Jaroslav StrharskTesting scope broadened (by Jaroslav Strharskýý))

Project:Major Telco company in CZExcel for RQs and TCsOriginally only 2 testers planned

Need of change:25 testers involvedExcel files too complicatedSeveral bugs got misrouted

Testing with HP Mercury Quality CenterTesters complained about difficulty of the tool and level of formalismMonitoring of users was necessaryFormalized and free tests were put in place

Benefits:Inform business testers about fixed bugs they discovered: they will be satisfiedTraining for testers takes approx. the same time regardless of a toolTesting with this tool is as quick as without a tool. But consider the benefits!

www.ness.com

Testing tools – Test Testing tools – Test Execution Execution – – screenshotsscreenshots

HP Mercury Quality Center

www.ness.com

Testing tools – Test ExecutionTesting tools – Test ExecutionNo time for new toolsNo time for new tools

Project:TCs and bugs maintained in Word and ExcelCommunication to developers was face-to-faceBugs were fixed almost immediately

Need of change:Model does not work under stressBig volume of reported bugs (+300%)Management required reporting

Only 3 weeks to deliver the testing!

Why JIRA from Atlassian?JIRA is used in NESS as internal tracking tool (over 750 users)Every employee is familiar with itSkillful administrators

Benefits :No more misunderstandingsReports for managementAll set up in less than 1 day

www.ness.com

Testing tools – Test Testing tools – Test Execution Execution – – screenshotsscreenshots

JIRA by Atlassian

www.ness.com

Testing tools – Test ExecutionTesting tools – Test ExecutionSummarySummary

www.ness.com

Summary? There is no perfect tool!► Every tool suits something► If the tool is TOO complex and TOO perfect: Is it still easy to maintain?

Is there anything else we should be aware of? Answer these questions about testing tools:► Easy to install?► Friendly to use? ► Even for non-IT users?► Any additional features?

• Notifications,• Attachments, screenshots• Reporting options• Upgrades• …

► Could you customize the tool?► When you tried it, does it serve you well?

Overall summaryOverall summary

www.ness.com

What do we use?► We – Lucie and Zdenek – like SPIRA (and several our customers too )

We use tools from our partners (Atlassian, HP, IBM, Inflectra, Microsoft…)

Use tool best for the purpose and for the client ► Excel? OK, but educate clients! And remember their maturity!

Which tool should you use? See questions on previous slide!► Summarize your needs

Don’t you know your needs?► Talk to someone who helps you define them

Most powerful tool: OPEN MINDS!

And what about testing at NESS Czech?And what about testing at NESS Czech?

www.ness.com

Complete QA solution in one package

Server and thin-client-based application

Very user-friendly environment

Rich reporting options

Document repository

Option to connect to LDAP

Positive feedback from clients

Even from business (non-IT) people

After one year over 30 projects and over 60 external users

SPIRA TEAM by Inflectra SPIRA TEAM by Inflectra

http://www.inflectra.com

www.ness.com

bug tracking

project management

agile software development

technical support and help-desk

used by more than 10 000 organizations

create any kind of issue in seconds

keep your projects and issues organized

track all activities in your project

designed with both business and technical users in mind

easy to administer and customize

Because you’ve got issuesBecause you’ve got issues

www.ness.com

Our question: What did you take from this presentation?► There is no perfect tool! ► Discuss and ask questions before you decide to use certain tool!► Ask those who know about this stuff…

We are ready for your questions…

… and for the chat about testing – later :-)

Questions and AnswersQuestions and Answers

It looks like end... but it is rather start