www.eu-etics.org

INFSOM-RI-026753

WP4: Testing tools and WP4: Testing tools and methodologies methodologies

István ForgácsIstván Forgács

4D SOFT4D SOFT

INFSOM-RI-026753

Team

• 4D SOFT’s staff – some changes4D SOFT’s staff – some changes– István Forgács - responsible for István Forgács - responsible for

– leading WP4leading WP4– 4D Soft’s own methods and tools4D Soft’s own methods and tools– Key scientific issues Key scientific issues – Cooperate with the coordinator, visiting PMBs, etc.Cooperate with the coordinator, visiting PMBs, etc.

– Anna Bánsághi, Anna Bánsághi, – Study and evaluate testing toolsStudy and evaluate testing tools

– Katalin Gergely, Katalin Gergely, (Thalmeiner Zsolt)(Thalmeiner Zsolt)– Study and evaluate testing methodsStudy and evaluate testing methods– Cooperates with Univ. of Wisconsin staffCooperates with Univ. of Wisconsin staff

– Andor Dirner Andor Dirner – DILIGENT connectionDILIGENT connection

– Gyöngyi KispálGyöngyi Kispál– Financial issuesFinancial issues

– Klára TauszigKlára Tauszig– Administrative issuesAdministrative issues

INFSOM-RI-026753

A4.1 requirements

A4.1A4.1            Collect requirements for unit, functional and Collect requirements for unit, functional and regression testing. There is a large number of research regression testing. There is a large number of research papers on the test adequacy criteria of unit testing. papers on the test adequacy criteria of unit testing. However, there are no related tools in the practice for However, there are no related tools in the practice for most of them. We are going to collect applicable and most of them. We are going to collect applicable and practical requirements based on our knowledge, practical requirements based on our knowledge, experience and the testing tools currently available and experience and the testing tools currently available and widely used such as JUnit. The task involves the widely used such as JUnit. The task involves the requirements and applicability criteria for test requirements and applicability criteria for test planning, dynamic testing and the testing tools.planning, dynamic testing and the testing tools.

INFSOM-RI-026753

Work has been done

• General requirements for test automation General requirements for test automation – Studying the literature especially the book of Fewster and Studying the literature especially the book of Fewster and

Graham: Software Test Automation Graham: Software Test Automation – Key result: test automation should be based on good test Key result: test automation should be based on good test

methodsmethods

• Requirements for test planningRequirements for test planning– Easy to learn,Easy to learn,– Easy to use Easy to use – Easy to maintainEasy to maintain– Can be automatedCan be automated– Can be generally usedCan be generally used– Needs more invention and less manual workNeeds more invention and less manual work

INFSOM-RI-026753

Methods

• Methods have been studied Methods have been studied – Method for the whole testing processMethod for the whole testing process

– V modelV model

– Test planningTest planning– Category-partitioning (detailed description tomorrow)Category-partitioning (detailed description tomorrow)– Syntax testing – can be automated, checks correct and incorrect Syntax testing – can be automated, checks correct and incorrect

syntax syntax – Equivalence PartitioningEquivalence Partitioning – input domain is partitioned into sub – input domain is partitioned into sub

domains so that one sub domain behaves uniquely w.r.t. the errorsdomains so that one sub domain behaves uniquely w.r.t. the errors– widely usedwidely used

– disadvantagesdisadvantages• Doesn't test every input Doesn't test every input • No guidelines for choosing inputs No guidelines for choosing inputs • Heuristic based Heuristic based • very limited focusvery limited focus

INFSOM-RI-026753

Methods

• Methods have been studied Methods have been studied – Test planningTest planning

– Boundary Value AnalysisBoundary Value Analysis – together with the equivalence – together with the equivalence partitioningpartitioning

– Well-known requirements that boundaries and close to the Well-known requirements that boundaries and close to the boundaries should be testedboundaries should be tested

– The boundaries of both valid and invalid partitions are consideredThe boundaries of both valid and invalid partitions are considered

– DisadvantagesDisadvantages• Does not test all possible inputs Does not test all possible inputs • Does not test dependencies between combinations of inputsDoes not test dependencies between combinations of inputs

– Error guessing – Error guessing – the experience of the tester is used to postulate the experience of the tester is used to postulate what faults might occur, and to design tests specifically to expose what faults might occur, and to design tests specifically to expose them.them.

– These three are used together. These three are used together.

INFSOM-RI-026753

Methods

• Methods have been studied Methods have been studied – Test planningTest planning

– Cause-Effect GraphingCause-Effect Graphing:: old fashioned, not maintainableold fashioned, not maintainable– State transition diagramsState transition diagrams: widely used if the software contains : widely used if the software contains

states and different events cause the transition from one to another states and different events cause the transition from one to another example ATMexample ATM

– Random testingRandom testing – random input for test cases. The – random input for test cases. The distribution distribution (normal, uniform, etc.) to be used in the generation of random input (normal, uniform, etc.) to be used in the generation of random input values shall be based on the expected operational distribution of values shall be based on the expected operational distribution of inputs. inputs.

– Ad hoc testingAd hoc testing unfortunately it’s widely used unfortunately it’s widely used– Fuzz input testingFuzz input testing - lots of arbitrary, mainly invalid input - lots of arbitrary, mainly invalid input– Smoke testingSmoke testing: : is a cursory examination of all of the basic is a cursory examination of all of the basic

components of a software system to ensure that they wocomponents of a software system to ensure that they workrk– Equivalence testingEquivalence testing this is the best method if you have two different this is the best method if you have two different

software aiming to do the same thingssoftware aiming to do the same things

INFSOM-RI-026753

methods

• Unit, component and integration testingUnit, component and integration testing– Black box methodsBlack box methods

– Test planning methodsTest planning methods– EBIT (tomorrow)EBIT (tomorrow)

– White box methodsWhite box methods– Test bedTest bed

– Test driver – missing callersTest driver – missing callers

– Test stubs – missing calleesTest stubs – missing callees

– Coverage analyses Coverage analyses – All statement criterionAll statement criterion

– All branch criterion – most widely used minimum requirementAll branch criterion – most widely used minimum requirement

– MC/DC criterionMC/DC criterion

– Data flow criteria – better but rarely appliedData flow criteria – better but rarely applied

– Reduced path testing – cycles are tested with 0,1 and more Reduced path testing – cycles are tested with 0,1 and more iterationsiterations

INFSOM-RI-026753

methods

• System testingSystem testing

– Scripting languagesScripting languages– Capture and play back techniquesCapture and play back techniques– Static analysisStatic analysis

– Dependence analysis (slicing)Dependence analysis (slicing)– MetricsMetrics– Exception handling analysisException handling analysis– Null pointer analysisNull pointer analysis– Non-initialised variable analysisNon-initialised variable analysis

INFSOM-RI-026753

methods

• Regression testingRegression testing– Dynamic regression testingDynamic regression testing

– Full regression testingFull regression testing– Selective regression testingSelective regression testing

– Selection modification-traversing testsSelection modification-traversing tests

– Classifying regression testsClassifying regression tests– At least one output has been influencedAt least one output has been influenced– Influences through predicatesInfluences through predicates– No influences existNo influences exist

– If necessary transforming tests to be effective If necessary transforming tests to be effective

– Static regression testingStatic regression testing– Original and modified sources are availableOriginal and modified sources are available– Dependence analysis considering the modificationsDependence analysis considering the modifications– Selecting the non-expected influencesSelecting the non-expected influences

INFSOM-RI-026753

A4.1 Collect requirements for tools

• Testing tool selectionTesting tool selection– Requirements can only be collected based on existing toolsRequirements can only be collected based on existing tools– We studied tools for different purposesWe studied tools for different purposes

– Test planning - No specification-based tools have been considered Test planning - No specification-based tools have been considered studying the specification, detailed design, etc. documents for studying the specification, detailed design, etc. documents for DILIGENTDILIGENT

– CatGenCatGen

– Unit testingUnit testing– JunitJunit

– GJTesterGJTester

– Functional, regressionFunctional, regression– vTestvTest

– Static analysisStatic analysis– INDUSINDUS

INFSOM-RI-026753

A4.1 Collect requirements for tools

• Testing tool selectionTesting tool selection– Acceptance testingAcceptance testing

– AutAT (web appl)AutAT (web appl)– Accessibility testingAccessibility testing

– CSE Lite (web appl test tool)CSE Lite (web appl test tool)– A-Prompt (web appl test tool for people with disabilities)A-Prompt (web appl test tool for people with disabilities)– PowerTools HTTP proxyPowerTools HTTP proxy– PowerTools Connection TestPowerTools Connection Test– PowerTools HTTP Request EditorPowerTools HTTP Request Editor– PowerTools Expression TestPowerTools Expression Test– PowerTools Encode/DecodePowerTools Encode/Decode

– Performance testing (load, stress)Performance testing (load, stress)– OpenSTAOpenSTA– TestMakerTestMaker– GrinderGrinder– JmeterJmeter– LoadRunnerLoadRunner– AnalogAnalog

– CoverageCoverage– JavaCovJavaCov

INFSOM-RI-026753

Immediate future plan

• More testing tools to studyMore testing tools to study– FunctionalFunctional

– Qengine (performance)Qengine (performance)– Enterprise Web Test (stress, reliability)Enterprise Web Test (stress, reliability)– AppPerfect WebTest (load)AppPerfect WebTest (load)– JFuncJFunc

– ConformanceConformance– SuiteRunner (unit)SuiteRunner (unit)

– Regression, smokeRegression, smoke– ITPITP

– IntegrationIntegration– iValidatoriValidator

– PerformancePerformance– TPTPTPTP

INFSOM-RI-026753

Immediate future plan

• More testing tools to studyMore testing tools to study– ComponentComponent

– ASP Component TestASP Component Test

– Code analyserCode analyser– PMDPMD

– CoverageCoverage– JcoverageJcoverage

– UnitUnit– AppPerfect DevTestAppPerfect DevTest

INFSOM-RI-026753

Immediate future plan

• We have finished studying testing methodsWe have finished studying testing methods• Requirement analysisRequirement analysis

– Starting pointsStarting points– Methods availableMethods available– Testing tools availableTesting tools available– Testing researchTesting research– Test books for testingTest books for testing

– Setting requirementsSetting requirements– Weighting requirementsWeighting requirements– Validating and comparing requirementsValidating and comparing requirements

INFSOM-RI-026753

Deliverables

• We haven’t any scheduled deliverables yetWe haven’t any scheduled deliverables yet• Future deliverables in this yearFuture deliverables in this year

– D4.1 - Distributed test execution system (prototype) in PM06. D4.1 - Distributed test execution system (prototype) in PM06. Lead partLead participant: icipant: UoWUoW

– D4.2 - Requirements for unit, functional and regression testing in D4.2 - Requirements for unit, functional and regression testing in PM06PM06. . Lead partLead participant: 4DSicipant: 4DS

INFSOM-RI-026753

metrics

• Number Number of testing tools of testing tools downloaded and studieddownloaded and studied– Plan 20-25Plan 20-25 current status 20current status 20

• NNumber of test methods studiedumber of test methods studied– Plan 20-30Plan 20-30 current status 23current status 23

• NNumber of quality metrics identified for test methods umber of quality metrics identified for test methods and tool selectionand tool selection– Plan 15-20Plan 15-20 current status 0current status 0

INFSOM-RI-026753

Future plan

• Selection of the testing tools and methodologies based Selection of the testing tools and methodologies based on the requirements derived in the previous parton the requirements derived in the previous part

• Working outWorking out special aid on how to implement and use special aid on how to implement and use these tools/methods with clear examples explaining the these tools/methods with clear examples explaining the main difficulties of the application. main difficulties of the application.