OHJ-306x: Software Testing – Introduction to the Course Project

Part 2:Project phases 3 & 4: System testing

Matti Vuori, course assistantAntti Jääskeläinen, leading course assistant

1

29/10/12

Testing objectives and generic strategy

• Testing that the jEdit editor is suitable for production work• Find out what defects it has so they can be corrected

• Testing done at UI level

• Black box, no access to source code

• The end user's view

• Tester & test designer need to understand what is important for the user and decide based on that

• What to test

• How to test it – with what kind of tests

2

29/10/12

Two phases

• The system testing will be done in two phases (like the unit testing)• Phase 3 – planning

• Phase 4 – testing

• The version of jEdit will be changed between the phases• For testing, you will get a version which has additional defects

seeded in, so you'll get more experience in finding errors and reporting them

• More about the phases in a moment

3

29/10/12

What are the features to be tested? 1/2

• The whole editor• All its functionality, all use cases

• Start with the most important features of any text editor• Prioritization is important as exhaustive testing would take too

much time

• As in unit testing, only functional testing is done

• We look into usage of the software now, not how it has been implemented

• Think of how a programmer would use an editor• What would be the use scenarios / use cases?

• What editor functions (UI functions) of the editor are used?

4

29/10/12

What are the features to be tested? 1/2What are the features to be tested? 1/2

What are the features to be tested?

• If we had a requirement specification, it would guide us• Instead, we use the use cases and the implemented software as our

guide

• That is common in practice, in companies, too

• Requirement specifications rarely describe common functionality anyway (like how opening of files or printing really happens)

5

29/10/12

Need for proof & trust

• System testing is carefully monitored in companies • By the managers, by the customers

• Need proof that the testing

• ...is well planned – right things, with good tests

• ...is well executed – metrics, sometimes logs

• ...has covered the right things – requirements coverage (code coverage is not usually monitored at this level)

• Documentation is therefore important

6

29/10/12

Working with other people – providing information

• Unit testing is usually done by the developer her/himself• System testing may be done by a separate tester or a team• Tests may be planned and executed by different people• And managers watch it carefully• Therefore, things need to be documented so that others can

understand them• Can someone else perform the testing based on the plans?

• Can someone else correct the errors based on the testers' reports?

• Can the managers understand by reading the reports, what the state of the application is? (What risks are there if it is released to market or deployed to the customer?)

7

29/10/12

Free to choose an approach

• For the system testing phases, all team are free to choose an approach that best suits and most interests them• Much more freedom than in the unit testing phases!

• In practical testing projects, many approaches could be used and the same applies in training too

• We need people with different skills• The basic approaches to choose from are

• Approach A: Systematic, planned manual testing• Approach B: Exploratory testing• Approach C: Traditional test automation• Approach D: Model-based testing• Approach E: Combination of approaches

8

29/10/12

Approach A: Systematic, planned manual testing

• The most traditional approach and most widely used in companies• The approach well documented in books, slides

• Phase 3:• Identify the most essential functions of the application• Plan a test suite• Design test cases

• Phase 4:• Execute the test cases• Report errors• Report the testing

• Essential:• Good planning of test cases that test the most essential features of

the application• Good test planning so others can be sure that the testing has a solid

ground• Good error reports so the developers can fix the errors

9

29/10/12

Approach B: Exploratory testing 1/3

• The approach used often in agile software development• Also used in systematic approaches to supplement it• Phase 3:

• Select a set of usage scenarios• Present an understanding of what is most important in the

application to work correcly• Present your refined approach to the exploratory testing

• Phase 4:• In the context of the scenarios, test the application using the

principles of exploratory testing• Keep a test log to document what has been tested and how• A checklist ”List of some usual things to test in an application” can

be used as and aid (see the project web page)• Report the errors found• Report the testing

10

29/10/12

Approach B: Exploratory testing 2/3

• Essential:• Choosing the right context for testing – assumptions about

usage, methodology, techniques, attitude• Making observations and interpretations of the behaviour of the

application• Using good test techniques at each point where something is

suspected• Documenting the work so that others can trust that you have

done a good job• A plan• The test log

● Good error reports so the developers can fix the errors

11

29/10/12

Approach B: Exploratory testing 3/3

• Approach to exploratory testing• There are plenty of approaches to this! There are no standards

for this

• Find one that is natural for you

• Lecture slides describe one approach

• You may find some others in literature

• You need to describe your approach – generic “exploring” is not enough

•

12

29/10/12

Approach C: Traditional test automation 1/2

• Phase 3:• Identify the most essential functions of the application• Plan how testing of them can be automated. The tools, the test

data etc…• Plan a test suite• Design automated test cases• Document briefly the testing system / environment

• Phase 4:• Execute the test cases using test automation• Report errors• Report the testing

13

29/10/12

Approach C: Traditional test automation 2/2

• Essential:• Finding a good tool with which the automation can be done

efficiently and so that the tests are easy to maintain• As test design will need to be done in similar manner to the

alternative A, we will not require a high number of test cases – just a demonstration of good understanding of how testing can be automated

● Good error reports so the developers can fix the errors

14

29/10/12

Approach D: Model-based testing

• Phase 3:• Identify the most essential functions of the application• Pick a suitable testing tool and modeling methodology• Create a test model

• Phase 4:• Generate and execute tests• Report errors• Report the testing

• Essential:• Deciding what to model and designing the test model carefully• Good error reporting so the developers can fix the errors

15

29/10/12

Approach E: Combination of approaches

• The approaches A, B, C and D (two or more of them) can be combined to create a balanced approach to testing

• This would often be most valuable in practical projects and demonstrating skills of all approaches is a proof of good expertise

• Examples:• Systematic manual testing of some functions and exploratory

testing of some others• Automated testing of some functions and exploratory testing of

some others• First exploratory testing, then systematic testing, and finally some

automated testing• Essential:

• Demostrating undertanding of key issues of the approaches used• Good error reporting so the developers can fix the errors

16

29/10/12

Test cases 1/2

• What is important• Positive and negative testing• Each test case must be uniquely identified.• Each test case must have one input.

• Each test case must have one output.

• All relevant sections listed

• If there are several input and outputs in one test case it is not a test case but test suite (sort of). Exception: data-based testing (run a list of values though a test case)

• Each test case must have priority.• So that the most important ones can be executed first and the

rest if there is time

17

29/10/12

Test cases 2/2

• On unit tests• The test code documents plenty of the tests

• Documenting is much like documenting the production code

• Inputs usually hard coded in test code

• On system tests, however

• The test steps are important – we must have and share an idea of how the tests are executed manually (or by an automated script)

• Inputs may vary – this way new bugs can be detected

18

29/10/12

Reporting

• Summary of what has been tested.

• Did you have to change your plan while testing?

• If you don’t change your plan when needed, your project will be a failure

• What were the test cases you had to change and why?

• In your opinion, was your testing complete enough.

• Did you find any faults or other peculiar things?

• Was your testing successful?

• Did you find faults, did you reach your goals etc.

• See the Master Test Plan for details for reporting requirements.

19

29/10/12

Test logs?

• In exploratory testing there are usually no test cases

• Test reports must give a view to what was tested by test logs

• When, what, how...

• In systematic testing Test Management tools (HP Quality Center, TestLink) are often utilized to track of what test cases have been executed and when, so no manual log-keeping is necessary

• If you do system testing in exploratory style, we need some simple test logs (forms are included in test report templates)

20

29/10/12

Supported automation tools 1/2

• Jemmy

• A Java testing tool for accessing GUIs of applications

• Can be used to automate execution of manually created or automatically generated tests

• http://java.net/projects/jemmy

• OSMO Tester

• A model-based test generation tool based on Java

• The behavior of the SUT is specified in Java code using annotations

• http://code.google.com/p/osmo/

• ModelJUnit

• Another model-based testing tool that defines test models as Java code

• http://www.cs.waikato.ac.nz/~marku/mbt/modeljunit/

21

29/10/12

Supported automation tools 2/2

• Basic instructions and examples for using these three tools are available at http://www.cs.tut.fi/~testaus/s2012/project/tools/

• No special environment is required, simply add the tool packages to your jEdit project

• Other automation tools may also be used, but are not supported by course personnel

22

29/10/12

Some other test automation tools for Java on desktop

• General• http://java-source.net/open-source/testing-tools

• Unit testing tool JUnit – used with libraries on higher test levels• The most famous xUnit test application

• http://www.junit.org/

• System / UI level ● jEdit project itself uses Marathon http://www.marathontesting.com/● FEST is also used (a library used with JUnit)

http://code.google.com/p/fest/

● UISpec4J http://www.uispec4j.org/● Jameleon http://jameleon.sourceforge.net/index.html

23

29/10/12

What we value in your work

• A sensible approach to testing• Understanding of how to carry out the approach and

concentration on essentials• Good plans and reports that give clear information to others• Testing does not need to be complete, but what is tested, should

be tested well• You need to learn how to test well and what it takes... testing

“more” is done at work

• In grading we look this time more into what you have done well than what the deficiencies are

• Especially in exploratory testing and test automation, in which we are not able to give that much support for you

24

29/10/12

End notes

• Good luck!

• Try to find an approach to your testing that is most suitable for your learning and interests!

25

29/10/12