department of information engineering 625 verification and validation verification –are we...

1Department of Information Engineering

Verification and validation

• Verification– Are we building the system correctly?– Eliminate bugs in the system

• Validation– Are we building the correct system? – Should already be handled by a

thorough requirements analysis based on use cases


Testing

• Purpose of testing– To find the as many bugs in the system as

possible

• What is the difficulty?– Developers are not keen to find their own

bugs

• What the industry does?– Use a team of testers whose main job is to

find fault in the system• typical tester : developer ratio ~ 1 : 3

– take up at least 30% of the development cost


Test levels

• Unit testing– To test one and only one unit, usually a

class

• Integration testing– Usually use cases based– Integration and unit tests can be done

together

• System testing– Testing the entire system, typically

from an end-user view


Unit testing

• Specification test (black box test)– Verify the unit’s externally observable

behavior– Given certain input and a particular

state, see if the output return is as expected

– Partition the range into equivalent classes to reduce the possible combinations


Equivalent partitioning

• An equivalent set is a set of conditions for which an object is supposed to behave similarly, so as to reduce the number of test cases

• Example– partition the state of a bank account into

•empty, positive and negative balance– partition the state of a stack into

•empty, half-full, full– Partition the expected input value (0 to

100)•To value at boundary, outside boundary

and normal•e.g. -10, -1, 0, 1, 40, 60, 99, 100, 110


Unit testing - structure test (white box test)

• Structure test – every statement

has to be executed at least once

• Test – the most

interesting paths – the least-know

paths – The high risk

paths

while ...

If a<b


Integration testing

• To test different units working together• Each time test only one use case • The sequence diagram is a good source to

specify the test case• For a user case, we may have the

following tests– basic course tests– alternative course tests– tests of user documentation


Integration testing

• Example: to test ATM withdraw use case• Test case: Withdraw $300 from Account

123456• Input

– User selects Withdraw option, key in the password number (888) and withdraw $300 from account 123456

– initial balance = $1000• Result

– Dispense $300– Print the new balance $700 of account

123456• Conditions

– no other use cases are allowed to access the accounts during this test case


Integration testing

• The test case using in the example can verify only one scenario

• a matrix format can be used to represent similar test cases that differ only in a single input or result

• test cases A test cases B Input /result input/result300/700 200/2040600/4000 . . .. . . . . .


System testing

• Test entire system as a whole– several use cases are executed in

parallel– the test can be divided into

•operation tests• full-scale tests•negative tests• tests based on the requirements

specification• tests of the user documentation


Testing techniques

• Regression test– Run the test every time you change the

system– Main purpose is to verify old functionality still

work– Important test, must be automated (e.g.

JUnit)

• Operation test– Test the system in normal operation over a

long period– Use the system in the intended manner– Only normal mistakes are made– Measure mean-Time-To-Failure (MTTF)


Testing techniques

• Full-scale test– run the system on its maximum scale– System is used by many users

• Performance test or capacity test– Measure the system performance under

different loads

• Overload test– Goes one step further than the full-scale

test to see how the system behaves when it is overloaded

– Should not expect normal performance but at least the system should not go down and catastrophe not to occur


Testing techniques

• Negative tests– Stress test that try to use the system in

ways it is not designed for, so as to reveal system weaknesses

– e.g. incorrect network configuration, insufficient hardware capacity, impossible work load

• Tests based on requirements– Tests based on requirement specifications

• Testing of the user documentation– to check the consistency between manuals

and system behavior


Testing techniques

• Ergonomic test– Test the man-machine interface– Is the interface consistent with the use

cases?– Are the menu logical and readable for

non-computer professionals?– Can users understand the failure

messages?


Testing techniques

• Acceptance tests– Alpha testing

•final check by the customers• test under real environment for

longer time than when the system was developed

– Beta testing•Product is not targeted to general

rather than specific customers•Product is tested by specially

selected customers


Testing techniques

• Installation tests– Verify that the system can be installed

on the customer platform and the system operates correctly

• Configuration tests– Verify that the system works correctly

in different configurations, such as different network configurations


C# attribute […]

• A C# attribute is something that is inside a square brackets, such as [serializable]

• Two ways of using the attribute– Simple way

•Use the attribute as a tag (a label)•The tag can be identified by

“reflection” and can do useful work, e.g. testing

– Advance way •Aspect programming, will be

discussed


Attribute uses as a simple tag

• [Serializable] //this class can be serialized

public class Dummy {

int x;

[NonSerialized]

int tmp; //no need to serialized this part

}

• The alternative is to add serializable code to the class, but this pollutes the class with non-core responsibility

• The “serializable” responsibility is factored (taken) out of the class


NUnit (an example of using attribute as tag)• Main philosophy

– Prove the quality of your software by testing

• Relentless testing (XP)– Test all methods using different scenario

• Repeat of all your test cases whenever you change your code– Regression test

• How to do this efficiently?


NUnit

• Based on the simple testing framework JUnit proposed by Beck and Gamma

• The cornerstone of Extreme Programming (XP)

• Download NUnit V2.1 from www.nunit.org– Study QuickStart.doc


Example of writing test cases

• The class to be tested by NUnit

public class Account {

double balance=0;

public void Deposit(double amount) {

balance += amount;

}

public double Balance {

get {return balance;}

}

}


JUnit by Beck and Gamma

TestRun()

TestSuiteTestCase

TestRunner

testSuite.Run()*

Your code YourTestCase

JUnit Framework


Note

• TestRunner has-a TestSuite• Your write the YourTestCases, add to TestSuite

• testSuite.Run() invokes all testcases

• class Test, TestSuite, and TestCase are Composite

• Test is a Command• TestCase is a Template

TestRun()

TestCaseRun() { SetUp() RunTest() TearDown()}

YourTestCase


Test-driven development (TDD)

• Write the test cases before you write the code !

• Work on one test at a time• Keep the test small

• TDD Golden Rule– Never write code unless you have a test

that requires it

• Ref: http://www.parlezuml.com/tutorials/tdd_nunit/index_files/frame.htm


How to improve an already very good testing tool• NUnit already makes regression testing

very easy

• But still need to learn the NUnit framework

• The current version of NUnit is very easy to use owing to attribute programming


Example of writing test cases

using NUnit.Framework;

[TestFixture]

public class AccountTest {

[Test] //the test case

public void Deposite() {

Account acc = new Account();

float balanceBefore = acc.Balance;

acc.Deposit(10.0F);

float balanceAfter = acc.Balance;

Assert.AreEqual(10.0F, balanceAfter-balanceBefore);

}

}


Assertions

• Assertions are central to unit testing

• NUnit provides a rich set of assertions as static methods of the Assert class– Comparison test

•Assert.AreEqual( 1.0, sum);– Condition test

•Assert.IsTrue(bool condition);•Assert.IsNull(object anObject);

– Utility methods•Assert.Fail(string message);


• Other common attributes– [SetUp]

•Method marked with [SetUp] will be run before every test

– [TearDown]•Method marked with [TearDown] will

be run after every test

• Compile the program• Start NUnit GUI• Select the .dll file• Run


Reflection

• NUnit simply loads the assembly– Find the attributes all classes/methods by

the reflection mechanism, and runs the methods

• Use of Reflection– Viewing metadata, perform type discovery – Dynamic invocation

• to invoke properties and methods on objects dynamically instantiated based on type discovery.

• ref:http://www.ondotnet.com/pub/a/dotnet/excerpt/prog_csharp_ch18/index.html?page=1


Create your own attribute TestFixture

//custom attribute, target includes class & method

[AttributeUsage(AttributeTargets.Class |

AttributeTargets.Method | AllowMultiple = true)]

public class TestFixtureAttribute : System.Attribute

{

string str;

TestFixtureAttribute(string str) { //constructor

this.str = str;

}

public string Message() { //define a property

get { return str; }

}

}


Reflection and attribute

• Load the assembly– Assembly a = Assembly.Load(AssemblyName)

• GetTypes() return an array of Type objects– Type[] types = a.GetTypes( );

• Check whether the type has attribute [TestFixture]– Object obj = type.GetCustomAttributes()[0];

if (obj is TestFixtureAttribute)

{ //found the attribute instance, do something

Console.WriteLine(“{0}”, obj.Message);

}


AOP (Aspect Oriented Programming)

• Limitation of OOP – Principle of divide-and-conquer– Decompose complex system into

simpler units– Each unit has a clear responsibility

• Question– What if some responsibilities are not

confined to any particular object but are instead scattered through out the system?


Classic example - logging facility

• How to trace the flow of operations?

• The traditional way– public class Foo {

protected Logbook log; public bar() {

log.enter(“bar() entered”); …

… \\business logic of bar() … log.enter(“bar() quitted”);

}}


Classic example - logging facility

• The problems– Messy, need to provide the code for

every objects, bad code reuse– Objects are overloaded with non-core

responsibilities

• The problem– Some responsibilities cannot be cleanly

encapsulated in an object or method– e.g. security, transaction, performance

evaluation

• The solution– Aspect-Oriented Programming (AOP)


Concerns

• Concern is a particular goal of a system

• A typical system has– Core concerns

•E.g. processing payments in bank applications

– System-level concerns•Logging, transaction, security, . . .

• System-level concerns tend to crosscut (share) by many modules– Such concerns are known as crosscutting

concerns


• OOP– Each object should have clear

responsibility– Good at addressing core concerns– BUT can’t handle crosscutting

concerns well

• Aspect-Oriented Programming (AOP)– Solve the problem of crosscutting

concerns by a pattern called Interception


AOP solution via C# attributes

• [Logbook]

public class Foo : ContextBoundObject {

protected Logbook log;

public bar() {

log.enter(“bar() entered”);

… \\business logic of bar()

…

log.enter(“bar() quitted”);

}

}


What is changed?

• Add an attribute [Logbook]– This attribute is associated with a component that carries out the logging operation

• Foo is now a subclass of ContextBoundObject– Foo now consists of its core business

logic only, no more crosscutting concerns


What happened? The Interception pattern

• C# compiler inserted a transparent proxy which intercepted all calls to Foo

clientTransparent

ProxyMessageSink

Foo

Logbook

Inserted by compiler


To handle more crosscutting concerns

• [Logbook, Security]

public class Foo : ContextBoundObject {

…

}

clientTransparent

ProxyMessageSink A

Foo

Logbook

Inserted by compiler

MessageSink B

Security


• Logging, security, etc are aspects of a system– Each aspect addresses a crosscutting concern– By combining aspects together, one can build

a highly configurable application rapidly• Core concerns are addressed by objects• Crosscutting concerns are addressed by aspects

• C#– Interception supported by compiler

• Java– Tools like AspectJ inserts code to source file


AOP basics

• Identify the concerns in a system• Separate the concerns into core and

crosscutting (usually system-level) concerns

• Implement each concern separately– e.g. concerns in a credit card application

•Business logic (core concern)•Logging (crosscutting concern)•Authentication (crosscutting concern)

• Integrate all the concerns by an aspect integrator (also known as aspect weaver)


The prism analogy

Requirements Concern Identifier

Business logic

logging

Security, persistence, transaction

Weaver

AspectualDecomposition

AspectualRecomposition

system


Configure a new system using aspects

Implementationmodules

Security

Transaction

Logging

Business logic


Noun, verb, and adjective

• If procedure is verb,and class is noun,then aspect is the adjectives that describes the noun

• Reference• See “Decouple Components by Injecting Custom

Services into Your Object’s Interception Chain (http://msdn.microsoft.com/msdnmag/issues/03/03/ContextsinNET)

• For overview, articles by Ramnivas (http://www.javaworld.com/javaworld/jw-01-2002/jw-0118-aspect.html)

department of information engineering 625 verification and validation verification –are we...

Documents

system testing

test levels unit testing

testing purpose of testing

system validation

entire system

correct system

class integration testing

number of test cases