1 automatic error prevention jtest c++test by shimrit tzur-david

1

Automatic Error Prevention

JTestC++Test

By Shimrit Tzur-David

2

Outlines

Automatic Error Prevention overview JTest Static Analysis - Coding Standards JTest Unit Testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test Static Analysis Freeware Tools Summery

3

Automatic Error Prevention overview

The key for developing reliable software on time and on budget is twofold:Reduce the opportunity for errors by following

coding standards.Thoroughly test each class, as soon as it is

developed to prevent small mistakes from growing into widespread, difficult-to-pinpoint problems.

4

Motivation

5

What is Automatic Error Prevention ?

Methodology for improving software quality and reliability.

It uses information gained from software testing, measurement, and monitoring to progressively improve the development process.

6

What is AEP ? – Cont.

Fixing problems where they occur in the manufacturing process, not only eliminates many quality problems in the finished product, but also promotes the ultimate goal of improving the quality of the manufacturing process.

By fixing the process itself, it's possible to prevent the same types of errors from occurring over and over again.

7

Five simple steps for AEP

1. Detect an error

2. Isolate the cause of the error

3. Locate the point in production that created the error

4. Implement practices to prevent the error from reoccurring

5. Monitor for improvement

8

Example

Inspectors on an auto assembly line discovered that seat bolts are not being tightened properly.

The cause: the bolts do not exactly fit the tool used to tighten them.

The corrective action: provide the proper fitting tool.

Monitoring the process is accomplished by closely inspecting the seat bolts for tightness.

9

Error Prevention vs. Error Detection

Error detection is the process of finding and fixing errors after an application is built; the flawed process that generated those errors is left uncorrected.

In the seat example, error detection would have simply tightened the seat bolts at the end of the assembly line. This action left the root of the problem embedded in the manufacturing process.

10

Testing Categories

There are few tests which can be performed in order to significantly reduce the number of errors in our code:Static AnalysisWhite-Box TestingBlack-Box TestingRegression Testing

11

Traditional coding standards

Rules which apply to constructs within the class under test.

A traditional coding standard might test whether or not a file’s source code contains a construct that has a high probability of resulting in an error.

For example, one traditional coding standard checks that we use “equals” instead of “==” when comparing strings.

12

Global coding standards

Rules that ensure that projects use fields, methods, and classes wisely.

They program correctness, ease of understanding, and ease of maintenance.

A global coding standard might check that a project does not contain logical flaws and unclear code (such as unused or overly-accessible fields, methods, and classes). These problems increase the probability of an error being introduced into the code, and might also make the code less object-oriented.

13

Unit testing

Focused test on the smallest possible unit of the software application.

By performing unit testing, we can catch errors as early as possible during development, when errors are easier to find and correct.

We are able to test parts of a project without waiting for the other parts to be available.

We will be able to test internal conditions that are not easily reached by external inputs in the larger integrated systems.

14

White-Box Testing

White-box testing doesn't test that the class behaves according to the specification, but instead, it ensures that the class doesn't crash and that it behaves correctly when passed unexpected input.

White-box involves looking at the class code and trying to find out if there are any possible class usages that will make the class crash (in Java this is equivalent to throwing an uncaught runtime exception).

White-box testing is a process that checks code using test data derived from the class's internal structure.

15

Black-Box Testing

Black-box testing checks that the class behaves according to the specification, meaning that the class produces the correct output for each input.

The tester does not know anything about how the program is coded. He simply takes any output, and compares it to what he thinks is the desired output.

We can view the program as a "black-box", whose behavior can be determined by studying its inputs and the related outputs.

16

Regression Testing

Regression testing checks that the class behavior doesn't break when the code is modified.

Regression testing checks that the class continues to perform correctly after it has been modified.

In regression we’re going back over our previous tests to ensure that the bug we previously found has been fixed and that no new bugs have been introduced.

17

Outlines

Automatic Error Prevention Overview JTest Static Analysis - Coding Standards JTest Unit Testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test Static Analysis Freeware Tools Summery

18

JTest

Parasoft's Jtest testing tool enables statically and dynamically analyzing Java code.

It is an Automated Error Prevention product that automates Java unit testing and coding standard compliance.

Jtest checks whether code follows over 400 coding standard rules (plus any number of custom rules) and can automatically correct violations of over 200 rules.

With the click of a button, developers can identify and prevent problems such as uncaught runtime exceptions, functional errors, memory leaks, performance problems, and security vulnerabilities.

19

JTest static analysis

Static analysis can be thought of as consisting of both an automatic code review and automatic coding standards enforcement.

Adhering to coding standards is an effective error prevention strategy. Coding standards are rules that ensure that code is written in a way that makes it less error-prone.

20

JTest static analysis – Cont.

By adhering to these standards, we significantly reduce the opportunities to introduce errors in the code.

Adhering to coding standards enforcement also makes code more readable and easier to maintain.

Jtest statically analyzes each class by parsing its Java source code and comparing it to a set of coding rules.

Jtest can automatically correct many of the coding standard errors it identifies.

21

Example

To check whether a class follows the default set of Jtest coding standards:

Select the class resource, and click the Play button in the toolbar.

By default, when you click this button, Jtest will check coding standards as well as perform unit testing.

22

The source file

23

The standard tab

A Jtest summary panel opens after the test completes. The Standards tab of this panel displays the following information about the coding standard test run: Elapsed time for the coding

standards checking. Number of files checked. Number of failed runs. Number of errors found. Number of suppressed errors. Number of rules violated.

24

Errors found window

25

Rules violation Text label ‘case10’ maybe typo for ‘case 10’ The Rule: Avoid using text labels in "switch"

statements. case 2 is missing either “break”, return, or /* falls through */ The Rule: Avoid a "switch" statement with a

bad "case".

Literal constant is used: 20 The Rule: Avoid using literal constants.Exceptional constants: -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0.0, 0L, 0.0F, 0.0D, 0xF, 0xD

Literal constant is used: -2 The Rule: Avoid using literal constants.Exceptional constants: -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0.0, 0L, 0.0F, 0.0D, 0xF, 0xD

26

Automatic fixing of coding standard errors

Jtest can automatically correct many of the coding standard errors it identifies.

27

After Automated fixing

28

Outlines

Automatic Error Prevention overview JTest Static Analysis - Coding Standards JTest Unit Testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test Static Analysis Other Tools Summery

29

JTest unit testing

Unit testing involves testing software code starting with its smallest functional point, which is typically a single class or method.

The objective of unit testing should be to verify the code's functionality and construction/robustness.

Ideally, unit testing should start applying the following types of tests as soon as a Java class compiles: Construction testing (also known as white-box testing) Functional testing (also known as black-box testing) Regression testing

30

JTest unit testing – Cont.

Jtest creates a jtest project ‘ProjectName.jtest’ with the same packages hierarchy as in the original project.

For each class ‘ClassName.java’ in the original project, it creates a class ‘ClassNameTest.java’ in the jtest project.

ClassNameTest.java contains test methods for each method in ‘ClassName.java’.

For the method ‘methodName’ in ClassName.java, we will have in ClassNameTest.java the methods: testMethodName1, testMethodName2, until testMethodNameN, where N can be any number greater than 0.

31

JTest unit testing – Cont.

Each testMethodName method in ClassNameTest.java does two things: White-box testing - It generates methodName in

ClassName.java with different parameters. Black-box testing - After the execution of methodName, it

uses the validation method assertEquals to ensure that the output of the function is as expected.

There are few more methods in each ClassNameTest.java for the tests start-up and shut-down and a main method to run the tests.

32

JTest unit testing – Cont.

33

Outlines

Automatic Error Prevention overview JTest static analysis - Coding Standards JTest unit testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test static analysis Freeware tools Summery

34

JTest White-Box Testing

White-box involves looking at the class code and trying to find out if there are any possible class usages that will make the class throwing an uncaught runtime exception.

Jtest executes the class under test using a symbolic virtual machine.

While executing each bytecode, it checks if there are solutions to the equations that will make the bytecode throw an uncaught runtime exception. If it finds a solution it reports an error for it.

35

Example

36

Errors found window

37

/* NullPointerException */public void testStartsWith3() throws Throwable {boolean RETVAL = Simple.startsWith(null, “0”);}

/* NullPointerException */public void testStartsWith1() throws Throwable {boolean RETVAL = Simple.startsWith(null, null);}

Errors

/* StringIndexOutOfBoundsException */public void testStartsWith4() throws Throwable {boolean RETVAL = Simple.startsWith(“”, “0”);}

38

Few Words on Design by Contracts…

In order to fix white box testing errors, Jtest uses the principle of DBC.

A program is correct if it performs according to its specification.

The principal idea of DBC is that a class and its clients have a contract with each other: The client must guarantee certain conditions before calling a method specialized on the class (the preconditions), the class guarantees certain properties after the call (the postconditions).

If the pre- and postconditions are included in a form that the compiler can check, then any violation of the contract between caller and class can be detected immediately.

39

Automatic fixing white-box testing

Jtest can automatically correct many of the white box testing it identifies.

40

After the fixing

41

Outlines

Automatic Error Prevention overview JTest static analysis - Coding Standards JTest unit testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test static analysis Freeware tools Summery

42

JTest Black-Box Testing

Black-box testing checks that the class produces the correct output for each input.

In general, any black-box testing tool requires the user to specify a set of inputs to try, along with the correct outputs for those inputs. The tool then runs the inputs automatically and checks that the correct outputs are generated.

Jtest automatically generates a set of sophisticated inputs; Jtest analyzes the class bytecodes and provides a minimal set of inputs that produce as much coverage as possible for the class.

43

JTest Black-Box Testing – Cont.

Jtest executes all of the inputs and provides the actual outcomes for those inputs.

The developer can see the outcomes and verify them with one click of a button.

when Jtest performs subsequent tests on this class, it will notify us if a different outcome is produced.

44

Example

45

Errors found window

46

Errors

Expected: <true> but was: <false>

Expected: <false> but was: <true>

47

Create user-defined test case

Let’s assume that we want to add our own test case to the method map, and to check what’s happening when we execute it with ’34’ as input.

All we need to do is to add to class SimpleTest.java the following method:

public void testMap() { assertTrue(Simple.map(34) == -2);

} And run the unit test again.

48

Outlines

Automatic Error Prevention overview JTest static analysis - Coding Standards JTest unit testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test static analysis Freeware tools Summery

49

JTest Regression Testing

When Jtest first runs a unit test, it takes a "snapshot of the current functionality of each method tested, and records its behavior in a JUnit-format test case.

Jtest provides automatic regression testing. Even if we don't specify what the correct outcomes are, Jtest remembers the outcomes from previous runs and compares them every time the class is tested and reports an error for any outcome that changes.

50

Example

Let’s assume that we’re changing the add method to reduce i2 from i1, it can be intentional, but it can also be a typo.

51

The error

Since we have the test case:

We will get the following error:

52

Automatic fixing

53

Outlines

Automatic Error Prevention overview JTest static analysis - Coding Standards JTest unit testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing C++Test static analysis Summery

54

C++Test

C++Test is an Automated Error Prevention product that automates C and C++ unit testing and coding standard analysis to help prevent software errors and ensure that code is structurally sound, reliable, maintainable, and portable.

C++Test checks whether code follows 500+ coding standard rules plus any number of custom rules.

Implementing C++Test as part of a team-wide AEP solution is a critical step in improving software quality and streamlining a team’s software development process.

55

General View

56

Running a test

57

Coding Standard Errors

58

After fixing

59

Suppressed Rules

60

Testing Multiple Files

Testing multiple files at once allows to create more realistic test scenarios by testing symbols from different files without breaking the links between them.

C++Test supports multi-file tests by combining a number of single files in a single test unit.

To combine few files, we need to create Test Unit and add the files to it.

When we test the unit, all the files in it will be tested.

61

Test Unit

62

Multi-Files Example

63

Errors

64

Creating Custom Rules

With C++Test we can create or modify a rule in order to create custom coding standards.

Custom coding standards are rules that are specific to a certain development team, or even a certain developer.

The RuleWizard feature lets you design custom coding standards by graphically expressing the pattern that you want to look for during coding standards analysis.

We can use this tool to modify some of the built-in coding standards and to add addition ones.

65

C++Test – Dynamic Analysis

C++Test also has the capabilities of dynamic analysis (like Jtest).

It also automatically creates unit tests and performs white-box, black-box and regression testing.

C++Test Dynamic analysis is out of this lecture scope.

66

It’s all about the money…

The two tools that we covered are Parasoft’s, they are very good (from my little experience) but also very expensive!!!

Since we’re students and probably won’t pay so much for these tools, I’ll mention shortly a company called Panorama.

Panorama has one main advantage: you can download its evaluation copy for free. It has evaluation copies for C/C++ (Panorama C/C++) and for Java (JavaAnalyzer).

67

Panorama JavaAnalyzer

Making a Java program much easier to understand (graphical view).

Making the quality of a Java program easier to measure (coding standard).

Making a Java program much easier to inspect and walk through (linking logic diagrams).

Making Java programs much easier to maintain and modify (code complexity analysis).

68

Panorama C/C++

Automatically generating design diagrams directly from the system specifications.

Provides less-bug and less-risk coding by enforcing the developer to follow the specified coding standard rules.

Efficient program review and inspection by making code traceable in all levels through whole system diagramming and diagram linkage

69

Outlines

Automatic Error Prevention overview JTest static analysis - Coding Standards JTest unit testing JTest White-Box Testing JTest Black-Box Testing JTest Regression Testing LintPlus static analysis Summery

70

Summery

AEP, with its comprehensive collection of "ready-to-use" error prevention practices and its unique embracement of automation, offers the software industry a prime opportunity to start making the necessary changes in a way that is as painless as possible.

With the tools presented in this lecture, developers can develop with as less errors possible and organizations can invest their efforts on the really important issues in the development process.

71

THE END