systematic testing and validation of simulation programs...

Master Thesis 2007-01-21 Final report

Systematic Testing and Validation of Simulation Programs for Combustion Processes Author: Richard Andersson Examiner: Professor Per Runeson, Department of Communication Systems

Supervisor: Martin Tunér, LOGE

Department of Communication Systems Lund Institute of Technology

Master Thesis 2007-01-21

2

Abstract

Software testing is a very resource demanding stage in the software development process. More and more companies have over the years spent increasing amounts of time and money to make testing more efficient and well planned, e.g. they have learnt that correcting faults early in the process can save a lot of resources. LOGE AB develops software for advanced chemical simulations and there is a need to make the testing structured and to define what is suitable to test. An important factor to have in mind is that the company is small with only three employees, and that only a basic ad-hoc testing of units and some result validation is currently being done. It is therefore necessary to make the work highly goal-oriented and well planned.

In order to collect the necessary information to determine what to improve in the current test process, a case study was performed. This research study identified improvement areas, and thus solutions could be derived and presented. The research was divided into two parts, an investigation part in which areas of improvements were identified, and a second part in which the result from the previous part formed the basis for the derivation of a solution which was validated in the second part. The outcome of this research has lead to the use of a customized quality framework that pinpoints certain critical areas in the test process. A major goal of this thesis was to ensure that it would not be too difficult for the organization to adopt the methods and techniques presented; the main task shall be development, but calling attention to the need for a more test oriented approach. Some testing tools and development environments of possible interest to the company in question were also evaluated.


3


4

Acknowledgments

Special acknowledgement goes to the following individuals who contributed to the creation of this master thesis and who have all supported me.

Thanks to my supervisors, Professor Per Runeson at the Department of Communication Systems for his support and encouragement during the process, and to Martin Tunér at LOGE for his everlasting help and interest over the course of the months.

I would also like to thank Karin Fröjd and Raffaella Bellanca at LOGE for their contributions.

Finally, I want to offer special thanks to my devoted and loving mother, Margareta, for proofreading my work as well as supplying me with a lot of constructive comments and feedback. Lund, December 12, 2006 Richard Andersson


5


6

Table of Contents

ABSTRACT 2

ACKNOWLEDGMENTS 4

TABLE OF CONTENTS 6

1.INTRODUCTION 10

2. TEST THEORY 12

2.1 IMPORTANCE OF TESTING 12

2.2 TERMS & DEFINITIONS 13

2.3 TEST STRATEGIES 13

2.3.1 BLACK BOX 13

2.3.2 WHITE BOX 14

2.4 TEST LEVELS 14

2.4.1 UNIT TEST 15

2.4.2 INTEGRATION TEST 15

2.4.3 SYSTEM TEST 16

2.4.4 REGRESSION TEST 17

2.4.5 ACCEPTANCE TEST 17

2.5 TEST DRIVEN DEVELOPMENT 17

2.6 QUALITY FRAMEWORKS 19

2.7 VERIFICATION & VALIDATION IN SIMULATION SOFTWAR E 19

2.7.1 INTRODUCTION 19


7

2.7.2 VERIFICATION 20

2.7.3 VALIDATION 20

3. QUALITY FRAMEWORK PROPOSITION 22

3.1 INTRODUCTION 22

3.2 MTPF 22

3.2.1 INTRODUCTION METHOD 22

3.2.2 OVERVIEW & PRACTICES 23

4. DARS 26

4.1 INTRODUCTION TO DARS 26

4.2 ARCHITECTURE OF DARS AND ITS COMPONENTS 29

4.2.1 COMBUSTION MODELLING 29

4.2.2 CHEMICAL MODELLING 30

4.2.3 REACTOR SET UP 31

4.2.4 ENGINE SET UP 32

5. METHODOLOGY 34

5.1 METHOD INTRODUCTION 34

5.2 RESEARCH STRATEGIES 35

5.2.1 CHOSEN RESEARCH METHOD 35

5.3 CASE STUDY 36

5.3.1 DESIGN 37

5.3.2 DATA COLLECTION 37

5.3.3 DATA ANALYSIS 38


8

6. CASE STUDY AT LOGE 40

6.1 CASE DESIGN 40

6.1.1 STUDY QUESTIONS 40

6.1.2 PURPOSE & PROPOSITIONS 40

6.1.3 UNITS OF ANALYSIS 40

6.1.4 LINKING DATA AND CRITERIA 41

6.2 COLLECTING DATA 42

6.2.1 FIRST ROUND OF DATA GATHERING 42

6.2.2 SECOND ROUND OF DATA GATHERING 43

6.3 DATA ANALYSIS 43

6.3.1 FIRST ANALYSIS 43

6.3.2 SECOND ANALYSIS 46

7. PROPOSED SOLUTIONS 48

7.1 INTRODUCTION 48

7.2 PROBLEM AND EXPERIENCE REPORTING 48

7.3 ROLES AND ORGANIZATION 50

7.4 VERIFICATION & VALIDATION 50

7.5 TEST ADMINISTRATION 52

7.6 TEST PLANNING 53

7.7 AUTOMATION 54

8. TOOL EVALUATION 56

8.1 INTRODUCTION 56


9

8.2 FUNIT – FORTRAN UNIT TESTING FRAMEWORK 56

8.3 ECLIPSE PHOTRAN 60

8.4 FORCHECK 62

8.5 STOPWATCH – A FORTRAN MODULE 63

8.6 PARASOFT JTEST® 65

9. CONCLUSION 68

10. REFERENCES 70

11. APPENDIX A – SOFTWARE TEST PLAN TEMPLATE 72

12.APPENDIX B – QUALITY ATTRIBUTE SHEET 78

13. APPENDIX C – INTERVIEW QUESTIONS 80

14. APPENDIX D – REQUIREMENT DOCUMENT TEMPLATE 84

15. APPENDIX E – MEETING AGENDA 88

16. APPENDIX F – TEST PLAN LOGE 90

17. APPENDIX G – TEST CASE DARS 98

18. APPENDIX H – TEST INCIDENT REPORT LOGE 100


10

1. Introduction

Lund Combustion Engineering (LOGE) develops software for simulation of complex combustion processes. The main product, Digital Assessment of Reactive Systems (DARS), is a development environment for creating chemical models for use in other software, for example engine simulation tools. The result of such a combination is simulations with detailed chemical kinetics. The systems are implemented in FORTRAN with a Java user interface for both Windows and Linux systems.

The software developed by LOGE has only undergone a few basic tests to ensure that the code and the models used are correct. Thus, there is a need to guarantee that a certain level of correctness and test quality is fulfilled. This is even more crucial when the product is being launched into the commercial market. Here, terms like validation and verification come into the picture. Validation is a process that determines if an organization is building the right product. This shall be done at the end of each software cycle. Verification on the other hand means the process of evaluating whether the product is correctly built. Major goals are to introduce these terms into the software development process and to adopt the methodology in continuous development.

The purpose and goal of this thesis is to illuminate and answer the following areas and questions:

• Define the testing maturity models that can be applied and to which extent.

• Determine what to test and how to test it. More specific questions to have in mind are:

o Does the software behave and work as intended in respect of quality attributes like accuracy, reliability, time behaviour, changeability, operability and understandability?

o How to verify that a product is ready for launch? • Can the testing be automated and what shall be automated?

o Develop a test framework and look into some testing tools that can be used. After the introductory chapter, Chapter 2 deals with the theory behind this master thesis followed by Chapter 3 which contains a description of the proposed quality framework. Chapter 4 is a short overview of the software being developed at the company. Chapter 5 deals with the chosen study methodology and presents the reason for selecting that particular methodology. Chapter 6 covers the case study analysis and results. Here a discussion of the result and chosen method is valid. Chapter 7 describes the proposed solutions based on the research study followed by an evaluation of various tools in Chapter 8. Chapter 9 contains the conclusions based on the result and each statement is addressed. Chapter 10 consists of the reference list and Chapters 11-18 are composed of the appendix.


11


12

2. Test theory

2.1 Importance of testing

Software development is a complex procedure that requires good planning and a structured way of working. Testing is a major part of the development phase and therefore an important cost factor. It is estimated that testing represents up to 50-60 % of a software project’s total costs [1], making it important to ensure that resources are spent in a way that guarantees making the most of the testing phase. This involves following well known principles, standards and methods and the use of existing tools in order to make the job for the organization a lot easier. Detecting defects in an early stage in the development saves a lot of resources later.

It is hard to define what is meant by software product quality and it is also difficult to measure and evaluate it. Testing can be described as being: “a process used for revealing defects in software and for establishing that the software has attained a specified degree of quality with respect to selected attributes” [10]. The next problem that arises is trying to make the testing phase as efficient as possible. This is important as it is a resource demanding process which if not carried out in a satisfactory way, will reflect negatively on the product. To aid the test organization, different quality frameworks for software testing can be used to make the work structured. More on quality models will be described later in this chapter.

The process of determining whether the software being developed meets the right requirements and specification is the verification and validation process [33]. This process is iterative and means that it is ongoing throughout the whole software development. At each level in the development, activities are used to check if the result is satisfactory. The validation process means checking to see if the development team is building the right product. The verification process, however, is checking if the product is correctly built. Two important techniques to be used in the verification and validation are software inspections and software testing. Software inspections are a static approach and here methods like inspections of requirement documents can be used. Software testing is a dynamic approach due to the fact that the software is executed.

The quality of a product can be viewed in different ways by people involved in the development process. Therefore, the most important quality attributes for the software must be decided upon and a type of prioritized list needs to be created. The ISO9126 standard [22] has a hierarchy of quality attributes like:

• Functionality

• Reliability

• Usability

• Efficiency

• Maintainability

• Portability

• Compliance

• Quality in use

These have a number of sub-levels with regard to other quality aspects, which are not stated here. Important questions to ask are which quality attributes are the most important and how


13

can they be measured? Making decisions and having a good understanding of the complexity of quality dimensions is a key issue for software engineers.

2.2 Terms & definitions

The following terms and definitions, which are used throughout this report, are stated and described below [10]:

Debugging The process of locating the fault or defect, repairing the code and re-testing it

Process Set of methods, practices, standards, documents, activities, policies and procedures that software engineering people can use to create and maintain a software system with its associated artefacts. Artefacts involve: project and test plan, design documents, code and manual

Error A mistake, misconception or a misunderstanding on the part of a software developer

Fault (defect) Introduced in the software as result of an error. An anomaly in the program that may cause the software to behave incorrectly and not according to its specification

Failure The inability of a software system or component to perform required functions within specified performance requirements

Test case Consists of a set of inputs, execution conditions as well as expected outputs

Metric A quantitative measure of the degree to which a system, component or process possesses a given attribute

Precondition Condition that must be true in order for a software component to operate properly

Postcondition Condition that must be true when a software component completes its operation properly

2.3 Test strategies

There are two basic test strategies that a tester can use to design test cases in order to reveal as many faults as possible in the software [10]. These are black box and white box test strategies. Both are necessary to use in order to ensure that as many defects as possible are found and taken care of. Both require a framework to be set up describing the focus of the tests, necessary test data and when the tests have reached a certain criteria to ensure well working software. They are described in detail in the subsequent sections.

2.3.1 Black box

Black box is a testing strategy also called functional or specification based testing [10]. When using this approach the tester has no knowledge of the inner structure of the software; the only information available is the input and expected output data. The software under test can span from being just a single module to an entire, complete system. To aid the tester in knowing what to test, the requirement documents, system specifications, domain knowledge, pre and post conditions are helpful. The black box strategy is suitable to locate requirement -and specification defects. Because testing is a time and resource consuming process, it is not possible to test everything within a product. Ways to select what to test, depending on the situation do exist. Different methods are available to aid the tester in the process, as listed below:


14

• Equivalence class partitioning

• Boundary value analysis

• State transition testing

• Cause and effect graphing

• Error guessing

2.3.2 White box

The white box strategy is an approach to test the inner structure of a software unit or class [10]. This can be carried out both before, during or after using black box. It is often done afterwards since the tester needs to know how the system is structured, which occurs in the later phase of the design process. Code or pseudo code must be available to base the tests upon. This strategy aims to locate anomalies within the design, code, logical and sequence statements. As with the black box strategy, it is not possible to test everything and white box is most suitable when testing small units and components. The reason why everything cannot be tested depends on the high level of detail required to make adequate test cases. Another important aspect to bear in mind is to have some sort of stopping rule (stop criteria), e.g. when the tester shall stop testing the software. The stopping rule can be seen as a representation for the minimal standards for the actual testing [10]. A test adequacy criterion is a framework to aid testers in the selection of properties of the program which is focused on during testing, selection of test data is based on the properties and works as an indication when one is finished. A typical adequacy criterion can be expressed like a statement and when a program has been adequately tested, all of the structural elements of that statement have been exercised. Adequacy criterion can be of different sorts, they can be expressed as program based or specification based criteria.

There are methods a person can use to carry out the tests, as listed below:

• Statement testing

• Branch testing

• Path testing

• Data flow testing

• Mutation testing

• Loop testing

2.4 Test levels

When one is about to conduct tests on a software product there are a number of testing levels, at each level specific testing goals are defined. Each level focuses on different parts of the software to be tested, and using different testing strategies. The levels shall not be seen as boxes where one goes from unit to integration testing and from integration to system testing in a strict order. The testing process is an iterative approach where the main focus in the beginning of the development is on the unit testing; however the unit testing is maintained during the entire development as well as the other levels of testing. The main goal is to understand that, for example system testing is not only done at the end of development and unit testing is not only performed in the beginning.


15

2.4.1 Unit test

A definition of a unit can be described as “A unit is the smallest possible testable software component” according to Burnstein [10]. The goal is to see whether the components meet their specifications. The meaning of unit differs from various programming languages. In a function based language a unit is a function or a procedure. However, in an object oriented language the actual definition is not yet settled, it can be a method or class that represents a unit [10]. It is important to include unit testing in the overall testing process, not only to test the basic components, but also the simplicity of size and details of the tests in this phase. It is also relatively easy to develop test data, record the tests and analyze it. As the overall test process needs a plan, so does the unit testing, in order to make it as effective and structured as possible. Auxiliary code being prepared is another important aspect in unit testing. This involves the code to be used to test the units and it is part of the test harness. A definition of test harness says; “The auxiliary code developed to support testing of units and components is called a test harness. The harness consists of drivers that call the target code and stubs that represent modules it calls” [10]. An illustration of this can be seen in Figure 1:

Figure 1: Test harness structure

2.4.2 Integration test

An integration test is the technique used to test units grouped together and the interface between them [10]. As with the previous section describing the testing of units, one must also consider the programming language when performing integration testing [10]. In the procedural environment the goal is to find defects in the interface between units; the units put together form a sub-system. It is important to test only those units that have passed the unit testing described above, and also to use an iterative approach [10]. When building the sub-system, one should as far as possible interact only one unit at a time and test upon that module to avoid any massive failures. There are two strategies for performing integration testing on procedures [10]:

• Bottom-up: An approach where one tests the lowest level of modules first and works up towards a complete system. The advantage is that low level modules get well tested in an early stage of the process. The disadvantage is however that the higher level of units and modules get less attention which may be sensitive when handling with safety critical parts of a system.

• Top-down: Works the other way round to the previous approach where the top modules are tested first. Stubs must be developed to simulate lower levels of components. The advantage being that the top layer gets tested well and early in the process, which is good if it is of a complex nature. The disadvantage is that this approach may require complex stubs to deliver data up in the module chain.


16

A test leader must in both cases carefully consider the planning of modules so that they are ready in time. This is due to the fact that modules are not always ready to be tested as planned. Regarding the object oriented language the tester cannot adopt the same approach as

described above. Here, objects are gathered in groups and the simplest group is tested first. Cluster is a well known name for a group of classes and the definition of it according to Burnstein [10] is: “A cluster consists of classes that are related, for example, they may work together (cooperate) to support a required functionality for the complete system”. One way to make the testing of cluster easier is to either integrate classes which support simple functions and run test or integrate those who send few or no messages.

To design test cases both black and white box strategies may be used. Due to the complexity of performing integration between components, it is important that the tester tests as many available input and output data as possible. Not only is the correct data necessary, it should also be in the right order. Black box is a good technique to aid the tester in the design of test cases where information like requirement documents and user manuals can come in handy [10].

2.4.3 System test

The objective of system testing is to ascertain if the system meets its predefined requirements. Both functional and quality requirements such as reliability, usability, performance and security must be achieved. The system testing is particularly useful for finding defects in external hardware as well as in software interfaces. This test technique is a highly resource demanding activity in the overall test process [10]. The planning of this test should begin at the same time as performing the requirement activity, as well as during the investigation of test plan activities. Regarding the testing team it is best if no developers are engaged in the process. There are several variations of system testing: functional, performance, stress, configuration, security, recovery, reliability and usability testing. Of course not everything can be tested, and as stated before, testing is a highly resource demanding activity. The test team has to consider what to use and to what extent. An important tool to use when testing quality requirements as well as performance and stress test is a load generator. The definition according to Burnstein [10] is: “A load is a series of inputs that simulates a group of transactions”.

In Figure 2, the various system testing strategies are illustrated, except usability testing. Here one can see that a fully integrated system arrives ready for system tests. Depending on the chosen test type, different documents can be used. The requirement documents are important when developing functional tests as well as the usage profile for configuration tests and the user manual for performance tests. After the tests have been completed and all have passed, the system is ready for acceptance testing with the user in centre.


17

Figure 2: The different system test strategies with the necessary documents

2.4.4 Regression test

This is the level of testing when one re-tests already tested software. This is a must when defects are repaired or changes have taken place. Regression testing can be performed at any testing level and it is especially useful when developing multiple software releases [10].

2.4.5 Acceptance test

The final level of testing is called acceptance testing; here users play a very important roll [10]. Users need to be active during most of the test design phase in respect of planning and development of test cases, but on this level they integrate with the product being developed. If a software system is designed for a special organization, test planners and users work together to form test cases that will be used during the acceptance testing. During the actual test, the users evaluate the product from their own point of view to determine whether it meets their goals. The software must be executed in a real environment and work in real conditions, if not the tests will not give proper information regarding system performance. The conditions can involve the software being executed a full working day and both legal and illegal inputs must be used.

For software being created for the mass market there is another approach to use. It is not possible to test for individual users, instead alpha and beta tests are used [10]. The alpha tests take place at the developer’s location and users are invited to perform tests. In beta test, the software is transmitted to the users own environment and any problems or defects are reported back. The completion of acceptance testing is a major milestone for the development team.

2.5 Test driven development

The test driven development (TDD) technique is a practice of the Extreme Programming paradigm [4,23,24]. TDD can be regarded both as a test design method and as part of a style in the programming practise [23]. Whether viewed as a design method depends on the fact that test cases are derived prior to writing a code. The following steps are included in the process [4,11,23]:

• Write a test that specifies certain functionality in the coming software system. It must be as simple as possible and be derived with the help of use cases and user manuals

• The test must fail the first time it is executed, this because no code has yet been implemented and in order to see that the test actually works


18

• Write code to make the test pass

• Re-factor the code to make it easier, more flexible, remove duplications and finally re-test to make sure no defects and failures occurred in the re-designed code

o The idea is to improve the internal structure and change how things are done instead of what is done [4]

o Duplications involve “death of good code” [4], which means that one wish to make the code as effective as possible

o To make re-factoring easier, automation of test is essential

In order to be consistent when using this approach, not a single line of code shall be written before a test has been created. In the beginning it is hard to adopt for an organization because it is hard not to implement anything prior to test creations. It is also important to have supporting tools when doing theses kind of tests, due to the amount of tests generated. A set of tools that can be used in Java and FORTRAN are JUnit and FUnit [4,11,23]. TDD cannot in any way be a substitute for “ordinary” testing, it can only be a complement to the unit testing level. Still, when all the tests have passed one can feel confident about the correct functionality being implemented according to the specification [4].

Questions to ask are: why is TDD beneficial to an organization and what motivates adopting the technique? TDD is a “think before you act” approach [4] where the goal is to think through the design first and from the test cases derived, develop code. Some reasons why TDD could be useful are [4]:

• In most cases, testing is done after a code has been written. Often the programmer responsible for the code has moved on to another project. This can, when defects are found, lead to difficulties with identifying problem areas and getting “back” in the code.

• Test cases are usually developed by people who have not been involved in the actual code development. There is a risk that certain issues and functionality may be overseen.

• Automation of tests is necessary to make them regular, frequent and executed in the same way each time. The automation of tests is an important part of the TDD approach.

The benefits of using TDD may be greater amongst smaller companies which do not have personnel that work exclusively with testing. TDD also assures that one can reach 100% coverage of the code since every module and line of code is executed in the tests. Advantages include [4]:

• Software being developed faster and better, this because the functionality of the system is first at hand before starting to write any code.

• Small steps (increments) are taken towards a complete system, the tests describes the system in an early stage.

• Defects and failures are likely to be found early on the unit level instead of higher levels such as system or integration which cost a lot more. This leads to a more robust system earlier in the development process.

• Feedback is given to the programmer continuously, which offers sufficient confidence to continue with the development in a satisfactory way.


19

Disadvantages include:

• If not done correctly the code written could be poorly constructed, it should be complemented with other techniques like reviews.

• Poorly designed tests reflect bad implementation. Important to have fully understood the tests so that they meet the correct requirements.

• Complicated to use TDD with GUI: s [4].

2.6 Quality Frameworks

Having some kind of measurements and framework to follow is necessary in order to achieve good quality testing. As mentioned before, testing and maintenance of a product is costly. The problem is that testing has very low priority in many organisations and companies, and is also very little discussed in literature and not very common in research. Developers are in a position to reduce the amount of testing when they see that maintenance costs will no longer affect them but instead the customer in question. However, not all problems have to do with a faulty testing structure or imperfect software process handling. The organizational model of a company is a very important factor in testing [1,10,20]. It is crucial to understand the environment in which the company’s testing processes is carried out before trying to improve the quality of it.

To create better conditions for planning, realization and evaluation of the software process there are different quality frameworks aimed at improving the process – Software Process Improvement (SPI) frameworks. These frameworks can, from a testing point of view, be divided in two categories, generic SPI frameworks and frameworks that specific address testing issues. Examples of common generic SPI frameworks are Capability Maturity Model (CMM) and ISO 9001. Examples of test specific frameworks are Testability Maturity Model (TMM), Test Improvement Model (TIM), Test Process Improvement (TPI) and the Minimal test practice framework (MTPF) [25].

If there were sufficient quality in software processes and especially in the testing field, there would be no need for quality frameworks. The lack of quality improvement of software systems has resulted in an increased demand for good frameworks and models. The continuously increasing workload has also called for an improvement of the test process

2.7 Verification & Validation in Simulation softwar e

2.7.1 Introduction

Simulation software, like any other software being developed, needs some sort of verification and validation (V&V) to acquire a certain level of quality. One way to determine quality in simulation software is to measure the accuracy of the simulation result [6]. Before going further into the subject, it may be useful to know the actual meaning of the terms model and simulation. A model is described as: “a representation or abstraction of something such as an entity, a system or an idea” [5]. Simulation can be described as: “the act of experimenting with or exercising a model or a number of models under diverse objectives including acquisition, analysis, and training” [5]. There are different approaches one can use to increase the accuracy in simulation software, stated by Balci [6] as “quality-centred assessment” approaches.

Some issues regarding V&V of this kind of software involve the main idea of simulations, e.g. simulating a real-world activity or event. The purpose of transferring the simulations into the computer depends on the fact that it is not always possible to do the calculations elsewhere. The downside however, is that it can be troublesome to test the results and to see


20

whether it is correct or not. If problems arise within the software it may depend both on the actual code, the developer having written incorrectly or malfunctioning code or are the models that represent the real-world not correctly captured. Here, the verification process can be seen as the activity to determine that the code works as intended and the validation process to ensure that the conceptual simulation model is correct [26]. It is also worth mentioning that it is not possible or desirable to acquire a totally correct model. The purpose is not that it should reflect the real-world model to 100% or else it would not be a simplified model [17, 26].

2.7.2 Verification

Within the verification process of simulations software, different techniques are used to make sure that the computer code is defect free. According to Kleijnen [26], the process involves using the following approaches:

• General good programming practice Use of the proper syntax, documentation, walk-through and modular coding.

• Verification of intermediate simulation output Can be done by calculating output manually and comparing against the simulated results. The term tracing can be used, which means “getting all intermediate results from a computer program automatically” [26]. The main point is not to do everything by hand.

• Comparing final simulation outputs with analytical results The simulation result can be verified by using a simplified version of the model with a well known analytical solution. But it is not always possible to find suitable test cases with the known model.

2.7.3 Validation

After the code is assured to be correct and all bugs and defects are located and fixed, it is time to se if the conceptual simulation model is an accurate representation of the system at hand. This is done in the validation process and the following is a way of assuring that the right model is employed [26]:

• Obtaining real world data Formulate the laws that the system will depend on, validate the model by measuring input and output data of the real system or already known results. It is sometimes impossible to gather the necessary data.

• Comparing simulated and real data After the real data is gathered, it is transferred into the simulation model. Simple tests are used to see if the data match each other in a satisfactory way.

• Doing a sensitivity analysis Investigations of what data is the most important by conducting a risk analysis and determine the risks for the input domain that can not result in any data.

Another way to look upon validation according to Fraedrich & Goldberg [17] is by asking: is the simulation good enough? How can this question be answered and are there any frameworks to follow to make the work structured? A methodological framework has been derived by Fraedrich & Goldberg [17] for validation of simulations and it includes five phases:


21

• Phase 1: Can the simulation be validated? Finding what method to use depending on the nature of the simulation, is it predictive or non-predictive? Validation is used for predictive simulations whereas for non-predictive simulations evaluation is used where the methods and standards are particularly use-specific.

• Phase 2: Conduct a priori test Performing experiments with the real system or environment can be costly, i.e. the reason for simulations. Huge costs are involved in collecting empirical data, thus there is a need to verify that the developed code represents the conceptual model [17, 26] and that the conceptual model is correct.

• Phase 3: Designing and executing the real-world experiment In this phase the goal is to see whether the experimental data meets the objective of the simulation. The predicted accuracy of the simulation must be determined and guidelines have to be set regarding requirements of the accuracy.

• Phase 4: Comparison of measured and predicted data, and assessment This phase involves determining a measure of the predicted accuracy and trying to compare it to the requirements of the simulation.

• Phase 5: Simulation improvement If the measured accuracy is not satisfactory enough, the simulation must be done once again and modified regarding the conceptual model. There may be specifications errors that are not located at the first execution.

As with any other software the V&V process for simulation software is not a trivial task. The difficult point is trying to define a certain degree of accuracy of the simulated result and compare it to the real-world measurements. This is not always possible due to the various complex conceptual simulation models. Having a clear mind about the conceptual model as well as a defecting free code is a successful combination for a well executed simulation.


22

3. Quality framework proposition

3.1 Introduction

From the analysis of the data gathered in the case study research, the general conclusion is that some type of structured test framework is needed in the organization. This is essential in order to make it easier for the company to set up some test guidelines to follow and also trying to reach some quality within the test process. A very important factor to consider is not to “drown” the organization in paper work and too much administration, since the company only has three employees. It is important for these employees to focus on the development, however with a more test oriented approach. The proposed framework is called a minimal test practice framework (MTPF) [25]. The reason why this framework has been derived has to do with the large amount of resources necessary to adopt any of the larger quality improvement models like TMM, TIM and TPI [25]. For a small or medium company this is neither feasible nor desirable.

3.2 MTPF

3.2.1 Introduction method

To make the integration of the quality framework easier, an introduction method has been developed. This includes a pre-analysis to make sure that the framework and existing knowledge is adaptable within the organization. The method consists of five steps: preparing, introducing, revising, performing and evaluating [25]. These steps are illustrated in Figure 3.

Figure 3: Introduction method of the MTPF framework

The big dashed rectangle is meant to symbolize the normal organization; label 1 shows that the current practices shall be evaluated frequently along with revision if needed, shown by label 2. When a new phase is about to be adopted in the organization, shown by label 3, preparations have to be made. This involves training material, database systems and so on [25]. Before the new practice is used, it is introduced to the employees in the company, this for getting valuable feedback. To make it as appropriate as possible, any feedback is used to revise the practice before the actual performance.


23

3.2.2 Overview & practices

The MTPF framework consists of five different practice categories, important for small and medium companies. These categories point out areas in the test process as well as in the test organization that are of significance [25]:

• Problem and experience reporting: Reporting and tracking of defects found.

• Roles and organization issues: The organizational structure of test functions and the roles determined within the organization for test purpose.

• Verification and validation: Activities to ensure some V&V in the product.

• Test administration: Administration of the test environment.

• Test planning: Planning of tests.

These categories consist of three sub-levels which correspond to the growth of the company and the number of employees. The first level is suitable for a company with approx 10 employees, the second level for approx 20 employees and the third level for approx 30, and more. These numbers are not meant to be strictly followed; they work solely as a guide. The categories with the connecting sub-levels are described in more detail below [25].

Problem and experience reporting

• Phase 1 – Define syntax: Common problem-recording syntax shall be introduced in the organization to facilitate for programmers and test personnel to document problems.

• Phase 2 – Create system: Develop and adapt a reporting and storage system based on the problem-recording syntax defined in the previous phase. Have defined routines for gathering, documenting, storing and using experience in every single project.

• Phase 3 – Maintain and evolve system: Ensure that the system from the previous phase can handle recording defects.

Roles and organization

• Phase 1 – Define responsibility: Make test responsibility clear and concise. This involves: test plan development, administrating test environment and problem-reporting, updating checklists and continuously assessing the testing practices.

• Phase 2 – Define roles: Appoint a test manager and assign other people to be testers. The testers have responsibilities that involve: handling walkthroughs, administrating test-case development, supporting the problem-reporting system and administrating experience reporting.

• Phase 3 – Define teams: Create a test team, which is independent of the development team. All roles are defined and this presumes that the other phases are well used and adapted or else it is going to be difficult to make it effective.

Verification and validation

• Phase 1 – Use checklists: Use checklists for the most common and important tasks regarding GUI and platform testing. Checklist shall be written before the projects start or if they exist already, they must be updated and revised.

• Phase 2 – Perform walkthroughs: Introduce walkthroughs into the organization, conducted preferably in the design phase before the software is ready for execution. Developers, designers and a tester is part of the team.


24

• Phase 3 – Perform inspections: Introduce inspections and replace the walkthroughs. When the company grows, a more formal way of making the reviews is necessary. The inspections must be prepared in advance.

Test administration

• Phase 1 – Basic administration of test environment: Having a test environment that will be available at all times. Some activities include: organizing a testing environment for each project, updating the environment as well as having documentation of the environment.

• Phase 2 – Test cases: Create test cases to cover the basic and most common functionality. This requires a lot of resources and several testers have to work on this task. Test cases cannot replace all testing; some must still be ad hoc.

• Phase 3 – Risk management: Involves identifying problem areas in the beginning of a project. This to minimize eventual risks that could appear within the specified areas. A problem database must exist together with experience reporting.

Test planning

• Phase 1 – Test plan: Create a test plan which contains all test related material and planning of the test process. A schedule with milestones is an important point to consider.

• Phase 2 and 3 – Coordinate Software Quality Assurance: Define routines for quality assurance, guarantees that the software has a certain quality level before release.


25


26

4. DARS

4.1 Introduction to DARS

DARS (Digital Assessment of Reactive Systems) is a development environment for creating chemical models for use in other software, like simulation programs of different kinds. The software can be seen as a combustion engineering program where chemical models can be created, manipulated, exported and simulated. The simulations are mainly based on combustion processes, e.g. the main goal of the program. However, in the future one can imagine other processes being implemented and simulated.

The chemical models describing the real environment are highly complex and hard to illustrate and may contain a substantial amount of data. This is why one wants to transfer everything into the world of computers to make calculations a lot easier and to simplify chemical models. This regarding in particular combustion processes, as they consist of complex models which are hard or even impossible to perform in laboratories and because there are many intermediate steps which are still unknown. Another reason for the need to simulate processes is that it is impossible to do it in the real world. An overview of the overall purpose with DARS is illustrated in Figure 4. A chemical model is created in the software based on experimental results; the model has to be reduced due to its complexity. With knowledge of the physical model like engine parameters, reactor tools can be used for simulations (combustion simulations). With the help of CFD (computer fluid dynamics) tools, it is possible to solve interactions between chemistry and turbulence.

Figure 4: Map of chemical simulation in DARS

The software consists of a number of modules and is dependent on other software components to make it a fully fitted development tool. When DARS is initiated a main window appears as shown in Figure 5.


27

Figure 5: Main frame of DARS In the main window, three smaller window areas are located. The first one, on the left, contains available modules that can be used. These modules are activities in the program such as Read Mechanism, Reduce Mechanism, Mechanism Overview, different simulations and last but not the least exports of mechanisms to certain predefined formats.

When wanting to read a mechanism, e.g. a set of chemical reactions describing a chemical environment, the “Read mechanism” module is chosen from the left window and dragged to the workbench located in the middle. After that, certain input files containing necessary data have to be connected to the module like gas phase reaction, surface reaction, molecular data and state functions data. This must be done to be able to define the mechanism and simulation settings. The “Mechanism Overview” module is dragged in the same way as the previous one to the workbench. This activity is used to view mechanisms to ensure that it has been loaded properly from the files and to ensure that a certain degree of prevention has been made towards syntax error. The visualization displays chemistry information surrounding elements, reactions, species, molecular properties, transport properties and state functions. The information is displayed in an editor, which will pop up at the very right of the main program window when the module is clicked upon. It is also possible to export work to different predefined formats, this is done by dragging the different “Export Mechanism” modules to the workbench (Export mechanism, GT-Power and ProStar). The formats available are dynamic library of FORTRAN modules (used in other programs like STAR-CD and GT-Power), ASCII files (reactions in a ChemkinTM format), FORTRAN files and link files which are used by DARS internally. Lastly there is an activity called “Simulation”. Here one can drag and create new simulation nodes to be connected to the chemical mechanism. The simulations available are:


28

• Reactor set up o Constant volume or pressure o Perfectly stirred o Plug flow

• Engine set up o Engine modules for the HCCI (Homogeneous Charge Compression Ignition)

and SI (Spark Ignition) engines

• Flame set up o Premixed o Counter flow flame set up

A more detailed description of the various simulations modes will be described in the subsequent section.

After that has been done, several cases can be created to the simulation. It is possible to have multiple simulations running at the same time and printed out both in the output panel, to a file and also be seen if one expands the node in the window at the very right of the main frame. Under the expanding node information like gas properties can be viewed as diagrams. This is illustrated in Figure 6.

As previously mentioned, the middle window in the main frame is the workbench where all modules are dragged. It offers an overview of the current mechanism and its connected activities. The workbench consists of a grid net with possibilities to zoom in and out.

The bottom window is an output panel where information such as messages, errors and activity log are printed out during runtime. Under the message tab information like file paths are printed and info about closing and opening of files.

The last window is located at the far right of main frame and it is a kind of an activity window. It pops up whenever detailed information is needed, for example if clicking on a “Mechanism overview”, a frame with data appears such as element, specie, reaction, molecular and state function overview.

Figure 6: Output window and activity window to the right


29

4.2 Architecture of DARS and its components

There are many components interconnected to make DARS a complete system. Figure 7 below attempts to create an understanding of the structure. The blue/dotted marking on the components show that they are part of the chemistry manipulation in DARS, this is the main goal. As described above, it is possible to read, reduce and optimize chemical mechanisms. However, the optimize mechanism function is not yet implemented in the current version of DARS. Once a chemical model is created it is possible within the program to export it to different formats. Being able to export to the GT-Power format makes it possible to couple models between the two programs. GT-Power is a tool for one dimensional flow calculations which is standard on the industrial market and used by numerous engine and vehicle manufacturers [18]. The internal file format being used in DARS (.dll/.txt) can also be used in software tools like STAR which is developed by CD-adapco and consists of a powerful software chain for simulation and development.

The red/solid marking, however, symbolises internal activities carried out in DARS. The possibility to export to other software increases the usability for the end users and makes it possible to couple with other software. The simulation available can be used in different ways. First it is possible to simulate and test the chemical model that has been created to ensure that it is correctly put together but one can also make simulations based on loaded chemical models directly from DARS without the necessity to create a new model. The simulation modules (reactor, engine and flame) available are described in the coming sections.

Figure 7: Architecture of DARS and its components

4.2.1 Combustion modelling

A feature in DARS is the possibility to simulate combustions of different kinds. A combustion process is chemical energy being transformed to heat. To be able to perform a process like this, fuel and an oxidizer are necessary which then go through a set of conditions. The result is the actual combustion with its bi-products. The simulation requires both physical and chemical models of the environment. The physical models are:

• Mechanical process which includes piston and valve movement.

• Thermodynamic process which includes heat transfer, gas flow and turbulence.


30

• Time administration which include synchronizing of processes and sub-time stepping to acquire better accuracy.

The chemical model is a description of how various species interact with each other under certain predefined conditions. Each fuel and oxidizer has a unique set of species and reactions. Chemical models can be more or less complicated, stretching from 10 to more than 3000 species.

4.2.2 Chemical modelling

A chemical model is a set of species interacting under certain defined conditions. Within DARS it is possible to calculate quantities like reaction speed and reaction rate. To develop a chemical model, certain steps need to be considered as described below:

• Define some start conditions where decisions have to be made regarding choice of fuel and oxidizer. Also the mixture between the two needs to be settled.

• Literature study to investigate what species and reactions are involved and also to see if there are any models already developed and if any experiments have been made to back them up.

• Define sub-mechanisms (part of a chemical mechanism), is there one to use?

• Validate the sub-mechanisms; are there any experiments done or known models?

• Connect the identified sub-mechanism to a complete model.

• Validate the model with experiments.

Example of a mechanism is the hydrogen-mechanism which consists of 9 species and 43 reactions. There are also more advanced models like n-heptane which has 860 species and 3600 reactions.

The reason for having a reduction mechanism function is to scale the chemical model down to a manageable size for a specific physical model. It may also be used for faster calculations although with the aim of not losing any accuracy. The techniques that can be used to accomplish reductions concern:

• Lumping where species with the same characteristics are put together.

• Skeleton-mechanism: Redundant species are removed.

• QSSA (Quasi Steady State Assumption): Short lived species are identified and removed with the help of a numerical solver, their concentration is determined with an algebraic and iterative method. Figure 8 below shows the procedure where the result is a matrix of fixed size.

∑ ∑= +=

+

⇒

−

⇒

n

i

n

kjjij

kkkk

k

nnn

n

n

xa

x

x

aa

a

aaa

reduction

QSSA

x

x

x

aaa

aaa

aaa

1 1

1

1

21

112112

1

1121

22221

11211

M

M

LL

MOM

MO

L

M

M

M

LLL

MOM

MOM

MOM

LLL

Figure 8: QSSA - reduction of a chemical model


31

In the examples described earlier, the hydrogen-mechanism can with the help of Lumped reduction be simplified down to consisting of 151 species and 1402 reactions. With the use of skeleton reduction the same mechanism can be further narrowed down to 64 species and 482 reactions.

In the reduce mechanism function information such as reaction flow analysis, sensitivity analysis and species lifetime analysis can be performed after a reactor, engine or flame setup calculation.

4.2.3 Reactor Set up

As stated by Bellanca [8], chemical reactors are “laboratory devices in which a change in composition of matter occurs by chemical reactions. They are used for scientific investigations since inside a reactor experiments can be performed under ideal conditions”. In DARS one can simulate these reactors under optimal conditions and get information such as the rate of the reaction, concentration of species and temperature. From this, quantities like ignition delay, heat release, pressure and volume can be measured. To be able to describe what happens during the reactor simulation, a mathematical formula is required to describe the chemical process. These differential equations are called conservation equations and there are equations for mass, momentum and energy. The equations describe what happens to the quantities during a certain time interval. State variables like: energy, pressure, entropy (degree of chaos), temperature and volume characterize a system, together with the conservation equation all the quantities are used to create a closed system which can be solved by integrating over the discretized domain [8]. To be able to simulate the chemical activity in the reactor, a kinetic model is required. These models consist of elementary reactions which are determined from theory and research.

In the first reactor set up, either the volume or the pressure is constant in the closed system. When simulating the combustion with constant volume, the pressure increases and there are no in or out flows. Here one is able to make calculations of the heat of formation for various fuels [8]. The second reactor type is called the Perfectly Stirred Reactor (PSR) and this is a kind of ideal reactor were it is possible to obtain a perfect mixture by either mechanical stirring or by using fuel injection and oxidant mixture. The PSR is used for studying combustion processes like flame stabilization, NOx formation and to calculate values for global reaction parameters. When performing the actual simulation, the reactor contains combusted gases together with unburned mixture. The third and last reactor type is called the Plug Flow Reactor (PFR). This reactor consists of an open tube where fuel mixture is inserted at one end and the gases are pushed through the tube to get a boundary layer flow which is fully developed.

There are two different reactor types. They are either homogenous or stochastic reactor models. The homogenous model is a zero-dimensional model and the same properties work throughout the whole closed system. The PFR reactor as well as reactors with constant volume and pressure are said to be homogeneous. The in data needed for this model are:

• Chemical model

• Data for the system such as: volume and wall surface areas

• Thermodynamic data such as heat transfer parameters, wall temperatures and start conditions like temperature and pressure

• Calculation data such as step-length of the time, start and stop time

• Starting conditions for the gas composition

Possible out data that can be obtained from the simulated model is fractions of species, pressure, heat release and fraction of the burnt fuel. Limitations surrounding this model


32

include that it can provide an over prediction regarding max temperature, max pressure and NOx but also provide an under prediction with HC and CO values. Other limitations present are no combustion durations and that local effects are not taken into account.

A more advanced model is a stochastic reactor which is a quasi-zero-dimensional model. Instead of having the same properties in the whole system, there are an arbitrary number of particles in the closed system. An example of such a reactor is the so called Partially Stirred Plug Flow Reactor (PaSPFR), both the volume and pressure can be constant. In data to this kind of model is:

• Chemical model which depends mostly of the actual kind of fuel being used

• Data for the system such as: volume and wall surface areas

• Thermodynamic data such as heat transfer parameters, wall temperatures and start conditions like temperature and pressure

• Statistical data such as statistic heat transfer coefficient and the number of particles

• Calculation data such as step-length of the time, start and stop time

• Starting conditions for the gas composition

Out data from this model includes information like global parameters which are calculated over time such as pressure, temperature heat release, and chemical composition. Also local parameters can be obtained where the calculation is on each single particle over time. These are mass, volume, pressure, temperature and chemical composition.

4.2.4 Engine Set up

In this simulation type an engine is to be simulated. When making a model of the engine it is possible to utilize a zone model. The approach used according to Bellanca [8] is “the engine is regarded as a homogeneous area and modelled as a single cell exactly as for reactors”. There are two different engine modules available, the first one is called the Homogeneous Charge Compression Ignition (HCCI) and the second one is called Spark Ignition (SI).

The cycle of an HCCI engine is illustrated in Figure 9:

Figure 9: The four-stroke cycle of an HCCI engine

The cycle works as follows: it starts with an intake stroke when the piston moves down (first piston from the left in picture above) and fuel and air is injected at the top of the cylinder. Then the piston moves back up (second piston from the left) and compression of the fuel mixture occurs, e.g. the compression stroke. When a certain temperature is reached, the mixture auto ignites (third piston from the left). During this stage in the cycle, intake conditions and engine parameters are matched frequently so that the ignition occurs close to the dead centre. A great amount of pressure is developed as a result of the ignition whereas the piston is pushed back down (fourth piston from left) and the chemical energy is transformed to mechanical work, e.g. the power stroke. In the final stroke, the piston moves up again (last piston from left) and the burnt charge is pushed through the exhaust valve. This type of engine can be described as a reactor being compressed and expanded by the


33

piston movement. Quantities like temperature, pressure and volume can be simulated and calculated for different fuel mixtures. Here the chemical reaction decides the combustion process and the temperature is usually low. This form of simulation is hard to simulate and to obtain control of.

The SI engine is a little different from the HCCI engine. They both have four strokes but in the SI case, the combustion starts with an electrical discharge when a flame front develops, this can be modelled with a two zone model. Here the unburned gas is compressed both by the piston and the flame front and the same quantities as for the HCCI engine can be calculated upon. The combustion process in the SI engine has a high temperature and the actual event which starts the ignition is a set of defined events, e.g. electrical discharge in cycle 2. This type of engine model is easy to control and can be simulated by using simple chemical models.


34

5. Methodology

5.1 Method introduction

In the process of carrying out the work of a master thesis, having a well defined method is of importance since the work needs defined frames and principles to follow in order to make the work structured. This involves everything from setting the goals of the thesis to ensure and validate the result. The method I have chosen is a combination of two different approaches. The aim of the first approach is to find out the current situation in the company organization with regard to testing and in what way it is performed today. The second approach involves solving and trying to adopt techniques and methods available in the company and how far it is possible to reach in this aspect. The work line adopted for this thesis is shown in Figure 10:

Figure 10: Work line for the master thesis

As seen in the picture, the first step is to evaluate the current situation at LOGE. This involves what is currently being done and which aspects of the testing process that can be improved. The investigations shall not only take a further look at the testing process, it shall also be viewed from the employee’s point of view. The second step involves prioritization of that which can be improved and introduced into the company based on the previous pre-study and theory. From the second step, a proposed solution can be developed and finally adopted within the organization.

When conducting the work of the thesis it is important to validate the work being carried out as well as the results. Terms to have in mind are:

• Reliability in the data gathering and analyzing process

• Validity between the objects one wishes to investigate and measure, triangulation can be used for this purpose


35

• Representatively: Ensure that the conclusions are general and to acquire a certain accuracy in the result

5.2 Research strategies

To be able to carry out the investigations of the company’s current practice, a research strategy must be used to structure the work. From the selected strategy, one can make a plan of how to carry out the investigation work. There are two different research designs available, fixed and flexible [32, p. 5-6]. Another way to describe the research designs proposed by Hammersley is that fixed designs are said to be quantitative whereas flexible designs are said to be qualitative [32, p. 5]. Examples of fix design methods are experiments and surveys [32]. The reason why they are called fixed depends on the fact that the information about how and what you are going to do must be defined before carrying out the actual research study. This involves developing a conceptual framework or use theory to know what to do. Also a certain degree of control of the researcher is required. For example, when developing a survey with questions it is not possible to change the order or the content of the questions after it has been sent out to the participants of the survey.

The second design is called flexible; case studies, ethnographic studies and grounded theory studies are said to be flexible [32, p. 164]. In contrast to other design methods, a smaller degree of information has to be determined before the actual study takes place. The design is evolved and developed throughout the entire research process.

In the process of choosing a relevant research strategy there is a framework proposed by [32, p. 81] with questions to have in mind when planning and performing the research:

• Purpose: What is the aim of the study and what shall be done? Meaning to describe, change or explain something? Trying to locate some difficulties and find a solution?

• Theory: Any theory available for guiding and helping you to perform the study? How can one understand the result of the study?

• Research questions: What questions can be derived to meet the goals of the research within the given time and with the available resources?

• Methods: What techniques to be used for gathering the data, e.g. interviews, observations and so on?

• Sampling strategy: From where and when will one get the data? The selection and balance of the gathering has to be considered.

5.2.1 Chosen research method

The first choice to make is whether to use the fix or the flexible approach. As stated by Robson [32, p. 5] fixed designs are “carried out in the real world settings and they require a developed conceptual framework or theory so that you know in advance what to look for, and extensive pilot work to establish what is going to be feasible”. In my opinion, it is hard to adopt a fix research design in a master thesis like this one, where as a flexible design is more appropriate. The research situation in the master thesis may change from time to time when new ideas and changes may be needed; this is not possible in a fix design. As the overall purpose of the research is to investigate the current test process, it matches the flexible design much better.

After the decision has been made to use a flexible research design, the next problem is to determine what research technique to use. Good research techniques for real-world studies involve, as described earlier, case studies, ethnographic studies and grounded theory studies. A comparison between them is illustrated in Table 1 [32, p. 165]:


36

Table 1: Overview of research techniques

Grounded theory Ethnography Case study Focus Developing a theory

based on data from field

Describing and interpreting a cultural and social group

Developing an in-depth analysis of a single case or multiple cases

Discipline origin Sociology Cultural, anthropology, sociology

Political science, sociology, evolution, urban studies

Data collection Typically interviews with 20-30 individuals to saturate categories and detail theory

Primarily observation and interviews during extended time in the field

Multiple sources – documents, archival records, interviews, observations, physical artefacts

Data analysis Open coding, axial coding, selective coding, conditional matrix

Description, analysis, interpretation

Description, themes, assertions

Narrative form Theory and theoretical model

Description of the cultural behaviour of the group

In-depth study of a case or cases

Due to the purpose of the master thesis, the ground theory can be ruled out; the idea is not to develop a theory. The alternatives are case studies or ethnography studies. A more descriptive overview of both case studies and ethnographic studies follows [32]:

Case Study: This research strategy is a well-established technique where a case is being focused on. A case can include a person, a group, a setting, an organization etc. Both quantitative and qualitative data can be collected.

Ethnographic studies: This technique is also well-established, here the focus is on the “the description and interpretation of the culture and social structure of a social group” [32]. Various methods can be used for gathering data; typically participant observations are used over a period of time.

The choice is to use a case study approach which meets the goals of the thesis in the most satisfactory way. The test process within LOGE is to be evaluated and the chosen strategy offers the most freedom when carrying out the study.

5.3 Case Study

Case studies are particularly useful when answering questions like “how” or “why” in organizational and management studies [34]. Case studies can be three different types - exploratory, descriptive and explanatory. Not only positive things are involved in the use of this technique, some drawbacks comprise [34]:

• Lack of rigor of case study research, findings and conclusions may be influenced by non relevant information.

• Provides little basis for scientific generalization.


37

• May take too long to conduct the research with case studies, could result in a lot of documents.

• No way to ensure or testing a person’s ability to write good case studies.

To be able to design a good case study, certain steps can be followed as described in the coming section.

5.3.1 Design

To make the most of a case study design, five steps should be taken into consideration [34]:

• Study questions: The study questions shall answer the “how” and “why” regarding the research. The questions will help the researcher to stay on course throughout the study.

• Propositions: The propositions determine what to study and examine, to ensure that the research stays in scope. The questions don’t show exactly what to study, they propose the direction of the study.

• Units of analysis: The unit of analysis is the actual case to be studied. The case can be an individual, event, organization etc.

• Logic linking the data to the propositions and criteria for interpreting the findings: This step involves data analysis and to link the data to the propositions. This is necessary to make the right conclusions about the findings.

5.3.2 Data collection

There are numerous techniques available to gather the relevant data for the research study. To make the most of the data gathering, a combination of as many techniques as possible is favourable [34].

Documentation is often useful to all case studies and can take many forms. Examples of different documents are: letters, agendas, administrative documents, formal studies and newspaper clippings [34]. Also archival records can be of use, examples are: service records, organizational records, maps and charts, survey data and personal records [34].

Interviews are a good way to address a certain subject and constitute an important part of the case study. There are a number of different interviews; open-ended, structured and semi-structured [27]. The structured interviews may be combined with a list of questions defined before the interview and they constitute the scope of the study. This technique can be seen as an oral questionnaire. The semi-structured interviews on the other hand contain a list of questions that will guide the interviewer. The order and the content of the questions can be modified depending on the situation. In the open-ended interviews it is entirely up to the person being interviewed to determine what is to bring up, the information must be in line with the scope of the current subject.

By making a visit directly into the field of the current “case” one can make direct observations, where information can be gathered both with formal or casual data collection techniques. Participant observations is another approach to be used, here the observer can take on different roles and be part of the case study in order to collect relevant information.


38

The strong and weak points of the six major techniques are described below [34]:

Source of Evidence Strengths Weaknesses Documentation Stable-can be reviewed

repeatedly Unobtrusive-not created as a result of the case study Exact-contains exact names, references, and details of an event Broad coverage-long span of time, many events, and many settings

Retrievability- can be low Biased selectivity-if collection is incomplete Reporting bias-reflects (unknown) bias of author Access-may be deliberately blocked

Archival records Precise and quantitative and [same as for documentation above]

Accessibility due to privacy reasons and [same as for documentation above]

Interviews Targeted-focuses directly on case study topic Insightful-provides perceived casual inferences

Bias due to poorly constructed questions Response bias Inaccuracies due to poor recall Reflexivity-interviewee gives what interviewer wants to hear

Direct observations Reality-covers events in real time Contextual-covers context of event

Time-consuming Selectivity-unless broad coverage Reflexivity-event may proceed differently because it is being observed Cost-hours needed by human observers

Participant observations [Same as for direct observations] Insightful into interpersonal behaviour and motives

[Same as for direct observations] Bias due to investigators manipulation of events

Physical Artefacts Insightful into cultural features Insightful into technical operations

Selectivity Availability

5.3.3 Data analysis

After data has been gathered it is relevant to make an analysis to draw the right conclusions. Without the conclusions, the whole case study research lacks point. It is important to make a proper analysis of the data to make it reliable, valid and representative to ensure that it reflects and meets the goals of the object being studied. Three components concerning the data analysis process include: data reduction, data display and drawing conclusions [32].

Data reduction: When gathering qualitative data it is easy to accumulate a large quantity. Being able to reduce this amount and gather the most important aspects is significant for the success of the research. The reduction is iterative, which means it shall start and be ongoing during the


39

whole research process. The idea is to keep the data manageable and to make early decisions on how to deal with it. This can be done by using summaries, memos, coding etc.

Data display: Just as important as reducing the data gathered is the need to display it in a simple and manageable way. One can use matrices, charts, networks and etc. The key is trying to show what the data means and how to read it, so that the right conclusions can be made.

Drawing conclusions: This is the final step in the analysis process; this one as well as the others are iterative and shall be carried out during the whole study. Conclusions are drawn from making the right interpretations of the derived result. The verification process of checking that data and conclusions are valid and reliable is a significant process during the research. This involves finding evidence confirming the goals of the study as well as using other methods to get results with substance.


40

6. Case Study at LOGE

6.1 Case design

An overall goal of this master thesis is to examine and investigate how the test process in LOGE is carried out today. This provides an idea of what improvements to suggest and what testing methods and techniques that can be adapted. It does not only concern the current test practice, but also what the employees at LOGE desire and require. Also organizational factors have to be examined and taken into consideration when drawing conclusions.

6.1.1 Study Questions

The following questions will be relevant for the research to take place:

• What testing is done today?

• What can be improved?

• What is the general approach to testing in the organization?

• What, for the people working at LOGE, is important regarding testing?

6.1.2 Purpose & Propositions

The purpose of the use-case research is to:

Investigate and find out what is currently being done, and what methods and techniques to use to be able to improve the test process at LOGE to make it as structured and effective as possible.

Propositions for the research involve the following areas:

• Find positive and negative aspects within the current test practice

• Among well known test methods and standards, apply those that can be of use.

6.1.3 Units of analysis

The units of analysis for this master thesis are:

• Test process and its current application.

• Improvement suggestions for the current process.

To look deeper into the units, certain attributes have to be derived for closer investigations; these are taken from another use case research study used in a master thesis by Sara Nordén [30]:

• From whom will the data be collected?

• When will the data collection take place?

• The scope of the data collection


41

The sampling strategy to be used in the research is described in Table 2:

Table 2: Sampling strategy overview

Units of analysis Sources Time of execution Scope The test process and its current application

Employees/developers

In the beginning of the first data collection round

Collect data giving an overview of the current test process. Gather information regarding areas of improvement and the employees’ approach to testing

Improvement suggestions for the test process

Employees/developers and theory

After improvement of the current test process based on conclusions from the previous data collection round

Collect data to see if the suggested improvements are satisfactory to the organization

6.1.4 Linking data and criteria

To be able to link the relevant data to the propositions, certain areas need to be analysed and connected. The area of interest which affects the improvement suggestion involves:

• Software test theory

• Development process

• Current test process

• Improved test process

• Employees’ opinions and attitude

These areas can be linked up as illustrated in Figure 11 and explained in Table 3:

Figure 11: Linking of areas of analysis and propositions


42

Table 3: Explanation of linking areas

Relation number Comments

1 Test theory will work as a base when developing the improvement test software model

2 Connection between the current test model and the development process

3 Connection between the development process and the improved test process, the idea is to determine when to start testing in the development

4 The positive aspects, if any, of the current test practice will be maintained and used in the developed test model

5 Employees may have improvement suggestions, their opinions are important. Get their approach and attitude towards the improved test process

6 Both positive and negative aspects from the employees’ point of view reflect on the current test process. Determine what is good to use within the new improved model

6.2 Collecting Data

Collecting the relevant data for the research is crucial for a well performed study. A number of different techniques can be used as described in the previous chapter. In the first data collection round, the goal was to get an understanding of how the current test process is executed in the organization. The second data collection round is to evaluate if the suggested changes are satisfactory for the people working at LOGE.

6.2.1 First round of data gathering

In order to gather data about the current test situation and about what could be improved, a semi-structured interview was used. It was guided by a list of questions that the interviewees took part of before the actual interview. The reason for this was to let them start thinking about their test situation and related issues, so that questions and any areas of confusion could be settled during the interview. The interview data was gathered by myself and written down; some of the questions were answered before the interview took place. The questions used can be seen in Appendix C.

Being able to understand what the employees consider important quality aspects when developing software is necessary and to determine this. Some quality attributes were gathered on a sheet which can be seen in Appendix B. To enable the interviewees to make a prioritized list of the three most important ones, this sheet was sent out prior to the interviews. By doing this, the organization was made aware of which quality aspects to have in mind when creating software.

Due to organizational factors within LOGE and the lack of a proper testing structure, the only way to perform the study was by using interviews and questions. This made it impossible to use other data collection techniques like test documentation and archival records. Another drawback concerns the few employees in the company, where the statistical data forming the basis of decision will not be as satisfactory as in a larger company. The reliability can in this case not be as high as in a bigger company with more employees, and which uses some sort of documentation and records. Some reliability was however achieved by sending out the interview material in advance to start a “mind” process regarding quality in software as well as the current test process. To ensure some validity of the gathered data, any misunderstandings


43

and areas of confusion were dealt with directly. This was the main reason for distributing the interview material in advance.

6.2.2 Second round of data gathering

From the first data gathering round as well as the analysis, a solution proposal could start to take form. The solution was based on the current structure of the organization, employees’ requirements and the testing theory derived from the research done in the beginning of the master thesis work. Because this work is of an exploratory and descriptive nature with the main focus on suggesting improvements, it is very important to verify and ensure that the results are valid and credible. Otherwise it is not meaningful to use the result for any decision making.

The evaluation is based on a presentation of the proposed solutions to the employees of the company. Without their opinions and approval the result would not be satisfactory to LOGE or me. To gather the relevant and necessary opinions and feedback for a final solution, a short PowerPoint presentation of the conclusions and suggestions was held. After the presentation there was a discussion about ideas and further improvements. Comments presented during this discussion were noted in writing by me. This presentation was not my only opportunity to get relevant feedback from the employees. During the whole work study there have been continuous discussions to get relevant input.

6.3 Data Analysis

The data analysis is carried out in two separate steps. The first one includes analyzing the information gathered in the first round of data collection. This leads to the second analysis of the suggested improvements. The data is reduced and displayed and conclusions will be made as described in the previous section. An overview of the process can be seen in Figure 12.

Figure 12: Overview of the analysis process

6.3.1 First analysis

The interviews were conducted with the three employees working at LOGE. None of them have any real experience in the software testing area. Their expertise is with physics and the first person is a PhD-student in combustion physics with an MSc in computer science, the second person has a PhD in combustion physics and an MSc in environmental physics and the third has an MSc in physics. All three works as developers in the company, some only with the FORTRAN code and others with the GUI, developed in JAVA.

The basic structure of the testing performed today is of an ad hoc nature, where the individual experience determines the amount of time and areas devoted to testing. Unit testing is the most frequently used testing technique to ensure that functions and classes work as intended. Validation and verification of the models implemented are done towards analytic data to see that they are correct.

Neither test documentation nor test planning is used in the organization, and there are not any people responsible for the testing process. Until June this year, one person was employed


44

to work solely with testing, but that is no longer the case. There are no problem reporting systems at hand, but one intended to use BugHost (www.bughost.com). For CM (content management) however, a CVS system is used today.

Tools currently used for testing are the debugging mode in Compaq Visual FORTRAN as well as the debugging mode in NetBeans. A collection of own written routines are used to test some functionality with in the FORTRAN code. From the inquiry into which tools that may be useful in the company, people wished for the following:

• Unit testing tool for the FORTRAN code.

• A java interface tool to run multiple simulations on models with pre-defined experiments and gather the result. After that, it shall compute the accuracy of the result and to be summarized in a report, for example a text file.

• A problem reporting tool

• A tool for testing GUIs, capture/replay tools.

From the inquiry into what the employees thought could be introduced regarding testing in the company, numerous subjects where identified involving:

• A basic test policy with targets.

• Goals to test against

• Test plan with pass/fail criteria

• Standard test cases, covering the basic functionality in the FORTRAN code

• Problem reporting syntax

• Checklists for test plan, GUI, integration and unit testing.

• Some sort of automation tool for the test cases.

• Test cases for the GUI.

From the distributed sheet of attributes, a prioritized list was derived as illustrated below: Attribute 1 Attribute 2 Attribute 3 Person 1 Accuracy: For

engineering applications it is vital to have accuracy to a specific level

Time behaviour: Calculations need to have optimal time behaviour

Operability: The user should feel that he/she understands the complexity of our software compared to others.

Person 2 Recoverability: A program that crashed is not accepted by users.

Understandability: The user should not be forced to spend too much energy on adapting to a new system when approaching the program.

FORTRAN: time behaviour JAVA: Changeability, Allowed to be changed to meet different customers’ requirements. LOGE both a consulting and adopting organization

Person 3 Suitability: Main goal of DARS is to give

Time behaviour: CPU time consuming

Changeability: Different customers


45

the user ability to calculate that which needs to be calculated

calculations to reach certain accuracy. Make program as fast as possible for a given accuracy.

have different goals, they change over time. DARS needs to have high changeability to meet the customer’s needs. Feels like this is a problem today

As seen in the table above there is a difference between those who develop code in FORTRAN and the person using JAVA. For the FORTRAN part of the software, the major importance is on accuracy and time behaviour of the calculations, and also the ability to offer customers the right functionality. On the other hand, it is equally important to have recoverability and understandability in the system, in order to make the software behave and work as intended. The conclusions that can be drawn from the quality attributes are that the software should:

• Offer fast calculations with a specified accuracy.

• Crash as little as possible.

• Be easy for users to understand the software and what functionality is offered.

• Be changeable, to meet goals from different users.

Another part of the research was to gain comprehension of what the employees in the organization thought was important for the creation of a successful testing process. The reason that I wanted to gather their opinions has to do with the overall approach towards testing and getting an insight into areas that they felt could be of use. The priority among the areas is illustrated in Table 4:

Table 4: Areas of importance for testing

Person One Person Two Person Three Test plan 2 1 1 Checklists 4 5 2 Test environment 3 2 4 Problem reporting 5 4 3 Walkthroughs 6 6 6 Common code stand. 7 3 5 Other (describe) 1 - -

In Table 4, the rating value of area one is the most important and seven is the least important according to the employees. To have some sort of test plan seems to be one of the most highlighted areas; person 1 stated under “others” that targets and objectives for testing was the number one alternative. This area, in my opinion, belongs to the test plan area. To have a well structured test environment and to use a problem reporting system will come second and third. However, one person thought that checklists were important to get a rating of two. Walkthroughs and common code standards are less significant to the organization and that I can agree with, depending on the size of the company. Walkthroughs take too much time to perform and a common code standard has no real impact when the development team only consists of three persons.

From the above analysis one can draw some preliminary conclusions regarding a list of things to introduce into the company, such as:


46

• Overall test plan with targets and pass/fail criteria.

• Checklists for various activities, for example test plan creation, integration, unit and GUI testing.

• Evaluation and creation of structured test cases, both for the FORTRAN code and the GUI, how this shall be done and what techniques that can be used are questions to answer. Unit testing tool for the FORTRAN code and a test tool for the GUI are also two important areas to consider.

• Investigation of automation tools and seeing what can be automated.

• Introducing a simple and basic problem reporting syntax.

• With the quality attributes in mind, developing and maintaining the DARS software to make it as effective as possible. This involves:

o Crashing as little as possible o Offering good accuracy in the simulation result o Usability o Changeable to meet future goals

6.3.2 Second analysis

The purpose of this analysis was to gather relevant feedback on my proposed solutions to the organization. This analysis is based on the presentation I held in the second data gathering round. The presentation lasted for about 40 minutes and the participants present involved employees and both my supervisors. After the presentation, a discussion was held regarding the material. The general solution presented is described very briefly below:

• Use a quality framework, in this case the minimal test practice framework (MTPF) to make the test process structured and organized.

• Use a common problem and experience reporting syntax, for reporting bugs and anomalies in the software. There are various tools for this, for example BugHost and BugZilla.

• Define responsibility in the organization regarding testing, update test plans and other relevant documents when necessary.

• Structured V&V process regarding system, integration and unit testing. Develop test cases from use cases when necessary as well as checking the accuracy of the simulation results. Define suitable integration modules and how these can be tested.

• Have a well organized test environment.

• Test planning; the test process must be planned and documented. Documents that must be created involve test plans, requirement specifications and test specifications.

• Investigation and presentation of various test tools.

The discussion did not only have to do with the solutions, it also concerned the goals that were decided upon in the beginning of this master thesis. Since the goals were set in the beginning and the work flow is dynamic in nature, reflections on the matter were useful, and more about this is found in the conclusion section. Most of the suggested areas were to a certain extent already known to the employees of company – it is difficult to work for so many weeks without presenting or revealing something. Conclusions that could be drawn from the presentation include:


47

• Seeing whether it would be possible to adapt the documentation to suit LOGE better, scaling it a bit more. This involves documents like test plan, requirements specification and test specification.

• Performing a workshop where I demonstrate the programs tested and so on.

• Making templates of the produced documents so they can be used by the company in the future.

The final solution based on input from the case study research is described in detail in the coming sections.


48

7. Proposed Solutions

7.1 Introduction

Due to the fact that LOGE is a small and a growing company, the MTPF framework matches the situation quite well. At the moment there are only three employees, and that should of course be considered when choosing what techniques to use. The benefit of the MTPF framework has to do with the fact that it is a lightweight approach [25], making it much easier to adapt than other “heavier” frameworks. MTPF also uses well proven practices and techniques giving it substance. From the investigation of the test situation at LOGE, the following improvement areas were identified:

• Problem-reporting syntax

• Someone responsible for test administration

• Checklists for various activities

• Test case investigation

• Overall test plan with targets and pass /fail criteria

These areas match many of the phases in the framework. It is however neither possible nor feasible to use every suggestion stated in the framework. That would make the test process complicated and hard to use throughout the organization. From the identified areas above and the different phases provided by the framework, a tailored version can be derived and adopted. This is illustrated in Figure 12. Tailored phase

Define syntax

Define responsibility

Use of certain checklists

Admin of the test environment, also some test case development

Test plan

Category Problem & Experience Reporting

Roles and organization

Verification & validation

Test administration

Test planning

Figure 12: Overview of the tailored MTPF

7.2 Problem and experience reporting

Having a common problem-recording syntax is essential to make the process of reporting defects and bugs located in the code effective. This is important because developers should be able to correct each others’ faults and defects. If the organization grows and staff is required to work with testing only, they will not be able to carry out any testing if the problem cannot be re-produced. It may also be useful to get a comprehensive picture of the seriousness/priority of problems.

The proposed tool or template to use is provided by BugHost [9] as it offers a basic web-service for reporting bugs in the code. An account was set up at the website for the company to use, and during the end of the master thesis work the employees started to use the service more frequently. To make the adoption threshold as low as possible, the tool was configured


49

to suite the situation at LOGE, and this involved setting up an application list for the different integration projects in the company. This is illustrated in Figure 13:

Figure 13: Integration modules in DARS

It is possible to assign a certain application to a user, that user will be fully responsible for correcting defects in that application area. Other features included are [9]:

• Submitting bugs

• Viewing and editing bugs

• Searching for bugs

• Setting security rights for different users

A submitted bug, found in one of the applications is shown in Figure 14.

Figure 14: Bug details


50

Not only a common problem-recording syntax may be useful, but also a common code standard for the FORTRAN part of the code. This is to simplify for external people (test personnel) to understand code segments, but it is also important for the employees at LOGE to understand each others’ code. It may not be necessary to follow every single rule in the code standard, but those regarding how to make proper documentation are probably the most important ones. By doing this misunderstandings and misconceptions are minimized. An example of a code standard is the “European Standards For Writing and Documenting Exchangeable Fortran 90 Code” [2]. These rules and guidelines were originally derived to aid developers in “European meteorological organizations and thereby to facilitate the exchange of code between them” [2]. However, many of the rules can be adopted by other organizations or used to create an awareness of how to make code more readable and maintainable. This standard involves areas such as [2]:

• External and internal documentation rules

• Coding rules for packages

• Guidance for the use of dynamic memory

• Coding rules for routines

• Banned Fortran features

• Style rules

• Use of new Fortran features

7.3 Roles and organization

Having one person responsible for the administration behind the test process makes it a lot easier. This also reduces the risk of misunderstandings and misinterpretations. Having someone who is responsible for the development of test plans for each new project and other test activities is also crucial.

This area can seem a bit trivial and obvious. However, the lack of an effective and structured test process often depends on a faulty organizational structure where roles and responsibility are not defined. This will lead to material, documentation and other related artefacts not getting any attention. It is important for every new project or release to update and revise matters like test plans, test cases and requirements specifications. These artefacts are the core of a structured test organization. As well as assessing the documents involved, it is equally important when using this type of quality framework to continuously look into the current test practises to see what is needed to be updated for the next phase in the development. As the organization grows the need for new techniques and methods will increase. To make sure that the targeted context has suitable practices the introduction method, described earlier, was developed.

7.4 Verification & Validation

Regarding the verification and validation category, the use of checklists is the best technique for the company on a more general level. The organization is considered too small to be using walkthroughs and inspections; it would simply take too much time. Checklist can be used on many different test levels, from GUI to unit testing [3]. The idea of using these checklists is to identify problem areas in the development phase and to make sure that these areas have been covered in a satisfactory way. As with the coding standards above, it is not desirable to use every rule or checklist question. The main thing is to find the subjects that match the company best. Examples of unit checklist questions are shown below:

• Check I/O interfaces and calls to see if they are correct


51

• Format specifications matching I/O statements correctly?

• Array sizes correct?

• Pointers de-referenced?

• All memory usage de-allocated when leaving a certain module or function?

The first thing to do is trying to determine what to validate and verify as it is not possible to test everything due resource limitations. By doing the research in this thesis it was possible to determine the relevant aspects of the software of importance to test and the less significant parts. The investigation lead to the derivation of the following schematics, shown in Figure 15.

Figure 15: V&V Overview

Figure 15 illustrates a prioritized list of the test levels and the connections with the system requirements specification and the master test plan. Before any testing can be done, a requirements specification must be at hand, e.g. to know what functionality to test. All test related material shall be gathered in the test plan.

The highest priority is given to system testing, to verify that the system meets the right requirements from customers and end users. Testing the system is done by analyzing the result to see if the accuracy is satisfactory. This is done by simulating scenarios and comparing it with real world data. Moreover, the use cases that are possible to perform in the software must be tested to ensure the correct functionality. These use cases can be used as a basis for creating test cases. An example of a typical test case in the DARS software; it can be seen in Appendix G. It is not only useful to create test cases as a quality mark for the customers, it is also helpful for the employees to think through the design of the software.

The second highest priority is given to unit testing of FORTRAN modules and units. In the research it was found that it was not equally important to unit test the Java code. From a test oriented point of view, the technique best suited to use was black box testing. This was the most useful approach to verify that the result from modules/units is correct and within a pre-defined tolerance level. To achieve this, a proposition to use a unit testing framework for


52

FORTRAN was given; a more detailed description of this framework can be seen in Chapter 8.2 of this report. Along with the black box testing, it was found that following the Test Driven Development paradigm (TDD) might be useful, which has been described earlier in the report.

Last on the list is integration testing, and the first thing was trying to determine what was suitable to define as an integration. This area was also part of the research investigation, and after a discussion with my supervisors a definition of an integration unit could be derived as a separate program that is a part of the DARS software. The available sub-programs are shown in Figure 16.

Figure 16: Integration modules in DARS

The idea is to test these programs separately as far as possible to ensure that they work before integrating them with the rest of the software. Checklist questions to have in mind are:

• Has the structure of global data been identified?

• Has drivers and stubs been defined and developed?

• Has the software architecture (software structure) been fully defined?

• Is any regression testing performed as new modules are integrated?

7.5 Test administration

Having a test environment that is available when necessary is an important factor as well as more administrative parts like keeping it up to date and having some sort of documentation. On the part of LOGE, this involves having all the necessary compilers available along with the right operating systems (Windows and Linux). The administration area is as important as the previously described areas, without proper test environment the test process will be ineffective and will reflect negatively on the software.

Being able to design and execute effective test cases is another area of interest to the organization. These must be saved for future use and updated if re-used. The test cases work as a quality mark on the software towards customers.


53

7.6 Test planning

Last but not least is the test planning category. This is probably the most important area and it is absolutely crucial to plan the test process in every detail to make it structured and goal-oriented. This makes the test process effective and people will continuously know what to do. Also, all test related material is gathered in one document. One request from the employees in the organization was to have targets to test against as well as pass /fail criteria and these are described in the test plan. Figure 17 describes the different documents’ roles in the development process.

Figure 17: Documentation process

There are three different document phases in the test process. In phase one a test plan is created based on test planning. The documents needed for deriving the test plan are requirements specifications and architecture documents. These artefacts must exist in order to know what to test and when to stop testing. In the second phase, test cases are created to cover the functionality set in the requirements specification. The test cases are described in detail in a test specification document. When all tests have been designed and specified, it is time to execute them and this is an iterative process, which means that it is ongoing until defects are no longer found. If a test fails, a problem report must be written to document what went wrong and why. When all test cases have passed, an evaluation shall be done in the next phase, e.g. phase three. The test process is finished with the creation of a test evaluation summary, describing the overall result of the different test activities involved.

The following are examples of these documents that have been created to show to the company:

• Test plan: For the coming release to Nissan Motors. A more detailed description of the template of a test plan can be viewed in Appendix A and the more specific test plan for LOGE can be viewed in Appendix F.

• Requirements specification: For the coming release to Nissan Motors. Due to confidential information in this document, only a template can be viewed in Appendix D.


54

• Test specification: Containing test cases from the unit testing of the Woschni module and the Newton Solver (more on this described in the coming section) as well as test cases for DARS ESM. Due to confidential information in this document it is not included in this report.

• Trouble report: Contains information about the problems found when testing the Newton solver. A more detailed description can be seen in Appendix H.

• Meeting agenda: Purpose of continuously having meetings to get feedback and evaluate what has been done, say every third week. An agenda template to follow can be seen in Appendix E.

7.7 Automation

Another sub-goal in this master thesis was finding out whether it was possible to automate any of the testing. Due to lack of time and the vast field of automation, it was not possible to automate anything. Also, it is not feasible at this point to use automation as it requires too much time and resources, resources that are not present at this time. Perhaps in the future, when and if the organization grows, automation could be a good and suitable alternative. However, the investigation of the DARS software and the division of integration modules reveals that it may be possible to automate some of the testing against these separate integrations.


55


56

8. Tool evaluation

8.1 Introduction

One of the goals of this master thesis was to look into some testing tools and development environments to determine whether any of them could work well for the company. There are a number of different tools on the market for the different stages of the test process. An important criterion is to make it as easy as possible for the company to adopt a tool, meaning that the adoption threshold must be as low as possible. Also, there is no interest in spending too much time learning a new tool, since development is the key concern and main focus. Another key factor to have in mind is that LOGE is a very small company with a few employees, making the budget for buying new software limited, and therefore the research focused on finding non-commercial tools that could be of use. This turned out to be difficult, especially for the FORTRAN language. There are not many free-ware test-tools for the FORTRAN code that were relevant to this case. The following sections contain a short presentation of some of the tools found, both commercial and non-commercial, for both the FORTRAN and the Java part.

8.2 FUnit – FORTRAN unit testing framework

One of the key issues for LOGE is having the possibility to make structured unit test cases for some parts of the FORTRAN code. The main focus was to validate that functions and modules produced the correct result and within a given tolerance level, e.g. Black-box testing. With a pre-defined set of input data to a function one can see whether the correct output data was generated.

FUnit [29] is a unit testing framework for FORTRAN modules, where “unit tests are written in Fortran fragments that use a small set of testing-specific keywords and functions. FUnit transforms these fragments into valid Fortran code and compiles, links, and runs them against the module under test” [29]. This framework resembles in many ways the JUnit framework that has become very popular for the Java language. The requirements to be able to use the framework is a Fortran 90/95/2003 compiler working together with the Ruby language (www.ruby-lang.org), which is a dynamic, open-source programming language. Also, the function or sub-routine under test must be located in a module, or else the framework will not work properly.

To show an example of how to use the framework, a decision was made to test both a module for calculating the heat-loss in a Woschni engine as well as a Newton solver used for calculating derivations in chemical systems. To limit the span of this report, only the example from the WoschniFunction-module will be shown as can be seen below:

MODULE WoschniFunction CONTAINS DOUBLE PRECISION FUNCTION HeatLossWoschni(p,t,Volume) !---------------------------------------------------------------------------------------------- ! This function calculates the heat loss /temperature in a SI-engine ! depending on the engine geometry, speed pressure and the ! temperature in the engine in [J/sec/K]. !----------------------------------------------------------------------------------------------- !USE universal_constants, ONLY: pi IMPLICIT NONE


57

! p: pressure [N/m^2] ! t: temperature [K] ! Volume: actual cylinder volume [m^3] DOUBLE PRECISION p,t,volume DOUBLE PRECISION PistonArea, WallArea, u_piston DOUBLE PRECISION v_characteristic DOUBLE PRECISION pi, Bore, Speed, C1, Ap0, DeltaV !------------------------------------------------------------------------------------------------- pi = 3.14 Bore = 0.135 Speed = 1000 C1 = 0.8 Ap0 = 1.2 DeltaV = 2.00d-03 PistonArea = pi * (Bore**2) /4.d0 u_Piston = DeltaV / PistonArea * 2.d0*Speed/60.0d0 v_Characteristic = u_Piston * C1 HeatLossWoschni = 3.26 * Bore**(-0.2d0) & & * (p/1.d3 * v_Characteristic)**(0.8d0) & & * t**(-0.53d0) WallArea = volume/PistonArea * pi*Bore + PistonArea*(1.d0 + Ap0) HeatLossWoschni = HeatLossWoschni * WallArea !----------------------------------------------------------------------------------------------------- END FUNCTION HeatLossWoschni END MODULE WoschniFunction

To be able to use this module separately from its original environment, one had to manually type in some input data for a number of variables like Bore, C1, DeltaV, etc. This is only to make the module much easier to handle. When a module has been chosen for testing, it is time to write some suitable test cases. These are located in a file named “WoschniFunction.fun”, observe that the “.fun” extension must exist to make the framework execute properly. The .fun file for the current module can be viewed below:

DOUBLE PRECISION :: p, t, volume beginsetup p = 0 t = 0 volume = 0 endsetup beginTest insertZero IsTrue(isNan(HeatLossWoschni(p,t,volume))) endTest beginTest valueTestCheckOne p = 138985 t = 600


58

volume = 1.0e-03 IsFalse(HeatLossWoschni(p,t,volume) < 0) endTest beginTest valueTestCheckTwo p = 138985 t = 600 volume = 1.0e-03 IsEqualWithin(1.4871176720156325,HeatLossWoschni(p,t,volume), 0.5) endTest beginTest valueTestCheckThree p = 30000 t = 3000 volume = 0 IsEqualWithin(0,HeatLossWoschni(p,t,volume), 0.5) endTest

As seen above, the first case initializes start values for the prior test case, e.g. insertZero. The tags: beginsetup, endsetup, beginTest and endTest are FUnit fragments that are used and must be written correctly for the framework to translate the test cases into valid FORTRAN code. To test conditions and various result intervals methods like IsFalse(), IsEqual(), IsTrue() and IsEqualWithin() are used. To get a more detailed description on available methods in the framework, the reader is referred to the FUnit API [29]. One of the most frequently used methods for LOGE part is IsEqualWithin(). This method checks that a function or sub-routine returns a value within a specified tolerance level; this is especially good for making test cases for various calculations and to validate the result.

When the WoschniFunction.fun file has been written, the next phase involves trying to translate it into valid FORTRAN code. This is done automatically by the framework by typing: “funit WoschniFunction” in the command prompt and be located in the catalogue where the files are saved. When this is done, two new files have been created: a file named WoschniFunction_fun.f90 and a file named TestRunner.f90. The first file contains the FORTRAN code used to test the module and the second is the main program. Example of the files is shown below:

Module WoschniFunction_fun use WoschniFunction implicit none logical :: noAssertFailed public :: test_WoschniFunction private integer :: numTests = 0 integer :: numAsserts = 0 integer :: numAssertsTested = 0 integer :: numFailures = 0


59

DOUBLE PRECISION :: p, t, volume contains subroutine insertZero ! IsTrue assertion numAsserts = numAsserts + 1 if (noAssertFailed) then if (.not.(isNan(HeatLossWoschni(p,t,volume)))) then print *, " *IsTrue failed* in test insertZero & &[WoschniFunction.fun:10]" print *, " ", "isNan(HeatLossWoschni(p,t,volume)) is not true" print *, "" noAssertFailed = .false. numFailures = numFailures + 1 else numAssertsTested = numAssertsTested + 1 endif endif numTests = numTests + 1 end subroutine insertZero program TestRunner use WoschniFunction_fun implicit none integer :: numTests, numAsserts, numAssertsTested, numFailures

print *, "" print *, "WoschniFunction test suite:" call test_WoschniFunction( numTests, & numAsserts, numAssertsTested, numFailures ) print *, "Passed", numAssertsTested, "of", numAsserts, & "possible asserts comprising", & numTests-numFailures, "of", numTests, "tests." print *, "" end program TestRunner

Both these files are generated automatically by the framework, which saves a lot of time. If any changes are made to input values or anything else, it is enough to change in the “.fun” file only and use the “funit” command once more and all files will be updated accordingly.

However, using this framework does not only offer positive sides, some drawbacks include:

• The compiler must be of a Fortran 90/95/2003 compiler.

• The Ruby language must be installed, which is not straightforward. Also the correct FUnit package must be initialized in Ruby.


60

• Only a limited number of test-function fragments are currently available in the framework.

• Installing the Cygwin software (www.cygwin.com), which is a Linux-like environment for Windows. The reason for this was to have the possibility to compile all the source files with the help of a makefile.

8.3 Eclipse Photran

Since LOGE develops software containing both Java and FORTRAN code and two separate development environments are used, e.g. NetBeans and Compaq Visual Fortran 6: Developer Studio, an investigation of the possibility to find a common environment to work in was valid. This was not as easy as it may seem. One should keep in mind that NetBeans is open-source and that a licence has already been bought for the Developer Studio. However, if it was possible to find an environment that would support both languages and at the same time be open-source it would save a lot of resources, both regarding money and organizational aspects. In the near future, some parts of the FORTRAN code will be translated to C/C++, this also has to be planned for. The idea of introducing a more common and widespread IDE (integrated development environment) is not only due to the constraints on the current one, it also has to do with the possibility to have well working tool integration. Eclipse supports a number of various test plug-in software.

The Eclipse IDE is an open-source and platform independent development environment [12]. It is written in Java and was from the beginning designed for building JEE applications [13]. However, over time several other languages have been supported in Eclipse, for example C/C++ and FORTRAN. The FORTRAN IDE based on Eclipse and CDT (C/C++ Development Tooling) is called Photran [14]. Photran includes [14]:

• Syntax-highlighting

• CVS support

• GUI interface to gdb (Gnu debugger)

• Makefile-based compilation

• Compiler error extraction

Other advantages with Photran contra Compaq Visual Fortran 6: Developer Studio is the possibility to:

• Have a shortcut menu to available functions and sub-routines in a file

• Change compiler, just by changing in the makefile

• The ability to chose what editor to use


61

The Photran environment is illustrated in Figure 18:

Figure 18: Eclipse main windows As seen in Figure 18, there is a shortcut menu to the right, where all sub-routines and functions in the current file are located. This is not possible in the current Fortran IDE, which makes it cumbersome to find a specific function in a large file with thousand lines of code.

The Eclipse Photran software was downloaded and installed on a Windows XP machine. Since my previous knowledge was based on software development in NetBeans, it did not take me very long to get familiar with the interface. When dealing with relatively small source files, the program performed very well and there were no problems executing files with the help of a makefile and shifting compilers within that file (from Fortran 95 to the Absoft compiler). Problems started to appear when trying to import larger files, with thousand lines of code. Eclipse took quite some time to load these files, and when shifting files in the interface this also took quite some time, which is not satisfying. Another problem, regarding the compiler support in Eclipse, is the lack of debugging functionality under Windows for the necessary compilers. Due to the last problem, it is impossible to replace the Compaq Visual Fortran: Developer Studio with Eclipse under Windows, but under the Linux environment it is possible. In the future when the company is gradually changing to use C/C++, Eclipse is a very good all-round IDE to use.


62

8.4 Forcheck

In the search of test tools for FORTRAN development, a tool called Forcheck was found [16]. Forcheck works as a compliment to the compilers to statically analyzing FORTRAN programs and units [16]. With the help of this software, bugs can be detected in an earlier stage in the code and resources can be saved. Features in Forcheck include [16]:

Static analysis In the analysis there are two different types, e.g. program-unit and global program analysis. In the first analysis type, the software checks for example the following items:

• Syntax and type checking

• Unreferenced and undefined items

• Comparing argument lists

• Illegal use of data types

• Indication of truncation and loss of precision

• Tune Forcheck to accept various extensions

In the latter analysis, the program checks for example the following items:

• Subprogram references

• Argument lists

• Detection of recursive I/O and subprogram references

• Consistency of common blocks

• Public module variables

• Consistent usage of include files

Documentation A documentation feature is also available in Forcheck where it is possible to generate [16]:

• Source code listing for program units

• Cross reference table for syntactic items

• Report consisting of a summary of messages and metrics like number of subprograms, number of statements and number of lines.

Extensions and compiler support The software supports the most common language extensions like FORTRAN 66/77/90/95 and High Performance FORTRAN. Moreover, all the necessary compilers are supported such as Absoft Fortran/Absoft Pro Fortran, Compaq Visual Fortran and Intel Visual Fortran and many more.

Forcheck can be a good complement to the current Fortran IDE environment; it supports a number of useful features to make sure that the code is as bug free as possible in an early stage of the development. The software works on the most common operating systems like Windows, Linux and Unix. Along with the Windows version of Forcheck, an optional IDE is provided. This IDE includes features like [16]:

• Project files displayed in an tree structure

• Edit, analyze and compile from IDE

• Viewing output and input source files concurrently

• Jumping into a specific location in source file by using the diagnostic message

• Browsing of the reference structure


63

8.5 StopWatch – A FORTRAN module

One important aspect when developing software, especially simulation software, is short calculation and response times. If these are too long, the software will not work particularly well for the end user. To help with the development to increase performance and time behaviour, a small FORTRAN module can be used called StopWatch. StopWatch is “a Fortran 90 module for measuring execution time of program segments. It is designed to be a portable, easy-to-use means of measuring execution time. It supports the wall clock, CPU clock, a breakdown of the CPU clock into user and system times, and returns all times in seconds” [28]. The clock-module works with compilers like: Elf90, F, FORTRAN 90 and FORTRAN 95. Like the FUnit framework above where the developer has to manually put in calls in the code, it is the same with StopWatch. The main sub-routine calls to put in your own code are the following:

• start_watch()

• stop_watch()

• reset_watch()

• read_watch()

• print_watch()

With the help of this module, multiple watches can also be created as well as the possibility to use watch groups where functions can operate on multiple watches at the same time.

In order to demonstrate how to use this module, one of the developers at the company helped me to find a useful program module. The chosen program to test is called “Ignition Catalyst”, and is a separate program-module integrated with DARS. This program contains numerous different files needed for the calculation of gas properties within a tube. The most important file for illustrating this example is the main program, e.g. we wanted to measure the total execution time. The files required for the clock to work are the following:

• stopwatch.f90: the module containing all the different clock functions, for example to create and stop clocks.

• cpusec.f95.f90: a separate sub-routine for returning the CPU time, this is required by StopWatch. Several versions of this sub-routine are included in the StopWatch package for different systems and in this case the FORTRAN 95 compiler version is used.

• IgnitionMain.f90: The main program for starting the Ignition Catalyst program.

The stopwatch.f90 file is too big to be included in this document, it is proposed that the reader go to [28] for a detailed description. The cpusec.f95.f90 file contains the following:

subroutine cpu_second (cpu,user,sys) implicit none real, intent(out) :: cpu,user,sys ! ------------------------------------------------ ! returns elapsed cpu time since start of job (sec) ! ------------------------------------------------ ! f95 version; uses the standard intrinsic subroutine from fortran 95. ! user and sys are set to -1. because these are not provided by cpu_time.


64

user = -1.0 sys = -1.0 call cpu_time(cpu) return end subroutine cpu_second

The IgnitionMain.f90 file is illustrated below with the StopWatch calls marked; it has been reduced, for easier reading:

PROGRAM ignition USE JacobianMethod0d, ONLY: jac USE FlameType, ONLY: ReactorType,ReactorNo USE flamemodeling, ONLY: redkin USE stopwatch <--------------------------------- statement for locating the right module IMPLICIT NONE

TYPE (watchtype) :: w <------------ declare variable of the right type, e.g. stopwatch DOUBLE PRECISION time REAL :: ProgramTime,StopTime,StartTime LOGICAL LastReactorReached call create_watch(w) <-------- watches must be created before they are used call start_watch(w) <----------- starting the watch… WRITE(6,*) 'Program IgnitionCatalyst PFR' WRITE(6,*) 'Programcode for calculation of chemistry processes in catalysts' WRITE(6,*) 'Version D1.3.18 September 2006' CALL GetLicense() CALL CPU_TIME(StartTime) time = 0.d0 jac = 0 ReactorNo = 0 CALL GetMecAndThermData() CALL InitializeGasSpecies(n_Species) ReactorLoop: DO CALL ReadOneReactor(ReactorNo,LastReactorReached) IF(LastReactorReached) EXIT ReactorLoop CALL InitializeReactorHelpRoutines() IF(redkin .AND. Reactor%CalcVisc)THEN CALL InitTransportData() ENDIF SELECT CASE(ReactorType) CASE(1) CASE(2) CALL Gate_PFR(time) END SELECT


65

CALL CPU_TIME(StopTime) ProgramTime = StopTime - StartTime WRITE (6,*) 'Program executing time [sec]: ',ProgramTime ReactorNo = ReactorNo + 1 END DO ReactorLoop call stop_watch(w) <--------------- this stops the watch! call print_watch(w) <--------------- this prints the measured time! call destroy_watch(w) <------------ destroys the watch to free memory Print*, 'Program exits normally' CALL Exit(0) END PROGRAM ignition

The output from this program can be seen in Figure 19:

Figure 19: Output from Ignition using StopWatch

8.6 Parasoft Jtest®

The FORTRAN part of DARS is the major one, although the Java part with the GUI and its pre and post processors are still important to test. Jtest is a commercial java testing tool developed by Parasoft Corporation [31]. Jtest “provides fast and easy ways to apply a comprehensive set of Java testing techniques. Helps “test early and often” so quality is built into the code, and bugs are exposed upon introduction.” [31]. There are many useful functions and utilities in Jtest, the major ones will be described in more in detail below.

A new technology and function introduced in the latest version of Jtest is BugDetective [7,19,31]. With the help of this utility it is possible to search the code for errors that can lead to runtime bugs and instabilities in the application. The execution of the software, under test, is simulated where execution paths are traced, even those that span over multiple methods, classes and packages [7,19,31]. It is possible, for example, to locate and find null-pointer exceptions, resource leaks (like non-closed sockets) and SQL injections [7,19]. This also means that GUI applications can be tested in a structured way [7].

Another useful feature is the Jtest tracer [7, 19, 31]. This utility generates test cases while the application is running and can then be used to test the current application. The test cases


66

generated are of JUnit type (www.junit.org). If you have a basic GUI application, it is possible when the program is executing for the user, to type in test data and then Jtest will create a test case of that scenario [7].

Other features offered by Jtest include [31]:

• Compliance check against 700 built-in rules of which 100 are security rules [7,19,31]

• Correcting violations of 250 rules

• Identifying and re-factoring duplicate and unused code

• Executing test cases for regression testing

• Monitoring test coverage and providing high coverage using branch coverage analysis

• Ability to go through tests with the help of a debugger

• CVS support

This is only a brief description of some of the functions and features offered by Jtest, it also supports the Eclipse IDE for plug-in. It was not possible to test the Jtest software due to the lack of an evolutional copy.


67


68

9. Conclusion

The main objective and goal of this thesis work is to make the test process more structured

for the organization and to identify the most important areas to test. Due to the company being small, it is neither possible nor feasible to test everything, thus locating the relevant aspects of the software is crucial for a well performed test activity. A major concern when looking into solutions was trying not to get too enthusiastic and wanting to do too much, as this would only lead to a high adoption threshold for the company and provide a result that would not be satisfactory to me or the company.

Having a quality framework for the organization to follow was an important part of the investigation. The answer was to use a “light” framework which is easier to adopt than heavier ones and which will continue to evolve as the company continues to grow. The “introduction” method in the framework helps to make the adoption level as low and easy as possible.

Defining what to test was not as trivial as it seems, this was also a significant part of the case study research. Since the software consists of various programming languages and developers are involved in different parts of the code, they naturally have different requirements in respect of the important areas. The research pointed out areas like unit testing of the FORTRAN code as well as result validation. The key issue is to guarantee certain accuracy in the result towards customers. In addition, some suitable integration modules were located and defined for testing.

During the research, I wanted to discover which quality aspects the employees at LOGE attach great importance to when developing software. This has to do with the fact that software quality depends on the effectiveness of the testing process. By trying to define the most important quality attributes from the company’s perspective, I could get an understanding of which areas to focus on and which of them the employees regard as the most important ones.

The master thesis work has included the production of several documents, for example master test plan, requirements specification and test specification. They serve as a quality mark towards customers and let them know what has been tested and so on. They also provide a plan of the work progress as well as goals for when to stop testing.

When this master thesis work started and the goals for further investigations were decided upon, there was a high level of optimism regarding what to achieve during a certain period of time. As a consequence there was not enough time to attend to everything stated in the goal description. One area that did not get much attention was automation. To automate testing requires a substantial amount of resources, and I do not think it is of much use at the moment. However, further work could include a more thorough investigation of the possibilities to automate as well as a database solution with test cases.

The suggestions which have been presented to LOGE need to be evaluated through practical use throughout the company to enable future adaptation to the company’s needs and requirements.


69


70

10. References

[1] Ahonen, J.J, Junttila, V., Sakkinen, V., ”Impacts on the Organizational Model on Testing: Three Industrial Cases”, Empirical Software Engineering, Kluwer Academic Publishers, Netherlands, vol. 9 2004, pp. 275-296.

[2] Andrews, P., Cats, G., Dent, D., Gertz, M., Ricard, J.L., “European Standards For Writing and Documenting Exchangeable Fortran 90 Code”, Version 1.1, http://www.metoffice.gov.uk/research/nwp/numerical/fortran90/f90_standards.html, last visited in December 18, 2006.

[3] APM, http://www.rspa.com/checklists/, R.S. Pressman & Associates, Inc, 2001, last visited in December 19, 2006.

[4] Astels, D., “Test-Driven Development: A Practical Guide”, ISBN 0-13-101649-0, Pearson Education, 2003, pp. 5-43, 171-174.

[5] Balci, O., “Verification, validation, and certification of modeling and simulation applications”, Proceedings of the 2003 Winter Simulation Conference, 2003, pp. 150-158.

[6] Balci, O., “Quality assessment, verification, and validation of modelling and simulation applications”, Proceedings of the 2004 Winter Simulation Conference, 2004, pp. 116-123.

[7] Bell, J., “JDJ Product Review - Parasoft Jtest 8.0”, JDJ™ - The World’s Leading Java Resource, November 17, 2006, http://java.sys-con.com/read/299985.htm.

[8] Bellanca, R., “BlueBellMouse, A Tool for Kinetic Model Development”, Division of Combustion Physics, Lund Institute of Technology, Doctoral Thesis, Lund 2004, pp. 1-26.

[9] BugHost, www.bughost.com, USA, 2006, last visited in December 19, 2006.

[10] Burnstein, I., ”Practical software testing: a process-oriented approach”, Department of Computer Science, Illinois Institute of Technology, ISBN 0-387-95131-8, Springer-Verlag New York, 2003, pp. 2-131.

[11] Damm, L-O., Lundberg, L., ”Results from introducing component-level test automation and Test-Driven Development”, The Journals of System and Software, Elsevier Inc, 2005, pp. 1001-1014.

[12] D´Anjou, J., Fairbrother, S., Kehn, D., Kellerman, J., McCarthy, P., “The Java™ Developer´s Guide to ECLIPSE”, 2nd edition, Addison-Wesley, Boston, 2005.

[13] Eclipse – an open development platform, http://www.eclipse.org/, last visited in December 12, 2006.

[14] Eclipse Photran - an eclipse-based integrated development environment for fortran, http://www.eclipse.org/photran/, last visited in December 12, 2006.

[15] Eriksson, M., Lilliesköld, J., “Handbok för mindre projekt”, 1st Edition, Liber, 2005, pp. 66.

[16] Forcheck - A Fortran analyzer and programming aid, www.forcheck.nl, last updated in March 31 2006 and last visited in December 13, 2006.


71

[17] Fraedrich, D., Goldberg, A., “A methodological framework for the validation of predictive simulations”, European Journal of Operational Research 124, 2000, pp. 55-62.

[18] Gamma Technologies, http://www.gtisoft.com/broch_gtpower.html, last visited in January 15, 2007.

[19] Grehan, R., “Jtest Continues Its Upward Ascent”, InfoWorld, October 9 2006, pp. 48.

[20] Herbsleb, J.D., Goldenson, D.R.., ”A Systematic Survey of CMM Experience and Results”, Proceedings of 18th International Conference on Software Engineering, Berlin Germany, 1996, pp. 323-330.

[21] IEEE Std 829-1998, “IEEE Standard for Software Test Documentation”, Engineering Technical Committee of the IEEE Computer Society.

[22] ISO/IEC 9126-1:2001(E), International Standard Software Engineering Product Quality Part 1: Quality Model.

[23] Janzen, D., Saiedian, H., “Test-Driven Development: Concepts, Taxonomy, and Future Direction”, IEEE Computer, IEEE Computer Society, vol. 38, no. 9, 2005, pp. 43-50.

[24] Janzen, D., Saiedian, H., “On the Influence of Test-Driven Development on Software Design”, Proceedings of the 19th Conference on Software Engineering Education & Training, 2006, pp. 141-148. [25] Karlström, D., Runeson, P., Nordén, S., ”A minimal test practice framework for emerging software organizations”, Software Testing, Verification and Reliability, vol. 15, no. 3, 2005, pp. 145-166.

[26] Kleijnen, J.P.C, “Verification and validation of simulation models”, European Journal of Operational Research 82, 1995, pp. 145-162.

[27] Lantz, A., “Intervjumetodik: Den professionellt genomförda intervjun”, Studentlitteratur, 1993.

[28] Mitchell, W., “StopWatch”, http://math.nist.gov/~WMitchell/StopWatch.html/, last updated in December 23, 2003 and last visited in December 07, 2006.

[29] NASA Open Source Agreement, “FUnit: a Fortran Unit Testing Framework”, http://funit.rubyforge.org/, last updated in September 15, 2006 and last visited in December 12, 2006.

[30] Nordén, S., “Developing a test method for use in a small software organization – a case study”, Department of Communication Systems, Lund Institute of Technology, June 2002.

[31] Parasoft Corporation, “Parasoft Jtest®”, http://www.parasoft.com, last visited in December 13 2006.

[32] Robson, C., “Real World Research: A Resource for Social Scientists and Practitioner-Researchers”, 2nd edition, Blackwell Publishers, Oxford, UK, 2002.

[33] Sommerville, I., “Software Engineering, 6th Edition”, ISBN 0-201-39815-X, 2001, pp. 420-438.

[34] Yin, R.K., “Case Study Research: Design and Methods”, 2nd edition, SAGE Publications, 1994.


72

11. Appendix A – Software Test plan template

Here is a template of how a Software test plan can look like according to [21] with a general description:

(Agency) (Project)

Software Test Plan

Date: mm/dd/yyyy Version: (n)


73

Purpose

To prescribe the scope, approach, resources, and the schedule for the testing activities. To identify the items being tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risks associated with this plan.

Outline

A test plan shall have the following structure: a) Test plan identifier b) Introduction c) Test items d) Features to be tested e) Features not to be tested f) Approach g) Item pass/fail criteria h) Suspension criteria and resumption requirements i) Test deliverables j) Testing tasks k) Environmental needs l) Responsibilities m) Staffing and training needs n) Schedule o) Risks and contingencies p) Approvals

The sections shall be ordered in the specified sequence. Additional sections may be included immediately prior to Approvals. If some or all of the contents of a section is found in another document, then a reference to that material may be listed in place of the corresponding contents. The referenced material must be attached to the test plan or be available to users of the plan.

1. TEST PLAN IDENTIFIER

Specify the unique identifier assigned to this test plan.

2. INTRODUCTION

Summarize the software items and software features to be tested. The need for each item and its history may be included.

1.1 Objectives

1.2 Background

1.3 Scope

1.4 References

References to the following documents, when they exist, are required in the highest level test plan:

a) Project authorization b) Project plan


74

c) Quality assurance plan d) Configuration management plan e) Relevant policies

f) Relevant standards

3. TEST ITEMS

Identify the test items including their version/revision level. Also specify characteristics of their transmittal media that impact hardware requirements or indicate the need for logical or physical transformations before testing can begin (e.g., programs must be transferred from tape to disk). Supply references to the following test item documentation, if it exists:

a) Requirements specification b) Design specification c) Users guide d) Operations guide e) Installation guide

Reference any incident reports relating to the test items. Items that are to be specifically excluded from testing may be identified.

2.1 Program Modules

2.2 Job Control Procedures

2.3 User Procedures

2.4 Operator Procedures

4. FEATURES TO BE TESTED

Identify all software features and combinations of software features to be tested. Identify the test design specification associated with each feature and each combination of features.

5. FEATURES NOT TO BE TESTED

Identify all features and significant combinations of features that will not be tested and the reasons.

6. APPROACH

Describe the overall approach to testing. For each major group of features or feature combinations, specify the approach that will ensure that these feature groups are adequately tested. Specify the major activities, techniques, and tools that are used to test the designated groups of features. The approach should be described in sufficient detail to permit identification of the major testing tasks and estimation of the time required to do each one. Specify the minimum degree of comprehensiveness desired. Identify the techniques that will be used to judge the comprehensiveness of the testing effort (e.g., determining which statements have been executed at least once). Specify any additional completion criteria (e.g., error frequency). The techniques to be used to trace requirements should be specified.


75

Identify significant constraints on testing such as test item availability, testing resource availability, and deadlines. 6.1 Conversion Testing

6.2 Job Stream Testing

6.3 Interface Testing

6.4 Security Testing

6.5 Recovery Testing

6.6 Performance Testing

6.7 Regression

6.8 Comprehensiveness

6.9 Constraints

7. ITEM PASS / FAIL CRITERIA

Specify the criteria to be used to determine whether each test item has passed or failed testing.

8. SUSPENSION CRITERIA AND RESUMPTION REQUIREMENTS

Specify the criteria used to suspend all or a portion of the testing activity on the test items associated with this plan. Specify the testing activities that must be repeated, when testing is resumed.

8.1 Suspension Criteria

8.2 Resumption Criteria

9. TEST DELIVERABLES

Identify the deliverable documents. The following documents should be included:

a) Test plan b) Test design specifications c) Test case specifications d) Test procedure specifications e) Test item transmittal reports f) Test logs g) Test incident reports h) Test summary reports

Test input data and test output data should be identified as deliverables. Test tools (e.g., module drivers and stubs) may also be included.

10. TESTING TASKS

Identify the set of tasks necessary to prepare for and perform testing. Identify all inter-task dependencies and any special skills required.


76

11. ENVIROMENTAL NEEDS

Specify both the necessary and desired properties of the test environment. This specification should contain the physical characteristics of the facilities including the hardware, the communications and system software, the mode of usage (e.g. stand-alone), and any other software or supplies needed to support the test. Also specify the level of security that must be provided for the test facilities, system software, and proprietary components such as software, data, and hardware. Identify special test tools needed. Identify any other testing needs (e.g. publications or office space). Identify the source for all needs that are not currently available to the test group.

11.1 Hardware

11.2 Software

11.3 Security

11.4 Tools

11.5 Publications

12. RESPONSIBILITIES

Identify the groups responsible for managing, designing, preparing, executing, witnessing, checking, and resolving. In addition, identify the groups responsible for providing the test items identified in section 3 and the environmental needs identified in section 11. These groups may include the developers, testers, operations staff, user representatives, technical support staff, data administration staff, and quality support staff.

12.1 System test group

12.2 Corporate payroll department

12.3 Development project group

13. STAFFING AND TRAINING NEEDS

Specify test staffing needs by skill level. Identify training options for providing necessary skills.

13.1 Staffing

13.2 training

14. SCHEDULE

Include test milestones identified in the software project schedule as well as all item transmittal events. Define any additional test milestones needed. Estimate the time required to do each testing task. Specify the schedule for each testing task and test milestone. For each testing resource (i.e., facilities, tools, and staff), specify its periods of use.


77

15. RISKS AND CONTINGENCIES

Identify the high-risk assumptions of the test plan. Specify contingency plans for each (e.g., delayed delivery of test items might require increased night shift scheduling to meet the delivery date).

16. APPROVALS

Specify the names and titles of all persons who must approve this plan. Provide space for the signatures and dates.


78

12. Appendix B – Quality attribute sheet

Quality attributes in software development [22]:

• Functionality

The ability to deliver certain functions that fulfils needs under defined circumstances. What does the system do to meet the needs?

o Suitability: What is the ability to fulfil a suitable set of functions for specific user goals?

o Accuracy: The ability to sustain the correct result within a given precision. o Interoperability: How well does the system coup with surrounding systems? o Security: Can the system protect data against unauthorized users?

• Reliability How good is the ability to sustain a certain specified function during specified circumstances?

o Maturity: A system with a high level of maturity has a low level of latent defects and a lower probability to malfunction.

o Fault tolerance: Can the system still perform despite of latent defects? o Recoverability: To witch extent can the system recover from failures?

• Usability How easy is the system to use for specific circumstances and for specific users?

o Understandability: Does the users understand what the system can do and how to solve a given problem?

o Learnability: How easy is it for users to learn the system? o Operability: How easy is it for users to control and maintain the system? o Attractiveness: How attractive is the system to the users?

• Efficiency The ability to maintain a certain performance in relation to the amount of resources needed under specific circumstances.

o Time behaviour: The ability to give short responses, low calculation times and high throughput rates.

o Resource utilization: The ability to use available resources of different kind like disc space and primary memory in a satisfactory way, e.g. capacity.

• Maintainability How easy it is to change the system, referring to adoptions, corrections and the adding of new functionality?

o Analysability: How easy is it to identify parts that need modifications? How easy it is to identify reasons of failure?

o Changeability: How easy is it to make a specific modification to the existing system? o Stability: How great is the risk that the modification will make undesirable effects?


79

o Testability: How easy is the system to validate and how easy it is to determine whether a modification meet the specification and expectations.

• Portability How easy is it to transfer the system to another environment?

o Adaptability: How easy is it to adopt the system without any other means except those offered by the system itself? What level of difficulty is there to scale the system?

o Installability: How easy to install the system in it’s right environment? o Co-existence: How easy is it for the system to coup with surrounding software in

the same environment using the same shared resources? o Replaceability: How easy is it to replace the current system with another specific

system with the same purpose in the same environment?

• Compliance For each of the quality attributes above, one can ask questions about how to achieve them according to standards, conventions, laws and so on.

• Quality in use How well does the system support users to achieve specific goals in a specific situation? Dimensions for describing the system as a component and where the user interacts with the different parts of the system.

o Effectiveness: What is the systems ability to support users in their achievement of goals in a specific situation?

o Productivity: How well does the system support users to spend a reasonable amount of resources (time and money) in relation to the achieved effectiveness?

o Safety: How well does the system support safety to ensure that humans, system, property or environment don’t get hurt?

o Satisfaction: The ability to satisfy users in a specific user situation.


80

13. Appendix C – Interview Questions Down below the questions from the interview are listed, questions 19-25 was collected from another master thesis which I thought was relevant for my work [30].

1. Name

2. Education:

3. Work title (developer/tester etc.):

4. How long have you worked at LOGE?

5. Work tasks:

6. Any testing performed today?

7. If yes to the pervious question, what methods and techniques are used and to which extent?

8. Any documentation regarding testing? 9. Any test planning within the organization?

10. Anyone in the company responsible for testing?

11. Any tools used for testing of any kind?

12. Any tools that you would like to use?

13. Is a quality model used within the organization?

14. Any test plan available in the company?

15. What would you like to introduce in the company regarding testing? Something to improve?

16. Are there any problem reporting syntax for the company when finding faults in the software? 17. Is either checklists or walkthroughs used?

18. From the list with quality attributes, list the three that you feel are the most important for your organization. One is the top priority and three the lowest. Also make a short description to each why you think that particular one is important:

1.

2.


81

3.

19. Arrange the following areas regarding their importance for successful testing! Start with the most important! A. Test plan B. Checklists C. Test environment D. Problem reporting E. Walkthroughs F. Common code standard G. Other (describe) 20. Arrange the following areas regarding the importance of their planning! Start with the most important! A. Test environment B. Time plan with milestones C. Approach D. Pass/fail criteria E. Change management F. Responsibilities G. Other (describe) 21. Would you be interested in planning the test phase? Motivate!

22. Arrange the following checklists regarding their usability in testing! Start with the most usable! Checklist for: A. GUI testing B. Walkthroughs C. Unit testing D. Integration testing E. The test plan F. Testing on different platforms (OS, browsers etc) G. Other (describe) 23. Arrange the following alternatives regarding their importance for creating an effective test environment! Start with the most important! A. Someone is responsible for the test environment B. The accessibility of the test environment C. That the test environment is realistic D. Other (describe)


82

24. What do you think about the reporting syntax template below? Motivate!

25. How easy/hard is it to orientate in code that someone else has written? A. I can understand the function of the code if a read it through but I find it hard to find and correct faults. B. I can correct a fault that has been pointed out by the author C. I can find and correct faults in code that someone else has written D. Other (describe)


83


84

14. Appendix D – Requirement document template

LOGE AB <Template>

Software Requirements Specification

Revision history

Revision Date Author Comment


85

Table of Content

1. Introduction

Purpose Overview Terminology References

2. General description of the system

Main functions System context

3. Functional requirements

4. Non-functional requirements

5. Additional requirements


86

1. Introduction

Purpose

<A short description of this document and the purpose behind it. Also a short statement of the content and the structure which involves how the numbering of requirements shall be presented.>

Overview

<An overview of the intended system is provided. Main functionality and purpose is stated.>

Terminology

<Specific words are explained.>

References

<List all other documents that will help clarify the requirements. This can involve customer wishes/expectations on the system as well as functionality, meeting protocols and UML diagrams.>

2. General description of the system

Main functions

<A list containing the main functions.>

System context

<Describe the systems relation and dependencies to other systems.>

3. Functional requirements

<State what the system shall do. Number the requirements so each may be identified easily, this because cross-reference is common. Example: SRS02-3.1.1 (SRS02 is the document name and 3.1 is first requirement).>

4. Non-functional requirements

<List non-functional constraints for the system like interface requirements, design constraints and so on.>

5. Additional requirements

<List additional requirements like quality attributes such as reliability and accuracy.>


87


88

15. Appendix E – Meeting agenda Here is a template of how a status report for a meeting can look like according to [15]:

Status Report Current situation

• What has happened since the last occasion?

• What is the current situation?

State of resources

• Comments on the amount of resources consumed and on how much that has been delivered.

• (In industrial projects also the financial situation is commented on)

Problems/proposed action

• Problems that Management should know about, and proposed action

Risks/proposed action

• Follow up the previously made risk analysis.

• Risks that Management should know about, and proposed action in order to minimise the respective risks.

Project changes compared with project plan:

• This is where project changes are made and where they are formally made visible. E.g. additions, organisational changes, documentation added/cancelled, review added/cancelled.

Enclosures

• Schedule (or scheduled milestones)

• Resource plan


89


90

16. Appendix F – Test plan LOGE

LOGE AB Nissan Motor

Software Test Plan

Revision history

Revision Date Author Comment 0.1 08/11/2006 Richard A. First draft 0.2 29/11/2006 Richard A. Second draft 0.3 12/12/2006 Richard A. Third draft


91

Table of Content 1. Introduction

1.1 Objectives 1.2 References

2. Test requirements

3. Test approach

3.1 Unit test 3.2 Integration test 3.3 System test 3.4 Tools and environment

4. Item pass / fail criteria

5. Trouble reporting

6. Progress and status reporting

7. Roles and responsibilities

8. Deliverables

9. Approvals


92

1. Introduction

Objectives

This test plan describes the overall test strategy being used for the release of the DARS software to Nissan Motor. The purpose of the test effort is to verify that the software fulfils the current requirements and to make the testing as effective and structured as possible. The following items are within the scope of this document:

• A strategy for the test process with criteria’s to know when to stop testing.

• A description of the requirements that must be fulfilled to verify some quality within the software.

• A list of documents derived from the test process.

The following items are outside the scope of this document:

• A budget plan containing cost estimation of man hours and time spent on the testing process.

• A schedule of the test process, hard to derive in an organization like this where the development and testing is very dynamic.

References

[1] Software Requirement specification for Nissan Motor, Doc. No. D0001, Version. 0.3 [2] Test specification, Doc. No. D0003, Version. 0.2 [3] Meeting agenda, Doc. No. D0004 [4] Test Incident Specification, Doc. No. D0005, Version. 0.3

2. Test requirements

All requirements are listed in the Software requirement specification [1].

3. Test approach

The purpose of this chapter is to present the approach to testing in the current project. The previous section, describes what will be tested and this section describes how the testing will be conducted.

The main consideration for the test strategy is the techniques to be used and the criteria for knowing when the testing is completed. The different levels of testing are prioritized according to Figure 1.


93

Figure 3 Test prioritization

The grey dashed area in Figure 1 is the important area for this test plan; it illustrates the different levels of testing that will be performed. The prioritization order will be the following:

• System testing: Ensure that the system will deliver the necessary functionality as well as result that are satisfactory to Nissan Motor.

• Module/unit testing: All new functions and modules shall be unit tested.

• Integration testing: Integration testing will be performed on the most critical merges of units and modules if there are any resources left.

3.1 Unit test

A unit is an individual function, class or module which is produced by a single developer or a close team of developers. The purpose of unit tests is to validate the function of all units prior to integration with other system components. Both internal functionality as well as the interface needs to be tested through unit tests.

Unit tests will be executed on as many critical functions and modules as possible and developers will perform the unit tests. It is not possible or feasible to test every function in the software.

After the tests have completed, a unit test report shall be written containing the result and an evaluation of the current stage. This is done by the test manager or the person responsible for the testing process.

Test design Test execution Input Requirement

specification [1] Test cases / checklist areas


94

Output Test specification [2] Test protocol

Entrance criteria Architecture design

Existing test cases

Exit criteria The test specification is completed

Test cases have been executed and checklists have been checked.

Responsibility of Developer/tester

Developer/tester

3.2 Integration test

In this next phase, we test the integration of functions put together and the interface between them. Integration can be performed on classes (clusters), modules and functions.

Integration tests will be executed on the most complex and critical collection of units and modules and developers will perform the integration tests. It is not possible or feasible to test all integrations in the software. The integration modules are illustrated in Figure 2:

Figure 4 Integration modules

The separate program functions consist of a number of modules and functions. The main software, DARS, consists of all these separate program connected to the GUI. The integration testing process is to ensure that the separate programs work as intended and can communicate with the GUI in a satisfactory way.


95

After the tests have been completed, an integration test report shall be written containing the result and an evaluation of the current stage. This is done by the test manager or the person responsible for the testing process.


specification [1] Test cases / checklist areas

Output Test specification [2]

Test protocol

Entrance criteria Architecture design

Existing test cases

Exit criteria The test specification is completed

Test cases have been executed and checklists have been checked


Developer/tester

3.3 System test

The system is the entire deliverable product, or parts of it, put together. The purpose of system tests is to validate that the system meets its implemented requirements, in both functional and structural ways.

The system tests will check that the correct functionality is implemented towards the end user and that the simulation models offer a certain predefined accuracy in the result.

After the tests have been completed, a system test report shall be written containing the result and an evaluation of the current stage. This is done by the test manager or the person responsible for the testing process.


specification [1] Test specification

Output Test specification [2] Test protocol

Entrance criteria Requirement specification [1]

A test specification exists as well as finished code to test upon

Exit criteria Test specifications, that will cover the whole system, is finished

Test protocols have been filled out for each system test specification


Developer/tester

3.4 Tools and environment

For the FORTRAN part of the code, the following tools are used:

• Own written test routines

• Debugging mode in Compaq Visual Fortran


96

• The FUnit framework for unit tests

• Developer Studio, IDE

For the Java part of the code, the following tools are used:

• NetBeans IDE The software is built for Window XP - 32, Linux Cent OS 4.2 - 32 and 64 bits. The test environment shall be executed on these operating systems. The supported FORTRAN compilers involve Compaq Visual Fortran 6.6C, Absoft 8.2 for Windows and Absoft 9.0 for Linux.

4. Item pass / fail criteria

The test effort will be approved for the Nissan Motor project when:

• All test specifications have been executed and approved and no major defects exist in the software.

• Test reports for the following activities have been created and approved by the project group:

• System testing

• Module/unit testing

• Integration testing

• Update documentation if necessary and approve.

5. Trouble reporting

Severe defects and problems found in the software shall be reported and documented.. The developer that found the fault is responsible for reporting and correcting the anomaly. BugHost (www.bughost.com) will be used as the problem-reporting system.

6. Progress and status reporting

Informal meetings, followed by [3], shall be held once every third week among the employees to reflect over testing and development. Notes from the meeting must be written down and saved in the project folder for future use.

7. Roles and responsibilities

The purpose of this section is to provide information on the responsibilities that are associated with each role. A role can come with many responsibilities but the execution of these may very well be delegated.

Role

Responsibility

Test manager / developer

• Test plan

• Test specification

• Test report Developers

• Execution and evaluation of tests


97

8. Deliverables

The test effort of this project will generate the following deliverables:

• Master test plan

• Unit test report

• Integration test report

• System test report

• Test specifications

9. Approvals Test Manager CEO ______________________ _________________________


98

17. Appendix G – Test case DARS

Test case: Plug Flow reactor (NOoxidization) Pre-condition: Initiate the DARS software and create a new project and import the necessary data files with the help of the “Read Mechanism” module (Gas.inp and Surf.inp).

Reactor properties

Pipe diameter 1 mm

Pipe length 15 mm

Initial Conditions

Inlet Gas Temperature 673 Kelvin

Inlet Gas Pressure 1.013e5 Pa

Inlet Velocity 0.8 m/s

Inlet Viscosity 3.1e-5 kg/(m*s)

N2 inlet mass fraction 0.99932

NO2 inlet mass fraction 0.00068

Chemical Mechanism Olsson et al [1]

Pt(S) site fraction initial guess

1.0

Flow of Events: Basic path

1. The use case starts when the user drags a “Plug Flow” reactor module into the workbench and connects it to the “Read mechanism” module. 2. The user double clicks on the reactor module and clicks on the panel that will appear, selects the option “Create a new single run” and name the test case. 3. The user right clicks on the appearing node and chooses the option “Set parameters”. 4. The user fills in the necessary data among the available options (the cursive means that a button is to push).

Solver Setting Window minimum time step size needs 10-13 Reactor Data window viscosity 3.1*10-5 kg/(m*s) pipe length 0.015 m Y 7.9*10-7 m2 hydraulic diameter 5*10-4 m wall area 3.1*10-3 m Gas Composition window gas temperature 673 K gas velocity 0.8 m/s gas composition / N2 0.99932 gas composition / NO2 6.8*10-4 Surface composition PT(S)


99

surface area per catalyst length 3.1*10-3 Output Options Window Writes species mass fraction profiles Check box

5. When it’s done and the user presses the “OK” button a warning message appear that states that all previous results files will be deleted if one presses the “OK” button. 6. If the user presses “OK”

a. The new settings are saved and a new node appears in the tree which will contain the simulation result. Go to step 8.

7. If the user presses “No” a. A new warning message appears and says “New results directory name is different

from the old one. You can’t keep old result files, go back and erase them”. Back to step 4.

8. The user right clicks on the test case node and chooses the alternative “Run simulation”. 9. A warning message appears and states: “Starting calculations. Click YES to proceed and wait until end window appears”. Contains a “YES” button and a “NO” button. 10. If the user presses the “YES” button

a. Simulation starts. Go to step 12. 11. If the users presses the “NO” button

a. Calculations are aborted, back to step 8. 12. Simulation is proceeding 13. Simulation is done; window appears which the user presses “OK”. 14. User analyzes the result. Post-condition: The simulation result has been saved in different result files.


100

18. Appendix H – Test Incident report LOGE

LOGE AB

Test Incident Report - Newton Solver -

Revision history

Revision Date Author Comment 0.1 04/12/2006 Richard Andersson First draft 0.2 12/12/2006 Richard Andersson Second draft 0.3 16/01/2007 Richard Andersson Third draft


101

Table of Contents

1. Summary

2. Incident description

3. Impact

4. References


102

1. Summary

A number of tests were executed towards the Newton Solver, to see if it could actually solve all the stated mathematical problems. The reason for deriving these tests depends on the fact that the solver has only undergone some basic tests in the past, but that there was interest in seeing how it could perform in other more complex cases.

The test cases with input and output data can be seen in detail in the test specification document [1]. The FUnit framework worked very well to make a comparison of both the input and the expected output data and to see if the result was in the given tolerance interval. These test cases may very well be saved for future use if changes are made to something in the solver.

2. Incident description

The tests were run with the help of the FUnit framework in the Developer Studio environment on a Windows XP machine. The actual testing process was conducted by myself during week 48, 2006. The original code had to be modified a little to suite the framework and to be run separately, these modifications were done by one of the FORTRAN developers at the company. Of 138 possible assertions, 162 passed the pre-defined tolerance level. A detailed description of the output from the test framework is as follows: GateSolver test suite: *IsEqualWithin failed* in test case17 [GateSolver.fun:342] -1.777027357140412D-01 ( -0.177702735714041 ) is not -0.177686181217214 within 1.000000000000000E-005 *IsEqualWithin failed* in test case26 [GateSolver.fun:465] -2.208575158908672D-15 ( -2.208575158908672E-015 ) is not -2.454497866885416E-005 within 1.000000000000000E-019 *IsEqualWithin failed* in test case27 [GateSolver.fun:475] 70.03731057008607D+00 ( 70.0373105700861 ) is not 70.0355627524440 within 1.000000000000000E-004 *IsEqualWithin failed* in test case29 [GateSolver.fun:496] 9.815017249707434D-11 ( 9.815017249707434E-011 ) is not 9.822105940392211E-011 within 1.000000000000000E-015 *IsEqualWithin failed* in test case31 [GateSolver.fun:517] 10.2726D+00 ( 10.2726000000000 ) is not 10.2723098844066 Within 1.000000000000000E-004 *IsEqualWithin failed* in test case32 [GateSolver.fun:527] 0.0D+00 ( 0.000000000000000E+000 ) is not -2.89501194361851 Within 1.000000000000000E-004 *IsEqualWithin failed* in test case33 [GateSolver.fun:539] 0.756245D+00 ( 0.756245000000000 ) is not 0.755621780315063


103

within 1.000000000000000E-004 *IsEqualWithin failed* in test case34 [GateSolver.fun:550] 0.695786D-04 ( 6.957860000000000E-005 ) is not 7.404269715510603E-005 within 1.000000000000000E-008 *IsEqualWithin failed* in test case35 [GateSolver.fun:561] -0.584253D-04 ( -5.842530000000000E-005 ) is not -6.123497818134016E-005 within 1.000000000000000E-008 *IsEqualWithin failed* in test case36 [GateSolver.fun:572] 0.667726D+00 ( 0.667726000000000 ) is not 1.18955739380848 within 1.000000000000000E-004 *IsEqualWithin failed* in test case37 [GateSolver.fun:583] -1.21774D+00 ( -1.21774000000000 ) is not 1.22726457973036 within 1.000000000000000E-004 ERROR: Convergency was not reached. To gain further information, please select "full output" under solver settings. END of error message. Passed 162 of 183 possible asserts comprising 27 of 38 tests. As can be seen above, there is 11 test cases of 38 tests that did not pass, but there are 21 assertions that didn’t go through. This is the case because if one of the assertions in a test case fails, the rest of assertions in the same case will not be executed. The most important question now is why didn’t the above test cases pass? This has to do with a number of things:

• Tolerance level: The level of tolerance might be to low, if one would increase the level, more tests might pass. The downside however is the loss of precision.

• Convergency: There can be several reasons why the solver can’t solve the mathematical problems and not reach convergence. One of these reason can be the actual implementation and that the own written function does not behave as intended. Another reason for not reaching convergence can be that no constraints on the result vector for the mathematical problems have been defined. This means that when iterating over time, the X-value can for example be negative.

• Solver settings: Some test cases might not pass due to bad tuning of solver settings. This is a cumbersome work to select the right values for optimized result.

• Solver: The implementation of the solver might simply be incorrectly implemented.

3. Impact

It is very important to understand why the solver cannot solve certain test cases. As previously mentioned, this can depend on various factors. In the process of trying to locate the problem, it could be useful to start by looking at the solver settings, to see if it is a tuning problem. The solver problem has a high severity because there might be customers that will


104

use input-data when using the software that will re-produce the faults. A prioritized list for the fault location process is:

• Check solver settings

• Adjust the fault tolerance level to see if additional test will pass

• Check the actual implementation of the solver, might have missed something.

Test cases and other relevant material have to be updated depending on the findings.

4. References [1] Test specification document, Doc. No. D0003