1

ATHABASCA UNIVERSITY

MARTS - Multi-Agent Based Regression Testing Suite

BY

Khalid Aziz Muhammadi

A project submitted in partial fulfillment

Of the requirements for the degree of

MASTER OF SCIENCE in INFORMATION SYSTEMS

Athabasca, Alberta

April, 2009

© Khalid Aziz Muhammadi, 2009

2

ABSTRACT

Software testing is one of the major phases in Software Development Life Cycle,

and typically adequate budget is allocation for this activity, however software

quality usually does not meet end-users expectations. This paper attempts to

achieve two goals. First, it highlights the fundamental issues with project planning

and management, which directly impact the quality of the end product.

Inadequate software quality and lack of user’s acceptance of the product, is not

only due to technical reasons, but the combination of technical and non-technical

issues, which includes lack of proper management and leadership, decisions

without appropriate user feedback, and incorrect resource allocation. This paper

highlights some underline testing issues, and how Managers, Developers,

System Testers, and End-User’s play its roles accordingly, to make matter

worse. Secondly, it proposes solution of software testing using Intelligent

Software Agents. Regression testing can be repetitive work, but does require

some level of intelligence; Software Agents are the best candidate for the

Automated Regression Testing. Proposed framework; MARTS (Multi-Agent

Based Regression Testing Suite) is a loosely coupled component-based testing

framework, which uses intelligent Software Agents to simulate system testers,

along with some capability to make autonomous decisions. Automated testing

reduces human error possibilities, and overall cost reduction in long run. MARTS

adopts the benefits of proven Distributed Computing solutions, along with the

flexibility and decision making capability of Intelligent Software Agents. For

3

components like Data-Generator and Reporting, preferred solution is to use off-

the-shelf components and integrate into MARTS. Overall goal is not to use latest

technology, and create cutting-edge technology product, but to address real

business problem’s, i.e. how to improve software quality and end-user

experience.

4

ACKNOWLEDGMENTS

Special thanks to Dr. Qing Tan, University of Athabasca, for his supervision in preparing this essay.

5

TABLE OF CONTENTS

CHAPTER I - INTRODUCTION ...................................................................................... 7

Statement of the Purpose .............................................................................................. 16 Research Problem ......................................................................................................... 16 Definition of Terms....................................................................................................... 17 Organization of the remaining chapters........................................................................ 18

CHAPTER II - REVIEW OF RELATED LITERATURE............................................... 19

Related Work ................................................................................................................ 22 Current Research........................................................................................................... 24 Summary ....................................................................................................................... 29

CHAPTER III - METHODOLOGY................................................................................. 30

Software Agents............................................................................................................ 30 JADE Platform.............................................................................................................. 32 Agent Communication Language (ACL)...................................................................... 34 Multi-Agent Based Regression Testing Suite (MARTS): An Overview...................... 34 MARTS Software Architecture .................................................................................... 36 An Online Shopping Store - Example .......................................................................... 47

CHAPTER IV - ISSUES, CHALLENGES, AND TRENDS........................................... 49

CHAPTER V - CONCLUSIONS AND RECOMMENDATIONS.................................. 51

Conclusions................................................................................................................. 51 Suggestions for Further Research .......................................................................... 52

REFERENCES ................................................................................................................. 53

6

LIST OF FIGURES

Figure 1.0: Software Development and Maintenance Costs (Boehm, 1981) 11

Figure 1.1: Software Development and Maintenance Costs in 487 Business

Organizations (Boehm, 1981) 12

Figure 1.2: Software Maintenance Production Function (Boehm, 1981) 13

Figure 2.0: Software Activity Variations – Percentage of Staff Effort by Activity

(Jones, 2002) 21

Figure 2.1: Test Agent System (Choi, 1999) 25

Figure 2.2: Implementing our testing approach upon JADE Platform - a low level

design view. (Coelho, 2006) 28

Figure 3.1: FIPA reference model of an Agent Platform (Bellifemine, 1999) 33

Figure 3.2: Software Architecture of JADE Agent Platform (Bellifemine, 1999) 33

Figure 3.3 MARTS Framework Architecture 37

Figure 3.4 Framework Controller 38

Figure 3.5 Agents Communication with Framework Controller 38

Figure 3.6 Reporting Component with Framework Controller 39

Figure 3.7 Logging Component with Framework Controller 40

Figure 3.8 Framework Controller with Test Data Generator 41

Figure 3.9 Framework Controller with Test Results Analyzer Service 41

Figure 3.10 Framework Controller with Test Scenario Agent 42

Figure 3.11 Test Scenario Agent with Test Data Generator 43

Figure 3.12 Test Scenario Agents with Logging Component 43

Figure 3.13 Test Scenario Agent with Unit Test Wrapper 43

Figure 3.14 Test Scenario Agent with Test Results Analyzer 44

Figure 3.15: SQLEdit DTM Data Generator (SqlEdit, 2009) 45

Figure 3.16: TurboData Data Generator (Turbodata, 2009) 46

7

CHAPTER I

INTRODUCTION

Software testing is one the most important, yet frequently ignored subject area in

IT industry. This is also one the major reason why software maintenance cost is

very high. Third party software providers or (ISP’s) are using better standards

and techniques to improve software quality. However, this is still a major problem

for in-house development. Project Manager’s tend to ignore this phase, or

assign this task to the people with low technical skills. When programmers do not

perform well, they are typically assigned testing work. Writing quality code

requires significant level of technical skills, however, testing the feature requires

in-depth understanding of the business, and the detail study of business

requirements for the application. Unfortunately, software testers usually do not

have adequate business knowledge of the application or understanding of

business requirements. Software testers are given list of test cases for execution,

which are written by the business users and typically reviewed by software

developers. One may think, if test cases are written by the business users and

reviewed by developers, than it should be adequate enough for the software

validation and achieving required software quality.

Writing test cases and creating test plans is the biggest documented blame game

in Software Industry. It’s well structured, where in the end everyone loses,

however no one can be held accountable for the potential disaster. Therefore it

8

can be called as Software Blame Game. Game is played by four major actors,

and each has predefined set of rules, which dictates how they will play the game.

The Project Manager; always concerned about the schedule and budget for the

project. To expedite the development, the manager will assign the testing to the

weakest person on the project. Appropriate time and budget is allocated;

therefore when the incompetent member of the team is not able to test properly,

Project Manager cannot be blamed. However, manager is aware from day one,

that person assigned for testing will not be able to achieve testing goals; hence

software quality will not be according to what is promised to the end-users.

The Business Users; always concerned about how this new application will affect

there day to day tasks. They will make every attempt to make things more

complicated, and sometime provide details which has nothing to do with the

actual application. Business users have day to day tasks, and when new IT

project is initiated, they are expected to contribute and make that project

successful. However, business users day to day tasks should not be impacted

either. Aggressive timelines force them to rush on identifying business

requirements, therefore business requirements tends to be vague, and can

sometime become impossible to create specific test cases to validate those

requirements. Business users are typically aware of IT department’s lack of

information about the business; however they tend to let it slide. If project

encounter issues in production, it can be safely blamed on the IT department.

9

This ensures job security for the business users, and potential re-write of the

project within five to ten years.

Software Testers; execute the test cases according to the predefined test plan

which is created by business users and reviewed by Software Developers. They

will run exactly what is asked for, and will try their best not use their mind and go

extra mile to validate the scenario, which is not documented. If software tester

attempts to go extra mile, they will be discouraged by the Developers and Project

Manager, since tester will spend more time testing, which could impact project

timelines. Therefore, it is not the fault of Software Testers if software quality is

not achieved, because they are not given enough time and development support

to go extra mile.

Software Developers; can also be identified as construction workers, work over

time, and meet their deadline, at the expense of product quality. They tend to

code the software feature without reading the business requirements. They never

accept their fault and can get offended easily when asked to unit test the code.

Overall maintenance cost it increasing exponentially, and regardless of

methodologies, and tools, product quality is not up to the client’s satisfaction. In

large organizations, appropriate best practices and software methodologies are

adopted; however, expected results cannot be achieved using available

resources and timelines. As Stutzke states, “Software processes must produce

10

software that can be gracefully evolved at reasonable costs. The choice of the

software architecture significantly influences modifiability and hence

maintainability.” (Stutzke, 1996). In 2001, “more than 50 percent of the global

software population was engaged in modifying existing applications rather than

writing new applications.” (Jones, 2002)

Since programs are not tested adequately, future enhancements are not easily

doable, and in some cases, software project requires potential re-work during

maintenance. “All programs should be written with the expectation that they will

be changed later. They should be judiciously modularized, well-commented, and

written in proper style so that the effort required to understand them later in order

to make changes is significantly less than that of starting over.” (Conte, 1986)

Software products with higher maintenance usually run into issues of estimation.

It is nearly impossible to accurately estimate the maintenance work on the

application, which was not tested properly in initial release. Some documented

functionality may not exist, and other may or may not work as documented.

“Development estimating is a difficult problem, but estimating maintenance and

enhancements is even more complex because of the very tricky relationship

between the base application that is being modified and the changes being made

to accommodate new features or repair defects.” (Jones, 1998). In order to

ensure that software product is reliable, low maintenance, and ease in future

enhancements, one need to make sure that product is properly tested. "Software

11

testing will remain one of the best tools we have at our disposal to ensure

software dependability." (Briand, 2007)

Figure 1.0: Software Development and Maintenance Costs (Boehm, 1981)

12

Figure 1.1: Software Development and Maintenance Costs in 487 Business Organizations

(Boehm, 1981)

13

Figure 1.2: Software Maintenance Production Function (Boehm, 1981)

There are quite a few methodologies that can be used for software testing. Unit

tests can prove to be very effective, when used for regression testing. This allows

testing to be performed at function level, and each component can be tested at

its core.

"A growing trend in software development is to use testing frameworks such as

the xUnit frameworks (for example, JUnit, JsUnit and NUnit) which allows the

code to conduct unit tests to determine whether various sections of the code are

acting as expected in various circumstances. Those frameworks do not fit for

testing distributed real-time systems." (Yu, 2007). Automated testing using unit

tests can help to improve software quality, and reduce the testing and

14

maintenance cost. For every software build, unit tests can be executed to ensure

no new defects are introduced. Automated testing also guarantees that test

cases are not skipped, and report can be validated to verify the results. As Liu

states that “automated test can run more test cases in a given time, make better

use of resource”. (Liu, 2003). This also reduces the overall testing cost. Test-

driven Development (TDD) is one of the popular evolving agile methods. “TDD

stresses the development of working code over documentation, models, and

plans. The suggestion of TDD is (paradoxically) that developers test the program

before it is fully written. A striking aspect of this approach is the idea that the

code that is implemented may not be behaviorally correct: it just has to pass the

test. The test is therefore the specification against which the code is checked."

(Ostroff, 2007)

As David Kung, in his paper highlights the fact that, Dynamic behaviors,

Heterogeneous representations, Uncertainty in execution and Novel data

handling mechanism in web based applications makes testing extremely difficult

to test. He states that, "web applications [are] much more complicated than

traditional applications. Traditional testing approaches are no longer adequate for

Web applications because they cannot handle such complicated situations. On

the other hand, most available approaches for Web applications testing only

focus on usability, conformance, performance, etc. The lack of mature and

systematic approaches and tools forces Web applications testing to rely on ad

hoc solutions and the experience of developers. This adversely affects the quality

15

of Web applications. Consequently, systematic as well as flexible and extensible

testing approaches and intelligent tools are in urgent demand." (Kung, 2005)

"Software testing remains the primary technique used to maintain quality of the

software products and to gain consumers’ confidence in software. Unfortunately,

it is well-known that testing software is a time-consuming and costly process.

Therefore, techniques that support the automation of software testing will result

in significant cost and time savings for the software industry. Automatic

generation of the test data is essential for the automation of software testing."

(Xu, 2005)

However, one major issue with automated testing is test cases are repeated

without any intelligence. Test Data changes frequently, and sometime results

from the automated tests provides very little or no value to the overall product

quality. Proposed solution to that problem is to utilize Intelligent Software Agents.

"Agents are objects that represent operations; they are effectively closures from

functional programming. Agents can be passed to different software elements,

which can use the object to execute the operation whenever they want. Agents

thus provide a way of separating the definition of a routine from its execution."

(Ostroff, 2005)

Software Agents are autonomous, and can make their own decisions up to

certain extend, based on their Belief, Desire, and Intension (Agent BDI Model).

"Agent technologies facilitate the automated software testing by virtue of their

high-level decomposition, independency, and parallel activation. The informal

16

interpretations of qualitative agent theories are not sufficient to distinguish agent-

based approaches from other approaches in software testing." (Dhavachelvan,

2005)

Statement of the Purpose

Appropriate level of testing not only ensures quality product, but assist in

reducing the maintenance cost. There is no difference of opinion regarding the

importance of testing, however, to achieve the adequate level of testing depends

on project timeline, allocated budget, and resources with appropriate level of skill

set. With new software methodologies, Project Managers allocate sufficient

amount of resources for the testing phase. However, software products are still

error prone with high maintenance cost. There are various testing methodologies,

and depending on the type of software product, different testing strategies can be

used to achieve optimal results. This paper proposes solution by using Intelligent

Software Agents for Regression Testing Framework. It utilizes unit tests and

allows automated testing with extensive reporting capabilities.

Research Problem

Automated testing provided initial results, however over the time; the value of

those results diminishes due to lack of intelligence in the execution. Automated

scripts may not fully accommodate change in environment, i.e. variations in

database schema and business requirements. Proposed Regression Framework

17

- MARTS; uses Intelligent Software Agents for executing unit tests, on JADE

platform (Java Agent Development Framework). Proposed solution only uses

agents for execution of test scenarios, in order to reduce the overhead of

creating every component as agent-based. Therefore proposed solution utilizes

the benefits from the proven distributed applications, along with the flexibility and

autonomous decision making capability of agent-based applications.

Definition of Terms

MARTS: Multi-Agent Regression Testing Suite – The proposed solution, which

uses Software Agents to execute unit test cases.

JADE: Java Agent Development Framework is a software Framework that

implements multi-agent systems through a middle-ware that complies with the

FIPA specifications.

FIPA: Foundation for Intelligent Physical Agents is an IEEE Computer Society

standards organization that promotes agent-based technology and the

interoperability of its standards with other technologies.

BDI: Agent Architecture - Belief, Desire, and Intension. BDI software model

implements the principal aspects of Michael Bratman's theory of human practical

reasoning.

18

Organization of the remaining chapters

Chapter 2 discusses different testing methodologies, and use of Software Agents

for application testing. Chapter 3 discusses Software Agents, brief overview of

JADE platform, and explains in detail the architecture of MARTS – Multi-Agent

Regression Testing Suite. Chapter 4 discusses the issues, challenges, and

trends in software testing, and using agents in regression testing. Chapter 5

discusses conclusion and future work in the field of software testing.

19

CHAPTER II

REVIEW OF RELATED LITERATURE

Software testing has been the field of interest for decades. Significant research

has been conducted, to enhance the over testing process, and especially

Software Testing Automation. With better testing, not only we achieve better

quality product, but also assists in meeting project timelines. As Jones states in

the article on software estimation, “reason software projects run late and exceed

their budgets may be that they have so many bugs they cannot be released to

users.” (Jones, 2002)

There are different approaches, one can adopt for software testing automation

however for the scope of this paper; we will discuss the possible solutions using

intelligent software agents. Software agents have been defined using various

definitions, and as David states that, "although there is no general accepted

definition, an agent can be viewed as an autonomous object that is designed and

implemented to complete certain tasks by cooperating with the environment and

other agents." (Kung, 2005)

Software agents have been used in various projects to automate software

testing. Depending on the type of application, agents can assists in not only

testing the functionality, but to automate the test case generation. As explained in

one research paper, "Agent technologies facilitate the automated software testing

by virtue of their high level decomposition, independency, and parallel activation.

20

Usage of agent based regression testing reduces the complexity involved in

prioritizing the test cases. With the ability of agents to act autonomously,

monitoring code changes and generating test cases for the changed version of

the code can be done dynamically." (Salima, 2007)

Coelho, in his research, used agents to test application in development phase,

an approach also sometime refer as Test Driven Development (TDD). He states

that, "Agent technology has emerged as a prominent technique to address the

design and implementation of these new and complex distributed systems. While

the asynchronous architecture of multi-agent systems (MAS) helps to address

current application requirements, it also brings many threats to software

dependability. [….] there are four complementary means to attain dependability:

fault prevention, fault tolerance, fault removal and fault forecasting. In this work,

we propose a MAS testing approach which aims at removing faults along the

application development." (Coelho, 2006)

Software product goes through an extensive development life cycle, and

depending on the type of software applications, some development phases

require more resources than others. Author Jones; create the comparison chart

for Web Projects, MIS projects, System Projects and Military Projects. Notice,

Military projects, where product quality is extremely important requires even

distribution of efforts at all phases of software development. On other hand, web

projects are usually simpler, and do not usually requires resources at all phases.

21

Figure 2.0: Software Activity Variations – Percentage of Staff Effort by Activity (Jones, 2002)

22

Related Work

Intelligent have been used in the field of software engineering for quite some

time. Agents are used in distributed applications, and in software testing. Since

agents can be enhanced to represent human behavior, they can play very

important roles in the field of software testing. As Coelho states, "an agent is an

autonomous, adaptive, and interactive element that has a mental state. The

mental state of an agent is comprised by: beliefs, goals, plans and actions. Every

agent of the MAS plays at least one role in an organization. One of the attributes

of a role is a number of protocols, which define the way that it can interact with

other roles." (Coelho, 2006)

Regardless of methods of software testing, quality of test cases is very important,

and directly impacts the quality of the software product. "A very important

consideration in program testing is the design and creation of effective test

cases. Testing, however creative and seemingly complete, cannot guarantee the

absence of all errors. Test-case design is so important because complete testing

is almost impossible; a test of any non trivial program must be necessarily

incomplete. The obvious strategy, then, is to try to make tests as complete as

possible. Given constraints on time and cost, the key issue of testing becomes:

What subset of all possible test cases has the highest probability of detecting the

most errors?" (Coelho, 2006). Here author highlights a very important point of

identifying the subset of critical test cases. It is not possible to execute all the test

cases every time new software built is created. Even with test automation, it is

23

very difficult to create the test data to execute the test cases. Once critical test

cases are identified, they can be executed with every software build as part of

built verification. This ensures software quality is maintained, and decreases the

project cost, by reducing the maintenance cost.

Software Agents are frequently used in e-learning systems, as Cabukovski states

that, "different students would have different content view. An idea of agent

based testing subsystem for estimation of the student’s grade level which is the

main measure for dynamic generation of the content view is presented in the

paper. … The agent based testing subsystem was applied in a distance

educational system for learning mathematics and informatics called MATHEIS."

(Cabukovski, 2006)

"MATHEIS is an educational system for learning mathematics and informatics for

pupils and students in primary and secondary schools. ... The system supports

easy XML based import/export between applications, as well as SCORM and

AICC based one-click import/export of the course metadata and content files.

Students use various types of services at MATHEIS: web based material

treasury, e-mail communication, discussion forums, chat, testing services, etc.; It

supports different learning styles, from self-paced material, to scheduled or

synchronous classes offline or online and collaborative and integrated learning."

(Cabukovski, 2006)

24

Current Research

Intelligent Agents are frequently used, especially in e-learning projects. This

allows applications to simulate human behavior, along with artificial intelligence

component. Although main stream industry has not fully adopted the concept of

agents, significant amount of research work has been done in the academic

world. “The concept of an agent has become important in computer science and

has been applied to a number of application domains, such as electronic

commerce and information retrieval.” (Choi, 1999)

There are few reasons why agents are best candidate for software testing. First,

testing has huge component of repetitive task, which is error prone for humans.

Secondly, it requires some sort of decision making component, along with the

repetitive task. Therefore, it is not possible to fully automate software testing

without using Artificial Intelligence. Based on these reasons, intelligent agents

are best fit for software testing. “Automated tests are helpful in performing tests

quickly and precisely, but they don't replace the human tester, who has to run the

tests, check that all is going satisfactorily, and consider what other items need to

be checked to ensure quality. It still takes considerable time to perform all the

selected test cases with an automated test tool. Therefore, we need a test tool

which replaces the role of a tester, one that plans the test, selects test cases,

and performs regression automatically, and also has the properties of an agent,

such as autonomy, social ability, and intelligence.” (Choi, 1999)

25

Figure 2.1: Test Agent System (Choi, 1999)

Figure 2.1 is "composed of three agents, a “User Interface Agent” which

exchanges information with the tester, a “Test Case Selection & Testing Agent,”

which selects test cases, and a “Regression Test Agent,” which performs

regression tests. Also, there are five modules, the “Information Management,” the

26

”Test Case History Management,” the “Test Item Management,” the “Test

Oracle,” and the ”Test Case Suit”.(Choi, 1999)

Choi further explains the system, and states that "the User Interface Agent

interconnects testers and the Test Agent System, allowing testers to exchange

information through this User Interface Agent. The purpose of this agent is to

search for test items from the test target inputted by the tester according to the

test level, and to see the test results. Once the test target program and

information about the function of that test target are input, the User Interface

Agent plans the testing process. The test process includes steps: identifying the

test level according to object-oriented test processes, looking for test items,

selecting test cases for each test item, performing tests, and reporting on the

results. The User Interface Agent identifies the test items according to the test

level. From the “Test History,” the User Interface Agent reasons whether the

input test target needs regression tests or not, based on the rules which will be

described in the next paragraph. The User Interface Agent neither selects test

cases nor performs regression tests. Because of this, the User Interface Agent

recognizes an agent able to select test cases and perform regression tests, and

communicates with that specific agent". (Choi, 1999)

“The Test Case Selection & Testing Agent is able to recognize consistent test

cases without redundancy autonomously through the knowledge base. During

this process, it has to select a test case that can produce greater test efficiency

27

from the numerous test cases made by the automated test tool. The Test Case

Selection & Testing Agent recognizes the agents or modules that can acquire

test items and test case pools, then secures them by corresponding with the

agents or modules.” (Choi, 1999)

Software Agents are also frequently used in web based system. "With the

prevalence of the Internet and the WWW, Web quality assurance becomes

increasingly important. This makes the testing and evolution especially important.

Automatic and intelligent testing is needed so as to improve the testing efficiency.

Intelligent Agent is a self-contained, autonomous software module in the

distributed systems that could perform certain tasks on behalf of its users. These

characters make Agent rather fit for the Internet and WWW platform, and Agent

has made effective progress in Information Retrieval (IR), user personalize

services, and etc." (Jiang, et. al, 2004)

Without artificial intelligence, software testing cannot be fully automated.

"Testers are also needed in the whole process because the tools have no

intelligent and cannot deal with the abnormity. If there are certain of intelligent

entities to assist the test case generation, selection, and execution, the testing

efficiency will be greatly improved. “(Jiang, et. al, 2004)

Figure 2.2 shows another example where mockup agents are used, using JADE

platform. As Coelho explains that "instead of creating a unit testing tool from

28

scratch to exercise the Test Cases defined in our approach, we decided to

extend JUnit framework to support JADE agents’ tests. The reason for that is to

lower the developers’ learning curve providing a simple, and widely used testing

framework architecture. JUnit was already ported to many other languages in

different paradigms (e.g. CppUnit, NUnit, PyUnit, XMLUnit). All these

frameworks, known as xUnit family of tools, have the same basic architecture."

(Coelho, 2006)

Figure 2.2: Implementing our testing approach upon JADE Platform - a low level design view.

(Coelho, 2006)

29

Summary

The purpose of this study was to identify the reasons why software quality is not

achieved, despite of using all latest Software Testing Methodologies and

recommend possible solution for the problem. Testing is important but

unfortunately never done properly. Despite tools and methodology, quality of

software still takes few releases to achieve expected results. Manual testing

cannot be eliminated, and human error/negligence cannot be resolved. Agents

are the best available solution which can represent the human testing,

eliminating human error.

30

CHAPTER III

METHODOLOGY

Software testing is the selective verification of the computer program, based on

the business requirements and pre-defined test cases. These test cases can be

created dynamically, or can be created manually, using the test scenario data. It

is very crucial that proper test cases are created, and executed every time

related code is updated. However, with code changes, it is very hard to predict

how other components are going to respond, until all components are properly

tested. Repeatedly executing test cases requires resources, which impacts the

overall project budget. Automated Regression Testing assist in maintaining the

product quality and keep the budget under control. Initial investment is required

to create the automated testing for the product, but it pays off in maintenance

phase. "Regression testing is the selective retesting of a software system that

has been modified to ensure that any bugs have been fixed and that no other

previously working functions have failed as a result of the reparations and that

newly added features have not created problems with previous versions of the

software." (Salima, 2007)

Software Agents

As defined by Bellifemine, “an agent is essentially a special software component

that has autonomy that provides an interoperable interface to an arbitrary system

and/or behaves like human agent, working for some clients in pursuit of its own

agenda. … Agents may interact with each other indirectly (by acting on the

31

environment) or directly (via communication and negotiation). Agent may decide

to cooperate for mutual benefit or may compete to serve their own interests.”

(Bellifemine, 2006)

Intelligent Software Agents possess certain key characteristics. They can

perform tasks on behalf of the user, and have an ability to communicate with

other agents. Since agents are autonomous, they can operate without direct user

intervention, and can monitor environment and can directly affect surroundings.

“Agent has such primary behavioral attributes as autonomy, cooperation, and

learning. Thus we find it is possible to apply Agent into Web testing so as to

improve the degree of atomization and intelligence of testing. For example, we

need to obtain the accurate and effective user visiting scenes in Web

performance testing and regression testing, and the main aspect of Agent

application is information retrieval, so it is natural to obtain users’ visiting actions

through Agent and determine virtual users’ actions and testing steps by multi-

Agents’ associations. In addition, Agents have the learning ability, and they can

adjust and improve their functions due to users’ feedback, so we can obtain

objective and exact results in Web usability evolution.” (Jiang, et. al)

An agent "is a programmed system that can handle tasks autonomously with

intelligence under some prescribed rules. To benefit from autonomous control

and reduced running costs, system functions are performed automatically. Agent-

oriented considerations are being steadily accepted into the various software

32

design paradigms. Agents may work alone, but most commonly, they cooperate

toward achieving some application goal(s). “(Salima, 2007). Some of the

common properties of intelligent agents are Sociability, Autonomy, and

Intelligence.

JADE Platform

Java Agent Development Framework (JADE) is a distributed middleware system

with a flexible architecture, which supports the development of complete agent-

based applications. It allows easy extensions to available add-on, most popular,

JADE-LEAP (Light Extensible Agent Platform). JADE was initially developed by

Research and Development department of Telecom Italia, however now its open

source community project, code distributed under LGPL license. JADE is fully

compliant with FIPA specifications; Foundations of Intelligent Physical Agents.

FIPA; formed in 1996, a Swiss based organization, produce software standards

specifications for heterogeneous and interacting agents and agent based

systems. As Bellifemine states about JADE framework that,"the goal is to simplify

development while ensuring standard compliance through a comprehensive set

of system services and agents. JADE can then be considered an agent middle-

ware that implements an Agent Platform and a development framework. It deals

with all those aspects that are not peculiar of the agent internals and that are

independent of the applications, such as message transport, encoding and

parsing, or agent life-cycle. (Bellifemine, 1999)

Following figure shows the FIPA reference model of an Agent Platform.

33

Figure 3.1: FIPA reference model of an Agent Platform (Bellifemine, 1999)

Figure 3.2 shows the Software Architecture of JADE Agent Platform

Figure 3.2: Software Architecture of JADE Agent Platform (Bellifemine, 1999)

34

Agent Communication Language (ACL)

Agent Communication Language (ACL), a proposed standard language for agent

communications. It was proposed by Foundation for Intelligent Physical Agents

(FIPA). JADE platform implements the FIPA-ACL for agent communications.

"Agent communication is based on message passing, where agents

communicate by formulating and sending individual messages to each other. The

FIPA ACL specifies a standard message language by setting out the encoding,

semantics and pragmatics of the messages. The standard does not set out a

specific mechanism for the internal transportation of messages. Instead, since

different agents might run on different platforms and use different networking

technologies, FIPA specifies that the messages transported between platforms

should be encoded in a textual form. It is assumed that the agent has some

means of transmitting this textual form. The syntax of the ACL is very close to the

widely used communication language KQML. However, despite syntactic

similarity, there are fundamental differences between KQML and ACL, the most

evident being the existence of a formal semantics for ACL which should eliminate

any ambiguity and confusion from the usage of the language." (Bellifemine,

1999)

Multi-Agent Based Regression Testing Suite (MARTS): An Overview

There are quite a few testing frameworks available which can be adopted, to

perform the regression testing. Most of them are based on distributed computing

concept. The existing frameworks, which utilize the Agent technologies, use

agents very extensively. Most of the components are designs based on the

35

agents, and organizations are hesitant to adopt, due to the fact that not many

organizations have skill set available to understand software agents. In this

paper, we go back to the basic, and attempt to find out why we need agents for

Regression Testing. "Testability can be defined as the property that measures

the ease of testing a piece of code or functionality, or a provision added in

software so that test plans and scripts can be executed systematically." (Yu,

2007)

In regression testing, one important perceptive of using agent would be to allow

agents test the different test scenarios, and make appropriate decisions at

runtime, to simulate testers. This does not mean that we have to create all

services, and entire framework architecture based on agents. We believe,

answer exists in the heterogeneous solution, where we use distributed computing

concept for the entire framework, and the test cases are executed by the test

scenario agents, which simulate the software testers. We will utilized existing unit

test cases, and integrate them into regression framework, so that test scenario

agents can be scheduled for every software release.

MARTS, the proposed framework uses intelligent agents to execute the test

scenarios, which are based on the existing unit test cases. It supports test case

versioning, reporting, logging, and results analysis service, which assists test

administrator to make better decisions, and predicts overall product quality.

Although Test Data Generator is integral part of the solution, it is loosely coupled,

and encouraged to use off-the-shelf third party solutions.

36

MARTS Software Architecture

The first main component is the graphical user interface. Testing administrator

will use this interface to variety of tasks. All user preferences and testing history

will be stored in the framework database. There were two options to store the

framework related data. First was to use xml files, and update them each time

user make changes. Second option, to store all the information in the separate

database, and entire user interface is loaded based on the framework database.

Second option was selected based on few reasons. As requirement for this

framework, user must have access to testing history, scenario agent’s schedules,

common reports, and list of available unit tests along with their metadata, which

includes versioning, parameter lists, expected behavior, and possible actions

based on the results. Since this is only high level design paper, not all

functionality has been discussed in detail; however architecture supports

scalability and flexibility for future enhancements, in order to meet the overall

vision of this framework.

37

Figure 3.3 MARTS Framework Architecture

Framework Controller is the central piece of this framework. Like name suggests,

word controller comes from the concept of Model View Controller (MVC) design

pattern.

Framework Controller. User performs actions using Suite’s user interface, and

framework controller updates user interface based on user’s actions. It stores

user preferences, test results, test scenarios, and other related information in

framework database. Figure 3.1 shows the interaction between controller, user

interface and framework database.

38

Figure 3.4 Framework Controller

User will be able to create test scenario, by selecting available unit tests.

Appropriate unit tests versions will be available for user to select, along with the

brief description. Once test scenario is created, user will have an option to

schedule to test scenario to be executed, or can be executed on demand. When

execution is required, framework will create the test scenario agent to execute

the test scenario. Agent will be able to make its own decisions based on the

results of each test case. At any given time, there can be multiple agents running

testing different test scenarios. Figure 3.5 shows framework interaction with

multiple test agents.

Figure 3.5 Agents Communication with Framework Controller

39

One major problem with test scenarios is it usually passes when executed

individually, however when executed along with other test scenarios, it shows

concurrency and performance issues. Multiple test agents test the application

simultaneously using different test scenarios. Software Agents are based on the

BDI (Belief, Desire, and Intension) software model. As explained on Wikipedia,

"Superficially characterized by the implementation of an agent's beliefs, desires

and intentions, it actually uses these concepts to solve a particular problem in

agent programming. In essence, it provides a mechanism for separating the

activity of selecting a plan (from a plan library) from the execution of currently

active plans. Consequently, BDI agents are able to balance the time spent on

deliberating about plans (choosing what to do) and executing those plans (doing

it). A third activity, creating the plans in the first place (planning), is not within the

scope of the model, and is left to the system designer and programmer."

(Wikipedia)

Figure 3.6 Reporting Component with Framework Controller

Another important functionality of MARTS is reporting component. On demand

reports are created, based on the user defined criteria. All information is stored in

40

the framework database, for future retrieval. This allows user to store the

common reports, and does not have to specify the report criteria again. Reports

can be scheduled, which will execute and user will be notified upon completion.

Here we get two benefits. First, user will receive the information regularly, and

does not have to request for information. For example, in Project Management

software’s, Project managers can execute reports to analyze the overall project

issues, however, most of them either forget to run them, or they don’t know if

certain type of report can be generated. Vision for MARTS is to create the

reports, and information when required. As future extension, agents will have an

ability to initiate the report, to inform user of potential issues that were

encountered during execution. These will be mostly issues that do not fail the test

case; however, it gives enough information regarding possible future issues.

Figure 3.7 Logging Component with Framework Controller

Like all software systems, MARTS logs all agent activities, for troubleshooting

purposes. Logs will be captured in normal text files, however, user will have

access to the logs using user interface.

41

Figure 3.8 Framework Controller with Test Data Generator

For each test scenario, appropriate test data will be required. A stand along Data

Generator service will be integrated with the MARTS, to provide the test data

based on the test scenario. Test Data Generator can either be custom built,

which requires construction for scratch, or can be off-the-shelf test data

generator. Creating test data is a complete topic by it self, therefore details are

left for future enhancements. Framework Controller will create test data for all

new test scenarios, and update data for existing scenarios. Test Data Generator

Service will be integrated with the application database, in order to provide the

appropriate data. Static data causes issues, mostly when database is updated.

Integration of the service with the application database ensures that schema

changes are reflected at execution time of test scenarios.

Figure 3.9 Framework Controller with Test Results Analyzer Service

42

Like Test Data Generator service, Test Results Analyzer will be stand along

service, to analyzes the test results, and provide appropriate feedback. Test

Results analyzer will interact with framework controller, and scenario agents.

Test Scenario Agent. Test Agent will be created by the framework controller at

execution time. For each test scenario, controller will create the agent. The base

structure will be the same, and execution steps will be provided by the

framework. However, for future enhancement, MARTS will have different types of

agents, with different default behavior. User will have an option to select the

agent type from user interface. For example, aggressive type agent will perform

the testing, using more strict rules, and have proactive behavior when interacting

with our agents, whereas defensive type agents will adopt more friendly behavior

when interacting with surrounding agents.

Figure 3.10 Framework Controller with Test Scenario Agent

Once test scenario is completed, agent will update the framework controller,

which will update framework database, and update user accordingly.

43

Figure 3.11 Test Scenario Agent with Test Data Generator

Test Scenario agent will interact with the test Data Generator service to retrieve

the test data for the unit tests. Every test scenario is assigned unique test id. All

unit tests are also assigned unique ids, which represent unit test id along with the

version number. Agent will send the information to test data generator, which

provides the appropriate data for the test agent.

Figure 3.12 Test Scenario Agents with Logging Component

All agents’ activities are logged by logging component, which updates framework

controller.

Figure 3.13 Test Scenario Agent with Unit Test Wrapper

44

Since agents will be using unit tests, it requires certain wrapper, such that an

agent does not have to be specific to test cases. A generic unit test wrapper will

be created to make sure agents can call the wrappers, and can use unit tests,

without getting into specific on JUnit or NUnit.

Figure 3.14 Test Scenario Agent with Test Results Analyzer

Test Scenario agent will frequently interact with the Test Result Analyzer service

to get test results validated. For each test results, a unique id will be created, and

stored in framework database, for future retrieval and tracking purposes.

Reporting component will use that information to create overall progress.

Reporting Component will also be able to generate the quality analysis report, to

analyze the rate of bugs reported by agents, and identify the overall health of the

software product. Since this is regression framework, agents will be scheduled to

run all the times, except for software build deployment and server maintenance.

Unit Test Wrapper. Test Scenario Agents will use the Unit Tests, to execute

the given test scenario. Unit Test Wrappers will be used to make sure generic

agents can be used, for all test scenarios. Test Wrappers can be a service, or

45

web service, depending on the application requirement. For complex

applications, it is better to have stand-alone unit tests wrappers, with no

dependencies to the agents.

Test Data Generator. Test Data Generator service will create test data, which

will be used by the scenario agents. Test Data can be predefined, or can be

create at runtime. This will require more construction time, based on the system’s

requirement; this can be off-the-shelf component, which can be easily integrated

with the MARTS framework. For example, DATA Generator from SQLEdit

provides similar functionality.

Figure 3.15: SQLEdit DTM Data Generator (SqlEdit, 2009)

46

Another third party component which can be potentially used is Turbo Data.

Figure 3.16: TurboData Data Generator (Turbodata, 2009)

In-house data generator will require more resources, whereas third party

components can have some integration issues.

Test Results Analyzer. Test Results Analyzer will receive the input from

agent, and create the analyses report based on pre-defined set of rules. Most of

the system requirements will be validated in this service. Agents will interact with

this, and can make decisions based on the results. For example, if critical test

47

case fails, agent may decide to terminate the test execution. Agent’s decision will

be based on the feedback from Test Results Analyzer.

An Online Shopping Store - Example

For an online shopping store, there can be multiple roles agents can play to test

the system. For example, user creates four test scenarios, first time customer,

existing customer, vendor, and web site administrator. All four agents can start

their test cases simultaneously. Moreover, user can create multiple instances of

same test scenarios, therefore at any given time; we could have n number of

customer agents, shopping online, while few vendor agents can validate their test

cases. Intelligent Agents provides much flexibility, and better simulate the human

testing.

Test Agent - Customer: Existing Customers will already have their accounts

created on the web portal, however for new customers; accounts will have to be

created. For all customers, once there accounts are setup; they will have there

preferences stored, which should customize the web look and feel based on their

choices. Test Agent, would be able to validate the preferences settings, and

continue to verify the available information for data accuracy. Using agent based

testing, all different combinations can be tested, and ensure that services

available on the web portal continue to respond properly, after each software

build. Customer agents will help in analyzing the performance testing, since all

scenario agents can use the web services simultaneously, and simulate the real

environment.

48

Test Agent – Vendor: Vendor agent should continuously monitor the stock

quantities for their respective products, and should be able to bid for the supplies,

as soon as it’s required. Vendor agent would load the vendor information based

on the vendor account information, and simulate the concurrent testing for

application, while customers and vendors are all participating at the same time.

Vendor agent should bid aggressively, and should ensure its goals are met, as

compared to the portals customers. Vendor would also get the runtime

information that is released by the site administrator.

Test Agent - Site Administrator: Site administrator, aside for monitoring the

overall site health and stick piles, should also compare the bids for from different

vendors and approve the lowest quotes, based on the quotes from different

vendor agents. It creates the runtime inventory of items and goods that are not

available on the site, and based on the vendor agent responses, can enhance

the web portal at runtime. For example, lots of customer agents search for the

same electronic item which is not available on the site. Site Administrator agent

would query the vendor agent for the quote. Once response is positive, it will

update the site information, and place orders to the vendor. This will help to

simulate how site administrator will monitor and place orders, and how vendor

will response. In order to perform these testing manually, it will require lots of test

cases, and large amount of human effort to test all possible scenarios. With

agent based-testing, it can run for few days, and log the errors as encountered.

49

CHAPTER IV

ISSUES, CHALLENGES, AND TRENDS

Software Testing has always been the most neglected phase of software

development life cycle. Inadequate resource allocation for software testing phase

tends to be one of the top reasons for high maintenance cost. Business Users for

large Enterprise Applications, typically mainframe users, spend most of the time

understanding new technology, and tends to compare the usability with

mainframe application interface. For example, on mainframe, users perform

specific sequence of steps to perform their task. With newer technology, steps

will definitely change; however, business users tend to accept the change with

the feeling as if IT dictating business. Inexperienced Project Manager and over-

technical IT Senior Management; usually unable to sell the idea appropriately,

results in majority of the user/acceptance testing time, spend in unproductive

meetings. Since technology cannot address these issues, but can help by

automating the testing, and validate the core business requirements.

Agent-based testing can simulate the user test; however, it depends on the

design of the software testing agent. It requires significant amount of time and

effort, to create an intelligent agent, which can make its own decisions, based on

the environment, and interaction with other agent. JADE framework is flexible,

and more important, easy to learn for new developers. More academic projects

tend to use JADE, due to fact that it is open source, and completely written in

JAVA.

50

Although Agent-based systems are more flexible, and scalable, it is quite

complex to develop, and hard to find resources with appropriate skills.

Implementing security model for agent-based system is more complex and error

prone, compared to non agent-based system.

Proposed solution attempts to address common problem of the software industry.

It uses Unit Tests, which are created by developers in construction phase, and

testing scenarios, which are created by tester’s in collaboration with business

users. Since test scenario agents can be scheduled for regular testing, tester’s

gets regular analysis feedback, to ensure software quality is maintained.

51

CHAPTER V

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

In this paper, an agent based regressing testing suite was proposed. It does not

only use agent for the sake of the technology, rather it integrates the benefits of

intelligent agents with the distributed systems. Its very high-level design, and

more detail research will be required to create a product, which can be used for

regression testing mid-size and large enterprise applications. Of all the available

options, Agent-based testing is the better, due to flexibility, autonomous in

decision making, and scalability. Poor quality software is released to customers,

due to excessive schedule pressure, lack of training programs for developers,

lack of code inspections due to time constraints and failure to address defects in

timely manner. With automated testing, regression testing can be performed

regularly, and quality of software is maintained throughout the project.

Proposed solution utilizes the benefits of the agent-based systems, along with

the proven solutions of distributed computing. In proposed solution, only test

scenarios are executed using agents, rather than entire agent-based system.

There were few reasons, why the approach was adopted. Firstly, skilled

developers for agent-based programming are not easily available. Our main goal

was to simulate human testing, along with some level of intelligence and

autonomous decisions making capability. Creating completely agent-based

system for the proposed solution will increase development overhead. Our goal

52

was to achieve simplified solution, using existing technologies, and address real

industry problem, i.e. unacceptable software quality, despite higher budget

allocated for the project.

Suggestions for Further Research

This paper is very high-level design proposal, and requires more detail analysis

of each proposed components. Major work will be required for creating

algorithms for intelligent agents, and agent templates, to allow user to select the

type of agent, to execute the given test scenario. Future implementation of

prototype of the proposed framework – MARTS, and create execute some beta

testing, to create survey results, based on end user’s feedback.

53

REFERENCES

Jones T.C. “Software Cost Estimation in 2002”, CrossTalk June 2002.

Qi, Y., Kung, D., and Wong, E. 2005. An Agent-Based Testing Approach for Web

Applications. In Proceedings of the 29th Annual international Computer

Software and Applications Conference - Volume 02 (July 26 - 28, 2005).

COMPSAC. IEEE Computer Society, Washington, DC, 45-50. DOI=

http://dx.doi.org/10.1109/COMPSAC.2005.42

Xu, B., Xu, L., Jiang, J. 2004. Applying Agent into Web Testing and Evolution.

ISSN 0302-9743 (Print) 1611-3349 (Online) . Volume Volume 3032/2004 ISBN

978-3-540-21988-0 Pages 794-798. DOI=

http://dx.doi.org/10.1007/b97162

Choi, J., Choi, B. 1999. Test Agent System Design Volume: 1, On page(s): 326-

331 vol.1 ISBN: 0-7803-5406-0 INSPEC Accession Number: 6422443

DOI= http://dx.doi.org/10.1109/FUZZY.1999.793259

Liu, L., Zhou, X., Gu, J., Yang, Z. 2003. Agent-based automated compatibility

software test for NLSF. Machine Learning and Cybernetics, 2003

International Conference on Volume 4, Issue , 2-5 Nov. 2003 Page(s):

1986 - 1989 Vol.4 DOI= http://dx.doi.org/10.1109/ICMLC.2003.1259828

Kung, D. 2004. Agent-Based Framework for Testing Web Applications. In

Proceedings of the 28th Annual international Computer Software and

Applications Conference - Workshops and Fast Abstracts - Volume 02

54

(September 28 - 30, 2004). COMPSAC. IEEE Computer Society,

Washington, DC, 174-177.

Dhavachelvan, P. Uma, G.V. 2005. Complexity Measures for Software

Systems: Towards Multi-Agent Based Software Testing. Intelligent

Sensing and Information Processing, 2005. Proceedings of 2005

International Conference on Publication Date: 4-7 Jan. 2005. On page(s):

359- 364 ISBN: 0-7803-8840-2 INSPEC Accession Number: 8703524

DOI= http://dx.doi.org/10.1109/ICISIP.2005.1529476

Xu, D., Li, H., and Lam, C. P. 2005. Using Adaptive Agents to Automatically

Generate Test Scenarios from the UML Activity Diagrams. In Proceedings

of the 12th Asia-Pacific Software Engineering Conference (December 15 -

17, 2005). APSEC. IEEE Computer Society, Washington, DC, 385-392.

DOI= http://dx.doi.org/10.1109/APSEC.2005.110

Cabukovski, V. E. 2006. An Agent-Based Testing Subsystem in an E-Learning

Environment. In Proceedings of the 2006 IEEE/WIC/ACM international

Conference on Web intelligence and intelligent Agent Technology

(December 18 - 22, 2006). WI-IATW. IEEE Computer Society,

Washington, DC, 622-625. DOI= http://dx.doi.org/10.1109/WI-

IATW.2006.34

Zhou, Z. Q., Scholz, B., and Denaro, G. 2007. Automated Software Testing and

Analysis: Techniques, Practices and Tools. In Proceedings of the 40th

Annual Hawaii international Conference on System Sciences (January 03

55

- 06, 2007). HICSS. IEEE Computer Society, Washington, DC, 260. DOI=

http://dx.doi.org/10.1109/HICSS.2007.96

Briand, L.C. A Critical Analysis of Empirical Research in Software Testing.

Empirical Software Engineering and Measurement, 2007. ESEM 2007.

First International Symposium on Publication Date: 20-21 Sept. 2007 On

page(s): 1-8 Location: Madrid, ISSN: 1938-6451 ISBN: 978-0-7695-2886-

1 INSPEC Accession Number: 9875968 DOI=

http://dx.doi.org/10.1109/ESEM.2007.40

Yu, L., Xu, L., Wang, G., Chi, C., Xiao, W., and Su, H. 2007. Testability and Test

Framework for Collaborative Real-Time Editing Tools. In Proceedings of

the Seventh international Conference on Quality Software (October 11 -

12, 2007). QSIC. IEEE Computer Society, Washington, DC, 322-327.

Salima, T. M., Askarunisha, A., and Ramaraj, N. 2007. Enhancing the Efficiency

of Regression Testing through Intelligent Agents. In Proceedings of the

international Conference on Computational intelligence and Multimedia

Applications (ICCIMA 2007) - Volume 01 (December 13 - 15, 2007).

ICCIMA. IEEE Computer Society, Washington, DC, 103-108. DOI=

http://dx.doi.org/10.1109/ICCIMA.2007.182

Ostroff, J., Paige, R., Makalsky, D., Brooke, P. E-Tester: a Contract-Aware and

Agent-Based Unit Testing Framework for Eiffel. Journal of Object

Technology, vol. 4, no. 7, September–October 2005, pp. 97–114, Web

Site: http://www.jot.fm/issues/issues 2005 09/article4

56

Javed, A. Z., Strooper, P. A., and Watson, G. N. 2007. Automated Generation of

Test Cases Using Model-Driven Architecture. In Proceedings of the

Second international Workshop on Automation of Software Test (May 20 -

26, 2007). International Conference on Software Engineering. IEEE

Computer Society, Washington, DC, 3. DOI=

http://dx.doi.org/10.1109/AST.2007.2

Holcombe, M., Coakley, S., Smallwood, R. A General Framework for Agent-

based Modelling of Complex Systems Department of Computer

Science,University of Sheffield North Campus, Broad Lane, Sheffield, S3

7HQ, UK

Coelho, R., Kulesza, U., von Staa, A., and Lucena, C. 2006. Unit testing in multi-

agent systems using mock agents and aspects. In Proceedings of the

2006 international Workshop on Software Engineering For Large-Scale

Multi-Agent Systems (Shanghai, China, May 22 - 23, 2006). SELMAS '06.

ACM, New York, NY, 83-90. DOI=

http://doi.acm.org/10.1145/1138063.1138079

Pugh, W. and Ayewah, N. 2007. Unit testing concurrent software. In Proceedings of the

Twenty-Second IEEE/ACM international Conference on Automated Software

Engineering (Atlanta, Georgia, USA, November 05 - 09, 2007). ASE '07. ACM,

New York, NY, 513-516. DOI=

http://doi.acm.org/10.1145/1321631.1321722

57

Wikipedia. Belief-Desire-Intention software model. Retrieved on March 21, 2009.

Web site: http://en.wikipedia.org/wiki/BDI_software_agent

SQLEdit DTM Data Generator. 2009. Retrieved on March 27, 2009. Web site:

http://www.sqledit.com

TurboData Data Generator. 2009. Retrieved on March 28, 2009. Web site:

http://www.turbodata.ca

Stutzke R.D. “Software Estimating Technology: A Survey”, CrossTalk May 1996.

Conte S.D., Dunsmore H.E. and Shen V.Y. “Software Engineering Metrics and

Models”, Benjamin/Cummings Publishing Company (1986).

Boehm B. “Software Engineering Economics”, Prentice Hall (1981)

Bellifemine, F., Poggi, A., Rimassa, G. 1999. JADE – A FIPA-Compliant Agent

Framework. Proceedings of the Practical Applications of Intelligent

Agents.

Bellifemine, F., Caire, G., Greenwood, D. 2007. Developing Multi-Agent Systems

with JADE. John Wiley & Sons, Ltd. ISBN: 978-0-470-05747-6.