evaluating information systems development methods · pdf fileevaluating information systems...

Z. Bellahsène, D. Patel, and C. Rolland (Eds.): OOIS 2002, LNCS 2425, pp. 311-322, 2002. Springer-Verlag Berlin Heidelberg 2002

Evaluating Information Systems Development Methods:A New Framework

Peter Bielkowicz, Preeti Patel and Thein Than Tun

Dept. Computing, Information Systems and MathematicsLondon Guildhall University

100 Minories, London EC3N 1JY, UK{bielkowi,patelp,thein}@lgu.ac.uk

Abstract. Our detailed investigation into various approaches to evaluateInformation Systems Development methods has shown that numerousattempts to assess these methods only yield inconclusive and question-able results. There are two general trends as to how various criteria forevaluation are organised. There are relatively ad hoc lists of criteria forevaluation, and systematically organised frameworks, which generallyprovide more authoritative assessment results. However, the frame-works investigated are too generic and disproportionate in their empha-ses on certain parts of a method. Our initial motivation was the devel-opment of a framework for assessment of Component-based SoftwareDevelopment methods. However, in response to the shortcomings inexisting approaches to method evaluation, a more generic frameworkthat can be used to evaluate various types of Information Systems De-velopment methods such as object-oriented methods, structured meth-ods etc, is presented in this paper. The proposed framework definesthree major elements of a method, namely, System Models, System De-velopment Process and Software Architecture. This paper discusses thetechnique for evaluation of System Models and due to limitations onlength of the paper, discussions on the evaluation of the other two ele-ments are not included.

1 Introduction

The primary purpose of this paper is to present some interesting aspects of our currentwork on a new approach to evaluation of Information Systems Development methods.The authors are involved in an academic research project to critically evaluate pub-lished Component-based Development methods in order to create a holistic approachto Component-based development. As part of our research, we first set out to establisha theoretical framework for evaluation of Component-based Development methods,e.g. [2]. However, our investigation into several existing approaches to evaluation andcomparison of different kinds of methods reveal that most attempts to evaluate meth-ods do not yield decisive answers.

312 Peter Bielkowicz et al.

Due to these weaknesses in existing approaches, we decided to shift our focus a lit-tle and present certain aspects of our framework that is applicable not only to Compo-nent-based Development methods, but also to other types of Information SystemsDevelopment methods such as object-oriented and structured methods.

Selected approaches and our criticism of each of them are summarised in the fol-lowing section. Section 3 presents our framework that deals with three major elementsof a method, namely, System Models, System Development Process and SoftwareArchitecture. In this paper, only the technique for evaluation of System Models isdiscussed due to shortage of space. Conclusions and references can be found in Sec-tion 4 and 5 respectively.

2 A Survey of Existing Approaches to Method Comparisonand Evaluation

In order to provide some indications of the state of the art, this section presents asummary of selected existing method evaluation approaches we investigated.

2.1 Wieringa�s Approach: A Framework for Surveying Methodsand Techniques

Wieringa [11] suggests an evaluation framework that is based upon the concept ofsystem, in particular, a system that interacts with its environment. These interactionscan be grouped into meaningful units called �functions�. Interactions are �by naturealways communications�. Functions, when ordered in time, become behaviour.

This approach regards functions, communication and behaviour as �system proper-ties�. These properties can be described at various levels of abstraction. System inter-action at the top level can be termed the �mission� of the system. At a more detailedlevel, are functions, and at the bottom level, there are �atomic transactions�. Also,behaviour of the system can be described at various levels and they are shown in the�refinement hierarchy�. Systems are composed of parts, and the composition of thesystem is shown in an aggregation hierarchy. Hierarchies of aggregation and refine-ment are also intrinsic, which can be represented in a magic square. Finally, the map-ping between system interactions and system components is shown in Function De-composition Table.

Based upon these concepts, Wieringa concluded that a method should offer tech-niques for four properties. They are:

1. function specification techniques,2. behaviour specification techniques,3. communication specification techniques, and4. decomposition specification techniques.

Since decomposed system parts are regarded as systems in their own right, the ap-proach states that the techniques for the specifications of function, behaviour and

Evaluating Information Systems Development Methods: A New Framework 313

communication are divided into �external� specifications, i.e. specifications for theentire system, and �internal� specifications, i.e. specifications for system components.

2.1.1 Comments

The framework does explain what kind of models one should expect from a methodand why. However, there are some weaknesses in the framework. For instance there isan overemphasis on system interaction, which is described from three different per-spectives. The framework does not highlight the importance of information that isbeing communicated, and information that needs to be stored inside the system.

Also, when references to �system� are made, it seems that only the �process of thesystem� is implied. This is evident from the assessment of the Entity Relationshipmodel, which is regarded as a system decomposition model. However, the concepts ofcommunication, function and behaviour cannot be applied to entities in the ER modeland the idea that decomposed units of a system should be regarded as �systems in theirown right� appears to be invalid in this sense.

We also believe that decomposition of the system and refinement of system inter-action is mutual, contrary to the suggestion made by the approach that it is possible todecompose a system without detailed knowledge of its interactions and vice versa.

Finally, there does not seem to be much need for communication, which for mostpurposes is similar to function.

2.2 Jayaratna�s Approach: NIMSAD

Jayaratna [7] proposed the Normative Information Model-based Systems Analysis andDesign (NIMSAD) framework, which is a generic framework for the evaluation of�any� method1, including Information Systems Development methods. The frameworksuggests that effective application of a method depends on three elements: the methoditself, the person who applies the method, and the context in which the method is ap-plied.

Evaluation of the three elements, namely, the problem-solving process, the problemsolver, and the problem situation will be carried out at three stages: �before interven-tion in order to minimise our efforts and effectiveness; during intervention because ofthe dynamic nature of the elements; and after intervention so that lessons can belearned about the three elements� [7].

2.2.1 Comments

The NIMSAD framework is rather unique in many regards. First, it highlights the factthat, practically speaking, effective problem-solving not only depends on the methodused, but also on the situation to which the method is applied and the person who usesthe method. The NIMSAD framework also treats the evaluation of a method as a dy-namic activity that is carried out before, during and after the application of themethod. This treatment is also a very original feature of the NIMSAD framework.

1 Jayaratna uses the term methodology. Some authors prefer �method� to �methodology�, and

others use the two interchangeably. We opted for the term method and for the limited pur-pose of the discussion, it can also be read as �methodology�.


The framework appears to be too generic in some areas. For instance, the frame-work assumes that a method is a �problem-solving process�. However, the frameworkdoes not attempt to define specifically what constitutes a method and how to compareelements of a method. The lack of which makes the framework less efficient for thetechnical and theoretical evaluation of very specific kinds of methods, e.g. Compo-nent-based Development methods.

2.3 Other Approaches

Other approaches are also of interest and were therefore investigated. Some are brieflydescribed here.

Avison and Fitzgerald [3] provide a list of criteria which includes philosophy,model, techniques and tools, scope, outputs, practice and product. The criteria how-ever suggest only what to compare in general terms, but not how to carry out the com-parison. How can one compare different philosophies of methods? Also some criteriaare too generic. For example, what exactly needs to be compared in models?

As mentioned in [3], Catchpole, Land, and Bjorn-Andersen, compile extensive listsof criteria (see [3] for references). We believe that comparison criteria should not be anever-ending list of randomly collected questions, but they should rather focus onimportant features of methods that affect the overall quality of the methods.

Hong et al [6] describe the following two-phased approach to compare object-oriented methods. Phase One consists of building meta-process and meta-data modelswhilst Phase Two is concerned with the comparison of analysis and design steps,concepts and techniques. Hong et al made some attempt to quantify the quality attrib-utes, but as Jayaratna [7] pointed out, if one can create a supermethodology, would westill be needing to compare imperfect methods?

2.4 General Findings of the Investigation

We believe that many attempts to compare or evaluate methods using lists of criteriaare unsuccessful because they tend to deploy criteria that are too subjective, ill-defined, and narrowly-focused. For instance, criteria such as �simplicity�, �symbols�,and �reusability� are rather common across many catalogues of criteria. Some of thesecriteria are, of course, important. The clear weakness of criteria lists is that they tendto be rather ad hoc and incomprehensive due to lack of congruent fundamental princi-ples. Therefore, assessment results produced by these criteria are often inconclusive.For example, if a method is not simple, does it necessarily make it a bad method?Some of the quality criteria are also hard to gauge let alone compare them. How doesone confidently measure the simplicity of a method?

A good alternative to such fragmented lists of criteria is assessment frameworks.Frameworks such as [7] and [11] are based upon rather sound logical reasoning, andthey provide some fairly objective and systematic ways to question the quality ofmethods. However, a potential drawback is that since the frameworks can be too ge-neric, there can be areas for subjectivity to creep in. The proposed framework willdraw from the fundamental rules on which these frameworks are built, but will alsosupplement them with specific mechanisms to gauge various quality attributes.


3 The Proposed Framework

As our investigation has shown, there are a number of weaknesses in the existing ap-proaches, which leads to the proposal of our framework. Frameworks such as [7] and[11], and �systems thinking� [10] in general have some bearing on this framework butit also sheds new light on other important areas of method evaluation such as catego-rising systems models, mapping their correlations etc. that are not explored by existingapproaches. Hence the proposed framework does not overturn these approaches, butrather synthesises and enhances them by providing more concrete guidelines.

Elements of Evaluation

At a very generic level, this framework agrees with the NIMSAD framework thatsuccessful evaluation of a method requires assessment of not just the method, but alsothe context in which the method is applied and various personal qualities of the personwho applies the method. This framework also agrees with NIMSAD that the evalua-tion of the three elements should be done at three stages, before intervention, duringintervention and after intervention. Since NIMSAD is designed for the evaluation of awide range of methods, its definition of method is very generic. The framework re-gards a method as �a problem-solving process�. It is very process-focused and for mostInformation Systems Development methods, it is a rather simplistic description. Inmodern methods, such as Component-based Software Development methods, there arespecific issues such as modelling techniques that are too important to be ignored.

Authors such as Song [9] have made various proposals on what constitute amethod. Based on these proposals and our observations of modern methods, the pro-posed framework suggests that a method has three major elements.

Three Major Elements of a Method

This framework proposes that a method has three main elements namely, SystemModels, System Development Process and Software Architecture (see Figure 1).

In fact these three elements are not totally independent of each other. For instance,in a good method, process stages can be expressed in terms of models produced. De-liverables from the process stages can be mapped to the elements defined in the ar-chitecture and (especially design) models should have some correspondence with thesoftware architecture.

Fig. 1. Three major elements of a method

Now we will look at the evaluation of System Models.


3.1 System Models

In order to understand the aspects of the system that need to be modelled, this frame-work has applied the principles of �systems thinking� to Information Systems.

Main Characteristics of Information Systems

Authors such as Waring [10], Wieringa [11] discuss various characteristics of systems.Based on such discussions, we argue that there are, among others, three main charac-teristics of Information Systems: Information, Process/Functionality and Interaction.

Information: Perhaps the most distinguishing feature of Information Systems is thatthey provide information to their users. In order to provide useful information, Infor-mation Systems have to carry out some tasks such as gathering, storing and manipu-lating information, which leads to the second characteristic Process/Functionality.

Process/Functionality: Information Systems perform actions such as storing, retriev-ing, manipulating, presenting data etc. Some of these actions require the system tocommunicate (to obtain or transmit information) with the outside world (users, othersystems etc) through system interactions.

Interaction: Information Systems communicate with the outside world by means ofInteraction. In a broad sense, Interaction means conveying of information from asource to a destination.

These three characteristics are by no means the only characteristics of InformationSystems. Properties such as emergence, control, and others also apply to InformationSystems. We believe that the three characteristics are the most defining characteristicsof Information Systems. Therefore, when modelling Information Systems, paramountattention should be paid to capture these three characteristics.

Two Types of Models

Methods provide various system models to express the method-users� reflection of the�situation of concern�. In terms of coverage, there are two types of models: GlobalModels and Contextual Models. Global models are the descriptions of the system inits entirety from a modelling viewpoint. Contextual models are the descriptions of thecontexts in which system components operate. The following sections examine bothmodels in detail.

3.1.1 Global Models

A global model is a description of a system as a whole from a given perspective. Inorder to produce a model that focuses on one particular characteristic of the system,the modeller needs to take a standpoint and study the system from a particular angle.These angles are called Modelling Viewpoints (see Figure 2).

It is clear from the discussion above that Information Systems have three main as-pects that need to be modelled, and therefore, we are likely to produce three globalmodels that capture these three characteristics. In general, one can practically have aglobal model for each major characteristic of the type of system that one is dealingwith.


Fig. 2. Modelling Viewpoint: A perspective from which a modeller perceives the system

In this framework, these characteristics are represented as three axes as in the fol-lowing IPI (Information, Process/Functionality & Interaction) Matrix, and we can mapglobal models suggested by methods onto the matrix (Figure 3). A line is drawn nearthe Process/Functionality axis to represent a process/functionality model that providesan overall view of the system�s functionality. Similarly, lines are drawn near the In-formation and Interaction axes to represent an information model that provides a rep-resentation of (or information about) interesting things in the real world, and an inter-action model that provides a high-level description of the system�s communicationwith the outside world respectively.

Models from every method may not neatly fit into these three categories. For in-stance, it is possible that a model that depicts the process/functionality view of thesystem also contains elements of system interactions. Such models can be accommo-dated in the matrix by tilting the lines towards the appropriate axis or shading an an-gular area.

Fig. 3. IPI Matrix for Global Models of Information Systems

In Figure 4, ER model is drawn very close to the Information axis because ERmodels only deal with the information aspect of the system. On the other hand, Classmodel has elements of processing: class operations. Therefore, it is aligned towardsthe Process/Functionality axis. DFD is essentially a process/functionality model butalso contains elements of interaction and information, and therefore it is shown as anangular area.


The three main regions of the figure are halved using dotted lines, which representa hypothetical model that contains equal amount of elements from the two neighbour-ing axes. Such models may not have a practical use, since each model should have aclear aim of what it intends to describe. Nevertheless, they provide a useful purposewhen attempting to plot other models appropriately.

Fig. 4. Global Models of various methods

Elements of a Global Model

These global models can be broken down into basic meaningful units. Basic units of aProcess/Functionality Model are processing units, basic units of an Information Modelare logical groups of data items, and basic units of an Interaction Model are events (anevent is an abstraction of some data flows/messages).

Model elements mentioned above are one of the three notable kinds of elements ina global model. They are:

1. Content Elements that substantiate the overall aim of the global model. Forexample functional units in a process/functionality model.

2. Structural Elements that show the relationships/dependencies between thecontent elements, the scope the model, its boundary, its environment etc. and

3. Overlap Elements are elements from other models to show linkages betweenelements of the models which are further explored in contextual models.

Content elements are essential parts of a global model. For instance if the globalmodel aims to describe the system�s functionality, content elements are units of proc-essing. If the global model is to depict the Interaction of the system, content elementsare units of interaction, such as data flows and events. Structural elements provideways to interconnect these content elements, e.g. class relationships. They also areused to add other information such as the boundary, limitations of the model etc. Be-cause of a need for traceability between models, most global models also containelements from other global models called overlap elements. (Alternatively, somemethods use extra models to show such overlaps e.g. Entity Event Model inSSADM [5].)


3.1.2 Contextual Models

Contextual models take each of these basic major units of global models and showhow these units take part in the running of the system. For instance, a contextualmodel can show how a unit of Process/Functionality uses Information units and Inter-action units to perform a specific task. In this regard, the use case realisation model inthe UML can be regarded as a contextual model [1] and [8].

Contextual models are shown as arrow-headed arches. There are two kinds of con-textual model. One is abstract (a direct relationship) and the other detailed. Directrelationships are represented by arches with dash lines, and the detailed relationshipsare represented by arches with thick full lines as shown in Figure 5.

The direct relationship is to represent a model or part of a model that shows an ab-stract correlation between one unit of the global model at the bottom end of the arrowwith one or many units of the global model at the pointed end. The other arrow repre-sents a model that details the abstract relationship. For example, a row in Entity EventMatrix is a contextual relationship showing in abstract terms how an event affects oneor many entities, and Effect Correspondence Diagram details exactly how the effectsare realised.

Fig. 5. Contextual Models show abstract and detailed relationships between elements of globalmodels

Contextual models describe the system view from individual component�s perspec-tive, contrast with global models that portray the entire system from outside. Globalmodels tend to be static, i.e. time-independent. It is largely because of the sheeramount information that needs to be presented in a highly abstract form. Contextualmodels, on the other hand, deal only with a small portion of the system, and hencethey are better suited to show the dynamics of various components of the system.

Contextual models present a good opportunity to check the validity of the globalmodels since time-dependent modelling is good for checking certain assumptionstaken in the global models by unrolling them.

Figure 6 shows complete examples of the IPI Matrices showing SSADM and UMLmodels.

3.1.3 Evaluation of Models

When evaluating a given method, in addition to the assessment of the coverage of theglobal models and how they are strengthened by various contextual models, this


framework also provides mechanisms to measure the rigour of various modellingtechniqes. This paper however, does not deal in detail with the second part of theassessment, which is discussed at length in [4].

Fig. 6. IPI Matrices showing all major UML and SSADM models

As we can see clearly, the UML does not have a global interaction model thatshows interactions with the outside world. There also seems to be too few contextualmodels in the UML. It means that crosschecking between models will be less rigorousin the UML, compared with SSADM which provide numerous contextual models.


What and How to Evaluate in Global and Contextual Models

In order to evaluate models of a given method, we need to complete the IPI matrix bydrawing appropriate lines and arches for global and contextual models provided by themethod. Then from the matrix, one can ascertain the following:

1. Ideally, there should be three global models in an Information Systems Devel-opment method, one for each major characteristic of the system. Each globalmodel should have a clear aim of what it intends to describe and the compo-nents of the models should reflect that.

2. As far as contextual models are concerned, ideally there should be four arches(two direct and two detailed) for each pair of global models. This will guaranteethat models have a good coverage and they are tightly knitted.

Evaluation of Modelling Techniques

Our previous paper [4] deals with a detailed approach to measure and compare tech-niques to specify what we called �data requirements�, which in the context of thisdiscussion means information models. In the paper, when assessing techniques tospecify data requirements, we first identified the desirable quality attributes of datarequirement specifications, and various elements of data requirement specifications.By applying all the quality attributes to each of the elements of a data requirementspecification, we came up with a detailed list of criteria. We then identified the dataspecification techniques and found out the sort of quality that one can achieve in datarequirement specifications using these techniques. There we used 0, 1, and 2 rigourscales. These criteria are recommended for the assessment of techniques used byglobal information models. Similar detailed criteria for other global modelling tech-niques are still under development.

4 Conclusions

The use of random lists of assessment criteria to evaluate methods does not tend toprovide conclusive answers. Frameworks, which are based upon more rigorous logicalfoundations, deploy a more systematic questioning approach. The existing approacheswe investigated are theoretically sound, but there are deficiencies in pragmatics.NIMSAD, for instance, is too generic and does not tell us, say, how to evaluate mod-elling techniques. Wieringa�s work is indeed very interesting but we believe that cer-tain aspects of the framework can be enhanced. In our proposed framework, we haveattempted to show how methods can be evaluated by looking at three major elements.To evaluate models, we provided a new assessment technique that is not founded ontrivia, such as symbols, simplicity etc, but on wider issues such as coverage models,rigour of their techniques etc. The framework proposed the diagrammatic IPI matrix,onto which various models of a method can be mapped, and qualities of the models ata high level can be ascertained from it. Many assessment approaches only concentrateon what we call �global models�, and very little attention is paid to the importance of�contextual models�. These contextual models are equally important because they dealwith very detailed analysis of the system and serve as crosschecks between global


models. Our framework emphasises that without these contextual models, it would bealmost impossible to validate the global models. The framework also deals with theother two elements of a method, System Development Process Model and SoftwareArchitecture, by providing mechanisms for evaluation (which we were not able topresent in this paper). These mechanisms can be used to select an appropriate devel-opment process and architectural model for a given application development. Webelieve that this framework provides a comprehensive and rigorous approach to theevaluation of Information Systems Development methods.

Acknowledgement

We would like to thank our colleague Islam Choudhury for comments on the UML.

References

1. Booch G., Rumbaugh J., and Jacobson I. (1999). The Unified Modeling Lan-guage User Guide, Addison-Wesley.

2. Allen P., and S. Frost. (1998). Component-Based Development for EnterpriseSystems: Applying the SELECT Perspective. Cambridge University Press/SIGSBooks.

3. Avison D. E. and Fitzgerald G. (1995). Information Systems Development:Methodologies, Techniques and Tools (2nd Edition). Berkshire, England:McGraw-Hill.

4. Bielkowicz P. and Tun T.T. (2001) A Comparison and Evaluation of Data Re-quirement Specification Techniques in SSADM and the Unified Process. InProceedings of CAiSE 2001, Interlaken, Switzerland, pp.46-59.

5. Goodland M. and Slater C. (1995) SSADM Version 4: A Practical ApproachMcGraw-Hill, London

6. Hong S., Goor G. van den and Brinkkemper S. (1993). A formal approach tothe comparison of object-oriented analysis and design methodologies. In Pro-ceedings of the 26th Hawaii International Conference on System Sciences,January, pp. 689-698.

7. Jayaratna N. (1994). Understanding and evaluating methodologies: NIMSADA Systemic Framework. Berkshire, England: McGraw-Hill.

8. Rumbaugh J., Jacobson I., and Booch G. (1999) The Unified Modeling Lan-guage Reference Manual, Addison-Wesley, Reading, Mass.

9. Song X. (1995). A Framework for Understanding the Integration of DesignMethodologies. ACM SIGSOFT Software Engineering Notes 20(1) pp. 46-54.

10. Waring A. (1996). Practical Systems Thinking. London, UK: InternationalThompson Business Press

11. Wieringa R. J. (1999). A Survey of Structured and Object-oriented SoftwareSpecification Methods and Techniques. ACM Computing Survey, 30(4), pp459-527.

evaluating information systems development methods · pdf fileevaluating information systems...

Documents