predicting the development effort of multimedia courseware

Information and Software Technology 1994 36 (5) 251-258

Predicting the development effort of multimedia courseware

I M Marshall, W B Samson, P I Dugard Department of Mathematical & Computer Sciences, University of Abertay Dundee, Bell Street, Dundee DD1 1HG, UK

W A Scott Marconi Simulation, The John Sutcliffe Building, Fulmer Way, Donibristle Industrial Park, Nr. Dunfermline, Fife KY11 5JX, UK

A recurring problem faced by multimedia courseware developers is the accurate estimation of development effort. This paper outlines a metrics based model for predicting the development effort of multimedia courseware. A composite model of multimedia courseware development effort is proposed which makes use of a Rayleigh curve and cost drivers. Initial analysis of cost drivers and delivery time are described along with future work to develop a rigorous model for multimedia courseware development.

Keywords: multimedia, courseware, metrics, courseware engineering

Educationalists have used computers to deliver instruction using interactive video and other media for over 10 years ~. From these pioneering days the major constraints on wide- spread use of multimedia was not the hardware but rather the development effort of preparing courseware. Today, the development effort required to produce multimedia courseware remains substantial despite the reduction in media costs and improved functionality of authoring software. A number of authors have reported development to delivery time ratios from 50:1 to 800:1 and beyond 2-s. With development to delivery ratios of this size it is essential to make realistic estimates of the development effort required to produce courseware.

There have been repeated calls for systematic research into the development effort required to prepare courseware 9-'2. In this paper, the potential for research into the development of courseware metrics to support the planning and production of cost effective multimedia systems for educational or training purposes is discussed. Early research has indicated that although multimedia development effort estimating systems exist, these fail one or more of Boehm and Wolverion's criteria for evaluating cost estimation models '3. In this paper initial research into multimedia courseware metrics is described. While the proposed model is targeted at education and training, it could be extended to the development of commercial multimedia for information systems.

The scope of s o f t w a r e me t r i c s

The original motivation for the development of software measures, models and metrics was managerial with the aim of predicting project costs at early phases in the software development life-cycle ~4. There are a number of alternative models for predicting software development effort and cost available to help both software engineers and managers predict the development effort and cost of a project. The study of software metrics attempts to quantify various aspects of software development. Fenton z4 described software metrics as an all-embracing term used to describe a wide range of diverse activities including:

• Cost and effort estimation models and measures. • Productivity measures and models. • Quality control and assurance. • Data collection. • Quality models and measures. • Reliability models. • Performance evaluation and models. • Algorithmic and computational complexity. • Structural and complexity metrics.

A typical example of a cost or effort estimation metric is Boehm's COCOMO or COnstructive COst MOdel 15. This is a composite modeP 6 which was developed using a combination of analytical equations, statistical data fitting and

0950-5849/94/050251-0g © 1994 Butterwonh-Heinemann Ltd 251

Predicting the development effort o f multimedia courseware: I M Marshall et al.

Figure 1 The waterfall model of software life-cycle

expert judgement. COCOMO is one of the best known and widely documented composite models. The basic equation for estimating the software development effort is given by:

Effort = a~(Size) bi m(X)

The Effort is measured in person months while Size repre- sents an estimate of the thousands of delivered source code instructions and m(X) is a composite multiplier which de- pends on 15 main cost driver attributes. The values of ai and bi depend on the mode of development. While there is some criticism of COCOMO concerning the subjectivity and independence of the cost drivers ~7-z°, it still forms the basis for a number of metric based estimating and planning tools.

M u l t i m e d i a courseware engineering models

Conte, Shen and Dunsmore indicate that the development of effort metrics is only possible within a well-defined phased base software development modeP 6. Figure 1 describes the waterfall model ~5 which is one of the commonly accepted models of the software development life-cycle. Each phase is well defined with start and end points.

Comparable models have been proposed for multimedia and computer based learning courseware development 2~-24. Figure 2 describes a proposed waterfall model for multimedia courseware development. The detailed objectives, target audience and the average training hours of the course are defined in the courseware specification phase. Discus- sion with developers indicates that this is the minimum information they require to begin the development process. The overall instructional design phase is based on discussions with subject matter experts, courseware designers and the client. Once agreement has been reached with the client, then detailed design is undertaken for the media to be used in the final course. In this phase, additional experts, such as media specialists, graphics artists, design editors and programmers, may be added to the team 2s.

During the multimedia development phase the graphics, audio, video, sound, simulation and courseware structure is developed. The courseware integration phase brings the various multimedia elements together on one platform and

Figure 2 The waterfall model of multimedia courseware development

rigorously tests them. In the testing phase the courseware is piloted with learners. Problems identified by the evaluation of the courseware during the testing phase are fed back into the model to improve the quality of the final product. Once the product finishes the testing phase it is then published. Normally in the maintenance phase the courseware is only changed or altered when serious problems are detected or the client requires changes in order to update the content.

The development o f mu l t imed ia metrics

Using the waterfall model, it then becomes possible to develop metrics to describe the multimedia courseware processes, products or resources. While it can be argued that the software programming aspects of multimedia courseware could use existing metrics, the instructional aspects, complexity of media development and interfacing do not match the characteristics of normal software development. Software engineering metrics were originally developed using data processing or, in some cases, real- time development systems data and development models. The incorporation of media design, development, and integration into the normal software development life-cycle reduces the likelihood that straight transfer of the software metrics will provide useful information on the various aspects of multimedia processes, products or resources.

A number of authors have investigated informally the factors which affect the development effort of multimedia courseware 2-8 Marshall, Samson and Dugard 26 indicated that most of these models failed the Boehm and Wolverton's criteria for evaluating cost estimation models ~3. There would appear to be an opportunity to establish multimedia courseware metrics to objectively measure aspects of the development process, products or resources.

Multimedia Effort Estimation Method

Boehm's COCOMO j5 was used as the basis for the development of the Multimedia Effort Estimation Method (MEEM). In common with COCOMO the underlying

252 Information and Software Technology 1994 Volume 36 Number 5

Predicting the development effort of multimedia courseware: I M Marshall et al.

vagcaocm ~

J 0Soun0

Figure 3 Rayleigh manpower loading curve

assumption is that staff utilization during development can be modelled to a Rayleigh c u r v e 14.

Figure 3 shows a Rayleigh curve for the overall duration of the project. This can be made up from a number of individual Rayleigh curves which represent the individual phases of the multimedia project development. The proposed formula for development effort is based on a composite model which incorporates a combination of analytical equations, statistical data fitting and expert judgement ~6. Several researchers have criticized the Rayleigh curve assumption for software development 27-29 but it is a reasonable starting point for this model.

Effort = a(Average Training Delivery Hours) b CD(X)

Effort is measured in person hours, and Average Training Delivery Hours is an initial estimate of the number of hours of training required. There is some debate as to the validity of using Average Training Delivery Hours as the basis of any metric based model of courseware development effor t 29. However, discussion with developers indicate that commissioning briefs will normally state target training delivery hours. The value of constants a and b are used to map data on to the proposed model and to convert average student hours into development staff hours. Cost driver CD(X) is dependent on a number of factors which affect the development of multimedia courseware and are based on expert judgement or analysis of the proposed project.

Proposed cost drivers in multimedia development

A number of authors have identified lists of potential cost drivers which may contribute to the overall development effort of multimedia courseware 2-8. Discussion with developers and analysis of existing factors produced the initial lists of proposed cost drivers in Table 1. It is likely that some of these cost drivers are not independent and they will be eliminated from the final effort esimation model.

Table 1. Multimedia cost drivers

Group Cost driver

Course Difficulty (CD)

Interactivity (IN)

Development Environment (DE)

Subject Expertise (SE)

Number of course objectives Level of course objectives Existing course material

Complexity of interface Level of interactivity Type of question feedback Majority question style Graphics requirements Graphics density Animation requirements Animation density Audio requirements Audio density Video requirements Video density Simulation requirements Simulation density

Production environment Instructional design, development

and delivery methodology Size of proposed development team Development team's subject matter

experience Development team's multimedia

experience

Subject matter expert's multimedia experience

Availability of subject matter expert

The proposed cost drivers are grouped together under four headings:

• Course Difficulty (CD). • Interactivity (IN). • Development Environment (DE). • Subject Expertise (SE).

Currently the cost drivers are defined in terms of an ordinal scale which will be validated through the collection of experimental data and statistical analysis. The values of each cost driver range from CD (very low) through to CD (very high). Each set of cost drivers will have its own independent range of values to be determined by data collection and analysis, but for compactness the values are listed under common headings in Tables 2 to 5.

Course Difficulty (CD)

Table 2 describes the cost drivers grouped under the Course Difficulty heading. These factors increase or decrease the effort required to develop multimedia courseware. This includes cost drivers which account for the number and level of objectives as well as the existence of prepared course material.

Number of course objectives. The number of objectives to

Table 2. Course Difficulty cost drivers

CD (very low) CD (low) CD (normal) CD (high) CD (very high)

Number of course objectives Less than 20 21 to 40 41 to 60 61 to 80 Greater than 80

Level of course objectives Concrete concepts Abstract concepts Lower order principles Higher order principles Problem solving

Existing course material Rewrite of existing Rewrite of existing Rewrite of written Rewrite of tutor New course multimedia material CBT material material delivered material

Information and Software Technology 1994 Volume 36 Number 5 253


~ I ~ ~

I ~ Ofdat ~ Or~¢

. . . . . i . ¢'~.,.,m,

• ] I - 1 e-z r,.7~7~ ¢,a,-.,zm,~

Figure 4 Hierarchy of course objectives

be achieved is normally well defined, or can be defined, prior to the development of the courseware. The proposed scale makes an assumption that as the number of objectives increases, the size of the cost driver increases.

Level of course objectives. The highest level of objective to be achieved in the multimedia courseware is assumed to contribute to the development effort 4. Gagne and Briggs '3° classification of objectives is used as the basis of the cost driver scale. The objectives are classified in terms of the hierarchy shown in Figure 4.

The lowest value is given to objectives which involve concrete concepts, which typically describe physical entities. Defined concepts describe objectives which try to

explain ideas such as inflation. Lower order rules are used to describe objectives which involve learning simple rules. Higher order rules are used to describe objectives which involve the application of complex rules. The highest value of cost driver is allocated to problem-solving objectives. At this level the student would be applying previously learned behaviours in new or unusual situations. The level and type of course objectives are also used by a number of other multimedia development effort estimation methods 3-5,7.

Existing course material. Kearsley 6 and Schooley 7 both indicate that the existence of well prepared course material can significantly reduce the amount of effort required to prepare multimedia courseware. Similarly, rewriting existing multimedia material for another platform will require less effort than the creation of completely new material 3-7.

lnteractivity fiN)

Table 3 describes the cost drivers under the Interactivity heading. Primarily this includes contributions to development effort due to the complexity of the interface, or media used in the creation of the courseware.

Complexity of interface. Kearsley 6 identified the complexity of the user interface as being important in deter- mining the amount of development effort required to create multimedia courseware. A simple text-based interface will typically require less effort to develop than a simple graphical interface or windowing environment.

Level of Interactivity. Cohen describes Interactivity in terms

Table 3. Interactivity cost drivers


Complexity of interface Simple text based Complex text based Simple graphical Complex graphical Windowing graphical based based based

Level of activity Linear Simple branching Complex branching Simple adaptive Complex adaptive

Type of question feedback None Right/Wrong Right/Wrong with right Right/Wrong with Right/Wrong with reme- feedback relevant feedback diation and feedback on

each wrong answer

Majority question style True-False Multiple choice Single words Limited free text Other

Graphics requirements None Existing artwork Simple original Complex original Extremely complex artwork artwork artwork

Graphics density Less than 1 per 20 1 per 20 frames 1 per 10 frames 1 per frame More than 1 per frame frames

Animation requirements None Existing animation Simple animation Complex animation Mathematically accurate animation

Animation density Less than 1 per 20 1 per 20 frames 1 per 10 frames ! per frame More than 1 per frame frames

Audio requirements None Existing audio Simple Complex original Extremely complex original audio audio audio

Audio density Less than 1 per 20 1 per 20 frames 1 per 10 frames 1 per frame More than 1 per frame frames

Video requirements None Existing video Simple original linear Complex original linear Complex original video clips video clips interactive video clips

Video density Less than 1 per 20 1 per 20 frames 1 per 10 frames 1 per frame More than 1 per frame frames

Simulation requirements None Existing simulation Simple original Complex original Realistic simulation simulation simulation

Simulation density Less than 1 per 20 1 per 20 frames 1 per l0 frames 1 per frame More than 1 per frame frames



of the richness of the 'dialogue' between the learner and instructional program 3~. Golas 4 states that the level of Inter- activity expected in the final courseware product affected the development effort. A linear sequence with little or no inter- action is given the lowest rating. Complex branching is used to describe programs in which feedback loops and a significant number of preprogrammed sequences are included. Adaptive is used to describe programs which include elements of intelligence and, perhaps, models of the student 32.

Audio requirement and density. The audio density is used to describe the average requirement while the audio requirements cost driver describes the development effort required to produce sound for the courseware. The existence of suitable sound clips considerably reduces the development effort and would result in a low value cost driver. Extremely complex audio clips with multiple voices and background sound effects would require more development time and are allocated a higher cost driver.

Level of question feedback. A number of authors have identified level of feedback as an important factor in predicting development effort 3'5'7. Questions with no feedback require less development effort than questions which provide relevant feedback and remediation on each wrong answer 3t.

Majority question style. Gery 3 found that the complexity of the questioning style used in the multimedia courseware development had a significant effect on the development effort. The proposed range of question style varies from simple true or false through limited free text input to other. The other category is used to describe complex interactions such as the definition of touch click areas.

Graphics requirements and density. These cost drivers attempt to describe the development effort required to produce graphical images for the multimedia courseware. The requirement scale describes the number of graphical images required in the proposed multimedia package. The graphical density is used to describe the average requirement for graphical images in the final multimedia product. Common sense indicates that the existence of suitable artwork considerably reduces the development effort required while the production of extremely complex artwork increases the development effort.

Animation requirements and density. The animation requirements and density cost drivers describe the development effort required to produce animation sequences for multimedia courseware. A lower requirements cost driver is allocated if suitable animation already exists. A higher value of cost driver would be given to the production of extremely complex photo realistic animation. The graphical density is used to describe the average requirement for animation in the courseware.

Video requirements and density. Bergman and Moore 33 indicated that the video requirements and density significantly affected the development effort required to produce multimedia courseware. The requirements scale is used to describe the average complexity of the video required in the proposed multimedia package. The video density is used to describe the average requirement for graphical images in the final multimedia product.

Simulation requirement and density. Schooley 7 classified the production of simulation as a major contributor to the development effort required to produce multimedia courseware. The simulation requirements cost driver attempts to describe the development effort required to produce simulation for the multimedia courseware. Realistic simulation involving accurate movement, sound and response to student actions is more difficult to produce than a simple animation which simulates the movement of the planets around the sun. The density cost driver is used to describe the average requirements for simulation in the final multimedia product.

Development Environment (DE)

The Development Environment heading groups together cost drivers which describe the production environment, instructional design method, and the size and experience of the team developing the courseware. Table 4 describes the cost drivers under this heading.

Production environment. A number of authors have indicated that the type of production environment affects the development effort required to prepare multimedia courseware 3-7. The type of production environment used to produce the courseware is assumed to have an effect on the development effort. Unfortunately, the exact effect is unclear.

Table 4. Development Environment cost driven

CD (very low) CD 00w) CD (normal) CD (high) CD (very high)

Production environment Authoring environment Authoring system Authoring language High level language Low level language

Instructional design, None Informal Formal first Formal second Formal third development and delivery generation generation generation methodology

Size of proposed More than 15 10-15 5-9 2-4 1 development team

Development team's subject Expert knowledge of Good knowledge of Some knowledge of Knowledge of related No knowledge of matter experience the subject the subject the subject subject subject

Development team's Extensive multimedia Some multimedia Extensive computer Some computer aided None multimedia experience development development aided learning learning experience

experience experience experience



Table 5. Subject Expertise cost drivers


Subject matter expert's Extensive multimedia Some multimedia Extensive computer Some computer aided None multimedia experience development development aided learning learning experience

experience experience experience

Unrestricted contact Daily contact Weekly contact Availability of subject matter expert

Monthly contact Restricted contact

Avner 2 found that suitability for the task was more important than the range and complexitiy of facilities available in the authoring system. The current scale ranges from intelligent authoring environment 34 which fully supports instructional design, development and delivery (I D3) of multimedia courseware, to the hand crafting of software using a low level language such as assembler. The scale assumes that the authoring system is suitable for the courseware being developed and the team have experience in using the tool.

Instructional design, development and delivery methodology. It is often stated that using a formal methodology improves the effectiveness of courseware production. However, De Diana and van Schaik 9 indicate ' . . . v e r y few studies are available that demonstrate that the use of specific design and development methods can in fact improve efficiency'. The cost driver assumes that the effect of using a formal instructional design, development and delivery methodology is to reduce the effort required to produce multimedia courseware. One of the goals of this research is to attempt to determine accurately the effect of a ID 3 methodology on the overall development.

Size of proposed team. Avner 2 found conflicting results on the effect of team size on courseware development effort. In a team which is communicating effectively, then the larger the team the more productive it will be, although the relationship is not linear. However, in teams which are not communicating, the larger the team the less productive it is. Similar results have been found in software development; small teams are more productive than larger teams because less time is spent in communication between team members 35'36. The scale assumes that the team is not communicating effectively which results in a drop in productivity as team size increases.

Development team's subject experience. Golas 4 identified the team's experience and knowledge in the subject area as an important factor in reducing development effort. This may be because the team members do not have to spend time learning the terminology used in the subject area and do not have to refer to an external subject matter expert.

Team multimedia development experience. A number of authors 3-7 have indicated that the more experience a team has in developing multimedia, the more productive the team will be in developing other courseware. The scale ranges from no experience--in which case the team will spend time on the current project learning from their mistakes--through to extensive multimedia experience.

Subject Expertise (SE)

The Subject Expertise heading groups together cost drivers which describe the effect of the subject matter expert on couseware production. This includes the subject matter expert's multimedia development experience and availability. Both of these factors were identified as important by courseware developers. Table 5 describes cost drivers grouped together under this heading.

Subject expert's multimedia development experience. A subject matter expert who is able to provide the development team with the information in a sequence and in a form which is suitable for multimedia delivery will reduce the development effort. The scale indicates that an author who has past experience of multimedia development will be able to contribute more effectively to the development than a novice author. Avner 2 found some evidence to support this assumption in his longitudinal studies of authors from their first courseware development.

Availability of subject matter expertise. This scale is based on discussions with multimedia developers who indicated that the availability of the subject matter expert for consultation significantly affects the overall development effort.

In i t ia l study

As indicated at the beginning of the paper, a number of these cost drivers are not independent of each other. For example, the level of Course Objective and the level of Interactivity may be related. Once an adequate data set has been collected, step-wise linear regression or principle component analysis will be used to identify the main cost drivers. In this initial study, data on 14 courseware development projects was available to explore the relationship of the cost drivers to courseware development. This is considered too few for a detailed investigation of the effects of the individual cost drivers, but sufficient to look at the Course Difficulty (CD), Interactivity (IN), Development Experience (DE) and Subject Expertise (SE) headings.

In each group a simple score was calculated by summing the ratings on each cost driver in each heading. This assigns equal weight to all cost drivers in the group, and also assumes that the scale, which is a set of ordered categories, may be considered to be approximated by an interval scale. Once more data is available, it will be possible to improve on both of these assumptions.

The four cost driver group scores, together with the delivery time and development time, are given for the 14 projects in Table 6.

As can be seen from Table 6, 11 of the 14 projects have a delivery time of 1 hour, but the development time for



Table 6. Mult imedia projects

Development Course Inter- Development Subject Delivery time (hours) difficulty activity Environment Expertise time

(CD) (IN) (DE) (SE) (hours)

28 7 22 16 7 0.167 80 8 18 19 7 1.000

100 9 17 17 7 1.000 100 8 25 18 7 1.000 180 9 21 18 7 2.000 180 9 23 15 6 1.000 200 9 18 19 8 1.000 220 8 18 19 8 1.000 250 10 24 16 6 1.000 320 10 26 17 6 1.000 400 10 19 19 7 1.000 435 7 19 18 6 1.000 500 11 37 19 8 1.000 590 9 37 16 6 3.000

these varies by a factor of more than 5. As a first step, these 11 projects were used to see if the variance in the development time could be explained using the four summary cost driver headings. Plots of delivery time against Course Difficulty, Interactivity and Development Environment are shown in Figures 5 to 7.

Figure 5 suggests that there is an approximately linear relationship between development time and Course Diffi- culty with the exception of the project with Course Diffi- culty = 7 which appears to have much too high a development time. The Interactivity and Development Experience

see

u o

1SO.

sO0.

sO ,

0

Figure 5

$

$ *

7 It g

Delivery time against Course Difficulty

Figure 6

ii " $ •

m

0 / I I I 16 21 26 31 31

~ ( 0

Development time against Interactivity

Figure 7

Ua

4 m

|=

0 :4

$

:. ," ,.' ,; DIIv .~.7@7,~ t Elwl ro t1~m (DE)

Development time against Development Environment

Table 7. Actual and predicted values of development time

Actual Predicted Percentage error

80 128.190 -60.2371 100 186.509 -86 .5086 100 97.500 2.5 180 125.129 30.4837 200 247.888 -23 .9440 220 128.190 41.7320 250 275.517 --10.2069 320 306.207 4.3103 400 367.586 8.1034 500 487.284 2.5431

scores for the project are not exceptionally high and it appears that nothing in the data available explains this project. For the other 10, a simple linear regression of development time on Course Difficulty explains nearly three-quarters of the variance. Adding Interactivity to the regression barely improves it: this is partly because the correlation between Course Difficulty and Interactivity is high at 0.65. However, adding Development Environment to the regression explains 85 % of the variance. The actual and predicted values of the development time are given in Table 7.

Addition of Subject Expertise to the regression did not improve the results. This is because there is little variation in the Subject Expertise scores for these projects (all lie between 6 and 8).

Discuss ion o f results

Using the grouped headings of the cost drivers enables 85 % of the variation in development time to be predicted. How- ever, the limited number of projects in the data set could be unrepresentative of multimedia development projects. Senbetta 8 asked 21 experienced developers of courseware to estimate the development effort for courseware based on requirements documentation. Table 8 shows the maximum and minimum and mean estimated development time for projects with delivery times between 1 and 20 hours. In common with our data, the development time estimates vary by a factor of about 5 for each delivery time.

Only three projects in the data set have delivery times other than 1 hour. Omitting the outlier with development time 435 and using the other 13 projects, 80% of the variance in development time can be accounted for by a regression on Course Difficulty, Interactivity and Delivery Time. The addition of Development Experience does not improve it.

C o n c l u s i o n and future work

The preliminary results are promising and suggest that,

Table 8. Expert estimation of courseware development time

Estimated delivery time

Delivery time n Minimum Mean (hours) (hours) (hours)

Maximum (hours)

1 21 90 240 365 2 21 150 380 700 5 21 250 800 1625

10 21 500 1400 2550 15 21 750 1950 3750 20 21 1000 2400 5000



even with the unweighted scoring of cost drivers used here, useful predictions can be achieved. A large and rich data set would allow more detailed investigations which would almost certainly lead to improved results. It is hoped that with a larger data set containing more projects with long delivery times, it may be possible to predict development time using Course Difficulty, Interactivity and Develop- ment Experience along with delivery time. Similarly, with a greater range of values for Subject Expertise it may be possible to identify its contribution to the prediction of development effort.

Given enough data, it should be possible to produce a software tool version of MEEM. The current model is, of course, simply a framework and cannot yet be used for estimation. It requires to be calibrated to determine the appropriate:

• coefficients • cost driver values

Boehm 15 used data from 63 projects to calibrate his original COCOMO model, and it is aniticipated that data from a similarly large number of multimedia projects will be needed. Work is currently underway to collect and analyse such data from multimedia projects. In addition, other potential cost drivers are being identified with the assistance of multimedia developers and from the litera- ture 3-7. Using data from a representative sample of projects, statistical analysis will help to show which cost drivers are the most significant.

The authors' experience with step-wise linear regression techniques suggest that in any development environment only three or four of the cost drivers have a significant bearing on the effort prediction. Robust statistics 36 have produced promising results in predicting significant cost drivers. Samson, Ellison and Dugard 37 have produced promising results through the use of neural nets to 'learn' how to predict effort from cost driver values. Once these cost drivers have been determined, it may be possible to further simplify data collection.

References

1 Stewart, A M and Bryce, C F A 'Multimedia multipurpose: is the quality of the learning experience being well served by the use of educational media?' in Percival, F and Ellington, H (eds) Distance learning and education Kogan Page (1981)

2 Avner, R A 'Is there an ideal size for courseware production teams?' Proc. 30th Association for the Development of Computer Based Instructional Systems (ADCIS), Philadelphia (1988)

3 Gery, G Making CBT happen: prescriptions for successful implemen- tation of computer-based training in your organisation Weingarten Publications, Boston, MA (1987)

4 Golas, K C 'Estimating time to develop interactive courseware in the 1990s' Proc. lmerservices/Industry Training and Education Conf. (1993)

5 Jay, J, Berustein, K and Gunderson, S 'Estimating computer-based training development times' AR/ Technical Report 765, Research Institute for Bebavioural and Social Sciences, USA (1987)

6 Kearsley, K 'The CBT advisor: an expert system program for making decisions abuot CBT' Performance and Instruction Vol 24 (1985) pp 15-17

7 Schooley, R E 'Computer-based training (CBT), cost estimating

algorithm for courseware (CEAC)' Proc. lnterservice Industry Training Systems Conf. (1988)

8 Sanbetta, G 'CBT time and cost estimation: What do the experts say?' Proc. lOth Annual CBT Conf, and Exposition (1982)

9 De Diana, I and van Scbaik, P 'Courseware engineering outlined: an overview of some research issues' ETH Vo130 No 3 (1993) pp 191-21 I

10 Chen, J W and Chen, M 'Towards the design of an intelligent courseware production system using software engineering and instructional design principles' J. Educational Technology Systems Vol 19 No 1 (1990) pp 41-52

11 Costello, G 'Developing computer-based instruction: the systems design approach' Training Officer Vol 28 No 2 (1992) pp 48-50

12 Jones, M K, Li, Z and Merrill, D M 'Rapid prototyping in automated instructional design' Educational, Training Research and Development Vol 40 No 4 (1993) pp 95-100

13 Boehm, B W and Wolverton, R W 'Software cost modelling: some lesson learned' J. Systems and Software Vol 1 (1980) pp 195-201

14 Fenton, N Software metrics: a rigorous approach Chapman-Hall (1991) 15 Boehm, B W Software engineering economics Prentice-Hall (1981) 16 Conte, S D, Shen, V Y and Dunsmore, H E Software engineering

metrics and models Benjamin/Cummings (1986) 17 Subrahmania, G H and Brelawski, S A 'A case for dimensionality

reduction in software development effort estimation' Internal Report TR-89-02 Department of Computer Science, Temple University, Philadelphia, USA (1989)

18 Kitchenham, B A 'Software project development cost estimation' JSS Vol 5 (1985)

19 Kemerer, C F 'An empirical validation of software cost estimation models' CACM Vol 30 No 5 (1987)

20 Cuelenaere, A M E, van Genuchten, M J I and Heemstra, H J 'Calibrating a software estimation model: why and how' J. Inf. and Soft. Technol. Voi 29 No 10 (1987)

21 De Diana, I and Collis, B 'Adaptable educational courseware: an antidote to several portability problems' J. Research on Computing in Education Voi 23 No 2 (1990) pp 225-241

22 Friedler, Y and Shabo, A 'An approach to cost-effective courseware development' British J. Educational Technology Vol 22 No 2 (1991) pp 129-138

23 Phillips, W A 'Individual author prototyping desktop development of courseware' Computers in Education Vol 14 No 1 (1990) pp 9-15

24 Maher, J and Iugrahem, A 'Software engineering and ISD: similarities, complementaries and lessons to share' Proc. Ann. Meeting of Assoc. Educan'onal Communications and Technology Dallas, Texas USA (1989)

25 Cohen, V B 'Interactive features in the design of videodisc materials' Educational Technology Vol 24 No 1 (1984) pp 16-20

26 Marshall, I M, Samson, W B and Dugard, P I 'Estimating multimedia courseware development effort' Technical Report Dundee Institute of Technology, UK (1994)

27 Parr, F N 'An alternative to the Rayleigh curve model for software development effort' 1EEE Trans. for Soft. Eng. Vol SE-6 No 3 (1980)

28 Jeffery, D R 'Time sensitive models in commercial MIS environment' IEEE Trans. Soft. Eng. Vol SE-13 No 7 (1987)

29 Fairweather, P and O'Neal, A 'The impact of advanced authoring systems on CAI productivity' J. Computer-Based Instruction Vol 11 (1984) pp 90-94

30 Gagne, R M and Briggs, L J Principles of instructional design (2nd edn) Holt, Rinehart & Winston (1979)

3l Cohen, V B 'A re-examination of feedback in computer based instruction: implications for instructional design' Educational Technology Vol 25 No 1 (1985) pp 33-37

32 Spohrer, J C 'Integrating multimedia and AI for training--examples and issues' Proc. IEEE Int. Conf. Systems, Man and Cybernetics, Los Angeles, USA (1990) pp 663-664

33 Bergman, R E and Moore, T V 'Managing interactive video/ multimedia projects' Educational Technology (1990)

34 Tennyson, R D and Elmore, R L (1993). 'Integrated courseware engineering system' Proc. NATA AS11993 - Automating Instructional Design, Development and Delivery Grimstad, Norway (In press)

35 Brooks, F P The mythical man-month Addison-Wesley (1975) 36 Kitchenham, B and Mellor, P 'Data collection and analysis' in Fenton,

N E (ed) Software metrics: a rigorous approach Chapman & Hall (1991) lap 89-110

37 Samson, W B, Ellison, D G and Dugard, P I 'Software cost estimation using an Albus Perceptor (CMAC)' Proc. 8th Int. COCOMO Estimating Meeting, SEI, Pittsburgh, USA (1993)


predicting the development effort of multimedia courseware

Documents