maintaining your quality management the answers software ... · 50 software quality management into...

Maintaining your quality management

system - what are the benefits?

S.D. Walker

ANSWERS Software Service, AEA Technology

Centre, Winfrith, Dorchester, Dorset, DT2 SDH,

UK

ABSTRACT

The ANSWERS Software Service of AEA Technology hasbeen certificated under the United Kingdom's Departmentof Trade and Industry TickIT scheme since September1992. The history of ANSWERS was described in a paperpresented at SQM-93 [1]. The need for softwaredevelopers to have ISO-9001 Certification is becomingsimilar to the requirement of a car driver to hold adriving licence - it is essential to have one to get onthe road ! This paper discusses our experiences inmaintaining a Quality Management System (QMS),particularly in assessing its effectiveness andimproving it so as to provide tangible benefits both tothe QMS users and to ANSWERS customers.

The ANSWERS QMS was established in 1981 and like mostearly quality initiatives, was something of an 'add on'to the day-to-day business operations of theorganisation. Through regular re-assessment of workingpractices, review and improvement of the procedureswhich form part of the QMS and increased staffparticipation, the QMS is now central to all workingoperations. The paper discusses the problems of QMSreview and of obtaining commitment to 'quality' from allstaff; it also examines the pressures and real benefitsof regular surveillance audits, as well as additionalassessments from overseas customers who do not recogniseISO-9001. We now view a surveillance audit as a valuablepolicing exercise which provides us with an element ofconsultancy in running a QMS. We look at the evolutionof our QMS particularly with respect to how our views on

Transactions on Information and Communications Technologies vol 8, © 1994 WIT Press, www.witpress.com, ISSN 1743-3517

48 Software Quality Management

its implementation have changed during the lifetime ofour TickIT Certification.

The paper concludes by examining the net beneficialeffect which an accepted QMS has had within ourorganisation, and by identifying the real benefits tocustomers of maintaining and improving such a system. AnANSWERS product developed during the lifetime of thecertificated QMS is used to identify the benefits whichhave been gained.

INTRODUCTION

In many software engineering areas, the developmentof a Quality Management System (QMS) commenced as thedefinition of procedures covering aspects of the workwhich required specific documentation due to recognitionof a product's significance in terms of its complexity,the consequences of its failure, the risk of itfunctioning incorrectly and so on. The ANSWERS SoftwareService started from just such a position.in 1984 andthe evolution of the QMS over a number of yearsculminated in the gaining of a TickIT Certificate inSeptember 1992. The history of the ANSWERS SoftwareService have already been described in detail byMaddison, Walker and Miller [1]. In summary, ANSWERSprovides validated physics applications codes to thenuclear and oil industries on a world-wide basis. Thesecodes are of vital importance to the customers who usethem, in some cases because they are applied in areaswhere there are safety-critical implications, and inother cases because of the cost-implications ofincorrect predictions. Some of these codes have theirorigins in the earliest days of mainframe computers, anddue to the substantial investment both in existing codeand expertise, the development of these programscontinues as a third generation language activity,normally FORTRAN-based.

This paper examines the continuing evolution of theANSWERS Quality Management System specifically withregard to the need to increase the effectiveness of thesystem both from the customer and user viewpoint. Therequirement to actively maintain the QMS within theTickIT Certification framework and to gain commitment to'quality' from all staff is discussed. First, however, abrief overview of why we think ISO 9001 Certification isdesirable is presented.


Managing Quality Systems 49

ISO 9001 CERTIFICATION - WHY IS IT DESIRABLE ?

ISO 9001 Certification under the TickIT umbrella isgaining in popularity all the time - but why is itdesirable ? In some senses, it is becoming similar tothe requirement for a car driver to hold a driver'slicence - it is essential to have one to get on the roadsince many organisations require certification as a pre-requisite for tendering. Other reasons are alsoapparent. Certification gives an independent check ofconformance to established international standards; ithelps to provide a consistent standard of quality acrossthe software industry; it should indicate that a QMS iseffective; it provides recognition and possiblycompetitive advantage; it can act as a motivating forcefor staff. This last point is significant and there isemphasis on the word can in this context, since withoutcareful management, the audit cycle necessitated bycertification can also act as a de-motivator.

Certification under a recognised scheme permits thepurchaser to trust the supplier without "the need toperform a second party audit.

EVOLUTION OF THE QUALITY MANAGEMENT SYSTEM

The ANSWERS QMS started life as the formulation of amethod of controlling large suites of software in anorganised manner - this is what we now call softwareconfiguration management. The basic requirement was tobe able to match a specific software build against agiven functional specification and set of accompanyingdocuments. There was nothing particularly novel in thisapproach, but in the late 1970s and early 1980s, thiscontrolled approach had not been applied to softwaredevelopment in the nuclear industry. The emphasis hadinstead been placed on validation by experimentalcomparison.

The system was gradually extended by documentingspecific technical and business processes; this resultedin a set of procedural documents covering thedevelopment of software using a standard life-cycle,documentation during software project lifetime,software replication and issue to customers, and errorreporting and change control functions.

This in itself did not form a QMS, but it did providea solid foundation on which to build. When integrated



into the management procedures for the AEA TechnologyReactor Services business, a complete Documented QualitySystem (DQS) was formed.

However, in spite of its relatively slow evolution,the system did not, as a matter of course, become partof the culture of the organisation. In other words, theclassic TQM problem of winning the 'hearts and minds'battle with the users of the QMS still remained. Severalreasons may be offered for this:

• the QMS was developed by a limited subset ofsoftware practitioners and 'imposed' on others.

• the perception that a QMS limits the creativity ofthe individual and also the output rate.

the perception that too much paperwork is involvedin the implementation of the QMS.

• the perception that a QMS limits flexibility.

• the perception that it is not necessary - thesystem worked before 'QA' was introduced.

the perception that the cost is too high

The question which naturally follows, therefore, ishow to increase the effectiveness of the QMS in order toinvolve all practitioners in its development and inorder to motivate users in its application. Beforelooking at this, however, it is worth considering thespecial factors which affect our developmentenvironment.

WHAT IS DIFFERENT ABOUT THE ANSWERS SOFTWARESERVICE ?

There are various factors which combine to make thesoftware development environment, within which ANSWERSfunctions, an unusual one :

• ANSWERS is a small part of a large scientific andengineering services organisation with its ownquality management structure within which ANSWERShas to operate.



• the codes are large, typically many tens ofthousands of lines, and yet the development teamsare small.

« the technical documentation is complex and oftensubstantial, and its production thus tends to lagbehind the production of the code itself.

« prototyping is frequently necessary to try out newsolutions; this can lead to the temptation ofavoiding a formal re-work.

• one man projects are not unusual, and thisfrequently leads to the problem of a lack ofdissipated knowledge and consequent difficultieswith independent review.

• the problem of incorporating the 'guru' into thesystem is a common one.

" the software has significant safety implications(although within the strict definition it is notsafety-critical)

* with much of the software being statisticallybased, execution times are long and thereforetesting is time-consuming, and analyticalverification/validation is difficult to achieve.

• the technical complexity makes the work hard toaudit, both for internal and external (TicklT)auditors. This can lead to the perception thataudit only examines the management aspects of thework and does not lead to any improvement intechnical quality.

• the number of customers for a given product ismeasured in tens not thousands.

The impact of these factors is addressed in thefollowing sections.



INCREASING THE EFFECTIVENESS OF OUR QMS

To effectively apply a documented quality system(DQS), it is necessary to have an understanding of, anda commitment to, quality from all members of anorganisation. The application of the DQS, involvingthorough development and quality planning, regularreview and feedback, and committed staff, gives aworking and effective quality management system. Inorder to create and maintain such a system there areseveral questions which have to be answered:

• how is the QMS reviewed effectively ?

• how is commitment to quality gained ?

• how does one avoid the QMS becoming stale ?

Review of the QMS

Formal review of a quality management system is oftentime-tabled as an event held at fixed intervals,frequently on an annual basis. We have found that thistends to over-formalise the review and can lead to a'going through the motions' problem. It is far moreimportant to try and build in continuous reviewthroughout the year. As an example, review may take theform of re-assessment of individual procedures, workinstructions and day-to-day working practices. This typeof assessment can be carried out on an on-going basis;specific items from a quality or procedures manual canbe passed to an individual for review and comment at anytime. The results can be assimilated and new issues madeon a regular basis.

As part of the review process, we canvass staffopinion both verbally and by seeking a written responseto, for example, a questionnaire.

We have found that regular meetings of seniormanagement and the QA team, to review outstandingissues, examine the findings of internal and externalaudits and to generally discuss quality problems, areeffective and beneficial.

To date, review of the QMS has revealed that we havedevoted excessive time to the production of formaldocumentation and placed an over-emphasis on rigidlyadhering to a given software life-cycle. Experience has



shown that it is more valuable to devote effort to theareas of requirements and design specification andtesting, rather than to formal programmer documentation;in most instances thorough working notes are morevaluable than a more superficial, yet polished,document. It is worth remembering that lack ofmaintenance documentation may hinder future development,but lack of testing is more likely to impact directly onthe user.

The review process is a feedback loop which is usedto improve the system, to make it more usable and moreefficient. It should be made clear that we attempt to dowhat is necessary to have a functioning and effectiveQMS - not what is necessary to address each individualclause of ISO 9001 per se. On-going review, involving asmany staff as possible, is vital to avoid the QMSbecoming stale.

Staff Commitment

In order to gain commitment to quality and the QMS,staff must be, and must feel, involved in the evolutionof the system. It is fair to say that this an area wherewe went wrong, almost inevitably, at the outset. This isbecause the introduction of a QMS has to be driven bysomeone and thus may always appear to somebody else asan imposition. Whilst one can endeavour to involve asmany staff as possible in the formation of the DQS, itis very difficult to include everyone.

Consequently, the review mechanisms discussed abovemust be used subsequently to draw in as many QMS usersas possible; this expands the ownership of the QMS. Itis vital that the QMS is not viewed as somethingestablished by, owned by and for the benefit of theQuality Manager.

The implementation of internal audits, described inthe section below, has been a key factor in gainingcommitment from a wider range of ANSWERS staff.

Internal Audits

In order to overcome difficulties in providingeffective internal audit, and the problems associatedwith peer review on one-man projects, we have introduceda system of Product Quality Audits (PQAs) with which we



address problems which affect the technical quality ofour products. At a higher level, we are still subjectedto additional internal audits which examine our QMS andits application. The PQAs, however, are briefexaminations of specific areas of work carried out bytwo auditors, one with a QA background (ideally withsome auditor training) and the other with a soundtechnical background in the area under scrutiny (the twoare not of course mutually exclusive ! ) . PQAs are usedto examine the workings of a project below the ProjectManager level whereas it is fair to say that ourstandard company internal audits tend to concentrate at,and above, Project Manager level. In many cases theynever get down to the technical issues

It is, of course, easier to examine the quality andmanagement aspects of a project than the technical ones.However, by assessing the completeness ofspecifications, designs, test plans and test records,and by checking that review processes have beenfollowed, confidence may be built up.

One of the principal benefits that ANSWERS has seenfrom the use of this type of internal audit is theincreasing involvement of a greater number of staff inthe audit process. This has helped to remove theperception that audit is a confrontational process wherethe auditors are trying to 'catch out' the auditees; ithas also lead to a greater cross-fertilisation of ideas.As the audit programme continues, more staff areinvolved on 'both sides of the fence. There is also anelement of peer pressure which works positively toincrease the commitment to quality.

Third Party Surveillance Audits

We undergo regular surveillance visits from BureauVeritas Quality International (BVQI) as part of themaintenance of our TickIT Certification. Eighteen monthshas elapsed since the original audit and therefore thesurveillance audit cycle is well established. We viewthese audits as highly beneficial, since they tend toconcentrate on specific areas allowing us to examine indetail the way in which our QMS functions. Inevitablysome measure of consultancy arises out of thesurveillance, although that is not the purpose of thevisit. There is of course pressure associated with eachaudit, but this acts to focus attention on ensuring that



key documents are kept up-to-date and that schedules areregularly reviewed.

Some of the useful improvements which have come aboutas a result of surveillance are in the areas of producttesting and product release; these are discussed below.

Second Party Assessment

ANSWERS has recently been subjected to second partyaudit both from UK customers and also from the US.Within the US nuclear industry, neither ISO 9000 nor itsTicklT implementation are recognised, and the NQA and10CFR series of standards hold sway. These second partyassessments, whilst giving an alternative perspective onquality requirements, can prove contentious due to thediffering standards applied in the US and the UK. Wehope that the introduction of the TicklT schemeequivalent will take off rapidly in the US in order tohelp overcome such difficulties.

The second party audit against US standards hashelped to improve our system for error reporting tousers, necessitating a higher specification andusefulness to users than is required by TicklT.

Focus on Testing Procedures

Much of the initial effort in establishing workinglevel software development procedures was devoted to theareas of documentation and change control. As discussedabove, following review of the system, the emphasis hasbeen changed to try an focus more effort on producttesting. One of the difficulties with the testing of theANSWERS software products, is the range of applicationof the codes and the mathematical complexity of theproblems being solved. Analytical solutions to all butthe simplest test cases are virtually impossible. Thismeans that validation against experiment, against knownphysical situations and against the results of othercomputer programs is vital.

We have recently introduced controlled field testingto allow users to test products on their own systems.This is not a simple undertaking since the trial use ofa new program version will impose additional costs onthe organisation performing the field test. This burdenmay, of course, be offset by the availability of newoptions and other improvements in the code, but within



the nuclear industry, the results produced by a so-called 'beta release' may not be acceptable for officialuse.

Recent field test results have greatly improved theusers' confidence in the completed product versions andhave allowed a variety of anomalies to be ironed outprior to formal issue.

Product Release Management Procedures

As discussed above, surveillance audit hashighlighted the need for various procedural improvementsand one of these has been in the area of our productrelease procedures. We have increasingly formalised therelationship between the development team and therelease management team in order to introduce a greatermeasure of independence into the review and testing ofthe product prior to its issue.

A pre-release audit has been introduced which againuses the concept of a QA and a technical auditor toreview the product package (i.e. the code and itsassociated documentation) prior to release. The job ofthe technical auditor is to review the functionalcapability of the code, to ensure that any new featuresare adequately specified and documented, to check thattest coverage is sufficient and to confirm that the Userdocumentation is comprehensive and clear.

The introduction of these procedures adds value atthe pre-release stage by increasing the checking of codeand documents prior to issue, and by reviewing theadequacy of testing before release. In the long term, itis our intention to increase the involvement of therelease management team during the earlier stages of thesoftware life-cycle to assist with reviewing and testingof documents and code.

'Harnessing the Guru'

The 'guru' can best be described as the acknowledgedtechnical expert in a given field who wishes to work ina free-thinking and unrestrained environment in which heis at liberty to experiment with new ideas as and whenhe thinks fit. It is virtually impossible to give the'guru' the flexibility required without compromising thequality standards which have been established. However,in an advanced technological area, the value of such aperson cannot be underestimated. We have to provide



additional support to such experts to ensure thatprototyped ideas are developed into solutions which meetthe requirements of the system as a whole. This maysound inefficient, but the cost to us is outweighed bythe productivity of the 'guru' and hence we are able tojustify this solution. It should be stressed that suchindividuals are rarities and within an organisation ofsome forty people, only a few qualify.

ASSESSMENT OF GAINS AND LOSSES

In order to assess the benefits of working under anestablished QMS, we take as an example, the MONKcomputer code [2] which is widely used as a nuclearcriticality assessment tool. This program has a historydating back to original development in the 1960's, andhas been the responsibility of the ANSWERS SoftwareService since 1987. MONK is used in an environment wheresafety is critical, and the code is often used tosupport safety cases submitted to the NuclearInstallations Inspectorate and the Department ofTransport. It has also been approved for use incriticality assessments by the US Nuclear RegulatoryCommission.

Over the past three years, the MONK program has beenunder development by a team of three experts. The code,which comprises some fifty thousand lines of FORTRAN,has been substantially modified. Although the basicfunctionality remains unchanged, there are few areaswhich have been left untouched. It has been ourintention to upgrade a product which, due to the age ofmost of the coding was in danger of becomingunmaintainable, into a code which can be supported wellinto the future. A further aim was to pool resourceswith other products in the ANSWERS range to gaincommonality in various key functional areas, such asgeometry processing, in order to become more efficient,particularly in code maintenance. From a user'sviewpoint, it has been essential to ensure that the newversion has sufficient new options to make itsapplication desirable and to ensure that backcompatibility is retained.

From an internal point of view, we would naturallyexpect to see a number of benefits from working under anestablished QMS, however, inevitably there are alsodrawbacks.



The Benefits - expectation and reality

• The QMS provides an environment within which itemsof work are managed more efficiently. We work inan environment where we need to be ever moreresponsive to our customer's needs; the AEA hasevolved into a fully commercial customer-drivenorganisation from its government laboratoryorigins. All items, from major deliverablesthrough review actions to minor ideas forimprovement, are logged in the system andmonitored. The QMS provides a framework withinwhich we are not allowed to forget tasks andtargets!

• The use of the QMS gives us an expectation ofimproved productivity due to more efficientworking practice and greater project control. Inreality, the increased requirement fordocumentation and formality in certain areasimposes an overhead which counterbalances some ofthe gains. However, experience with MONK showsthat this is a short term effect for severalreasons. Firstly, practitioners become familiarwith the QMS and as they climb the learning curvethey become more efficient. This ascent has takenat least two years to achieve. Measurement ofefficiency is difficult, but in general a moreprofessional attitude to software development isnow displayed by all practitioners. Secondly, theimprovement in future-proofing has begun to takeeffect; during a prolonged period of softwaredevelopment, the benefits of tighter specificationand design have begun to be seen as updates becomeeasier to include due to the improved structure ofthe code and documentation. Thirdly, we haveachieved a greater commonality of resourcesbetween products which reduces the duplication ofwork between our code products.

• The application of improved testing proceduresgives us the expectation of a better validated andmore reliable product. Recently introduced fieldtesting has undoubtedly allowed the program to beexercised over a greater range of realistic userproblems. This allowed a selection of errors andanomalies to be removed, but has delayed the finalrelease of the program. We need to be able todemonstrate worth to customers and making testingmore visible is one way of achieving this. As far



as MONK is concerned, we have needed customerinvolvement in the testing process in order toincrease user confidence in the released versionof the program. Although this involvement requiresthe input of resources from the customer, we havefound it to be of mutual benefit and valuableexperience in the management of field testing hasbeen gained.

Standardisation of working practice leads us toexpect a more cohesive software developmentenvironment within which staff may operate moreflexibly. This is a key expectation in an areawhich has shrunk over the years with the reductionof investment in the nuclear industry. It is vitalto the future of our company and its products tohave staff who can move between areas of work inthe knowledge that they may 'slot in' to afamiliar environment with familiar procedures. ForMONK, the system has evolved to the point wherethe reality is starting to approach theexpectation as the QMS 'shakes down' to aframework within which more staff feel at ease,and within which we have motivated staff whounderstand the meaning of true quality.

A spin-off of more standardised working practicehas been the ability to work in a collaborativeenvironment with some of our customers who alsofunction within the bounds of quality managementsystems. Some of the new features of the MONKprogram have been developed at British NuclearFuels pic, with whom AEA Technology has acollaboration known as Nuclear Codes Development(NCD). The smooth functioning of MONK developmentwithin NCD has been facilitated by the existenceof the established QMS.

An additional benefit of an early involvement withQMS has been the ability of ANSWERS to influenceoverall company policy and assist in the ISO 9001Certification process for a wider portion of theorganisation. The AEA Reactor Services businesshas recently been certificated against ISO 9001,and ANSWERS has been able to provide specialistadvice in the TickIT area.



The Drawbacks

• One criticism levelled at the system is that itnow takes too long to build in error correctionsor minor improvements due to the complexity of therelease process and constraints of the proceduresin general. At present there is some validity inthis suggestion since the emphasis has been placedheavily on full regression testing irrespective ofthe simplicity of the modification which has takenplace. We overcome this problem to some extent byissuing 'development' versions of programs whichhave been through the full release procedure priorto their issue. In the future, with a greaterautomation of test facilities, we expect this tobe less of an issue.

• It has been argued that the application of formalprocedures leads to inflexibility and the loss of'pioneering spirit'. This, however, very muchreflects the recognition that in areas wheresafety is all important, and where the cost offailure is substantial, rigorous procedures mustbe applied to ensure the quality and correctfunctionality of the product.

• The time required to maintain both the QMS and ISO9001 Certification should not be under-estimated.Effort is absorbed in QA management activitiessuch as internal audit preparation and execution,surveillance audit preparation and follow-up, QMSreview and so on. Whilst these are all necessaryand worthwhile activities, the pay-back period interms of improved, lower-maintenance product ismeasured in years, not months, for a program suchas MONK. This can damage the perceived benefits ofthe QMS and of Certification in the shorter term.



CONCLUSIONS

Maintaining a quality management system is not simplya matter of carrying out a formal review once a year andpassing a report to the managing director. A QMS is adynamic system which needs to be constantly tuned to theneeds of those who work within it, in order that it maycontinue to be effective and efficient. We expect to seeimprovements both in the products developed under theauspices of the QMS, and in the way in which theoperations of our organisation function on a day-to-daybasis. Experience has shown these benefits are beginningto filter through and that these gains outweigh anylosses which have been incurred.

REFERENCES

1. Maddison,R.j.,Miller,P.C.,Walker,S.D.,"The ANSWERSService for the quality management of scientificsoftware for the nuclear industry", in SQM-93 (Ed.Ross,M., Brebbia,C.A., Staples,G., Stapleton,J.),pp.107-114, Proceedings of the First InternationalConference on Software Quality Management, Southampton,U.K., 1993.

2. Smith,N.R., "A Summary of the Current Status of theMONK6 Code and its Validation for CriticalityApplications", pp. lv-69 to IV-76, Proceedings of theICNC '91 International Conference on Nuclear CriticalitySafety, Oxford, UK, 9-13 September 1991.

© AEA


maintaining your quality management the answers software ... · 50 software quality management into...

Documents