toward standardization of an automated software quality model: the grid-qcm

22
www.eu-etics.org INFSOM-RI-026753 Toward standardization of Toward standardization of an automated software an automated software quality model: the Grid- quality model: the Grid- QCM QCM Adriano Rippa Adriano Rippa Engineering Ingegneria Informatica S.p.A. Engineering Ingegneria Informatica S.p.A. ETICS Final Review ETICS Final Review CERN, Geneva - 15 February 2008 CERN, Geneva - 15 February 2008

Upload: helmut

Post on 06-Jan-2016

41 views

Category:

Documents


5 download

DESCRIPTION

Toward standardization of an automated software quality model: the Grid-QCM. Adriano Rippa Engineering Ingegneria Informatica S.p.A. ETICS Final Review CERN, Geneva - 15 February 2008. Summary. Introduction to QA concepts The starting point of the study - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Toward standardization of an automated software quality model: the Grid-QCM

www.eu-etics.org

INFSOM-RI-026753

Toward standardization of an Toward standardization of an automated software quality automated software quality

model: the Grid-QCMmodel: the Grid-QCM

Adriano RippaAdriano RippaEngineering Ingegneria Informatica S.p.A.Engineering Ingegneria Informatica S.p.A.

ETICS Final Review ETICS Final Review CERN, Geneva - 15 February 2008CERN, Geneva - 15 February 2008

Page 2: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 2INFSOM-RI-026753

Summary

• Introduction to QA conceptsIntroduction to QA concepts• The starting point of the studyThe starting point of the study• The The Grid Quality Certification ModelGrid Quality Certification Model (Grid-QCM) (Grid-QCM)• Possible Questions, Timeline and feedbackPossible Questions, Timeline and feedback• ConclusionsConclusions

Page 3: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 3INFSOM-RI-026753

Quality Assurance refers to several concepts

• Quality of the implementation Quality of the implementation process process – High level steps of the software High level steps of the software

production cycle suggesting what production cycle suggesting what the organization must do (not how) the organization must do (not how) to have effective development to have effective development processes that “processes that “may leadmay lead” to good ” to good software.software.

• Quality of the requirements Quality of the requirements management management – Correct collection/management of Correct collection/management of

requirements and relation with the requirements and relation with the customer and stakeholders, to customer and stakeholders, to reduce the percentage of failures reduce the percentage of failures due to misinterpreted requirements.due to misinterpreted requirements.

• Quality of the service Quality of the service – Performances of the servicePerformances of the service

• Finally quality of the software…Finally quality of the software…

QUALITY

Quality of Product

Quality of Process

Grid-QCMGrid-QCMCMMCMM

ITILITILISO – 900xISO – 900x

Page 4: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 4INFSOM-RI-026753

Some QA Terminology

• According to ISO 9126 documentation we can define:According to ISO 9126 documentation we can define:

• Measure: Measure: the number or category assigned to an the number or category assigned to an attribute of an entity by making a measurement attribute of an entity by making a measurement (sometimes used as synonymous of metric)(sometimes used as synonymous of metric)

• MetricMetric: : The defined method to measure an attribute and The defined method to measure an attribute and the scalethe scale

• MeasurementMeasurement: : The use of a metric to assign a value The use of a metric to assign a value (which may be a number or category) from a scale to (which may be a number or category) from a scale to an attribute of an entity)an attribute of an entity)

Page 5: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 5INFSOM-RI-026753

Quality is a matter of measure!

• State-of-the-art provide hundreds of metricsState-of-the-art provide hundreds of metrics– The most commonThe most common

– Cyclomatic complexityCyclomatic complexity

– Lines of CodeLines of Code

– Function PointsFunction Points

– Mean Time Between (To) FailureMean Time Between (To) Failure

– Bugs densityBugs density

– ……

– Other approaches (Goal Question Metric - GQM) promote user defined Other approaches (Goal Question Metric - GQM) promote user defined metricsmetrics

– Anomalies distributionAnomalies distribution

– Effort used to solve anomaliesEffort used to solve anomalies

– Cost of not founded anomaliesCost of not founded anomalies

– ……

• An exhaustive list is provided within deliverable D5.7An exhaustive list is provided within deliverable D5.7

Page 6: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 6INFSOM-RI-026753

Quality Standards

• There are There are many standardsmany standards to asses the quality of the processes to asses the quality of the processes of an organization of an organization – CMMCMM– ISO family (e.g. ISO 9126, ISO 900X)ISO family (e.g. ISO 9126, ISO 900X)– ITILITIL– AQAPAQAP

• But QA means initial investments and managing QA means But QA means initial investments and managing QA means devote resources to it!devote resources to it!

• Several studies show that lot of companies (e.g. many SMEs) Several studies show that lot of companies (e.g. many SMEs) can’t afford the initial effortcan’t afford the initial effort and don’t recognise the promised and don’t recognise the promised increase of valueincrease of value..– Only ~70 companies in the world are certified at level 5 Only ~70 companies in the world are certified at level 5

– 50 of them are in India50 of them are in India– source: Gartnersource: Gartner

– Just 25% of the companies in the world are level 2 or above Just 25% of the companies in the world are level 2 or above – source: Kulik, Weber: “Software Metrics Best Practices – 2001”source: Kulik, Weber: “Software Metrics Best Practices – 2001”

and “Software Metrics State of the Art – 2000”and “Software Metrics State of the Art – 2000”

Page 7: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 7INFSOM-RI-026753

QA in non commercial short-live project (e.g. research projects) is missing!

• Current quality assurance standards are useful butCurrent quality assurance standards are useful but– They need lot of time to be applied. The organisation need to They need lot of time to be applied. The organisation need to

be structured and certified. be structured and certified. What for the short-live consortiaWhat for the short-live consortia??

– They provide onlyThey provide only theoretical guidelines theoretical guidelines which need to be which need to be adapted and implemented. adapted and implemented. What for homogeneity and What for homogeneity and comparability comparability of results?of results?

– It’s It’s hard to systematically verify hard to systematically verify goodness of results: goodness of results: managing tools neededmanaging tools needed

– They They need resourcesneed resources to be devoted to to be devoted to

– People needPeople need training training and certification needs and certification needs inspections and inspections and time time to be achievedto be achieved

Page 8: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 8INFSOM-RI-026753

QA in grids: our solution

• Grid-QCM is aGrid-QCM is a modelmodel forfor quality assurance that isquality assurance that is

– fully automatablefully automatable in measuring and verifying activities to in measuring and verifying activities to reduce investments and management effort,reduce investments and management effort,

– not subjectivenot subjective, to certify the object not the process nor the , to certify the object not the process nor the organization,organization,

– product orientedproduct oriented, not process oriented,, not process oriented,

• ……easily adoptable within (Grid) short-live Projectseasily adoptable within (Grid) short-live Projects

Page 9: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 9INFSOM-RI-026753

Grid-QCM: Preliminary Remarks (1/2)

• Using the ETICS tool people can have Grid-QCM Using the ETICS tool people can have Grid-QCM implemented for free in the b&t application.implemented for free in the b&t application.

• Grid-QCM has been developed within a Grid project Grid-QCM has been developed within a Grid project and to asses the quality of grid software research and to asses the quality of grid software research projects but it projects but it can be used for any software can be used for any software applicationsapplications . .

• Grid-QCM has been developed Grid-QCM has been developed according to according to – The feedback received from expert people and potential usersThe feedback received from expert people and potential users – several standards:several standards:

– Grid-QCMGrid-QCM has been described has been described according to ISO standardsaccording to ISO standards (e.g. (e.g. ISO/IEC ISO/IEC 25000, 1459825000, 14598))

– Grid-QCMGrid-QCM has been restructured has been restructured according to ISO 25041according to ISO 25041– Quality attributesQuality attributes has been named using the same terminology as has been named using the same terminology as

ISO 9126ISO 9126

Page 10: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 10INFSOM-RI-026753

The ETICS vision

Grid-QCMGrid-QCM

ETICS SW (v.2.0)ETICS SW (v.2.0)

ETICS grid infrastructureETICS grid infrastructure

CERN, INFN, UoW (NMI)CERN, INFN, UoW (NMI)

Defines Defines

metricsmetrics

RunsRuns

measuresmeasures

Allows Allows

Code analysisCode analysis

Allows Allows

automationautomation

Page 11: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 11INFSOM-RI-026753

Grid-QCM: Structure

• Grid-QCM is structured in Grid-QCM is structured in Evaluation Modules (EM)Evaluation Modules (EM)..

• The set of evaluation techniques are grouped in The set of evaluation techniques are grouped in families. Every family is an Evaluation Modulefamilies. Every family is an Evaluation Module

• 5 Evaluation Modules:5 Evaluation Modules:

– Static analysisStatic analysis– Coding styleCoding style– Structural testingStructural testing– Functional testingFunctional testing– Standards complianceStandards compliance

Page 12: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 12INFSOM-RI-026753

Evaluation Modules 1/2

• Static analysisStatic analysis– Quality characteristics:Quality characteristics:

– Reliability – maturityReliability – maturity– Maintainability – analysabilityMaintainability – analysability– Maintainability – changeabilityMaintainability – changeability– Maintainability – testabilityMaintainability – testability

– Static analysis of classes. Statistics on measures are used as Static analysis of classes. Statistics on measures are used as predictor of quality characteristics.predictor of quality characteristics.

• Coding styleCoding style– Quality characteristics:Quality characteristics:

– Maintainability – analysabilityMaintainability – analysability

– Static analysis of the source code. Static analysis of the source code.

Page 13: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 13INFSOM-RI-026753

Evaluation Modules 2/2

• Structural testingStructural testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Reliability – maturityReliability – maturity

– Structural testing to classes identified more likely to have many errors. Structural testing to classes identified more likely to have many errors.

• Functional testingFunctional testing– Quality characteristics:Quality characteristics:

– Functionality – accuracyFunctionality – accuracy– Functionality – interoperabilityFunctionality – interoperability– Reliability – maturityReliability – maturity– Portability – adaptabilityPortability – adaptability– Portability - installabilityPortability - installability

– Platform compliance and to functional abilities of the software Platform compliance and to functional abilities of the software

• Standards complianceStandards compliance– Quality characteristics:Quality characteristics:

– Functionality – standards complianceFunctionality – standards compliance

Page 14: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 14INFSOM-RI-026753

Grid-QCM: Final Score

• Final score should be provided according to the following Final score should be provided according to the following schema. The items which should be available for the users are:schema. The items which should be available for the users are:– A A table summarizing the resultstable summarizing the results– A A list of passed and failed testslist of passed and failed tests– All the All the important informationimportant information as: as:

– Software product (e.g. name, version, executable code, documentation..)Software product (e.g. name, version, executable code, documentation..)

– Platform (name, version, date)Platform (name, version, date)

– Quality characteristics (name, evaluation result, evaluation module Quality characteristics (name, evaluation result, evaluation module identification)identification)

– Standard compliance (for each standard: name, version, date)Standard compliance (for each standard: name, version, date)

– Identification of evaluation report (organization, report number, date)Identification of evaluation report (organization, report number, date)

– Identification of certification body (organization, contact information)Identification of certification body (organization, contact information)

– Certification data (dates, certification number)Certification data (dates, certification number)

– Electronic signature of certification record Electronic signature of certification record

Page 15: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 15INFSOM-RI-026753

Possible Criticism, Our Answer

• I can’t add any overhead to my projectI can’t add any overhead to my project– This model (and the capability of automate) reduce the effort in This model (and the capability of automate) reduce the effort in

performing continuous build and test activities (e.g. coverage performing continuous build and test activities (e.g. coverage tests) on different SW releases.tests) on different SW releases.

• What about the cost?What about the cost?– Using the ETICS tool people can have the model implemented Using the ETICS tool people can have the model implemented

for free!for free!

• My organisation is certified ISO/CMMI so I…My organisation is certified ISO/CMMI so I…– The model is a standalone quality certification model.The model is a standalone quality certification model.– However it can be easily integrated in yet ISO/CMMI certified However it can be easily integrated in yet ISO/CMMI certified

organisations.organisations.

Page 16: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 16INFSOM-RI-026753

Current ETICS metrics and Grid-QCM

MetricsMetrics TypeType Programming Programming languagelanguage

Grid-QCM: INVOLVED EMsGrid-QCM: INVOLVED EMs

complexitycomplexity staticstatic JavaJava, Python, Python EM CODING STYLEEM CODING STYLE

EM STATIC ANALYSIS EM STATIC ANALYSIS

design qualitydesign quality staticstatic JavaJava EM CODING STYLEEM CODING STYLE

EM STATIC ANALYSIS EM STATIC ANALYSIS

nr of nr of „possible”„possible” bugsbugs

staticstatic Java, Java, C/C++, C/C++, Python, Perl, Python, Perl, PHP PHP

EM CODING STYLEEM CODING STYLE

EM STRUCTURAL TESTINGEM STRUCTURAL TESTING

nr of nr of „possible”„possible” bugsbugs

dynamicdynamic C/C++C/C++ EM STRUCTURAL TESTINGEM STRUCTURAL TESTING

unit testunit test dynamicdynamic JavaJava, Python, Python EM FUNCTIONAL TESTINGEM FUNCTIONAL TESTING

EM STRUCTURAL TESTINGEM STRUCTURAL TESTING

coveragecoverage dynamicdynamic JavaJava EM FUNCTIONAL TESTINGEM FUNCTIONAL TESTING

EM STRUCTURAL TESTINGEM STRUCTURAL TESTING

profiling profiling informationinformation

dynamicdynamic Java, C/C++Java, C/C++ EM FUNCTIONAL TESTINGEM FUNCTIONAL TESTING

Page 17: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 17INFSOM-RI-026753

The timeline

OCTOBEROCTOBER DECEMBERDECEMBER

ECHOGRID/EUChinagridECHOGRID/EUChinagrid

Conference 24-25 AprilConference 24-25 April

Beijing (CHINA)Beijing (CHINA)

MAYMAYFEBRUARYFEBRUARY

OGF 20/EGEE UFOGF 20/EGEE UF

7-11 May7-11 May

Manchester (UK)Manchester (UK)

Belief ConferenceBelief Conference

25-28 June25-28 June

Rio de Janeiro (BRA)Rio de Janeiro (BRA)

OGF 21OGF 21

15-19 Oct15-19 Oct

Seattle (USA)Seattle (USA)

QUALIPSO QUALIPSO ConferenceConference

16-17 Jan 200816-17 Jan 2008

Rome (IT)Rome (IT)

NowNow

EGEE ’07EGEE ’07

1-5 Oct1-5 Oct

Budapest (Hun)Budapest (Hun)

EELA 3° Conference EELA 3° Conference 3-5 Dec 20073-5 Dec 2007

Catania (IT)Catania (IT)

ESA 3rd GRID & e-ESA 3rd GRID & e-Collaboration Workshop Collaboration Workshop

16-17 Jan 200816-17 Jan 2008

Frascati (IT)Frascati (IT)

Page 18: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 18INFSOM-RI-026753

Grid-QCM: feedback

• OGF 20OGF 20• Possibility to automatePossibility to automate

• Unclear formal name of the modelUnclear formal name of the model• Criticism: GQACM was difficult to rememberCriticism: GQACM was difficult to remember

• Solution: New name: Grid-QCMSolution: New name: Grid-QCM

• Belief/EELA ConferenceBelief/EELA Conference• Automation, CMMI/ISO compatibility Automation, CMMI/ISO compatibility

• Structure of some parts of the modelStructure of some parts of the model• Criticism: Metrics not well split according to their roleCriticism: Metrics not well split according to their role

• Solution: Organisation in Evaluation ModelsSolution: Organisation in Evaluation Models

• EGEE’07EGEE’07• Automation, ETICS + Grid-QCM = free, people asked for Automation, ETICS + Grid-QCM = free, people asked for

specific information specific information

• Metrics for the processMetrics for the process• Criticism: Lack of process metricsCriticism: Lack of process metrics

• Solution: it’s out of the scopeSolution: it’s out of the scope

Page 19: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 19INFSOM-RI-026753

Grid-QCM: feedback

• OGF 21OGF 21• Automation, CMMI/ISO compatibility, people asked for Automation, CMMI/ISO compatibility, people asked for specific specific

informationinformation

• ISO9000 compatibilityISO9000 compatibility• Criticism: no mention about ISO9000 compatibilityCriticism: no mention about ISO9000 compatibility

• Solution: it’s out of the scopeSolution: it’s out of the scope

• QualiPSo ConferenceQualiPSo Conference• Automation, Used Standards, Automation, Used Standards,

ETICS + Grid-QCM = free ETICS + Grid-QCM = free Many people interested in specific information Many people interested in specific information

Page 20: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 20INFSOM-RI-026753

Conclusions

• Grid-QCM is a certification modelGrid-QCM is a certification model– AutomatableAutomatable– Implemented for free in the ETICS toolImplemented for free in the ETICS tool– Not in contradiction with classical standardsNot in contradiction with classical standards– Ready to interact with classical standards Ready to interact with classical standards – Not limited to research projectsNot limited to research projects– Not limited to grid softwareNot limited to grid software

• Grid-QCM require less human effort to be used because it is Grid-QCM require less human effort to be used because it is almost fully automatablealmost fully automatable

• ETICS tool is ready to implement Grid-QCMETICS tool is ready to implement Grid-QCM

• During the ETICS 2 project, if approved, During the ETICS 2 project, if approved, – Grid-QCM will be validated on-the-field with at least four projectsGrid-QCM will be validated on-the-field with at least four projects– Grid-QCM will be proposed for standardisation under ISO.Grid-QCM will be proposed for standardisation under ISO.

Page 21: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 21INFSOM-RI-026753

Q&A

http://www.eu-etics.org

Page 22: Toward standardization of an automated software quality model: the Grid-QCM

ETICS Final Review - Grid-QCM - CERN, 15 February 2008 22INFSOM-RI-026753

Grid-QCM: Summarizing Table example

Consolidated eval. result

EM Static

EM Coding

EM structural

EM Functional

EM Std compliance

EM eval. resultEM eval. result MM MM GG GG GG YY

Functionality G G G

Accuracy G G G

Interoperability G G

Compliance Y (Y)

Reliability M M G G

Maturity M M G G

Maintainability M M G

Analyzability M M G

Changeability M M

Testability M M

Portability G G

Adaptability G G

Installability G G

E = ExcellentG = GoodM = MediumF = FairP = Poor