wojciech sliwinski wojciech.sliwinski@cern.ch beams department, controls group cern
Post on 24-Feb-2016
24 Views
Preview:
DESCRIPTION
TRANSCRIPT
CONTINUOUS INTEGRATION AND QUALITY ASSURANCE FOR THE ACCELERATOR
CONTROLS CODEBASE AT CERNJINR/CERN/MEPHI COMPUTING SCHOOL
DUBNA, 24TH OCTOBER 2011
Wojciech SliwinskiWojciech.Sliwinski@cern.chBeams Department, Controls GroupCERN
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 2
Outline
Defining Software Quality
Continuous Improvement
Context – Accelerator Controls Codebase
Applying QA & CI for the Controls Codebase
Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 3
Outline
Defining Software Quality
Continuous Improvement
Context – Accelerator Controls Codebase
Applying QA & CI for the Controls Codebase
Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 4
Software Quality (SWQ)…
We know it’s “a good thing” ™
People think…“I know it when I see it”“I know the value of it”
But what are the actual characteristics of SWQ? Do we have a common understanding of SWQ?
The lack of it normally appears as consequences…
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 5
1985: Therac-25 – Software issues…
Radiation Deaths linked to Software ErrorsKilled 2 peopleInjured dozens of othershttp://www.ccnr.org/fatal_dose.html
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 6
1996: Ariane 5 – floating point conversion error
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 7
2007: Airbus A340 – Software issues …
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 8
80s – The Kano Model (Marketing)C
usto
mer
Sat
isfa
ctio
n
Product Features
Basic Needs MetLinear P
erformance
Delight, Surprise,
Innovate
24th October 2011
Satisfied
Dissatisfied
Needs wellfulfilled
Needs notfulfilled
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 9
Early and Other Quality Definitions
Zen and the art of motorcycle maintenance, Robert Pirsig, 1974:“If quality exists in an object, then you must explain why scientific
instruments are unable to detect it”“On the other hand, if quality is subjective, [existing only in the eye of
the observer] then this Quality is just a fancy name for whatever you'd like.”
“Quality is not objective. It doesn't reside in the material world [...] Quality is not subjective. It doesn't reside merely in the mind.”
McCall et al, 1977 and Boehm et al, 1978 First elaborate studies on the notion of 'software quality’Quality factors – only indirectly measurable like “reliability”Quality criteria – objective aspects that support the factors, like
“uptime”.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 10
IEEE SWQ Standards
IEEE Glossary of Software Engineering Terminology defines it as“the degree to which a system, component, or process meets
customer or user needs or expectations” IEEE Std 1061-1992 IEEE Standard for a Software Quality
Metrics MethodologyDoes not actually prescribe any metrics!Defines method to evaluate metrics that could be applicable to
a particular case Plenty more – eg. IEEE Std 730 and friends
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 11
ISO Quality Standards
ISO 9000 series of standards for quality management systems:Process based“the degree to which a set of inherent characteristics fulfils the
requirements”
ISO 9126 Software Quality Characteristics and Metrics, 2001Comprehensive but … large & complex6 key quality attributes:
○ Functionality, Reliability, Usability○ Efficiency, Maintainability, Portability
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 12
SWQ – Huge Set of CharacteristicsFunctionality Suitability Accuracy Interoperability Security Functionality compliance Reliability Maturity Fault tolerance Recoverability Reliability compliance Usability Understandability Learnability Operability Attractiveness Usability compliance Efficiency Time behavior Resource utilization Efficiency compliance Maintainability Analyzability Changeability Stability Testability Maintainability compliance Portability Adaptability Installability Co-existence Replaceability Portability compliance Availability Maintainability Efficiency Portability Flexibility Reusability Integrity Testability Interoperability Reliability Robustness Usability Effectiveness Safety Productivity Satisfaction
• Only a subset applies to all projects• Choose those that suit the project goals
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 13
McCalls’ categories of SWQ factors
Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality,RADC-TR-77-369, 1977, US Dep. of Commerce.
Operation: Correctness Does it do what I want?Reliability Does it do it accurately all of the time?Efficiency Will it run on my hardware as well as it can?Integrity Is it secure?Usability Can I run it?Revision: Maintainability Can I fix it?Testability Can I test it?Flexibility Can I change it?Transition: Portability Will I be able to use it on another machine?Reusability Will I be able to reuse some of the software?Interoperability Will I be able to interface it with another system?
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 14
Internal & External Quality Attributes
Important Primarily to Users Important Primarily to DevelopersAvailability Maintainability
Efficiency Portability
Flexibility ReusabilityIntegrity Testability
Interoperability Reliability
Robustness
Usability
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 15
The SW Quality “Iceberg”
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 16
The Maintainability Index (MI)
Parameter Name Measures
aveV Average Halstead complexity Computational density
aveV (g )′ Average extended cyclomatic complexity Logical complexity
aveLOC Average count of lines of code Code size
PerCM Average percent of lines of comments Human insight
MI = 171 – 5.2 x ln(aveV) - 0.23 x aveV(g’) - 16.2 x ln(aveLOC) + 50 x sin √2.4PerCM
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 17
MI applied to FreeBSD codebase
24th October 2011
Kernel Codebase User Programs
Modules with MI < 40 are rejected !
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 18
Applying Quality
Computers don’t care about quality!
Metrics are tests for quality characteristics Quality “clusters”
Testing is ensuring a quality of conformanceBugs occur in clusters too!
Like testing, if quality is low, builds should fail
Quality characteristics must be part of the whole development lifecycleQuality products only come from quality requirements, quality design, quality
code, …24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 19
Applying Quality (Assurance)
Software Quality consists of:1. Engineering Methods (proper analysis, design, …)2. Standards and Procedures (common to all)3. Technical Reviews (eg. code reviews)4. Configuration management (repositories, versioning)5. Measurement (code, metrics)6. Testing (unit, system, integration, acceptance)7. Improvement (of the items above)
24th October 2011
• Quality is a way of life• If you think Quality is expensive, try an accident…
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 20
Capability Maturity Model (CMM) Levels*
1. Initial (chaotic, ad hoc, individual heroics)○ the starting point for use of a new process.
2. Repeatable○ some processes are repeatable, possibly with consistent results
3. Defined ○ the process is defined/confirmed as a standard business process, and
decomposed to levels 1 and 2
4. Managed (quantatively)○ using process metrics, management can identify ways to adjust and adapt
the process to particular projects
5. Optimized○ process management includes deliberate process
optimization/improvement.
* Capability Maturity Model from SE Institute.24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 21
Outline
Defining Software Quality
Continuous Improvement
Context – Accelerator Controls Codebase
Applying QA & CI for the Controls Codebase
Conclusions
24th October 2011
Continuous Improvement (CI) – Origins
Deming’s Plan-Do-Check-Act (PDCA) The Toyota Production System
“Stop the line” quality control About eliminating “waste” (obstacles) A learning, non-judgmental whole team approach Adopted by the Agile movement for SWD Caveat: Development is an “empirical process” Introduced when an organization recognizes it has
arrived at some impasse
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 22
CI - How it works
“People (at all levels) freely discuss the information, issues, ideas, evaluate them, choose, plan and execute improvements”
Focuses on how things get done:Standardized (and ing) common tasksOnly fixing possible issuese.g.: repeated mistakes, time sinks, quality holes
Very suitable where tools, technology, external outcomes and requirements change often
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 23
CI - What it relies on
Objective information:To evaluate current situation and processesAssess applied improvementsJudge outcome as worthwhile, or retryDemonstrate added value of CI against overhead
Improvement culture:“Free speech” – everyone’s ideas are welcome A real desire to try it and improveAccepting there will not be total agreementReplacing skeptical inaction with real experimentation
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 24
CI – It can fail!
No one “owns” responsibility for implementation No charter to make changes No time allocated for improvement No follow up by team or sponsors
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 25
Improvement Practices
Change AgentsAKA: Coaches, Monitors, Consultants, ChampionsOne or more floating people searching for improvements
Kaizen time aka “renovation days”Team plans some improvementsAll actions executed by all at same time
Reflection workshopsRegular meetings to decide on improvementsApply one or more ideasKeep what works
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 26
CI - Summary
Team driven incremental changes Team evolves and improves together Drives increase in quality, efficiency and satisfaction Not a tool or technique – it is an attitude Not leader oriented, instead peer based consensus Keeps us ahead in best practice and professionalism
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 27
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 28
Bibliography
Software Requirements
K. Wiegers, Microsoft Press, 2003
Code Complete: A Practical Handbook of Software Construction S. McConnell, Microsoft Press, 2004
Software Engineering: Principles and Practice H. van Vliet, John Wiley & Sons, 2008
Wikipedia page on SWQ ISO standard 9126.
24th October 2011
Bibliography
The Toyota Way, J. K. Liker, 2004 Agile and Iterative Development, C. Larman, 2004 Sustainable Software Development, Kevin Tate, 2006 Implementing Lean Software Development, M. & T. Poppendick,
2007 Rapid Development, S. McConnell, 1996 The Enterprise and Scrum, K. Schwaber, 2007 Collaboration Explained, J. Tabaka, 2006 Peopleware Papers: The Human side of Software, L. Constantine,
2001
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 29
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 30
Outline
Defining Software Quality
Continuous Improvement
Context – Accelerator Controls Codebase
Applying QA & CI for the Controls Codebase
Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 31
Context - CERN Accelerator Complex
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 32
Context - The CERN Control Centre
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 33
TCP/IP communication services
TCP/IP communication services
PUBLIC ETHERNET NETWORK
TIMING GENERATION
TT
T
T
T
T T T T
OPERATORCONSOLES
OPERATORCONSOLES
FIXEDDISPLAYS
CE
RN
GIG
AB
IT E
THER
NE
T
TEC
HN
ICA
L N
ETW
OR
K
FILE SERVERS
APPLICATION SERVERS
SCADA SERVERS
RT Lynx/OSVME Front Ends
WORLDFIPFront Ends PLC
LHC MACHINE
ACTUATORS AND SENSORSCRYOGENICS, VACUUM, ETC…
QUENCH PROTECTION AGENTS,POWER CONVERTERS FUNCTIONSGENERATORS, …
BEAM POSITION MONITORS,BEAM LOSS MONITORS,BEAM INTERLOCKS,RF SYSTEMS, ETC…
Wor
ldFI
P
SE
GM
EN
T (1
, 2.5
MB
its/s
ec)
PR
OFI
BU
S
FIP
/IO
TCP/IP communication services
OP
TIC
AL
FIB
ER
S
Controls Software Infrastructure
Client tier
Server tier
Resource tier
CTRL CTRL
DB
Business Layer
Device Layer
Presentation Layer
CMW
A 3-tier architecture• Presentation Layer
– Graphical interactive applications
– Fixed Displays• Business Layer
– General purpose and specific Application servers
• Device Layer– Front End
Device servers• Communication to the
equipment goes through Controls MiddleWare CMW
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 34
Controls Software Codebase - Situation
Complex Accelerator domain
~6 million LOC Java/J2EE
~200 developers (CERN + other labs) ~1000 projects/modules
C/C++ ~35 developers (CERN + other labs) +20 major projects
Oracle database Eclipse IDE + plugins SVN + Maven + CommonBuild( Atlassian Suite (JIRA, Fisheye, Crucible, etc.)
Nowadays encourage Scrum Focused on Software Quality
GUI Applications
Control Logic
Middleware
Control System
24th October 2011
Controls Software Codebase - Situation
Lots of Code~6 million lines, 20’000+ classes.… and still growing
24th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 35
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 36
Situation – Self Assessment and Outlook
Unknown qualityQuality uncheckedNo mentoring or guidance?Limited standards
High Staff turnoverStudents, Project Associates, Fellows …
More projects coming (PS renovation, …) More service provision (looking after running services) Will this cause future problems like maintenance?
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 37
Outline
Defining Software Quality
Continuous Improvement
Context – Accelerator Controls Codebase
Applying QA & CI for the Controls Codebase
Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 38
Objectives – improve Software Quality
Introduce quality improvement as an integral part of the everyday development work
Leverage tools to automate the process as much as possible
Establish guidelines and metrics to measure progress
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 39
Quality in the Development Cycle
For each stageAgreed mandatory activities and deliverablesTools identified and integrated with IDE (Eclipse), giving
immediate feedback
Design
Implement, Test,
Document
Deploy, Maintain
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 40
Design Reviews
Design meetings with external members
To verify the soundness of design in an early stage
To identify overlapping functionality
Promote a set of common components and libraries
At the level of sub-components, main classes and design patterns
UML: class and sequence diagrams
Design
Implement, Test,
Document
Deploy, Maintain
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 41
Code Reviews & Code Analysis
Goal: Identify defects Ensure maintainable code Verify conventions are followed
Static code analysis tools to identify common mistakes and bug patterns: FindBugs Checkstyle Eclipse warnings
Person-to-person time consuming Only for critical code Mentoring of junior
developers Light-weight person-to-
person code reviews with FishEye + Crucible
Design
Implement, Test,
Document
Deploy, Maintain
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 42
Files being reviewed
Author and
reviewers Comments inline
24th October 2011
43
Reviewing Code (Crucible)
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 44
A list of ‘bugs’
The ‘bug’ line indicated
‘Bug’ explained
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 45
Testing & Continuous Integration
General agreement on benefits of testing and CI
To increase level of testing, unit tests mandatory deliverable of each project >30% test coverage for non-trivial
classes, measured with Clover
To discover inter-project issues early, Continuous Builds with Bamboo: CI server from Atlassian Compiles and runs unit tests Builds fail on:
○ Low test coverage○ Basic PMD rules
Design
Implement, Test,
Document
Deploy, Maintain
24th October 2011
46
Continuous Integration and Test (Bamboo)
Test coverage for project Code metrics
Classes with highest risk
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 47
Red = not covered
Green= covered
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 48
Triggered by changes in a dependencyaccsoft-
commons-core
accsoft-common
s-util
accsoft-commons-
concentration
accsoft-commons-
io
Top/flop list generated (Work in progress)
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 49
Integration and System Testing with CO Testbed
CO Testbed - mini-accelerator labHardware and SoftwareCore components from the Accelerators Controls duplicated
into this Test Environment
System and Integration tests are executed by Bamboo CI serverAutomatic execution of tests with “real” systemAutomatic deployment of Release Candidates
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
50
CO Testbed Hardware in placeTIMING
FEC03
FEC01 FEC02
FEC04FEC05
SERVER06SERVER07
24th October 2011
51
Integration Tests done through Client APIsBamboo CI Server
CMW Proxy CMW Directory Svc
RBAC A1 Service
ConfigDB
Client APIs (JAPC, CMW, RBAC)
Timing
CMW, RBAC A2, FESA
CMW, RBAC A2, FESA
PPC/LynxOS Linux/i386
TESTS
Woj
ciec
h S
liwin
ski:
CI a
nd Q
A fo
r the
Acc
eler
ator
C
ontro
ls C
odeb
ase
at C
ER
N
JMS
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 52
CO Testbed controlled by Bamboo CI server
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 53
1.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 54
1.
2.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 5524th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 56
Challenges
Think of and organize us as one big team, not many small ones
The code belongs to the group, not to a project or an individual developer
The guidelines and standards established and agreed by everyone
Chal
leng
esAgree on standards and configurations
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 57
Encourage developers to invest time on quality
Quality-related tasks part of the yearly objectives for a project
Progress measured against the level a project adheres to guidelines – top/flop lists
Focus the effort where the effect is highest, rely on tools as much as possible
‘CI days’ – common days with a common goal
Chal
leng
es
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN 58
Conclusions
Integrated quality improvement in the development cycle
Introduced guidelines/standards and supporting tools
Increased developer motivation through Immediate feedback when developingOfficial tracking of progress and top/flop lists
Increased awareness of benefits, application in individual projects and confidence when making changes
Agile approach reflect, identify ”waste” and improve24th October 2011
top related