product line engineering lecture – quality assurace...koala – component definition component is...
TRANSCRIPT
© Fraunhofer IESE
1
Schedule - Lectures
© Fraunhofer IESE
2
Schedule - Exercises
© Fraunhofer IESE
Product Line Scoping
--- Recap ---
Component-basedPLE
© Fraunhofer IESE
4
Component-based Development
Development(for reuse)
Assembly(with reuse)??
© Fraunhofer IESE
5
Towards Systematic Reuse
(De-)Composition
* divide and conquer
* plug and play
* component technology asintegrationmechanism
Development for Reuse* generalization* parameterization
Development with Reuse* instantiation* adaptation
Development of generic
components
Development of specific
components
Instantiation of generic
components
Adaptation of specific
components
© Fraunhofer IESE
6
Component Modeling – Generic Component Model Suite
Specification
Realization
FunctionalModel
BehavioralModel
Structural Model
Structural Model
Activity Model
Interaction Model
DecisionModel
DecisionModel
© Fraunhofer IESE
7
The Principle of Locality
Run-time composition structures are traditionally described within global models
inhibits reuse
non-component oriented
The principle of locality states that
content and organization of software artifacts should be relative to a single component
Global artifacts
should be used as little as possible
should be generated recursively
© Fraunhofer IESE
8
Calendar Specification: Structural
© Fraunhofer IESE
9
Component Specification: Behavioral
© Fraunhofer IESE
10
Component Specification: Functional
© Fraunhofer IESE
11
Role of a Component Realization
To describe the way in which a component realizes the services specified in the specification
Defines a component’s
private subcomponents
architecture and design
Is defined in the context of the component’s specification
the realization models expand upon the information in the specification model
© Fraunhofer IESE
12
Role of Realization Activities
Data Activities
StructuralModeling
InteractionModeling
ActivityModeling
© Fraunhofer IESE
13
KOALA – component definition
Component is a unit of design, development and reuse
Interact between the environment or other components ONLY through explicit interfaces
Koala components are defined in its:IDL (interface definition language)CDL (component definition language)DDL (data definition language)
Syntactically: KOALA components are defined in ADL-like language
Semantically: KOALA components are units of computation and control connected together in an architecture
© Fraunhofer IESE
14
Component Interfaces
C
m
p : I
interface I{
int Max(int x, int y);float Sin(float x);
}
component C{
provides I p;contains module m;connects p = m;within m{
p.Max(x,y) = x > y ? x : y;// p.Sin implemented in C
…}
Provides Interfaces
Has-A rather than Is-A
Ports rather than Inheritancecode-carrying
model ☺
© Fraunhofer IESE
15
Connectors
C1
C2
C1
C2
C1
C2 C3
Switch Glue ModuleDirect
r
p
r
p1
r
pp2
m
© Fraunhofer IESE
16
Quality Assurance (QA)Overview
© Fraunhofer IESE
17
Product Line Quality AssuranceMotivation
Product lines are based on the systematic, large-scale reuseof components and other software development workproducts
Product line assets developed for reuse must have a high level of quality
because otherwise, quality problems in an asset would not just lead to a single end product with low quality, but it would propagatethis low quality into all products making use of the asset
because low-quality assets would not be reused in the long term
More than for traditional, single-system software development,quality assurance becomes a crucial part of product line engineering
© Fraunhofer IESE
18
Product Line Quality AssuranceState-of-the-Art
Research in the field of product lines has mainly focused on development(analysis, design, and implementation)
Unique issues of product line quality assurance have been investigated and addressed by only few researchers so far
General techniques, methods, models, and tools for quality assurance in the context of product lines are still missing
© Fraunhofer IESE
19
Product Line Quality AssuranceState-of-the-Practice
Quality assurance techniques and strategies for traditional single systemsare largely applied
The overall quality assurance effort should not be overly compromized by additional product line quality assurance
There is time pressure, and limited resources are available for quality assurance
© Fraunhofer IESE
20
Product Line Quality AssuranceProblems in Practice (1/2)
Quality assurance cannot keep pace with accelerated developmentin software product line engineering
Quality assurance is already the bottleneck in single system development
Technical issues: genericity and integration
Organizational issues: more stakeholders, asynchronous communication, and distributed responsibilities
© Fraunhofer IESE
21
Product Line Quality AssuranceProblems in Practice (2/2)
Quality assurance is very time and effort consuming and there are too few resources available for assuring the quality of all the products and possible combinations in a product line
Scaling of quality assurance resources often impossible
Benefits of product lines will not be achieved if quality assurance effort is increased too much
Trade-off between time-to-market and effort vs. quality
© Fraunhofer IESE
22
Product Line Quality AssuranceGeneral Challenges
Make best possible use of the limited time and resources available for quality assurance
Balance the quality assurance effort between product line assetsand products
Minimize the quality assurance effort per product as well as the overall product line quality assurance effort
Select or define a quality assurance strategy most appropriate for available resources, given time schedules, and quality requirements and taking into account risks
© Fraunhofer IESE
23
Quality Assurance (QA)What is quality assurance and why do we need it for product lines?
© Fraunhofer IESE
24
Quality Assurance -A subprocess of Quality Management
Quality Management/Engineering/Assurance are often used as synonyms
Here, Quality Assurance is a sub-process of Quality Management
Quality management [ISO9000:2005]
Controlled activities to direct and control an organization with regard to quality
Process [IEEE829-2008]
Set of interrelated activities, which transform inputs into outputs
Quality [ISO9000:2005]
Degree to which a set of inherent characteristics fulfills requirements
Legend
(sub)activitycontrol flow
Quality ManagementQuality
PlanningQualityControl
QualityAssurance
QualityImprovement
© Fraunhofer IESE
25
Quality Management –Concerns
At organizational level
Establishing a framework of organizational (development) processesand (reqirements, design & code) standards for high-quality software
At project level
Application of specific quality processes, checking that these processes have been followed, ensuring that project outputs conform to project standards
Establishing a quality plan (quality goals & standards)
© Fraunhofer IESE
26
Quality Assurance –Characteristics
Quality Assurance [IEEE12207-2008]
All the planned and systematic* activities implemented within the quality system, and demonstrated as needed, to provide adequate confidence that an entity will fulfill requirements for quality
Quality assurance impactsall processes in the softwaredevelopment life-cycle,not just implementation
* systematic implies: repeatable & consistent
Quality ManagementQuality
PlanningQualityControl
QualityAssurance
QualityImprovement
Software Development
Quality Assurance
RequirementsEngineering Design Implemen-
tation
© Fraunhofer IESE
27
Application Engineering (AE)
Family Engineering (FE)
Core Asset Base
RequirementsEngineering Design Implementation
RequirementsEngineering Design Implementation
Product Re-quirements
Product
Recap: Product Line Development Process
Requirements Architecture Code
© Fraunhofer IESE
28
Application Engineering
Family Engineering
Core Asset Base
RequirementsEngineering Design Implementation
RequirementsEngineering Design Implementation
Product Re-quirements
Product
Product Line Development & Quality Assurance
Requirements Architecture Code
Quality Assurance
QA Assets
Quality Assurance
© Fraunhofer IESE
29
QA Assets
Quality AssuranceSub-Activities / Techniques
QA sub-activities
Inspections
Measurement
Testing
Simulation
… and correspondingproduct line assets
Quality Assurance (in AE or FE)
Inspections TestingMeasure-ment
…Simulation
Inspect.Forms Metrics Tests
Simu-lations
…
© Fraunhofer IESE
30
Development Qualities –Single Products (AE) vs. Product Line Assets (FE)
Single Product / Component
Functionality – does the product produce the right outputs?
Efficiency – does the product consume an appropriate amount of space & time?
Ease-of-use – how easily can a product be used in development?
Product Line / Reuse Asset
Usability – does the asset have appropriate functionality, efficiency & ease-of-use?
Generality – can the asset be reused in all required product instances?
Adaptability – how much work must be done to configure the asset?
© Fraunhofer IESE
31
Product Line InspectionsHow can we review product line assets?
© Fraunhofer IESE
32
Inspections (Reviews)Characteristics and Benefits
Characteristics
static analysis technique that relies on visual examination of software development workproducts to detect errors, violations of development standards, and other problems [IEEE610-1990]
used to check that certain quality aspects of a software item are fulfilled
can be performed on every workproduct that is created in the software development life-cycle (e.g. requirements, design, test cases, code, etc.)
allow identification and correction of defects earlier in the software development cycle compared to testing
Benefits
Early defect detection
Reduction of costs early in the life cycle
Facilitates early feedback
Quality Assurance
Inspections TestingMeasure-ment …
© Fraunhofer IESE
33
Recap: Traditional Document & Code Inspection
Steps:
Techniques:
Ad-hoc
Checklist-based
Perspective-based
Planning
Detection
Meeting
Correction
Inspector
Organizer
Author
Author orModerator or
Inspector
Document
PlanningForm
DataSummary
DefectForm
DefectCorrection
Form
IssueReport Form
CorrectedDocument
© Fraunhofer IESE
34
InspectionsProduct line-specifics
Application inspections (inspections in AE)are similar to traditional single systems inspections
because they only deal with examining single products
Differences lie in
Family inspections (inspections in FE)
Must consider product line issues => variability
Product line inspection assets
Contain variability
Do not necessarily have to deal with functionality,but may also deal with construction issues (see later)
© Fraunhofer IESE
35
Product Line InspectionsTwo Extreme Approaches
Brute Force Strategy
Idea: Perform all inspection activities for all possible products in family engineering, none in application engineering
Disadvantages
large amount of inspection assets
overhead
Pure Application Strategy
Idea: Perform all inspection activities for all possible products in application engineering, none in family engineering
Disadvantages
no advantages over single systems inspections
overhead
AE RE Des. Imp.
FE RE Des. Imp.Inspections
Inspections
AE RE Des. Imp.
FE RE Des. Imp.Inspections
Inspections
© Fraunhofer IESE
36
Product Line InspectionsViable Approaches
Sample Application Strategy
Idea: Use one or a few sample applications to inspect the core assets. Application inspections are still needed for each application
Pros&cons: low learning effort, but overhead
Commonality & Reuse Strategy
Idea: Inspect common parts and prepare inspection assets for variable parts in family testing. Reuse both kinds of inspection assets for inspecting individual products in application inspections
Pros&cons: low loverhead, but learning effort
Combinations
Balances individual pros&cons
© Fraunhofer IESE
37
Product Line MeasurementHow can we measure product line qualities?
© Fraunhofer IESE
38
Measurement –Characteristics and Benefits
Characteristics
Process by which numbers or symbols are assigned toquality attributes of software entities (components, systems, processes)in the real world [Atkinson++02]
Process output: Metrics
Measures used to indicate progress or achievement [Leon05]
Types: control (e.g. avg. time for activity) or predictor (e.g. code) metrics
Benefits
Objective comparison
to make judgements about software qualities
to access the effectiveness of processes, tools, and methods
Quality Assurance
Inspections TestingMeasure-ment …
© Fraunhofer IESE
39
Project Goal
Question
MeasurementGoal
Metric Raw Data
Data Collection Procedure
Results
Project Results
Interpreted ResultsValidation
0,7
3,1415926
π42
0,7
3,1415926
π42
GQ
MGQM Plan
MeasurementPlan
MeasurementResults
Measurement –Goal-Question-Metric (GQM) Paradigm [vanSolingen++02]
© Fraunhofer IESE
40
Measurement –GQM Characteristics
Most popular mechanism for goal-oriented measurement
Process steps:
1. Formulate goals
Output: Hierarchy of goals and sub-goals
2. Refine goals to questions
Questions answer to what extent the goal has been reached
Output: List of questions
3. Quantify questions by metrics
Output: metrics suite
© Fraunhofer IESE
41
Product Line Measurement –An example (1/2)
Task: evaluating product line code complexity in FE (development/evolution)
1.Sustainable PL Code Evolution
2. PL Code Complexity Adaptation
3. Size Reduction
4. Shape Alignment
5.Variability Emphasis
6. VarMgtConsistency
7. Reuse Efficiency
1. Formulation of goals
Output: goal hierarchy of 7 goals
software developerfrom the viewpoint of the
variable partswith respect to
emphasizingfor the purpose of
code of software product linesAnalyze the
Q14: How many variable parts are visible at the module level? (How many should be?)Q15: How many variable parts are visible module-internally? (How many should be?)Q16: How many variable parts are indistinguishable from common code?
2. Refinement of goalsto questions
Output: 23 questionsexcerpt for goal 5:
5.Variability Emphasis
© Fraunhofer IESE
42
Product Line Measurement –An example (2/2)
3. Refinement of questions to metrics
Output: metrics suite of 21 PL complexity metrics
excerpt for goal 5 / questions 14-16:
G Q Metric name Description
5 Variability emphasis
14 NVPrte Number of externally visible variable parts
15 NVPrti Number of internally visible variable parts
16 NVPrta Number of ambiguous variable parts
Other PL metrics:reuse ratio# of variation
points# of defaults
code churn among alternativesdepth of reuse hierarchywidth of reuse hierarchy
© Fraunhofer IESE
43
TestingHow can we test product line assets?
© Fraunhofer IESE
44
Testing –Characteristics
Characteristics [IEEE829-2008]
Process of uncovering evidence of defects in software systems
Necessary part of any quality assurance process
Defect (here: general term used for all types of quality deficits)
Non-fulfillment of a requirement, related to an intended or specified use [ISO9000:2005]
Other defect types [Jalote05]
Error: Discrepancy between a computed, observed, or measured value and the true, specified, or theoretically correct value
Failure: Inability of a system or component to perform its required functions according to its specifications
Fault: Condition that causes a system to fail in performing its required function
Quality Assurance
Inspections TestingMeasure-ment …
© Fraunhofer IESE
45
Test ProcessTraditional (V-Model)
RequirementsEngineering
Requirements
Architecture
Code UnitTests
IntegrationTests
SystemTests
Test LevelsTest Items
IntegrationTesting
SystemTesting
Design
Implemen-tation
DevelopmentProcesses
TestProcesses
Legend
activitywork productproduct flowcontrol flowv&v 1 flow
1 verification & validation(Verific.: Are we building the product right? Valid.: Are we building the right product?)
Unit Testing
© Fraunhofer IESE
46
TestingConcepts
Test Level Separate test effort that has its own documentation and resources [IEEE829-2008]
Test ItemSoftware or system item that is an object of testing [IEEE829-2008]
Test Case Set of test inputs, execution conditions, and expected results developed for a particular objective [IEEE610-1990]
Test Oracle Mechanism, different from the program itself, that can be used to check the correctness of the output of the program for the test cases. The output of the [program/software under testing and the test oracle] is compared to determine if the program behaved correctly for the test cases [Jalote05]
© Fraunhofer IESE
47
Test Process
TestingInputs, Process, and Output
Test Inputs Test Output
TestCases
TestItems
TestOracle
Comparator
TestResults
© Fraunhofer IESE
48
TestingProduct line-specific process issues
Application testing (testing in AE)is similar to single systems testing
because it only deals with the functionality of single products
Differences lie in
Family testing (testing in FE)
Must consider product line issues => variability
Product line test assets
Contain variability
Do not necessarily have to deal with functionality (execution time behaviorof complete components), but may also deal with construction issues(of incomplete system parts)
© Fraunhofer IESE
49
Product Line TestingConcepts
Application Testing
Sub-process of application engineering where family test assets are reused to uncover the evidence of defects in the application
Family Testing
Sub-process of family engineering where the existence of defects in product line development assets is uncovered and where reusable test assets are created
Test Assets
Inputs and outputs of the family test process containing plans, specifications, test cases, test oracles, and test results
Are reused in application testing
© Fraunhofer IESE
50
Product Line TestingTwo Extreme Approaches [Pohl++05]
Brute Force Strategy
Idea: Perform all test activities for all possible products in family engineering, none in application engineering
Disadvantages
large amount of test assets
overhead
Pure Application Strategy
Idea: Perform all test activities for all possible products in application engineering, none in domain engineering
Disadvantages
no advantages over single systems testing
overhead
AE RE Des. Imp.
FE RE Des. Imp.Testing
Testing
AE RE Des. Imp.
FE RE Des. Imp.Testing
Testing
© Fraunhofer IESE
51
Product Line TestingViable Approaches
Sample Application Strategy [Pohl++05]
Idea: Use one or a few sample applications to test the core assets. Application testing is still needed for each application
Pros&cons: low learning effort, but overhead
Commonality & Reuse Strategy [Pohl++05]
Idea: Test common parts and prepare test assets for variable parts in family testing. Reuse both kinds of test assets for testing individual products in application testing
Pros&cons: low loverhead, but learning effort
Construction Testing [IESE]
Idea: Do not test the functionality of test items (“execution testing”), but the way they are constructed and compiled. A test succeeds if the test item compiles
Pros&cons: applicable in parallel with all other test approaches, low learning effort, low overhead, applicable to non-executable test items, behavior untested
[1] Pohl++05: Software Product Line Engineering
© Fraunhofer IESE
52Execution Testing
Construction Testing
Product Line TestingGeneral Process
ExecutionTest Cases
ExecutionTest Oracle
ExecutionComparator
ExecutionTest Results
ConstructionTest Cases
ConstructionTest Oracle
ConstructionComparator
ConstructionTest Results
PL TestItems
© Fraunhofer IESE
53
Summary
© Fraunhofer IESE
54
Summary
Quality assurance is a sub-process of quality management
Quality assurance techniques are inspections, measurement, testing, etc.
Software testing is a necessary part of quality assurance, uncovering theevidence of defects in software systems
Quality assurance and testing exist for all development phases, not just for
the implementation phase
Inputs of test processes are test cases, test oracles and test items, theoutput is test results
Product line testing differs from single systems testing by testing activities
in family engineering, and test assets that may contain variability
Viable test approaches are execution tests (Sample Application Strategyxor Commonality & Reuse Strategy), and construction tests (which
succeedif the test item can be constructed/preprocessed/compiled)
© Fraunhofer IESE
55
References
[ISO9000:2005] European Standard – Quality Management Systems
[IEEE829-2008] Standard for Software and System Test Documentation
[IEEE12207-2008]Standard for Software Life Cycle Processes
[Jalote05] P. Jalote: An Integrated Approach to Software Engineering
[IEEE610-1990] Standard Glossary of Software Engineering Terminology
[Pohl++05] Software Product Line Engineering, Springer-Verlag
[vanSolingen++02] GQM Approach, Encyclopedia of SE