pendar webinar 1 with notes
TRANSCRIPT
© Predictable Network Solutions Ltd 2017 www.pnsol.com
PEnDAR Focus Group
Performance Ensurance by Design, Assuring RequirementsFirst Webinar
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 2
Goals• Consider how to enable Validation & Verification of
cost and performance • For distributed and hierarchical systems • via sophisticated but easy-to-use tools• Supporting both initial and ongoing incremental
development
• Provide early visibility of cost/performance hazards• Avoiding costly failures• Maximising the chances of successful in-budget delivery
of acceptable end-user outcomes
Partners
Supported by:
PEnDAR project
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 3
Our offer to youLearn what we know about the laws of the distributed world:• Where the realm of the possible lies;
• How to push the envelope in a way that benefits you most;
• How to innovate without getting your fingers burnt.
Get an insight into the future:• Understand more about achieving
sustainability and viability;• See where competition may lie in the future.
Our ‘ask’Help us to find out:• In which markets/sectors/application
domains will the benefits outweigh the costs?• Which are the major benefits?
• Which are just ‘nice to have’?
• How could this methodology be consumed?• Tools?• Services?
Role of the focus group
© Predictable Network Solutions Ltd 2017 www.pnsol.com
The system delivery challenge
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
5
System delivery
challengeSystem
Requirements
Tim
e pr
essu
reCom
plexity
Cost/resource constraints
Forces hierarchical decomposition
Forces parallel development
Change during development
Subcontracted subsystems
Use of COTs componentsVague/contradictory
Leave ‘hard’ problems to the end
Existing asset re-use constraints
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 6
How to specify and verify subsystem
requirements/acceptance criteria
Functional outcomes
Performance outcomes
Resource constraints
MEE
T
Risks
MANAGEPeak Loads
Integration testing
Rework
MIN
IMISE
Post-deployment scaling
System delivery challenge
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
7
Constrained performance hazards
Unconstrained performance
hazards
Distributed
Centralised
Dedicated SharedResources
Stru
ctur
e
Traditional avionics
Cloud apps
Traditional mainframe
Standalone microcontrollers
Modern avionics
Client-server systems
Established approaches running out of steam?
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Dimensions of the challenge
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
9
Out
com
es
Deliv
ered
Resources Consum
ed
Individual impact
Variability
Exception/failure
Syste
m im
pactSelf-induced
Externally created
Indi
vidua
l impa
ct Mitigation
Correlation
System impact
Propagation
Scale
Schedulability
CapacityDistance
Number
Tim
e
Space Density
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
10
Out
com
es
Deliv
ered
Resources Consum
edVariability
Exception/failure
Scale
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
11
Out
com
es
Deliv
ered
Resources Consum
edVariability
Exception/failure
Scale
Distance
Number
Tim
e
SpaceSchedulability
Capacity
Density
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
12
Out
com
es
Deliv
ered
Resources Consum
edVariability
Exception/failure
Mitigation
Propagation
Scale
Schedulability
CapacityDistance
Number
Tim
e
Space Density
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
13
Out
com
es
Deliv
ered
Resources Consum
edVariability
Exception/failure
Self-induced
Externally created
Mitigation
Correlation
Propagation
Scale
Schedulability
CapacityDistance
Number
Tim
e
Space Density
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
14
Out
com
es
Deliv
ered
Resources Consum
edVariability
Exception/failure
Self-induced
Externally created
Individual impactIn
divid
ual im
pact Mitigation
Correlation
Syste
m im
pact
System impact
Propagation
Scale
Schedulability
CapacityDistance
Number
Tim
e
Space Density
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
15
Out
com
es
Deliv
ered
Resources Consum
ed
Individual impact
Variability
Exception/failure
Syste
m im
pactSelf-induced
Externally created
Indi
vidua
l impa
ct Mitigation
Correlation
System impact
Propagation
Scale
Schedulability
CapacityDistance
Number
Tim
e
Space Density
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Practical methodology
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 17
Support stages of the SDLC• Design
• Feasibility analysis• Hierarchical decomposition• Subsystem acceptance criteria
• Verification• Checking delivery of quantified outcomes• Evaluating resource usage• Re-verification during system lifetime
• Validation• Quantification of performance criteria• Checking coverage and consistency
Quantify hazards• Failure to meet outcome requirements
• Physical constraints• Schedulability constraints• Supply chain constraints
• Failure to meet resource constraints• Scaling• Correlations
Interaction with System Development Life Cycle
© Predictable Network Solutions Ltd 2017
18
www.pnsol.com
Quantifying intent
• The key challenge is to establish quantified intentions• For outcomes/resources/costs
• Then a variety of mathematical techniques can be applied• queuing theory• large deviation theory• ∆Q algebra
• This is “only rocket science”• Not brain surgery!
© Predictable Network Solutions Ltd 2017
19
www.pnsol.com
POLL
Imagine two technical scenarios:• In the first, a client asks for “a replacement for its phone system”.• In the second, a client asks for “a VoIP system with an average MOS of
4.2, the ability to take 30 concurrent calls, a failover system for when it hits capacity (e.g. voicemail/recorded message), first priority for emergency calls and second priority for current calls”.
If the first scenario is 1 and the second is 5, where on a scale of 1 to 5 would you place the requirements you generally encounter in your field?
© Predictable Network Solutions Ltd 2017
20
www.pnsol.com20
Quantifying timelinessOutcome requirement:Suppose we could get the marketing department to specify how long it’s acceptable to wait for the first frame of a video:• 50% of responses within 3 seconds• 95% of responses within 10 seconds• 99.9% of responses within 15s• 0.1% failure rate
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 21
• Suppose the black line shows the delivered CDF• From measurement, simulation or
analysis
• This is everywhere above and to the left of the requirement curve• This means that the timeliness
requirement is satisfied• If not, there is a performance
hazard
Meeting a timeliness requirement
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Questionnaire analysisWho are you and what do you think?
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
23
012345
Management System Design / Architect
Capability development
/ Research
Marketing / Sales
0
1
2
3
4
0
2
4
6
8
What field do you work in?
What is your role/what level do you work at?
Which parts of the product lifecycle are you involved in?
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
24Where do system/product requirements come from?
1 2 3 4 50
1
2
3
4Precise and unambiguous?
1 2 3 4 50
1
2
3
4 Quantifiable and quantified?
0
2
4
6
© Predictable Network Solutions Ltd 2017© Predictable Network Solutions Ltd 2017 www.pnsol.com
25How / when does your organisation determine whether requirements have been / will be met?
0246
What is the typical response to the discovery that requirements will not be / have not been met?
0
1
2
3
© Predictable Network Solutions Ltd 2017
26
www.pnsol.com
Importance of performance and resources
For the systems / products you produce / operate, how critical is performance to success?
4.4
To what extent is your role accountable for aspects of performance that are critical for success?
3.2
To what extent do you, in your role, have influence/control over how resources are shared?
2.8
Given that resources are shared, other parts of the system / other systems may consume resources on which you rely: to what extent is this a concern?
4.0
Given that resources are shared, your system / product may consume resources on which others rely: to what extent is this a concern?
3.8
My role is primarily concerned with the performance impact on the individual users / uses
My role is concerned with both the overall performance and the impact on individual users / uses
System performance is not a concern of my role
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Summing up
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Click to edit Master title style 28
Benefits• Avoid infeasible developments
• 'fail early’• prune blind alleys
• Address scalability early• including real-world constraints• avoid 'heavy tail’
• See the whole risk landscape• not just the 'first problem’
• Be able to write a safety case• Even for shared-resource distributed systems
Costs• Have to ‘quantify intent’
• may meet resistance
• Effort in adopting new tools and techniques• Upfront work needed before “real”
development starts• may not fit expectations/metrics of
‘progress’
Expected benefits and costs of this approach
© Predictable Network Solutions Ltd 2017 www.pnsol.com
Thank you!