fitman webinar 2015 06 verification and validation methodology

116/06/2015 FITMAN Webinars, Online

FITMAN Verification & Validation Methodology

FITMAN WebinarsJune 16th, 2015

Dr. Fenareti LampathakiNational Technical University of Athens (NTUA)

What is Verification & Validation?

Verification Validation

Definition [IEEE Std 1012-2004]

The process of providing objective evidencethat the software and its associated productsconform to requirements (e.g., forcorrectness, completeness, consistency,accuracy) for all life cycle activities duringeach life cycle process (acquisition, supply,development, operation, and maintenance);satisfy standards, practices, and conventionsduring life cycle processes; and successfullycomplete each life cycle activity and satisfy allthe criteria for initiating succeeding life cycleactivities (e.g., building the softwarecorrectly).

The process of providing evidence that thesoftware and its associated products satisfysystem requirements allocated to software atthe end of each life cycle activity, solve theright problem (e.g., correctly model physicallaws, implement business rules, use theproper system assumptions), and satisfyintended use and user needs.

Question Are we building the product right? Are we building the right product?

Objective To ensure that the product is being builtaccording to the requirements and designspecifications.

To ensure that the product actually meets theuser’s needs, the specifications were correctin the first place and the product fulfils itsintended use when placed in its intendedenvironment.


Bringing together agile and waterfall software

engineering philosophies

FITMAN V&V Method purpose

Infusing a crowd assessment mentality in

the V&V activities

Including both business andtechnical perspectives

Providing initial practical guidelines

An all-inclusive framework for verifying, validating and evaluating a software product from its conception to final release and implementation in real-life

Key message: A general, holistic method including techniques to be applied

for each step and recommendations for its application by the trials

A method reusable beyond FITMAN: • Each trial and development team can streamline the method

according to their own needs


FITMAN WP2 D2.1 4

FITMAN V&V Methodology – Overall Picture

Trial Specific Perspective: Steps T-1 & T-2

Business Validation (T-2) to assess whether the overall trial solutioneventually offers sufficient added value to the trial. Demonstrating that the software developed has clear benefit to the trials, allowing

them to operate more efficiently (usually in terms of cost, time or quality) thanbefore, or supporting them to do something they couldn't do before.

User Acceptance Testing

Simplified ECOGRAI Methodology

Trial Solution Validation (T-1) to guarantee that the overall trial solutionsatisfies intended use and user needs. Validation of each trial solution from a technical and functional point of view only.

5

5 TIs

BPIs


6

Product Specific Perspective: Steps P-(1-5)

Product Validation (P-5) to examine whether theproduct satisfies intended use and user needs.

User Acceptance Testing

Release Verification (P-4) todetermine whether therequirements of the finalproduct release are met.

Backlog Verification (P-3) todetermine whether therequirements of the productafter each sprint are met.

Model Verification (P-2) tocoordinate the alignmentbetween design andrequirements, as well asbetween design and code.

Code Verification (P-1) to ensure functionality,correctness, reliability, and robustness of code.

Alpha Testing

Regression Testing

Traceability Analysis

White Box Testing

Developer Acceptance Survey

7+2 TIs

Test Cases Documentation


Assessing Business Impact

Business Performance Indicators (BPI) are identified for each trial through a simplified ECOGRAI Method.

Characteristics of a Business Performance Indicator (BPI):

• easy to be interpreted, to put in work, to use or to exploit

• easily measurable, quantifiable

• representative of the objective which it measures

For each Business Performance Indicator, the trials are required to report:

• The current value (AS-IS)

• The target value they want to achieve (Target) and

• The actual achieved values after the solution implementation (TO-BE)


Initial Technical Indicators (filled in by trials)

8

Fulfilment of requirements

“The solution fulfils the Trial requirements”

Learnability “It is easy to start to use the

solution and learn functionalities”

Understandability “The solution is easy and self-clear to understand and the

concepts and terminology are understandable”

User’s attraction level “The solution is attractive to the

user. I feel satisfied and comfortable when using it”

Efficiency “The time and resources required

to achieve the objectives of the solution are reasonable, the

solution is fast enough and does not require too many steps”

Openness “Ensuring that specific people

groups may access the software for free with specified rights”

Interoperability maturity

“The capability of the software to interact with other systems”

Ease of application “A measure of the applicability of

the software in the particular environment in terms of amount

of work and extra actions or means”

Spe

cifi

c En

able

rsTr

ials

So

luti

on

• Strongly agree• Agree• Neither agree

nor disagree• Disagree• Strongly

disagree

Dif

fere

nt

Leve

ls p

er T

I


Additional Technical Indicators

Technical Indicator

Definition Value (in 3 levels)

Correctness The degree to which the software is free from defects in its specification and implementation

High (no defects detected) / Medium (relatively minor defects detected) / Low (severe defects detected)

Efficiency The capability of the software to provide appropriate performance, relative to the amount of resources used

High (performance beyond expectations, with reasonable resources consumption) / Medium (expected performance,even with small increase of resources utilization) / Low(Marginally accepted performance and/or overuse of available resources)

Reliability The capability of the software to maintain a specified level of performance when used in the factory settings

High (totally stable performance) / Medium (marginal performance divergences) / Low (the performance of the software varies significantly depending on the conditions)

Sustainability How easy it is to maintain the software and apply required modifications for adapting to changes in the operating environment

High (software maintenance and modification are very easy) / Medium (software maintenance requires reasonable effort) / Low (software maintenance is very complex and modifications difficult to be applied)

Additional calculated indicators (not to be assessed by trials nor inserted in SurveyMonkey):

Versatility: to be estimated according to the formulas provided in WP2

Portability: to be estimated on a cross-trial basis using existing indicators

Existing indicators (already assessed for GEs/SEs): Openness / Ease of application / Interoperability maturity

Trial Journals

10

Unstructured Information

Unstructured information for the experience of each Trial is collected.

Technical Journal:

Registration of the implementation issues encountered in the implementation of the Trial systemRegistration of the operational resilience of the Trial (e.g. major bugs, blocking errors, etc.)

Business Journal:

Collection and analysis of the most important operational issues faced in the implementation of the system in the Trial, e.g. organizational and business difficulties, degradation of the business system.


Overall V&V in practice

11

Developer Acceptance

QuestionnairesValidation of SEs by anyone who reuses them

4+


An example of Self-certification for SEs


An Example for “Release Verification” documentation


An example for “Product Validation” documentation


Lessons Learnt so far…

• Experience in FITMAN was collected from trial owners and technical partners:

– The users mainly considered the method sufficient

– Training, support and the online V&V tool are crucial

– Definition of BPIs was not trivial, but it was worth the effort

• Challenges with the definition of objectives and the BPIs, initially not well formulated / measurable

• Confidentiality issues

• V&V from technical and business perspective must be an ongoing and iterative process…


FITMAN V&V Decalogue

I. Have a deep knowledge of the requirements and the ends sought in each trial.

II. Infuse a V&V culture to your trial.

III. Create a concrete V&V plan to be followed in each step, tailored to your real needs and competences.

IV. Engage from the very beginning the required stakeholders, assigning them with appropriate responsibilities and decision power.

V. Tap the power of data to effectively conduct V&V activities.

VI. Proceed with each V&V step on the basis of at least 1 recommended technique.

VII. Keep the balance between business and technical V&V aspects.

VIII. Invest on crowd-sourcing techniques aligned to the trial philosophy.

IX. Keep a complete V&V log and document in detail the V&V findings.

X. Treat V&V as a continuous, iterative procedure.


Thanks for your attention!

Dr. Fenareti Lampathaki (NTUA)

Join Us!

17

fenareti.lampathaki @[email protected]


fitman webinar 2015 06 verification and validation methodology

Science

software product

product validation p

right product

intended use

user needs

life cycle activity

overall trial solution

business validation