December 3, 2014 Sam Siewert

SE420

Software Quality Assurance

Lecture 12 – Final Review

SQA FIRST HALF

MSQA Chapters 1-6, Notes Through Week 6

Sam Siewert 2

Coverage – As Noted on SyllabusMurali Chapters 1 to 5– Chapter 1 – Quality Gurus, SQA Department or Integrated Process

and Policies (Dijkstra, Demming, Boehm)

– Chapter 2 – Dimensions of Quality (Specification, Design, Development, Conformance)

– Chapter 3 – Black-box, White-box

– Chapter 4 – Role of SQA Department

– Chapter 5 – Verification (Validation), Walkthroughs, Inspections

Validation Definition and Test Scenarios in Chapter 6

In-Class Notes and Assignments– Waterfall, Evolutionary Spiral and Extreme Programming

– Boehm’s V

– Agile Manifesto and Evolutionary Process

– Unit Tests, Drivers, and Black-box / White-box Tools & Methods

– Acceptance Testing

– Integration & Test – Unit/Component, Sub-system, System, System of Systems

Sam Siewert 3

Knowledge and Concepts DayTrue / False, Multiple Choice, Short Answer

Concepts, Terminology, and Understanding of SQA Methods of Validation and Verification

Focus on Boehm’s V – Unit Level and Acceptance

Knowledge of I&T and System Test Concepts Only

Sam Siewert 4

What, How, Why, When, Who

What, Why, When

Day 1 & 2

Day 1

Test TypesAcceptance Tests [Subset of Unit, I&T, System Tests]

– Test as Product Will Be Used

– Out of Box Experience

– Pre-Ship Test

Unit Tests1. Function and Feature

2. Performance Tests [Operations / second]

3. Error Handling and Recovery [Failed API calls, Bad Return Codes]

4. Unit Reliability Stress Tests, Soak Tests [Overnight, Memory Leak, Resource Exhaustion]

5. Corner and Coverage [Initial Conditions, Input Extremes, Pattern]

Regression Tests (Code checkout, Build, Tests at All Levels)– Verify Code Fixes Remove Defect(s)

– Verify that Code Changes Do Not Introduce New Bugs or Anomalies

Integration and System Tests1. Diagnostics [Built-in Self-Test]

2. Ease of Use and Provisioning

3. Interoperability

4. Upgrades

5. Sub-system and System Resource Testing

– Unit Tests Adapted for Sub-System and System

E.g. Features, Performance, Error Handling

E.g. Reliability Stress & Soak Integrated

– Reliability with External Fault Injection – On Interfaces and at Hardware/Software Interface

• E.g. Cable Pull on Disk Drive for HW Fault

• E.g. Data Corruption on Buffer for Function Call Interface

Sam Siewert 5

Theory, Implementation, Methods

Black-box Unit Level

– Test Cases in Test Driver

– Code Under Test in Separate Modules

– Modify Code Under Test as Little as Possible

– Use of Gcov and Lcov

– Exit Criteria

White-box Unit Level

– Basic Blocks

– Disassembly Methods (-S option for GCC)

– DDD Single Step Debugger

– Software in Circuit Tools – Software Debugger, Gcov

– Hardware in Circuit Tools - JTAG

Sam Siewert 6

Familiarity with Test Scenarios

RAID 10, 50– XOR Encoding

– Recovery of Lost Disk Drive (File or Segment) with XOR of Remaining

– Normal Write and Read Operations

– Read with Recovery on Fly [Degraded Mode]

– Write During Degraded Mode

– Lost Segment Rebuild

Encryption and Decryption of Messages– Simple Substitution

– Transposition [Block Cypher]

– Mathematical Hashing Function for Substitution or Transposition

– Combinations of Methods

– Reverse Operation to Decrypt

Image Processing Transformation– Load Image File to Buffer

– Apply Pixel Operations to Buffer [X, Y, Channel]

– Write Buffer to Image File

– Preview Display to User

Sam Siewert 7

Walkthrough vs. Inspection

What is the Difference? [Informal, Formal Gate]

Why are Both Done? [Validation, Verification, Standards]

When is Each Done Ideally? [Phase of Lifecycle]

Who is involved in Each? [Peer Engineers,

Management, SQA]

Where is Each Done? [Office one-on-one,

Meeting/Conference Room with Committee]

Sam Siewert 8

SQA SECOND HALF

Scan MSQA Chapters 7-9, Notes Through Week 14

Sam Siewert 9

Integration and TestIntegrate Software Modules [units] and Hardware Components into Sub-systems

Test Focus on Interfaces [Function, Message, Shared Memory, Hardware], Protocols, and Interoperability of Modules

Sam Siewert 10

Test TypesPositive Tests– Functional Software Interface Tests

Functions calling Functions – API

Message Passing – Local Message Queues, Network, Client-Server

Shared Memory – Synchronization, Buffers

– Hardware Interface TestsDrivers and Device Interfaces

Firmware [ROM Code, Run out of Reset]

Negative Tests– Software Interface Faults

– Hardware Interface Fault Injection

Diagnostics [Built-in Self-Test]

Unit Interoperability– Sub-system Resource Testing – Memory, CPU, I/O, Storage, Power

– Protocols – Message Acknowledgement, Command/Response, Background Commands, Peer-to-Peer, etc.

Sam Siewert 11

Outline for Every Integration Test1. Check out Specific Source Code Test Configuration – CMVC Tools, Git

– Collection of Modules [Units] Tagged by Revision Control

– OR Current

2. Build and Link Modules (*.o) and Libraries (*.a) into Sub-system to Test

3. Load / Install Sub-system Code onto Test Hardware Platform of Known Configuration

– Record key hardware configuration parameters

– E.g. for I/O HW config - lspci, lsusb,

– General config - hwinfo

– Linux OS kernel build config - uname –a

– cat /proc/meminfo

– cat /proc/cpuinfo

4. Run Integrated Test(s) [with Gcov, Lcov, Gprof]

5. Review of Expected Syslogs, Output to Terminal, for Each Feature

6. Review Performance Profiles

7. Track Bugs, Anomalies, and Disposition as Defects

Sam Siewert 12

Negative Tests – Fault Injection

Hardware Layer

– E.g. Cable Pull for Storage [SAS, SATA]

– Cable Pull for Networking [Ethernet]

– Power Loss

– Analog Circuit Methods [Over/Under Voltage, Short, Open, Mis-

route]

Software Layer

– Buffer Corruption [Shared Memory Corrupt, Message Loss,

Corrupt, L from Test Client]

– Network Layer 2, 3, 4 Packet Corruption - Wireshark

– Bad Command Generation [Test Client]

– Bad Pointer [Buffer]

Sam Siewert 13

Fault Injection Methods

(IP Network Layer Forwarding)

Sam Siewert 14

Sockets API

MPEG Program

VLC Stream Service

UDP

IP

IEEE 802.3z - GigE

Cat-6 UTP

Sockets API

MPEG Program

VLC Player

UDP

IP

802.11abgn - Wireless

Radio Transmission

L5

L6

L7

L4

L3

L2

L1

IP Forwarding

[Enable/Disable]

802.3z -

GigE

Cat-6 UTP

802.11abgn -

Wireless

Radio

Transmission

Fault Injection Methods

(Application Layer Message

Forwarding)

Sam Siewert 15

Sockets API

MPEG Program

VLC Stream Service

UDP

IP

IEEE 802.3z - GigE

Cat-6 UTP

Sockets API

MPEG Program

VLC Player

UDP

IP

802.11abgn - Wireless

Radio Transmission

L5

L6

L7

L4

L3

L2

L1

Sockets API

MPEG Program

Store and Forward

[Corrupt, Delay, Drop]

UDP

IP

802.3z -

GigE

Cat-6 UTP

802.11abgn -

Wireless

Radio

Transmission

Root-Cause Analysis

Field Issue - Anomaly, Reported Bug, Data Corruption,

L

– Software Defect?

– Hardware Reliability

– User Error

Reproducibility – Capture Conditions via Logging

– Recreate Scenario in SQA / QA Lab

Trace to Root-Cause– Assert

– Analysis Triggers

– Propose Fixes

– Apply and Regression Test

– Release Maintenance Patch

Sam Siewert 16

Case Study – Mars Pathfinder Story

JPL Mission Flow to Mars, Landing on July 4th, 1997

Pathfinder Rolling Resets on Final Approach to Mars

Capture Orbit

VxWorks RTOS Used

Reproduction of Anomaly on the Ground

Root-Cause Analysis

Proposed Fix

Sam Siewert 17

Bug Open/Close Rates and ReadinessControversy – Bug Counts, Closure and Prediction of Phase Transition Readiness– E.g. Unit to I&T to System Test to Acceptance Test to Shipment

– Can Be inaccurate due to Unsatisfactory Testing or Lack of Criteria

– Guideline for Project Management [Compared to Guessing!]

– Not all Reported Bugs Become Defects [Test Case Errors, Human Error]

Sam Siewert 18

http://www.testandverification.com/DVClub/24_Jan_2011/Greg_Smith.pdf

Test C

ase C

overa

ge

[E.g

. C

ode P

ath

Covera

ge]

Bug C

ounts

[Report

ed,

Not

Verified a

s D

efe

ct]

I&T Summary

Integration of Modules [Units]

Configuration of Sub-systems and Systems

Sub-systems built from Modules [units] and Components

Positive Tests

Negative Tests

Interoperability of Modules and Components

I&T Precedes System Test, Where Sub-systems are L– Fully Integrated

– Configured Similar to Deployment [Perhaps Not Exact – E.g. Spacecraft in Thermal-Vac Testing]

– Stimulated with Tests Replicating Operations

Sam Siewert 19

System TestIntegrate Sub-systems into System Configuration

Test Focus on End-to-End, Black-box Testing and Architecture

Validation, System Requirements Specification

Sam Siewert 20

System Test Types – Extends I&T

Emphasis on Full Configuration End-to-End Testing

Usability and HCI Testing – GUI, CLI, API

Performance Tests – Profiles and Traces

Workload Testing – Normal, Peak

Stress Testing – Accelerated Lifetime

Security Testing

Scalability Testing – Volume [Capacity, Nodes, Size]

Nightly Build and Smoke Test [Regression]

Installation Testing – Out of Box Experience

High Availability Recovery Testing

Mission Critical Fail Safe and FDIR Testing

Compatibility and Interoperability Testing

Sam Siewert 21

Software End-to-End System

Sam Siewert 22

Software Units

Flight Computer [DATA]1. Low Rate Downlink, Uplink Tasks

2. Flight Command/Response [FRTE]

3. Device I/O

Instruments [SXEE, FARUS, SXEE]– Recycled from LASP, Integrated Firmware

Ground Support System [GSFC ACCESS]– Low Rate Downlink/Uplink Interface

– Ground Real-Time Engine

– Selective Telemetry Monitor

– Spacecraft Command Language [Scripts, Rules, Constraints]

– Data I/O Server [Kerberos]

– OODBM Telemetry Database

– Operator GUI

Payload Operations Center [CU-Boulder]– Data I/O Client [Kerberos]

– DATA-CHASER Planning and Scheduling

– Payload Operations GUIs

Remote Science and Engineering [JPL]– Remote Engineering GUIs

Sam Siewert 23

Regression Testing and Test

AutomationRe-test Units After Defects are Fixed

Re-run I&T and System Tests

Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results

Sam Siewert 24

Regre

ssio

n T

esting

(Test A

uto

mation)

General V&V Automation

Many Tools Exist

DejaGnu – a short

introduction

Go Beyond Batch

Test and Visual

Inspection

Expect / DejaGnu –

Automates V&V

Reports in Addition

to Running Drivers

Sam Siewert 25

Batch is Execution

Automation Only

To Run Drivers in

A Sequence

Test Automation and SEI CMMCoverage Criteria, Bug Open/Close Rate Metrics, Pass/Fail Statistics Provide Quantitative Management [Level 4]

Automation is a Step Toward Optimization – Necessary, but Not Sufficient (Metrics for Feedback Process Improvement)

Sam Siewert 26

Ad Hoc, “Code & Fix” – Driven by

Individual Styles, Un-coordinated,

More of an Art than Practice

Clear Phases with Key

Deliverables – Analysis,

Design, Units, Test Plans

E.g. Boehm’s “V” and

Spiral Model, Applied to

All Projects - Repeatable

Entry/Exit Criteria for

Phases, Metrics Drive

Decisions in Process [Gates]

Feedback Used to

Improve Process with

Automation and Innovation

SEI CMM Levels

http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm

Cyberphysical Systems

Sam Siewert 27

Infamous System Software Defects

Infamous & Famous Field Defects

1. Toyota ABS Software Recall, 2010

2. Windows Genuine Advantage Outage, 2006

3. Mars Climate Orbiter Units Interoperability, Loss, 1999

4. Mars Pathfinder Priority Inversion, Recovered, 1997

5. Ariane-5 501 Cluster Launch Software Failure, 1996

6. Pentium FPU Bug – Software Acceleration Co-Processor Recall, 1994

7. ATT 4ESS Upgrade, Outage, 1990

8. Therac-25 Radiation Therapy Patient Overdosing, 1985-86

9. NORAD False Alarms, User Error for Test Mode, Device Failures, 1979/1980

10. Apollo 11 Guidance Computer Overload, 1969 - Recovered

Sam Siewert 28

http://en.wikipedia.org/wiki/Software_bug

RCA, Patch, Test, Fix!

http://history.nasa.gov/SP-350/ch-11-4.html

Rear Admiral Grace Hopper

Week 14 & 15

Week 14 – Code Walkthrough Practice in Class

Week 15 – Exam #2

Sam Siewert 29