1022005 maplddesign integrity concepts what’s the exit strategy? verifying a design

12005 MAPLD Design Integrity Concepts

What’s the Exit Strategy?Verifying a Design


Agenda – Design Verification

• Concepts and implications

• The specification connection

• Verification before test

• Test thoroughness

• Tiered verification

• Concluding the process

• Summary


Verification Concepts• Verification – Confirmation, through the provision of

objective evidence, that the specified requirements have been fulfilled (Q9000-2000)

• Confirmation – the act of ratifying or corroborating (AH)


Verification Implications• The verification process is not intended to be a “craps

shoot”– Verification is primarily intended to ratify correctness– Verification is NOT primarily intended to reveal incorrectness

• The mindset is important!– Doubt that a design will work expresses a lack of confidence in the

designer and the design process– Waiting until “verification” to guarantee correctness is an excuse

for being sloppy

• We should expect our designs to work! Verification simply puts the seal of confirmation on the expectation


Verification Implications (cont.)

• Verification addresses two processes– Design

• Primarily a one-time, iterative process in which the developed concept is proven sound

• Fundamental correctness can be proven analytically– Inspection confirms logical correctness in all circumstances

» Peer reviews are a manual form of inspection» Functional simulation is an automated form of inspection» Inspections must be tailored to design

– Analysis verifies correctness in the presence of variability» Reliability (WC, Derating, FMEA, etc.)» Timing over environments and age



• Verification addresses two processes (cont.)– Instantiation

• Particular instance must be sound• Particular correctness must be demonstrated experimentally

– Inspection confirms that instructions are followed» Correct components installed» Correct configuration selected» Correct processes performed

– Test and demonstration confirms that operation matches expectations

» Functionally and parametrically» Predicted by nominal analytical predictions


Verification Implications (cont.)• Verification “after the fact” is too late

– Analysis and test are NOT equivalent– Design analysis and inspection must proceed in lockstep with

design processes– Implementation tests and inspections should simply confirm what

is known

• Design efforts fail because they occur in a vacuum– Creativity takes primacy over correctness– Functionality comes first with operability being “shoehorned”

after the fact • The “monster” simulation syndrome

• Conclusion: Verify as you proceed



• Correctness is a matter of personal integrity for the designer– Verification is not simply externally imposed

task– Verification is something that the designer must

want to do (a part of his/her ethos)– Verification is confirmation that the designer is

“worth his/her salt”


The Specification Connection• “Requirements” are confirmed during verification

– The designer does not have a free hand with respect to how a design is confirmed

– Specification provides the constraint set under which verification is prosecuted

– Therefore, verification must be defined prior to start of design• Not simply in a general manner• Specifically

– Since specification is a customer and vendor document• Both customer and vendor must be involved in the verification

process• “Trust me” is not a reasonable phrase when it comes to verification


The Specification Connection (cont.)

• Specification contains a complete [ideally] set of requirements for the design– Some requirements are very specific

• “All discrete outputs shall have a source impedance of 2 kOhms.• An adequate test is fairly obvious

– Some requirements are not very specific• “The unit shall provide 512 kbytes of SRAM”• Note the implied requirement – The SRAM shall function correctly• What is an adequate functional test for this requirement? (walking 1’s, 0’s

addressing, etc?)– Some requirements lend themselves to quantitative analysis – “The unit

shall provide a .1° C accuracy at end of life”– Some requirements lend themselves to simple inspection – “The unit shall

provide 4 analog output channels”– When do we decide adequacy of the method of verification and the

individual verification cases? – Before we start designing


The Specification Connection (cont.)• Therefore, verification planning must begin with

specification creation– Each requirement must be assigned a method or

methods– Each requirement must be assigned a test or analysis

case• Must answer question – What is an adequate verification?• Must establish answer to question – When are we done?

– Each requirement must be verified at one or more levels [FPGA, board, box, sub-system, system]

– Customer must concur with decision• Note that modern verification databases make this

relatively straightforward



• Advantages to this type of verification planning– More verifiable / verified designs

• Up-front focus on programmatically difficult verification can be instituted

– Earlier analysis

– Simplification of overly complex circuit designs

• Guidance for lower-level verification efforts can be established– Sub-module vs. module level

– Module vs. unit level

• Integral self-test provisions can be included in design

• Earlier simulation vector definition (FPGAs / logic)



• Advantages (cont.)– More robust test sets

• Early knowledge regarding end-item function communicated

• Capability of test set matched to need– Precision (timing, voltage, current, etc.)

– Speed

– Test level (component, unit, system)

• Test flow design considerations included in test set design

– Knowing verification requirements early makes the entire development process more efficient



• Defining requirements and test cases at the same time as the specification clarifies and simplifies the FPGA verification process– Allows early start to simulation vector design– Creates early knowledge of additional functional

models (SRAM, peripherals, etc.) needed– Clarifies decision regarding what can be functionally

simulated and what must be simulated with back-annotation

– Defines levels at which simulations must be run


Verification Before Test• It is a relatively bad thing to discover a design flaw during

test (better than nothing)– Late in the development cycle– Correction is more complicated / expensive– Personal stress is higher

• Yet … a surprising lack of verification occurs early• Types of pre-test verification

– Inspection - conformity evaluation by observation and judgment accompanied, as appropriate by measurement or testing (ISO)

• Personal self-check• Peer Review• Basic functional verification (Signal audits, functional simulation,

etc.)


Verification Before Test (cont.)• Types of pre-test verification (cont.)

– Analysis – Conformity evaluation of the predictions produced by realistic models of the system

• Worst case analysis (voltage, current, frequency, time) using models that incorporate real world degradation of function

• Derating analysis using models that reduce stress• Other

• Note that pre-test verification is fundamentally conceptual– Not tool dependent– Not design dependent– Can be accomplished many ways


Verification Before Test (cont.)• Two approaches to pre-test verification• Verify full design at once (one big peer review, code walkthrough,

analysis, simulation)– Benefits

• Complete design reduces need to develop assumptions• When proved correct, does not need to be repeated• When design is solid, less work• Reduce final documentation for verification

– Drawbacks• Allows small design flaws to propagate for a long time• Complexity reduces visibility (verification more prone to mistakes and

oversights)• When design is flawed, it increases work• Leads to corrections that are global (hard to test effectively and predict side

effects)• May take longer to execute than an aggregate of smaller verification activities


Verification Before Test (cont.)• Two approaches to pre-test verification (cont.)

– Verify incremental designs• Benefits

– Supports development of “known good” sub-designs– Meshes with a partitioned design approach– Facilitates visibility (fewer mistakes, clearer goals)– Allows small flaws to be fixed quickly (minimizes downstream

impact)• Drawbacks

– Is more work in the ideal case (no/few flaws)– Increases documentation if formality is vital

• Either approach can be made to work, but one or the other must be chosen and implemented


Test Thoroughness• Principles for test thoroughness

– Every “shall” in the specification that meets the following criteria must be tested

• Items that may be affected by the instantiation process (subject to fabrication error)

• Items for which the only extant verification is model based • Items for which the testing does not pose unacceptable risk (radiation,

overstress, etc.)– Every instance of a “shall” must be tested

• 18 interfaces require 18 specific tests– Requirements that have an exclusive character

• Must be tested for correct operation • Should be tested at some level for abnormal operation• Example: If you write a 5Ah to something to set it on, you should verify that

writing other values does not set it on– Requirements must be tested over a representative range of conditions


Test Thoroughness (cont.)

• A thorough test is complicated and time consuming– Requires some level of automation to be

comprehensive– Requires some level of compromise to be

practical– Requires team agreement to be acceptable


Test Thoroughness (cont.)• Ensuring thorough, yet practical, tests

– Define which tests require manual interaction and which can be automated

• Output impedance or analog fine precision may need to be manual• Signal level or analog coarse precision may be automatable

– Define which tests must be performed over environmental conditions and which can be performed at room temperature only

• Make decision based on character of measurement• Do not decide based purely on practicality

– Define level of complexity for all measurement sets• Example: 24 analog inputs (3 eight channel muxes)• Test full range of values for 24, or limited range for 7 of 8 mux inputs

and full range for remaining– Define need for repetition of measurement at higher level of

integration (tiered verification)• Example: voltage level / signal quality / clock jitter at board level• Repeat at box level?


Test Thoroughness (cont.)• Test thoroughness issues affect everything

– Customer relationship– Test set design (board, box, system)– Schedule / cost

• An adequate test program is not trivial– Cannot wait until system level– Must incorporate lowest level of design– Must be a constant background activity during design process

• All programs must balance perfect and good enough• A good test program

– Will ensure that the specification is met– Will verify everything that can be affected by fabrication– Will build on the analysis and inspection effort– Will maximize automation without sacrificing test accuracy


Tiered Verification• Tiered verification is the process of matching the correct type of

verification to the appropriate level of integration• Tiered verification incorporates all types of verification (before and

during testing)• Tiered verification applied to the test regime attempts to have

thoroughness with practicality– Test a requirement at the lowest conceivable level– Determine which tests must be repeated at higher levels by

• Establishing the purpose of a test at a particular level• Determining whether the next level has the potential to change previous

measured quantities• Determining what test set capabilities are

• Tiered verification cannot be retrofitted into the process– Must be planned up front– Must be implemented throughout the development process


Tiered Verification - Example• Need: A set of 16 high-level pulsed discretes to control

latching relays used to change the spacecraft power configuration

• Basic Requirements– Quantity: 16– Level: +30V +/- 2 V– Load: 2 kOhm, 4 mH– Duration of pulse: 50 ms– Rise / Fall time max: 1 ms– Command capability: Unique bit pattern, only one at a time,

arm first– S/W requirements: Various (beyond scope of

example)


Tiered Verification – Example (cont.)

• Tier 1 – Basic Verification (functional simulation)– Verify each channel fires for a specific bit pattern /

address

– Verify cannot be fired without an arm command

– Verify that other complementary patterns don’t accidentally fire the pulse

– Analyze drive capability of FPGA and input / output requirement / capability of translation chip

– Evaluate glitch hardness of logic selected



• Tier 2 – Board level– Test to confirm signal level between FPGA and

driver chips is compatible– Test rise, fall, duration, level with dummy load– Repeat test for all 16 outputs– Replace dummy load with relay, verify that the

relay switches for all 16 outputs– Repeat over temperature if necessary



• Tier 3 – Box level– Attach GSE (verifies with relays)– Functionally pulse all outputs

• Verify relay switches• Verify gross drive duration

• Tier 4 – Box and operational software level– Verify functionality commanded through software– GSE verifies correct relay switch

• Tier 5 – System– Verify gross functionality as needed for system

operation


Tiered Verification - Notes• Most complex and detailed functionality is verified at

lower levels– Performance– Error cases– Predictive of future operation

• Higher levels focus on functionality / operation• Lower level verification requires more manual touch• Higher level verification is more automated• STE and test procedure developed as appropriate

– Purpose is maximum test completeness– As well as “bang for the buck”


Concluding the Process• (“… through the provision of objective evidence…”)• The importance of traceability

– Something is only verified (formally) when it is documented– The tie between verification item and specification must be made

(verification matrix or database)• The importance of planning

– Establish the links between requirement, method, level, test case early

– Use the established structure to develop verification items• The importance of buy in

– All responsible must complete verification and report results– Systems engineers must track and close– People must keep up with the process

• Only personal commitment from all involved will ensure that the process is successful


Summary• Verification is to prove success, not avoid failure• Verification planning is an integral part of the project lifecycle

– Must be initiated with specification (including customer buy-in)– Must be included in design effort– Must be used to develop documentation and appropriate test hardware

• Verification is more than final test– Incorporates analysis and inspection throughout design process– Begins at lowest possible level– Develops proof of compliance at all levels of operation

• A thorough test is complicated– Requires extensive work– Must trade perfect for practical (a tiered test approach can help)

• Verification processes must be tracked and documented– The whole team must buy in

1022005 maplddesign integrity concepts what’s the exit strategy? verifying a design

Documents