2007tehran university1 functional verification of hdl models introduction to verification

2007 Tehran University 1

Functional Functional Verification of HDL Verification of HDL

ModelsModels

Introduction to VerificationIntroduction to Verification


Reference BookReference Book

• Janick Bergeron Janick Bergeron [WTB][WTB]

Writing Testbenches:Writing Testbenches:

Functional Verification Of HDL Functional Verification Of HDL Models.Models.

• First Edition, Kluwer, 2000, ISBN: 0-First Edition, Kluwer, 2000, ISBN: 0-7923-7766-47923-7766-4

• Second Edition, Kluwer, 2003, ISBN: Second Edition, Kluwer, 2003, ISBN: 1-4020-7401-81-4020-7401-8


Other referencesOther references

• Comprehensive Functional Verification: Comprehensive Functional Verification: The complete industry cycleThe complete industry cycle– B. Wile, J. Goss, W. Roesner [IBM]B. Wile, J. Goss, W. Roesner [IBM]

• Principles of Functional VerificationPrinciples of Functional Verification– Andreas MeyerAndreas Meyer

• Functional Verification Coverage & Functional Verification Coverage & Measurement and AnalysisMeasurement and Analysis– Andrew PizialiAndrew Piziali

• Advanced Verification Techniques: A Advanced Verification Techniques: A SystemC Based Approach for Successful SystemC Based Approach for Successful TapeoutTapeout– Leena Singh, L. Drucker, Neyaz KhanLeena Singh, L. Drucker, Neyaz Khan


Course GoalsCourse GoalsAimAim: : To familiarise you with the routine tasks in To familiarise you with the routine tasks in

verificationverificationmanagementmanagement, and to give you the , and to give you the technical background technical background

plus some of the plus some of the practical skills practical skills expected from a Design expected from a Design Verification Engineer.Verification Engineer.

• PrerequisitePrerequisite: Hardware Language programming: Hardware Language programming

On successful completion of this unit, you will be able On successful completion of this unit, you will be able to:to:

• understand the complexities and limits of verification;understand the complexities and limits of verification;• carry out functional verification and determine its carry out functional verification and determine its

effectiveness;effectiveness;• set appropriate verification goals, select suitable set appropriate verification goals, select suitable

verification methods and assess the associated risks;verification methods and assess the associated risks;• compile a verification plan that fits into the flow of a compile a verification plan that fits into the flow of a

design projectdesign project


19741974

Brian W. KernighanBrian W. Kernighan• Debugging is twice as hard as Debugging is twice as hard as

writing the code in the first place. writing the code in the first place. Therefore, if you write the code as Therefore, if you write the code as cleverly as possible, you are, by cleverly as possible, you are, by definition, not smart enough to definition, not smart enough to debug it.debug it.


Bugs: What is the Cost?Bugs: What is the Cost?

• Time to MarketTime to Market is affected is affected• Bugs found early in design have little cost!Bugs found early in design have little cost!• Finding a bug at chip/system level has Finding a bug at chip/system level has

moderate cost:moderate cost:– Requires more debug time and isolation time.Requires more debug time and isolation time.– Could require new algorithm, which could effect Could require new algorithm, which could effect

schedule and cause board rework.schedule and cause board rework.

• Finding a bug in System Test (test floor) Finding a bug in System Test (test floor) requires re-spin of a chip.requires re-spin of a chip.

• Finding a bug after customer delivery costFinding a bug after customer delivery cost millons


Cost of Bugs Over TimeCost of Bugs Over Time

system

Number of bugs found

chipmodule

Cost of finding bugs

Timecustomer


Importance of Importance of VerificationVerification

• 70% of ASIC / IP / SoC design effort goes to 70% of ASIC / IP / SoC design effort goes to verificationverification– Is on the critical pathIs on the critical path– Has effects on Schedule, Cost, QualityHas effects on Schedule, Cost, Quality– Fewer revisions through fabrication process Fewer revisions through fabrication process

means lower cost and faster time-to-marketmeans lower cost and faster time-to-market

• Twice more Verification engineers than RTL Twice more Verification engineers than RTL designerdesigner

• Re-spinning cost time and moneyRe-spinning cost time and money• Getting the design right first time is an artGetting the design right first time is an art

– Due to chip complexity it is more and more Due to chip complexity it is more and more difficultdifficult


Silicon ValidationSilicon Validation

• Silicon Validation Silicon Validation times increase as times increase as features size decreasesfeatures size decreases

• The difficulty is The difficulty is expected to increase at expected to increase at 65nm and below 65nm and below because existing ad-because existing ad-hoc methodologies do hoc methodologies do not scale with the not scale with the unprecedented levels unprecedented levels of SoC device of SoC device complexity. complexity.


SoC VerificationSoC Verification

• Complete system-level verification of a Complete system-level verification of a complex SoC at 90 nm or below is not complex SoC at 90 nm or below is not feasible pre-silicon feasible pre-silicon – Even the most sophisticated SoC design Even the most sophisticated SoC design

methodology cannot fully account for all the methodology cannot fully account for all the parameters that impact silicon behavior, or parameters that impact silicon behavior, or for all logic "corner cases" that occur in the for all logic "corner cases" that occur in the real life of a chip working at speed and in real life of a chip working at speed and in system [Miron Abramovici, Sep. 2007 ]system [Miron Abramovici, Sep. 2007 ]

• post-silicon validation has become an post-silicon validation has become an essential step in the design essential step in the design implementation methodology implementation methodology


Importance of Verification Importance of Verification (Cont.)(Cont.)

• International Technology Roadmap International Technology Roadmap for Semiconductors identified for Semiconductors identified Verification as a crisis area Verification as a crisis area (http://public.itrs.net/)

• Shortfalls in qualified engineers and Shortfalls in qualified engineers and software-trained designerssoftware-trained designers

• Verification can be reduced through Verification can be reduced through parallelism, abstraction and parallelism, abstraction and automationautomation


Course OutlineCourse Outline

Lecture Topics:Lecture Topics:• What is Verification? What is a Testbench?What is Verification? What is a Testbench?• Verification Tools (including Verilog basics)Verification Tools (including Verilog basics)• Verilog HDL coding and Verification PlanVerilog HDL coding and Verification Plan• Assertion-based VerificationAssertion-based Verification• Advanced Testbench Design Methodology Advanced Testbench Design Methodology • Functional Formal Verification and Functional Formal Verification and

Property CheckingProperty Checking• Labs (Verilog, ModelSim, C/C++, SystemC)Labs (Verilog, ModelSim, C/C++, SystemC)


Introduction to IC Design Introduction to IC Design ProcessProcess

1.1. SpecificationSpecification2.2. Behavioral DescriptionBehavioral Description

Architectural SynthesisArchitectural Synthesis3.3. RTL DescriptionRTL Description

Logical SynthesisLogical Synthesis4.4. Gate Level DescriptionGate Level Description

Physical SynthesisPhysical Synthesis5.5. Layout and tape-outLayout and tape-outIn practice all steps almost start at the In practice all steps almost start at the

same time and run in parallelsame time and run in parallel


Steps in IC Design Steps in IC Design ProcessProcess

Architectural SynthesisArchitectural Synthesis Input: behavioral Input: behavioral descriptiondescription

• generates architecture for design at RTLgenerates architecture for design at RTL– data path (interconnection of adders, multipliers, etc)data path (interconnection of adders, multipliers, etc)– control unit (signals to control data path)control unit (signals to control data path)

• scheduling and bindingscheduling and bindingLogic SynthesisLogic Synthesis Input: RTL descriptionInput: RTL description• converts into network of logic primitives (depending on converts into network of logic primitives (depending on

designdesign• style)style)• Two groups:Two groups:

– sequential/combinatorialsequential/combinatorial

Physical DesignPhysical Design Input: gate level Input: gate level descriptiondescription

• generates geometric patterns that define physical layout of generates geometric patterns that define physical layout of chipchip


Verification Role in IC Verification Role in IC DesignDesign

IC design is a complex process:IC design is a complex process:• Conflict: design time (decreased) versus complexity Conflict: design time (decreased) versus complexity

(increased)(increased)• Aim: correct by construction, right-first-time designAim: correct by construction, right-first-time design

More and more time-consuming to obtain an More and more time-consuming to obtain an acceptable level of confidence in correctness of acceptable level of confidence in correctness of designdesign

• Design time << Verification timeDesign time << Verification time• Needs its own trained verification engineersNeeds its own trained verification engineers• 80% of written code is for the verification 80% of written code is for the verification

environmentenvironment


How does a designer know How does a designer know a circuit is correct?a circuit is correct?

concept

Specification

HDL design (RTL)

Tape out

Silicon

Verify: What you specified is what you envisioned

Verify: What you designed is what you specified

Verify: What you taped out is what the RTL described

Verify: What you manufactured is what you taped out

There is no single methodology that covers allThere is no single methodology that covers all!


Some DefinitionsSome Definitions

• What is Verification?What is Verification?– A process used to demonstrate the functional A process used to demonstrate the functional

correctness of a design in its implementationcorrectness of a design in its implementation

Hence the term Hence the term Functional VerificationFunctional Verification

• What is a Testbench?What is a Testbench?– Code used to generate predetermined input Code used to generate predetermined input

sequence to a designsequence to a design– Implemented using Verilog, VHDL, Implemented using Verilog, VHDL, ee or Open or Open

Vera, also include C routinesVera, also include C routines– Always refers to test case/scenarioAlways refers to test case/scenario


Testbench and DUVTestbench and DUV

• Completely closed systemsCompletely closed systems• No inputs or outputs go in or outNo inputs or outputs go in or out• Generic Structure of a testbench and Generic Structure of a testbench and

DUV:DUV:

DesignUnder Verification

Testbench

Stimulus Generator Checker

Supply stimulus to checker for comparison with the specification


Verification is on critical Verification is on critical pathpath

Need to minimise verification time e.g. by Need to minimise verification time e.g. by using:using:

• ParallelismParallelism: Add more resources: Add more resources• AbstractionAbstraction: Higher level of abstraction (i.e. : Higher level of abstraction (i.e.

C vs Assembly, TLM)C vs Assembly, TLM)– This often means a reduction of control!This often means a reduction of control!

• AutomationAutomation: Tools to automate standard : Tools to automate standard processesprocesses– Requires standard processes/methodology.Requires standard processes/methodology.– Usually a variety of functions, interfaces, Usually a variety of functions, interfaces,

protocols, and transformations must be verified.protocols, and transformations must be verified.– Not all (verification) processes can be automated.Not all (verification) processes can be automated.– Randomization can be used as an automation toolRandomization can be used as an automation tool


Reconvergence ModelReconvergence Model

• Conceptual representation of the Conceptual representation of the verification process: verification process: Illustrate what Illustrate what is exactly being verifiedis exactly being verifiedTransformation

Verification

Purpose of verification is to ensure that the result of some transformation is as intended or as expected.


Source of Source of ambiguity/errorambiguity/error

• Human Factor: Human Factor: – How a designer read and interpret the How a designer read and interpret the

specification specification – Specification can be Specification can be ambiguousambiguous– Designer can introduce errors by Designer can introduce errors by

(mis-)interpretation (mis-)interpretation (re-convergence model)(re-convergence model)

DANGERDANGER: When a designer verifies her/his own : When a designer verifies her/his own design - then she/he is verifying her/his own design - then she/he is verifying her/his own interpretation of the design - not the interpretation of the design - not the specification!specification!


How to Reduce Human How to Reduce Human ErrorsErrors

Reduce human errors by:Reduce human errors by:• AutomationAutomation

– Take the human out but not really feasibleTake the human out but not really feasible– Process not well defined enough, not formal Process not well defined enough, not formal

enoughenough• Make mistake-proof the human intervention Make mistake-proof the human intervention

by reducing it to simple steps (Poka-Yoka)by reducing it to simple steps (Poka-Yoka)• RedundancyRedundancy

– Two individuals check each other’s workTwo individuals check each other’s work– Costly mechanism but still cheaper than re-Costly mechanism but still cheaper than re-

design and chip recalldesign and chip recallBut never let a designer verify his own designBut never let a designer verify his own design


What is being verified?What is being verified?

• Choosing a common origin and the Choosing a common origin and the reconvergence points determines what is reconvergence points determines what is being verified.being verified.– Determined by the tool used to perform the Determined by the tool used to perform the

verificationverification

• The following approaches have different The following approaches have different origin and reconvergence points:origin and reconvergence points:– Formal VerificationFormal Verification– Functional VerificationFunctional Verification– Testbench GeneratorsTestbench Generators


Formal VerificationFormal Verification

This is This is notnot a tool that mathematically a tool that mathematically determines the correctness of a design determines the correctness of a design without having to write testbenches!without having to write testbenches!

Build a formal mathematical model of some Build a formal mathematical model of some aspect of the system and calculate aspect of the system and calculate whether or not it has some desired whether or not it has some desired propertyproperty

• 2 types of Formal Verification2 types of Formal Verification– Equivalence CheckingEquivalence Checking– Model CheckingModel Checking


Equivalence CheckingEquivalence Checking• Compares two models to check for equivalenceCompares two models to check for equivalence

– Mathematically proves that both are logically Mathematically proves that both are logically equivalent equivalent

• Commonly used on lower level of designCommonly used on lower level of design

• Make sure that scan chain, clock tree insertion Make sure that scan chain, clock tree insertion did not change the functionalitydid not change the functionality

• It is only interested in comparing boolean and It is only interested in comparing boolean and sequential logic functions – no tech mappingsequential logic functions – no tech mapping

• Why can’t we entirely trust the synthesis tool?Why can’t we entirely trust the synthesis tool?

Equivalence Checking

Synthesis

RTL Gate


Model CheckingModel Checking

• Assertions or characteristics of a design are Assertions or characteristics of a design are formally proven or disprovedformally proven or disproved

• Use to check for generic problems or violations of Use to check for generic problems or violations of user-defined properties of the behavior of the user-defined properties of the behavior of the designdesign

• Usually employed at higher levels of design process

• Properties are derived from specification• Properties are expressed in some (temporal) logic• Checking often involves a (finite) state machine

model


Model CheckingModel Checking

Problems:Problems:• Knowing Knowing which properties which properties to check and to check and how to how to

express them express them needs expertise.needs expertise.– Very few designers are also good mathematicians.Very few designers are also good mathematicians.– (Verification engineers should be closing this gap.)(Verification engineers should be closing this gap.)

• Very few user-friendly tools for model checking.Very few user-friendly tools for model checking.• Tools that do exist can only check small designs.Tools that do exist can only check small designs.• There is a lot of There is a lot of researchresearch to make to make Model Model

CheckingChecking more practically usable. more practically usable. For an For an introduction see ”Model Checking” by Clarke, introduction see ”Model Checking” by Clarke, Grumberg and PeledGrumberg and Peled


Functional VerificationFunctional Verification

• Ensure that a design implements Ensure that a design implements intended functionalityintended functionality

• Without it, one must trust that the Without it, one must trust that the transformation of a specification into transformation of a specification into RTL code was performed correctlyRTL code was performed correctly

• We can only prove the presence of We can only prove the presence of bugs, but we cannot prove their bugs, but we cannot prove their absence!absence!


Testbench GeneratorsTestbench Generators• Tools to generate stimuli to exercise code or expose bugs• Designer/Verification engineer input is still required.• RTL code is the origin and there is no reconvergence point

• Verification engineer has responsibility to determine if the testbench applies valid stimuli

• If used with parameters, one can control the generator in order to focus the testbench on more specific scenarios

• 2 input ”and” gate - ... combinations of inputs (exhaustive)• Instruction stream generators to verify processor cores.Is exhaustive verification (always) feasible?

RTL Testbench

Testbench Generation

Code coverage/proof Metrics


Functional Verification Functional Verification ApproachesApproaches

There are 3 complementary There are 3 complementary approaches:approaches:

• Black-Box VerificationBlack-Box Verification• White-Box VerificationWhite-Box Verification• Grey-Box VerificationGrey-Box Verification


Black Box VerificationBlack Box Verification• The black box has inputs, outputs, and performs some The black box has inputs, outputs, and performs some

function.function.• The function may be well documented (or not ...).The function may be well documented (or not ...).• To verify a black box, you need to To verify a black box, you need to understand the understand the

function function and be able to predict the outputs based on the and be able to predict the outputs based on the inputs.inputs.

• The black box can be a full system, a chip, a unit of a The black box can be a full system, a chip, a unit of a chip, a module, etc.chip, a module, etc.

• No knowledge of the actual implementation is requiredNo knowledge of the actual implementation is required• Independent of implementation.Independent of implementation.• Lacks controllability and observability.Lacks controllability and observability.• Difficult to locate source of problem, only exposes effectsDifficult to locate source of problem, only exposes effects• Example: round-robin arbiterExample: round-robin arbiter


White Box VerificationWhite Box Verification

• Opposite of Black BoxOpposite of Black Box• Full visibility and controllability of the Full visibility and controllability of the

internal structure and implementation of internal structure and implementation of DUVDUV

• Easy to set up interesting conditionsEasy to set up interesting conditions• But testbench is tied to a specific But testbench is tied to a specific

implementation – needs to be changed if implementation – needs to be changed if design modifieddesign modified

• Best for module or system level verificationBest for module or system level verification• Example: counter roll-overExample: counter roll-over


Grey Box VerificationGrey Box Verification

• Compromise between BB and WBCompromise between BB and WB• Like BB, it observes and controls top level Like BB, it observes and controls top level

interface signalinginterface signaling• But increases coverage metrics by adding But increases coverage metrics by adding

some non-functional modification to increase some non-functional modification to increase visibilityvisibility– Include additional software-accessible registers to Include additional software-accessible registers to

observe internal statesobserve internal states

• In practice for SoC most environments are In practice for SoC most environments are GB because prediction of results are GB because prediction of results are sometimes impossible without looking at sometimes impossible without looking at internal signalsinternal signals


Perfect VerificationPerfect Verification

• To fully verify a black box, you must show that the logic works correctly for all combinations of inputs.

This means:• Driving all permutations on the input

lines.• Checking for proper results in all cases.• Full verification is not practical on

large designs, but the principles are valid across all verification approaches


Verification versus Verification versus TestingTesting

• Often Confused:Often Confused:– Test: show the design was Test: show the design was

manufactured correctly (manufactured correctly (post-siliconpost-silicon) – ) – verifies internal nodes can be toggled verifies internal nodes can be toggled by applying test vectorsby applying test vectors

– Verification: Ensure the design meets Verification: Ensure the design meets its functional intent (its functional intent (pre-siliconpre-silicon))

Spec Gates Silicon

HW Design Manufacturing

Testbench Testing

HDL

Equivalence CheckingEquivalence Checking

SynthesisSynthesis


Verification versus Testing Verification versus Testing (Cont.)(Cont.)

• One test method is scanning: One test method is scanning: Design For Design For TestTest– Links all registers into a long shift register (chain)Links all registers into a long shift register (chain)– Chain accessible from the I/O pins, observe and Chain accessible from the I/O pins, observe and

control internal statescontrol internal states• Scan-based testing restricts the design Scan-based testing restricts the design

(sync, no gated clock, single clock) but keeps (sync, no gated clock, single clock) but keeps cost low (reference book: Digital System cost low (reference book: Digital System Testing and Testable Design, Abramovici)Testing and Testable Design, Abramovici)

Research topicResearch topic: “Design for Verification”!: “Design for Verification”!Consider what the design is supposed to do? Consider what the design is supposed to do?

How will it be verified?How will it be verified?


Design and Verification Design and Verification ReuseReuse

• You won’t use what you don’t trust.You won’t use what you don’t trust.• How to trust it? (Verify it)How to trust it? (Verify it)• For reuse, designs must be verified with more For reuse, designs must be verified with more

strict requirementsstrict requirements– All claims, possible combinations and uses must be All claims, possible combinations and uses must be

verified.verified.– Not just how the design is (intended to be) used in Not just how the design is (intended to be) used in

one environmentone environment

• Verification/testbenches re-usable to some Verification/testbenches re-usable to some degreesdegrees– BFMs have different requirements at block or system BFMs have different requirements at block or system

levellevel– From project to project From project to project


Cost of VerificationCost of Verification• It is always too long and costs too much but It is always too long and costs too much but

is absolutely necessaryis absolutely necessary– It does not generate revenue!It does not generate revenue!– As number of bugs decreases, the cost and time As number of bugs decreases, the cost and time

for finding the remaining ones increases: for finding the remaining ones increases: When When is the verification done?is the verification done?

• Yet indispensable:Yet indispensable:– To create revenue, design must be functionally

correct and provide benefits to customer.– Proper functional verification demonstrates

trustworthiness of the design.– Right-first-time designs demonstrate

professionalism and ”increase” reputation of design team


When Is Verification When Is Verification Done?Done?

Never truly done on complex designs.Never truly done on complex designs.• RememberRemember: Verification can only show presence of : Verification can only show presence of

errors, not their absence.errors, not their absence.• Skill and experience are needed to:Skill and experience are needed to:

– Determine WHAT needs to be verified.Determine WHAT needs to be verified.• Be selective/exhaustive.Be selective/exhaustive.• Identify corner cases.Identify corner cases.

• Given enough time, errors Given enough time, errors will will be uncovered.be uncovered.• Critical question:Critical question:

– Is the error likely to be severe enough to warrant the Is the error likely to be severe enough to warrant the effort spent to find/correct it?effort spent to find/correct it?

Verification is similar to statistical hypothesis testing.Verification is similar to statistical hypothesis testing.• Hypothesis ”under test” is: Hypothesis ”under test” is: Is the design Is the design

functionally correct?functionally correct?


Hypothesis MatrixHypothesis Matrix

• Type I mistakes: Easy to identify - found error where none exists.

• Type II mistakes: Most serious - verification failed to identify an error!– Can result in a bad design being shipped unknowingly!

• Knowing where you are in the verification process is much easier to estimate than how long it will take to complete the job.

Type II(False Positive)

Type I(False Negative)

Bad Design

Good design

Fail Pass


SummarySummary• Seen what a testbench is• Understood how important verification is and why

it is so important.• Seen that parallelism, automation and abstraction are

strategies to reduce the time necessary to implement testbenches.

• Used reconvergence models to help us identify what is verified.

• Been introduced to the various verification approaches that exist.

• Investigated the importance of verification for design reuse.

• Discussed the cost of verification• Next: Verification tools, especially simulators,

waveform viewers and simulation languages (Verilog)

2007tehran university1 functional verification of hdl models introduction to verification

Documents

verification engineers

verification plan

design verification

cost time

suitable verification

lower cost

moderate cost

little cost