a multi-paradigm verification flow dave whipp. copyright © nvidia corporation 2005 multiple...
TRANSCRIPT
![Page 1: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/1.jpg)
A Multi-Paradigm Verification Flow
Dave Whipp
![Page 2: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/2.jpg)
Copyright © NVIDIA Corporation 2005
Multiple Paradigms
Analog
Mixed Signal
Digital
Formal
Semi-formal
Simulation
Emulation
Directed tests
Directed random
Constrained random
Self checking
Model based
Cycle based
Transaction based
Unit level
System level
Plan driven
Coverage driven
Pure Verilog
SystemVerilog
VHDL
SystemC
E / Vera
HDCaml / Confluence
![Page 3: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/3.jpg)
Copyright © NVIDIA Corporation 2005
A Multi-Paradigm Verification Flow
What do I mean by Multiple Paradigms?
Why would we want one?
Engineering a [ Multiple Paradigm ] Flow
Was it worth the effort?
![Page 4: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/4.jpg)
Copyright © NVIDIA Corporation 2005
Two Philosophies
ConformityEmphasize Standardization
DiversityEmphasize Innovation
Synergy or Conflict?
![Page 5: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/5.jpg)
Copyright © NVIDIA Corporation 2005
Context
Large Project
Multiple Teams
Long Legacy
People move between projectsbenefit of familiar environment
exposure to alternative approaches
![Page 6: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/6.jpg)
Copyright © NVIDIA Corporation 2005
Understanding Variation
To understand what to standardize:you need to understand what not to standardize
Personal Preferences
Technical Aspects of the Designs
Supporting Legacy
Seeking the Next Big Thing
![Page 7: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/7.jpg)
Copyright © NVIDIA Corporation 2005
Personal Preferences
Emacs!
vi
![Page 8: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/8.jpg)
Copyright © NVIDIA Corporation 2005
Personal Preferences
Choice of editor doesn’t affect othersAt least, not much
Choice of scripting language has greater impactBut is encapsulated
A script’s users don’t see the implementation language
Choice of HVL affects whole teamCan’t write “E” tests for a “Vera” testbench!
But a unit testbench isn’t seen by other units
A good flow will allow encapsulation of preferencesI can go to any unit and build & run its tests
Enables rapid localization of infrastructure issues
![Page 9: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/9.jpg)
Copyright © NVIDIA Corporation 2005
Technological Differences
![Page 10: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/10.jpg)
Copyright © NVIDIA Corporation 2005
Technical Characteristics
PCIE Video Frame
Buffer
Off-chip Memory
Graphics Pipe
![Page 11: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/11.jpg)
Copyright © NVIDIA Corporation 2005
Unit Level System Testing
Stage N-1
Stage N+1
Stage N
Graphics Pipeline (Transaction Model)
RTL Unit DIFFDIFF
![Page 12: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/12.jpg)
Copyright © NVIDIA Corporation 2005
Transactions Vs Cycles
Data min_val (Addr a1, Addr a2){ Data d1 = mem_read(a1); Data d2 = mem_read(a2);
if (d1 < d2)return d1;
elsereturn d2;
}
Address
Data
a1
d1
Pipelined Bus
a2
d2
t1 t2 t3 t4 t5 t6
![Page 13: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/13.jpg)
Copyright © NVIDIA Corporation 2005
Reuse Vs Stagnation
Reuse considered GoodAvoid reinventing the wheel
Build on the shoulders of giants
Reuse invites InertiaReuse can propagate dependencies
Dependencies make things harder to change
Resistance to change is known as inertia
Inertia can lead to StagnationImproper reuse accumulates dependencies
Reused code that is not understood will bit-rot
To avoid stagnation, inject agitation
![Page 14: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/14.jpg)
Copyright © NVIDIA Corporation 2005
New Challenges
New Tools
New Platforms
New People
New Ideas
Refactoring
Testability
D.R.Y.
Avoiding Stagnation
![Page 15: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/15.jpg)
Copyright © NVIDIA Corporation 2005
Are Single Paradigm Projects Possible?
Project 1
Unit A
Unit B
Unit C
Project 2 Project 3
Unit D
Unit B
Unit C
Unit B
Paradigm 1
Paradigm 2
Paradigm 3
time
![Page 16: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/16.jpg)
Copyright © NVIDIA Corporation 2005
1999
GeForce 25622 Million Transistors
2002
GeForce463 MillionTransistors
2003
GeForce FX130 Million
Transistors
2004
GeForce 6 Series222 Million
Transistors1993
NV11 Million Transistors
2005
GeForce 7 Series302 Million
Transistors
A Rich Legacy
A project must support at least three approachesLegacy
Mainstream
Future
![Page 17: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/17.jpg)
A Paradigm-Agnostic Verification Flow
![Page 18: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/18.jpg)
Copyright © NVIDIA Corporation 2005
What is a Verification Flow?
The User Interface to VerificationCommand Line Interface
Graphical Interface
Web (browser) Interface
Programmers Interface
Used ByVerification Engineers
Designers
Managers
Robots
![Page 19: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/19.jpg)
Copyright © NVIDIA Corporation 2005
Purpose of a Flow
Conceptual FrameworkWhere do things live?
How do things work?
Keep out of the wayDon’t make life difficult for people
Define MetaphorsSubtly influence future directions
Anchor for VariationGive people something to complain about
![Page 20: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/20.jpg)
Copyright © NVIDIA Corporation 2005
Concepts
A well defined interface behaves as expectedObvious how it works, just by looking (“affordances”)
As consistent as possible, but no more so
Options are decision points
Users don’t read the manual
![Page 21: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/21.jpg)
Copyright © NVIDIA Corporation 2005
Designing a Flow (Interface)
Invent some Users
Figure out the important activities
Create the user model
Sketch a first draft
Iterate
Watch some real users; and iterate again
![Page 22: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/22.jpg)
Copyright © NVIDIA Corporation 2005
Invent some Users
Tim Nalu
Dusk Mike
![Page 23: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/23.jpg)
Copyright © NVIDIA Corporation 2005
Figure out the important Activities
TimburyNew Hire
Needs to get started
Checkout Tree
Run Sanity Regression
Simple Debug
DuskLead Designer
Intensive Debug
Modify RTLQuick Regressions
Analyze Coverage
Uses Emacs
NaluTesting Guru
Creates Testbenches
Hacks the Flow
Intensive Debug
Setup Regressions
Uses vi
MikeManager
View ProgressTests Written
Pass/fail counts
Coverage
Windows® User
![Page 24: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/24.jpg)
Copyright © NVIDIA Corporation 2005
Task Summary
build testbenchesrun quick regression tests (sanity tests)
run specific testrerun specific test with debug options
rerun specific phase(s) of a test
Add a testAdd a testbench
overnight regression testingSummarize results (web page)
![Page 25: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/25.jpg)
Copyright © NVIDIA Corporation 2005
User Model for Running Tests
Build Testbench
Generate Tests
Run Predictor
Run DUT
Compare Behaviors
![Page 26: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/26.jpg)
Copyright © NVIDIA Corporation 2005
The Implementation
“Make” driven flowMakefiles define the build
Automated testbench creation
Compile/generate tests where appropriate
“makepp” is a better “make”
Perl scripting for the gutsEach test is a Perl object
Small number of standard classes
Individual units define additional classes
![Page 27: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/27.jpg)
rdistnvmk run_unit
GEN DIFF
VSIM
CSIM
tbgen
fmtbgen
fmgen
run_testgen
compare
“grok”
randomizetestlist
rand_regress
mod
stgen
ttgen
build_testlist
tjob
LSF
vcs
An Early Sketch
![Page 28: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/28.jpg)
Copyright © NVIDIA Corporation 2005
Command Line Interface
nvmknvmk sanity
nvmk regress
nvmk all
nvmk open_bugs TESTNAME=bug42
nvmk open_bugs TESTNAME=bug42 VSIM=1 DUMP=1
nvmk open_bugs TESTNAME=bug42 CSIM=gdb
![Page 29: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/29.jpg)
Copyright © NVIDIA Corporation 2005
Detailed Triage and Debug
cd rundir/open_bugs/bug42
ls *.loggen.log csim.log
vsim.log diff.log
replay vsim.log +foo +bar
replay –gdb csim.log
![Page 30: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/30.jpg)
Copyright © NVIDIA Corporation 2005
Testing the Flow
Considered a hard problem“If it works, then it works”
Testability is a featureStart testing early
Some tests are better than none
Create a multi purpose “Example Project”A stable testing environment
Acts as documentation
Use the “Invented Users” to create activity scripts
Keep it up-to-date
![Page 31: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/31.jpg)
Copyright © NVIDIA Corporation 2005
Watch some Real Users
Image courtesy of DAZ Productions
![Page 32: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/32.jpg)
Copyright © NVIDIA Corporation 2005
How to “Watch” Users
Meetings and Discussions
Coffee-Break Grousing
Bug Reports
Keep Track of Support Requestscreate FAQs
VNC (Remote Desktop)
Instrumentation
![Page 33: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/33.jpg)
Copyright © NVIDIA Corporation 2005
Biggest Problems
Performancemake-driven user interface was subjectively slow
too much dependency checking
users know what file’s they’ve changed
how to communicate this to the tools?
Too much flexibilityOptions imply decisions
Lack of sufficient understanding
So pervasive cut & paste reuse
![Page 34: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/34.jpg)
Copyright © NVIDIA Corporation 2005
Build Time Distribution: 10,000 per sample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
> 1 hour
< 1 hour
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
< 20 sec
![Page 35: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/35.jpg)
Copyright © NVIDIA Corporation 2005
Build Time Distribution: 1000 per sample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
> 1 hour
< 1 hour
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
< 20 sec
![Page 36: A Multi-Paradigm Verification Flow Dave Whipp. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation](https://reader038.vdocuments.net/reader038/viewer/2022103006/56649eb25503460f94bb8e75/html5/thumbnails/36.jpg)
Copyright © NVIDIA Corporation 2005
Summary
Verification Engineers do more than write tests
A flow is a user interfaceEncourage user feedback
Add instrumentation
A flow is an enabler, not disablerencourage innovation
don’t tell users what not to do
Without diversity, a flow stagnates
Without standardization, a flow flounders