heuristic stimuli generation for coverage closure ... · [email protected]. goal • to...

Heuristic Stimuli GenerationHeuristic Stimuli GenerationFor Coverage Closure For Coverage Closure

Exploiting Simulation FeedbackExploiting Simulation FeedbackGiovanni Squillero

Politecnico di Torino - ItalyCAD Group ( )

[email protected]

GOALGOAL

• To propose a methodology for coverage-directed stimuli generation based on simulation feedback

• Such stimuli could be added as new content to improve existing validation suites

DVClub 2010-01-18 [email protected] 2

AcknowledgementsAcknowledgements

• Danilo Ravotto• Ernesto Sanchez• Matteo Sonza Reorda• Alberto Tonda

• + many others


OutlineOutline

• Proposed methodology• Case studies• Conclusions


Design ChoicesDesign Choices

• Being able to tackle real problems• Develop a versatile and broadly applicable

methodology – Compatible with different environment– Compatible with any coverage metric

• Minimize effort to change goal/target– Exploit common aspects


Feedback-Based ApproachFeedback-Based Approach

• Simulation-based approach• Exploits feedback from simulation • Incremental improvement/refinement of the

solution (trial-and-error)• Trade-off between computational resources

and confidence• May exploit heuristics or problem-specific

knowledge


Proposed MethodologyProposed Methodology


StimuliGenerator

System

Stimuli

Feedback

StimuliStimuli


StimuliGenerator

System

Stimuli

Feedback

StimuliStimuli

• Sequences of bits• Sequences of keys • Full fledged assembly language programs• VHDL test case• External world• …


Stimuli GeneratorStimuli Generator


StimuliGenerator

System

Stimuli

Feedback

Stimuli GeneratorStimuli Generator

• Exploit an Evolutionary Algorithm to generate stimuli to maximize a given function


Evolutionary AlgorithmsEvolutionary Algorithms

• Meta-heuristic optimization algorithm based on the concept of population and exploiting some principles of natural evolution



• Succession of random and deterministic steps– A systematic way of throwing dices– Better than pure random

“The great effect produced by the accumulation in one direction, during successive generations, of differences absolutely inappreciable by an uneducated eye”



• Population– Multiple solutions considered in each step– More resistant than pure hill-climbing– Different solutions may interbreed



• Why using an evolutionary algorithm?– Adaptative– Able to find unexpected solutions– Better than randomBetter than random



• Problem: Fitness function– GOAL: Optimize the wheel– FITNESS: Minimize the number of “bumps”



• Problem: Black magic


µGP (MicroGP)µGP (MicroGP)

• CAD Group general-purpose evolver– 3 versions (only 2 released under GPL)– Project started in 2002– 11 developers + contractors, students, …

• Current version– ≈ 300 file, > 40,000 lines in C++



Evolutionary Optimization: the µGP toolkitE. Sanchez, M. Schillaci, G. Squillero

Springer, 2010

ISBN: 978-0-387-09425-0




http://ugp3.sourceforge.net/

MicroGP++ (aka. ugp3, µGP3)• Information• Download• Credits

System & FeedbackSystem & Feedback


StimuliGenerator

System

Stimuli

Feedback

SystemSystem

• Strongly problem dependant• Model via simulation/emulation– HDL (netlist to high-level)– HW accelerated (e.g., exploiting FPGA)– Architectural simulator– ISA simulator

• Real device


Feedback (examples)Feedback (examples)

• From simulation– Code coverage metrics (e.g., instruction coverage)– HW specific metrics (e.g., toggle coverage)– High-level information (e.g., FSM coverage)

• From the real system– Performance counters– Physical measures (e.g., temperature, time, power

consumption)


OutlineOutline



Design VerificationDesign Verification

• Devise a set of programs maximizing different coverage metrics


FeedbackFeedback

• Code coverage metrics– Statement coverage (SC)– Branch coverage (BC)– Condition coverage CC)– Expression coverage (EC)

• HW specific metric– Toggle coverage (TC)


SystemSystem

• DLX/pII– Cleaned and simplified MIPS intended primarily

for teaching purposes– 32-bit load/store architecture, 5-stage pipeline– VHDL RTL description• 4,558 statements• 3,695 branches • 193 conditional statements (1,764 expressions)• 8,283 logic bits


Experimental ResultsExperimental Results




230Kinstructions

2,978instructions



20h 33h

27h

37h

95h

≈1 week

Post-silicon VerificationPost-silicon Verification

• Generate functional test programs for post-silicon verification

• Activity performed in collaboration with the ETM Group, Intel (Phoenix)

• Intel Pentium 4– 42-55 millions of transistors– 2GHz clock– NetBurst architecture


System & FeedbackSystem & Feedback

• Real Pentium 4 (no simulation) • Performance counters as metric– Introduced in 1993 in IA32 architecture – 48 event detectors + 18 programmable counters– Instruction-tagging (for discriminating non-

speculative performance events)



• Two sets of experiments:– Mispredicted branches over the total branches– Clock cycles in which the trace cache is delivering

µops to the execution unit instead of decoding or building traces (hard metric!)

• Intrinsic features of the µarchitectural design• Time (both): 12h/program

Haifa 2008 G. Squillero 33

Mobile PhoneMobile Phone

• 12m-activity performed in collaboration with Motorola Research Center in Turin (“Automation of Current Drain Measures”)


SystemSystem

• Real prototypical mobile phone– P2K platform– GSM/3G networks– Complex applications

• Radio Communication Analyzer, anechoic chamber

• Controllable power supply• Custom hardware


StimuliStimuli

• Sequence of keys• Power supply• Network status


FeedbackFeedback

• Physical measures– Power consumption (+ autocorrelation)– Network status

• FSM transition coverage



• Major bugs discovered in the prototype– Test infrastructure– LCD display– Fake deep-sleep state


OutlineOutline



ConclusionsConclusions

• Simulation-based & feedback-based• Evolutionary algorithm• Broadly applicable• Human resources– Limited (set-up )

• Computational resources– Easily parallelizable (generation level)– Trade-off quality vs. effort


ConclusionsConclusions

• May exploit other methodologies– Useful starting point– Effective completion

• May be coupled with other methodologies– Rule-based instruction randomizers– Simulation-based approaches


heuristic stimuli generation for coverage closure ... · [email protected]. goal • to...

Documents