code coverage formal unreachability analysis

…personal…portable…connected

Code Coverage Formal Unreachability AnalysisRicardo DantasDialog Semiconductor

Metric-Driven Verification

Formal Unreachability Analysis: the Challenge

Divide & Conquer: the Solution

Configuration Script

Results

Conclusions

Dialog Semiconductor © 2015 2

Agenda

Metric-Driven Verification


• Modern digital verification methodologies, especially constrained random ones (e.g. UVM), strongly rely on feedback metrics, which indicate how thoroughly a design has been verified. Two popular metrics are:

• Functional Coverage:• Captures the occurrence of complex scenarios during simulation, based on

coverage items provided by the verification engineer, e.g. SystemVerilogcovergroups;

• Code Coverage:• Captures parts of the design that have been exercised during simulation, i.e.

block, expression, toggle, fsm;• Requires no user inputs but is not aware of complex scenarios;• 100% code coverage is not an indication that the verification is done (complex

scenarios may be missing).

• So, why do we still care about code coverage?

Why Code Coverage?


• 100% code coverage is typically hard to achieve and it is not an indication that the verification is done, but low code coverage figures indicate that:

• more stimuli are needed (new tests, more runs with different seeds) and/or• part of the design is unreachable.

• The so called unreachable code can be due to several reasons, including defensive code, improved readability, parameterization, legacy or dead code. For instance, a perfectly reachable code might become unreachable due to a bug upstream, therefore the importance of analysing unreachable code.

• Manual identification of unreachable code can be extremely laborious and tedious, not to mention error-prone. Therefore the use of a formal tool to automated this process is very welcome, although visual analysis of the identified unreachable code is still necessary.

A tool might pinpoint the location of unreachable code but it is still necessary to understand why it is not exercisable.

UNR Analysis with Cadence IEV


• Cadence Incisive Enterprise Verifier (IEV) was used for the formal unreachability analysis (UNR), which, starting from an existing code coverage database, inspects the coverage holes and adds automatic properties to be formally proven. If a property proof succeeds (passes) the code is reachable and extra stimuli

needs to be provided; If a property proof fails the code is unreachable and can be waived after

review; The tool might not be able to formally prove (in reasonable time) complex logic,

thus resulting in ‘explored’ properties (neither passed nor failed).

• A database with all unreachable holes is created, which can then be merged with the original coverage db.

• Unreachable coverage holes can be inspected inside IMC before being waived.

UNR Analysis in simple steps (I)


The following requirements are needed by IEV for the unreachability analysis:

1. the RTL design;

2. a code coverage database• the higher the coverage, the faster the unreachability analysis runs;• depending on the design complexity, a number of properties might be explored

(neither proven nor failed);

3. the hierarchical level definition where to start the analysis from• the top-level DUT or any internal block instance;

4. list of modules to be black-boxed, if any.

No constraints were added for the results presented in the next slides but there is the possibility to load them from a Tcl script. The goal was to create a robust and simple to use flow to be adopted by all digital design groups.

UNR Analysis in simple steps (II)


irun -iev -sv -64bit -f <compilation options>-top $top-inst_top $blk_inst -covdut $top-bb_list $blk_bblist -covtest ${unr_results}_{blk_name}-covoverwrite -coverage all -covfile cov.cf -covdb $covdb -l ${unr_results}_{blk_name}.log -licqueue -input unr.tcl

IEV arguments:• -top: entity to be considered top-level design unit for formal verifier, e.g. testbench;• -inst_top: hierarchical name to be considered as the top-level for the formal build

(top-level DUT or any internal block instance);• -bb_list: file containing the design units and libraries to be blackboxed;• -covdb: simulation coverage database for coverage unreachability flow;

set engine auto_distdefine auto_dist_max 3define effort lowproveexit

Constraints or additional IEV options can be loaded via an

external script

UNR Analysis: a practical approach


• Pros:• Quick set-up and straightforward flow;• Good results (no explored properties) can be achieved from a few minutes to a

couple of hours;• Unreachable code is highlighted when visualized with IMC and, if needed, can

be completely waived;

• Cons:• UNR analysis of a complex design, e.g. top-level, can be extremely slow

(several days), ending up in poor results due to the increasing amount of explored properties;

• As an alternative, it is possible to divide the design in smaller parts and tackle them individually. The problem with this approach is the reduced awareness of an isolated block, i.e. other blocks in its cone of influence are not considered during UNR analysis.

• The challenge:• How to maximized the results (all unreachable code found and no explored

properties) whilst keeping reasonable running times (hours, not days)?

UNR Analysis: Divide & Conquer (I)


A divide & conquer strategy consists of breaking up the design for quicker runs and merging the results afterwards. The way the design is split will influence the results and can be divided into the following strategies:

1. Standalone blocks UNR analysis is launched for all internal blocks individually (big internal blocks can be

divided further), i.e. IEV –inst_top set to the block instance under analysis; a “hollow” top-level run is executed with all internal blocks black-boxed; running times are reasonable but individual blocks are not influenced by their neighbours

(low awareness).

2. Hierarchical blocks (sub-systems) UNR analysis is executed multiples times from the top-level, each time with one internal

block visible, whilst the others are black-boxed (top-block awareness); sub-systems can be created by letting a few blocks visible (intra-block awareness); typically slower than the previous method, as the top-level is re-analysed for every run; depending on the complexity and size of the sub-systems, running times can be kept

short or can increase drastically, ending up several explored properties. Sub-system definition requires design knowledge.

UNR Analysis: Divide & Conquer (II)


TOP

Wrapper

Block BBlock A Block C

Block EBlock D Block F

TOP

Wrapper

Block BBlock A Block C

Block EBlock D Block F

Standalone blocks1) Individual block-level UNR analysis of all internal blocks:

A, B, C, D, E, and F;-inst_top: TOP.Wrapper.Block_<x>-bb_list: [ ]

2) Top-level analysis (starting from TOP) with everything under Wrapper black-boxed;-inst_top: TOP-bb_list: Block_A, Block_B, Block_C,

Block_D, Block_E, Block_F]

3) Merge of all results from (1) and (2).

Hierarchical blocks1) Top-level analysis (starting from TOP) with all blocks but one

under Wrapper black-boxed;-inst_top: TOP-bb_list: Block_A, Block_B, Block_C,

Block_D, Block_E, Block_F]2) Re-run (1) for all other internal blocks: A, B, C, D and E;3) Merge of all results from (1) and (2).

UNR Analysis: Divide & Conquer (III)


Hierarchical blocks with sub-systems1) UNR analysis can run on sub-systems made of several blocks;2) Key blocks, such as a central register bank, fsm or clock generator, can be included in all sub-systems;3) Running times can increase drastically, hence different configurations of the same blocks are needed;4) Single blocks can still be analysed individually and the results merged with other runs.

TOP

Wrapper

Block_B

GlueDFT

Block_C

Block_D

Block_E

Block_F

Block_GBlock_H

Block_I

Block_A

Example: sub-system ‘green’ (blocks D/H/I):-inst_top: TOP-bb_list: [blk_A, blk_B, blk_C,

blk_D, blk_E, blk_F,blk_G, blk_H, blk_I]

Configuration script


%ANALYSE_LIST = ( "$TOP.u_block_A" => [()],"$TOP.u_block_B" => [()],"$TOP.u_block_C" => [()],"01$TOP“ => [(@major_modules)],"02$TOP" => [(rm_element(\@all_modules,[('block_A')]))],"03$TOP" => [(rm_element(\@all_modules,[('block_B','block_C')]))]);

no black-boxes

As will be shown in the following slides, the divide & conquer approach is able to generate good results in reasonable times, but the chief difficulty is to configure all the runs, collect the results in an organized way and merge everything into a final database.

A script was developed to ease the set-up and handle all different configurations in a single run. Each individual UNR analysis is entered as an element of a hash, which contains the hierarchical level where it should start from, as well as an array of black-box modules.

Auxiliary array of modules can be created to be shared amongst the runs.

@all_modules array without modules block_B and block_Chierarchical level + optional ID(hash keys have to be unique)

Top-level results


0,00%

0,20%

0,40%

0,60%

0,80%

1,00%

1,20%

1,40%

Standalone blocks Sub-systems Whole designTop-level 1,23% 1,37% 0,38%

Unr

each

able

cod

e (a

vera

ge o

f all

met

rics)

Top-level unreachable code

The chart illustrates the amount of unreachable code identified in the whole design using 3 different approaches:

1. Divide & conquer of standalone blocks(~8 hours);

2. Divide & conquer of sub-systems(~12 hours);

3. Single run for the whole design(~10 days).

Block by block results: linear scale


0,00%

0,50%

1,00%

1,50%

2,00%

2,50%

3,00%

3,50%

4,00%

4,50%

Unr

each

able

cod

e (a

vera

ge o

f all

met

rics)

From the biggest (left) to the smallest (right) block

Unreachable code relative to the block sizeStandalone blocks

Sub-systems

Whole design

Block by block results: log scale


1

2

4

8

16

32

64

128

256

512

1024

Unr

each

able

cod

e (a

vera

ge o

f all

met

rics)

From the biggest (left) to the smallest (right) block

Unreachable code relative to the block size(normalized log scale)

Standalone blocks

Sub-systems

Whole design

Running times and explored properties


0

200

400

600

800

1000

1200

0:00:00

3:00:00

6:00:00

9:00:00

12:00:00

15:00:00

18:00:00

21:00:00

24:00:00

27:00:00

30:00:00

33:00:00

36:00:00

39:00:00

42:00:00

45:00:00

48:00:00

0 2000 4000 6000 8000 10000 12000 14000

Num

ber o

f exp

lore

d pr

oper

ties

Tota

l tim

e

Total number of holes per block/sub-system

Performance results

Time

Explored PropertiesThe number of explored properties seems

to increase drastically after a certain threshold, leading to sub-systems to be revisited and big blocks to be divided.

Conclusions


• The advantage of doing formal unreachability analysis is to reduce the time spent investigating whether a given hole is reachable or not. The tool findings need to be reviewed by the verification team before being waived. However, the tool indication helps to focus on the right holes.

• Information about unreachable code can be useful for the block designer, as it can potentially show areas with dead code that could be removed or parts of the code that are unreachable due to a bug.

• The Divide & Conquer approach showed itself manageable and capable of identifying a good deal of unreachable code. Unless a design is so small that could be analysed thoroughly by IEV, the D&C approach represents a good compromise.

• Knowledge of the design is necessary in order to define the sub-systems with some reasonable criteria, which need to be revisited in case of several explored properties.

• Effort was set to low (10s) but the total amount of properties to be proven can be so big that an increase in the effort would increase even more the running times. It is preferable to review the sub-systems and try to break them up.

…personal…portable…connected

The power to be . . .


Thank you

[email protected]

code coverage formal unreachability analysis

Documents

identified unreachable

location of unreachable

defensive code

dead code

unreachable coverage

low code coverage figures

functional coverage

coverage items