A Validation Methodology for Graphics Processors

Author: Valtteri RantalaSupervisor: Prof. Raimo KantolaInstructor: M. Sc. Harri Syrjä

Thesis Work Presentation 10.10.2006

Slide 2

Content

•Background•Objectives & methodology•Graphics processors & design process•Validation in general•Validation methodology•Validation of the Vector Graphics unit•Conclusions•Future study

Slide 3

Background

•ATI technologies Finland (formerly Bitboys oy) designs IP cores for mobile devices

•Quality is extremely important in processor design•Errors are extremely costly to repair after the design is implemented

in silicon•Customer dissatisfaction

•Company saw it necessary to reinforce the quality process•Validation was a part of the quality assurance that needed

improvement

Slide 4

Objectives & methodology

•Objectives To develop the validation process for graphic processors To specify documentation for the validation process•To find suitable testing techniques to be used in the validation

process

•Scope•Only validation aspect of quality assurance is inspected•Only validation of one unit is inspected in the thesis• Individual test cases are not introduced

•Methodology•Literature survey•Discussion with experts

Slide 5

Graphics processors & design process

•Graphics processors•Used to accelerate the calculation tasks in

different stages of the 3D pipeline•Specialised in calculating computer graphics

algorithms•Contains a set of arithmetic units that are

design for specific tasks •The structure of the graphics processors is

based on the use of units

Direct-X 3D-pipelineIllustrative structure of a graphics processor

Slide 6

Graphics processors & design process

Design process for graphics processors Graphics processor roadmap definition

Defines what kind of products are to be developed

Core specification Feature set, architecture, size,

performance, quality requirements Unit specifications

Usage and operational details Validation TLM – Transaction Level Model Verification

The models are verified against each other and in some cases against the specifications

•RTL – Register Transfer Layer

Design flow of a graphics processor

Slide 7

Validation in general

•Validation is a part of the quality assurance•The goal of the validation is to discover defects as early as

possible •Validation is defined as “The process of evaluating a system or

component during or at the end of the development process to determine whether it satisfies specified requirements”(IEEE1012)

•Consists of analysis, evaluation, review, inspection and testing of products

•Different validation approaches•Walk-through•Formal methods•Test development•Prototyping

Slide 8

Validation methodology

•Validation process was proposed•The validation process is divided into five different phases

Product Validation Plan Unit Validation Plan Test Case Design and Procedures Validation Report•Validation Feedback

•Validation sub process is executed for each unit•Each phase has input and outputs documents•Each phase has specific tasks

Slide 9

Validation methodology

•Product Validation Plan•To employ the validation resources efficiently, to monitor, to

control, and to allow the identification of each participant’s role and responsibility

•The purpose of this document is to describe the validation program for the product on a high level

•Validation Program Management•Describes the program schedule, milestones and goals for each

milestone, defect and reporting management, and staffing and risk management

•Concept Level Validation•Describe on a system level what the system must be able to do

• Performance, power consumption, acceptance use cases and APIs

•Hardware Level Validation•Describes feature level validation •Hardware features are introduced on a high-level and the overall

validation methodology for each feature group is described

Slide 10

Validation methodology

•Unit Validation Plan•For every unit•Describes how the specific unit will be validated•Contains to tasks: feature analysis and validation goal analysis

•Feature analysis•All the features that are to be implemented are analysed•All features must be: clear, unambiguous, internally consistent

and testable•Priority class is given for each feature

•Validation goal analysis• features are analysed and the validation goals for each feature is

presented Feature spreadsheet is made

Target feature, validation items, methodology description, number of test cases

Slide 11

Validation methodology

Test Case Design and Procedures All testing work is done in this phase

Test case execution Result analysis Defect reporting

Test Case Design Principles•Test cases should uses the same coding standards as product

developers•Only to test a feature•Size is 128x128 pixels or smaller, RTL is 100 to 1000 times slower

to run than TLM designs.•Test Execution Framework

•Test environment•Validation environment

Slide 12

Validation methodology

Test environment•All test cases are executed in test environment Contains three implementations: Reference, TLM, RTL Each implementation generates a bitmap image•Third party software implementation can be used as a reference •Compiles test cases and executes them in hardware models Command stream input is used to hold register level data, which is

used as an input to hardware models

Slide 13

Validation methodology

Validation environment•The fully automated validation environment

Execute tests•Compare images from different sources•Produce a report from executed test cases

Validation environment

Report structure

Slide 14

Validation methodology

•Defect Reporting•Defect reporting system was used (JIRA)

•Centralized location, defect count, reporting, issue handling

•The following information should be available for each defect Defect Id, date of detection, product/project id, defect criticality Location of the defect (file name, code line if possible) The version of the file/design/release Description of the defect Name of the test case showing the defect

•Proper reporting = faster fixing times

Defect handling diagram

Slide 15

Validation methodology

•Validation Report•The purpose of this phase is to give a good understanding of the

current validation status•Readership is all stakeholders•Generated weekly basis -> produces history data

•Validation report contains the following information: Status tracking of various variables

Test case implementation progress Test cases passing rates for different design models•Defect tracking results

Test case coverage The goal is to show that all the features are tested

•Code coverage analysis•Statement coverage is used•Coverage for each file

Slide 16

Validation methodology

•Validation Feedback

•The purpose is to get information about how well the process has worked

•The Feedback is analysed and recommendations for improving the validation process are done based on the feedback

•Collecting Feedback

•From the process results Low failure rate/high pass rate, defect density, inconsistent results

•To make questionnaires and interview different stakeholders

•Change Management

•To have a planned approach to change implementation•Major/minor changes

Slide 17

Validation of the Vector Graphics unit

•Validation process was used to validate the Vector Graphics unit•All validation process phases and tasks were executed

•The feature analysis revealed a few errors in specification documentation

•Test case design based on the unit specification •The test case principles guide the designing of the test cases•Code coverage analysis revealed unspecified features•Quality issues: what is accurate enough?

Slide 18

Conclusions

•Importance of the validation was realised•Written process for the validation of the graphics processor with

documentation templates•Predictability of the validation process was improved due the

validation reporting and proper planning•No major rework was made in the project•Defects found in TLM after validation dropped from 30% -> 5%•There is now real history data to be used in the future projects•The documented progress increased confidence towards the

product

Slide 19

Future study

•The work for improving the validation process will continue after this thesis

•The areas for future work are the areas that received most negative feedback

•The use of validation metrics and the use of history data must be stressed in future projects

•The quality of test vectors will be stressed in future projects

Slide 20

Any questions?

Slide 21

Thank you!