www.ischool.drexel.edu info 630 evaluation of information systems dr. jennifer booker week 4 –...

www.ischool.drexel.edu

INFO 630 Evaluation of Information Systems

Dr. Jennifer Booker

Week 4 – GQ(I)M and Ishikawa’s Tools

1INFO630 Week 4

www.ischool.drexel.eduINFO630 Week 4 2

Why Care About Measurement?

• To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts."

• We are seeking to identify things 1) which can be counted, and 2) which count toward achieving our goals


Reasons for Measurement

• Measurements are required by all major process and quality models (CMM, ISO 9000, etc.) for four good reasons:– To Characterize or understand the current

status of activities or products– To compare that understanding to our

objectives, and Evaluate whether the current status is “good” or “bad”


Reasons for Measurement

– To Predict future performance, based on past trends

– To form the basis for measuring Improvement• You won’t know if you improved if you don’t know

where you started!


Where do we get Information?

• Metrics are often used to support decision making (the Evaluate step in the previous slide)

• Decisions should be based on quantified information• To get that information, we calculate measures from raw

data called data elements• The data elements each come from a data source

DataSource

DataElements

Measures Information


How do we Choose What to Measure?

• A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement– It is based on GQM work by Victor Basili

(first reported circa 1988-89)


How do we Choose What to Measure?

• The GQ(I)M method uses ten steps to describe measurements systematically

• The steps don’t have to be followed in the order presented– Their main purpose is to help ensure that

measurements have been fully thought out, so you can start at the top, or the bottom, or the middle

– We’ll discuss them in a top-down approach, just to be linear


1. Identify Business Goals

• These are big, vague, lofty desired accomplishments or objectives for the organization:– Reduce cycle time– Improve customer satisfaction– Develop detailed process history– Respond to changing business environment


1. Identify Business Goals

– Reduce overhead– Improve competitive position– Increase market share– Improve product quality

• Think of what you’d find cited in a company’s annual report – those are often high level business goals


2. Identify Desired Knowledge

• Break down each goal into products, resources, and activities (processes) needed to meet that goal

• Think of questions like:– What activities do I manage or execute?– What do I want to achieve or improve?– To meet this, I will need to …


3. Identify Subgoals

• Set Subgoals (objectives) for each of the entities you manage

• What do you want to know about the results of step 2? What kind of information is important to you?

• How big, fast, expensive, complex, or much time will a process, product, tool, or resource take?


4. Identify Entities and Attributes

• Formulate questions to identify entities (document, product) created by your process, and the attributes of them you are interested in (size, quality, duration, cost)– Entities in this sense are often part

of the inputs, outputs, or process associated with an activity



• Don’t get too detailed at this point - just identify the type of information desired (what are you studying), and the subject of that information (what about it do you want to know)– Then once the entities have been described,

find the particular characteristics of interest (attributes)



Consists of…Holds...

Input(receives)

Output(produces)

<Process name>

entities entities entities

attributes attributes attributes

(products)

(traits)


5. Define Measurement Goals

• Form structured statements of the measurement goals for each attribute– This step is the heart of defining a metric in the

form of a sentence• Two types of goals:

– Active goals: reduce or improve something– Passive goals: identify, assess, understand



• Lower maturity organizations start with passive goals, then work on active goals– First measure existing trends, before predicting

improvements• Description of a measurement goal includes

the target entity, a purpose, a perspective, the environment and constraints



• It should include the quantity to be measured, the active or passive verb, describe the independent variable (e.g. time), and for active goals, quantify a desired amount or level– DO NOT report traits of individual people (fear

of judgment) unless it’s a known part of their job



• Also need to balance how often measurements are made– More frequent measurement gives finer

control, but excessive measurement wastes time and slows the process being measured

• Can measure per release, per component, or some basis other than time



• Examples of measurement goals:– Passive: “Measure the number of requirements which

changed each month.”“Identify the voluntary turnover rate per month for programmers and software engineers.”

– Active: “Reduce the defect rate of developed code over time to under 20 defects/KLOC”“Improve the percent of satisfied customers after 30 days of product use to 95% or more”



• The general format is: <verb> <measure> <qualifier(s)> [objective]

• Where the <qualifiers> indicate the scope or time frame of the measurements (a.k.a. independent variables), and [objective] is only given for active measurement goals


6. Quantify Questions and Indicators

• Pose questions to address your measurement goals (quantifiable ones, if addressing active goals)– Active: “Can we resolve customer emergencies, on

average, in under 24 hours?”– Passive: “How many requirements do we have at the

end of the Requirements Definition phase?”


6. Quantify Questions and Indicators

• Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle

• Identify indicators to show the results effectively, such as:– Pie charts– Bar graphs– Scatter plots, etc. (see Ishikawa later)


7. Identify Data Elements

• Identify the data elements and equation(s) needed to prepare (calculate) each indicator – E.g. “defect rate by module each month”

needs a table of defects found, for the last month, with the module each came from

• Determine the source for each data element


8. Define Measures

• Describe how the data elements will be used to produce the indicators (i.e. what is the equation for the measure)– Even the most obvious measure could be

defined many ways, so state the definition of each measure clearly• E.g. does your ‘turnover rate’ include all project

employees, or only salaried ones?


8. Define Measures

• Define exactly what you mean by each measure - use a checklist if needed to show what is and isn’t included in its definition– Include rules, assumptions, constraints, and

environment– Cite source if an unusual measure is used; or

if you made it up, explain why


An Aside

• The first eight steps of the GQ(I)M approach define traceability from the business goals to the exact definition of each measure, and the way it is calculated from its data elements

• The last two steps focus on the broader issue of planning a measurement approach for a project


9. Identify Measurement Implementation

• Analyze what measures are currently collected (if anything) and how they’re being used

• Diagnose how well the current measurements meet your goals – ask ‘what’s missing?’

• Act on implementing new measurements, possibly in a phased approach based on priorities


10. Prepare Measurement Plan

• Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project

• This is generally called a Metrics Plan or a Measurement Plan


Summary of Core Steps

• Goal (the big picture this is describing)• Subgoal (objective; why collect this metric)• Question(s) (answered by this metric)• Indicator (how display metric)• Measurement (the actual metric and its definition)• Data Elements (used to calculate various metrics)• Source (of each data element)


Indicators

• Indicators are the means used to present measures, such as charts, graphs, etc.– (yes, ‘indicator’ is an odd term for it, but

pretend it makes sense)


Indicators

• To choose a good indicator, consider: 1.The Amount of Data to be presented for each

interval (e.g. one measure at a time, or five different survey responses at once), and

2.The Number of Intervals to be shown, such as time units, modules of code, etc.


Indicators

• Different indicators are better at different Amount or Number characteristics– Consider also how your data will be presented

- in color or B/W, live or printed?– Will your audience see pristine originals, or

will it be copied a zillion times?


Indicators

• For graphs:– The X-axis of a graph (the horizontal line)

is the independent variable• Is often a <qualifier> you choose before the measurement,

such as time, severity, etc. • If X is time, describe how often measurements

are made (weekly, monthly, every release, etc.)

– The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure)


Pie Chart• The lowly pie chart is good for presenting a limited amount of

information attractively– % of customers satisfied and not satisfied– % of defects by severity at this moment

• Amount of Data: Shows a few data points (2-10)• Number of Intervals: One - it generally shows only one

moment in time, or one set of measurementsSales ($M)

1.2

2.4

1.1

1.7

4.6

West

Northeast

South

Midwest

East


Ishikawa’s Seven Basic Tools

• Developed circa 1950 for manufacturing production– Used widely in manufacturing quality control– Focuses on project level concerns; is this

batch good enough to accept?• Not very useful for research; has little

theoretical basis


Ishikawa’s Seven Basic Tools

• From a software perspective, these tools are often best for managers – focuses on identifying process and/or quality control issues– Not typically helpful for individual developers


1. Check sheet

• Used to gather data easily, consistently, and in a standard format– Helps to define key parts of a process, and

make sure they are all performed– Examples include code inspection checklist,

detailed test procedures• A check sheet used to help the quality of a

process or product is a “checklist”


2. Pareto Diagram• Used to identify problem areas - where to fix first; what are

the biggest fires to put out• Defects tend to cluster in portions of code e.g. plot % of

defects by the type of defect (logic, data definitions, etc.), and plot the total % of defects above it

• Must list categories in descending order of frequency

Pareto Diagram of Component Defect Rate

22.3

15.613.3

11.18.9

5.6

0

5

10

15

20

25

Interface Query Core Import Export Reports

Component

Def

ect

Rat

e (d

efec

ts/K

LO

C)

Helps look for the few components where most

defects reside

X axis must be a nominal variable


3. Histogram• A bar chart is used to break down data by an ordered

category (e.g. defect severity, satisfaction rating, etc.)• Can choose to put bars next to each other, or stack them.

Stack when they add to a constant (e.g. 100%), or when the total is also a useful measure

• Can show limited time spans, e.g. a few time intervals

No. of Problem Reports by Status

23

234

15 834

50

0

50

100

150

200

250

New Closed Pending Withdrawn Analysis Testing

Status

No

. o

f P

rob

lem

Rep

ort

s


3. HistogramProblem Reports in each Status over Time

0

100

200

300

400

500

Jan-02 Feb-02 Mar-02 Apr-02 May-02

Time

Nu

mb

er o

f P

rob

lem

R

epo

rts

Testing

Analysis

Withdrawn

Pending

Closed

New

Problem Reports in each Status over Time

0

50

100

150

200

250

300


Time

Nu

mb

er o

f P

rob

lem

R

epo

rts

New

Closed

Pending

Withdrawn

Analysis

Testing

Stacked and cluster bar

graphs for the same data set


3. Histogram

• A true histogram uses ranges of X axis values to show, for example, the number of events or data points in that range of values

Histogram

0

1

2

3

4

5

6

7

18 30 40 50 60 More

Bin

Fre

qu

ency

Example shows how many customers were in age ranges of 0-18, 19-30, 31-40, 41-50,

51-60, and 61+


4. Run Charts

• Purpose is to plot something important versus time• Often compare values to a desired or target value,

especially at higher process maturity levels (CMM Level 3 and up) and for comparing to active goals

• Special case: The “S” curve plots (cumulative % completion of something) versus time

Number of Defects Found by Severity

05

1015202530354045


Time

Nu

mb

er o

f D

efec

ts F

ou

nd

Minor

Major

Total

Usually uses a line chart


5. Scatter Diagram

• Is used to plot two measures against each other to see if there’s a correlation between them– E.g. Defect rate per module

versus module complexity, or productivity versus experience

– When we say ‘plot Blah versus Ick,’ Blah is the Y axis, and Ick is the X axis

Current Salary vs. Educational Level

EDUCATIONAL L EVEL

2220181614121086

CU

RR

EN

T S

AL

AR

Y

60000

50000

40000

30000

20000

10000

0


5. Scatter Diagram

• If there’s enough data, can try to add curve fitted lines – we’ll cover this in the regression analysis discussion

• This and control charts are the most powerful types of graphs for understanding your data– Many estimation rules and equations are

derived from scatter diagrams


6. Control Chart

• Is a key Statistical Process Control tool• Hard to apply to software development

– Specifications poorly defined– Each project takes a long time– Too many uncontrolled process variances– Process-quality relationship not well defined– Too many processes used– Rapidly changing technology

• There are many varieties of control chart


6. Control Chart

• Control charts are often used at very high levels of process maturity (CMMI levels 4 & 5) Control Chart: Percent Passed

Sigma level: 3

121110987654321

93.000

89.000

85.000

81.000

77.000

Percent Passed

UCL = 92.3343

Average = 85.0833

LCL = 77.8324


6. Control Chart

• Pseudo-control charts include:– The ‘u’ chart for defect rates by component, BMI,

etc. versus time– The ‘p’ chart for percentages, such as inspection

effectiveness or customer satisfaction rating• Upper and Lower Control Limits (UCL and LCL)

are +/- 3 sigma from the mean (average)– Can add a warning limit at +/- 2 sigma


7. Cause and Effect Diagram

• A.k.a. the Fishbone chart– Is the least used Ishikawa tool, in my experience – Is not an Indicator per se, just a tool for capturing

thoughts about a problem• Used with brainstorming to trace the causes of some

outcome or result (good or bad)• Ask “what causes that” or “what influences that” to

determine the major types of causes, then break them into more detailed events


7. Cause and Effect Diagram

• Sample fishbone diagram to analyze causes of Incorrect Deliveries

Note the choices of types of causes (communication, skills, transport, procedures) vary from one problem to the next

www.ischool.drexel.edu info 630 evaluation of information systems dr. jennifer booker week 4 –...

Documents

bad slide

linear slide

data source slide

business environment

jennifer booker week

previous slide decisions

measurement measurements

ishikawas tools 1info630