LeanSigma® Facilitator Training

Module 12 – Measuring Success

Topics

Measuring Success

• Process performance measures……………………………………… 5 - 17

• Process capability ………………………………………………………… 18 - 23

• Program performance measures ………………………………… 24 - 34

2

Lean Sigma has six steps to optimize processes.

3

Introduction

• We’ll look at several types of measures:− Process performance: cycle time, KPIs, DPMO and yield

− Process capability: capability indices

− Program performance: pace, cost savings, implementation progress

• Selected process measures should follow from CTQs

• We want quantitative measures of a process because they’re much more objective than qualitative or ad hoc data.

• Performance data can help us to set a baseline, so we know when to intervene with more process improvement work.

4

Cycle time is our most common measure of process improvement.

• We create an estimate of future state cycle time in kaizens based on the improvements we plan to make.

• After the future state is implemented, you’ll measure the actual cycle time improvement.

• This validates how much time we’ve redeployed to other work. You’ll capture this measure in the Lean Sigma PMO

5

Annualized Labor Hours Saved

IMS FTE Reduc-tion

Outsourced FTE Reduction

Capacity redeployed

You will receive a demo of the PMO within the next few weeks.

Other common process improvement measures

• Reduction in lead time− May support a reduction in DAP; but often process improvements

must first be implemented in other, related processes

• Process improvement measures are often not measured on an on-going basis, but rather before and after process changes (2x)

6

VA Tas

ks

Non-V

A Ta

sks

Delay

s

Loop

back

s

Hand-

Offs

Decisio

n Po

ints

Lead

Tim

e

Cycle

Tim

e

0%10%20%30%40%50%60%70%80%90%

100%

Percent Improvement

Typical kaizen event measures

Key performance indicators (KPIs) are tracked to monitor the health of the process over time.

• Identify the “vital few” measures that tell the story of process performance

Define Collect Analyze Report Take Action

On-time delivery, query responsiveness, defect rate, throughput, etc.

Clear metrics definitions are important!

• Each KPI must have an Operational Definition:− An exact definition of the measure

− How it should be measured

− Measurement frequency

− Who should measure

− The threshold or expected service level

• Reporting: KPIs are evaluated in 30 / 60 / 90 day post-kaizen reviews.

• KPIs are monitored on an on-going basis by Process Owners, who are accountable for taking the appropriate action when results fall below corresponding thresholds

8

Defects per unit and per million opportunities are common process measures.

Term Definition

Defective Any transaction that contains at least 1 defect (error)

Defect Any part of a single transaction that has an error

Opportunity Any part of a single transaction that can cause a defect• A transaction could have any number of opportunities contained within it, a measure of

complexity• Every chance to do something either “right” or “wrong”

DPU Defects Per Unit – Calculates the percentage of the time in decimal form how many defects occurred out of the total transactions evaluated

DPMO Defects Per Million Opportunities – Calculates the number of times out of a million a defect would occur based upon the total opportunities for error

Evaluated # Total

Defects of # TotalDPU

000,000,1*iesOpportunit Total

Defects of # TotalDPMO

9

Calculating DPMO

• A project team is working on improving invoice accuracy

• The number of defect opportunities per invoice is 10. The team evaluated 100 invoices and discovered 35 defects

DPMO = (# of defects x 1,000,000) (# of Defect Opportunities/Unit) x # of Units

DPMO = 35 x 1,000,000 10 x 100

• “Six Sigma quality” means DPMO ≤ 3.4

10

Calculating DPMO

• A delivery process of a courier company has 3 defect opportunities (timeliness, accuracy of delivery, accuracy of invoice)

• The company has delivered 5000 mail pieces, 8 of them were too late, 3 of them delivered to the wrong address, and 2 had a wrong amount on the invoice

• The DPMO is

866000,000,1*5000*3

238

DPMO

11

Process Yield Concepts

• LeanSigma Yield concepts are:

−Throughput Yield – one process step: ratio of defects vs. defect opportunities

−Rolled Throughput Yield (RTY) – multiple process steps: multiplication of Throughput Yields

−Normalized Yield – multiple process steps: root of the multiplied Throughput Yields (“average process yield”)

• “Traditional” Yield concepts:

−First Time Yield – ratio of defectives vs. units processed

−Final Yield – ratio of defectives (after rework) vs. units processed

12

First Time Yield

• FTY – the percentage of units that pass through an operation without any defects

• If you describe the yield as 80%, the true performance of the process is not measured

• FTY = 20%, therefore the percentage defective = 80%

13

FTY + p(d) x 100% = 100%

Percentage Defective

“Traditional” Yield Metrics

• First Time Yield (FTY)

• Final Yield (FY)

Focus on Defectives and units

No. of units: 1057

No. of defective: 58

FTY: 94.5%

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

No. of units: 1057

No. of defective: 45

FY: 95.7%

Rework

Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

14

FTY Example

• 1056 invoices were processed in one day. 924 passed and 132 failed for various reasons

• # of bad units + # of good units = Total # of units

• # of bad units can be expressed as a probability of a defective, p(d): or a percentage defective: p(d) x 100% = percent defective

15

%1001056

132

1056

924

%100%5.12%5.87

FTY + percent defective = 100%

FTY = 87.5%, percent defective = 12.5%

LeanSigma Yield Metrics

• Throughput Yield (TY)

− Focus on Defects and defect opportunities

No. of defect opportunities per unit: 4

No. of units processed: 268

No. of total defect opportunities (4x268): 1072

No. of defects: 34

Throughput Yield: 96.8%

Step 1

16

Rolled Throughput Yield

• What is the probability of producing this product error free?

17

Invoice Generation

Invoice Processing

Billing Output

TY = 97%

Op1 x

TY = 98%

Op2 x

TY = 91%

Op3= 86.5%

86.5% probability an invoice will pass through defect-free

How can we establish thresholds for selected process measures?

• Thresholds or “specifications” should be established for all KPIs to flag the need for adjustment to the process

• Specifications are usually based on:− Historical results

− Compliance/regulatory requirements

− Benchmark data

− Management judgment

• In most cases, IMS specs are one-sided limits defined by internal customers

− Coding Accuracy: Error (defect) rate < 1%

− Panel fulfillment > 90%

• Specifications are often documented in Service Level Agreements (SLAs)

18

How can we determine if our process measures can routinely comply with established specs?

• Process capability is the inherent ability of a process to meet the specifications− The performance of the process when it is stable and predictable,

or operating in a state of statistical control, i.e., the mean and the variation in the process are not shifting

− Assumes no “special cause” variation, e.g., a “market event”, or a hurricane and resulting production delays. Verify this!

• Process capability measures provide insight as to whether the process has:− A centering issue relative to the specifications− A variation issue− A combination of centering and variation issues− Inappropriate specifications

• Process capability measures provide a baseline assessment

19

When specifications cannot be changed, actions must be taken to improve process capability

• LSL = Lower Spec Limit• USL = Upper Spec Limit• Target is usually the

midpoint between the specs

20

Standard deviation describes how far results in a process vary from the mean result.

• Processes with higher standard deviations have more variation.

• They are less predictable and less stable than healthy processes.

• Processes with less variation will have a higher “sigma level” and fewer defects.

21

Variation(Dispersion)

1

)( 2

n

XXs

NormalDistribution(“Bell”)Curve

StandardDeviation =

We assess process capability using process capability indices.

• Cp = =

− What is possible if your process is perfectly centered

− The process potential

• Cpk = minimum ( , )

− The reality of your process performance

− How the process is actually running relative to the specifications

− The “k” is the penalty for being off-center

specification width

process width

(USL – LSL)

6s

(USL – X)

3s

(X - LSL)

3s

22

The “sigma level” of the process provides another measure of process capability.

• σlevel = minimum ( , )

− Number of standard deviations between the center of the process and the nearest specification

• “Six-sigma quality” means:− Cp ≥ 2.0

− Cpk ≥ 1.5

− σlevel ≥ 4.5

(USL – X)

s

(X - LSL)

s

Process capability should be periodically reassessed, especially following the implementation of process improvements.

23

Program metrics help us monitor the health and benefits of the LeanSigma program.

• Are we achieving our mission to deliver more value to our customers and stakeholders by identifying and implementing measureable process improvements?

• Regular Lean governance meetings help us: − Align on priorities and targets− Validate results meet expectations− Ensure support− Mitigate risks

24

Costs:• Lean facilitator & SME time• Travel $• Implementation costs

Benefits:• Enable IMS to deliver on

strategic priorities• Transform the culture

Lean program reporting summarizes results from all Lean events.

• Cadence

− Pace of events by month, department & geography

• Benefits

− Net EBIDTA savings

− Annualized hours saved (capacity redeployed, FTE savings)

− Percent improvement cycle / lead time, delays, etc.

− Highlights of exceptional events, case studies

• Implementation

− Percent of identified savings removed from budget

− Percent of kaizen action items completed

• Focus areas

− Prioritization of processes based on exec sponsor discussions

− Pipeline of Lean events

• Cultural entrenchment

− Number of employees trained / certified

− Number of employees participating in kaizens

− Communications

25

95% of reporting is enabled by web queries against the Lean PMO

26

Sample DashboardU.S. Events by Month

n= 110completed

events

Process Focus to Date

Annualized Labor Hours Saved= 31,086

IMS FTE Re-duction 8,040

hrs; 26%

Outsourced FTE Reduction

7,236 hrs; 23%

Capacity re-de-

ployed15,810 hrs; 51%

2012 U.S. EBIDTA Contribution: $817K*100% of savings realized

Production & Data Coll.

26%

Del ivery & Support

29%

PD / Stat Svcs3%

US BU6%

HR26%

Finance6%

GPS2%

Legal1%

Feb Aor Ju

nAug Oct

Dec Feb

Apr Jun

Aug Oct0123456789

pla

nn

ed

2011 2012

Training grants23%

Delivery & support

46%

Producton & Data Coll

31%

80.23% of all U.S. kaizen action items completed

Cost Savings Detail Example

27

Qualitative Reporting Sample

28

Case Study: U.S. Market Research Client Support

29

February 2011Sponsor: Manny TriggianoProcess Owner: Brenda CapobiancoLean Process Lead: Carissa WaltenbaughTeam: Kevin Downing, Chris Davies, Cathy Pileggi-Jefferson, Katrina Kulp, Kyle Burke, Michele Agnoli, Joy Cross, Gerard Avillo, Lisa Cooney, Jeff Northern

Key Kaizen Findings

Issues Noted Process Changes

Some CSAs less able to handle complex questions.

CSAs sometimes call multiple times to refine the client question.

CSAs sometimes do unnecessary research.

Triage client questions and assign CSA resources to a Level 1 pool (simple) or Level 2 pool (complex).

Create a checklist to guide CSAs when refining details of a client’s question.

Create a decision tree to identify what research is required for a particular question. Improve how info is organized on ARC.

90 Days After the Kaizen

Clients are assigned a CSA even if they call only a few times a year.

Pool CSA resources so questions are answered by the first available CSA.

A pilot was completed to test the central support pool for smaller clients.

The centralized model went live June 13, 2011.

QMS was enforced as a central information system.

Tier 1 (simple) and 2 (complex) resource pools were created.

Checklists and decision trees were created to clarify info needs.

Case Study: U.S. Market Research SupportFindings 15 months after implementation

30

Results:• The central service center remains in place successfully.• Clients talk to CSAs directly more often.• Very few clients have questioned the change.• Three positions were eliminated.• In addition, the team took on additional work without adding

headcount. (SMART roll-out). • Cycle time savings were estimated at 65% initially. The savings

observed are about 50%.

Critical factors made the change possible: Pilot to test client feedback Excellent communication to the team and clients about changes Extreme organization and focus

Improve process workflows for custom deliverables

Improve quality & consistency of RFP responses

Reduce time spent screening & validating data

Reduce admin burden for GTM resources

Reduce time to implement

Reduce admin burden

Improve the process for IC implement-ations

APR 16

Focus areas – Completed Events

MAR 6-8

Resu

lts

Obje

ctiv

es

MMS Screening & Validation

MAS RFP Response mini

kaizen

Billing Initiation Process

MAS Custom Project Process

Performance Analytics

Deployment Standardization

Opportunity Management /

CRM Improvement

SAS Incentive Compensation

Implementation Process

MAY 1-3 MAY 15-17 MAY 15-17 JUN 11-12 JULY 17 -19

31% reduction in cycle time

Process improve-ments imple-mented

Identified 3 improve-ment work streams

Results to be rolled out at July 10 report out

Delivery time reduced by 25%

Eliminated time spent entering detail into early stage opps

31

Process Priorities – Sample from Operations

32

Gather / enter customer-specific order requirements

Collect dataPerform

statistical data transformation

Manage offering

production

Plan / QC the order

Perform delivery supportStandardize / cleanse data

Schedule / run customer-specific

production

Create output / deliver order

Track and update order

Define & communicate order

management procedures

Perform customer-specific

QC on requirements

Perform quality control on final

output

Build / Deliver the order

Test & certify supplier

Receive, cleanse & bridge/code

transaction inputs

Create / maintain statistical

methodology

Define & communicate

product creation procedures /

methodologies

Add new supplier to database

Perform transaction quality

assurance

Define / perform global quality management

Schedule production

Establish ongoing data collection

process

Build transaction database

Project / impute transaction data

Execute production jobs

Maintain reference database

Validate projected data

Manage rerun / rework

Perform change management

Maintain product related data

Maintain historical data

Perform in-process

production quality control

Provide customer training

Define & communicate

customer service & support procedures

Close loop to Marketing & Sales

Receive / escalate customer inquiry Service Center

Receive / escalate customer inquiry

Call Center

Analyze & resolve inquiries / complaints

Provide field services and

support

Manage credits & invoices

Process changes & maintain customer order information

Maintain customer support systems

and data

Measure & report customer delivery

performance

Manage customer delivery vendors /

subcontractors

Manage customer delivery information

systems

Data Collection Production Delivery

High Priority Process

Evidence of Cultural Entrenchment

• Lean continues to be a high priority for Seyed and teamLeadership

• U.S. has held more Lean events than any other country (110 events since Feb 2011)

• 30 additional events are in the pipelineLean Events

• More than half of all certified Lean facilitators are in the US

• 19 certified and 48 in progressSkill Development

• Over 25% of Lean’s global savings come from the U.S.

• Implementation is strong (>80% kaizen action items complete)

Results

33

Lean Measurement & Reporting Tools

• Lean Event Executive Summary Template:− http://arc.imshealth.com/gm/document-1.9.1639940

• Guidelines for 30 / 60 / 90 Day Reviews− http://arc.imshealth.com/gm/document-1.9.2276470

• Lean PMO− http://arc.imshealth.com/gm/leansigmaroadmap

• PMO Users Guide:− http://arc.imshealth.com/gm/folder-1.11.1540558

• Project Statistics (PMO Web Query):− http://arc.imshealth.com/gm/folder-1.11.1517854

• Guidance on standard metrics and metrics-monitoring tools− Under development. Contact Tamra Kirkpatrick.

34

US & Canada: ISO Compliance

• In the US and Canada, the Kaizen management process is maintained in compliance with ISO standards

• Facilitators are required to comply with the Kaizen Management procedure− http://arc.imshealth.com/gm/document-1.9.2241809

− The procedure is written to provide a great deal of flexibility in selecting, planning and conducting kaizen events

• Process effectiveness is assessed:

• Results are reviewed in quarterly internal management reviews, and periodically, by external ISO auditors

35

Number of kaizen events per quarter, based on a rolling

four quarters15≥