presh - metrics

8/4/2019 PresH - Metrics

1/21

"Quality Metric, Scorecards andDashboards" presented by DianeKulisek

October 11th, 200

[email protected];www.capatrak.com

(805) 522-5005 W / (805) 320-7879 C Page 1 of 2

Quality Metrics,Quality Metrics,

Scorecards andScorecards andDashboardsDashboards

Presented October 11th, 2008

Presented byDiane Kulisek

www.capatrak.com805-522-5005 O805-320-7879 C

[email protected] Pinewood Place

Simi Valley, CA 93065

October 11, 2008 "Quality Metrics, Scorecards and Dashboards"presented by Diane Kulisek

2

Quality Metrics, Scorecards and DashboardsQuality Metrics, Scorecards and Dashboards


2/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 2 of 2


3


TOPICS

Some Definitions What are Quality Metrics? What are KPIs? What are Dashboards? What are Scorecards?

Quality Management System Considerations What Do Standards and Regulations Require with Regard to

Quality Metrics? How do Quality Metrics directly support the Quality Management

System? Where in the QMS should Quality Metrics be established?

Tips for Selection of Quality Metrics

Scorecards and Dashboards Elements of an Effective Scorecard Elements of an Effective Dashboard Scorecard and Dashboard Pitfalls to Avoid

Quality Metrics References

Some Quality Metrics Tools

What willbeaddressed:


4


What are Quality Metrics?

What are KPIs?

What are Dashboards?

What are Scorecards?

SomeDefinitions


3/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 3 of 2


5


What are Quality Metrics?

A metric is a measure.

Quality is something a customer defines.

A Quality Metric, therefore, is a measure ofquality as defined by the customer.

NOTE 1: A customer might be definedas anybody with an expectation ofreceiving something of value inexchange for something else of value,either external to or internal to anorganization.

NOTE 2: Not all Metrics are QualityMetrics

SomeDefinitions


6


What are KPIs? KPI = Key Performance Indicator

There are MANY definitions:

Chosen factorsthat directly and indirectly influence theeffectiveness of a product or process.

A significant measureused on its own, or incombination with other key performance indicators, tomonitor how well a business is achieving its quantifiableobjectives.

A proxy measure of the successof part of anorganization, or a manager of that part. A type ofindicator. The future of the unit or person depends onachieving a satisfactory figure.

Also known as Key Success Indicators (KSI). Financialor non-financial metricsused to reflect the criticalsuccess factors of an organization.

NOTE 1: All KPIs are also Metrics, but not allMetrics are KPIs.

NOTE 2: Not all KPI Metrics are Quality Metrics.

SomeDefinitions


4/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 4 of 2


7


What are Dashboards?

A dashboard is a dynamic set ofindicators about the state of aprocess, piece of equipment, orbusiness metric at any specific pointin time.

A Quality Metrics Dashboard isanalogous to the dashboard in a caror the cockpit in a plane but, insteadof driving a car, the user is driving a

process.

SomeDefinitions


8

Quality Metrics, Scorecards and DashboardsQuality Metrics, Scorecards and Dashboards Dashboard Example 1(http://www.primedonline.com/PulseMonitor.htm )

SomeDefinitions


5/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 5 of 2


9

Quality Metrics, Scorecards and DashboardsQuality Metrics, Scorecards and Dashboards Dashboard Example 2

(http://www.enterprise-dashboard.com/sitemap/)

SomeDefinitions


10

Quality Metrics, Scorecards and DashboardsQuality Metrics, Scorecards and Dashboards Dashboard Example 3

(http://dashboardspy.com/templates-wireframe-coolblue.html)

SomeDefinitions


6/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 6 of 2


11


What are Scorecards? A scorecard is part of a broader corporate

methodology or management discipline andis a report card of how a given person,business unit or entity performed withrespect to certain goals over a previouslydefined period of time(such as quarterly orannually).

An Executive Scorecard is analogous to astudents Report Card.

There are two popular formats for BalancedScorecards (BSCs): 1.) Customer, Employee, Shareholder,

Process

2.) Customer, Learning/Growth, Financial,Internal Business Process/Operation

SomeDefinitions


12


What are Scorecards? (continued) A popular scorecard concept model

(http://www.balancedscorecard.org/basics/bsc1.html ):

SomeDefinitions


7/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 7 of 2


13


What are Scorecards? (continued) Another scorecard concept model

(www.capatrak.com):SomeDefinitions


14


What are Scorecards? (continued) Another scorecard concept model (detail):Some

Definitions


8/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 8 of 2


15


Scorecard Example 1

(http://www.primedonline.com/PulseMonitor.htm)

SomeDefinitions


16


Scorecard Example 2(http://www.hospitalsoup.com/public/scorecardshell.pdf)

SomeDefinitions


9/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 9 of 2


17


What Do Standards Require with Regard toQuality Metrics?

Compliance with ISO 9001 and ISO 13485 are addressed in thefollowing sections of the standard:

5.4.1 Quality objectivesTop Management shall ensure that quality objectives,including those needed to meet requirements for productare established at relevant functions and levels within theorganization. The quality objectives shall bemeasurable and consistent with the quality policy.

7.1 Planning of product realization In planning product realization, the organization shall

determine the following, as appropriate:a) quality objectives and requirementsfor the product;

c) records needed to provide evidence that the realization

processes and resulting product meetrequirements.

QualityManagementSystemConsidera-tions


18


What Do Regulations Require with Regard toQuality Metrics?

Compliance with the Code of Federal Regulations governing QualitySystems for Medical Device & Equipment/Software Manufacturers,21 CFR Part 820, is addressed in the fol lowing subparts and subsectionsof the Code:

820.20 Management responsibility(a) Quality policy. Management with executiveresponsibility shall establish its policy andobjectives for, and commitment to, quality.

(b) Organization.(3) Management Representative. shall have responsibility for:

(i) Ensuring that quality system requirements areeffectively established and effectively maintained inaccordance with this report; and

(ii) Reporting on the performance of the qualitysystemto management with executive responsibilityfor review.



10/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 10 of 2


19


What Do Regulations Require withRegard to Quality Metrics? (continued)

Compliance with the Code of Federal Regulations governingQuality Systems for Medical Device & Equipment/SoftwareManufacturers, 21 CFR Part 820, is addressed in the followingsubparts and subsections of the Code:

820.20 Management responsibility(c) Management Review. shall review the

suitability and effectiveness of the qualitysystem at defined intervals and with sufficientfrequency according to established proceduresto ensure that the quality system satisfiesthe requirements of this part and themanufacturers established quality policy andobjectives.



20


What Do Regulations Require withRegard to Quality Metrics? (continued)

Compliance with the Code of Federal Regulations governingQuality Systems for Medical Device & Equipment/SoftwareManufacturers, 21 CFR Part 820, is addressed in the previouslynoted and following subparts and subsections of the Code:

820.70 Production and processcontrols. General. Each manufacturer shall develop,

conduct, control, and monitor production

processes to ensure that a deviceconforms to its specifications.



11/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 11 of 2


21


How do Quality Metrics directlysupport the Quality ManagementSystem?

1.) Quality Metrics can be floweddown from the Quality Policy of anorganization into every majorelement of the Quality ManagementSystem and the Procedures thatsupport them.

2.) Quality Metrics can be flowedfrom Customer Requirements into

Product and Process Specifications.



22


Where in the QMS should Quality Metrics beestablished?

Break the Quality Policyinto discrete subparts. Establish Quality Objectivesfor each subpart of

the Quality Policy. Create Scorecards for Quality Objectives that

involve multiple processes. These will be used inQuality Management Reviews.

Establish Quality Metrics requirements withinOperating Proceduresthat support each QualityObjective.

When feasible, consider establishing Dashboardrequirements within Production Procedures,Work Instructions, or Process, Material andProduct Specificationsfor critical QualityMetrics.

When products or processes are outsourced, flowrequirements for Quality Metrics to Suppliers viatheir Contracts or Purchase Orders.



12/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 12 of 2


23


Tips for Selection of QualityMetrics

Use the S.M.A.R.T. criteria:

Tips forSelectionof QualityMetrics


24


Tips for Selection of Quality Metrics(continued)

Avoid being UN-S.M.A.R.T. :


13/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 13 of 2


25


Tips for Selection of Quality Metrics(continued)

Other helpful selection criteria(http://web.mit.edu/tdqm/www/tdqmpub/PipinoLeeWangCACMApr02.pdf):


26


Elements of an Effective Scorecard(Ref.: The Certified Manager of Quality/OrganizationalExcellence Handbook)

Tracks total Organizational Performance againsta Strategic Plan and is Management-driven

Represents longer term intervals (weeks,months, quarters)

Includes Goals for every Metric Balances Financial Considerations against all

Other Considerations (Quality, Schedule, etc.) Emphasizes Use of Metrics that are Leading

Indicators Note:

Leading Indicators Predict FuturePerformance

Lagging Indicators Report PastPerformance

ScorecardsandDashboards


14/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 14 of 2


27


Elements of an Effective Dashboard(http://www.datamanagementgroup.com/Resources/Articles/Articles.asp?offset=50/ )

Provides Tactical Indicators about the State of aProcess, Piece of Equipment or Business Metricand is Worker-driven

Represents Near Term Time Intervals (minutes,hours, days)

Information is Dynamic and is Real-time (or asclose to Real-time as possible)

Information is Very Visible and Is Used Closeto the Process that Generates it

Trend Lines are Used Goals Are Not Necessary but Action Limits Are



28


Scorecard and DashboardPitfalls to Avoid(http://www.dmreview.com/white_papers/library.html )

Measurement Pitfalls

Approach Mistakes

Technology Red Flags



15/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 15 of 2


29


Scorecard and Dashboard Pitfalls to Avoid(continued)

Measurement Pitfalls Information Overload (keep it simple)

Functional Bias (be crossfunctional)

Hindsight (look forward)

Limited Perspective (see the forest ANDthe trees)

Inconsistent Definitions (collaborate)

Irrelevant Metrics (use metrics that canbe controlled by the user or could

significantly impact action taken by theuser)



30



Approach Mistakes

Strategic Confusion (NO Scorecardswithout Strategic OBJECTIVES)

Tactical Confusion (NO Dashboardswithout Action Limits)

Destination Unknown (focus as muchupon where to go as to where theorganization has been)

No Executive Mandate (See Next Slidefor Explanation)



16/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 16 of 2


31


Scorecard and Dashboard Pitfalls toAvoid (continued)

Approach Mistakes (continued)ScorecardsandDashboards


32



Approach Mistakes (continued)

Misalignment (Prioritize, Harmonize,Integrate, etc.)

Big BANG (Avoid One Hit Wonders,Adopt Metrics as the Way of DoingBusiness Throughout the Organization)

Incorrect Results a.k.a. SolutionSuicide (Use only Validated Data from

Qualified Sources) Unguided Exploration (Train Everybody

and Make How-To Information ReadilyAvailable)



17/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 17 of 2


33



Technology Red Flags

Complex User Experience (Again, Keepit Simple, Avoid Esoteric Metrics,Unnecessary Barriers to Access,Challenging On-Line Navigation, etc.)

Only Used for ad hoc Research (Again,Avoid One Hit Wonders, Adopt Metricsas the Way of Doing BusinessThroughout the Organization)

No Knowledge Retention (Collect andUse Lessons Learned)



34



Technology Red Flags (continued)

Inflexible (Enable the Metrics to Changeas the Organization Changes withoutLosing Data Integrity or Access to PastMetrics Information)

Segregated (No standalone systems likeExcel spreadsheets, integrate with other

existing systems like MRP and CRM,include financial reporting)



18/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 18 of 2


35



Technology Red Flags (continued)

Dead-end Results (Finish What YouStart! Maintain Forward andBackward Traceability for InformationUsed and Actions Taken)

Outdated Information (use near real-time data to support as many metricsas possible, establish alerts for

instances where action limits areapproached)



36


Scorecards and Dashboards When All ElseFails(http://www.dummies.com/WileyCDA/DummiesTitle/Excel-2007-Dashboards-Reports-For-Dummies.productCd-0470228148.html)


Excel 2007 Dashboards& Reports ForDummies

By Michael Alexander

ISBN: 978-0-470-22814-2

Pages: 336 PagesPub. Date: March 2008Price: $24.99


19/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 19 of 2


37


Quality Metrics References havebeen cited throughout thispresentation but are summarized

on the next few pages

Dilbertisms: www.comics.com (paidmembers can sort comics by key words)

Definitions: Do a Google search usingDefinition of ____ (www.google.com )

QualityMetricsReferences


38


Quality Metrics References Summary(continued)

Dashboard Examples: http://www.primedonline.com/PulseMonitor.htm

http://www.enterprise-dashboard.com/sitemap/

Scorecard Models and Examples: http://www.balancedscorecard.org/basics/bsc1.ht

ml

http://www.capatrak.comhttp://www.primedonline.com/PulseMonitor.htm

http://www.hospitalsoup.com/public/scorecardshell.pdf



20/21


October 11th, 200


(805) 522-5005 W / (805) 320-7879 C Page 20 of 2


39


Quality Metrics References Summary

(continued) Tips for Quality Metrics Selection:

http://web.mit.edu/tdqm/www/tdqmpub/PipinoLeeWangCACMApr02.pdf

Elements of an Effective Scorecard: Ref.: The Certified Manager of

Quality/Organizational Excellence Handbook fromASQs Quality Press

Elements of an Effective Dashboard: http://www.datamanagementgroup.com/Resource

s/Articles/Articles.asp?offset=50/ Scorecard and Dashboard Pitfalls to Avoid

http://www.dmreview.com/white_papers/library.html

http://www.dummies.com/WileyCDA/DummiesTitl

e/Excel-2007-Dashboards-Reports-For-Dummies.productCd-0470228148.html



40


QualityMetricsTools Youcan Put ToUseImmediately

The following free templates will be provided toyou via email upon request (send request [email protected]) and are also availablefrom the CAPAtrak Download pages atwww.capatrak.com. These include:

Strategic Quality Plan Template Operating Procedure Template

for Inclusion of Quality Metrics Scorecard Template

Dashboard Template A .pdf version of the handouts for this

presentation is also available for download fromthe CAPAtrak download page.


21/21


October 11th, 200

[email protected];t k

THANK YOU!THANK YOU!

May I answer YourMay I answer Your

Questions?Questions?Presented by

Diane Kulisek, President

www.capatrak.com805-522-5005 O

805-320-7879 [email protected] Pinewood PlaceSimi Valley, CA 93065

presh - metrics

Documents