welcome, examiners!

Welcome, Examiners!

Washington State Quality Award

Examiner Training

2010

Thank you!!

• THANK YOU for your time, your efforts and expertise!

• WSQA’s mission: Improving the way we live, learn, and work in Washington by helping organizations improve through the use of the Baldrige Criteria for Performance Excellence

2

Objectives For This Class

• Learn to examine a WSQA application through all stages of the assessment process

• Do this with confidence • In yourself as an Examiner and • In the assessment process

3

Agenda: Session 1

1. WSQA organization overview2. Baldrige criteria overview3. WSQA Assessment process4. Online scorebook preview5. Independent review,

step-by-step• Preparing• Key Factors• Process review• Results review

Mostly exercises

Mostly presentation

4

Agenda: Session 26. Ethics, code of conduct7. Consensus, step-by-step8. Key Themes

9. Site visit preview10. Final scorebook preparation11. “Lite” vs. “Full”12. Summary

Mostly exercises

Mostly presentation

5

Introductions

• Name, organization, what you do• Questions and expectations?• What do you like to do when

you’re not working?

6

Workbook contents

Table of ContentsSession 1 Slides TAB 1Session 2 Slides TAB 2Ethical Standards and Conflict of Interest Determination TAB 3Considerations for Small Organizations TAB 4Comment Guidelines and Scorebook Editor TAB5Case Study TAB 6Scorebook and Feedback Report TAB7Independent Review Blank Worksheets TAB 8Sample Evaluations TAB 9Homework Instructions and Code of Conduct Exercise TAB10

7

Signed Ethical Statements

• Collect Signed Statement

8

For a Successful Course…For a Successful Course…

• Be an active participantBe an active participant• Give only constructive feedbackGive only constructive feedback• Be courteous to your colleaguesBe courteous to your colleagues• Honor diverse opinionsHonor diverse opinions• Seek to understandSeek to understand• Ask questionsAsk questions• Have fun!!Have fun!!

9

1. WSQA organization overview

10

What is WSQA?

• Created by State law to:• Promote excellence• Recognize achievement• Educate and train

• Patterned after the Baldrige National Quality Award• Awards presented annually by the Governor • Awarded to private, public, and not-for-profit

organizations in manufacturing, service, education, and healthcare

11

WSQA VisionBe recognized as a premier resource for advancing

organizational excellence.

WSQA Mission

Make Washington State a better place to live, learn, and work by helping organizations achieve superior results through the use of the Baldrige

Criteria for Performance Excellence.

12

Funding

WSQA is a 501(c)(3) non profit organization supported by

• Corporate donations• Individual memberships• Workshop , collaboratives and conference fees• Many volunteer hours

13

Learning Opportunities

• WSQA offers workshops• Introduction• Self Assessment• Case Study• Preparation for Application

• Annual conferences• Consulting• Performance Excellence Collaborative• Become an Examiner

14

Who is WSQA?

• Examiners• Lite Assessment• Full Assessment• Certification follows assessment

• Senior Examiners• Level 3 Team Leads• Scorebook Editors

• Panel of Examiner & Process Development (PEPD) Mentors• Panel of Judges• Board of Directors• Executive Director and support team

15

2. Baldrige criteria overview

16

Objectives

• Learn a little Baldrige history• Get a high-level understanding of the criteria• What’s in the booklet?

• YOUR EXAMINER TRAINING INCLUDES ATTENDANCE TO INTRODUCTION TO PERFORMANCE EXCELLENCE WORKSHOP!

17

Baldrige award history

• Created by law in 1987, named for Malcolm Baldrige, Secretary of Commerce from 1981-1987

• Purpose is to improve American productivity and quality

• Originally just for manufacturing, now includes education, healthcare, nonprofit organizations

• Public-private partnership

18

Public / Private Partnership

Private Public AffiliatedPrivate Public AffiliatedPrivate Public

Foundation

Board of Examiners

U.S. Dept of Commerce

NIST State Award Programs

Trade Associations

Foundation


ASQFoundation


Foundation


Foundation


Foundation


Board of Examiners

Foundation

Board of Overseers

Board of Examiners

Foundation

Private

Foundation

Board of Examiners

Private

Foundation

Board of Overseers

Board of Examiners

Private

Foundation

Award Recipients19

Baldrige Framework - A Systems Perspective

Leadership Triad

Results Triad

Supportive Underpinning

2StrategicPlanning

1Leadership

3Customer and Market Focus

4Measurement, Analysis, and Knowledge Management

5Workforce Focus

7Results

6Process

Management

Organizational Profile: Environment, Relationships, and Challenges

20

2121

Core Values and Concepts

• Visionary Leadership• Valuing WF Members

and Partners• Managing for

Innovation• Focus on Results and

Creating Value• Customer-Driven

Excellence

• Agility• Management by Fact• Systems Perspective• Organizational &

Personal Learning• Focus on the Future• Societal

Responsibility

Criteria Characteristics• Non-prescriptive and adaptable• Created and improved through a consensus-building

process with the user communities• Capture the leading edge of validated management practice• Changes incremental in the short term, but significant over the long

term

• Support improvement in overall organizational performance practices and capabilities

• Focus on sustained improvement in results• Facilitate communication and sharing of best practices

22

Six-Step Evaluation Process

23

Ground yourself in the Criteria Item

24

**Item Question Organization**

Glossary

25

Understanding the Questions

• The quality of the feedback provided depends on understanding of the meaning and significance of the questions

• The Criteria book provides aids to understanding

• A Glossary of Key Terms is provided; Terms with definitions are identified with CAPs

• Open your criteria, find a few Key Terms and look up the definition

• Notes are provided to clarify key requirements, give instructions, and indicate important linkages

• Read N1 on Item 1.1 (page 7) to find an expectation concerning the applicant’s Vision

• Category and Item Descriptions discuss the purpose for the questions and comment on expectations for answers

• Read the first Comment in Section 1.1 and consider how it will affect the applicant’s discussions in 1.1

26

3. WSQA Assessment Process

Objective:

Introduce the Assessment process

27

What Examiners Do…

Analyze Response

Read Criteria

Read Applicant Response

Meet with co-examiners to

consolidate analysis

Visit the Applicant to clarify/verify

Co-write an analysis report

Record Analysis

28

Stages of the Assessment Process

No

Receive applications

Stage 1:Independent Review

Individual Scorebooks

Stage 2:Consensus Review

Consensus Scorebook

Lite or Fullapplication?

Selected forsite visit?

Full

Judges recommend award level to WSQA Board

Lite

Stage 3:Site Visit

Yes

Feedback report

Executive Briefing

WSQA Symposium & Awards Ceremony

Final product

Editor updates

scorebook

Lead & Mentor meet with Judges Panel

WSQA updates

scorebook

Lead & Mentor meet with

Judges Panel

Consensus Scorebook

Site visit scorebook

29

Preparing for independent review

• Read the criteria• Read the application• Familiarize yourself with the online toolhttp://www.wsqa.net/extraining.php• Check in with Team Lead or Mentor after preparing

a few Key Factors

30

Select the relevant Key Factors for the Item

31

4. Independent Review:Key Factors

Objectives:

1. Know how to prepare for independent review

2. Be able to identify and record Key Factors

32

Key Factors topics

• Role and importance of the Organization Profile

• What are Key Factors?• Role of Key Factors in the assessment

process

33

Organization profile

• Introductory section of the Criteria and the application

• WSQA criteria requires applicant to complete the entire Baldrige profile

• Applicant describes• Organizational environment• Organizational relationships• Competitive environment• Strategic context• Performance improvement system

34

Organizational profile

• Not evaluated or scored as are categories• Examiners use as a basis for evaluation

• What do we need to know and understand about the applicant?

• What’s important to the applicant?• Applicant can use as an initial self-assessment to

identify gaps/conflicts in knowledge• Evaluation: How well does the applicant address its

Key Factors?

35

What is a “Key Factor”?

• “Significant attributes of an organization that influence the way the organization operates”

• Facts about the organization and key strategic decisions made that shape the organization

36


• Examples• Mission, vision, and values• Employee/staff profile• Customer and market segments

and customer requirements• Competitive position and

critical success factors• Strategic challenges• Governance structure

• These are all key factors because they are facts or attributes that affect the way the organization operates.

37


• Non-examples• Complaint management process• Strategic planning approach

• These are not key factors; rather, they are processes that the applicant uses in managing its work

38

How to identify key factors

• Responses to the Organizational Profile• Responses to the Criteria Items—

it is important to consider key factors wherever they are found

in the application. • Eligibility forms

39

Key Factor Format

• Not full sentences• No need to specify “the applicant”• First word is Bold• Example:Workforce: 1500 ees, 80% technical, 20%

administrative, 30% minority, 25% work remotely

40

Exercise 1:

Key FactorsYou will need:

• Flip chart and marker• Case study application• Criteria

Steps:• For assigned Section of the Organization Profile, identify 2-3 key factors• Review Key factors in the Case Study Scorebook for your section to see identify similarities.

41

5. Independent review:Process Items

Objective: Be able to diagnose a process item

for independent review

42

Read the relevant section of the application

43

Read the application• Begin Analysis of the Response

• Which questions are the applicant answering?• Which evaluation factors [Approach, Deployment,

Learning, Integration] are present in the response?• Make notes on your findings

• Sticky notes, highlighter, comments in the margins• Identify any results that could be generated by the

activity and reported in Category 7• If you find misplaced information (key factors, responses to

other criteria items), put references to those responses in the appropriate section of the applicants response

44

Independent review

1. Select the most relevant Key Factors (from list)2. Read the criteria for the category3. Read (and annotate) the relevant section of the

application4. Identify and record the applicant’s processes5. Record observations. Note missing responses and

significant observations6. Recommend a scoring range for the category

45

Draft around six strengths and opportunities for improvement

46

Independent review steps - process

1. Review related criteria2. Read the category under review, take notes3. Select relevant key factors from list4. Identify & record processes5. Record observations on worksheet under strengths

and Opportunities6. Indicate

• Approach• Deployment• Learning• Integration

47

Identify and record processes

• “Process” includes• Inputs• Steps (related activities)• Time frames• Outputs• End users (internal/external customers)• Standards• Key measures to evaluate and improve the process

• Not always formal or named – you might have to read between the lines

• Look to criteria for guidance

48

“Systematic” refers to processes that• Are well-ordered• Are repeatable• Use data and information so learning is possible• Build in the opportunity for evaluation,

improvement, and sharing


49

• “Effective” refers to how well a process or measure addresses its intended purpose

• Determining effectiveness requires evaluating• How well the approach is aligned with the organization’s

needs• How well it is deployed, and/or• How the evaluation of the outcome of its measures are

used, i.e., for learning and improvement


50

Indicating ADLI- Approach

• “Approach” refers to• Methods used to accomplish the process• How appropriate the methods are to the item

requirements• The effectiveness of their use• The degree to which the process is repeatable

and is based on reliable data and information (i.e., systematic)

51

Indicate “ADLI”- Deployment

“Deployment” refers to the extent to which• The approach is applied in addressing relevant

and important item requirements• It is applied consistently• It is used by all appropriate work units

52

Indicate “ADLI”- Learning

“Learning” refers to • Refining the approach through cycles of

evaluation and improvement• Encouraging breakthrough change to your

approach (innovation)• Sharing refinements and innovations with other

relevant work units and processes

53

Indicate“ADLI”- Integration

“Integration” refers to the harmonization of• Plans• Processes• Information• Resource decisions• Actions• Results• Analyses

to support key, organization-wide goals

54

Indicate ADLI- Integration

Questions to ask in analyzing “Integration” • Do the individual components of a performance

management system operate as a fully interconnected unit?

• Is the approach aligned with the organizational needs identified in the Organizational Profile?

55

ADLI-key concepts

Integration examples:

• Alignment of objectives and action plans with strategic challenges AND mission, vision, values

• Alignment of product/service delivery methods with KEY customer requirements

• Alignment of key process measures with KEY customer requirements

56

Scoring Categories 1 – 6 (Process Categories)

• Examiners select a score (0-100) to summarize their observed strengths and opportunities for improvement (OFI’s)

• Scoring Guidelines are provided for Approach, Deployment, Learning, and integration

• Scores are assigned for each ITEM

57

Scoring Guidelines for Approach SCORE PROCESS

0% -5% No SYSTEMATIC APPROACH is evident; information is ANECDOTAL. (A)

10% - 25% The beginning of a SYSTEMATIC APPROACH to the BASIC REQUIREMENTS of the Item is evident. (A)

30% - 45% An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the BASIC REQUIREMENTS of the Item, is evident. (A)

50% - 65% An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the OVERALL REQUIREMENTS of the Item, is evident. (A)

70% - 85% An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the MULTIPLE REQUIREMENTS of the Item, is evident. (A)

90% - 100% An EFFECTIVE, SYSTEMATIC APPROACH, fully responsive to the MULTIPLE REQUIREMENTS of the Item, is evident. (A)

58

Category/Item Question Organization

BASIC

OVERALL MULTIPLE

59

Scoring Guidelines for DEPLOYMENTSCORE PROCESS

0% -5% Little or no DEPLOYMENT of an APPROACH is evident.

10% - 25%The APPROACH is in the early stages of DEPLOYMENT in most areas or work units, inhibiting progress in achieving the BASIC REQUIREMENTS of the Item.

30% - 45% The APPROACH is DEPLOYED, although some areas or work units are in early stages of DEPLOYMENT.

50% - 65% The APPROACH is well DEPLOYED, although DEPLOYMENT may vary in some areas or work units.

70% - 85% The APPROACH is well DEPLOYED, with no significant gaps.

90% - 100% The APPROACH is fully DEPLOYED without significant weaknesses or gaps in any areas or work units.

60

Scoring Guidelines for LEARNING

SCORE PROCESS

0% -5% An improvement orientation is not evident; improvement is achieved through reacting to problems.

10% - 25% Early stages of a transition from reacting to problems to a general improvement orientation are evident.

30% - 45% The beginning of a SYSTEMATIC APPROACH to evaluation and improvement of KEY PROCESSES is evident.

50% - 65%A fact-based, SYSTEMATIC evaluation and improvement PROCESS and some organizational LEARNING are in place for improving the efficiency and EFFECTIVENESS of KEY PROCESSES.

70% - 85%Fact-based, SYSTEMATIC evaluation and improvement and organizational LEARNING are KEY management tools; there is clear evidence of refinement and INNOVATION as a result of organizational-level ANALYSIS and sharing.

90% - 100%Fact-based, SYSTEMATIC evaluation and improvement and organizational LEARNING are KEY organization-wide tools; refinement and INNOVATION, backed by ANALYSIS and sharing, are evident throughout the organization.

61

Scoring Guidelines for INTEGRATION

SCORE PROCESS

0% -5% No organizational ALIGNMENT is evident; individual areas or work units operate independently.

10% - 25% The APPROACH is ALIGNED with other areas or work units largely through joint problem solving.

30% - 45%The APPROACH is in early stages of ALIGNMENT with your basic organizational needs identified in response to the Organizational Profile and other Process Items.

50% - 65% The APPROACH is ALIGNED with your organizational needs identified in response to the Organizational Profile and other Process Items.

70% - 85% The APPROACH is INTEGRATED with your organizational needs identified in response to the Organizational Profile and other Process Items

90% - 100% The APPROACH is well INTEGRATED with your organizational needs identified in response to the Organizational Profile and other Process Items.

62

Overall Score for Categories 1 – 6 Items

• An applicant’s score for approach [A], deployment [D], learning [L], and integration [I] depends on their ability to demonstrate the characteristics associated with that score

• Although there are 4 factors to score for each Item, only one overall score is assigned

• The examination team selects the “best fit” score, which is likely to be between the highest and lowest score for the ADLI factors

63

“Best Fit” ExampleConsider two applicants, both scoring 60% for A, D, and I,

and 20% for L• Applicant Market Growth Technology

Competitors A 10%/year Stable Stable B 2X/year Major change Many new

• Item Score:

• Applicant A: 50% (Learning could lead to incremental results improvements) • Applicant B: 30% (Learning needed to sustain the organization)

64

Step 5: Draft feedback-ready comments

65

Step 6: Determine the scoring range and score for the Item

66

Exercise 2: Independent Process Review

• Exercise 2, Independent Review, Process• You will need:

• Exercise handout• Blank process worksheet• Criteria• Case Study• Process diagnosis attributes (workbook)

67

Exercise 2– Table Assignments

Table Item #

Table 1 Item 1.1

Table 2 Item 1.2

Table 3 Item 2.2

68

6. Independent review:LeTCI – Results Items

Objective: Be able to evaluate a results item

for independent review

69

Independent review steps - results

1. Review the Criteria to identify required results2. Review the relevant process in the application to

identify KEY results important to the applicant (use KF’s, application, your diagnosis)

3. Review the Results in the application• What IS there?• What ISN’T there that should be?

4. Evaluate & score on the basis of LeTCI

70

Using a Table of Expected Results to Identify Category 7 OFI’s

Purpose:

Identify results that the applicant hasn’t provided

WHY?

Because applicants like to show results that are favorable, but may not always show results that are important.

71

Table of Expected Results

Title of Expected Result

Source Reference

Category 7Reference

Result Reference

(Describe expected result and segments)

(List where in the application you learned this was important to report in Cat 7)

(List where in Cat 7 the result belongs)

(Identify the page or Figure that contains the results)

Examples: Examples: Examples:

Status of Action Plans (% of Action Plans on target)

OP pg.7 1, 2.1a2 Not found

Workforce segments OP p6 7.4 Expect to see segmentation by FT, PT and on-call staff

Found in 7.4-4, 6, 10Not found in 7.4-7, 8, 9

72

Process for using a Table of Expected Results

As you read the application, pay attention for processes that the applicant cites as important, measures that are discussed, segments that are used, etc.

As you find examples of these, include them in the table of expected results in columns 1-3.

When you review Category 7, review it for the expected results and complete column 4.

Based on the comments in Column 4 of the table, an OFI, or OFI’s can be drafted citing missing expected results.

Note: These are results that the applicant has introduced an expectation for in the Organizational Profile and Categories 1-6. They are NOT results that the examiner would like to see.

73

Evaluation Factors for Category 7 (results)

• Le = Performance LevelsNumerical information that places an organization’s results on a meaningful measurement scale. Performance levels permit evaluation relative to past performance, projection goals, and appropriate comparisons.

• T = TrendsNumerical information indicating the direction, rate and breadth of performance improvements. A minimum of 3 data points is needed to begin to ascertain a trend. More data points are needed to define a statistically valid trend.

• C = ComparisonsEstablishing the value of results by their relationship to similar or equivalent measures. Comparisons can be made to results of competitors, industry averages, or best-in-class organizations.

• I = IntegrationConnection to important customer, product and service, market, process and action plan performance measurements identified in the Organizational Profile and in Process Items.

• G = Gaps Absence of results addressing specific areas of Category 7 Items, including the absence of results on key measures discussed in Categories 1–6

74

Results Evaluation Factors

45

50

55

60

65

70

75

80

85

90

95

93-94 95-96 97-98 99-00 01-02 03-04

Per

cen

tile

MCB Percentile Top 10%National Mean MCB Trend

Le

Trend

Comparison

Denotes “Good” Trend

direction

75

Results strengths and OFI;s

• Strengths are identified if the following is observed:• Performance levels [Le] are equivalent or better than

comparatives, and/or benchmarks• Trends [T] show consistent improvement, and• Results are linked [Li] to key requirements

• Opportunities for improvement are identified if:• Performance levels [Le] are not as good as comparatives• Trends [T] show degrading performance• Comparisons are not shown• Results are not linked [Li] to key requirements• Results are not provided [G] for key processes and/or

action items

76

Scoring a Results Item

• Scoring a Results item is similar to a Process Item

• Scoring Guidelines for Le, T, C, and I

• A single “best fit” score is selected for each results item

77

Results Scoring for Level [Le]SCORE RESULTS

0% -5%There are no organizational PERFORMANCE RESULTS or poor RESULTS in areas reported.

10% - 25%A few organizational PERFORMANCE RESULTS are reported; there are some improvements and/or early good PERFORMANCE LEVELS in a few areas.

30% - 45%Improvements and/or good PERFORMANCE LEVELS are reported in many areas addressed in the Item requirements.

50% - 65%Improvement TRENDS and/or good PERFORMANCE LEVELS are reported for most areas addressed in the Item requirements.

70% - 85%Current PERFORMANCE LEVELS are good to excellent in most areas of importance to the Item requirements.

90% - 100%Current PERFORMANCE LEVELS are excellent in most areas of importance to the Item requirements.

78

Results Scoring for Trend [T]SCORE RESULTS

0% -5% TREND data either are not reported or show mainly adverse TRENDS.

10% - 25%Little or no TREND data are reported, or many of the trends shown are adverse.

30% - 45% Early stages of developing TRENDS are evident.

50% - 65%No pattern of adverse TRENDS and no poor PERFORMANCE LEVELS are evident in areas of importance to your organization’s KEY MISSION or business requirements.

70% - 85%Most improvement TRENDS and/or current PERFORMANCE LEVELS have been sustained over time.

90% - 100%Excellent improvement TRENDS and/or consistently excellent PERFORMANCE LEVELS are reported in most areas.

79

Results Scoring for Comparison [C]SCORE RESULTS

0% -5% Comparative information is not reported.

10% - 25% Little or no comparative information is reported.

30% - 45% Early stages of obtaining comparative information are evident.

50% - 65%Some TRENDS and/or current PERFORMANCE LEVELS—evaluated against relevant comparisons and/or BENCHMARKS—show areas of good to very good relative PERFORMANCE.

70% - 85%

Many to most reported TRENDS and/or current PERFORMANCE LEVELS—evaluated against relevant comparisons and/or BENCHMARKS—show areas of leadership and very good relative PERFORMANCE.

90% - 100%Evidence of industry and BENCHMARK leadership is demonstrated in many areas.

80

Results evaluation: key concepts

• Integration• Results align with key factors, e.g.,

• Strategic challenges, • Workforce requirements• Vision, mission, values

• Results presented for• Key processes• Key products, services• Strategic accomplishments

• What examples can you think of?• Strong integration?• Not-so-strong integration?

81

Results evaluation: key concepts

• Strong integration: results presented for • Key areas addressing strategic challenges, • Key competitive advantages • Key customer requirements

• Not-so-strong integration:• Results missing for the above• Results presented that the Examiner can’t match to

process items or Organizational Profile

82

Results Scoring for Integration [I]SCORE RESULTS

0% -5%RESULTS are not reported for any areas of importance to your organization’s KEY MISSION or business requirements.

10% - 25%RESULTS are reported for a few areas of importance to your organization’s KEY MISSION or business requirements.

30% - 45%RESULTS are reported for many areas of importance to your organization’s KEY MISSION or business requirements.

50% - 65%Organizational PERFORMANCE RESULTS address most KEY CUSTOMER, market, and PROCESS requirements

70% - 85%Organizational PERFORMANCE RESULTS address most KEY CUSTOMER, market, PROCESS, and ACTION PLAN requirements.

90% - 100%Organizational PERFORMANCE RESULTS fully address KEY CUSTOMER, market, PROCESS, and ACTION PLAN requirements.

83

“Key” results

• Refers to elements or factors most critical to achieving the intended outcome

• In terms of results, look for• Those responsive to the Criteria requirements• Those most important to the organization’s success• Those results that are essential elements for the

organization to pursue or monitor in order to achieve its desired outcome

84

Step 4:Draft around six strengths and opportunities for improvement

85

Draft feedback-ready comments

86

Results worksheet example

• REFER TO EXAMPLE IN MANUAL

87

Exercise 3– Table Assignments

Table Item #

Table 1 Item 7.5

Table 2 Item 7.3

88

Evaluation Process

89

Software for Home work• Link to softwarehttp://www.wsqa.net/extraining.php• Username and Password: firstlast name• Link for software manual and taped webinarhttp://www.wsqa.net/Extimetable.phpClick on scorebook, select Key factors, then

select Item evaluation Save Frequently, Save OFTEN, Save A Lot!-

save in software- not on your browser

90

The Scorebook Navigator™• Licensed by the Alliance for Performance Excellence to facilitate the

evaluation process for state-level Baldrige program examiners. • The software will help you organize and share your findings. It will

streamline both independent review and the consensus process. • As with all software applications, Scorebook Navigator ™ is a work in

progress. Please help us continue the improvement cycle by contacting your Award Program Staff with the OFIs (and best practices!) that you encounter.

Browser Requirements• This software has been designed to work with Microsoft’s Internet

Explorer (IE), Firefox, and Safari. If you use another browser, some of the features may not work properly. Firefox is the recommended browser to use with a Mac.

91

Logging in

• When you log in for the first time, follow the steps below. If you already logged in during training to complete your pre-work in Scorebook Navigator, ™ your account information (username and password) will remain the same during an actual evaluation.

• Go to scorebook.baldrigepe.org. • Input your username and password (assigned by your Award

Program Staff)• On the Terms of Use Screen click on the accept button for access

authorization. This limits your use of the Scorebook Navigator ™ to award program purposes – not for your personal or business use.

92

Setting Up Your Account

93

Stage 1- Independent Review

94

Iconology

95

Key Factors Screen• Your first step of independent review is to draft a

list of key factors. The Key Factors Screen allows you to document key factors, which will later be linked to your Item evaluation as you complete that portion of the scorebook.

• The first time you open your scorebook, the Key Factors Screen will appear. You can also access the Key Factors Screen by clicking Key Factors on the menu bar.

96

Key Factors

97

Use these Key Factors Fields to type in each Key Factor.

Saving your Work!• Save Frequently!• It’s important to click the Save icon before leaving the Key Factors Screen.

You must save before you move to another area on the Key Factors Screen, and before clicking any other tab. Otherwise, your work will NOT be saved.

• In addition, since the data is saved using the internet, there are many ways to lose your connection to the database. When this occurs, any information you have entered into your scorebook since the last time you saved will be lost.

• Save-Reminder feature. After working on a page for a fixed time (5 minutes is the default), the white lines between rows and columns will briefly flash blue and the Save Icon will turn red. You can continue to work while the icon is red, but should save as soon as practical. (On the My Account page, in the lower left corner, you can adjust the reminder time to 5, 10, 15, or 20 minutes; you can also turn the reminder off)

98

Item Evaluation Screen 2010 Baldrige Process

99

This is the top portion of the evaluation section on the Item Evaluation Screen. It continues as you scroll down the page.

Components of Individual Review

• The evaluation section is composed of five panels that you will use to complete each Item in your evaluation:

• Key Factor References• Strengths• OFIs• Feedback Ready Comments• Scoring

100

Strengths/Gaps & OFIs Sections• List approximately 6 of the most important OFIs/Gaps

and Strengths for this Item in order of their importance to the applicant.

• Base these findings on the applicant’s response to the Criteria requirements and its key business/organization factors. Refer to figures when appropriate.

• Use the arrows to arrange the order of the strengths, starting with the most important feedback to give the applicant.

101

Comment Columns• “KF Ref.” column. Use this column to identify the applicable key factors by putting

a check-mark in the box next to the relevant key factors. Hover your mouse over the Item number to see the key factor text.

• ++/-- Columns: Click on this box if the finding is very significant to the applicant.• Strength/Gaps & OFI columns: Enter your findings as a brief statement of an

approach. Findings should represent the response of the applicant to the criteria, given its Key Factors.

• As Evidenced by: Give the evidence from the application that supports the finding,

• A/D/L/I columns: Click on the box for each evaluation factor that applies to the finding

• “Item Ref.” column. The appropriate Item reference from the Criteria is checked on the Evaluation Factors Panel. Hover your mouse over the Item to reference the Criteria requirements.

• “Key Themes” column. If you believe an observation may be related to a key theme, check the box in this column.

102

Feedback Ready Comments• Choose one strength finding and one OFI finding and write a feedback-ready

comment for each. Each strength or OFI should include: specific Criteria language—language that refers to the Criteria requirement being addressed; the applicant`s terminology, examples, and figure numbers—language that refers to the approach or results given in the applicant`s response to Criteria questions, evaluation factors—language from the findings of your analysis.

• Type Column: prefilled for one strength and one OFI.• Double Column: click the box if the comment is very significant to the

applicant.• Comment Column: Enter your feedback-ready comment.• “Item Ref.” column. The appropriate Item reference from the Criteria is

checked on the Evaluation Factors Panel. Hover your mouse over the Item to reference the Criteria requirements.

• “Key Themes” column. If you believe an observation may be related to a key theme, check the box in this column.

103

Scoring Grid

104

What we’ve covered today

• WSQA overview• Baldrige / WSQA criteria overview• WSQA Assessment process• Independent review step-by-step

• Key Factors• Process review• Results review

• Scorebook Navigator Overview

105

Your questions?

106

1. Instructions and files will be e-mailed2. Complete independent review sheets for sections

1.1, 3.1, 6.1, 7.1 based on the Case Study3. Bring a hard copy4. Put aside 20 hours for the assignment5. Observe the time guidelines in the instructions

even if that means you complete fewer sections/comments

Homework assignment

107

Your feedback, please!

• Please complete surveys before leaving

108

THANK YOU!!

Your support and participation as Examiners helps us all by helping WSQA

fulfill its mission!

109

Welcome back, Examiners!

Washington State Quality Award

Examiner Training

2010

Session 2

2010 Examiner Training

111

First things first!

• THANK YOU again!• Our mission: Improving the way we live, learn, and

work in Washington by helping organizations improve through the use of the Baldrige Criteria for Performance Excellence

112

Objectives For This Class

• Learn to evaluate a WSQA application through all stages of the assessment process

• Do this with confidence • In yourself as an Examiner and • In the assessment process

Examiner Development Model: On-BoardingCoaching Process

Preparation Application Evaluation

Selection Notification

BNQP Orientation

Evaluation Process

Orientation

Case Study

New Examiner Training

Examiner Preparation

Training

Team Assignment Development Planning

Receive Welcome E-mailSelect Training Week

Coaching Meeting

Complete Online ModulesReview Sample Feedback Report

Read about Award RecipientsCoaching Meeting

Complete Online ModulesComplete “Are You Ready”

Coaching Meeting

Read the CriteriaComplete a Partial IR Scorebook

Coaching Meeting

Examiner OrientationPeer Learning, Reflection,

& Feedback

Prepare for the Award Evaluation Process

Peer Learning, Reflection, & Feedback

Complete ½ IR Scorebook (3 Process Categories,

3 Results Items)Peer Learning, Reflection,

& FeedbackTeam Leader Coaching &

Feedback

Reflect on ExperienceSeek Feedback from Peers, Monitor, &

Team LeaderCreate Development Plan

What are you learning?What are your strengths and

opportunities?What are your next steps?

113

PERFORMANCE EXPECTATIONS for an EXAMINER’s INDEPENDENT REVIEW

114

Performance along the Development ContinuumExaminer’s Performance Level

Entry With guidance

CompetentIndependently

Expert Independently

Job Aids Uses the evaluation tools and Step-by-Step instructions

Knows the resources available on the BNQP Web site

Uses the Help Line

Makes a plan that includes an estimated time schedule

Works with the guidance of a coach and/or team leader

Uses the current evaluation tools

Has adapted the Step-by-Step instructions to his/her personal work style

Works independently

Based on experience, accurately assesses the time required

Provides guidance

Demonstrates cycles of improvement in adapting the Step-by-Step instructions

Criteria Applies knowledge of the Criteria (e.g., Categories, Items, Areas to Address, and basic, overall, and multiple requirements)

Sequentially relates the applicant’s Item responses to the Criteria, key factors, other Items, and Core Values

Utilizes an in-depth understanding of the Criteria intent

When reading an application, simultaneously makes connections within the application and to linkages in the Criteria, key factors, and key themes

Demonstrates how the Criteria offer value to the specific applicant

115

Award Application

Identifies the elements of an application

Matches the response to the Criteria requirement

Describes the organization

Recognizes and, as appropriate, gives credit for misplaced responses

Recognizes evidence to support strengths and opportunities

Appropriately applies the “benefit of the doubt” concept

While reading the application, concurrently makes initial assessments and evaluates the applicant’s maturity levels

Identifies value creation and leveraging opportunities for the applicant

Builds support for cross-cutting and other key themes

Key Factors Selects key factors related to Item requirements

Selects key factors within the context of the application

Applies business acumen and sector knowledge when selecting relevant key factors

Develops key factors that combine data from the Organizational Profile and Eligibility Certification/Application Forms to highlight areas of potential leverage

Strengths and Opportunities for Improvement

Interprets facts, including comparing and contrasting them within individual Items

Drafts opportunity statements that may focus on a lack of response and poor trends. Provides enough information to enable the writing of a comment

Sees patterns and makes connections

Identifies strengths and opportunities

Prioritizes the most relevant, important feedback

Assesses the vulnerability or risk to the applicant of not meeting Criteria requirements

Develops a new thought by relating knowledge from different areas

Develops statements that demonstrate the systems perspective of the performance excellence framework.

Develops statements that support anticipated key themes

116

Feedback-Ready Comments

Identifies the elements of a well-written comment

Writes a concise, fact-based comment with the guidance of a team leader or coach

Follows all writing guidelines, avoiding the introduction of personal criteria or operational standards

Drafts a full comment without a detailed re-write of the applicant’s response

Demonstrates relevance to the applicant

Generalizes from Item level to key themes

Modifies comments based on consensus feedback

Prioritizes, combines, and separates comments as needed

Includes appropriate key factors in comments

Develops comments that add value and provide insight

Identifies well-written comments; assesses the value of comments and revises them as needed

Links comments to other Items, as appropriate

Develops key themes that diagnose and reflect critical strengths and vulnerabilities

Score With the guidance of a team leader or coach, identifies the “most descriptive” range in the scoring guidelines

Selects the “most descriptive” scoring range and appropriate score

Ensures score is aligned with the comment

Independently modifies scoring based on consensus comments

Assesses appropriate score and provides constructive feedback

Agenda: Session 1

1. WSQA organization overview2. Baldrige criteria overview3. WSQA Assessment process4. Online scorebook preview5. Independent review,

step-by-step• Preparing• Key Factors• Process review• Results review

Mostly exercises

Mostly presentation

117

Agenda: Session 26. Ethics, code of conduct7. Consensus, step-by-step8. Key Themes

9. Site visit preview10. Final scorebook preparation11. “Lite” vs. “Full”12. Summary

Mostly exercises

Mostly presentation

118

5. Ethics, Code of Conduct

Objectives:

1. Understand the importance of ethics in this process

2. Interpret and apply the Code

2010 Examiner Training 119

120

Conflict of interest issues

• WSQA seeks to avoid conflict of interest to protect process integrity

• Conflicts are frequent!• First step in assignment process: determine

any conflicts

121

Code of Ethical Conduct

• Document is in your workbook• Why are ethics of utmost importance to

WSQA and this process?

122

Exercise 4:

Code of Ethical Conduct Homework

Lessons Learned from Home work

Questions on Conflict of Interest

Individual Review – Review of Homework

• Homework complete?• Do you have any questions?• What went well?• What did you struggle with?• While completing homework, what

tools/docs do you want to have at hand?

123

Writing Well Written Comments


125

How do good comments add value?

Criteria

Applicationcontent

OrganizationProfile

The most valuable Comments/Key Themes

126

The Structure of Well Written Comments

Well written comments incorporate criteria language, ADLI/LeTCI, scoring language, and application language.

127

Well Written Comment Guidelines

Category 1-6 Does the comment link to the Criteria? Does it consist of 2-3 grammatically correct sentences

(about 1000 characters)? Does it use examples from the application that illustrate the

OFI/Strength? Does it include a so what if it is an OFI? Is the statement non-judgmental and non-prescriptive? Does the comment link to the scoring guidelines?

128

Well Written Comment Guidelines • Tie your comments to the basic, overall, or multiple level Criteria

requirements that are most important to the applicant. • Write process comments so they contain: a subject identified from

the criteria, the application, or the scoring guidelines; verb(s) and requirements from the Criteria; examples from the application, and figure numbers, as appropriate.

• Use the evaluation factors (ADLI or LeTCI), and scoring language, to clearly articulate the areas of the applicant’s response to clarify the strength or provide insights that will help the applicant improve overall organizational effectiveness and capabilities.

• Draw linkages between an Item and the applicant's Organizational Profile.

• Ensure that the comment does not contradict other comments in the same or other Items or in the Key Themes Worksheet.

129

The Style of Well Written Comments• Use active voice and present tense (e.g., 'completes' rather than 'is

completed'). • Use a polite, professional, and positive tone. • Use vocabulary/phrasing from the Criteria and the Scoring

Guidelines. • Describe what is missing if something "is not clear". “Not clear”

comments must be resolved if the applicant has a site visit. If they don’t do it based on site experience then say they don’t do it.

• In lieu of the applicant’s name, use such phrases like "the applicant" or "the organization" to refer to the applicant in the independent and consensus scorebooks. Use the applicant's name in final feedback reports.

• Use the applicant's language/terminology when appropriate.

130

“Don’ts” to avoid in comments

1. Avoid parroting2. Avoid being prescriptive 3. Avoid going beyond the criteria

131

“Do’s” for well-written comments

(refer to comment guideline in workbook)

132

Well Written Comments Don’t….

• Go beyond the requirements of the Criteria or assert your personal opinions.

• Be prescriptive by using "could", "should", or "would". • Be judgmental by using terms such as "good", "bad", or

"inadequate". • Tell the applicant that they aren’t doing enough of something• Comment on the applicant's style of writing or data

presentation. • Use jargon or acronyms unless they are used by the applicant • Only “Parrot” the application or the Criteria. Provide only

enough language to add clarity—seek to add value rather than restate information.

133

How Could This Comment Be Improved?Application language:• Share Food uses a biennial 12 step Strategic Planning Process that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the “Current State” step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. The Strategic Planning Process was initiated in 1997 and restructured in 2004 as a result of benchmarking analysis of food banks and other nonprofit organizations, and feedback from the state award process.

Comment: • Share Food uses a biennial 12 step Strategic Planning Process (SPP) that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the Define step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. This is proof that the organization has a great systematic approach to the SPP and extensive cycles of learning in 1997, and 2004.

134

How Could This Comment Be Improved?Application language:• The manager’s vision helps Our Store to maintain a strong

focus on customer satisfaction. The manager has worked hard to instill the belief that leadership and responsibility should be present at all levels. This rewards everyone at Our Store with the responsibility of planning, carrying out, and evaluating their own work.

Comment:• Our Store is in the beginning stages of building an approach to

setting a vision for customer satisfaction, this includes the manager working very hard to instill the beliefs in leadership and responsibility. Everyone in the store receives a reward and this will ensure that everyone will have ownership in the organization and the ability to evaluate their own work, which should improve performance overall.

135

How Could This Comment Be Improved?Application language:• Our Hospital builds relationships with customers through a variety of

methods including patient enrollment, which serves as an orientation to enabling services; Patient-Family Advisory Boards, which have recommended and implemented improvements to patient care; and high-visibility community health activities. Using patient surveys, conducted randomly after discharge, Our Hospital determines the issues that lead to poor relationship results and uses this information to implement new relationship-building processes. The Patient-Family Advisory Board is an example of the results of this process.

Comment:• Customer relationships are built through a variety of methods including

Patient enrollment, Patient-Family Advisory Boards, and community health activities. Surveys are conducted randomly after patients are discharged. The Patient-Family Advisory Board is used to determine issues that lead to poor relationships and implements new relationship-building processes.

136

How Could This Comment Be Improved?Application language:• Customer groups are identified annually as part of the Strategic

Planning Process using Our Hospital’s vision/mission/values as a focus. By analyzing demographic data from multiple sources, the cross-location Service with Spirit Team (SWST) is able to identify the gaps, look at disparities, and identify potential customers. Deployment of this process through multiple cycles has resulted in the development of several services designed to meet the unmet needs of customers in the Our Hospital’s service area.

Comment:• Through the SPP, customer groups are annually identified using the

MVV as a focus. The SWST uses demographic data to identify gaps, disparities and identify potential customers. This process has been deployed throughout the organization, and through multiple cycles of refinement the customers needs in the service area have been met.

137

Well Written OFI Comment Guidelines• May be used to give credit to the applicant for some areas that

HAVE been addressed• “While the applicant does x, y, z, it is not evident that a, b,

and c are performed.”• May begin with a statement about what is missing:

• “There is no evidence that…”• “There is no indication of how…”• “It is unclear…”• “It is not apparent that there is a …”

• Must tie to criteria language, (ADLI), and/or to Key Factors that the applicant has identified in the Organizational Profile

• Must include a “so what” (Why does the applicant care about this? Why would they take action?)

• Usually these come from Key Factors / Org Profile. The closer the ‘so what’ can be to identifying the applicant’s specific needs, the better.

138

How Could This Comment Be Improved?Application language:• Share Food uses a biennial 12 step Strategic Planning Process that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the “Current State” step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. The Strategic Planning Process was initiated in 1997 and restructured in 2004 as a result of benchmarking analysis of food banks and other nonprofit organizations, and feedback from the state award process.

Comment: • Although Share Food uses a biennial 12 step Strategic Planning Process

(SPP) that involves community leaders, volunteers, member agencies, and donors, there is no evidence of how blind spots are identified, nor how time horizons are set and how the SPP addresses those horizons. If the applicant implemented a more extensive benchmarking process that identified blind spots and included time horizons, there may be more of an opportunity for this organization to have a more successful Strategic Planning Process.

139

How Could This Comment Be Improved?Application language:• The manager’s vision helps Our Store to maintain a strong

focus on customer satisfaction. The manager has worked hard to instill the belief that leadership and responsibility should be present at all levels. This rewards everyone at Our Store with the responsibility of planning, carrying out, and evaluating their own work.

Comment:• Although the manager has worked hard to instill the belief

that leadership and responsibility should be present at all levels, there is no indication of a systematic approach, nor is there deployment at any level, no integration, nor are there any cycles of learning.

140

How Could This Comment Be Improved?Application language:• Our Hospital builds relationships with customers through a variety of

methods including patient enrollment, which serves as an orientation to enabling services; Patient-Family Advisory Boards, which have recommended and implemented improvements to patient care; and high-visibility community health activities. Using patient surveys, conducted randomly after discharge, Our Hospital determines the issues that lead to poor relationship results and uses this information to implement new relationship-building processes. The Patient-Family Advisory Board is an example of the results of this process.

Comment:• There is no systematic approach to building customer relationships, or in

conducting patient surveys. And although there is a Patient-Family Advisory Board there is no approach described for how issues are determined and how new relationship-building processes are developed.

141

How Could This Comment Be Improved?Application language:• Customer groups are identified annually as part of the Strategic

Planning Process using Our Hospital’s vision/mission/values as a focus. By analyzing demographic data from multiple sources, the cross-location Service with Spirit Team (SWST) is able to identify the gaps, look at disparities, and identify potential customers. Deployment of this process through multiple cycles has resulted in the development of several services designed to meet the unmet needs of customers in the Our Hospital’s service area.

Comment:• Although customer groups are identified annually in the SPP, and the

Spirit Team (SWST) identifies gaps, disparities, and identifies potential customers, there is no indication how this information is deployed throughout the organization and how the information is used to meet the unmet needs of customers in the area.

142

Comment Writing Exercise• Each person at the table select an independent

review item (Cat 1-6 strength or OFI)• Write a well written comment• Swap scorebooks• What feedback would you give to this examiner

• Use the comment checklist for examples • Practice giving feedback to each other• Report out the most significant key learnings• (If time swap with to another person at the

table)

143

Category 7 Comment GuidelinesCategory 7

Are specific results (numbers) used rather than generalities?

Have trends, comparisons and/or appropriate benchmarks been considered?

Does the comment include figure numbers? Have we summarized multiple charts with a consistent

theme (trends, missing, comparisons, etc.)

144

(Well Written Result Comment Example

There is a lack of comparative data in a number of the Results sub-categories; as an example, no comparative data is available for measures such as Discharge Weight of ELBW Infants (7.1-4), Improvement % of ELBW Infants (Figure Figure 7.1-5), % of Breast Milk Available (Figure Figure 7.1-6), Outpatient Lactation (7.1-7) and Catheter Related Bloodstream Infections (7.1-8), Survival to Discharge by Rhythm (Figure Figure 7.1-17), and Well Child Visits Figure (7.1-19). The lack of comparative results may impact the value of Excellence - setting and achieving the highest standards.

145

How Could This Comment Be Improved?

The applicant’s market share results for key service lines is strong, exceeding all comparative data across Discharges; Inpatient Surgeries; the applicant's Heart Services; and Neurosurgery Market. The applicant enjoys market dominance in these service lines and has shifted its focus to adolescent care and services to new geographic markets. The applicant has received numerous recognitions and awards for their leadership in these markets and service lines. These successes are important to their goal of achieving national recognition as a premier hospital, a tertiary and quaternary hub, and preferred provider of regional health care services.

146

How Could These Comments Be Improved?

• The applicant's Net Income from Operations (7.3.8) is above the comparison median and trending up over the past three years and overall applicant’s turnover rate has been decreasing.

• The applicant’s Net Sales between 2006 and 2008, have increased by 10% in 2006 to over 12.2% in 2008. The applicant is consistently above the goal of 9% for all three years, but is below the Baldrige comparison of 15% for all three years.

• Although the turnover rate for RNs has been increasing between 2006 and 2008. The turnover rate for RNs was 7.7% in 2006 and 10.3% in 2008. There have been substantial improvements in physician satisfaction from 13% to over 25% during the same time period.

147

How Could These Comments Be Improved?

• The applicant's Net Income from Operations (7.3.8) is below the comparison median and trending down over the past three years and overall applicant’s turnover rate has been increasing between 2006 and 2008, from 10% in 2006 to 12.2% in 2008.

• The turnover rate for RNs has been increasing between 2006 and 2008. The turnover rate for RNs was 7.7% in 2006 and 10.3% in 2008.

148

Comment Writing Exercise• Each person at the table select an independent

review item (Cat 7 strength or OFI)• Write a well written comment• Swap scorebooks• What feedback would you give to this examiner

• Use the comment checklist for examples - p.89• Practice giving feedback to each other• Report out the most significant key learnings• (If time swap with to another person at the

table)

6. Consensus, step-by-step

Objectives:

1. Understand consensus stage activities, roles, responsibilities

2. Be able to develop valuable comments


150

Topics

• Consensus stage activities: • Developing process and results comments• Recommending scores

• Comment guidelines – do’s and don’t’s

151

No

Receive applications

Stage 1:Independent Review

Individual Scorebooks

Stage 2:Consensus Review

Consensus Scorebook

Lite or Fullapplication?

Selected forsite visit?

Full

Judges recommend award level to WSQA Board

Lite

Stage 3:Site Visit

Yes

Feedback report

Executive Briefing

WSQA Symposium & Awards Ceremony

Final product

Editor updates

scorebook

Lead & Mentor meet with Judges Panel

WSQA updates

scorebook

Lead & Mentor meet with

Judges Panel

Consensus Scorebook

Site visit scorebook

Stages of the Assessment Process

151

152

ApplicationCriteria

Consensus planning call

Independent review

scorebooks

Editor or Lead

1. Consolidate scorebooks

Single consensus scorebook

Category Lead

2. Review and group team’s observations

Grouped observations

Category Lead

3. Select relevant Key Factors

4-6 (3-5) Relevant Key

Factors

Observations (from Ind. Review)

Key Factors

For your category/categories only

1

Numbers in parentheses apply for Lite assessment only

WSQA Consensus: Step-by-step

152

153

Category Lead

4. Select and develop comments

About 6 comments

Grouped observations

1

Below-the-line observations

Category Lead

5. Write rationales for below-the-line observations

Rationales

Below-the-line observations

Category Lead

6. Exchange feedback with Back-up, update

comments

Updated comments

Back-up’s comments &

feedback

2

Lite: 4-6

Scoring recommendation


154

2

Team

7. Review team comments, prepare

notes, enter key theme ideas

Notes

Consensus-ready comments

Team

8. Hold consensus call(s) or meetings

Updated consensus scorebook

Draft consensus scorebook

Draft Key Theme topics

Editor

9. Prepare consensus scorebook

Updated Comments, score

Key themes, site visit

recommendation

FULL only

End

Consensus scorebook

FULL only


155

Example Strength Comment• The applicant identifies their core competencies as the

participating dentist network; expert claims administration; extraordinary customer service and performance culture. A SWOT analysis is used in the annual strategic planning process to identify core competencies. Correlation analysis is performed on satisfaction survey data and is used to determine key drivers of customer satisfaction. Initiative action plans accentuate core competency strengths. Deployment is evidenced by the use of bi-annual CAG meetings, ensuring alignment with the mission and the integration of data for easy access across all departments and customer segments

• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor

156

Example OFI Comment

• Although cross functional teams are chartered and aligned with the SPP, there is no indication of processes used by the teams or otherwise by the applicant to design and innovate its work processes to meet key requirements. There is no indication of how design and innovation processes address opportunities for agility and improvements in cycle time and productivity. It is also unclear how the team work process improvement approach is evaluated for annual process changes or improvements. An annual evaluation of strategic team alignment and improvements may provide efficiency and effectiveness in the accomplishment of strategic objectives, initiatives, and action plans.

• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor

Consensus Comment Writing

• Review all IR comments• If there are comments that address the issue,

start with the best of those• Build on that comment with input from other

IR worksheets• Compare with Example Strength Comment

157

Exercise – Consensus Comment Writing

• For item 6.1, as team, prepare a consensus strength comment and OFI, based on your independent reviews

158

159

Consensus preparation and call(s)

• Each Examiner responds with agreement, suggestions for changes, or disagreement, stating rationales

• Category Lead facilitates consensus• Team Lead and/or PEPD mentor leads final

scoring discussion• Can take anywhere from 4-8 hours

(depending on what?)

160

Consensus and conclusion

• If using online tool during consensus, changes may be made in real time

• Category Lead notifies Team Lead and Scorebook Editor when done

• Scorebook Editor compiles consensus scorebook with final edits, notifies WSQA

• Consensus scorebook should be finalized ASAP

161

Comment, scoring considerations

• “Benefit of the doubt”• Give credit for what’s in the application

• Don’t penalize for incidental exceptions• Not every process must show complete “DLI”• Not every results example must include “TCI”

162

Comment, scoring considerations

• For LITE, only one score for entire results Category 7• Propose a score for results related to your

categories• Be prepared to reach team consensus for all of

Category 7• Using scoring language can help the

applicant understand comments and score

163

Category feedback

• Check and provide feedback on whether Cat. Lead’s comments:• Include criteria requirements• Provide meaningful feedback• Address ADLI or LeTCI• OFI’s provide relevant “so what” (at least some of them)• Results cite figure numbers/titles and specifics• References to application are correct• Recommended score reflects comments in content and

balance (strengths:OFI’s ratio)• Back-up does this for full, team does this for lite

164

Information resources

• WSQA comment guideline handout• Your back-up, team lead, PEPD mentor• Reference materials in

• Your workbook: comment and scorebook examples• http://www.wsqa.net/extraining.php• Baldrige web site

http://baldrige.nist.gov/Examiner_Resources.htm

7. Key themes


166

How do good comments add value?

Criteria

Applicationcontent

OrganizationProfile

The most valuable Comments/Key Themes

167

What is a Key Theme?A key theme is a perception or observation that recurs

throughout the scorebook, across processes and results, reflecting major strengths, opportunities, or vulnerabilities.

A “key theme” is a strength or OFI that • Addresses a central requirement of the Criteria,• Is common to more than one Item or Category

(crosscutting), • Is especially significant in terms of the applicant’s key

factors, and/or • Addresses a Core Value of the Criteria

What is a Key Theme?

A high level perception or observation, relevant to the applicant’s key factors, that recurs in the scorebook and is expanded upon with evidence during a site visit. Key themes are often common to more than one category or item. They may address one or more Baldrige core value.

A role model result, practice, process, or innovation, that might be used by other organizations, or an observation so important, strength or OFI, process or result, that it should be brought to leadership’s attention.

The executive summary of the feedback report. It may be the only part of the feedback report that some people in the organization read.

168

169

Developing Key Theme comments

• Address all double strengths and OFI’s individually and/or in groups

• Address related comments across categories• Develop 6-10 (Guideline!) key themes each for

significant• Process strengths• Process OFI’s• Results strengths • Results OFI’s

• See example Key Themes in the workbook for this class

Core Values-The Foundation of the Baldrige Criteria

• Visionary Leadership• Customer driven excellence• Organizational and personal learning• Valuing workforce members and

partners• Agility• Focus on the future• Managing for innovation• Management by fact• Societal responsibility• Focus on results and creating value• Systems perspective

• If you think of the 7 Baldrige categories as the vertical beams in a house frame, the 11 Baldrige Core Values are the cross members that provide strength to the frame.

170

Process for using Baldrige Core Values to Develop Key Themes

• Each Team Member:• Review the applicant • Read the Baldrige Core Values• Write 11 post-its rating the applicant against each Baldrige Core

Value with a +, 0, or – and write a few short thoughts for each core value explaining why you rated the applicant as you did.

• Key Themes Lead• Sum the plusses and the minuses. Core Values that have many

plusses, or minuses, may be candidates for Key Themes. Typically only a few will stand out.

• Report results to team. • Draft Key themes for these high scoring values using feedback

report comments and post-it thoughts as inputs.

171

Using Core Values to ID Key ThemesBaldrige Core Value Score (+,0,-)

Visionary Leadership +,+,+,+,+,+ (6+) Probable KT

Customer driven excellence +,+,0,0,0,-

Organizational and personal learning +,+,+, 0,0,0

Valuing workforce members and partners

+,+,+,+,+,- (5+) Possible KT

Agility +,+,0,0,-,-

Focus on the future +,+,+,0,-,-

Managing for innovation 0,0,-,-,-,-

Management by fact -,-,-,-,-,-, (6-) Probable KT

Societal responsibility +,+,+,+,+,0 (5+) Possible KT

Focus on results and creating value 0,-,-,-,-,- (5-) Possible KT

Systems perspective +,+,0,0,-,-

172

Caveat to this approach

• While the Baldrige Core Values are the foundation upon which the Criteria are based, they are not examination items. Key themes are written based on comments developed relating to the criteria. The feedback report comments MUST support the Key Themes.

173

9. Site visit preview

Objectives:

• Understand value of site visit

• Set expectations

174

175

Site visit preview

• Site visit is awarded based on• Examiners’ recommendations• Scoring band (Leadership & higher is automatic)• Judges’ final determination

• Purpose of site visit• Verify strengths• Clarify opportunities for improvement• Improve quality of feedback report

176

Site visit preview

Preparation• Site Visit JIT training• Planning calls, team assignments• Develop Site Visit Issue worksheets• Request documents• Lead drafts interview schedule as possible

177

Site visit preview

Day 1: Sunday (planning day)• Applicant provides requested documents,

equipment (printer, shredder, etc.)• Draft and/or update interview schedules• Review and update SVI worksheets• Develop walk-around questions

178

Site visit preview

• Monday – Wednesday ON-SITE• Introductory meeting (meet & greet only)• Interviews – individual, teams, groups• Team caucuses, lunches• Walk-around questions• Visit all possible sites and shifts

• Back at the hotel…• Work on SVI worksheets, close out as possible• Team reviews: “Walk-the-wall”• All issues closed by end of day Wednesday

179

Site visit preview

• Thursday – Friday • Draft comments based on updated SVI

worksheets• Review and finalize comments• Review and finalize Key Themes• Determine final scoring recommendation• Assemble final scorebook• Celebrate!

• Saturday: return home!

10. Final scorebook preparation

Objective: understand preparation of WSQA final product

180

10. Final scorebook preparationTopics:

1. What does the team do?

2. What do the Team Lead, Editor, and Judges do?

3. What does WSQA do?

4. Final scorebook preview

181

182

What does the team do?

• Everything possible to save your Editor work!• Follow the comment guidelines (refer to

examples)• Update comments during or immediately after

consensus• Let your Team Lead and Editor know when your

comments have been updated• Continue to use “applicant” throughout, not

the applicant’s actual name

183

What does the Editor do?

• Attends Senior Examiner training• Corrects/edits the scorebook for consistency

with comment guidelines• See workbook for details

184

What do the Team Lead,PEPD Mentor, and Judges do?

• Lead and Mentor prepare for presentation of final scorebook to Judges

• Lead and Mentor present team’s scorebook at Judges’ review• Summary/highlights of comments• Summary of Key Themes (Full)• Scoring recommendation

• Judges determine award level based on team’s recommendation

185

What does WSQA do?

• Copies team scorebook into Feedback Report template

• Replaces “applicant” with applicant’s actual name throughout

• Sends Feedback Report to applicant• Coordinates Executive review meeting date

and participants

186

Final scorebook preview

• (Open scorebook file)• Case Study final scorebook is online

11. “Lite” vs. “Full”

Objective: Understand distinctions in

• Application Process

• Examination Process

187

188

Lite vs. Full application processTopic Lite (Assessment) Full Examination

Application Deadlines Quarterly Once annually

Intent to Apply/Application Form

Required. Accepted throughout the year. Intent to Apply forms must be submitted at least 4 months prior to the submission of your Lite Application.

Required. See Web site for due date.

Criteria Lite (Assessment) Criteria, A subset of questions from the Full criteria

Full version Baldrige criteria

Version Current or previous year Current year only

189

Lite vs. full application processTopic Lite (Assessment) Full Examination

Organizational Profile Required, up to 5 pages Required, up to 5 pages

Page limit, application body (not including profile, org chart, glossary)

2007: 15 2008: 20 50 pages

Examiner participation At least one required At least one required

Recognition None As determined by Judges

Executive briefing Yes Yes

190

Lite vs. full: examination processTopic Lite (Assessment) Full Examination

Team size 2-3 plus PEPD Mentor 5-10 plus PEPD Mentor

Stage 1: Independent review

(Same) (Same)

Stage 2: Consensus

Omits Key Themes Includes Key Themes

Judges Panel review

None Recommends site visit OR determines award level

Stage 3: Site visit None As determined by Judges

Post-site Judges panel review

None Determines final scoring and award level

Feedback report Final report prepared by WSQA Final report prepared by WSQA

TIME TABLE

• Always refer to WSQA website

• Examiner Schedule for Both Full and Lite Schedules http://www.wsqa.net/extimetable.php

• Applicant Schedulehttp://www.wsqa.net/timeTable.php

191

192

Schedule & Time Commitment

Activity Full Lite

Training 20 hours 20 hours

Individual review 20-40 16

Consensus preparation: comment writing 8-16 4

Participate in consensus meeting(s) 8 4

Add if Team Lead, Back-up, Editor 16-24 4-8

Site visit including preparation & training 60+ None

Total: Many! (not as many)

193

Discussion question

Considering “Full” vs. “Lite” criteria, what are the trade-offs for each

• From the Applicant’s point of view?

• From the Examiner’s point of view?

12. Summary

Objectives:

Summarize what we’ve learned

Answer any remaining questions


195

High-level objectives

• Learn to evaluate a WSQA application through all stages of the evaluation process

• Do this with confidence in yourself and in the assessment process

196

What we’ve covered: Session 1

1. WSQA organization overview2. Baldrige criteria overview3. WSQA Assessment process

4. Independent review, step-by-step

• Preparing• Key Factors• Process review• Results review

197

What we’ve covered: Session 25. Ethics, code of conduct6. Consensus, step-by-step7. Key Themes

8. Online scorebook preview9. Site visit preview10. Final scorebook preparation11. Executive briefing12. “Lite” vs. “Full”

198

Managing your time

• Put aside time in your calendar (refer to time commitments)

• Work with your lead & team• Set meetings early• Coordinate• Communicate

• Stories to tell?

199

Managing your time

• Schedule work hours• Break tasks into little

pieces• Reward yourself for wins• Move quickly past

setbacks• Call your lead• Call your back-up

200

Your questions?

201

Your feedback, please!

• Please complete surveys before leaving

202

THANK YOU!!

Your support and participation as Examiners helps us all by helping WSQA

fulfill its mission!

welcome, examiners!

Documents

wsqa visionbe

wsqa application

malcolm baldrige

organizational excellence

little baldrige historyget

baldrige award historycreated

state law

final scorebook preparation11