Welcome, Examiners!

Washington State Quality Award

Return Examiner Training

2010

First things first!

• THANK YOU!• WSQA’s mission: Improving the way we live, learn,

and work in Washington by helping organizations improve through the use of the Baldrige Criteria for Performance Excellence

2

Objectives For This Class• Learn changes to process• Be aware of upcoming changes from

National• Using Baldrige values to build key themes• Reinforce comment writing• Using the expected results• Reinforce scoring

3

Introductions

• Name, organization, what you do• Your experience with WSQA

4

Training Materials

•Slide set

•Sample review worksheets

•Independent review blank worksheets

•Comment writing guideline

5

Signed Ethical Statements

• Collect Signed Statement

6

5/10/06 Washington State Quality Award

Council

7

Learning Opportunities

• WSQA offers workshops• Introduction• Self Assessment• Case Study• Preparation for Application

• Annual conferences• Consulting• Performance Excellence Collaborative• Become an Examiner

Discuss the linkage between “Key Factors” and “Key Themes.”

Over the years, some of us have learned that when there are multiple questions in an “Area to Address” then generally the first question is more important than the second one, which is more important than

the third question, etc. Is this true?

Can you describe “work systems” more fully and

perhaps compare/contrast with “work processes” and “core

competencies.”

Are “projections” the same or different than “targets or goals?” (See page 58 of the criteria, which seems to say that projections can be targets.) Do performance projections impact the scoring in Cat. 7?

How does an Examiner assess innovation in an application?

How to determine if a results item should be given credit in

7.5 rather than 7.1?

What are the likely changes to the 2011-2012 criteria?

Describe the fact-based process the Board of Overseers uses in shaping and improving the Baldrige criteria.

Harry’s Session on Upcoming ChangesProcess Changes• Category 3: Rewritten to improve logic flow,

also to discuss more of the organization—less of the content

• Category 5: Consolidation/reorder to improve the logic flow

• Category 6: Reorganized around “Operations Focus” with better separation between 6.1 and 6.2. Examiners were struggling between Work Systems and Work Processes.

12

Upcoming Changes to Results

• Category 7: Will have only 5 items – 7.1 and 7.5 are to be combined and 7.4 will now have to address Governance and Societal Responsibilities

• Performance Projections will only be a factor up in the 90-100% scoring ranges. The “hows” will be expected to be addressed in Categories 2 and 4.

13

General Upcoming Changes

• The criteria will now “gently introduce” both topics of Social Media and Intelligent Risk.

• The sub-areas listed within the items will now have Titles to help improve the outlines and as a guide for better organization.

14

What is a Key Theme?

A high level perception or observation, relevant to the applicant’s key factors, that recurs in the scorebook and is expanded upon with evidence during a site visit. Key themes are often common to more than one category or item. They may address one or more Baldrige core value.

A role model result, practice, process, or innovation, that might be used by other organizations, or an observation so important, strength or OFI, process or result, that it should be brought to leadership’s attention.

The executive summary of the feedback report. It may be the only part of the feedback report that some people in the organization read.

Cross Cutting Themes

• In the early days of the Baldrige, Key Themes were called Cross Cutting Themes, or Cross Cutting Issues, because they described topics that cut across the categories and items.

Core Values-The Foundation of the Baldrige Criteria

• Visionary Leadership• Customer driven excellence• Organizational and personal learning• Valuing workforce members and

partners• Agility• Focus on the future• Managing for innovation• Management by fact• Societal responsibility• Focus on results and creating value• Systems perspective

• If you think of the 7 Baldrige categories as the vertical beams in a house frame, the 11 Baldrige Core Values are the cross members that provide strength to the frame.

Process for using Baldrige Core Values to Develop Key Themes

• Each Team Member:• Review the applicant • Read the Baldrige Core Values• Write 11 post-its rating the applicant against each Baldrige Core

Value with a +, 0, or – and write a few short thoughts for each core value explaining why you rated the applicant as you did.

• Key Themes Lead• Sum the plusses and the minuses. Core Values that have many

plusses, or minuses, may be candidates for Key Themes. Typically only a few will stand out.

• Report results to team. • Draft Key themes for these high scoring values using feedback

report comments and post-it thoughts as inputs.

Using Core Values to ID Key ThemesBaldrige Core Value Score (+,0,-)

Visionary Leadership +,+,+,+,+,+ (6+) Probable KT

Customer driven excellence +,+,0,0,0,-

Organizational and personal learning +,+,+, 0,0,0

Valuing workforce members and partners +,+,+,+,+,- (5+) Possible KT

Agility +,+,0,0,-,-

Focus on the future +,+,+,0,-,-

Managing for innovation 0,0,-,-,-,-

Management by fact -,-,-,-,-,-, (6-) Probable KT

Societal responsibility +,+,+,+,+,0 (5+) Possible KT

Focus on results and creating value 0,-,-,-,-,- (5-) Possible KT

Systems perspective +,+,0,0,-,-

Caveat to this approach

• While the Baldrige Core Values are the foundation upon which the Criteria are based, they are not examination items. Key themes are written based on comments developed relating to the criteria. The feedback report comments MUST support the Key Themes.

Six-Step Evaluation Process

Step 1: Ground yourself in the Criteria Item

Item Question Organization

Glossary

Understanding the Questions

• The quality of the feedback provided depends on understanding of the meaning and significance of the questions

• The Criteria book provides aids to understanding

• A Glossary of Key Terms is provided; Terms with definitions are identified with CAPs

• Open your criteria, find a few Key Terms and look up the definition

• Notes are provided to clarify key requirements, give instructions, and indicate important linkages

• Read N1 on Item 1.1 (page 7) to find an expectation concerning the applicant’s Vision

• Category and Item Descriptions discuss the purpose for the questions and comment on expectations for answers

• Read the first Comment in Section 1.1 and consider how it will affect the applicant’s discussions in 1.1

3. WSQA Assessment Process

Objective:

Introduce the Assessment process

What Examiners Do…

Analyze Response

Read Criteria

Read Applicant Response

Meet with co-examiners to

consolidate analysis

Visit the Applicant to clarify/verify

Co-write an analysis report

Record Analysis

Preparing for independent review

• Read the criteria• Read the application• Familiarize yourself with the online toolhttp://www.wsqa.net/extraining.php• Check in with Team Lead or Mentor after preparing

a few Key Factors

27

Independent review:Process Items

Objective: Be able to diagnose a process item

for independent review

Independent review steps - process

1. Review related criteria2. Read the category under review, take notes3. Select relevant key factors from list4. Identify & record processes5. Record observations on worksheet under strengths

and Opportunities6. Indicate

• Approach• Deployment• Learning• Integration

29

Identify and record processes

• “Process” includes• Inputs• Steps (related activities)• Time frames• Outputs• End users (internal/external customers)• Standards• Key measures to evaluate and improve the process

• Not always formal or named – you might have to read between the lines

• Look to criteria for guidance

30

“Systematic” refers to processes that• Are well-ordered• Are repeatable• Use data and information so learning is possible• Build in the opportunity for evaluation,

improvement, and sharing

Identify and record processes

31

• “Effective” refers to how well a process or measure addresses its intended purpose

• Determining effectiveness requires evaluating• How well the approach is aligned with the organization’s

needs• How well it is deployed, and/or• How the evaluation of the outcome of its measures are

used, i.e., for learning and improvement

Identify and record processes

32

Indicating ADLI- Approach

• “Approach” refers to• Methods used to accomplish the process• How appropriate the methods are to the item

requirements• The effectiveness of their use• The degree to which the process is repeatable

and is based on reliable data and information (i.e., systematic)

33

Indicate “ADLI”- Deployment

“Deployment” refers to the extent to which• The approach is applied in addressing relevant

and important item requirements• It is applied consistently• It is used by all appropriate work units

34

Indicate “ADLI”- Learning

“Learning” refers to • Refining the approach through cycles of

evaluation and improvement• Encouraging breakthrough change to your

approach (innovation)• Sharing refinements and innovations with other

relevant work units and processes

35

Indicate“ADLI”- Integration

“Integration” refers to the harmonization of• Plans• Processes• Information• Resource decisions• Actions• Results• Analyses

to support key, organization-wide goals

36

Indicate ADLI- Integration

Questions to ask in analyzing “Integration” • Do the individual components of a performance

management system operate as a fully interconnected unit?

• Is the approach aligned with the organizational needs identified in the Organizational Profile?

37

ADLI-key concepts

Integration examples:

• Alignment of objectives and action plans with strategic challenges AND mission, vision, values

• Alignment of product/service delivery methods with KEY customer requirements

• Alignment of key process measures with KEY customer requirements

38

Scoring Categories 1 – 6 (Process Categories)

• Examiners select a score (0-100) to summarize their observed strengths and opportunities for improvement (OFI’s)

• Scoring Guidelines are provided for Approach, Deployment, Learning, and integration

• Scores are assigned for each ITEM

Scoring Guidelines for Approach SCORE PROCESS

0% -5%No SYSTEMATIC APPROACH is evident; information is ANECDOTAL. (A)

10% - 25%The beginning of a SYSTEMATIC APPROACH to the BASIC REQUIREMENTS of the Item is evident. (A)

30% - 45%An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the BASIC REQUIREMENTS of the Item, is evident. (A)

50% - 65%An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the OVERALL REQUIREMENTS of the Item, is evident. (A)

70% - 85%An EFFECTIVE, SYSTEMATIC APPROACH, responsive to the MULTIPLE REQUIREMENTS of the Item, is evident. (A)

90% - 100%An EFFECTIVE, SYSTEMATIC APPROACH, fully responsive to the MULTIPLE REQUIREMENTS of the Item, is evident. (A)

Category/Item Question Organization

BASIC

OVERALL MULTIPLE

Scoring Guidelines for DEPLOYMENTSCORE PROCESS

0% -5% Little or no DEPLOYMENT of an APPROACH is evident.

10% - 25%The APPROACH is in the early stages of DEPLOYMENT in most areas or work units, inhibiting progress in achieving the BASIC REQUIREMENTS of the Item.

30% - 45%The APPROACH is DEPLOYED, although some areas or work units are in early stages of DEPLOYMENT.

50% - 65%The APPROACH is well DEPLOYED, although DEPLOYMENT may vary in some areas or work units.

70% - 85% The APPROACH is well DEPLOYED, with no significant gaps.

90% - 100%The APPROACH is fully DEPLOYED without significant weaknesses or gaps in any areas or work units.

Scoring Guidelines for LEARNING

SCORE PROCESS

0% -5%An improvement orientation is not evident; improvement is achieved through reacting to problems.

10% - 25%Early stages of a transition from reacting to problems to a general improvement orientation are evident.

30% - 45%The beginning of a SYSTEMATIC APPROACH to evaluation and improvement of KEY PROCESSES is evident.

50% - 65%A fact-based, SYSTEMATIC evaluation and improvement PROCESS and some organizational LEARNING are in place for improving the efficiency and EFFECTIVENESS of KEY PROCESSES.

70% - 85%Fact-based, SYSTEMATIC evaluation and improvement and organizational LEARNING are KEY management tools; there is clear evidence of refinement and INNOVATION as a result of organizational-level ANALYSIS and sharing.

90% - 100%Fact-based, SYSTEMATIC evaluation and improvement and organizational LEARNING are KEY organization-wide tools; refinement and INNOVATION, backed by ANALYSIS and sharing, are evident throughout the organization.

Scoring Guidelines for INTEGRATION

SCORE PROCESS

0% -5%No organizational ALIGNMENT is evident; individual areas or work units operate independently.

10% - 25%The APPROACH is ALIGNED with other areas or work units largely through joint problem solving.

30% - 45%The APPROACH is in early stages of ALIGNMENT with your basic organizational needs identified in response to the Organizational Profile and other Process Items.

50% - 65%The APPROACH is ALIGNED with your organizational needs identified in response to the Organizational Profile and other Process Items.

70% - 85%The APPROACH is INTEGRATED with your organizational needs identified in response to the Organizational Profile and other Process Items

90% - 100%The APPROACH is well INTEGRATED with your organizational needs identified in response to the Organizational Profile and other Process Items.

Overall Score for Categories 1 – 6 Items

• An applicant’s score for approach [A], deployment [D], learning [L], and integration [I] depends on their ability to demonstrate the characteristics associated with that score

• Although there are 4 factors to score for each Item, only one overall score is assigned

• The examination team selects the “best fit” score, which is likely to be between the highest and lowest score for the ADLI factors

“Best Fit” ExampleConsider two applicants, both scoring 60% for A, D, and I, and 20% for L• Applicant Market Growth Technology

Competitors A 10%/year Stable Stable B 2X/year Major change Many new

• Item Score:

• Applicant A: 50%

(Learning could lead to incremental results improvements)• Applicant B: 30% (Learning needed to sustain the organization)

Step 5: Draft feedback-ready comments

Step 6: Determine the scoring range and score for the Item

Independent Process Review

• Refer to samples

49

6. Independent review:LeTCI – Results Items

Objective: Be able to evaluate a results item

for independent review

Using a Table of Expected Results to Identify Category 7 OFI’s

Purpose:

Identify results that the applicant hasn’t provided

WHY?

Because applicants like to show results that are favorable, but may not always show results that are important.

Table of Expected Results

Title of Expected Result

Source Reference

Category 7Reference

Result Reference

(Describe expected result and segments)

(List where in the application you learned this was important to report in Cat 7)

(List where in Cat 7 the result belongs)

(Identify the page or Figure that contains the results)

Examples: Examples: Examples:

Status of Action Plans (% of Action Plans on target)

OP pg.7 1, 2.1a2 Not found

Workforce segments OP p6 7.4 Expect to see segmentation by FT, PT and on-call staff

Found in 7.4-4, 6, 10Not found in 7.4-7, 8, 9

Process for using a Table of Expected Results

As you read the application, pay attention for processes that the applicant cites as important, measures that are discussed, segments that are used, etc.

As you find examples of these, include them in the table of expected results in columns 1-3.

When you review Category 7, review it for the expected results and complete column 4.

Based on the comments in Column 4 of the table, an OFI, or OFI’s can be drafted citing missing expected results.

Note: These are results that the applicant has introduced an expectation for in the Organizational Profile and Categories 1-6. They are NOT results that the examiner would like to see.

Evaluation Factors for Category 7 (results)

• Le = Performance LevelsNumerical information that places an organization’s results on a meaningful measurement scale. Performance levels permit evaluation relative to past performance, projection goals, and appropriate comparisons.

• T = TrendsNumerical information indicating the direction, rate and breadth of performance improvements. A minimum of 3 data points is needed to begin to ascertain a trend. More data points are needed to define a statistically valid trend.

• C = ComparisonsEstablishing the value of results by their relationship to similar or equivalent measures. Comparisons can be made to results of competitors, industry averages, or best-in-class organizations.

• I = IntegrationConnection to important customer, product and service, market, process and action plan performance measurements identified in the Organizational Profile and in Process Items.

• G = Gaps Absence of results addressing specific areas of Category 7 Items, including the absence of results on key measures discussed in Categories 1–6

Results Evaluation Factors

45

50

55

60

65

70

75

80

85

90

95

93-94 95-96 97-98 99-00 01-02 03-04

Per

cen

tile

MCB Percentile Top 10%National Mean MCB Trend

Le

Trend

Comparison

Denotes “Good” Trend

direction

Results strengths and OFI;s

• Strengths are identified if the following is observed:• Performance levels [Le] are equivalent or better than

comparatives, and/or benchmarks• Trends [T] show consistent improvement, and• Results are linked [Li] to key requirements

• Opportunities for improvement are identified if:• Performance levels [Le] are not as good as comparatives• Trends [T] show degrading performance• Comparisons are not shown• Results are not linked [Li] to key requirements• Results are not provided [G] for key processes and/or

action items

Scoring a Results Item

• Scoring a Results item is similar to a Process Item

• Scoring Guidelines for Le, T, C, and I

• A single “best fit” score is selected for each results item

Results Scoring for Level [Le]SCORE RESULTS

0% -5%There are no organizational PERFORMANCE RESULTS or poor RESULTS in areas reported.

10% - 25%A few organizational PERFORMANCE RESULTS are reported; there are some improvements and/or early good PERFORMANCE LEVELS in a few areas.

30% - 45%Improvements and/or good PERFORMANCE LEVELS are reported in many areas addressed in the Item requirements.

50% - 65%Improvement TRENDS and/or good PERFORMANCE LEVELS are reported for most areas addressed in the Item requirements.

70% - 85%Current PERFORMANCE LEVELS are good to excellent in most areas of importance to the Item requirements.

90% - 100%Current PERFORMANCE LEVELS are excellent in most areas of importance to the Item requirements.

Results Scoring for Trend [T]SCORE RESULTS

0% -5% TREND data either are not reported or show mainly adverse TRENDS.

10% - 25%Little or no TREND data are reported, or many of the trends shown are adverse.

30% - 45% Early stages of developing TRENDS are evident.

50% - 65%No pattern of adverse TRENDS and no poor PERFORMANCE LEVELS are evident in areas of importance to your organization’s KEY MISSION or business requirements.

70% - 85%Most improvement TRENDS and/or current PERFORMANCE LEVELS have been sustained over time.

90% - 100%Excellent improvement TRENDS and/or consistently excellent PERFORMANCE LEVELS are reported in most areas.

Results Scoring for Comparison [C]SCORE RESULTS

0% -5% Comparative information is not reported.

10% - 25% Little or no comparative information is reported.

30% - 45% Early stages of obtaining comparative information are evident.

50% - 65%Some TRENDS and/or current PERFORMANCE LEVELS—evaluated against relevant comparisons and/or BENCHMARKS—show areas of good to very good relative PERFORMANCE.

70% - 85%

Many to most reported TRENDS and/or current PERFORMANCE LEVELS—evaluated against relevant comparisons and/or BENCHMARKS—show areas of leadership and very good relative PERFORMANCE.

90% - 100%Evidence of industry and BENCHMARK leadership is demonstrated in many areas.

Results evaluation: key concepts

• Integration• Results align with key factors, e.g.,

• Strategic challenges, • Workforce requirements• Vision, mission, values

• Results presented for• Key processes• Key products, services• Strategic accomplishments

• What examples can you think of?• Strong integration?• Not-so-strong integration?

61

Results evaluation: key concepts

• Strong integration: results presented for • Key areas addressing strategic challenges, • Key competitive advantages • Key customer requirements

• Not-so-strong integration:• Results missing for the above• Results presented that the Examiner can’t match to

process items or Organizational Profile

62

Results Scoring for Integration [I]SCORE RESULTS

0% -5%RESULTS are not reported for any areas of importance to your organization’s KEY MISSION or business requirements.

10% - 25%RESULTS are reported for a few areas of importance to your organization’s KEY MISSION or business requirements.

30% - 45%RESULTS are reported for many areas of importance to your organization’s KEY MISSION or business requirements.

50% - 65%Organizational PERFORMANCE RESULTS address most KEY CUSTOMER, market, and PROCESS requirements

70% - 85%Organizational PERFORMANCE RESULTS address most KEY CUSTOMER, market, PROCESS, and ACTION PLAN requirements.

90% - 100%Organizational PERFORMANCE RESULTS fully address KEY CUSTOMER, market, PROCESS, and ACTION PLAN requirements.

“Key” results

• Refers to elements or factors most critical to achieving the intended outcome

• In terms of results, look for• Those responsive to the Criteria requirements• Those most important to the organization’s success• Those results that are essential elements for the

organization to pursue or monitor in order to achieve its desired outcome

64

Step 4:Draft around six strengths and opportunities for improvement

Step 5: Draft feedback-ready comments

Results worksheet example

• REFER TO EXAMPLE IN MANUAL

67

Writing Well Written Comments

69

How do good comments add value?

Criteria

Applicationcontent

OrganizationProfile

The most valuable Comments/Key Themes

70

The Structure of Well Written Comments

Well written comments incorporate criteria language, ADLI/LeTCI, scoring language, and application language.

71

Well Written Comment Guidelines

Category 1-6 Does the comment link to the Criteria? Does it consist of 2-3 grammatically correct sentences

(about 1000 characters)? Does it use examples from the application that illustrate the

OFI/Strength? Does it include a so what if it is an OFI? Is the statement non-judgmental and non-prescriptive? Does the comment link to the scoring guidelines?

72

Well Written Comment Guidelines • Tie your comments to the basic, overall, or multiple level Criteria

requirements that are most important to the applicant. • Write process comments so they contain: a subject identified from

the criteria, the application, or the scoring guidelines; verb(s) and requirements from the Criteria; examples from the application, and figure numbers, as appropriate.

• Use the evaluation factors (ADLI or LeTCI), and scoring language, to clearly articulate the areas of the applicant’s response to clarify the strength or provide insights that will help the applicant improve overall organizational effectiveness and capabilities.

• Draw linkages between an Item and the applicant's Organizational Profile.

• Ensure that the comment does not contradict other comments in the same or other Items or in the Key Themes Worksheet.

73

The Style of Well Written Comments• Use active voice and present tense (e.g., 'completes' rather than 'is

completed'). • Use a polite, professional, and positive tone. • Use vocabulary/phrasing from the Criteria and the Scoring

Guidelines. • Describe what is missing if something "is not clear". “Not clear”

comments must be resolved if the applicant has a site visit. If they don’t do it based on site experience then say they don’t do it.

• In lieu of the applicant’s name, use such phrases like "the applicant" or "the organization" to refer to the applicant in the independent and consensus scorebooks. Use the applicant's name in final feedback reports.

• Use the applicant's language/terminology when appropriate.

74

“Don’ts” to avoid in comments

1. Avoid parroting2. Avoid being prescriptive 3. Avoid going beyond the criteria

75

“Do’s” for well-written comments

(refer to comment guideline in workbook)

76

Example Strength Comment• The applicant identifies their core competencies as the

participating dentist network; expert claims administration; extraordinary customer service and performance culture. A SWOT analysis is used in the annual strategic planning process to identify core competencies. Correlation analysis is performed on satisfaction survey data and is used to determine key drivers of customer satisfaction. Initiative action plans accentuate core competency strengths. Deployment is evidenced by the use of bi-annual CAG meetings, ensuring alignment with the mission and the integration of data for easy access across all departments and customer segments

• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor

77

Well Written Comments Don’t….

• Go beyond the requirements of the Criteria or assert your personal opinions.

• Be prescriptive by using "could", "should", or "would". • Be judgmental by using terms such as "good", "bad", or

"inadequate". • Tell the applicant that they aren’t doing enough of something• Comment on the applicant's style of writing or data

presentation. • Use jargon or acronyms unless they are used by the applicant • Only “Parrot” the application or the Criteria. Provide only

enough language to add clarity—seek to add value rather than restate information.

78

How Could This Comment Be Improved?Application language:• Share Food uses a biennial 12 step Strategic Planning Process that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the “Current State” step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. The Strategic Planning Process was initiated in 1997 and restructured in 2004 as a result of benchmarking analysis of food banks and other nonprofit organizations, and feedback from the state award process.

Comment: • Share Food uses a biennial 12 step Strategic Planning Process (SPP) that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the Define step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. This is proof that the organization has a great systematic approach to the SPP and extensive cycles of learning in 1997, and 2004.

79

How Could This Comment Be Improved?Application language:• The manager’s vision helps Our Store to maintain a strong

focus on customer satisfaction. The manager has worked hard to instill the belief that leadership and responsibility should be present at all levels. This rewards everyone at Our Store with the responsibility of planning, carrying out, and evaluating their own work.

Comment:• Our Store is in the beginning stages of building an approach to

setting a vision for customer satisfaction, this includes the manager working very hard to instill the beliefs in leadership and responsibility. Everyone in the store receives a reward and this will ensure that everyone will have ownership in the organization and the ability to evaluate their own work, which should improve performance overall.

80

How Could This Comment Be Improved?Application language:• Our Hospital builds relationships with customers through a variety of

methods including patient enrollment, which serves as an orientation to enabling services; Patient-Family Advisory Boards, which have recommended and implemented improvements to patient care; and high-visibility community health activities. Using patient surveys, conducted randomly after discharge, Our Hospital determines the issues that lead to poor relationship results and uses this information to implement new relationship-building processes. The Patient-Family Advisory Board is an example of the results of this process.

Comment:• Customer relationships are built through a variety of methods including

Patient enrollment, Patient-Family Advisory Boards, and community health activities. Surveys are conducted randomly after patients are discharged. The Patient-Family Advisory Board is used to determine issues that lead to poor relationships and implements new relationship-building processes.

81

How Could This Comment Be Improved?Application language:• Customer groups are identified annually as part of the Strategic

Planning Process using Our Hospital’s vision/mission/values as a focus. By analyzing demographic data from multiple sources, the cross-location Service with Spirit Team (SWST) is able to identify the gaps, look at disparities, and identify potential customers. Deployment of this process through multiple cycles has resulted in the development of several services designed to meet the unmet needs of customers in the Our Hospital’s service area.

Comment:• Through the SPP, customer groups are annually identified using the

MVV as a focus. The SWST uses demographic data to identify gaps, disparities and identify potential customers. This process has been deployed throughout the organization, and through multiple cycles of refinement the customers needs in the service area have been met.

82

Well Written OFI Comment Guidelines• May be used to give credit to the applicant for some areas that

HAVE been addressed• “While the applicant does x, y, z, it is not evident that a, b,

and c are performed.”• May begin with a statement about what is missing:

• “There is no evidence that…”• “There is no indication of how…”• “It is unclear…”• “It is not apparent that there is a …”

• Must tie to criteria language, (ADLI), and/or to Key Factors that the applicant has identified in the Organizational Profile

• Must include a “so what” (Why does the applicant care about this? Why would they take action?)

• Usually these come from Key Factors / Org Profile. The closer the ‘so what’ can be to identifying the applicant’s specific needs, the better.

83

Example OFI Comment

• Although cross functional teams are chartered and aligned with the SPP, there is no indication of processes used by the teams or otherwise by the applicant to design and innovate its work processes to meet key requirements. There is no indication of how design and innovation processes address opportunities for agility and improvements in cycle time and productivity. It is also unclear how the team work process improvement approach is evaluated for annual process changes or improvements. An annual evaluation of strategic team alignment and improvements may provide efficiency and effectiveness in the accomplishment of strategic objectives, initiatives, and action plans.

• Red-Criteria Green-ADLI Blue-Application Pink-Key Factor

84

How Could This Comment Be Improved?Application language:• Share Food uses a biennial 12 step Strategic Planning Process that

involves community leaders, volunteers, member agencies, and donors. Strategic challenges are identified in the “Current State” step which includes inputs from the SWOT analysis, environmental scan, and funding mandate review. The Strategic Planning Process was initiated in 1997 and restructured in 2004 as a result of benchmarking analysis of food banks and other nonprofit organizations, and feedback from the state award process.

Comment: • Although Share Food uses a biennial 12 step Strategic Planning Process

(SPP) that involves community leaders, volunteers, member agencies, and donors, there is no evidence of how blind spots are identified, nor how time horizons are set and how the SPP addresses those horizons. If the applicant implemented a more extensive benchmarking process that identified blind spots and included time horizons, there may be more of an opportunity for this organization to have a more successful Strategic Planning Process.

85

How Could This Comment Be Improved?Application language:• The manager’s vision helps Our Store to maintain a strong

focus on customer satisfaction. The manager has worked hard to instill the belief that leadership and responsibility should be present at all levels. This rewards everyone at Our Store with the responsibility of planning, carrying out, and evaluating their own work.

Comment:• Although the manager has worked hard to instill the belief

that leadership and responsibility should be present at all levels, there is no indication of a systematic approach, nor is there deployment at any level, no integration, nor are there any cycles of learning.

86

How Could This Comment Be Improved?Application language:• Our Hospital builds relationships with customers through a variety of

methods including patient enrollment, which serves as an orientation to enabling services; Patient-Family Advisory Boards, which have recommended and implemented improvements to patient care; and high-visibility community health activities. Using patient surveys, conducted randomly after discharge, Our Hospital determines the issues that lead to poor relationship results and uses this information to implement new relationship-building processes. The Patient-Family Advisory Board is an example of the results of this process.

Comment:• There is no systematic approach to building customer relationships, or in

conducting patient surveys. And although there is a Patient-Family Advisory Board there is no approach described for how issues are determined and how new relationship-building processes are developed.

87

How Could This Comment Be Improved?Application language:• Customer groups are identified annually as part of the Strategic

Planning Process using Our Hospital’s vision/mission/values as a focus. By analyzing demographic data from multiple sources, the cross-location Service with Spirit Team (SWST) is able to identify the gaps, look at disparities, and identify potential customers. Deployment of this process through multiple cycles has resulted in the development of several services designed to meet the unmet needs of customers in the Our Hospital’s service area.

Comment:• Although customer groups are identified annually in the SPP, and the

Spirit Team (SWST) identifies gaps, disparities, and identifies potential customers, there is no indication how this information is deployed throughout the organization and how the information is used to meet the unmet needs of customers in the area.

88

Comment Writing Exercise• Each person at the table select an independent

review item (Cat 1-6 strength or OFI)• Write a well written comment• Swap scorebooks• What feedback would you give to this examiner

• Use the comment checklist for examples • Practice giving feedback to each other• Report out the most significant key learnings• (If time swap with to another person at the

table)

89

Category 7 Comment GuidelinesCategory 7

Are specific results (numbers) used rather than generalities?

Have trends, comparisons and/or appropriate benchmarks been considered?

Does the comment include figure numbers? Have we summarized multiple charts with a consistent

theme (trends, missing, comparisons, etc.)

90

(Well Written Result Comment Example

There is a lack of comparative data in a number of the Results sub-categories; as an example, no comparative data is available for measures such as Discharge Weight of ELBW Infants (7.1-4), Improvement % of ELBW Infants (Figure Figure 7.1-5), % of Breast Milk Available (Figure Figure 7.1-6), Outpatient Lactation (7.1-7) and Catheter Related Bloodstream Infections (7.1-8), Survival to Discharge by Rhythm (Figure Figure 7.1-17), and Well Child Visits Figure (7.1-19). The lack of comparative results may impact the value of Excellence - setting and achieving the highest standards.

91

How Could This Comment Be Improved?

The applicant’s market share results for key service lines is strong, exceeding all comparative data across Discharges; Inpatient Surgeries; the applicant's Heart Services; and Neurosurgery Market. The applicant enjoys market dominance in these service lines and has shifted its focus to adolescent care and services to new geographic markets. The applicant has received numerous recognitions and awards for their leadership in these markets and service lines. These successes are important to their goal of achieving national recognition as a premier hospital, a tertiary and quaternary hub, and preferred provider of regional health care services.

92

How Could These Comments Be Improved?

• The applicant's Net Income from Operations (7.3.8) is above the comparison median and trending up over the past three years and overall applicant’s turnover rate has been decreasing.

• The applicant’s Net Sales between 2006 and 2008, have increased by 10% in 2006 to over 12.2% in 2008. The applicant is consistently above the goal of 9% for all three years, but is below the Baldrige comparison of 15% for all three years.

• Although the turnover rate for RNs has been increasing between 2006 and 2008. The turnover rate for RNs was 7.7% in 2006 and 10.3% in 2008. There have been substantial improvements in physician satisfaction from 13% to over 25% during the same time period.

93

How Could These Comments Be Improved?

• The applicant's Net Income from Operations (7.3.8) is below the comparison median and trending down over the past three years and overall applicant’s turnover rate has been increasing between 2006 and 2008, from 10% in 2006 to 12.2% in 2008.

• The turnover rate for RNs has been increasing between 2006 and 2008. The turnover rate for RNs was 7.7% in 2006 and 10.3% in 2008.

94

Comment, scoring considerations

• For LITE, only one score for entire results Category 7• Propose a score for results related to your

categories• Be prepared to reach team consensus for all of

Category 7• Using scoring language can help the

applicant understand comments and score

95

Category feedback

• Check and provide feedback on whether Cat. Lead’s comments:• Include criteria requirements• Provide meaningful feedback• Address ADLI or LeTCI• OFI’s provide relevant “so what” (at least some of them)• Results cite figure numbers/titles and specifics• References to application are correct• Recommended score reflects comments in content and

balance (strengths:OFI’s ratio)• Back-up does this for full, team does this for lite

96

Information resources

• WSQA comment guideline handout• Your back-up, team lead, PEPD mentor• Reference materials in

• Your workbook: comment and scorebook examples• http://www.wsqa.net/extraining.php• Baldrige web site

http://baldrige.nist.gov/Examiner_Resources.htm

Six-Step Evaluation Process

Key themes

Objectives:

1. Be able to identify Key Themes

2. Be able to develop a Key Theme comment

99

How do good comments add value?

Criteria

Applicationcontent

OrganizationProfile

The most valuable Comments/Key Themes

100

What is a Key Theme?A “key theme” is a strength or OFI that • Addresses a central requirement of the

Criteria,• Is common to more than one Item or

Category (crosscutting), • Is especially significant in terms of the

applicant’s key factors, and/or • Addresses a Core Value of the Criteria

101

Developing Key Theme comments

• Address all double strengths and OFI’s individually and/or in groups

• Address related comments across categories• Develop 6-10 (Guideline!) key themes each for

significant• Process strengths• Process OFI’s• Results strengths • Results OFI’s

• See example Key Themes in the workbook for this class

Key Theme Development using the Baldrige Core Values

Purpose:

Create Key Themes at the end of the consensus meeting or at the end of a site visit

WHY?

to identify Key Themes that cut across categories and items

103

Managing your time

• Put aside time in your calendar (refer to time commitments)

• Work with your lead & team• Set meetings early• Coordinate• Communicate

• Stories to tell?

104

Managing your time

• Schedule work hours• Break tasks into little

pieces• Reward yourself for wins• Move quickly past

setbacks• Call your lead• Call your back-up

Software• Link to softwarehttp://www.wsqa.net/extraining.php• Username and Password: firstlast name• Link for software manual and taped webinarhttp://www.wsqa.net/Extimetable.phpClick on scorebook, select Key factors, then

select Item evaluation Save Frequently, Save OFTEN, Save A Lot!-

save in software- not on your browser

105

Your questions?

106

Your feedback, please!

• Please complete surveys before leaving

107

THANK YOU!!

Your support and participation as Examiners helps us all by helping WSQA

fulfill its mission!

108