Baldrige National Quality Program National Institute of Standards and Technology Technology Administration Department of Commerce

2007 Independent Review Toolkit

CONTENTS Page

What Is a Key Factor?........................................................................................................1

How to Identify Key Factors…..........................................................................................2

Considerations for Small Organizations............................................................................3

Four Common Acronyms for Accreditation and Regulation.............................................5

Process Item Evaluation Factors .......................................................................................7

Attributes of Meaningful, Well-Written Process Observations.........................................9

Results Item Evaluation Factors ......................................................................................10

Attributes of Meaningful, Well-Written Results Observations........................................12

Independent Review Worksheet Annotated Example: Item 3.1.......................................13

Independent Review Worksheet Annotated Example: Item 7.2.......................................17

Additional Example: Item 7.6...........................................................................................21

Results Matrix...................................................................................................................26

WHAT IS A KEY FACTOR? Definition Key Factors are significant attributes of an organization that influence

the way the organization operates. Examiners will use key factors to focus their assessment on what is important to the applicant.

Customers Examiners use key factors to provide more accurate and meaningful feedback to the applicant understand the Criteria areas of greatest importance to the applicant begin to understand the key points to investigate during the

consensus and site visit stages of the Award process Judges use key factors to

quickly acquaint themselves with the applicant focus on what is most important

Examples Mission, vision, and values Employee/staff profile Customer and market segments and customer requirements Competitive position and critical success factors Strategic challenges Governance structure

These are all key factors because they are facts or attributes that affect the way the organization operates.

Nonexamples Complaint management process Strategic planning approach

These are not key factors; rather, they are processes that the applicant uses in managing its work.

1

How To Identify Key FactorsLook for key factors in . . .

responses to the Organizational Profile responses to the Criteria Items—it is important to consider key

factors wherever they are found in the application. eligibility forms

Key Factors Worksheet Guidelines

Limit the worksheet to 1–2 pages. Group the key factors under the five Areas to Address in the

Organizational Profile (e.g., Organizational Environment, Organizational Relationships).

Write in phrases rather than complete sentences. Delineate each key factor with a bullet. Revise the worksheet throughout the evaluation process as additional

information and insights are gained during the consensus and site visit stages.

Use the following steps in developing the Key Factors Worksheet.Step Action

1 Read the eligibility forms.2 Read the Organizational Profile.3 Consider the facts that most affect the way the organization operates.4 List these facts in accordance with the guidelines on the Key Factors

Worksheet.5 Revise the list as you identify additional facts in the application.6 Review the list after you complete your evaluation of the application

to determine if key factors should be added or dropped.

Independent Review Worksheet Guidelines

Consider what key factors would impact individual Item requirements. Select the 4–6 key factors that are most relevant for each Item—these will

be a subset of the Key Factors Worksheet. List these at the top of each Independent Review Worksheet.

2

Considerations for Reviewing Small Organizations

All applicants are reviewed in the context of their individual key factors. In the case of small organizations, size is a significant key factor. While an organization’s size does not affect the applicability of the Baldrige Criteria, it does need to be factored into the assessment of an applicant’s responses in its Baldrige Award application. Therefore, Examiners with large-organization frames of reference should be careful not to apply operational and procedural requirements as they review small organization applications.

Some guidelines are given below for understanding the context for reviewing a small organization:

Small organization applicants are defined as those with 500 or fewer employees. Also noteworthy is the significant difference in resource availability between a 450-person organization and a 50-person organization.

Social responsibility and community involvement must be viewed in the context of the applicant’s size. A large organization might have impacts on a national or international basis; a small organization will frequently focus its involvement on a local community.

The issues of fiscal and managerial accountability, ethical behavior, and legal compliance are as pertinent to a small organization as they are to a large one, and the responses of management to these issues are equally important. A small organization, however, will necessarily address these issues in the context of its size, ownership (many are privately held or family-owned), and responsibilities. Good governance practices are still an imperative.

While large organizations frequently have complex computer/information systems for data management, a small organization (depending upon how small) may perform data and information management with a combination of personal computer- or work station-based data management systems and manual methods.

Due to limited workforce and funding resources, benchmarking and competitive comparison information in a small organization environment may be based largely on literature/trade association information and comparisons with best practices in the local geographic area.

In the context of a small organization, systems for workforce involvement and process management may rely more on informal verbal communication than on formal written communication and documentation. However, all applicants have the same requirement to demonstrate that their processes are repeatable, can produce the desired results, and are deployed fully and systematically throughout the organization.

The ability of a small organization to leverage key suppliers, particularly large suppliers, has to be viewed in the context of workforce availability and the volume of business that it does with the supplier.

3

The ability of a small organization to obtain customer and market knowledge through independent third-party surveys, commissioned studies, extensive interviews, or focus group techniques is limited by its resources. The important consideration for Examiners is to assess whether the applicant, given its resources, is using appropriate mechanisms to gather and use information to improve its customer and market focus and satisfaction.

The expectation that large organizations will segment their results data with regard to various customer and workforce segments may require modification in small organizations, depending on the complexity of these groups and the level of resources needed to gather and analyze the data.

4

Four Common Acronyms for Accreditation and Regulation

This sheet is designed to provide some basic information on sector-specific accrediting organizations and/or regulations that may be mentioned in applications you evaluate for the Baldrige program.

K-12 Education

NCLB - the No Child Left Behind Act was signed into law January 8, 2002. It is the latest revision of the 1965 Elementary and Secondary Education Act (ESEA). The overall purpose of the law is to ensure that each child in America is able to meet the high learning standards of the state in which he or she lives by the target date of 2014. A key provision of NCLB is that schools, school systems, and states must demonstrate continuous improvement by achieving Adequate Yearly Progress (AYP) both as a whole and for each subgroup -- major racial groups, students with disabilities, economically disadvantaged students, and students with limited English proficiency -- on the state identified achievement instruments. Each state is required to report on the AYP status of every school and district annually to parents and students. A school that does not meet AYP for two consecutive years is identified as being in “improvement”, a status that requires corrective action and may lead to supplemental services and school choice for students and parents.

Health Care

HIPAA (Health Insurance Portability and Accountability Act) of 1996. The Standards for Privacy of Individually Identifiable Health Information (“Privacy Rule”) to establish a set of national standards for the protection of certain health information. The U.S. Department of Health and Human Services (HHS) issued the Privacy Rule to implement the requirements of HIPAA. The Privacy Rule standards address the use and disclosure of the health information of individuals (called “protected health information” by organizations called “covered entities”) subject to the Privacy Rule, as well as standards for individuals’ right to understand and control how their health information is used.

A major goal of the Privacy Rule is to ensure that individuals’ health information is properly protected, while allowing the flow of health information needed to provide and promote high quality health care and to protect the public’s health and well-being. The Privacy Rule is intended to strike a balance that permits important uses of information, while protecting the privacy of people who seek care and healing. Given that the health care marketplace is diverse, the Privacy Rule is designed to be flexible and comprehensive to cover the variety of uses and disclosures that need to be addressed.

JCAHO (Joint Commission on Accreditation of Healthcare Organizations) evaluates healthcare facilities’ compliance based on a focused set of requirements that are considered essential to the delivery of patient care. JCAHO’s mission is to continuously improve the safety and quality of care provided to the public through the provision of health care accreditation and related services that support performance improvement in health care organizations. JCAHO is

5

voluntary. However, it is required for organizations that receive federal funds, like Medicare and Medicaid. JCAHO is an independent, nonprofit organization established more than 50 years ago. It is governed by a board that includes physicians, nurses, and consumers. JCAHO sets standards by which health care quality is measured in America and around the world, and evaluates the quality and safety of care for more than 15,000 health care organizations. To maintain and earn accreditation, organizations must have an extensive on-site review by a team of JCAHO health care professionals. The purpose of the review is to evaluate the organization’s performance in areas that affect patient care.

Four Common Acronyms for Accreditation and Regulation

Business (and limited Health Care and Education)

ISO (International Organization for Standardization) 9000, ISO 9001 and ISO 14000 - ISO 9000 is an international reference standard for quality management system requirements in business to business dealings. ISO 14000 is a set of standards developed to assist organizations meet their environmental challenges.

The ISO 9000 and ISO 14000 standards are known as generic management system standards. Generic means that the same standards can be applied to any organization, large or small, whatever its product - including whether its "product" is actually a service - in any sector of activity, and whether it is a business enterprise, a public administration, or a government department. "Management system" refers to what the organization does to manage its processes, or activities. "Generic" also signifies that no matter what the organization is or does, if it wants to establish a quality management system or an environmental management system, then such a system has a number of essential features which are spelled out in the relevant standards of the ISO 9000 or ISO 14000 families.

ISO 9001 is specifically concerned with conformity assessment, meaning what the organization does to enhance customer satisfaction by meeting customer, product specification, and applicable regulatory requirements, and to improve continually its performance in this regard. Customers frequently require their suppliers to be “ISO 9001 certified.” This means an accredited independent auditor has certified in writing that ISO’s conformity assessment standard has been met. An organization can publicly advertise that it is ISO 9001 certified in an effort to gain credibility with potential customers. ISO 14000 is primarily concerned with environmental management, meaning what the organization does to minimize harmful effects on the environment caused by its activities and to improve continually its environmental performance.

6

PROCESS ITEM EVALUATION FACTORS What are the evaluation factors?

Feedback to Award applicants and the scoring of their responses to Criteria Items are based on two evaluation dimensions: (1) Process and (2) Results. In responding to Criteria requirements, applicants need to furnish information relating to these dimensions. The specific factors for Process—Approach, Deployment, Learning, and Integration (ADLI)—are described below.

ApproachDefinition “Approach” refers to the methods used by an organization to address the Baldrige

Criteria Item requirements in Categories 1–6. Features Appropriateness

EffectivenessUse of a systematic approachInnovation

Questions to ask in analyzing approach

What approach or collection of approaches is discussed? What areas of the Criteria Item does the approach address (e.g., 1.1a, 1.1b)? Is the approach systematic (with repeatable steps, inputs, outputs, and time

frames and designed to allow evaluation, improvement, and sharing)? Is there evidence that the approach is effective? Is this approach (or collection of approaches) a key organizational process? Is

the approach important to the applicant’s overall performance? (If yes, clearly state why it is important, and cite the key factors used to support your position.)

Are any of the multiple requirements of the Item that are not addressed (gaps) relevant and important to the applicant?

DeploymentDefinition “Deployment” refers to the extent to which an approach is applied in addressing the

requirements of a Baldrige Criteria Item. Deployment is evaluated on the basis of the breadth and depth of application of the approach to relevant work units throughout the organization.

Features RelevanceComplete coverageConsistencyBreadth across all work unitsDepth through multiple levels

Question to ask in analyzing deployment

What information is provided to show what is done in different parts of the organization to confirm the approach is deployed (shared or spread) throughout the organization

Is the approach in the early stages of deployment, well deployed but with some variation among areas/work units, well deployed with no significant gaps, or fully deployed?

LearningDefinition “Learning” refers to new knowledge or skills acquired through evaluation, study,

experience, and innovation. Organizational learning is achieved through research and development, evaluation and improvement cycles, employee and customer ideas and input, best-practice sharing, and benchmarking. Learning should be embedded in the way an organization operates.

7

Features Embedded in operations

Questions to ask in analyzing learning

Has the approach been evaluated and improved? If yes, is the evaluation and improvement conducted in a fact-based, systematic (e.g., regular, recurring, data-driven) manner?

Is there evidence of organizational learning (i.e., evidence that the learning from this approach is shared with other organizational units/other work processes)? Is there evidence of innovation and refinement from organizational analysis and sharing (e.g., evidence the learning is actually used to drive innovation and refinement)?

Are the measures, information, and improvement systems complementary across processes and work units?

IntegrationDefinition “Integration” refers to the harmonization of plans, processes, information, resource

decisions, actions, results, and analyses to support key organization-wide goals. Effective integration goes beyond alignment and is achieved when the individual components of a performance management system operate as a fully interconnected unit.

Features Begins with alignmentCulminates in interconnectivity

Questions to ask in analyzing integration

How well is the applicant’s approach aligned with the organizational needs identified in the other Criteria Items and the Organizational Profile?

How well is the approach integrated with these needs? (Examples of needs are strategic challenges, objectives, and related action plans; organizational mission, vision, and values; key processes and measures; key customer/market segments and requirements; and employee groups and requirements.)

How do I use these evaluation factors?

To ensure a more holistic evaluation of Process Items (Categories 1–6), answer the questions when drafting your comments. You should be able to strengthen and clarify comments using insights gained from answering the questions. Provide evidence from the applicant’s response to support your comments. Use unique applicant terminology when it’s provided.

8

Attributes of Meaningful, Well-Written Process Observations

Process observations should tell other Examiners…

Whether you think the Approach is systematic. If it is, observations should include evidence that the

approach is repeatable, such as steps, time frames, inputs, and outputs

Show evidence of whether the method is appropriate, in the context of the organization’s key factors

Whether you think the approach is in the early stages of Deployment, well deployed, or fully deployed without gaps

Include specifics about the breadth and depth of deployment of the approach

Include lack of evidence of deployment to a specific work unit or employee or customer group

Whether you think there is evidence of Organizational Learning,

Include evidence of evaluation and improvement cycles, best-practice sharing, benchmarking, and innovation.

Whether you think there is evidence that the approach is Aligned and Integrated with the organization’s needs

Consider and include, if appropriate, evidence of alignment with strategic challenges, objectives, action plans; mission, vision, and values; key processes and measures; key customer/market segments and requirements; and employee groups and requirements.

What key Criteria requirements or key factors were not addressed and why it matters to the applicant

What Overall Item observations you have noted, if any.

Observations should not…

Be feedback-ready comments Consist of check marks, question marks, one-word statements

such as, “systematic,” “deployed,” or “mature” Include applicant name

Observation boxes may…

Be left blank if there is no evidence to record Include a + or – with the observation to help other Examiners

understand how you view the reported evidence Be bolded if a particular observation is very significant to your

assessment of the process

9

RESULTS ITEM EVALUATION FACTORS What are the evaluation factors?

Feedback to Award applicants and the scoring of their responses to Criteria Items are based on two evaluation dimensions: (1) Process and (2) Results. In responding to Criteria requirements, applicants need to furnish information relating to these dimensions. The specific factors for Results—Levels, Trends, Comparisons, and Linkage—are described below.

LevelsDefinition “Levels” refers to numerical information that places or positions an

organization’s results and performance on a meaningful measurement scale. Performance levels permit evaluation relative to past performance, projections, goals, and appropriate comparisons.

Features Meaningful scaleRelated to what the applicant described in the Organizational Profile and Process Items

Questions to ask in analyzing levels

Do the results provided address the Criteria requirements? Is the performance level excellent? Good? Poor? Are the results on a meaningful measurement scale?

Trends Definition “Trends” refers to numerical information that shows the direction and rate

(slope of trend data) and breadth (how widely deployed and shared) of performance improvements. A minimum of three data points generally is needed to begin to ascertain a trend. More data points are needed to define a statistically valid trend.

Features Direction and slope of results over timeNumber of data points dependent on cycle time of process being measured

Questions to ask in analyzing trends

Are results tracked over time? Are results sustained? Is there consistency of results—favorable, unfavorable, or no change? If there is no change, is this satisfactory because the organization’s results show sustained high performance?

Do you understand the trends (remember that decreasing numbers can be better and the applicant may not have indicated the direction of desired results)?

Do the results include segmentation? If results are not segmented, does the applicant explain why not? What is happening within each segment? What is happening for a selected segment throughout the Item? If segmentation varies in the figures, why? Are the key segments appropriate? Are the results segmented in a way that can help the applicant improve what it is doing?

10

ComparisonsDefinition “Comparisons” refers to establishing the value of results by their relationship

to similar or equivalent measures. Comparison can be made to results of competitors, industry averages, or best-in-class organizations. The maturity of the organization should help determine what comparisons are most relevant.

Features RelevanceConsistency

Questions to ask in analyzing comparisons

Are comparisons provided for a few, some, or many to most figures? (Applicants do not need to provide a comparison for every figure.)

Are comparisons provided for the most important results? Are the comparisons provided relevant? Are the comparisons with local organizations, industry/sector leaders, or

best-in-class organizations?

LinkageDefinition “Linkage” refers to a connection to important customer, product and service,

market, process, and action plan performance requirements identified in the Organizational Profile and in Process Items.

Features Relate to key requirements as described by the applicant Listed in Item notes

Questions to ask in analyzing linkage

Depending on the particular Results Item requirements and what the applicant stated was important, what would you expect to see (e.g., key measures, linkage to key segments, linkage to key processes, linkage to action plans)?

Do the results link to responses in Items or Areas to Address cited in the Item Notes?

How do I use these evaluation factors?

To ensure a more holistic evaluation of Results Items (Category 7), answer the questions before finalizing your comments. You should be able to strengthen and clarify comments using insights gained from answering the questions. Also determine how much information is missing. Is there a pattern in these “gaps,” or is one key segment or area generally missing? Provide details related to gaps and evidence, including exact numbers and details from figures, to support your comments.

11

Attributes of Meaningful, Well-Written Results ObservationsResults observations should tell other Examiners…

How you grouped various Key Results, for example Results related to specific Criteria requirements Results based on similar performance

Whether you think the Levels and Trends shown represent favorable or unfavorable performance and why

Include enough detail to ensure other examiners will know which results you are referring to

Include a summary assessment of the levels and trends without retyping what the applicant has already provided

Whether you think the Comparisons provided are appropriate and represent favorable or unfavorable performance and why

If you have grouped results, be sure it is clear which comparisons belong to which results by use of figure reference or other means

Whether you think the results demonstrate appropriate Alignment and Linkage to the applicant’s key factors and key process requirements

Consider the applicant’s stakeholder groups, employee types, organizational demographics

Consider the applicant’s strategic challenges, advantages, and objectives

Consider process requirements learned in your assessment of the Process Items

What Missing Results you noted and why it matters to the applicant, including expected results based on Criteria, Process, or other Stakeholder requirements

What Overall Item observations you have noted, if any. Be sure to consider results imbedded in the text as well as the

charts and graphs provided

Observations should not…

Be feedback-ready comments Consist of check marks, question marks, or one-word statements

such as, “Weak,” “Strong,” or “Average” Include applicant name Reiterate all or even most of the results presented by the applicant

Observation boxes may…

Be left blank if there is no evidence to record Include a + or – with the observation to help other Examiners

understand how you view the reported evidence Be bolded if a particular observation is very significant to your

assessment of the results

12

Independent Review Worksheet Annotated ExampleItem 3.1: 4-6 Most significant key factors (KFs) for this Item4. Paid employees and volunteers acquire, warehouse, transport, and distribute food to the food-insecure through partnerships with 58 member agencies.18. Serves 6-county metropolitan statistical area (MSA) population of 500,000 with 75,000 client contacts annually, 8% of population <5 years old (Mostly urban), 13% >65 years old (mostly rural), 10% of people below federal poverty rate, 13% unemployment rate, 1/3 of clients receive federal food assistance services, 9.5% food insecurity rate (above state average) in 2 counties27. Key changes - shifting groups of temporarily food-insecure persons, seasonal needs (school, migrant), Hispanic and Hmong populations doubled in past year. Nutritional quality of food, along with quantity, has become a key consideration in past decade.

30. Strategic challenges: Ensure food reaches those most in need when they need it most; Respond to member agency needs17. Four major customer/market segments: Customers: member agencies. Stakeholders: county, city, state, federal governments; FBA, FBLC,

IFBA; regional network of food relief agencies; taxpayers. Requirements: accountability; cost efficiency; administrative cost reduction; dependability; flexibility

Community Segments: County residents; community groups; community leaders; Chamber of Commerce; businesses; education entities; volunteers.

Donors/Suppliers: charitable foundations; corporations/grocers/restaurants; individuals; TEFAP; corporate contributors.

Item ref. Key process, subprocess, or methods used to address Item requirements

Relevant KF

Evidence of systematic approach

Evidence of extent of deployment

Evidence of systematic improvements

Evidence of alignment and integration

a(1) (how market/customer segments are determined)Segmentation Process

4, 18, 17, 30

+ four key steps outlined on pg. 12; based on information from listening and learning process-does not address differing requirements of various and changing food insecure groups (e.g., seasonal, Hispanic, Hmong)

+ fully deployed through SPP—BOD, senior leaders, employees, volunteers; covers all key services

+ PDCA process drives improvements

+ part of SPP process-/+ only biennial so may not be as responsive as needed for shifting groups

13

Good example: Note that in the key process cell, the specific a1 requirement is listed to help clarify the Criteria requirement being addressed in the observation.

In this case, the applicant has a clearly described process - Segmentation Process - that responds to this area, so it is listed as well

a(2-4) (how use voice of the customer to determine requirements, needs, expectations; how use voice of customer to better satisfy and innovate; keep current)Critical to Quality (CTQ)Determination Process

17, 18, 27, 30

+ five step, closed-loop process + listening methods vary for each customer group- While 5th step of CTQ process is to evaluate and improve, applicant does not address use of data to make improvements, stay current, better satisfy needs, or innovate

+ deployed unclear + data are systematically fed into SPP process- unclear how sharing requirements builds loyalty and referrals

a(2) (how listening methods vary for different market/customer segments) Communication Methods, 5th step of CTQ Process(Figure 3.1-2)

17, 18, 27

+ a collection of communication methods are used at regular intervals; and are tailored to each customer/market segment

+ processes appear to be fully deployed across market/customer segments

- unclear how communication methods are systematically improved,though a short Web-based survey is being developed; could this population access it?

-not described if measures and listening approaches are integrated into the measurement system described in 4.1a(1)

14

Note that in this case, the key process is responsive to a collection of the multiple requirements.

Poor example: No detail provided regarding to which groups or units the process is deployed.

Poor example: The observation “unclear” does not provide enough content to communicate a complete thought to another Examiner during consensus.

This piece of the larger CTQ process is significant enough that it is analyzed separately

Good examples: In these observations, specific concerns are noted that relate to either a key factor or an expectation based on the integrated Baldrige Performance Excellence Framework

Were any key Criteria requirements or relevant key factors not addressed? Item ref. Requirement or key factor not addressed Why is this gap significant for the applicant?

a(1) Customers of competitors were not addressed in market determination

Applicant is competing for donations and volunteers: this information might be helpful to ensure efforts keep competitive.

Innovation, keeping current Without ongoing improvement to voice of the customer approaches, the methods used may become ineffective over time as the environment changes

Use of extensive voice of the customer information not described

Limits opportunities for customer satisfaction, new business, innovation, improvement

Optional: Overall Item Notes This is a place where you can capture a critical thought that relates to the assessment of the Item. It is NOT intended to be used to capture the full set of comments for the Item or key themes. You will prepare your comments for this Item for the first time during the consensus review—not during the independent review.Lack of actual use of extensive voice of the customer data and analysesLittle evidence of improvement cyclesMay be building a theme around lack of innovation

15

Good examples: Gaps and notes on importance are helpful in scoring the Item, and in understanding shared viewpoints and developing comments during consensus.

Good example: Noted here are holistic observations that may help the Examiner recall important issues to review in other Items, or to consider in drafting key themes.

Item 3.1 ScoringFactor 0-5% 10-25% 30-45% 50-65% 70-85% 90-100%

Approach No systematic approach to Item requirements is evident; information is anecdotal

The beginning of a systematic approach to the basic requirements of the Item, is evident

An effective, systematic approach, responsive to the basic requirements of the Item, is evident

An effective, systematic approach , responsive to the overall requirements of the Item, is evident

An effective, systematic approach, responsive to the multiple requirements of the Item, is evident

An effective, systematic approach, fully responsive to the multiple requirements of the Item, is evident

x

Deployment Little or no deployment of any systematic approach is evident

The approach is in the early stages of deployment in most areas or work units, inhibiting progress in achieving the basic requirements of the Item

The approach is deployed, although some areas or work units are in the early stages of deployment

The approach is well deployed, although deployment may vary in some areas or work units

The approach is well deployed with no significant gaps

The approach is fully deployed without significant weaknesses or gaps in any areas or work units

x

Learning An improvement orientation is not evident; improvement is achieved through reacting to problems

Early stages of a transition from reacting to problems to a general improvement orientation are evident

The beginning of a systematic approach to evaluation and improvement of key processes is evident

A fact-based, systematic evaluation and improvement process and some organizational learning are in place for improving the efficiency and effectiveness of key processes

Fact-based, systematic evaluation and improvement and organizational learning are key management tools; there is clear evidence of refinement and innovation as a result of organizational-level analysis and sharing

Fact-based, systematic evaluation and improvement and organizational learning are key organization-wide tools; refinement and innovation, backed by analysis and sharing, are evident throughout the organization

x

Integration No organizational alignment is evident; individual areas or work units operate independently

The approach is aligned with other areas or work units largely through joint problem solving

The approach is in early stages of alignment with your basic organizational needs identified in response to the Organizational Profile and other Process Items

The approach is aligned with your organizational needs identified in response to the Organizational Profile and other Process Items

The approach is integrated with your organizational needs identified in response to the Organizational Profile and other Process Items

The approach is well integrated with your organizational needs identified in response to the Organizational Profile and other Process Items

xGuidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most descriptive of the organization’s achievement level for the Item.

Item 3.1—Overall Score

0–5%10–25%

x 30–45%50–65% Item 3.1 Score 40 %70–85%90–100%

16

Note that the score is achieved through an iterative process: first to determine a score for each evaluation factor, then to determine a holistic range score, then finally to determine a percent score that is a multiple of 5.

Independent Review Worksheet Annotated ExampleItem 7.2:

4-6 Most significant key factors (KFs) for this Item4. Paid employees and volunteers acquire, warehouse, transport, and distribute food to the food-insecure through partnerships with 58 member agencies.17.Four major customer/market segments:

Customers: member agencies. Requirements: timeliness; quality/variety/quantity of food; competency/consistency; access to nutritional food; continuity of service

Stakeholders: county, city, state, federal governments; FBA, FBLC, IFBA; regional network of food relief agencies; taxpayers. Requirements: accountability; cost efficiency; administrative cost reduction; dependability; flexibility and attachments A, B, and D.

27. Key changes - shifting groups of temporarily food-insecure persons, seasonal needs (school, migrant), Hispanic and Hmong populations doubled in past year. Nutritional quality of food, along with quantity, has become a key consideration in past decade.28. Comparative and competitive data sources - FBA national comparisons, FBA state comparisons, Assistance Now Finder, collaborative FBLC regional data and best practice sharing.30. Strategic challenges: Ensure food reaches those most in need when they need it most; Respond to member agency needs

Item ref. Key results—include figure references(group the results as appropriate)

Relevant KF

Level and trends (timeframe, direction, rate of change)

Comparisons used and appropriateness

Linkage to key segments/groups, process requirements, action plan requirements

7.2a1 Customer satisfaction:w/timeliness (7.2-1)w/communications (7.2-2)w/hours of operation (7.2-3)w/food selections and quality (7.2-4)overall of member agencies (7.2-5)Capital and Planned gift donor satisfaction (7.2-9)satisfaction of communities served (7.2-11)

4, 27, 28, 30, 17

+Satisfaction with timeliness/dependability and communication improving several years with one group reaching IFBA best +Generally good results for others for 3 years

+Appropriate regional comparisons+Capital and Planned gift donor satisfaction (7.2-9) are approaching IFBA and FBLC best performance.

+Segmented by member agency-not clear how these results are linked to customer requirements outlined in P1-4-performance of remaining member agencies is not clear

17

Good example: this key factor was focused for the Item, removing community segments and donors/suppliers.

Good examples: similar performance grouped.

Good example: strong performance tracks to a specific figure. Good examples: cites the

results that should be tracked but are not provided.

Good example: similar results grouped in one box, which may be most appropriate.

7.2a1 Complaints (7.2-6) 17, 27, 28, 30

+Reversed unfavorable trend for availability of food items stocked in 2006; number of complaints generally declining despite doubling of demand in past year-Variable levels and no clear improvement for hours of access, location in 4 years

+Internal comparison of rates of various types of complaints

+Linkage to key customer requirements-No segmentation provided

7.2a2 Customer loyalty, retention, referral, relationships:Likelihood to refer (7.2-7)

27, 28, 17

+Likelihood to refer increasing for 3 groups presented for 4 years

-No external comparison either within or outside applicant’s industry

-Decrease projected in satisfaction among two of the three member agencies reported

Were any key requirements, relevant key factors, or key metrics identified by the applicant not addressed? Item ref. Requirement, key factor, or key metric not addressed Why is this gap significant for the applicant?

7.2a1 Recognition of efforts is an expectation of 3 of 4 key customers, stakeholders, and market segments (P.1-4), but a measure of satisfaction with recognition is missing from strategic objectives and results.

These are key expectations that are not being tracked

7.2aResults were not provided for measures indicated in Figure 2.2-2 such as Number of Media Messages or Number of Corporate Contributors, or those shown in Figure 3.1-1 such as Comment/Assessment Cards, hotline suggestions, number of grants, or representation within community based organizations.

Additional measures of satisfaction could provide validation of assumptions about areas of importance or areas for improvement.

18

Good examples: specific information about what is missing and why it is important.

Good example: clearly cites which results are being referenced.

Poor example: should cite examples of the segmented results that you expected to see, based on what the applicant stated in the Organizational Profile or the process Item description.

7.2a Some relevant comparisons missing for key indicators such as Overall Number and Type of Complaints Received (Figure 7.2-1); Likelihood to Refer Others to [the Organization] (Figure 7.2-7); Ways [the Organization] Provides Information to the Food-Insecure (Figure 7.2-8); Satisfaction of Communities Served (Figure 7.2-11)

Applicant is a member of FBA, FBLC, and IFBA so should have access to comparative data; in addition, demand for services has doubled in the last three years so should be helpful to better understand relative performance or identify areas of excellence

Optional: Overall Item NotesThis is a place where you can capture a critical thought that relates to the assessment of the Item. It is NOT intended to be used to capture the full set of comments for the Item or key themes. You will prepare your comments for this Item for the first time during the consensus review—not during the independent review.Don’t forget the small organization considerations.

19

7.2 ScoringGuidelines 0-5% 10-25% 30-45% 50-65% 70-85% 90-100%

First Element(Levels/trends)

There are no organizational performance results or poor results in areas reported

A few organizational performance results are reported; there are some improvements and/or early good performance levels in a few areas

Improvements and/ or good performance levels are reported in many areas addressed in the Item requirements

Improvement trends and/or good performance levels are reported for most areas addressed in the Item requirements

Current performance levels are good to excellent in most areas of importance to the Item requirements

Current performance levels are excellent in most areas of importance to the Item requirements

x

Second Element(Trends/sustained results)

Trend data are not reported or show mainly adverse trends

Little or no trend data are reported, or many of the trends shown are adverse

Early stages of developing trends are evident

No pattern of adverse trends and no poor performance levels are evident in areas of importance to your organization’s key mission or organizational requirements

Most improvement trends and/or current performance levels have been sustained over time

Excellent improvement trends and/or consistently excellent performance levels are reported in most areas

x

Third Element(Comparisons/ benchmarks)

Comparative information is not reported

Little or no comparative information is reported

Early stages of obtaining comparative information are evident

Some trends and/or current performance levels—evaluated against relevant comparisons and/or benchmarks—show areas of good to very good relative performance

Many to most reported trends and/or current performance levels—evaluated against relevant comparisons and/or benchmarks—show areas of leadership and very good relative performance

Evidence of industry and benchmark leadership is demonstrated in many areas

x

Fourth Element(Importance of results)

Results are not reported for any areas of importance to your organization’s key mission or organizational requirements

Results are reported for a few areas of importance to your organization’s key mission or organizational requirements

Results are reported for many areas of importance to your organization’s key mission or organizational requirements

Organizational performance results address most key customer/patient/student, market, and process requirements

Organizational performance results address most key customer/patient/student, market, process, and action plan requirements

Organizational performance results fully address key customer/patient/student, market, process, and action plan requirements

x

Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most descriptive of the organization’s achievement level for the Item.

Item 7.2—Overall Score

0–5%10–25%

x 30–45%50–65% Item 7.2 Score 45 %70–85%90–100%

20

Independent Review WorksheetItem 7.6: 4-6 Most significant key factors (KFs) for this Item1. Serves as a food bank for 58 member agencies in a four urban and two

rural county area of central Iowa.18. Eight full time and 5 part time employees operating from one

warehouse with two trucks and one van to collect food and monetary donations. In addition to 20 ‘core’ volunteers and 500 volunteers SF is a nutrition and management internship site for the Des Couers University along with fellows from local and state governments and foundations

19. Principal success factors include: ability to feed the food-insecure through member agencies; network food suppliers, tight management of overhead costs; and volunteer base.

20. Key challenges include: Ensure that food reaches those most in need, when they need it most; optimize human resources and partnerships; respond to the needs of member agencies; obtain and maintain adequate financial resources; recruit volunteers from a broad range of age segments.

21. Volunteers contributed about 28,600 hours in 2006. Creating a challenge to maximize their potential while reducing the liability of having them represent the organization.

22. Diversity of employees demands tight controls to maintain food and personal safety of employees and volunteers.

23. Governed by a 12 member Board of Directors from the communities being service. The board has five sub-committees to provide more specific oversight and operational direction. The applicant is one of the largest food banks in Iowa.

24. It handles approximately 50% of food dispersal in the state. While the 500,000 population base of the six county service has remained stable , 8% are children under 5 and 13% are over 65. Approximately 10% live below the federal poverty rate. 94% of clients are short term users of the service to supplement other food sources or bridge temporary gaps.

Item ref. Key results—include figure references(group the results as appropriate)

Relevant KF

Level and trends (timeframe, direction, rate of change)

Comparisons used and appropriateness

Linkage to key segments/groups, process requirements, action plan requirements

7.6a(1) Accomplishment of strategy and action plansFigure 7.6-1 Strategy and Action Plan Results

3,4 38% of action plans are 5% or more below expectations. Information reported only for current year – no trend

No comparison but action plan accomplishment is not particularly appropriate for comparison

No strategic objectives or action plans directly related to the principle success factor of tight management of overhead.

7.6a(1) Figure 7.6-2 Significant Accomplishments

3,4 Consistent trend from 1997 to 2005 of actions to reach strategic objectives

Comparison not relevant

7.6a(1)7.6a(2)

Figure 7.6-3 Recent Increases in Funding; Figure 7.3-9 Share Food Income; Figure 7.6-7 Increasing Number of Grants and Funding;

3,4,8 Increasing trends each year since 2003 at approximately 5% (goal) for the past two years Public support $s

Although none provided, it might be possible to compare % increase in funding with food banks of

Linked to action plan

21

Figure 7.6-9 Corporate Contributors’ Donations to SF.

have grown from 03 through 06 and are projected to grow for 07. Overall funding has increased the past three years from $6mm in 05 to $6.4 in 06 and a projected $7 mm in 07. Aggregate value of grants has increased from $17K in 04 to $32K in 06 while donations from corporate contributors has declined $7K over the same period.

comparable size.

7.6a(1) Figure 7.6-4 Number of Ethics Hotline Calls and Outcomes

3,4,5 2 in 04 and 1 in 05. Zero for other years from 02 through 06. Low numbers for the 500 plus volunteers when reducing liability is a key challenge suggests that this measure may not be capturing ethical concerns.

Not provided Linked to action plan, key challenge and principal success factor

7.6a(2)Ethical behavior, stakeholder trust and organizational governanceFigure 7.6-5 Volunteers Completing Ethics/Values Self-Assessments

2,6,8 In the five years reported 02-06 there has been an increasing trend in completed self-assessments to a total of approximately 430 of 500 volunteers. The number has been short of each year’s goal

Not provided

Figure 7.6-6 Standards and Regulatory Agency Requirements

2,6,8 Zero findings or passed scores with the exception of USDA, FBA, FBLC, IFBA (Within 5%), and OSHA at .08% lost time injuries.No trend reported however numbers suggest stability

Figure 7.4-14 Sanitation and Food Handling and Disposal show 2006 performance just under FBLC and FBA best. Figure 7.4-15 Compliance Rating for Safety, Certifications, and Records

Compliance with FBA model protocols and standards

22

Maintenance show performance at 3 – 10% under FBLC and FBA best for 2006. No comparisons are provided for the other regulatory compliance measures although at zero findings comparisons are not significant.

7.6a(3) Fiscal accountabilityFigure 7.6-8 Performance to BudgetFigure 7.6-9 Corporate Contributions to Share Food

3,7,8 A slightly improving trend at a 06 number of 99.85% for performance to budget. A slightly negative trend in the Assistance Now ratings.

Performance to budget has been approximately 1% better than the 98.62 federal government average for the period 04-06.

Linked to action plan, key challenge and principal success factor

7.6a(4) Regulatory and legal complianceFigure 7.6-6 Standards and Regulatory Agency RequirementsFigure 7.6-10 Internal Audit ProgramFigure 7.6-11 Volunteer Audit ProgramFigure 7.6-12 External Audit Program

3,4,6,7 Internal and Volunteer program performance reflect a proactive attempt to identify and correct potential issues. The impact of these issues is reflected in the absence of external findings for the period 03-06 and the identification of only one discrepancy for each year 05-06.

Not provided although low numbers suggest potential good performance against benchmarks

7.6a(5) Organizational citizenshipFigure 7.6-13 Employee Participation in Community Ways of Connection Efforts

1,8 Percentage of employees contributing hours to Ways of connection has increased to 100% from approximately 85% in 04. Contributions by the board, and volunteers were not reported. Although since these people are already donating time to SF it is not clear how contributions to other agencies would fit

At 100% participation by employees comparative data may not be needed.

23

SF’s goals.Figure 7.6-14 Recycled Paper and PalletsFigure 7.5-5 Percentage of Food Products Requiring Repackaging and Percentage of Food Product WasteFigure 7.5-6 Average Percentage of Packaging Waste

1.8 Percentage of packaging waste has decline from approximately 17% in 04 to approximately 7% in 06. The product waste from repackaging has shown an even more aggressive decline from approx 25% in 04 to approx 8% in 06 This appears to parallel the increase in pounds of paper recycled from approximately 3K in 02 to approximately 11K in 06.

Not provided. Product requiring repackaging and product waste during repackaging are approximately 14% and 6% over the FBA best for 06.

Contribute to managing overhead costs

Figure 7.6-15 FEED Iowa Partnership Proposals

1,8 A positive trend in proposals from 1 in 04 to 4 and an IFBA best in 06.

IFBA Aligned with hunger free Iowa vision.

Were any key requirements, relevant key factors, or key metrics identified by the applicant not addressed? Item ref. Requirement, key factor, or key metric not addressed Why is this gap significant for the applicant?

Optional: Overall Item NotesThis is a place where you can capture a critical thought that relates to the assessment of the Item. It is NOT intended to be used to capture the full set of comments for the Item or key themes. You will prepare your comments for this Item for the first time during the consensus review—not during the independent review.

24

7.6 ScoringGuidelines 0-5% 10-25% 30-45% 50-65% 70-85% 90-100%

First Element(Levels/trends)

There are no organizational performance results or poor results in areas reported

A few organizational performance results are reported; there are some improvements and/or early good performance levels in a few areas

Improvements and/ or good performance levels are reported in many areas addressed in the Item requirements

Improvement trends and/or good performance levels are reported for most areas addressed in the Item requirements

Current performance levels are good to excellent in most areas of importance to the Item requirements

Current performance levels are excellent in most areas of importance to the Item requirements

X

Second Element(Trends/sustained results)

Trend data are not reported or show mainly adverse trends

Little or no trend data are reported, or many of the trends shown are adverse

Early stages of developing trends are evident

No pattern of adverse trends and no poor performance levels are evident in areas of importance to your organization’s key mission or organizational requirements

Most improvement trends and/or current performance levels have been sustained over time

Excellent improvement trends and/or consistently excellent performance levels are reported in most areas

X

Third Element(Comparisons/ benchmarks)

Comparative information is not reported

Little or no comparative information is reported

Early stages of obtaining comparative information are evident

Some trends and/or current performance levels—evaluated against relevant comparisons and/or benchmarks—show areas of good to very good relative performance

Many to most reported trends and/or current performance levels—evaluated against relevant comparisons and/or benchmarks—show areas of leadership and very good relative performance

Evidence of industry and benchmark leadership is demonstrated in many areas

X

Fourth Element(Importance of results)

Results are not reported for any areas of importance to your organization’s key mission or organizational requirements

Results are reported for a few areas of importance to your organization’s key mission or organizational requirements

Results are reported for many areas of importance to your organization’s key mission or organizational requirements

Organizational performance results address most key customer/patient/student, market, and process requirements

Organizational performance results address most key customer/patient/student, market, process, and action plan requirements

Organizational performance results fully address key customer/patient/student, market, process, and action plan requirements

X

Guidance: The overall score is not intended to be a numerical average of the elements above; you should select the range and score that are most descriptive of the organization’s achievement level for the Item.

Item 7.6—Overall Score

0–5%10–25%30–45%

X 50–65% Item 7.6 Score 50 %70–85%

90–100%

25

Results MatrixItem ______

Expected results based on Organizational Profile and Process Items:

Performance levels or Trends Comparisons Linkage: Results address areas of importance

Gaps

Area or sub-area to Address

Title of Measure and figure number

Timeframe(from-to)

Level or Rate (from # to #)

Direction + favorable = no change- unfavorable

-None (N)-Average (A)-Industry Best (I)-Key Competitors (K)-Best-in-class/best-in industry (B)

-Performance vs. competitors-Lagging (Lg)-Good/Very Good (G)-Strong (S)-Leader (L)

Measure or indicator links to:- key segments (student/ customer/stakeholder groups, employee/staff groups, locations),

- process rqrmnts

- action plan rqrmnts

Key requirements not addressed.

Missing measures and indicators (discussed in Categories 1-6, measures tied to key factors, strategic objectives, action plans, key customers, markets, processes, etc.):

26