Ten Steps to Building a Learning Evaluation

Strategy

Godfrey Parkin and Sarah WardTuesday, April 11th, 2006 – SA Ballroom BISPI 44th Annual International Performance

Improvement Conference, Dallas, TX

Can One Document Drastically Improve Evaluation?

Radically

How Can You?

• Focus your overall evaluation program for consistent, high-impact results?

• Bring all your disparate elements into alignment with a single, powerful document?

• Get your policies, procedures, plans, instruments, and reporting tools into perfect harmony?

• Uncover and communicate your true value to the business—and build it further?

The Key:

Learning Evaluation Strategy

(LES)

Plot

1. Premise: A LES will Radically Improve Eval.2. Fight Back Against Futility3. Back to Our Roots: Our Purpose?4. What a LES is and Should Do5. The 10 Steps6. Start with a Strategy Statement7. Case Studies8. Obstacles and Overcoming Them9. Vision for Next 2-3 Years 10. Conclusion: LES is More!

Plot

1. Our Premise:

A LES will Radically

Improve Evaluation

Five Frank Questions About Your Evaluation Program

1. Are you still using a 10-question smile sheet? Yes No

2. Does your evaluation stop at Level 2? Yes No

3. Does most of your performance data disappear into a black hole?

Yes No

4. Does trying to calculate ROI give you sleepless nights?

Yes No

5. Does your evaluation output fail to have high-level impact?

Yes No

Plot

2. Fight Back Against Futility

If you can’t measure it, it doesn’t exist.

Persistently Annoying Business Maxim

Most Training is Futile because It is Not Evaluated

Most Evaluation is Futile because it is Not Well Done

If you’re properly aligned to the business needs and the organisation recognises the

value of the training and development there shouldn’t be any need to be obsessed

with the figures after the event.

Martyn Sloman, Adviser Learning, Training and Development, CIPD, U.K.

(Chartered Institute of Personnel & Development)

Plot

3. Back to Our Roots:

What is Our Purpose?

What is the Purpose of Training?

Performance Improvement

What is the Purpose of Evaluation?

To Measure Changes

in Performance

Plot

4. What a LES is and Should Do

• Elevate the Status of Learning in the Organization

• Improve Training Operations• Improve Evaluation Efforts• Improve Organizational Performance

What is a LES?

It provides you with the context and overarching approach

within which evaluation should take place.

A LES Creates the Structure for High-yield, Low-cost, Meaningful Evaluation

What Should a LES Do?

• Set out organizational context, high-level goals, & overall evaluation approaches

• Tie these to organization’s strategy & goals

• Guide decisions about what data to collect, how to interpret it, how to act on the findings

• Form the foundation of policy

Plot

5. The 10 Steps to Building a LES

MAJOR PHASESPrepare Team / Collect InputsResearch & AnalysisCreate the Strategy & DocsImplement Your Vision

The Elements of a LES

• Strategy – describes the overarching structure, goals, and context

• Policy – sets the standards & approaches

• Plan – lays out the implementation including roadmap and schedule

• Procedures – specify step-by-step evaluation practices

Planslay out

implementation

Creates the Structure for High-yield, Low-cost,

Meaningful Evaluation

Learning Evaluation Strategy

(LES)

Proceduresspecify how to

evaluate step-by-step

Policysets standards

and approaches

Collects/ReportsHow is data obtained,

interpreted, and communicated?

Clarifies and Routinizes the Evaluation Process

Learning Evaluation Strategy

(LES), cont’d.

Delivers ValueHow are outcomes understood, valued

and acted upon?

Targets DataWhat information is needed for decision

making?

Input to a LES

Learning Evaluation Strategy

Organizational Mission, Vision

and Goals

Legal and Regulatory

Requirements

Employee Performance

Factors

Customer Satisfaction

Factors

Training Department

Mission, Vision and Goals

Developers of a LES

The LES Team

Core Project Team

Evaluation Project ManagerConsultants/SMEsISD Manager (or Representative)Key Line Managers

ChampionsHead of TrainingHis or Her Boss (Preferably a Senior VP, VP or CLO)

AdvisorsHuman Resources, IT,Market Research, Legal

The 10 Steps to Building a Learning Evaluation Strategy

Step 1Know Your Limits

Your first evaluation strategy is often the hardest to create: ask for help.

Step 2Convene a Task Force

Your team should include stakeholders that need to give buy-in.

Step 3Look Around for Best Practices

Look at other organizations’ evaluation strategy documents, and interview them to identify best practices and pitfalls.

Step 4Map the Process

Have your team identify, map out, and critique the current processes that learning evaluation uses.

Step 5Conduct a SWOT Analysis

Specify strategic priorities, goals, potential obstacles, and immovable constraints

The 10 Steps to a LES, cont’d.

Step 6Develop Your Strategy

Broadly identify alternative strategic approaches.

Step 7Document Your Strategy

Cover environment scan; values; vision & mission for the evaluation service; priorities; goals; & evaluation strategies.

Step 8Gain Stakeholder Buy-in

Review the draft LES document with all stakeholder groups, solicit their comments, and then make revisions.

Step 9Policy, Plan, and Procedures

Develop the policies, plan (schedule for implementation), and optimal procedures and instruments to carry them out.

Step 10 Roll-Out

If a radical departure, also develop a phase-in plan over several years. Prepare a pitch that “sells”. Include means of measuring your success.

Plot

6. Start with a Strategy Statement

Every LES Starts with a…

• Strategy Statement– Overarching description of your

evaluation initiative, its goals, and the approach for attaining those goals.

1. Overarching Concerns/Issues

2. Goals

3. Approach

LES Sample Statements #1

• Overarching Concern– We are underrated in our ability to prepare

our employees and command an increasing share of our market. Currently evaluation is neither strong nor comprehensive and needs to be implemented wherever training resources are invested.

LES Sample Statements #1

• Goals (one)– To support changes in performance

stipulated by the business and departmental goals laid out in the mission and vision statements.

LES Sample Statements #1

• Approach (one)– Utilize existing sales and marketing data

for evidence of training achievement supplemented by sample-based surveys derived from customer interviews and employee satisfaction survey.

LES Sample Statements #2

• Overarching Concern– We face increasing regulatory scrutiny

and need to ensure that our training addresses government-mandated competencies and repels the frequency and consequences of lawsuits.

LES Sample Statements #2

• Goals (one)– Be able to measure and report on

compliance with regulatory performance requirements.

LES Sample Statements #2

• Approach (one)– Tie government-mandated

competencies to curricula and measurements of performance compliance.

LES Strategy Worksheet

1. Identify 1 Element or Issue

2. Describe the Overarching Concern

3. State the Goals

4. Specify the Approach

5. Work Together into a Paragraph Statement

Plot

7. Case Studies

Case Studies

Sample Evaluation Strategy for a Division of American ExpressCurrently there is no consolidated training

evaluation practice or system at all within….

Sample Evaluation Strategy for Your OrganizationCurrently training evaluation is ….

Plot

8. Obstacles and Overcoming Them

ObstaclesPersistent Concerns About Conducting Evaluation

1. No Payoff There is no payoff; nobody cares

2. No Support Management doesn’t support evaluation activities

3. No Know-howWe don’t know how to do it; our staff has limited

skills

4. No TimeEvaluation projects are chaotic, ad hoc “collateral

duties” that take too much time

5. No SystemOur courses are not developed using good

instructional design, so they are hard to evaluate

6. No Use Nobody uses evaluation results

7. No Respect We might look bad and lose credibility

Obstacles & Overcoming Them

• Selling the LES– Politics are Unavoidable– Good Change Agentry Principles– Thorough Stakeholder Identification– Forge Relationships at Multiple Levels– Explain & Publicize the Benefits

Obstacles & Overcoming Them

• Selling the LES, cont’d.– Learn About & Counter Perceived Roadblocks– Establish a Program Champion– Understand How Integration will Occur– Explain the Implementation Plan– Articulate the “WIIFM” at All Levels

Plot

9. Vision for Next 2-3 Years

Vision for 2-3 Years Down the Line2005 2006 2007 2008 and Beyond

L1 – Begin conducting for 100% of all courses and learners from 10/31/05

L2 – Begin conducting continuously on a course by course basis as each course is completed by outside vendor and hosted on the LMS

L2 – Be evaluating 100% of courses by 12/31/06

L3 Formative – Start performing formative evaluation on selected skill building courses (chosen by risk, scale and cost) starting in 4Q05

L3 Formative – Conduct on all skill building courses and curricula as they are being developed (during pilots)* or substantially revised (using same criteria)*

Conduct as courses and curricula are developed or revised

L3 Summative – Begin conducting across all competencies using BOS and Mystery Shopping beginning in 3Q06 or 1 year following deployment of training re-organization

If not done in 3Q06, conduct BOS and Mystery Shopping twice in the first full year following deployment of the training re-organization; otherwise conduct once ~6 mos. later

Conduct BOS and Mystery Shopping once per year thereafter, as revisions take place, or as critical issues arise

The key decision factors are: risk from course failure, scale of deployment, and the cost of development and operation.* If alpha and beta pilots are not conducted, then the first 5% of participants will be used for formative evaluation purposes.

Plot

10. Conclusion:

LES is More!

Conclusion: LES is More!

• Elevates the Status of Learning in the Organization

• Improves Your Training Operation

• Improves Your Evaluation Efforts

• Improves Your Organizational Performance

Elevates the Status of Learning in the Organization

1. Supports Management Decision-making with Demonstrable, Undeniable and Pervasive Evidence of Performance

2. Makes the Case for Dramatically Improving Learning Effectiveness

3. Links Measures Learning Outcomes to Business Goals

4. Clarifies the Value of Your Unit

Improves Your Training Operation

1. Positions Instructional Designers and Their Contributions as Key

2. Makes Instructional Designers’ Jobs Easier by Making Clear What will Be Evaluation, How, and When

Improves Your Evaluation Efforts

1. Standardizes Evaluation by Creating a Strategy Statement, Policies, Procedures, Schedule, Plans, and Job Aids

2. Prescribes Simple, Straightforward Practices That Can Be Conducted by Modestly-trained Evaluators

Improves Your Evaluation Efforts

3. Ensures That Only Necessary Data are Collected or Mined in the Most-Appropriate, Least Intrusive and Most Cost-Effective and Efficient Manner

4. Reports Outcomes and Recommendations to Appropriate Decision-makers in Formats and Language Suited to Their Needs

Improves Your Organizational Performance1. Positions Evaluation as an Essential

Management Tool

2. Sets Up an Accountability Process for Acting on LES Recommendations

3. Provides a Mechanism for Continuous Improvement and Evaluation of the LES

4. Establishes a Process for Ongoing Quality Improvement of Organizational Performance in Areas Impacted by Training

LES is More!• Evaluation can be Trapped in a Low-Level,

Mimimalist Regimen– Requires and delivers little– Burns resources and garners ill-will

• Should be Structured & Empowered– Prevent Waste of Significant Amounts of $

• Poor Evaluation Risks– Handicapping Performance– Eroding Competitiveness– Putting Learning Investments at Risk

You Can Elevate Evaluation to a Profoundly Useful, Important Tool with a Learning Evaluation

Strategy (LES)

LES is More!

LES is More!

Watch for Our Article in PIQ

Download Slides from Our Websitewww.alter-inc.com (under Evaluation)

Thanks to Karen Medsker!!!