ibm … creating the future of learning ibm learning solutions © 2004 ibm corporation learning...

IBM … creating the future of learning

IBM Learning Solutions © 2004 IBM Corporation

Learning Effectiveness Measurements

Lauge SorensenFaye WanklingJune 17, 2004


IBM Learning Solutions © 2004 IBM Corporation2

Agenda

IBM Learning Learning Life Cycle Performance Value Chain Relationship Discussion

How do you make the linkage between learning and business results?

What gets in the way of doing effective business impact measurements?

How do you use Learning Effectiveness Measures in your organization?

Our Continued Challenges & Lessons Learned Learning Measurements: The Positive Way Questions



IBM Learning Process

People Level Requirements: Audience segmentation: current/desired behaviorDrivers (Measurements, skills, values, tools/info)

Business Level Requirements: Business problem statement

Detail level solution design: Offerings within cluster

Deployment, Delivery and Execution

Development of Offerings within cluster

Deployment & Delivery

Requirements

Design & Development

AgreementCollaboration

AgreementCollaboration

High level solution design: Total cost of learning, cluster design

Business AnalysisBusiness Impact Objectives

Gap AnalysisRoot Cause Analysis

Solution IdentificationSolution DevelopmentTotal Cost of LearningMeasurement Design

Solution ImplementationSolution Measurement

Continuous ImprovementNew Requirements, Enhanced Design

Measurem

ents

Measurements

Measurem

ents



Learning Development Cycle & Performance Value Chain

Business Impact

Behavioral Impact

Learning Gain

Readiness

Phase 6:

MEASUREMENTS &REPORTING

Phase 5: DELIVERY

Phase 4: DEVELOPMENT

Phase 1: BUSINESS PROBLEM TARGETING BUSINESS RESULTS

Phase 2: REQUIREMENTS

Phase 3: DESIGN

EVALUATE



Performance Value Chain Relationship

Learners’ intention, readiness, expectations, career aspirations, etc.

Design DeliveryManagement

Support

LearningSatisfaction

LearningGain

LearningApplied

Job Performance

Business Impact/ROI(e.g. Revenue, Employee Satisfaction, etc.)

LearningEffectiveness

BusinessImpact

LearningSupport

Readiness &Participation

L1/2/3+

L4+

L4/5+

L1+

L1+

Business Impact

Behavioral Impact

Learning Gain

Readiness



In your organization how do you make the linkage between learning and business results?

Honestly - We don’t… “Not very well” “We just know if we do some education or training, it’s bound to have results” “We try but we haven’t really been able to connect business results to learning –

at least not ONLY to learning – so many other factors contribute”We know what we need to do – but aren’t doing it… “Behavior is often viewed as results so a deep understanding of employees

environment (processes, tools, etc.) and skills/gaps helps us get there” “We link learning with talent management (job placement, etc.)” “We must tell the full story at each level – we must know that some learning is

being applied before we can then attribute to business results” “It is challenging and just because it is difficult to measure the impact it doesn’t

mean there isn’t a real linkage to driving lasting behavior change and ultimate business results”

“Performance = Results therefore we need to pursue gaps in results; every training request should be tied to a measurable gap in results”

“We need to move up from only job level performance because seldom will this alone impact organizational results”

We try – but we are not sure what method to use….. “We use Performance Value Chains” “We use Impact Maps” “We do Causal Analysis”



What Gets in the Way of Doing Effective Business Impact Measurements

It’s complicated… “ROI seems hard/complex” “Reaction data makes us feel good (“this was a great class”) – why get more” “Data collection is difficult, data is inconsistent, systems don’t support data

collection in a consistent way; Internal systems don’t support” “Difficult to effectively isolate the impact variable” “We measure what is easily measured instead of what is really important to

measure”We don’t have a process… “Failure to collect baseline data so nothing to compare to” “We don’t start off solving a business problem so measuring a business result

may be impossible” “Lack of standards – each developer thinks their offering is unique” “We often fall short of defining true business impact measurements and instead

look for evidence of some application of skills – but forget to ask ourselves “so what””

“We prefer to focus on achievement and forget about the power of creating an environment of continuous improvement”



What Gets in the Way of Doing Effective Business Impact Measurements

There’s Enterprise/Organizational barriers… “There are geography infrastructure/process inconsistencies making

overall global consistency difficult” “Deployment of business impact measurements is a major

organizational change” “Privacy issues; legal inhibitors both real and imagined”Unclear priorities…. “Business Impact Measurements drive increased accountability for

Learning organizations and business stakeholders – who wants that?!”

“Pressure to show results prevents us from taking time up front and at the back end of an intervention (front end to get started, back end to implement and get on with the race)”

“Unrealistic expectations of what can or can’t be measured” “Inability to uncover the root causes of the gap in results so we don’t

have a concrete business result to measure” “Lack of commitment from learning teams – especially to take time

at the front end”



How do you use Learning Effectiveness Measures in your organization?

?????....... “Use ‘em? For what?” “To look good to our CEO” “To feel good about what we’ve delivered” “Whatever we need to provide on the company scorecard”We know what we should do…. “We demonstrate our value/ROI for every dollar we spend on learning” “To justify continued/more investment in learning” “To make learning investment decisions in terms of allocating scarce

resources (invest in high impact learning only)” “We don’t really want to measure the business impact of all initiatives but we

sure want to use those ones that are a significant investment and make sure we report to our leadership team”

“We need to ensure more emphasis on incentives, reinforcement guidelines, etc. to ensure the measures keep driving the change”

“Specific to an individual organization but with standards and an informed statistical basis”



Our Continued Challenges

Shifting the perception Earn a seat at the table when problems are discussed and strategies

developed

Move from delivering training to solving business problems

Traditional thinking – some “get it” and some don’t

Data collection; privacy issues, global infrastructure

On demand learning Work embedded, makes separation of learning from performance

impossible; how will we measure



Our Lessons Learned

Just Do It Solve business problems, find opportunities to link learning to business results and speak in

business terms

Call out business and operational barriers to performance even if you can’t solve them

Learning is never in a vacuum; establishing a correlation or contribution to a result may be the best we’ll ever do

Reduce complexity of measurements and “just do it” using practical methods

Do it Enterprise Wide Tell the whole story; Participation and Reaction measures matter; but the full picture must be

told with business results

Anticipate major resistance; focus on strategic change management acknowledging that this is a major cultural change

Establish standards and methodology and watch for data distorted to tell only a happy story

Report results in all the right places Drive shared accountability for business results; share data with business stakeholders



A Positive Approach to Learning Measurement

The Guiding Principles: Continuous improvement; Collaboration; Consistency; Confidentiality; Objective; Ease of Use; Flexibility

IBM Learning Measurement is different from traditional learning measurement, as shown below:

Traditional Learning MeasurementTraditional Learning Measurement IBM Learning MeasurementIBM Learning Measurement

Fault-finding, judgmental Business Impact-Driven

Audit/Control Continuous Improvement

Imposed Collaborative

Add-on, after the fact Established in Requirements; Built into the design

Sporadic Continuous

Subjective Objective

Difficult to understand Easy to understand

Focus on individual solution Focus on all levels



Questions?Gartner, Inc., has recognized IBM in the "Leaders" quadrant in the "2004 E-Learning Suite Magic Quadrant”, the “2004 Learning Management System Magic Quadrant” and the "2003 E-Learning Content Magic Quadrant."

IBM #1: TRAINING magazine conducted extensive research on 606 companies to determine the top providers of employee training and development. IBM was honored as #1 in the 2004 TOP 100 listing. In addition, IBM's "Basic Blue for Managers" blended program was highlighted among only 8 "Best Practices" -- formal learning programs deemed most innovative or successful.

IBM received 2003 ASTD BEST Award, and was ranked highest among information technology companies.

IBM’s Shades of Blue program Recognized by ASTD as a 2003 Excellence in Practice

Managing@IBM was awarded two 2002 ASTD Citations for Organizational Learning and Workplace Learning and Development

IBM received a 2003 Corporate University Xchange Excellence award for Leadership Development sponsored by Fortune Magazine.

IBM Lotus Learning Management System awarded a Top Training Product of 2003 award from HR Executive Magazine

IBM LearningSpace Everyplace wins 2003 Silver Award

Ibm.com Americas – “Blended e-Learning in Call Centers wins 2002 Bronze Award

IBM's Edvisor wins 2001 Gold Award

ibm … creating the future of learning ibm learning solutions © 2004 ibm corporation learning...

Documents

learning measurements

business problem statement

development phase

requirements phase

enhanced design measurements

delivery phase

measurements reporting

employee satisfaction