the evolution of learning analytics at jetblue: key ingredients for business impact

#CLOwebinar Sponsored by

The presentation will begin at the top of the hour.

A dial in number will not be provided.

Listen to today’s webinar using your computer’s speakers or headphones.

Welcome to the webinar!


Speaker: Teri Schmidt Manager-Assessment, Measurement, & Evaluation (AME) JetBlue Airways

Moderator: Sarah Sipek

Associate Editor Chief Learning Officer magazine

The Evolution of Learning Analytics at JetBlue: Key Ingredients for Business Impact


Tools You Can Use Audio Control

–  A dial in number will not be provided.

–  Adjust the volume by sliding the indicator in the Media Player box to the right.

–  Also check your computer’s volume for external speakers or headsets.


Tools You Can Use

Speaker Bios

Resources

Media Player

Q&A

Slides

Twitter

Help

Post Event Evaluation Share This

Group Chat

HRCI

CLO Webinars


Tools You Can Use Q&A and Live Group Chat

–  Enter your first name and initial and join the live discussion with other webinar attendees

–  Enter technical or content-related questions into the Q&A box

–  Type your question in the space at the bottom.

–  Click “Submit.”

Q&A

Group Chat


Tools You Can Use

Twitter -  Click “Post” in the Twitter widget.

#CLOwebinar @CLOmedia


1. May I receive a copy of the slides? YES! Click on the resource list located on the top left portion of your

screen.

2. May I review the webinar recording at a later date? YES! You may log in again using today’s link to review the presentation

on-demand.

3. Is this webinar HRCI certified? YES! The HRCI certification code will appear in the box to the right of the

slides after the required watch-time has elapsed.

Frequently Asked Questions


Sarah Sipek Associate Editor Chief Learning Officer magazine



Teri Schmidt Manager-Assessment, Measurement, & Evaluation (AME) JetBlue Airways



TERI SCHMIDT | MANAGER, AME

11

Agenda

§  Data Invasion §  In Your Organization §  In Human Resources §  About JetBlue §  Key Ingredients and JetBlue's Journey

ü Common Metrics ü Collaborative Data Reviews ü Capability Building

§  Q &A

12

The White House Names Dr. DJ Patil as the First U.S. Chief Data Scientist

Big data: The next frontier for competition

13

POLL

Select an option below to fill in the blank. §  The focus on using data for decision-making in my

organization had ________________ in the past five years. a.  Decreased significantly b.  Decreased slightly c.  Remained steady d.  Increased slightly e.  Increased significantly

15

“It’s just so important. Data today, information today, is so prevalent. How do we use it to improve the customer experience and also to improve our operational performance…”

ABOUT US

17

POLL

What is JetBlue’s vision? A. Get you where you need to be B. Give ordinary people the opportunity to fly C. Inspire humanity D. Make flying fun

SECTION BREAK

30 million+ Customers 16,000+ Crewmembers 87 cities 825 daily flights 6 Focus Cities 3 types of aircraft: - A321, A320, E190 3 Support Centers: - Long Island City - Orlando - Salt Lake City

INSPIRE HUMANITY

19

Preparing Crewmembers for exceptional career success at JetBlue

20

Learning Shared Services Admin/Logistics, AME, Learning Design, Learning Systems, Lodge, Uniforms

EVP Operations

College

of Flight

College

of Tech Ops

SVP Safety, Security, & Training

Learning Systems

College

of Inflight

College

of Sys Ops

Learning Design

College

of Airports

College

of Customer Support

Learning Support

(Admin & Logistics, Uniforms, Lodge)

Learning Analytics

& Strategic

Evaluation

Director Technical Training

Director Inflight & Sys Ops Training

Director Customer Service Training

Manager AME

AME’s Contribu/on

Assessment, Measurement, and Evaluation (AME) Supporting leaders in making evidence-based decisions

22 22

POLL

Who handles learning analytics in your organizations? A.  No one B.  Courseware developers and/or instructors C.  A dedicated team or individual D.  Someone outside of the training department E.  Other

KEY INGREDIENTS

24

Key Ingredients

Capability Building

1

2

3

Common Metrics

Collaborative Data Reviews

Capability Building

25

2005

2006

2007 2009

•  Standard Level 1 •  New technology •  Workload increases

•  Pair of Hands to Consulting •  “Franchising Model” •  Centers of Excellence •  AME •  Defined influential partners

•  MTM •  College-specific evaluation strategies •  Business Alignment Process

2010 2008

•  Data Into Action Process •  AME Roundtable •  College-specific annual reviews

2011

•  Mid-year action reviews •  Data visualization

•  Crewmember Experience & Impact Team formed •  AME

Strategy •  Executive Reporting

2012

•  Strategic Evaluation •  PMO •  Evaluate Evaluation

2013

AME History

2014

•  Learning Analytics • Strategic Evaluation & Education

1

3

2

3+

26

#1 Common Metrics

JBU Data Type Use Recommendations

Supported Decision(s)

-  Learning experience improvements (ICE) -  Targeted curriculum improvements -  Policy, process, and/or procedure changes

-  Training time re-distribution -  Assessment (e.g., test, checklist.) changes -  Revisions to objectives to better align with operation

-  Work environment changes (tools, expectations & feedback, leadership support, rewards &recognition)

-  Additions/ revisions to JBU catalog of offerings

When to Use -  Lower cost data collection is needed

-  As a data source for cause analysis into performance and result data

-  When the investment of resources required to collect, analyze, and report on the data is justified (by criticality, visibility, and resources investment)

-  Monitoring: Continuous

-  Detailed analysis: When justified (by criticality, visibility, and resources investment)

28

Data Inventory

29

Research and Analysis Plans (RAPs) Decision/Question Analysis Frequency/Date Courses Sources Ongoing Monitoring How satisfied are participants in JBU’s training programs?

JBU Customer Satisfaction Trend Analysis

Monthly All that are in MTM

MTM

What courses have the greatest opportunities for improvement? What courses are most successful? What factors contribute?

JBU Customer Satisfaction Trend Analysis and comment analysis

Monthly; Faculty Meetings; Quarterly; Annually

All that are in MTM

MTM

Context-driven analyses What impact does changing an individual assessment to Train-to-Proficiency have on crew assessment performance?

Chi-sq: pass rates pre & post TTP

Q1 2015 Inflight Qualification Training

Inflight AQP database

What are the drivers of attrition? Attrition by course, instructor, day of class; chi-sq: comparison of attrition pre/post revisions to assessments

Q1 2015 College of Airports Instructor-led training courses

PPDMS, Attrition report, Portal

30

Strategic Evaluation

31

#2: Collaborative Data Reviews

“Feedback (data) itself at best can only initiate change; it cannot bring it about. People in groups making use of feedback can create change.”

-Nadler, D. A. (1977). Feedback and organizational development: Using data-based methods. p. 144.

Participants

Department Heads

Managers

AME

PRIORITIZE COMPARE EXPLAIN ACT

Monitoring of college-specific data reports and dashboards

Quarterly College Data Meetings

Quarterly JBU Data Actioning Team

Quarterly Department Head Data Meetings JBU Annual

Review Meetings •  Managing by

exception

•  Deeper dives as necessary

•  Review of JBU metrics

•  Discussion to action critical risks and successes

JBU’s Data Into Action Process

•  Review of JBU metrics and insights gained

•  Actions generated and support needed for action items

•  AME shares insight into the past quarter’s data

•  College leadership and analysts share insights, actions taken, and potential upcoming data-influencers

33

Data Review Meeting Agenda

34

#3: Capability Building

“Organizations that want employees to be more data oriented in their thinking and decision making must train them to know when to draw on data and how to frame questions, build hypotheses, conduct experiments, and interpret results.” Marchand, Donald A., & Peppard, Joe (2013). Why IT Fumbles Analytics. Harvard Business Review. Retrieved from https://hbr.org/2013/01/why-it-fumbles-analytics

•  Identifying & involving stakeholders •  Defining expectations •  Defining roles & responsibilities •  Managing risk (including potential

biases)

Manage Stakeholder Involvement

•  Identifying research questions/decisions

•  Identifying measures •  Choosing methodologies •  Developing tools

•  Administering data collection tools for maximum participation

•  Developing themes •  Conducting effective data

review sessions •  Developing actionable data

visualizations

•  Defining the gap between current & desired results

•  Determining root causes •  Selecting appropriate

solutions •  Evaluating solutions

Plan for Data Collection

Collect Data

Analyze, Interpret, &

Communicate Data

Use Data to Improve

Performance

Intro

Continue

THANK YOU Teri Schmidt [email protected]


Please complete the webinar evaluation.


Join our next Spotlight webinar! Learning’s in BLOOM – How On-Demand

Video Learning’s Transforming Blue Cross Blue Shield of North Carolina

Tuesday, April 28, 2015

Webinars start at 2 p.m. Eastern / 11 a.m. Pacific

Register for all upcoming Talent Management Webinars at www.clomedia.com/webinars

OR click the icon on the widget bar!

the evolution of learning analytics at jetblue: key ingredients for business impact

Recruiting & HR