using evidence-based career pathways with employer-defined ... · 4 •evidence-based career...

1 • Evidence-Based Career Pathways • Kelly Wilczak • Bill Guest • Jessica White-Hatinger •

Using Evidence-Based Career Pathways

with Employer-Defined Career Portfolios

for Job Seeker Success

Steve Bennett, Humanity 2.0, Washington DC;Bill Guest, Metrics Reporting, Inc., Byron Center, MI; Kelly Wilczak, Mercy Health – West Michigan, Grand Rapids, MI; Jessica White-Hatinger, West Michigan Works, Grand Rapids, MI

Friday: 8:15-9:00 White River A and 10:30-11:15 Room 304/305


Agenda

• Kelly – Mercy Health evidence-based selection process (EBSP) outcomes

• Bill – The science behind the outcomes … why did it work?

• Jessica – Organizing community collective impact to build evidence-based career pathways that help job-seekers.


Mercy Health, a Regional Health Ministry of Trinity Health, implemented the

evidence-based selection process (EBSP).


The Mercy Health Journey:• 2010 to 2011 – Planning, best practice research, process

redesign. Job analysis started for first 5 job families.

• 2012 to 2013 – Implemented the evidence-based selection

process (EBSP) across all staff job families (ES to PHRM). Job

Analysis methods evolved from ACT method to the ONET-

Based Confirmatory Job Analysis Process. Initial results

were good. Refined the structure of job families.

• 2015 to 2016 – Increased focus on evidence-based sourcing.

Tightening job market and a desire to continue progress on

diversity led to stronger focus on sourcing. Mercy hosted

“Building Talent Bridges” annual events, launched the

Career Portfolio project, & joined Health Career Pathways.

©2016 Trinity Health - Livonia, Mich. 5

Employed Physicians 3,900

Affiliated Physicians23,900

of all babies in America are delivered at Trinity Health facilities

1.7%

*Owned, managed or in JOAs or JVs.

**Operations are organized into Regional Health Ministries ("RHMs"), each an operating division which maintains a governing body with managerial oversight subject to authorities.

***Includes multiple locations for Trinity Home Health Services, Trinity Senior Living Communities and PACE facilities.

Home Care & Hospice Locations Serving 116 Counties49 Continuing Care

Facilities61PACE Center Locations14

Our 21-State Diversified Network

Hospitals* in 21 Regional Health Ministries**90 Mission Health

Ministries4 National Health Ministries***3++


Mercy Health – West Michigan

• West Michigan

• 8200 Employees

• 5 Hospitals

• 60+ Locations and

Physician Practices


Mercy Health – Key Performance Outcomes

• Reduced first-year turnover from a baseline of 25.3% down to 18.7% (Quality-of-Hire)

• Reduced Time-to-Fill from a baseline of 37 days down to 31 days

• Increased Diversity – Non-white new hires rate increased from an 18.4% baseline in 2010 to 30.2% in 2015

• Increased Diversity – Non-white employment increased from a baseline of 13.4% to 18.5% in 2015

• Financial Impact – annual savings of $3,259,420

9 • Evidence-Based Career Pathways • Kelly Wilczak • Bill Guest • Jessica White-Hatinger •Note: The 3-County area (Kent, Muskegon, Ottawa) is 21.6% non-white, including Hispanic or Latino

West Michigan 21.6%


Diversity – Regional Trends by Job Family

Note: Diversity increased in each of the 10 major job families over the period of 2010 to 2015.


Breakdown of "Non-white" as of December 31, 2015

Non-White Quantity

American Indian 38

Asian 154

African American 539

Hispanic 323

Not Specified 443

Pacific Islander 6

Total Non-White 1503


Supporting Data – Hiring Rate and Employment

2010 BASELINE FY 2013 FY 2014 FY 2015 Dec-15

NON-WHITE 838 1102 1138 1374 1503WHITE 5402 6030 6120 6352 6618TOTAL 6240 7873 8028 8658 8121

NON-WHITE % 13.4% 14.0% 14.2% 15.9% 18.5%

1503= 1.79

8381503 - 838 = 665


Unconscious Bias

All of us have implicit biases to some degree.

Acknowledging and understanding this implicit

response and its value and role is critical to

informed decision-making and is particularly

critical to those whose decisions must embody fairness and justice.


Selection Criteria

Cognitive Elements Character Elements

• Judgment & Decision Making

• Critical Thinking

* Reading Prose (procedures)

* Reading Documents (Charts,

Graphs, Forms)

* Math (Quantitative Reasoning)

• Overall Job Fit (personality and behavior)

* Recruiter SIG - (Perception, Service

Orientation, Active Learning, and

Office Administration)

* Manager SIG - (Time Management,

Influence, Teamwork, Critical Thinking)

* Reference Checks (360 Blind

Reference)


Selection Score Elements Raw Score Ranges Normalized

1 Reading Prose 0-500 R-Y-G 1-2-3

2 Reading Documents 0-500 R-Y-G 1-2-3

3 Quantitative Reasoning 0-500 R-Y-G 1-2-3

4 Overall Fit Index 0-100 R-Y-G 1-2-3

5 Recruiter SIG 0-20 R-Y-G 1-2-3

6 Manager SIG 0-20 R-Y-G 1-2-3

7 References 1-6 R-Y-G 1-2-3

Total Score 0-5 Stars

TRUE Score – Selection Evidence


Structured Interview Guide Rating Example

5

Expert

Expert or master level of proficiency. Can apply the competency in

exceptionally difficult situations. Candidate has served as a key

resource and advised others. Ideal answer.

4

Advanced

Exceeds expectations. Can apply the competency in considerably

difficult situations. Candidate requires no guidance. Very strong and

detailed answer.

3

Average

Meets expectations. Can apply the competency in difficult situations.

Candidate may require occasional guidance. Average answer.

2

Basic

Approaches expectations. Can apply the competency in somewhat

difficult situations. Candidate will require frequent guidance. Weak

but acceptable answer.

1

Limited

Limited level of proficiency and below expectations. Can apply the

competency in the simplest situation. Candidate requires close and

extensive guidance. Generally unacceptable answer.


Science

The science behind the success.


Comprehensive Competency Model

Occupational Competencies

Job Specific Competencies – also known as Tasks

or Work Behaviors – are job specific work activities.

Foundational Competencies

Industry-Wide Competencies – industry specific

work activities that are common across multiple jobs.

Cognitive

• Listening

• Speaking

• Reading

• Writing

• Reasoning

• Math

• Information Skills

• Judgment &

Decision Making

Linking

Foundational

competencies are

linked to

occupational

competencies to

establish their

relationship to job

performance

Character

• Achievement

Orientation

• Leadership

• Interpersonal

Orientation

• Adjustment

• Conscientiousness

• Independence

• Practical Intel.

Physical

• Fine Manipulation

• Control

Movements

• Reaction & Speed

• Strength

• Endurance

• Flexibility, Balance

& Coordination

• Vision


Validity

Validation requires measurement and quantitative analysis.


Job Families

O*NET Data: Code, Occupation, Education, and Wages


Predictive Validity – Job Performance

3

1

5

2

6

4

R

A

N

K

The Validity and Utility of Selection Methods in Personnel Psychology:

Practical and Theoretical Implications of 85 Years of Research Findings


Reading + Math + Locating (NCRC) orProse + Document + Quantitative (Literacy) =

Critical Thinking and Problem Solving

NCRC = Critical Thinking and Problem Solving

Most employees are problem solving specialists. They specialize in a particular set of problems. So, critical thinking, problem solving, and judgment and decision making are key competencies for most jobs.


The Five Factor Model of PersonalityA summary of the factors of the Big Five and their constituent traits:

Openness to experience: (inventive/curious vs. consistent/cautious). Appreciation for art, emotion, adventure, unusual ideas, curiosity, and variety of experience. Openness reflects the degree of intellectual curiosity, creativity and a preference for novelty and variety a person has. It is also described as the extent to which a person is imaginative or independent, and depicts a personal preference for a variety of activities over a strict routine. Some disagreement remains about how to interpret the openness factor, which is sometimes called "intellect" rather than openness to experience.

Conscientiousness: (efficient/organized vs. easy-going/careless). A tendency to show self-discipline, act dutifully, and aim for achievement; planned rather than spontaneous behavior; organized, and dependable.

Extraversion: (outgoing/energetic vs. solitary/reserved). Energy, positive emotions, surgency, assertiveness, sociability and the tendency to seek stimulation in the company of others, and talkativeness.

Agreeableness: (friendly/compassionate vs. analytical/detached). A tendency to be compassionate and cooperative rather than suspicious and antagonistic towards others. It is also a measure of one's trusting and helpful nature, and whether a person is generally well tempered or not.

Neuroticism: (sensitive/nervous vs. secure/confident). The tendency to experience unpleasant emotions easily, such as anger, anxiety, depression, or vulnerability. Neuroticism also refers to the degree of emotional stability and impulse control and is sometimes referred to by its low pole, "emotional stability".

Source: Wikipedia – Big Five personality traits, January 24, 2014


The Ten Aspects of the Big Five

Openness to experience:• Intellect

• Openness

Conscientiousness:• Industriousness

• Orderliness

Extraversion:• Enthusiasm

• Assertiveness

Agreeableness:• Compassion

• Politeness

Neuroticism:• Volatility

• Withdrawal

Source: Between Facets and Domains: 10 Aspects of the Big Five


y = 2000.8x - 68638R² = 0.5024

-50000

0

50000

100000

150000

200000

250000

0 10 20 30 40 50 60 70 80 90 100

Mean

Wag

es $

Critical Thinking

Mean Wages vs. Critical Thinking

n=765 … BLS wage data and ONET importance rating


y = 1902.2x - 47338R² = 0.5092

-50000

0

50000

100000

150000

200000

250000

0 10 20 30 40 50 60 70 80 90

Mean

Wag

es $

Complex Problem Solving

Mean Wages vs. Complex Problem Solving


y = 2078.2x - 59947R² = 0.4969

-50000

0

50000

100000

150000

200000

250000

0 10 20 30 40 50 60 70 80 90 100

Mean

Wag

es $

Judgment & Decision Making

Mean Wages vs. Judgment and Decision Making


y = 1631.6x - 58626R² = 0.3113

-50000

0

50000

100000

150000

200000

250000

0 20 40 60 80 100 120

Mean

Wag

es $

Achievement/ Effort

Mean Wages vs. Achievement/ Effort (4.01)


y = 1557.3x - 55620R² = 0.2886

-50000

0

50000

100000

150000

200000

250000

0 20 40 60 80 100 120

Mean

Wag

es $

Persistence

Mean Wages vs. Persistence (4.02)


y = 1501.1x - 56357R² = 0.2507

-50000

0

50000

100000

150000

200000

250000

0 20 40 60 80 100 120

Mean

Wag

es $

Initiative

Mean Wages vs. Initiative (4.03)


2012 2013 2014 2015

1,000,000

600,000


Career Pathways

Organizing community collective impact to build evidence-based career

pathways that help job-seekers.


Collective Impact – Health Career Pathways

Initiatives must meet five criteria in order to be considered collective impact:

• Common Agenda: All participating organizations (government agencies, non-profits, community

members, etc.) have a shared vision for social change that includes a common understanding of the

problem and a joint approach to solving the problem through agreed upon actions.

• Shared Measurement System: Agreement on the ways success will be measured and reported

with a short list of key indicators across all participating organizations.

• Mutually Reinforcing Activities: Engagement of a diverse set of stakeholders, typically across

sectors, coordinating a set of differentiated activities through a mutually reinforcing plan of action.

• Continuous Communication: Frequent communications over a long period of time among key

players within and across organizations, to build trust and inform ongoing learning and adaptation of

strategy.

• Backbone Organization: Ongoing support provided by an independent staff dedicated to the

initiative. The backbone staff tends to play six roles to move the initiative forward: Guide Vision and

Strategy; Support Aligned Activity; Establish Shared Measurement Practices; Build Public Will; Advance

Policy; and Mobilize Funding

Collective Impact is the commitment of a group of actors from different sectors to

a common agenda for solving a specific social problem, using a structured form of

collaboration. The approach of collective impact is placed in contrast to “isolated

impact,” where organizations primarily work alone to solve social problems.


Finding the Right Career Path – Career Coaching


My Career Pathway

Education & Job Training



Stackable Credentials



Job 1Getting Started

Job 2Moving Forward

Job 3Achieve

Goal

Name: _______________________________________________________________________________________________________________________

1

98

7

65

2

10

3

4

The 10-step path that leads beyond surviving to thriving.


Industry Sector Development – HEALTHCAREers Council

Catalyst for Change• Multiple Healthcare Organizations who are part of the same discussions• Talent supply & demand data

Core Ingredients:• Industry similar employers experiencing the same need• Partnerships between education and training that focus on workforce needs• Clearly defined pathways and defined hiring requirements• Utilize various funding options for optimal efficiency• Stakeholders willing to collaborate for the good of the community and business

= Regional, employer-driven partnerships in Healthcare to solve talent demand


Wellbeing: The Five Essential Elements

These are the universal elements of wellbeing that differentiate a thriving life from one spent suffering:

• Career Wellbeing is about how you occupy your time or simply liking what you do every day.

• Social Wellbeing is about having strong relationships and love in your life.

• Financial Wellbeing is about effectively managing your economic life.

• Physical Wellbeing is about having good health and enough energy to get things done on a daily basis.

• Community Wellbeing is about the sense of engagement you have with the area where you live.

Wellbeing is about the combination of our love for what we do each day, the

quality of our relationships, the security of our finances, the vibrancy of our

physical health, and the pride we take in what we have contributed to our

communities. Most importantly, it’s about how these five elements interact.

Social Wellbeing

Community Wellbeing

Financial Wellbeing

Physical Wellbeing

Career Wellbeing

Tom RathJim Harter

2010, Gallup


Job Analysis

The ONET-Based Confirmatory Job Analysis Process is a method to define and

document job-related competencies.


Task List

SME Consensus of Importance Ratings

Competency LinkingTools & Technology

Individual Ratings


Job Analysis – Report A. List of SMEs with demographic information

B. Sign-in sheet for SME session

C. Job Family O*NET Codes

D. Job Family CIP Codes

E. Task Families

F. Competency Families

G. Occupational Credentials

H. Task List

I. Tools and Technology List

J. Competency Linking Examples

K. SME Consensus Ratings

L. Job Family Portrait

M. JOFI Foundational Competency Data

N. SME and O*NET Ratings Alignment


Job Analysis – Process The job analysis is completed in three core phases that are preceded by preparation and followed by analysis and reporting:

• Phase 1 includes job family research using www.onetonline.org, www.ncptac.org, and www.talxcellenz.com, competency model research, and preparation of initial drafts of the occupational credentials list, task list, tools and technology list, and competency linking examples.

• Phase 2 includes job shadowing of subject matter experts (SMEs) to refine the initial drafts and prepare final drafts of the occupational credentials list, task list, tools and technology list, and competency linking examples based on observations and discussions with SMEs.

• Phase 3 is the SME session. The SME session is facilitated by a job analyst with support from employers and community partners. In that session the SMEs finalize the occupational credentials list, task list, tools and technology list, competency linking examples, and reach consensus regarding the ratings for each foundational competency.


Define & Measure – Validity

Validity quantifies the relationship between what a test actually measures and what it is intended to measure or predict.

• Educators want to measure mastery of content

• Employers want to predict job performance


Validity for Educational and Psychological Testing

Standards for Educational and Psychological Testing is sponsored by the American

Educational Research Association (AERA), the American Psychological Association (APA), and the

National Council on Measurement in Education (NCME).

• The purpose of the Standards is to provide criteria for the development and evaluation of tests and

testing practices and to provide guidelines for assessing the validity of interpretations of test scores for

intended test uses.

• Sources of evidence may illuminate different aspects of validity, but they do not represent distinct types

of validity. Validity is a unitary concept. It is the degree to which all the accumulated evidence support

the intended interpretation of the scores for the proposed use.

Validity refers to the degree to which evidence and theory support the interpretations of

test scores for proposed uses of tests. Validity is, therefore, the most fundamental

consideration in developing tests. The process of validation involves accumulating relevant

evidence to provide a sound scientific basis for the proposed score interpretations. It is the

interpretations of test scores for proposed uses that are evaluated, not the test itself. When

test scores are interpreted in more than one way (e.g., both to describe a test taker's current

level of the attribute being measured and to make a prediction about a future outcome), each

intended interpretation must be validated. Statements about validity should refer to particular

interpretations for specified uses. It is incorrect to use the unqualified phrase “the validity of

the test.”


Job Performance – Validation Criterion References

Responses to the following Job Performance constructs will be collected (quintiles format):

• General Task Performance – Behaviors that are formally included in the employee’s

duties. Examples include completing tasks on time and showing attention to detail.

• Safety and Rule Compliance – Being perceptive of and competing work in a safe

manner and complying with applicable policies and regulations.

• Customer Service – Serving and helping customers internal to or external to the

organization. Examples include responding to customers quickly and giving customers

helpful suggestions.

• Empathy, Caring, and Compassion – Being perceptive of and responsive to patients’

needs. Making patients feel safe and respected.

• Work Discipline – Following organizational norms and considering the organization’s

needs. Examples include coming to work on time and not wasting time or organizational

resources.

• Teamwork – Working well with others. Examples include helping others with their work

loads and sharing resources with coworkers.

• Proactive Work Behavior – Showing motivation to do a good job. Examples include

suggesting procedural improvements and exceeding performance standards.

• Overall Job Performance


Career Navigation

System Guidebook

An introduction to the essential elements of the evidence-based career coachingprocess and evidence-based career portfolios.


Stakeholder Guidebook

An introduction to essential elements and step-by-step guidance for creating sector initiatives that build demand-driven evidence-basedcareer pathways.


Talent Excellence

System Guidebook

An introductionto Talxcellenz processes and tools for job analysis and validation studiesto support robust competency-basedcareer pathways.


Copyright © 2015 and 2016 by Bill Guest, Metrics Reporting, Inc.

All rights reserved.

Portions of this work contain copyrighted and trademarked materials that are used with permission.

Contact Information:Bill Guest, 616-430-0828, [email protected]

Jim Sharf, 571-274-2154, [email protected] Cleveland, 616-240-9751, [email protected]

Talxcellenz® is a registered trademark of Metrics Reporting, Inc. The ONET-Based Confirmatory Job Analysis Process and related methods are

the intellectual property of Metrics Reporting, Inc. All rights are reserved.