assessment center: evaluation of a developmental...

Assessment Center: Evaluation of a Developmental Assessment

Center in Veterans HealthcareCenter in Veterans Healthcare

Boris Yanovsky, Robin L. Graff-Reed, Kasey Kruer, Katerine Osatuke, and Sue R. Dyrenforth

Traditional Vs. Developmental

• Assessment Centers (ACs) are popular tools for selection and promotionselection and promotion

• Also used for developmental purposes – Developmental Assessment Centers (DAC)( )

• These two types are very different from each other:• Prediction vs Development

• DAC used by Veterans Health Administration within a leadership development program for aspiring executivesexecutives

• We were interested to see to what extent our DAC was indeed developmental

What Did We Examine?

• Differences in scoring between ratersDifferences in scoring between raters (Mentors and Preceptors)

• Perceived learning that occurred across i i iactivities• Developmental properties of dimensions (VHA

competencies targeted by the DAC)competencies targeted by the DAC)

• Construct validity of dimensions• Construct validity of dimensions

VHA Core Competencies1. Interpersonal Effectiveness2 Customer Service2. Customer Service3. Systems Thinking4. Flexibility/Adaptability5. Creative Thinkingg6. Organizational Stewardship7 Personal Mastery7. Personal Mastery8. Technical Skills

Assessment Center Exercises Used to Develop Competencieseve op Co pe e c es

1. Meeting Management• Interpersonal Effectiveness, System Thinking, and

Technical Skills

2 Performance Based Interview2. Performance Based Interview• All eight competencies are assessed

3. Executive Team Simulation3. Executive Team Simulation• All eight competencies are assessed

4. Press Conference Simulation4. Press Conference Simulation• All but Technical Skills and Creative Thinking

5. Inbox Simulation• Not aligned with Core Competencies

Diff B t R tDifferences Between Raters: Mainly Non Significanty g

Executive Team Simulation Rater Differences: All Three Dayse e ces: ee ys

5

3

43.

96 4.00

3.81

3.96

3.96

4.01

3.99

4.03

4.05

4.10

3.94

4.04

3.94

3.87

3.88

4.01

2

3

Mentor

1Preceptor

0ctiveness

Thinking

l Mastery

ical Skills

ardship

daptability

er Service

Thinking

Interpersonal Effectiv

Systems Th

Personal MTechnica

Organizational Stewar

Flexibility/Adapt

Customer

Creative Th

Executive Team Simulation Rater Differences: Mondaye e ces: o d y

5

3

43.

733.

89

3.58

3.89

3.88

3.90

3.81

3.98

3.9

4.07

.75

4.04

.73

3.80

3.76

4.01

2

3

Mentor

3 3 3 3 3 3

1Preceptor

0ctiveness

Thinking

l Mastery

ical Skills

ardship

daptability

er Service

Thinking


Systems Th

Personal MTechnica


Flexibility/Adapt

Customer

Creative Th

Executive Team Simulation Rater Differences: Tuesdaye e ces: uesd y

5

3

43.

95 3.99

3.8 3.

87

3.83

3.96

3.95

4.07

3.94

3.99

3.95

3.87

3.9

3.68

3.8

3.91

2

3

Mentor

3

1Preceptor

0ctiveness

Thinking

l Mastery

ical Skills

ardship

daptability

er Service

Thinking


Systems Th

Personal MTechnica


Flexibility/Adapt

Customer

Creative Th

Executive Team Simulation Rater Differences: Thursdaye e ces: u sd y

5

1 6 0 9 .25

43

44.

31 4.20

4.17

4.21

4.27

4.26

4.3 4.

09

4.43

4.30

4.22

4.2

4.31

4.

4.21 4.

14

2

3

Mentor

1Preceptor

0ctiveness

Thinking

l Mastery

ical Skills

ardship

daptability

er Service

Thinking


Systems Th

Personal MTechnica


Flexibility/Adapt

Customer

Creative Th

Press Conference Rater Differences

5

3

43.

6 3.65

3.69 3.

69

3.81

3.84

3.94

4.08

2

3

Mentor

3 3

1

2Preceptor

0king ing

kills ure

Systems Thinking

Ethical Reasoning

Listening Skills

Maintains Composure

Construct Validity of Dimensions: yDimensions Within Exercises Do

Not Differentiate From Each OtherNot Differentiate From Each Other

ETS All 3 Days with Press Conference

Pres

s

Pres

s

ess C

onf

Pres

s

ETS

ETS ET

S

v. in

ETS

ty in

ETS

ETS

tem

s Thin

king

in P

Conf

ical R

easo

ning

in Co

nfen

ing S

kills

in Pr

ent

ains C

omp.

in P

Conf

tem

s Thin

king

in E

Stew

ards

hip in

Esto

mer

Ser

vice

in

r-per

sona

l Effe

ctiv

xibilit

y / A

dapt

abilit

yso

nal M

aste

ry in

Syste

Ethic

Liste

Main

Syste

Org S

Cust

Inte

r

Flex

i

Pers

Systems Thinking in Press Conf Pearson r XEthical Reasoning in Press Conf Pearson r .569(**) XListening Skills in Press Conf Pearson r .670(**) .581(**) XMaintains Comp. in Press Conf Pearson r .684(**) .438(**) .732(**) XSystems Thinking in ETS Pearson r .415(**) .323(*) .438(**) .279(*) XOrg Stewardship in ETS Pearson r .451(**) .423(**) .320(*) 0.238 .622(**) XCustomer Service in ETS Pearson r .460(**) .420(**) .373(**) .372(**) .573(**) .626(**) XInter-personal Effectiv. in ETS Pearson r .456(**) .455(**) .481(**) .437(**) .545(**) .632(**) .616(**) XFlexibility / Adaptability in ETS Pearson r .515(**) .362(**) .469(**) .504(**) .629(**) .696(**) .656(**) .711(**) XPersonal Mastery in ETS Pearson r .421(**) .300(*) .385(**) .375(**) .449(**) .637(**) .576(**) .738(**) .617(**) X**. Correlation is significant at the 0.01 level (2-tailed).

* C l ti i i ifi t t th 0 05 l l (2 t il d)

Sample size fluctuated between 57-61

*. Correlation is significant at the 0.05 level (2-tailed).

ETS Monday with Press Conference

ss C

onf

Pres

s

ss C

onf

ress

ETS

TS ETS in E

TS

y in

ETS

ETS

tem

s Thin

king

in Pr

esica

l Rea

soni

ng in

PCo

nfte

ning

Skil

ls in

Pres

inta

ins C

omp.

in P

reCo

nfte

ms T

hink

ing

in ET

Stew

ards

hip

in E

TSst

omer

Ser

vice

in E

r-per

sona

l Effe

ctiv.

xibilit

y / A

dapt

abilit

y so

nal M

aste

ry in

ET

Syste

Ethic

Liste

Mai

n

Syst

Org

Cust

Inte

r

Flex

Pers

Systems Thinking in Press Conf Pearson r XEthical Reasoning in Press Conf Pearson r .569(**) XListening Skills in Press Conf Pearson r .670(**) .581(**) XMaintains Comp in Press Conf Pearson r 684(**) 438(**) 732(**) XMaintains Comp. in Press Conf Pearson r .684(**) .438(**) .732(**) XSystems Thinking in ETS Pearson r 0.247 0.126 0.248 0.084 XOrg Stewardship in ETS Pearson r .297(*) .302(*) 0.136 0.009 .616(**) XCustomer Service in ETS Pearson r .261(*) .347(**) 0.249 0.106 .549(**) .694(**) XInter-personal Effectiv. in ETS Pearson r 0.233 0.190 .279(*) 0.177 .560(**) .614(**) .639(**) XFlexibility / Adaptability in ETS Pearson r .265(*) 0.241 .297(*) .276(*) .646(**) .624(**) .549(**) .727(**) XPersonal Mastery in ETS Pearson r 0.163 0.145 0.152 0.089 .497(**) .600(**) .593(**) .722(**) .692(**) X


**. Correlation is significant at the 0.01 level (2-tailed).

*. Correlation is significant at the 0.05 level (2-tailed).

Sample size fluctuated between 56 61

ETS All 3 Days with Meeting Managementge e

Mgm

t

Mgm

t

n M

tngM

gmt

hinkin

g in

Mtn

g M

Skills

in M

tng

Mg

vene

ss i

n ET

STh

inking

in E

TSSk

ills in

ETS

I/P S

kills

in

Syst

em T

h

Tech

nica

l S

I/P E

ffect

iv

Syst

ems T

Tech

nica

l S

I/P Skills in MtngMgmt Pearson r XSystem Thinking in Mtng Mgmt Pearson r 856(**) XSystem Thinking in Mtng Mgmt Pearson r .856(**) XTechnical Skills in Mtng Mgmt Pearson r .735(**) .836(**) XI/P Effectiveness in ETS Pearson r -0.072 0.097 0.154 XSystems Thinking in ETS Pearson r -0.156 -0.037 0.109 .545(**) X

S PTechnical Skills in ETS Pearson r -0.146 0.048 0.125 .599(**) .651(**) X

N=61**. Correlation is significant at the 0.01 level (2-tailed).

Meeting Management with Press ConferenceCo e e ce

Conf

onf

Mgm

t

nkin

g in

Pre

e Co

kills

in P

ress

Co

king

in M

tng

Mg

alSk

ills in

Mtn

g

Syst

ems

Thin

List

enin

g Sk

ilSy

stem

Thi

nkIn

terp

erso

nal

Systems Thinking in Press Conf Pearson r XSystems Thinking in Press Conf Pearson r XListening Skills in Press Conf Pearson r .670(**) XSystem Thinking in Mtng Mgmt Pearson r 0.170 0.177 XInterpersonal Skills in Mtng Pearson r 0.155 0.112 .856(**) X( )


**. Correlation is significant at the 0.01 level (2-tailed).

Developmental Properties of Dimensions: Scores Increased

Across AdministrationsAcross Administrations

ETS Scores Across 3 Days5

3

4

2

3

MondayTuesday

1

TuesdayThursday

0ctiveness

s Thinking

al Mastery

nical Skills

wardship

daptability

er Service

e Thinking

Interpersonal Effecti

Systems T

Personal MTechnic


Flexibility/Adap

Customer

Creative T

Inbox and PBI Scores Across 2 Days

1 5

0.8

0.9

4

0 5

0.6

0.7

Day 1

3

Day 1

0.3

0.4

0.5 Day 1Day 2

2

Day 1Day 2

0

0.1

0.2 1

0Inbox 0

PBI

Summary• Differences Between Raters: Mainly non-

significant• Raters (Mentors and Preceptors) tended to agree

with each other on ratees’ (Candidates’) ratings• Perceived Learning/DevelopmentPerceived Learning/Development

• Participants improved on scores in activities with multiple administrations, showing evidence for the d l t l t f th t t d t idevelopmental nature of the targeted competencies

• Construct Validity• Competencies did not measure separate constructsCompetencies did not measure separate constructs• This finding on our DAC is consistent with previous

research on DACs

Future Considerations

• Further training using behavioral anchors for raters

• Would allow for raters to better diff ti t b t di idifferentiate between dimensions

• Increases construct validitySi ll C did t l t d b• Since all Candidates were evaluated by separate Raters, true inter-rater reliability could not be assessedreliability could not be assessed

• Techniques exist for breaking down variance and could be used for future DACvariance and could be used for future DAC evaluation

assessment center: evaluation of a developmental...

Documents