continuing saga of adaptive testing principles applied to performance and personality measurement

Get Fitted for High Performance

PDRI Proprietary 1

Continuing Saga ofAdaptive Testing Principles Applied to

Performance and Personality Measurement

Walter C. BormanPDRI, an SHL Company

and University of South Florida

Presented toGateway Industrial-Organizational Psychologists

April 3, 2013


PDRI Proprietary 2

Outline

• Description of the “adaptive test” concept• Development of computer-adaptive rating

scales for measuring job performance• Development of NCAPS and GPI-Adaptive

personality inventories• Initial validation results• Faking issue• Conclusions and next steps


PDRI Proprietary 3

Description of the “Adaptive Test” Concept

• Initial ability domain application• Our work began in the performance

domain• Forced-choice Computer Adaptive Rating

Scales (CARS)• Laboratory study showed lower standard

error of measurement and higher validity and accuracy for CARS


PDRI Proprietary 4

CARS Performance Estimation Process


PDRI Proprietary 5

Canadian Forces Project

• Performance rating scales for 8 leadership competencies for officers and CSMs in four rank clusters• Action Orientation and Initiative• Analytical Thinking• Behavioral Flexibility/Change Leadership• Commitment to Military Ethos• Communication• Developing Self and Others• Results Management• Teamwork


PDRI Proprietary 6

Development of Competency Scales

• Workshops with officer and CSM groups to generate items

• Editing process• Items assigned to organizational level• Retranslation by I/O psychologists at PDRI and

CF

• Result is four item pools with 484 to 576 items


PDRI Proprietary 7

Development of Competency Scales(continued)

• Example items from Analytical Thinking• Consistently provides insightful observations

and analyses regarding the organization and solutions to problems (M = 6.12)

• Gathers and then analyzes information from a variety of sources to develop effective and timely solutions to problems (M = 5.13)

• Finds creative solutions to problems but is unable to translate these to relevant, realistic, and practical recommendations (M = 3.08)

• Has considerable trouble analyzing even straightforward problems (M = 1.29)


PDRI Proprietary 8

Development of Competency Scales(continued)

• Example items from Action Orientation• Finds appropriate ways of accomplishing almost all

tasks through initiative and hard work, and follow-through is typically outstanding (M = 6.23)

• In most cases, takes the initiative to complete tasks on or ahead of time (M = 5.12)

• Is not proactive in moving toward objectives, but usually achieves mission success by making steady progress(M = 3.52)

• Is reactive to situations, slow to respond, and rarely seeks to resolve the larger issues (M = 1.43)


PDRI Proprietary 9

Forced Choice Format: Example Item Pairs

Click on the behavior that is more descriptive of the ratee:

Is a good role model for others in the CF related to personal conduct and military ethos (5.69)

vs.

Usually accepts responsibility for own and subordinates’ actions (4.59)


PDRI Proprietary 10



Takes pride in serving the interests of the organization (5.42)

vs.

Actively and consistently promotes the vision and values of the CF; maintains exemplary conduct consistent with this vision (6.76)


PDRI Proprietary 11



Always ensures that subordinates follow policies, regulations, and orders (5.65)

vs.

Fully embraces the military profession and takes pride in the history, traditions, and values of the CF (6.47)


PDRI Proprietary 12

Development of NCAPS

• Idea was to apply CARS concept to non-cognitive testing

• 19 potential personality constructs were identified

• Psychologists rated their importance for 79 Navy jobs

• 10 constructs selected based on means and SDs


PDRI Proprietary 13

Constructs Identified for NCAPS

• Achievement• Social Orientation• Stress Tolerance• Adaptability/Flexibility• Attention to Detail

• Dependability• Dutifulness/Integrity• Self Reliance• Willingness to Learn• Vigilance


PDRI Proprietary 14

Development of NCAPS(continued)

• PDRI generated 1725 items at all trait levels

• Construct and trait level “retranslation” was conducted

• 1494 items survived, 106-199 per construct


PDRI Proprietary 15

Example Items for Social Orientation

• It is easy for me to find something in common with any person I meet (M = 6.36)

• It takes real effort for me to hide my impatience with people who aren’t very bright (M = 1.49)

• I am able to make friends when I put some effort into it (M = 3.63)


PDRI Proprietary

Validation Results

• Concurrent validation study designed with 110 first tour Navy Sailors

• NCAPS and a conventional personality inventory administered and supervisor performance ratings gathered on nine “Navy-wide” dimensions

16


PDRI Proprietary

Validation Results(continued)

• Unit-weighted composite of 10 NCAPS conventional against composite overall performance (r = .13 uncorrected;r = .18 corrected)

• Unit-weighted composite of 10 NCAPS adaptive against composite overall performance (r = .27 uncorrected; r = .37 corrected)

17


PDRI Proprietary

Validities Associated With Different Numbers of Item-Pairs (Dependability)

0.00

0.05

0.10

0.15

0.20

0.25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Item-Pairs

Valid

ity C

oeff

icie

nt

18


PDRI Proprietary

Overview of Global Personality Inventory-Adaptive

• General assessment of normal adult personality with a focus on workplace applications

• Selection, development, classification of employees across levels and industries

19


PDRI Proprietary 20

• Targeted a mid-level taxonomy similar in scope to NCAPS and with similar inclusion criteria:

• Comprehensiveness and breadth• Unidimensionality• Level of specificity• Expectation of criterion-related and construct validity• Also global applicability

• Review of dominant taxonomies resulted in 13 dimension taxonomy:

Measurement Taxonomy

• Self Development• Flexibility• Collaboration• Thoroughness• Reliability• Sense of Duty

• Achievement• Composure• Confidence and Optimism• Independence• Sociability• Influence• Innovation


PDRI Proprietary

Statement Identification and Development

• Specification of score distribution and statement bank size• 7-point scale• Goal = 150 to 200 statements per trait • Facets specified as an item writing guide and to ensure

construct coverage

• Similar methodology to NCAPS statement development• Recruited experienced personality item writers • Item writing training• Pilot items reviewed by project team• Item writing assignments by trait; targeted to parts of the trait

continuum to result in coverage across trait range

21


PDRI Proprietary

Manager-Level Validation

• Concurrent validation study with incumbents in first line leader roles

• Consortium of 8 organizations contributed 14 samples• Diverse set of organizations representing insurance,

telecommunications, financial services, healthcare, retail industries

• N = 1109 supervisors of hourly employees• N = 240 managers of salaried professionals

• The same job performance rating form provided consistent performance criteria across samples• 27 performance dimension rating areas • 7 global rating items

22


PDRI Proprietary

Validation Results

• Concurrent Validity Study Against Supervisor Performance Ratings (N=1349)

• Unit-Weighted Composite Against Overall Performance (r=.28, corrected for criterion unreliability)

• Validities Asymptote at 5-8 Item Pairs• Testing Time ½ that of Conventional

Inventory

23


PDRI Proprietary

Faking ResearchUnderhill, Lords, & Bearden (2006)

• Investigate fake resistance of NCAPS • First study to evaluate the extent to which participants can

deliberately elevate their personality scores on NCAPS• NCAPS and non-adaptive/traditionally-formatted versions used

• Students (N = 148)• T1: respond honestly• T2: deliberately fake to make the best impression possible for

acquiring a job

• Differences in personality scores from honest to faking were compared for each instrument • Adaptive NCAPS: no significant mean differences between

honest and faking scores • Traditional NCAPS: significant mean differences on all traits

measured

24


PDRI Proprietary 25

Conclusions and Next Steps

• Performance rating application shows promise but field study needed: Canadian Forces

• Personality measurement application also promising• Modest validity improvement over non-

adaptive• Shorter testing time by ~50%• Faking not as serious as feared


PDRI Proprietary 26

Conclusions and Next Steps(continued)

• Paired comparison judgment process and iterative IRT-based algorithm with raters or test takers may be an important factor in generating valid information for performance ratings and self-report personality reports

• More research comparing the forced choice adaptive format applied to performance ratings and personality testing to non-adaptive formats should proceed

continuing saga of adaptive testing principles applied to performance and personality measurement

Documents

straightforward problems

example item pairsclick

item pools

timely solutions

creative solutions

personality measurementwalter

hard work

complete tasks