current methods, problems, and solutions - paul barrett's · pdf file ·...

Paul Barrettemail: [email protected]

http://www.liv.ac.uk/~pbarrett/paulhome.htm

Affiliations: Chief Scientist at The State Hospital, CarstairsSenior Research Fellow at the Dept. of Clinical Psychology

Univ. Of LiverpoolMarch 30th , 2001

Paul Barrettemail: [email protected]

http://www.liv.ac.uk/~pbarrett/paulhome.htm

Affiliations: Chief Scientist at The State Hospital, CarstairsSenior Research Fellow at the Dept. of Clinical Psychology

Univ. Of LiverpoolMarch 30th , 2001

Integrity Testing BCS-ISSG Oxford, March 2001

Current Methods, Current Methods, Problems, and SolutionsProblems, and Solutions


The Collins English Dictionary, 3rd edition (1991)defines an honest person as one “not given to lying, cheating, stealing etc., trustworthy; not false or misleading, genuine; and characterised by sincerity and candour”. Integrity is defined as “adherence to moral principles, honesty; the quality of being unimpaired”.


Psycho-physiology Interview Biodata

Integrity Assessment

CovertCQT GKT P300 Arousal Covert and Overt Overt

Psychometrics

Control Question Test. Measures the

difference in response to a "critical" question

vs a more "neutral" question

Personality Trait measures

re-weighted to maximise prediction

of some criterion.

Specialist Integrity tests that are highly face-valid (in that

what is being assessed is obvious

to the test-taker

The P300 is a positive voltage evoked potential - related to information processing of features

of a stimulus - at around 300ms after the

stimulus has been presented

Psychopaths and some individuals associated

with delinquent behaviours show a

reduced level of arousal/reactivity to

"startle-inducing" stimuli

Guilty Knowledge Test. Measures the response

to information embedded in a question - the Peak of Tension (POT) method is an

ordered series of such information-embedded

questions

Milliseconds

Mic

rovo

lts

-8

-6

-4

-2

0

2

4

6

8

0 50 100 150 200 250 300 350 400 450 500

N1 or N100

P2 or P200Brain-Stemactivity

P3 or P300Late positivecomplex

P1



! Investigative methodology that is characterised by the construction of variables which possess a quantitative structure. ! A variable which possesses quantitative structure is required to satisfy the 9 conditions of ordinality and additivity (the 9 uniformities of co-existence from J.S. Mill)

Quantitative Science


The 9 axioms of Quantitative Measurement .1

From Michell (1990, p.52):Let X, Y, and Z be any three values of a variable Q. Then Q is ordinalordinal if and only if:

1. If X ≥ Y and Y ≥ Z then X ≥ Z (transitivity)2. If X ≥ Y and Y ≥ X then X = Y (antisymmetry)3. Either X ≥ Y or Y ≥ X (strong connexity)

A relation possessing these three properties is called a simple order, so Q is ordinal if and only if ≥ is a simple order on all its values.


All quantitative variables are simply ordered by ≥ , but not every ordinal variable is quantitative, for quantity involves more than order, it involves additivityadditivity.AdditivityAdditivity involves a ternary relation, symbolized as “X+Y=Z”. Let Q be any ordinal variable such that for any of its values X, Y, and Z …

4. X+(Y+Z) = (X+Y)+Z (associativity) 5. X+Y = Y+X (commutativity)6. X ≥ Y if and only if X+Z ≥ Y+Z (monotonicity)7. If X > Y then there exists a value of Z such that X=Y+Z (solvability)8. X+Y > X (positivity)9. There exists a natural number n such that nX ≥ Y

(where 1X = X and (n +1)X = nX + X) (Archimedean condition)

The 9 axioms of Quantitative Measurement .2


! Measurement in quantitative science is defined as the identification of a magnitude of a quantitative variable relative to some standard unit magnitude of that variable. ! This measurement uses numerical relations to express the ratio of magnitudes.! These numerical relations are the real-valued, positive number system.! There is thus an isomorphic (one-to-one) relation between the numbers used to represent magnitudes, and the standard unit for a variable.! The concatenation of standard units for a variable is additive – given a fixed-property unit.


So?

! Integrity measurement has no standard unit.! Any measurement that uses conventional, classical psychometric quantitative procedures is required to assume that such a standard unit exists.! Science requires that the meaning of the proposed standard unit is subsequently explored, tested, and better understood.! This does not mean that integrity cannot, or is not, being measured by various procedures, but rather, the laissez-faire attitude toward measurement taken by most psychologists renders the measurement of integrity as a somewhat ambiguous issue to be approached with care.


Reliability

The measurement remains stable over time (test-retest), and the components of a composite measure (test score) all measure the same attribute (internal consistency). Associated with internal consistency reliability is the concept of a measure being a measure of a single, unidimensional attribute

Validity

Does the test actually measure what it purports to measure?


Corrections for Unreliability and Restriction of Range

Most quoted correlations between Integrity measures and Job Performance, offences, or counter-productive behaviour are corrected for unreliability of measurement in the criterion or test and/or restricted range of measurement in the test and/or the criterion.


Taken from:Ones, D.S., Viswesvaran,C., and Schmidt, F.L (1993) Comprehensive Meta-Analysis of Integrity Test Validities: Findings and Implications for Personnel Selection and Theories of Job Performance, Journal of Applied Psychology, vol.78 (4) 679-703.

Ones, D.D., Schmidt, F.L. and Viswesvaran, C. (1993) Nomological net for measures of integrity and conscientiousness. Paper presented at the 8th annual conference of the Society for Industrial and Organizational Psychology , San Francisco.

*Personality Composite = agreeableness, conscientiousness, and emotional stability

Measures Uncorrected (actual)

Corrected (True-Score)

Integrity Test vs Job Performance 0.33 0.47 Personality Composite vs Job Performance 0.25 0.39


N=2000 cases, Bivariate Normal sample, Correlation=0.43

Dishonesty Score (high score = more dishonest)

Job

Perf

orm

ance

6

8

10

12

14

16

18

20

22

24

25 35 45 55 65 75 85 95

Region of InterestIndividuals with low Dishonesty scores possess the same Job Performance rating as those with high scores


Filtered Dataset (no case above population mean score)N=1023, Correlation = 0.34

Dishonesty Score, High Score = more Dishonesty)

Job

Perf

orm

ance

6

8

10

12

14

16

18

20

22

25 35 45 55 65 75 85 95


Filtered Dataset - Integrity Scores between 46 and 64N=1400, Correlation = 0.32

Dishonesty Score (High Score - more Dishonesty)

Job

Perf

orm

ance

6

8

10

12

14

16

18

20

22

25 35 45 55 65 75 85 95


Uncorrected Corrected Population Correlation 0.43 - High Scorers Clipped 0.34 0.52 Low and High Scorers Clipped 0.32 0.49

Corrected vs Uncorrected Correlations


But, do we want to use correlations at all as an indicator of the “utility” or “validity” of a measure?

Let’s look at some simulated Integrity Score prediction data – where we have a base rate of 10% “undesirables” – i.e. those who are classified by their behaviours as “lacking integrity”.

First, we see how well we can predict the undesirables if we were to just randomly reject 1 out of 10 candidates.


*Simulated data, 10% base rate, no test – random selection


*Simulated data, 10% base rate, Integrity Test Selection


If the cost of a False Negative (someone we predict to be “honest” and so employ, but in fact they turn out to commit dishonest behaviours) is £10,000 on average (taking into account all the costs associated with fraud detection, HR issues, and job-replacement), then by selecting an instrument solely upon its “validity” correlation coefficient, we might choose the one with a validity coefficient of 0.3325 (Pearson r) – which would cost us £290,000 in terms of “failure”. If we chose the other one with a coefficient of 0.3331, it would cost us £110,000 –a cost saving of £180,000. As the costs of fraud near £25,000 per individual, savings approach £½million.


Hard and Soft Criteria?

Hard … Stealing of goods, Absenteeism, Aggressive Incidents, Financial Fraud, Shrinkage

Soft … Supervisor Ratings, Workplace Attitudes, Performance Targets, Dress


Social Desirability not a problem?

♦ When candidates distort their responses, this can be systematicsystematic in that scale scores are elevated by some constant across all candidates (everybody tends to increase their scores say on conscientiousness). This kind of distortion has no effect upon the affected trait scale score and some criterion ….

Systematic Variation: Hypothetical Applicant and Non-Applicant Data Correlation between Job Criterion score in each group = 0.52

Conscientiousness Sten score

Job

Cri

teri

on S

core

0

2

4

6

8

10

12

14

0 1 2 3 4 5 6 7 8 9 10

Non-Applicants

Applicants


Brown and Barrett, 1999


Social Desirability not a problem?

♦ Alternatively, the distortion may be nonnon--systematicsystematic, with certain candidates obtaining elevated scores whilst others remain static. This kind of distortion has unpredictable consequences upon trait-criterion correlations. An example below shows what happens when “true low-scorers on conscientiousness” tend to fake-good at a rate relative to the size of their low scores, whilst average to high scorers maintain their “true” score.


Non-Systematic Distortion of Conscientiousness scores (Faking Good)Non-Applicant correlation = 0.52, Applicant Correlation = -0.08

Conscientiousness Sten Score

Job

Cri

teri

on S

core

0

2

4

6

8

10

12

14

0 1 2 3 4 5 6 7 8 9 10

Non-Applicants

Applicants

Brown and Barrett, 1999


Key “competing” References:

Ones, D. S., Viswesvaran, C. & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: the red herring. Journal of Applied Psychology, 81, 660-679.

Barrett, P.T. and Hutton, R. (2000) Personality and Psychometrics. Selection and Development Review, 16, 2, 5-9


A simple question that should be asked by anyone thinking of using any psychological test …

How accurate is this test?How accurate is this test?

And …

Is a correlation/validity coefficient a Is a correlation/validity coefficient a sufficient answer to this question?sufficient answer to this question?

Answer = a resounding Answer = a resounding No!No!


Dose-Response Curve for a Continuous/Ordered-Category criterion variable

Integrity Score

Prob

abili

ty o

f a C

ount

er-P

rodu

ctiv

e B

ehav

iour

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1 2 3 4 5 6 7 8 9

��

��

��

��

��

��

��

��

��


Rates of Violent Recidivism for subjects at each of 9 risk levels7 year recidivism follow-up - Taken from Rice (1997)

Violence Risk Level (VRAG)

Rat

e of

Vio

lenc

e R

ecid

ivism

1171 101

111

11696

74

29

9

0.00.10.20.30.40.50.60.70.80.91.01.1

0 1 2 3 4 5 6 7 8 9

These are the numbersof patients who obtainedthe VRAG score

Predicting risk of violent recidivism in forensic mental health


The VRAG dichotomous/binary outcome decision table


" ROC analysis permits an investigator to evaluate a fixed or “to be inferred” criterion value to make accurate discrimination between two outcomes (whether they be 2 different stimuli, stimulus vs no-stimulus, violent offence vsno violent offence, or risk factor vs no risk factor). The criterion value may be a test score, level of risk factor, drug dosage, cognitive judgement, or any other kind of variable that can possess at least a binary categorical, multiple ordinal, or even equal-interval magnitudes). " The ROC curve, in every case, consists of plotting the True PositiveTrue Positive rate on the ordinate (vertical, y-axis) and the FalseFalse--AlarmAlarm rate (False PositiveFalse Positive Proportion or rate) on the abscissa (horizontal, x-axis).


ROC Curve for Quinsey et al (1994) VRAG instrumentUsing 7 yr recidivism probabilities

False Alarm, False Positive Rate (1-Specificity)

Sens

itivi

ty (P

rob.

of c

orre

ctly

pre

dict

ing

offe

nce)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

50/50 Chance level

VRAG Score = 5 VRAG Score = 4


Interpreting a classification functionA Classifier function as utilised in the ATH-1

The Honesty Variable

Prob

abili

ty

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

xc

xc The decisioncriterion value

False PositivesFalse Negatives

True PositivesTrue Negatives

The Prisoner Score Distribution

The General PublicScore Distribution


The weighting of Integrity Test Scores in a Multi-Attribute Selection Battery

! That is, is it the case that regardless of any other test scores, if the integrity score signals “low degree of honesty”, then the candidate’s application is rejected?! Psychologists talk about an integrity test as just one of many indicators of a prospective employee’s attributes – but this is not how they are used. If you are told that a candidate’s “honesty” score is very low, would you want to employ them, even if they meet other targets on your person specification?


The Meaning of Integrity Test Scores?

! Conscientiousness and Integrity: Ones et al (1993) … she demonstrated that although Conscientiousness and Integrity were moderately correlated (0.26 uncorrected and 0.39 corrected for unreliability and restriction of range in both tests), job performance was best predicted by integrity test scores. ! Eysenck Personality Questionnaire – Tough-Minded vs ATH-1 Integrity = 0.62 (uncorrected) – a study carried out by myself, and published by Permetric (the original ATH-1 test publisher) as a data addendum in 1989


! Permetric also found that many entrepreneurial managers and executives had ATH-1 scores much closer to prisoners than to the general public. However, this makes a great deal of sense in that entrepreneurs are mould-breakers by definition – they are not compliant, and do not always follow others’ rules except if they are congruent with their own business needs. This interpretation is borne out by the EPQ P-scale correlation noted above.


! All interventions within a business environment must be subject to explicit quantitative evaluation of outcome! The optimal information about integrity tests is embodied in decision-theoretic statistics and full prediction error diagnostics. ! Stay away from correlations and validity coefficients, unless they also come with a portfolio of diagnostic statistics that allow you to properly evaluate a cost-benefit analysis for the assessment strategy.! Think how you will provide feedback to a rejected candidate – for you will have to (APA and BPS guidelines).As you do this, you will suddenly realise just how critical it is to be able to justify any form of cut-off or region of rejection.

current methods, problems, and solutions - paul barrett's · pdf file ·...

Documents