Title VII Case Study:

Plaintiff v. U.S. City

Defending Interviews Against a Title VII

Adverse Impact Claim

February 16, 2011

Visit BCGi Online

• If you enjoy this webinar,

– Don’t forget to check out our other training opportunities through the BCGi

website.

• BCGi Standard Membership (free)

– Online community

– Monthly webinars on EEO compliance topics

– EEO Insight Journal (e-copy)

• BCGi Platinum (paid) Membership ($199/year)

– Fully interactive online community

– Includes validation/compensation analysis books

– EEO Tools including those needed to conduct AI analyses

– EEO Insight Journal (e-copy and hardcopy)

– Members only webinars and training and much more…

www.BCGinstitute.org

HRCI Credit

• BCGi is an HRCI Preferred Provider

• CE Credits are available for attending

this webinar

• Only those who remain with us for at

least 80% of the webinar will be eligible

to receive the HRCI training completion

form for CE submission

44

About Our Sponsor: BCG

• Assisted hundreds of clients with cases involving Equal Employment

Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and

defense)

• Compensation Analyses / Test Development and Validation

• Published: Adverse Impact and Test Validation, 2nd Ed., as a practical

guide for HR professionals

• Editor & Publisher: EEO Insight an industry e-Journal

• Creator and publisher of a variety of productivity

Software/Web Tools:

– OPAC®(Administrative Skills Testing)

– CritiCall®(9-1-1 Dispatcher Testing)

– AutoAAP™ (Affirmative Action Software and Services)

– C4™ (Contact Center Employee Testing)

– Encounter™ (Video Situational Judgment Test)

– AutoGOJA®(Automated Guidelines Oriented Job Analysis

®)

– COMPare: Compensation Analysis in Excel

Title VII Case Study:

Plaintiff v. U.S. City

Defending Interviews Against a Title VII

Adverse Impact Claim

February 16, 2011

Contact Information

Dan A. Biddle, Ph.D.CEO

(800) 999-0438 x 113

dan@biddle.com

6

AgendaAgenda

7

• Interviews in the Courts

• Characteristics of defensible Interviews

• Adverse Impact review

• Case Review

• Validation Process

• Case Outcome

DisclaimerDisclaimer

8

• Discrimination, in both “allegation” and “court

concluded” forms, is a serious issue.

• BCG works on both sides of discrimination

lawsuits and believes that Title VII currently

provides the means for conducting a balanced

investigation of adverse-impact based

discrimination to be investigated.

• Validation should be a process used for building

fair and effective employment devices, and not as

a means for “just defending tests that are

potentially discriminatory.”

Interviews and the Courts

• The question is still sometimes asked…

– “Are Interviews really tests”?

– Yes, they are really tests

• Any Practice, Procedure, or Test (PPT) that separates two groups (e.g., men/women) based on two possible outcomes (e.g., pass/fail) is classified as a “test” under the Uniform Guidelines

Interview Defensibility:

Some General Characteristics…

UnstructuredUnstructured StructuredStructured

Single Rater Single Rater Multiple RatersMultiple Raters

Generic Generic ““one size fits allone size fits all”” Job SpecificJob Specific

Open Scoring/No Scoring Open Scoring/No Scoring BARSBARS

Least DefensibleLeast Defensible Most DefensibleMost Defensible

Interview Validity:

When Interviews are correlated to job

performance, here’s what we typically see:

Unstructured StructuredUnstructured Structured

rr= .11 = .11 -- .18 .24 .18 .24 -- .34 .34

Low Validity Low Validity High ValidityHigh Validity

Litigation Involving Interviews

• Is there a connection between Interview type and success

in court?

• Williamson et al. (1997). Employment interview on trial:

Linking interview structure with litigation outcomes.

Journal of Applied Psychology, 82 (6), 900-912.

– Study involving 84 disparate treatment and 46 disparate

impact cases where interviews were litigated

– 17 interview characteristics were evaluated (e.g.,

objective, subjective, standardized, etc.).

– Study resulted in clear findings that revealed the three

primary ingredients for successful interview validity

defense

Litigation Involving Interviews: Key

Defensibility Characteristics

• The Three Primary Factors Are…

– Interview objectivity and job relatedness, such as:

o Objective and specified criteria

o Trained interviewers

o Validation evidence

– Standardized administration, including:

o Scoring guidelines

o Minimal rater discretion

o Common questions

o Consistency

– Multiple Interviewers

o Implies a shared decision making process

o Rater reliability

What is Adverse Impact?What is Adverse Impact?

Adverse Impact is defined as a Adverse Impact is defined as a substantially substantially

different ratedifferent rate of selection in hiring, of selection in hiring,

promotion, or other employment decision promotion, or other employment decision

which works to the disadvantage of members which works to the disadvantage of members

of a race, sex, or ethnic groupof a race, sex, or ethnic group

14

Adverse Impact: Selection Rate

Comparison (1991 CRA) & UGESP

Amends Section 703 of the 1964 Civil Rights Act (Title VII)

(k)(1)(A). An unlawful employment practice based on

disparate impact is established under this title only if:

• A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and

consistent with business necessity; OR,

• A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate

employment practice, and the respondent refuses to adopt such alternative employment practice.

15

Adverse Impact

Selection Rate Comparison

2 X 2 Table

Used to evaluate hires, promotions and terminations

Statistical Significance:

• 5%

• 0.05

• 1 chance in 20

• 1.96 Standard Deviation

16

Now Let’s Review the Specific

Recent Interview Case…

• Claim from a female minority member holding “Position I” that the city discriminated against her when she applied to be promoted to “Position II”

– The claim of discrimination was based on race using an “Adverse Impact” approach

• A structured oral interview process was used as a primary means of selection

• Plaintiff claimed selection process was not valid (i.e., job related and consistent with business necessity)

• While this case occurred in the public sector context, the process and concepts are identical for private sector employers covered by Title VII

Four-Fifths (80%) Rule of Thumb

• Court indicated that a violation of

the 80% rule of thumb alone is not

sufficient to establish a prima facie

case

• A statistically-significant difference

in selection rates needs to be shown

Position II Selection HistoryPosition II Selection History

Year Status Focal Group Reference Group

2001 Hired x x

2001 Applied x x

2003 Hired x x

2003 Applied x x

2005 Hired x x

2005 Applied x x

2007 Hired x x

2007 Applied x x

• Data from different testing sessions

needs to be combined appropriately

How Should the Data Be Combined?How Should the Data Be Combined?

• Add the years together?

• Conduct adverse impact for each year

and average?

• Use a special statistical analysis

specifically used for combining different

versions of interviews across year?

– Hint! Multiple Event Probability

Test (MEEPT)

SimpsonSimpson’’s Paradox Examples Paradox Example

Simpson's paradox is an apparent paradox in which a correlation

(trend) present in different groups is reversed when the groups are

combined.

Using Appropriate Statistics

• Hiring combined

by simple adding

• Multiple Events

Exact Probability

Test

MEEPT Years

Fisher Exact Mid-P

Probability Value

2001, 2003, 2005, 2007 0.088

2001, 2005, 2007 0.147

Pooled Years

Fisher Exact Mid-P

Probability Value

2001, 2003, 2005, 2007 0.127

2001, 2005, 2007 0.235

The “true” relationship is different for different interviews across

a period of time

PlaintiffPlaintiff’’s Statistical Expertss Statistical Experts

• Mistakenly added a minority group member to the analyses who did not participate in the selection process

• BCG found Plaintiff’s statistical expert’s calculation error

• The Plaintiff’s expert admitted they added the numbers incorrectly when conducting their analyses – When correct data was used, they rested their claim on the fact that minority group members were promoted at a rate less than 80% of whites

– However, the difference in selection rates were not statistically significant

What is Validation?What is Validation?

Definition:

• Traditionally, validation is making sure a

practice, procedure, or test actuallymeasures

what it is designed to measure.

• In a legal realm, a selection procedure is valid

if it can be proven by an employer that it is

“job related and consistent with business

necessity.”

24

Why Validate?Why Validate?

• It is good business

– Validation often results in selection devices that are better at helping to select qualified applicants

– Can reduce the amount of time spent training newly-hired employees

– Sends a message to the community and your employees that you care about doing the “right thing”

– Minimizes potentially negative legal actions

• It is required under certain circumstances

– If there is adverse impact against a protected group of test takers

– If challenged in court or during a government audit

25

Job

Duties

Content Validation Process

Operationally

defined KSAOs

Other KSAOs

Selection

Devices

(e.g., Structured

Oral Interview) Where these

three

overlap is content

valid

Job Analysis: The Foundation for CV Job Analysis: The Foundation for CV

DefensibilityDefensibility

Low: Rely on job

descriptions with

no job analysis

about what is

required on the

job; no links

between duties,

KSAOs, and PPTs

High: Full &

comprehensive

job analysis,

including input

from job experts &

supervisors; links

between duties,

KSAOs required at

entry, and PPTs

How the Job Analysis was ConductedHow the Job Analysis was Conducted

Reviewed Historical

Documents

Feedback from

Incumbent Employees

Individual Telephone

Interviews

Job Analysis

Job Duties

KSAPCs

Surveyed

Job

Expertsusing

AutoGOJA

Analyzed Data

Important/Critical Job

Duties

Important/Critical

KSAOs required upon

entry

Job Analysis was Conducted RemotelyJob Analysis was Conducted Remotely

• A BCG consultant conducted telephone

interviews of Position II incumbents (SMEs),

their supervisors, and a Human Resource

staff member

• A search of the literature was made

regarding the positions

• Previous job postings and job descriptions

were examined

What Ratings Does BCG Collect During What Ratings Does BCG Collect During

a Contenta Content--related Job Analysis?related Job Analysis?

• For Job Duties we collect ratings for:

– Frequency* (Guidelines, 14.C[2])

– Importance* (Guidelines , 14.C[2])

– Differentiating* (Guidelines , 14.C[2])

– Relative difficulty

– Fundamental (Americans with Disabilities Act)

– Assignable (Americans with Disabilities Act)

* = required under federal Guidelines

• For knowledge, skills, abilities, and other

characteristics (KSAOs) we collect ratings for:

– Frequency* (Guidelines, 14.C[2])

– Importance* (Guidelines, 14.C[2])

– Performance Differentiating

– Minimum v. Desirable Qualifications

– Level needed upon entry

– Level needed for success (knowledge only)

What Ratings are Collected during a What Ratings are Collected during a

ContentContent--related Job Analysis?related Job Analysis?

* = required under federal Guidelines

Targeting Key KSAOs for Test Building

• Only certain types of KSAOs can be tested when using content-related validity

o The PPT measures and is a representative sample of a knowledge, skill, or ability and is

o Used in and is a necessary prerequisite to performance of critical or important work behaviors (i.e., shown through a link to job duties)

o Should closely approximate an observable work behavior, or its product should closely approximate an observable work product

• Skill, abilities, & other characteristics being measured should be “operationally defined” in terms of observable work behaviors

Identify Job Duties

Rate Job Duties

Identify KSAOs

Rate KSAOs

Link Important Job Duties and KSAOs

Link Important Job Duties & KSAOs

to Selection Device

Develop Selection Device

Content Validity Flow Chart

Structured Interviews

• Interview questions asked of job candidates were based on an examination of the job– Job Analysis and validation study confirmed the job-relatedness

of the questions

– Job Analysis foundation rested on a review of city documents, telephone interviews, and ratings collected from a panel of Job Experts

• Interview panels were trained by City’s Office Human Resource Agent

• Consensus discussions by the raters were held after each interview was completed

• Candidates were rank ordered based on scores– Effectively used to band – not strict rank order- candidates

• Different oral interview questions were asked of candidates during each year of the selection process

Validity Findings

% of Acceptable Questions

# Testing Standard 2001 2003 2005 2007

1. Is the question CLEAR and UNDERSTANDABLE? 100% 100% 100% 100%

2. Is the question written at an APPROPRIATE DIFFICULTY LEVEL? 100% 100% 100% 100%

3.Is this question FAIR to all groups of candidates (based on their

race, age, and/or gender)?90.90% 100% 100% 88.89%

4.Is the question job related (does it represent a situation that has

occurred or is very likely to occur on the job)? 100% 100% 87.5% 88.89%

5.

Will applicants be able to provide a sufficient response withoutpossessing job-specific knowledge that they will receive during training once they are hired as a Position II?

100% 100% 100% 100%

6. What KSA is measured by this question?

All of the interview questions were linked by the Job Experts to at least one important or critical

KSA.

DonDon’’t Questions Have to Linked to t Questions Have to Linked to

Important or Critical Job Duties?Important or Critical Job Duties?

• Section 14C(2) of the federal Uniform Guidelines indicates that the work behaviors being measured “should be critical work behavior(s) and/or important work behavior(s) constituting most of the job.”

• An interview question to important or critical work behavior (Job Duty) linkage can be inferred in the current study since each interview question was linked to KSAs and we had Job Experts indicate the strength of the linkages between KSAs and Job Duties

Plaintiff Also ClaimedPlaintiff Also Claimed……• Claim: A separate validation study was needed

before scores from interviews could be used for “rank ordering” of candidates

• Response: The process of ranking and/or score banding is permissible under the federal Uniform Guidelines and does not need to be validated using a “separate validation study.”

• Rather, the process can be validated using the job analysis and validation evidence that exists from the current study. The Uniform Guidelines allow for using a selection procedure for ranking job candidates if the user can specify evidence showing that a higher score on the selection procedure is likely to result in better job performance (see Section 14C9).

Even More ClaimsEven More Claims……

• Plaintiff moved to exclude BCG as the City’s Expert based on the fact that the job analysis and validation was conducted after the litigation commenced

• Plaintiff claimed that since the consultant who was facilitating the job analysis did not visit onsite, it was invalid

• Plaintiff claimed interview questions were written to avoid knowledge and skills and to “cater to the favorites who had been pre-selected”

• The job analysis and validation study was conducted after the claim of discrimination was made

• Even though courts and auditing agencies are likely to more strictly scrutinize validation studies that are conducted after adverse impact is determined, this court ruled that it was acceptable to conduct the study after a claim of discrimination was made

EveEve--ofof--Trial StudyTrial Study

The Court RuledThe Court Ruled……

• Court ruled that to successfully challenge the validation study, Plaintiff would need to submit expert testimony or other objective evidence that tended to refute the findings of the study

– Plaintiff did not provide any expert testimony to rebut the City’s validation study

– Plaintiff did not challenge qualifications of Dr. Biddle or the reliability of his methodology or analyses

– Based on these failures, the court denied the Plaintiff’s motion to exclude this study

Adverse Impact During a Specific Test or Adverse Impact During a Specific Test or

During the Entire Process?During the Entire Process?

• Court indicated Plaintiff must identify a

specific test, requirement or practice that has

an adverse impact on race

– Smith v. City of Jackson, Miss., 544 U.S. 228, 229

(2005)

– Plaintiff failed to identify a specific employment

practice that had adverse impact

• For this reason, the court focused only on the

Interviews given for the position, rather than

the promotional process as a whole

Take Take AwaysAways……

• Selection procedures should be job related

(i.e., valid) and fair to all qualified applicants

• Evidence of validity should be available for

review if challenged

• Written records of testing should be

maintained

• The appropriate statistical tests should be used

• Having EEO experts in your corner can really

help to defend your testing practices

Helpful Validation ResourcesHelpful Validation Resources

• Federal Uniform Guidelines on Employee Selection Procedures

– www.uniformguidelines.com

• U. S. Department of Labor’s Testing and Assessment: An Employer's Guide to Good Practices

– www.onetcenter.org/dl_files/empTestAsse.pdf

• Adverse Impact and Test Validation: A practitioner’s guide to valid and defensible employment testing – Author: Dan A. Biddle, Ph.D.

43

44