selection and assessment in safety-sensitive...

Selection and Assessment in Safety-Sensitive Industries

Selection System Development and Validation Portfolio

Lynne R. Sherman, Ph.D. Industrial and Organizational Psychologist

Selection System Development and Validation Portfolio i Lynne R. Sherman, Ph.D.

Table of Contents Introduction ................................................................................................................................. 1 Selection and Assessment Testimonials ..................................................................................... 2 Selection System Development and Validation Projects ............................................................. 3 Selection System Validation Documentation ............................................................................... 4 Training Programs that Support the Selection System Development and

Administration Processes ................................................................................................ 7 Expert Witness Testimony and Litigation Support Activities ........................................................ 8 Sample Job Knowledge Test Items ............................................................................................. 9

Reading Comprehension Test ......................................................................................... 9 Mathematical Ability Test ............................................................................................... 10 Instrumentation Reading Test ........................................................................................ 11 Troubleshooting Test ..................................................................................................... 13

Selection System Development and Validation Portfolio 1 Lynne R. Sherman, Ph.D.

Introduction Selecting candidates in safety-sensitive industries – such as healthcare, aviation, and public safety – presents a number of unique challenges not found in other industries, the most notable of which is the potential catastrophe associated with making a bad hiring decision. Public health can be at risk, fines can be levied, negative press can result, large-scale property damage can occur, and most importantly, lives can be lost. Thus, it is critical that selection systems used for hiring employees in safety-sensitive industries follow all professional, legal, and regulatory requirements that apply to selection system development and validation. Included in this portfolio are several documents that highlight my expertise in selection and assessment in healthcare, aviation, and public safety organizations. These documents include:

Selection and Assessment Testimonials

Selection System Development and Validation Projects: Organization x Selection System Component Grid

Selection System Validation Documentation

Training Programs that Support the Selection System Development and Administration Processes

Expert Witness Testimony and Litigation Support

Sample Job Knowledge Test Items For additional information, feel free to contact me at 203-641-6212 or at [email protected].

mailto:[email protected]


Selection and Assessment Testimonials

“Lynne, you have earned the respect of all the people you work with, from front-line mechanics and pilots to your colleagues in the People division and senior leaders. In your role as an expert witness for United, you saved the organization hundreds of thousands of dollars in employment litigation costs and penalties. In short, you exemplify the best in us all: you can always be counted on to deliver a professionally-developed, high-quality product when we need it.” Dennis Armstrong, Ph.D. Director, Organizational Effectiveness United Airlines – World Headquarters

“Lynne is regarded as an asset by those who have worked with her, and her skills are clearly in line with our needs. She is an exceptionally competent practitioner in her field and has had a tremendous impact on helping us become a more highly functioning organization. Personally I really enjoy working with Lynne!” Elizabeth Conrad Vice President of Human Resources Yale-New Haven Hospital / Saint Raphael Campus

“On behalf of the Airframe Maintenance SFOOV Training Department, I want to express my sincere appreciation for all that you have done for us. As a direct result of your expertise and hard work, we have shown an immediate return on investment in the maintenance division as a result of the new skills assessments and structured interviews you developed and validated for use in hiring new airframe mechanics. We have seen a higher quality, better prepared workforce with an improved work ethic. These results are especially important in light of the recent heightened safety awareness among major air carriers. Without your tenacity and quest for accurate maintenance knowledge, we would not be able to hire and promote such high quality candidates. Lynne, you have proven your worth over and over again in our quest for quality. We’ve also learned a great deal from you!” Tony Barker Team Leader, SFOOV Training Department United Airlines – Maintenance Operations Center, San Francisco


Selection System Development and Validation Projects Organization x Selection System Component Grid

Organization

Beh

avio

rally-B

ased

S

tru

ctu

red

In

terv

iew

Jo

b K

no

wle

dg

e

Te

st(

s)*

Skills

Assessm

en

t

Wri

tin

g S

am

ple

Assessm

en

t

Wo

rk S

tyle

s /

Pers

on

ality

In

ven

tory

Assessm

en

t C

en

ter –

In-B

asket

Exerc

ise

Assessm

en

t C

en

ter –

Ora

l B

riefi

ng

Ph

ysic

al A

bil

itie

s

Te

st

Valid

ati

on

Do

cu

men

ts o

n F

ile

Su

ccessfu

lly

Wit

hs

too

d L

eg

al

Ch

all

en

ge

Waterbury Hospital

Registered Nurse n/a

Patient Care Associate (CNA) n/a

Unit Clerk n/a

OD Consultant n/a

Saint Raphael Healthcare System (now Yale-New Haven Healthcare)

Patient Care Manager n/a

Registered Nurse n/a

Graduate Nurse n/a

Support Associate n/a

United Airlines

Flight Officer (Pilot / Copilot / Flight Engineer)

Yes

Airframe and Powerplant Mechanic

Yes

Radio and Electric Base Mechanic

Yes

Radio and Electric Line Mechanic Yes

Radio and Electric Shop Mechanic

Yes

Powerplant Base Mechanic n/a

Powerplant Line Mechanic n/a

General Mechanic n/a

Mechanic Helper n/a

Storekeeper Yes

Painter n/a

Technical Recruiter – Flight Operations

Yes

Technical Recruiter – Fleet Operations

n/a

Flight Officer Training Database Manager – Fight Safety

n/a

Senior Staff Analyst – Flight Safety

n/a

Data Entry Associate – Flight Safety

n/a

Administrative Assistant – Flight Safety

n/a

Director of Maintenance Operations

n/a

* Job Knowledge Tests include:

Critical Thinking Test

Data Entry Test

Floor Plan Reading Test

Following Directions Test

Instrumentation Reading Test

Map Reading Test

Mathematical Ability Test

Mechanical Reasoning Test

Med Administration Test

Problem, Cause, Solution Test

Radio and Electric Mechanic Test

Reading Comprehension Test

Troubleshooting Test


Organization

Beh

avio

rally-B

ased

Str

uctu

red

In

terv

iew

Jo

b K

no

wle

dg

e

Te

st(

s)*

Skills

Assessm

en

t

Wri

tin

g S

am

ple

Assessm

en

t

Wo

rk S

tyle

s /

P

ers

on

ality

In

ven

tory

Assessm

en

t C

en

ter –

In-B

asket

Exerc

ise

Assessm

en

t C

en

ter –

Ora

l B

riefi

ng

Ph

ysic

al A

bil

itie

s

Te

st

Valid

ati

on

D

oc

um

en

ts o

n F

ile

Su

ccessfu

lly

Wit

hs

too

d L

eg

al

Ch

all

en

ge

General Electric

Collections Adjuster n/a

Greater Cleveland RTA

Upholsterer n/a

Merchant Bank

Account Processor n/a

Bank Teller n/a

Clerk Typist n/a

Collections Adjuster n/a

Encoder n/a

Exception Item Processor n/a

Sorter Operator n/a

City of Akron, OH

Fire Department – All ranks Yes

Police Department – All ranks Yes

City of Cleveland, OH



City of Chicago, IL



City of Jacksonville, FL



City of Kent, OH



City of Lakewood, OH


Building Inspector Yes

Clerk Typist Yes

Wastewater Plant Operator Yes

Water Treatment Plant Operator Yes

City of Mansfield, OH



City of Willoughby, OH



* Job Knowledge Tests include:

Critical Thinking Test

Data Entry Test

Floor Plan Reading Test

Following Directions Test


Map Reading Test

Mathematical Ability Test

Mechanical Reasoning Test

Med Administration Test

Problem, Cause, Solution Test

Radio and Electric Mechanic Test

Reading Comprehension Test

Troubleshooting Test


Selection System Validation Documentation The following documents are routinely developed as part of the selection system validation process and provided to clients / organizations as appropriate.

Technical Report A formal technical report is written for each selection system that is developed, and it contains detailed information about the steps that were taken to create and validate the selection system. In most cases, a multiple hurdle selection system is developed in which case, the development and validation of each component of the selection system is included in the report. This document serves as the foundation of legal defensibility. Several appendices are typically referenced in and attached to the report; these appendices may include the following information:

Job description with major work behaviors, task analysis, and cognitive decision points Note: In the case of positions for which there are variations or specializations (e.g., Emergency Department Registered Nurse vs. Behavioral Health Registered Nurse vs. Critical Care Registered Nurse, etc.), a core job description is developed which represents commonalities across all similar positions and targeted job descriptions representing different specializations supplement the core job description.

Job specifications including minimum qualifications

ADA documentation

Challenges and responses to job knowledge test items for Civil Service exams

Job Analysis Data

Demographic information for each person interviewed / observed during the job analysis process

Job analysis notes generated from supervisor and incumbent interviews

Job analysis notes generated from on-the-job observations

List of source material which represents knowledge, skills, and abilities required prior to job entry

List of resources available to incumbents

Draft job descriptions reviewed and approved by incumbents

Statistical Analyses

Item analyses

Analyses used to determine cut scores for job knowledge tests, assessment center exercises, skills assessments, and physical abilities tests

Adverse impact analyses (4/5ths Rule and chi-square)


Behaviorally-Based Structured Interviews

List of source materials used for content development

List of behavioral dimensions to be assessed in each selection system component with definitions / descriptions of each dimension and behavioral anchors associated with varying levels of performance (e.g., minimally acceptable performance, below minimally acceptable performance, and above minimally acceptable performance)

Structured, behaviorally-based interview questions for each behavioral dimension

Rating forms for each dimension assessed

Voluntary withdrawal form

Job Knowledge Tests

Test plan for job knowledge tests including domains assessed and number of questions for each domain

List of source material used to generate test items

Item documentation (i.e., keyed test items with documented rationales for both correct and incorrect alternatives, citations for each item and alternative)

Cut score development process

Statistical analyses to support use of individual items / tests


Skills Assessments

List of skills assessed linked back to KSAs required at time of job entry

Detailed steps outlining how to perform skill

Subject matter expert ratings of varying performance levels

Behavioral criteria to objectively evaluate performance


Statistical analyses to support use of cut scores

Rating forms

Involuntary withdrawal / disqualification form due to safety concerns / violations


Assessment Centers – In-Basket Exercises

List of KSAs assessed linked back to KSAs required at time of job entry

Storyboards / scenarios for each exercise

Subject matter expert ratings for varying performance levels



In-basket exercise materials (e.g., memos, reports, phone messages, performance reports)



Assessment Centers – Oral Briefing Exercises

List of KSAs assessed linked back to KSAs required at time of job entry

Subject matter expert ratings for varying performance levels



Oral briefing materials (e.g., memos, reports, phone messages, performance reports)


Physical Abilities Test

List of activities assessed in test linked back to KSAs required at time of job entry

Course layout / map of course for participants

Equipment required provided for test (e.g., bunker coat, fire hose)

Subject matter expert ratings of varying performance levels



Statistical analyses to support use of cut scores

Rating forms

List of safety precautions taken before, during, and after administrations (e.g., medical personnel at site, change in start time due to dangerously hot weather, food and water stations throughout course)

Involuntary withdrawal / disqualification form due to safety concerns / violations


Selection System Administration Manual

Each component of the selection system has its own administration manual

Selection System Scoring Manual

Each component of the selection system has its own scoring manual


Training Programs that Support the Selection System Development and Administration Processes

The ability to clearly demonstrate that all relevant personnel have received formal training as needed to complete the selection system development, administration, and/or scoring processes is critical to the defensibility of both the selection system used as well as the administration process. Listed below are the most common training programs provided to organizations,

Cognitive Job Analysis Training Participants include:

Industrial and Organizational Psychologists

Human Resources professionals

Item Writing Training Participants include:


Human Resources professionals

Technical item writers (e.g., education professionals, commercial airline pilots, airframe and powerplant mechanics, radio and electric mechanics)

Behaviorally-Based Structured Interview Training Participants include:

Human resources professionals

Management personnel

Technical interviewers (e.g., commercial airline pilots, airframe and powerplant mechanics, radio and electric mechanics)

Selection System Administration Training A separate manual is created for each component of the selection system. Participants include:


Human Resources personnel

Management personnel

Assessment center administrators and raters


Expert Witness Testimony and Litigation Support Activities

Expert Witness Testimony Cases in which expert witness testimony, reports, and opinions were provided are listed below.

Marion S. Bullington v. United Air Lines, Inc., United States District Court District of Colorado, Case No. 97-WY-240-AJ

Alleged discrimination in the flight officer selection system / process based on age and gender; case included charges of adverse impact and disparate treatment.

David Ebeling v. United Airlines, Inc., United States District Court Western District of Washington, Case No. C97-347C

Alleged discrimination in the flight officer selection system / process based on age; case included charges of adverse impact and disparate treatment.

Litigation Support Litigation support includes reviewing case / court documents, developing case summaries, engaging clients’ in-house and/or trial attorneys in strategy development activities, critiquing opposing experts’ arguments and developing counterarguments, conducting literature searches and compiling relevant literature, compiling data for Discovery, writing direct and cross-examination questions for depositions and trial, conducting statistical analyses, and writing reports. Cases for which litigation support was provided are listed below.

Adams v. Burroughs-Wellcome Company

Adams v. City of Chicago

Adams v. SouthWest Publishing

Bettes v. Jenkens Gilchrist

Brown v. City of Chicago

Carmichael v. Martin Marietta

Gilner v. Saint-Gobain / Norton Industries

Porter v. McKinsey & Co.

Skalka v. Fermco

Slover v. Steris Corporation

Sperling v. Hoffman-LaRoche

United Black Firefighters Association v. City of Akron

Ware v. GCRTA

Woods v. Green Tree Financial Corporation

Wright v. South Central Bell


Sample Job Knowledge Test Items As with all components of a selection system, the constructs assessed (e.g., reading comprehension, mathematical ability) must be factors that are required at the time of job entry. Support for the inclusion of these constructs can be found in the job description for the target position. The following test items were included in a battery of tests for the position of Water Treatment Plant Operator for a city in Lakewood, Ohio. The test items presented below were used as SAMPLE questions for the candidates to review prior to starting the actual test and are no longer in use, thus their presentation below does not, in any way, compromise the integrity of the actual tests. Instructions on how to complete each test are not included here; however, detailed instructions were included in the actual test booklets.

Reading Comprehension

SAMPLE PASSAGE

Regulating every chemical that may appear in drinking water and that theoretically may adversely affect health in some remote circumstances is impractical and irrational. What is needed is some priority determination for regulation so that a reasonable number of contaminants of sufficient concern can be addressed in regulations that will advance the goals of the SDWA and provide definitive guidance to address potential human health effects of exposure to hazardous materials in drinking water. The most relevant considerations in selection of contaminants are (1) the occurrence or potential for occurrence in drinking water, and (2) the potential health risk.

SAMPLE 1: Plant Operator Smith is conducting a series of tests on a sample of water to determine

what, if any, chemicals are present in the water that may adversely affect people’s health. Operator Smith knows that trying to identify and regulate every chemical that may be found in drinking water is:

A. compatible with the goals of the SDWA. B. both irrational and impractical. C. the most relevant consideration in water treatment.

The best answer is alternative B, “both irrational and impractical.” The first sentence in the passage states that “Regulating every chemical that may appear in drinking water and that theoretically may adversely affect health in some remote circumstances is impractical and irrational.”


Mathematical Ability Test SAMPLE 1: A plant operator must determine the amount of a chemical that has been used during the

past two days. Plant records indicate that on Monday 14.60 lbs. of the chemical were used, and on Tuesday 15.70 lbs. of the chemical were used. Based on the information contained in plant records, the plant operator determined that the amount of the chemical used during the past two days is __________ lbs.

A. 29.30 B. 30.30 C. 31.30

The best answer is alternative B, “30.30” because 15.70 plus 14.60 equals 30.30.

USE THE FOLLOWING FORMULAS TO ANSWER SAMPLE 2

FORMULA A: Average Daily Flow = Total Flow Number of Days

FORMULA B: Average Flow, gpm = Total Flow Number of Minutes

SAMPLE 2: A plant operator must determine the average daily flow over a 5-day period. The total

flow, in terms of millions of gallons (MG) during that 5-day period was 21,000,000. Based on this information, the plant operator determined that the average daily flow during that 5-day period was __________ MG.

A. 2,050,000 B. 3,600,000 C. 4.200,000

The best answer is alternative C, “4,200,000.” Because the question specifically asks you to determine the average daily flow, the most appropriate formula to use is FORMULA A. Take the total flow of 21,000,000 and divide that number by the number of days over which the flow rate must be determined; in this case the target time period is 5 days. Thus, the required calculation is 21,000,000 divided by 5 which equals 4,200,000. Therefore, the best answer to SAMPLE 2 is “4,200,000.”



CARBON DIOXIDE (CO2)

FEED RATE

SAMPLE 1: Plant Operator Smith is taking a reading from a carbon dioxide (CO2) tank. The current

reading is __________ units.

A. 7.4 B. 7.6 C. 7.8

The best alternative is A, “7.4.” The shaded area in the instrument represents the amount of the substance (i.e., carbon dioxide) being measured, and each horizontal line or hash mark on the instrument corresponds 0.2 units. The uppermost level of the substance contained in the instrument corresponds with the number 7.4; therefore, the current reading for this instrument is 7.4 units.

1

2

3

4

5

6

7

8

9

10

11

12

13

14


CARBON DIOXIDE (CO2)

FEED RATE

SAMPLE 2: Plant Operator Smith needs to determine whether the amount of carbon dioxide (CO2)

being fed into the water treatment system needs to be adjusted. Operator Smith knows that readings between 2 and 12 units of CO2 are normal, and NO adjustments are needed to the feed rate. If the feed rate is below 2 units of CO2, then the feed rate must be increased, whereas if the feed rate is above 12 units of CO2, then the feed rate must be decreased. Based on the current reading, Operator Smith must __________ the CO2 feed rate.

A. increase B. decrease C. make NO adjustments to

The best alternative is B, “decrease.” The shaded area in the instrument represents the amount of the substance (i.e., carbon dioxide) being measured, and each horizontal line or hash mark on the instrument corresponds 0.2 units. The uppermost level of the substance contained in the instrument corresponds with the number 12.2 which is outside the desired feed rate range of 2 to 12. More specifically, the feed rate is greater than 12. Therefore, the plant operator needs to decrease the feed rate.

1

2

3

4

5

6

7

8

9

10

11

12

13

14


Troubleshooting Test Rules for Reading Diagrams 1. Water always flows from left to right unless indicated otherwise by an arrow. 2. For each question, always assume water is running through the water treatment process. 3. The cause of a problem in the normal water treatment operations will always occur prior to the

problem or earlier in the water treatment process. 4. In measuring rate of flow, you will always be measuring the flow from the component that is

located just prior to the point at which the rate of flow measurement was taken. 5. Each of the components of the water treatment system is labelled on the diagram, and the

numbers that correspond to each of the tasks performed at the water treatment plant are also provided. The diagram also shows where in the process certain chemicals and gases are added to the water. A Water Treatment Plant Operator can adjust the amount of chemicals and gases added to the water at those specified locations.

SAMPLE DIAGRAM

IRON AND MANGANESE TREATMENT

Contact Tank

Aeration Filtration Clearwell

Backwash

Distribution System Backwash Drain

to Waste

Waste Sludge

Groundwater Well

Chemical Oxidant Added

Disinfectant Added

1 1 2 2 2 2


SAMPLE TABLE

IRON AND MANGANESE TREATMENT

Component Number Measure /

Test Control / Adjust

Water Characteristic

Groundwater Well 1 X X Rate of flow

Aeration 2 X Water level

Contact Tank 2 X X Water level

Filtration 1 X X Rate of flow

2 X X Water level

Clearwell 2 X X Water level

SAMPLE 1: Water Treatment Plan Operator Smith is responsible for measuring the rate of flow in

different locations in the iron and manganese process. Based on the SAMPLE TABLE, Operator Smith will measure the rate of flow at __________ different locations.

A. 1 B. 2 C. 4

The best answer to SAMPLE 1 is alternative B, “2.” Based on the information presented in the SAMPLE TABLE, rate of flow is measure at the groundwater well and at filtration.

SAMPLE 2: Water Treatment Plant Operator Smith has measured the water level at aeration and

notices an unusually high water level. Operator Smith concludes that the high water level must be caused by an increase in the:

A. rate of flow between groundwater well and aeration. B. water level in clearwell. C. rate of flow between filtration and clearwell.

The best answer to SAMPLE 2 is alternative A, “rate of flow between groundwater well and aeration.” Based on the information presented in the rules of water flow, SAMPLE DIAGRAM, and SAMPLE TABLE, an increase in water level at aeration must be due to an earlier process or occurrence. Therefore, the only process before aeration that could increase the water level at aeration is an increase in rate of flow between groundwater well and aeration.

selection and assessment in safety-sensitive...

Documents