This panel addresses the assessment tools that should be used, how agencies can implement an assessment strategy that results in the best hires in a timely and cost efficient manner and how agencies can measure and report on the speed and quality of their hiring processes.
TRANSCRIPT
The Path to Improving and Measuring Hiring Results
Today’s Presenters
Beverly Dugan, Ph.D.Vice President, HumRRO
Elaine Pulakos, Ph.D.Chief Operating Officer,
PDRI2
Mike Heil, Ph.D.Assistant Vice
President, Aon Hewitt
Agenda
• Federal Hiring Reform
• How Can Objective Assessments Help Meet These Requirements?
• Use of Objective Assessments in the Hiring Process
• Selecting or Developing Effective Assessments
• Evaluating Hiring Process and Outcomes
• Use of Assessments in Federal Hiring
• Questions and Answers
3
Presenter
Presentation Notes
Today we are going to focus on the measurement aspects of hiring reform. I’ll start with an overview of Hiring Reform, then talk about how to evaluate hiring reform with a focus on hiring efficiency. Elaine will talk about quality of hires, and she will present some very impressive outcomes from organizations that implemented objective assessments. Jasmin and Eric will present an excellent case study that describes how Riverside County transformed their hiring process. They will tell us what they did, how they measured outcomes, and what the results were. The next slide will give you a little preview of their story.
4
Federal Hiring Reform
Presenter
Presentation Notes
As we noted in the introduction, you can look at different types of measures in an evaluation. Two types that we will focus on today are evaluating the hiring process, and evaluating the outcomes of the hiring process. Speed and effectiveness are key to hiring reform, and these are characteristics of the process. I’ll talk about these and then Elaine will talk about evaluating the outcomes of the hiring process.
Federal Government Hiring
• Unprecedented numbers of applicants
• Current systems too confusing and time consuming– Baseline for hiring is 130 days
• Technology helps efficiency but not quality
• Ineffective hiring assessments are “weakest link”– Do not effectively screen and reduce applicant pools – Do not identify best qualified– Unqualified candidates getting through
• Partnership for Public Service, 2010
5
Presenter
Presentation Notes
We’ll start with some well-known facts about Federal hiring. The Federal government will need to fill 600,000 positions over the next 3 years (http://www.govexec.com/dailyfed/0510/051110d1.htm). To hire this many individuals, agencies will need to screen millions of applicants. We all know that the current system is confusing and time consuming. Some applicants are so put-off by the length and complexity of the hiring process that they don’t follow through on their applications – but we don’t know how many don’t apply or drop out mid-way. Technology is part of the solution. Many Federal agencies have implemented technology to help streamline and simplify the hiring process, but one of the points we will make today is that technology alone does not improve a hiring system. Technology itself does not tell you anything about the applicants. Assessments used to screen applicants are critical to identification of strong applicants and successful employees. The Partnership for Public Service called hiring assessments the “weakest link” in the process. Elaine will talk more about effective hiring assessments in a minute.
6
President Obama’s Hiring Reform Memorandum
• Improve hiring process and quality of hires – Eliminate KSA essays – Reduce time to hire mission-critical and commonly
filled positions– Measure speed and effectiveness of hiring process– Measure quality of hires– Analyze causes of hiring problems – Implement actions to reduce problems
Quality, Speed, and Measurement are the Key Requirements of Hiring Reform
Presenter
Presentation Notes
President Obama’s hiring reform initiative attempts to overcome these problems with the hiring system. Key provisions of hiring reform, shown on this slide, include…… It is clear from this list that agencies must make changes to the hiring process, the assessment tools, and also implement measurements that will indicate how well changes to the system are working. We think the requirement to measure the effectiveness of hiring reform implementation is important for several reasons: 1.First, evaluation it is essential to ensure that the objectives are met and the agency gets a return on the time and effort that will be required to modify the hiring program. Measures are key to understanding what modifications are needed to the hiring reform implementation plan 2. Evaluation results provide HR, the “owners” of hiring reform, to tie the results of hiring reform to important organizational outcomes. Here is an opportunity to demonstrate HR’s contribution to operational performance on a large scale. Elaine and Riverside will have specific examples of how this can play out.
7
How Can Objective Assessments Help Meet These Requirements?
Presenter
Presentation Notes
As we noted in the introduction, you can look at different types of measures in an evaluation. Two types that we will focus on today are evaluating the hiring process, and evaluating the outcomes of the hiring process. Speed and effectiveness are key to hiring reform, and these are characteristics of the process. I’ll talk about these and then Elaine will talk about evaluating the outcomes of the hiring process.
Multiple Choice QuestionsWhich response best describes your experience preparing written documents?
1. No experience.
2. In training situations but not on a job.
3. Under supervision, prepared documents in which the content was determined by others.
4. Independently prepared documents in which conclusions were based on my analyses.
5. Independently prepared complex documents that integrated information from many sources to decide on operations or policy.
8
Narratives no longer permitted as first step in hiring process
Why Not Use Self-Report Information?
Applicant Prepared Narratives Preparing written documents
In my previous jobs, I have had a great deal of experience preparing written documents. I served as a research assistant. My primary job duty was to conduct literature reviews and write the results of these. Oftentimes, there were 100 or more research articles that I had to review, some of which contained contradictory findings. I have to look through the articles for common findings and offer explanations base don my analyses as to why different studies may have obtained what appeared to be contradictory results. This required collecting a large amount of information, organizing it in a ….
Presenter
Presentation Notes
Most certs are currently developed by evaluating applicants’ past experience on KSAs – knowledges, skills, and abilities – that are required to perform successfully on the job. Just to clarify, the Presidential memorandum says that the KSA narratives are no longer permitted as a first step in the hiring process. Agencies can still can and should continue collecting information on KSAs or competencies that applicants need to perform the job but this needs to be done using something other than the applicant narratives that describe their past experience on important KSAs. Because narratives are burdensome for applicants, some agencies have already moved away from them and they are instead using multiple choice questions to gather information about applicants’ past experience, like the one shown on the right. Multiple choice questions are a lot more efficient for applicants to complete than narratives, because all they need to do is select the response options than best describe their past experience. Whether an agency is using written narratives or multiple choice questions, however, both of these have a limitation because both are self report assessments. What this means is that we’re relying on applicants to provide accurate information about their KSAs rather than objectively assessing these. The problem with any self-report assessment is that even if people intend to be truthful, some are legends in their own minds and other people don’t say how good they actually are, so we can’t count self-report information to be accurate.
Why Not Use Multiple Choice Self-Reports?
• Often Inaccurate – Some are legends in their
own minds – Others underestimate
themselves
• Can Disadvantage Applicants Who – Lack access to experience – Truthfully report lack of
experience
9
• Time Consuming– Takes a lot of work to develop
good items – Customized for each vacancy – Often yields unwieldy item banks
• Manager Complaints – Do not sufficiently reduce
applicant pools to reasonable number to interview
– Unqualified applicants on certs – Qualified applicants not on certs
Presenter
Presentation Notes
Multiple choice questions are better than relying on resumes alone because although applicants can misrepresent themselves on these, they focus on critical KSAs for the job and information is collected in a structured and standardized manner from all applicants, so it provides more of an apples to apples comparison. However, research and experience with multiple choice self report questions has uncovered a number f problems that limit their effectiveness for applicant screening. We’ve already discussed the fact that we can’t be certain about the accuracy of any self-report information. But in addition, basing hiring decisions on demonstrated past experience on KSAs can actually disadvantage applicants who may not have had access to prior experience but nonetheless possess the skills to perform the job. Someone can have very effective customer service skills without having prior experience in a customer service job. We do not gain much efficiency using multiple choice questions because they are often customized for each and every vacancy, and the back and forth that is required to do this lengthens the hiring time. Finally, experience has shown that multiple choice questions do not effectively reduce large applicant pools to a reasonable number – after applicants are screened using multiple choice questions, there are often too many qualified applicants to interview and managers have had a lot of complaints about the certs they get based on multiple choice questions. So while agencies that use KSA narratives today may be tempted to move to multiple choice KSA assessments, these are not going to help improve the efficiency or effectiveness of the hiring process that everyone is looking for.
Why Not Use Resumes?
• Impractical when there are large numbers of applicants
• Even if resumes could be screened efficiently (e.g., automated tool), they are not effective hiring tools – Also rely on self-report information– Information not standardized across applicants so difficult
to assess
10
Presenter
Presentation Notes
We think agencies are going to be tempted to do one of two things to screen applicants under hiring reform – they will either attempt to make decisions based on reviewing resumes alone or they will use multiple choice self-report assessments like the ones we’ve been discussing. Regarding the use of resumes, it’s difficult and time consuming to review a large number of resumes. And if the job market continues to show weakness and changes are made that make the federal hiring process easier for applicants, the numbers of applicants for government jobs is only likely to grow, which will make resume reviews increasingly impractical as a first step in the hiring process. Even if agencies implement a more efficient way to review resumes, like an automated resume assessment tool, the reality is that the information from resumes alone is not useful for screening applicants and it is less actually much less effective for screening than the information you get from either self-report narratives or multiple choice questions. This is because the information on resumes is completely unstructured and unstandardized from applicant to applicant. Because there is no consistency in the information provided and gaps from person to person, it impossible to accurately assess which applicants are most capable of performing a job by using their resumes alone.
Why Use Objective Assessments?
Objective Assessments: Systematic, accurate measurement of a job-relevant
skill, knowledge area, or other competency
vs.Tell Me Show Me
11
Presenter
Presentation Notes
It’s amazing how many people think that just because something is automated, it’s good. However, one of the things that automating government hiring processes has done is to actually illuminate what a poor job current assessment tools are doing with respect to in identifying quality candidates for a job. In a recent report by the Partnership for Public Service, the poor quality of the assessments the government is using today was discussed extensively -- and recognized by federal HR officers and hiring managers – to the extent that assessments were called out as the weakest link in the hiring process. The Partnership and others have thus correctly concluded that the answer to improving the quality of hires lies in using better assessment procedures. The assessments that have not fared so well are what we are calling “Tell Me” assessment. They either have applicants prepare narratives of their experience and proficiency in KSAs that are required for the job. Or, they use automated systems in which candidates respond to multiple choice questions that likewise ask them about their experience and proficiency in KSAs or competencies that are required for the job. Neither of these types of assessments are particularly good because they rely on self-report information, which is notoriously inaccurate. In order to identify those who are best qualified to perform a job, it is not enough to ask applicants about their self-perceived competency proficiency. Instead, applicants need to be assessed using objective measures of their competencies. It’s the difference between asking someone how effectively they solve problems versus given them a problem to solve and then seeing how well they actually do this. Or, using the example on the slide – the difference between asking applicants how effectively they give briefings versus having applicants give a briefing and seeing how well they actually do this.
12
Use of Objective Assessments in the Hiring Process
Presenter
Presentation Notes
As we noted in the introduction, you can look at different types of measures in an evaluation. Two types that we will focus on today are evaluating the hiring process, and evaluating the outcomes of the hiring process. Speed and effectiveness are key to hiring reform, and these are characteristics of the process. I’ll talk about these and then Elaine will talk about evaluating the outcomes of the hiring process.
Placement of Assessments in Hiring Process
13
Job Posting
Application
Remote Assessments Automated and Unproctored
On-site Proctored Verification Assessments and Interview Assessment
Great Hiring Decision
Presenter
Presentation Notes
When assessing applicants, today it is a best practice to use remote, automated assessments administered over the internet early in the selection process. This approach has a number of advantages: First, the assessments are extremely accessible. They can be taken by a regionally disperse group of applicants when their schedules permit. This allows you to reach more applicants than would have been possible wit proctored assessments. Second, these types of assessments are automatically administered and scored, making the process extremely efficient. Finally, we have found that these assessments yield accurate information about applicants’ capabilities to perform a job. Now, there are a few drawbacks to using unproctored assessments, but there a number of strategies for overcoming these. For example, we don’t know if the person taking the test is actually how he or she says they are or just their smart friend. But the reality is, that same problem exists for the KSA narratives that have been used in the past. To mitigate this, it is a best practice to interview applicant who make it through the initial testing or to administer proctored verification assessment at the time of the interview to verify the applicants standing on the competencies. Another draw back is that when we administer the assessments online in unproctored settings, it is possible for applicants to copy down the items and then place them on Amazon or some other site. We have a number of strategies to mitigate this as well, and as I talk about the various types of assessments that can be used, I will highlight what we do to mitigate cheating and item exposure.
Assessment Administration Options
1. Application-based recruitments, proctored paper-and-pencil assessments per job opening
2. Resume-based recruitments, proctored web-based assessments on a continuous basis
3. Web resume-based recruitments, unproctored web-based assessments on a continuous basis
Cheating, content exposure, and applicant identity all mitigated
with security measures
15
16
Selecting or Developing Effective Assessments
Presenter
Presentation Notes
As we noted in the introduction, you can look at different types of measures in an evaluation. Two types that we will focus on today are evaluating the hiring process, and evaluating the outcomes of the hiring process. Speed and effectiveness are key to hiring reform, and these are characteristics of the process. I’ll talk about these and then Elaine will talk about evaluating the outcomes of the hiring process.
Decide What Needs to be Assessed
• Competencies
– The knowledge, skills, abilities, and other attributes required for effective performance on the job
• Competency Model
– Collection of required competencies in an occupation or organization
17
Presenter
Presentation Notes
I am sure we have all heard of the terms competencies and competency models I the past. But to make sure we are all on the same page when we talk about these concepts, I want to provide you with our definitions. A competency is the collection of knowledge, skills, abilities and other attributes required for effective performance on the job. You can see from this definition, we are not getting rid of the traditional KSAs. We are simply grouping similar KSAs into competencies and defining them slightly differently. On the next slide, I will show you a couple of examples of competencies. A competency model is the collection of required competencies for an occupational series or organization.
18
Auditing CustomerService
Critical Thinking CommunicationProductive
Engagement
ReadingReasoning Math ReasoningProblem Solving Decision Making Planning/EvaluateStrategic Thinking
Self Management FlexibilityStress Tolerance
WritingPresenting Information OrallyPersuading and Influencing
Interpersonal
Example Competencies
Presenter
Presentation Notes
This slide presents some example competencies. The competencies to be assessed can be foundational competencies required across a number of occupational series such as critical thinking, interpersonal skills, or communication. Alternatively, they can be technical in nature and required by a single occupational series or group of occupational series. For example auditing may be assessed for the auditing series or for groups of finical occupations. Competencies can also be broken down into sub-competencies which can also be assessed. For example, there are a number of aspects of communication – writing, oral communication, influencing and persuading. And we may want to assess these when evaluating applicants for positions. Identifying the competencies to be assessed may seem like a lot of work and a departure from KSAs. But there are quite a few resources available that can be leveraged. There have been a number of major competency development efforts in the Federal government such as the OPM MOSAIC models and the vast majority of agencies have developed competency models. All of this content can be used to help identify the competencies to be assessed. We started by reviewing major competency development efforts in the government such as the OPM mosaic model. We cross walked these models and ended up with an integrated model that contained both core competencies such as communication, teamwork, and customer service that were required for effective performance across all jobs, as well as job series specific technical competencies, such as architecture, auditing, budgeting and finance, and so forth.
19
Identify the Most Important Competencies
Administer validation survey to job incumbents
to identify important
competencies
Presenter
Presentation Notes
When we are developing objective competency assessments, we want to focus on assessing those competencies that are the most important for performing the job. So we need some method for identifying the important competencies. To do that, we administer what we call job analysis surveys. During these surveys, incumbents or job experts are asked to rate the importance of each competency and work behavior for their specific jobs. The data are analyzed across respondents to identify the job’s critical work behavior and competency requirements. Those competencies found to be the most important are those that will be the focus of the assessments that are developed. Collecting data of this type has often been an expensive and time consuming process. But there are a number of web-based survey tools such as the one shown here, that can be sued to significantly reduce the time an effort required to collect the data needed to identify the important competencies.
There are a number of different types of assessments that can be used to assess eh competencies. Different assessments are more or less effective for assessing different competencies as shown here. I am going to walk you through these various assessment types so you can see what types of assessments you have available.
21 21
Cognitive Assessments
Measure mental competencies, such as critical thinking, math,
reading, problem solving
To protect against publishing of items,
computer selects different items for
each applicant
Virtually no two applicants receive
same items
Presenter
Presentation Notes
The first type of assessment I want to talk about are computer adaptive assessments. These are typically used to measure cognitive competencies such as critical thinking, math, reading and problem solving. These are some of the most effective assessments we have available for identifying applicants who will likely do well on the job. Assessments of these types of competencies have clearly identifiable right and wrong answers. For example, we know that 2 + 2 = 4. As a result, for a long time, these types of assessments have not been used in an unprotored setting. However, the advent of Computer Adaptive Testing, or CAT, has allowed for their use in unproctored settings. With CAT assessments, you start with a very large pool of items and all applicants are given an initial item that is of approximately the same difficult. However, the next item the applicant receives is based on his or her response to the fist item. If he or she got the first item right, a harder item is administered. If he or she got the first item wrong, an easier item is administered. The computer administers items in this way until we have enough data to accurately evaluate the applicant’s standing on the competency. Because virtually no to applicants receive the same test, no applicant sees all of the items. This protects the security and helps prevent them form being published on amazon.
22 22
Work Styles Assessments
Applicants choose which item is most descriptive of them
Provides assessment of soft skill competencies,
such as self management, initiative, flexibility, teamwork, stress
tolerance, and others
Difficult to cheat because scoring is
not obvious
Presenter
Presentation Notes
The second type of assessment is a work styles assessment. These types of assessments can be used to assess a rage of soft skills competencies such as self management, initiative, flexibility, teamwork, stress tolerance, and others. The applicant is presented with two items, and is asked to select the item in the pair that is most descriptive of him or her. Unlike the CAT assessments I just showed you, the scoring is not obvious – it is not obvious to the applicant what the right answer is. So its very difficult for the applicant to cheat. Now, you may think that it is easy for an applicant to have an inflate perception of themselves or they may answer to present themselves as positively as possible. But there as a great deal of research that suggests that these assessments are predictive of performance even when applicants “fake good.”
23
Realistic problems like encountered on
job
Applicants select most and least effective way to handle situation
Difficult to cheat because scoring is
not obvious
Positive applicant reactions
Situational Judgment Assessments
Presenter
Presentation Notes
The next assessment is a situational judgment assessment. The items in an SJTs present applicants with realistic situations that are similar to those the might encounter on the job. The situations can be presented in writing, in video or in an animated format. Each situation is followed by a series of options for how to respond to the situation and the applicant is asked to select the most and least effective way to respond to the situation. SJTS are slightly different from typical multiple choice tests in that unlike the MC test, which have an obvious right and wrong answer, the answers in the SJT are all plausible – they al represent possible ways of responding. But some are more or less effective than others. Because the scoring is not obvious, it is difficult for the applicant to cheat which allows for their use in unprotorced settings. In addition to being predictive of job performance, Situational Judgment assessments have a number of advantages which make them particularly effective. First, they can be used to measure a range of competencies such as customer service, interpersonal skills and judgment and decision making. Second, because they present realistic situations, they are uniformly well received by applicants – applicants really like these assessments. This is even more true when the situations are video taped or animated. Finally, because they are administered and scored automatically, they do not require assessors to administer and score and can help reduce applicant time.
24
Please describe why you are a good fit for this role, including relevant background experiences, and
what you feel is your greatest strength.
Compose free text response to question
Response automatically scored on clarity, grammar,
persuasion, content, and others
Writing Assessments
Presenter
Presentation Notes
In our interactions with managers, one of the most common complaints we hear is their concern that new hires are not proficient in writing. One thing that can help is to include a computer scored writing assessment in the selection process. This is a really cool assessment that uses some pretty innovative technology. Using these assessments, the applicant is asked to compose a free text response to a question. That response is then automatically and immediately scored by the computer. The computer can actually score any and dimension that a human can score – from grammar and syntax to actual content. This is a far more cost effective and efficient way to evaluate applicant writing than using human assessors. Additionally, the computer is more reliable because it doesn’t get tired or have a bad day. You may me a little skeptical of using computers to score writing. And believe me I was for a very long time too. But we have been effectively using these assessments in government to assess writing and language translation. For those of you who have college aged kids, this is the same technology used to score the writing portion of the SAT and GRE.
25
Realistic problems like encountered on job
Can assess many competencies
Applicant reactions uniformly positive due to
high job relevance
Work Simulation Assessments
Presenter
Presentation Notes
More the assessment looks like the job the more positive applicant reactions tend to be. We have a great deal of exp developing web-based job simulations that can be customized to specific job requirements.
26
Technical Proficiency Assessments
Information Technology
Assessments
MS Office Assessments
Presenter
Presentation Notes
There are number of assessments available for assessing technical proficiencies…. NEED TO ADD MORE HERE
27
Job interviews are also assessments
For reliable and valid assessment:
One set of competency questions
should be used for each vacancy and
asked of all candidates
All candidates should be evaluated against
standard rating criteria
Job Interview Assessments
Presenter
Presentation Notes
Many of you are probably already using interviews as part of your selection process. Interviews, if done right, are actually very effective assessments. However, to be effective, all applicants for a given vacancy should be asked the same set of questions. And those questions should job-related. That is they should assess those competencies that have been identified as required for effective job performance. Finally, all applicants should be evaluated using a common set of rating criteria. If you are in a situation where you are hiring only one or two people a year, the use of a structured interview may be a really effective assessment for you to use.
28
Evaluating Hiring Process and Outcomes
Presenter
Presentation Notes
As we noted in the introduction, you can look at different types of measures in an evaluation. Two types that we will focus on today are evaluating the hiring process, and evaluating the outcomes of the hiring process. Speed and effectiveness are key to hiring reform, and these are characteristics of the process. I’ll talk about these and then Elaine will talk about evaluating the outcomes of the hiring process.
Getting Started
• Define measures– Hiring reform requirements– Agency strategic goals– Most critical impact areas (reduce costs, reduce turnover, etc.)
• Determine measurement strategy for stable, credible results– Collect baseline measures before changing your process – Collect measures at regular intervals during and after
implementation
29
Presenter
Presentation Notes
So how do you go about evaluating hiring reform? A solid evaluation goes beyond completing reporting requirements as a “check the box” exercise. A solid evaluation of hiring reform starts with defining the measures that assess: the extent to which hiring reform requirements are met the extent to which hiring reform contributes to accomplishment of strategic goals the extent to which hiring reform has an impact on critical outcomes, such as reducing costs or turnover. Once you have defined measures, it’s important to design a data collection schedule so that you can detect change as a result of hiring reform. For example, you will want to collect baseline measures before changing the process, so that you know the level of cost or turnover before and after implementation. You also want to think about the schedule of data collection to take into account variations in the system, such as peaks and valleys in hiring volume. Such variations indicate that you need to collect data at regular intervals so that you account for fluctuations.
Hiring Process Evaluation
• E2E metrics focus on speed and effectiveness of hiring– % employees hired within 80 days– % applicants given information on status of
application– % new employees reporting satisfaction with
hiring process– % new employees reporting regular
communication during hiring process• Other example hiring process metrics
– Total time HR devotes to hiring process– Total time manager devotes to hiring process– Manager satisfaction with hiring process
30
• Process measures answer the question: How well is the new hiring process working?
Presenter
Presentation Notes
Process measures are essential indicators of how well a new process is working. These measures provide critical feedback to implementers that indicate where system changes are not achieving their intended effects. Without such data, agencies will not know where the implementation needs to be modified to achieve its objectives. OPM has developed metrics for the E2E initiative that apply to all the steps required to bring employees into the system…... The metrics that apply to the hiring function are shown here…….as you can see, they track the % of employees hired within 80 days, communications about the applicant’s status, and satisfaction with the hiring process. Other examples of process measures are shown at the bottom of this slide. These measures provide feedback to implementers. For example, if the agency’s goal is to hire 60% of employees within 80 days and the evaluation shows that the agency is hiring 40% of employees within 80 days, that finding signals that something is not working. Further examination of the process may be needed to figure out what should be changed to achieve the goal. These measures assess the process itself, not the quality of hires or other outcomes associated with quality of hire.
Improving the Hiring Process
• Technology is key “Many problems agencies face are information technology-related,” said Tim McManus, vice president of education and outreach at the nonprofit Partnership for Public Service. For example, agencies have in place automated systems to sort through knowledge, skills and abilities essays, but have not yet developed assessments based on shorter questionnaires and specific positions and qualifications.
31
Excerpt from: Hiring reform deadline creates some headaches for agenciesBy Emily Long [email protected], September 16, 2010
“How do you assess in a way that's automated?” he said. “It's hugely important in this conversation because of the volume of applicants agencies get -- hundreds and even thousands of applications for a single position.”
Justice, Treasury and Veterans Affairs departments all identified delays caused by limited or antiquated IT systems.
Presenter
Presentation Notes
Attempts to streamline the hiring process pre-date hiring reform. Agencies have been working on streamlining the hiring process for several years, and frequently the chosen strategy involves technology. Technology is key, given the volumes of applicants that Federal agencies must process. Technology solutions will get even more attention with hiring reform, as the quote from Tim McManus suggests. He points out that “…agencies have in place automated systems to sort through knowledge, skills and abilities essays, but have not yet developed assessments based on shorter questionnaires and specific positions and qualifications.” Agencies will need to acquire or retrofit technology to administer the new assessments required by hiring reform.
• Automated tools enable applicants to apply anywhere, anytime
• Easy, efficient process attracts a larger number of qualified applicants
• Real time, immediate category scores– Not Qualified, Qualified, or Best
Qualified
32
Presenter
Presentation Notes
Automated tools can take over much of the process, including receipt of applications, tracking large numbers of applicants, and administering and scoring assessments. I won’t steal Riverside County’s thunder, but they will give you real-life examples of how Technology-delivered assessments enable applicants to apply anywhere, anytime, and How an easy, efficient process attracts large numbers of applicants
Caveat from MSPB
• “Plan carefully when implementing automated tools designed to support the hiring process. Do not assume that technology alone will fix already-broken processes. For instance, automating an assessment tool that is a low predictor of job success will not improve the tool’s predictive ability.”
33
Excerpted from: MSPB report “Reforming federal hiring – beyond faster and cheaper” –September 1, 2006
Presenter
Presentation Notes
But we end this section on a cautionary note: we have worked with many agencies that expected technology to solve all of the problems in the h irning process. The Merit Systems Protection Board published a report about 4 years ago that reviewed agency hiring systems and spoke very eloquently about this problem…..”Do not assume that technology alone will fix already-broken processes. For instance, automating an assessment tool that is a low predictor of job success will not improve the tool’s predictive ability.” This is similar to the garbage/in garbage/out problem that often afflicts technology solutions. Elaine will now pick up on this theme as she talks about quality of hire and how that can impact important organizational outcomes.
These are some of the organizational impacts that a robust hiring program can have. The private sector provides abundant examples of such outcomes associated with a strong hiring program – which is one that uses the right technology solution coupled with effective, objective assessments.
Example Results from Private Sector
35
Presenter
Presentation Notes
Here are examples of outcomes from three private sector organizations that should be equally important for government agencies. Use of objective assessments substantially decreased training time at AirTran, substantially increased customer satisfaction at BJ’s, and substantially decreased the interview to hire ratio at Blue Cross/Clue Shield of North Carolina.
36
34% more reach quality goals
$4 million realized from improved work efficiency
18 days of additional productivity per employee annually
35% reduced average handle time
43% more likely to pass exam
8% more calls completed
40% reduction in handle time18% more effective and 28% more likely to hire again
Call center agents
Logistics staff
At home agents
Call center agents
Claims adjustors
Insurance agents
Call center agents
Claims adjustors
EFFICIENCY AND QUALITY OUTCOMES
Increased Efficiency and Quality
Presenter
Presentation Notes
Now let’s take a look at more actual results that organizations achieved from implementing effective, objective assessments. These are all outcomes that reflect improved quality and efficiency. While these are from private sector studies, several of them relate to key mission requirements of federal agencies. For example, measures of handle time are important to agencies that process benefits claims, tax forms, and passport requests. Pass rates on exams are directly relevant to agencies that use tests to certify employee proficiency. Ratings of effectiveness and likely to hire again are measures of manager perceptions of workforce quality that can be used by any agency. Incidentally – the reason why our example results are all from the private sector is because objective assessments are used extensively for hiring in the private sector but they are just now starting to get interest and some traction in the federal government, so there should be similar results to report from the federal government at some point.
37
46% reduced turnover
$2 million saved in reduced repeat service calls
23% reduced turnover
33% reduced turnover
83% reduced turnover
78% million in reduced shrinkage
50% reduced turnover
63% reduced turnover
Nurses
Technicians
Call Center Agents
Reservations Agents
Call Center Agents
Store Employees
Production Workers
Call Center Agents
COST REDUCTION OUTCOMES
Reduced Costs
Presenter
Presentation Notes
We all know that the costs of turnover are significant. A number of factors affect employee turnover, and one of them is the fit between the employee and the job requirements. Employees who are more capable of performing a job and are a good fit for the job are more likely to stay, so the use of objective assessments is also frequently associated with reduced turnover.
Improved Leadership
38
300% higher performance ratings
57% more likely to be promoted
8% higher sales and profit
59% higher employee engagement
30% more likely to be promoted
Frontline Managers
Branch Managers
Territory Managers
Branch Managers
Senior Managers
LEADERSHIPEFFECTIVENESS
OUTCOMES
Presenter
Presentation Notes
A recent report by the MSPB, titled A Call to Action: Improving First-Level Supervision of Federal Employees, concluded that the Federal government needs to improve the selection of supervisors. The report found that current selection of first-level supervisors emphasizes technical expertise, not interest in and aptitude for leadership. More effective leaders – who can be identified by objective assessments -- can have a large impact on important organizational outcomes, as these results show. And now we’re going to let you hear first hand how Riverside County used both technology and effective assessments to implement a hiring program that yielded very significant effeciency and quality of hire outcomes.
Effective Hiring Outcomes Recap
39
Effective Work Behaviors
Fit
Hires with job-relatedabilities, skills
and traits
Knowledge
Skills
Work Styles
Competencies
ObjectiveAssessments
Turnover
Productivity
Engagement
Cost Savings
ManagerSatisfaction
Quality of Hire
Show Me vs. Tell Me
Process Effectiveness
and Speed
Time to Hire
Applicant Satisfaction
Applicant Communication
Hiring System
TechnologyImprovements
ProcessImprovements
Presenter
Presentation Notes
On this slide, then, at the top we show that the right technology is important for improving important hiring system outcomes like time to hire and applicant satisfaction. Then on the bottom, we show the important quality of hire outcomes that objective assessments have been shown to yield – things like increased productivity, decreased turnover, increased employee engagement, and others that I wil show you more examples of next.
40
Use of Objective Assessments in Federal Hiring
Federal Agencies Using Objective Assessments
• ATF • DEA• Department of Commerce• Department of State• DoD – uniformed services• DHS - CBP• FAA• FBI• IRS• Intelligence Community• OPM• Postal Service• TSA• Veterans Benefit Administration
41
Federal Agencies Using Unproctored Assessment
• Intelligence Community– Assessments include writing, foreign language, work styles,
computer skills, English skills, math, proofreading, typing speed and accuracy, among others
• OPM– Assessments include reading, reasoning, math reasoning, work
styles, and situational judgment
• U.S. Postal Service – Assessments including situational judgment and personal
characteristics
42
Q & A
Questions
Why Objective Assessments “Aren’t for Me”
• I’m only hiring one person for one position– But …agency may be hiring hundreds or thousands– Competency assessments (e.g., writing) useful across jobs– Can get thousands of applicants for one job – how do you find
the ONE?
• Too complex, time consuming, expensive – Quality assessments available from OPM – Commercial assessments available– Usually easier to implement competency assessments than self
report experience items
44
Presenter
Presentation Notes
Objective competency assessments have not been used as much for federal hiring as they have been in the private sector, although several agencies do use them. There are several reasons for this that boil down to various misperceptions that people have about objective assessments. For example, each federal vacancy tends to be viewed as unique and separate, and people think they need a highly customized assessment to get someone with the unique and specific experience that is needed for that one position. It would not make sense or be economically feasible to develop different objective assessments for each and every federal position that is filled. However, the reality is that the same basic competencies are required for many, many jobs. For example, if we think about writing or customer service skills, or critical thinking skills – these competencies are needed for many jobs. Jobs within a job series have many more competencies in common than they do unique requirements. So if we focus on what’s common across jobs within a series, which is the lion’s share of what’s required to perform them, competency assessments provide a viable, efficient, and effective approach for screening many applicants on the common and basic capabilities they need to do the job. Once you know applicants possess the core capabilities to perform a job, it them makes sense to look for the unique experience or knowledge that you may be looking for in a specific position. Another myth is that people think that objective competency assessments are more complex, time consuming, and expensive than self-report KSA experience assessments. High quality objective competency assessments will be available from OPM, as Andrea described, at no cost. Agencies can also purchase commercial assessments or develop their own competency assessments for their specific technical knowledge or skill requirements. While investments will need to be made to do this, we’ve already pointed out that developing multiple choice self-report items take a great deal of time and back and forth to capture what the manager is uniquely looking for, and this is done over again for every vacancy, which translates into both time but costs. Once a competency assessment is developed, it can be used over and over again without investing additional time or costs. So, objective competency assessments actually end up being the most efficient and effective to screen applicants.
How Does Category Rating Work?
• Purpose– Increases number of qualified applicants from which to select– Provides flexibility to select without regard to “rule of three”– Preserves Veterans preference rights
• Process– Evaluate minimally qualified applicants – Place all minimally qualified applicants into two or more pre-
determined quality categories– Combine quality categories if fewer than three in highest category – Send names of all in highest category to hiring official – Make selections form highest quality category
45
How Does Veterans Preference Work Under Category Rating?
• Applied after assessment and category placement• No longer assign points• Preference eligibles receive absolute preference
– Listed ahead of non-preference eligibles in the category
• Preference eligibles with compensable service-connected disability of 10% must be placed in highest category (except for scientific or professional positions GS-9 and above)
• Agencies must request to pass over a preference eligible
