understanding the statistical model - futurework systems...statistical model in the negotiations...

Understanding the

Statistical Model

Presented by the U.S. Department of Labor

Employment and Training Administration

Office of Policy Development and Research

And Its Use in Negotiations

Today’s Objectives

3/23/2017

U.S. Department of Labor, Employment and Training Administration

U.S. Department of Labor, Employment and Training Administration 2

Understanding the Statistical Adjustment Model

The WIOA statute requires the use of a Statistical Adjustment Model in

establishing performance goals for core programs.

This workshop will help you learn about:

The design of the statistical regression model

Its use in establishing performance metrics

Its application at the national level

Its practical use at the state level.

Today’s Agenda

3/23/2017



Overview of the requirement to use a Statistical Model

Discuss the phased-in approach to the Model and baseline indicators

Present the elements of the Model and the variables used

Explore the process of using the model to Negotiate Levels of

Performance

Explore the Model’s role in Determination of Performance Failure

Explore next steps in the development and use of the Model

Quick Poll

3/23/2017 U.S. Department of Labor, Employment and Training Administration 4

What is your level of involvement with the Statistical

Adjustment Model (SAM) and negotiations?

a. Integral

b. Minimal

c. Huh, who is SAM?

The Regulations

3/23/2017



Workforce Innovation and Opportunity Act; Joint Rule for Unified and

Combined State Plans, Performance Accountability, and the One-

Stop System Joint Provisions

https://www.gpo.gov/fdsys/pkg/FR-2016-08-19/pdf/2016-15977.pdf

https://www.gpo.gov/fdsys/pkg/FR-2016-08-19/pdf/2016-15977.pdf

Guidance

3/23/2017



TEGL 26-15 Negotiating Performance Goals for the Workforce

Innovation and Opportunity Act (WIOA) Title I Programs and the

Wagner-Peyser Employment Service as amended by Title III, for

Program Year (PY) 2016 and PY 2017

https://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=9363

TEGL 10-16 Performance Accountability Guidance for Workforce

Innovation and Opportunity Act (WIOA) Title I, Title II, Title III and Title IV

Core Programs




Today’s Agenda

3/23/2017







Performance



Overview of

Requirements

3/23/2017



WIOA 116(b)(3)(A) Performance Accountability requires the use of the

statistical model in the negotiations process: (v) Consideration of four factors when negotiating levels of

performance for each indicator;

(viii) The model is used to make adjustments to the negotiated State

levels of performance; and

(iv) The model is to be used in negotiations for the first two program

years (PY2016 and PY2017).

Negotiations

Requirements

3/23/2017



The negotiated levels for each indicator are based upon:

Comparison with performance levels of other States;

The results of an objective statistical model;

Continuous improvement; and

The extent to which performance levels assist in reaching the

GPRA goals

WIOA specifically requires use of the statistical model in negotiations for

the first two program years (PY2016 and PY2017).

Statistical Model

Requirements

3/23/2017



The Model must consider States’ actual economic conditions, including;

Differences in Unemployment Rate

Differences in job losses or gains in particular industries

The Model must also consider the characteristics of participants at the

time of enrollment;

Indicators of poor work history

Lack of work experience

Lack of educational or

occupational skills attainment

Dislocation from high-wage or

high-benefit employment

Low levels of literacy or English

proficiency

Disability status

Homelessness

Ex-offender status

Welfare dependency

Adjustment

Requirements

3/23/2017



Is this my final answer?

The model must be used on the back end to

adjust the negotiated State levels of

performance for:

Actual participant characteristics, and

Actual economic conditions

Today’s Agenda

3/23/2017







Performance



Phase-In of the

Statistical Adjustment Model

3/23/2017



As past outcomes and other labor market data becomes available, the

model will be updated over time with new data.

The original model was used in the negotiation of PY2016 and PY2017

goals for indicators where data was available.

The revised model (once updated with recent data) will be used to

re-negotiate PY2017 levels of performance, if a state opts to re-open

negotiations.

Generally, indicators for Titles I and III will be estimated in the model,

where data through PY 2015 is available, by the spring of 2017.

Data Availability and

Indicator Estimates

3/23/2017



Indicators are categorized in the model based upon when data are

available.

Those indicators where there are insufficient data yet to support the

statistical model have been designated as Baseline Indicators.

PY2018 is the first year that the Baseline Indicators will be estimated

using the statistical adjustment model.

For PY2017 re-negotiations, these indicators will remain incompletely

populated with data, because some data is not yet available.

Baseline Indicators

3/23/2017



Baseline Indicators, by core program title, include:

For Title I:

Measurable Skill Gains,

Effectiveness in Serving Employers,

Median Earnings (Youth program only)

For Title II:

All primary indicators (EXCEPT Measurable Skill Gains)

For Title III:

Effectiveness in Serving Employers

For Title IV:

All primary indicators

Today’s Agenda

3/23/2017







Performance



Elements of the Model

3/23/2017



The model is required to include two elements:

Participant Characteristics – variables are presented

as percentages of the total exiters for the particular

cohort.

Economic Conditions – variables are presented as

percentages of total employment level by industrial

sector and the State unemployment rate.

Elements of the Model (cont’d)

3/23/2017



Variables representing Participant Characteristics rely on existing

WIA data

Workforce Investment Act Standardized Record Data

(WIASRD) from PY 2005 through 2014 was used in the

development of the model and was the basis for PY 2016

and 2017 negotiations

The model is being updated with the PY 2015 WIASRD and

will be available for those States that wish to renegotiate

PY 2017 levels of performance


3/23/2017



Gender

Age

Race/Ethnicity

Educational Level

Employment Status at

Participation

For reference, details of the variables can be found here: be found here:

https://www.doleta.gov/performance/guidance/docs/WIOA_St

atistical_Model_Methodology_Report-6-24-2016.pdf

https://wdr.doleta.gov/directives/attach/TEGL/TEGL_26-15-

Attachment-II_Acc.pdf

In addition to required elements, other relevant characteristics of

participants served are captured, including:

https://www.doleta.gov/performance/guidance/docs/WIOA_Statistical_Model_Methodology_Report-6-24-2016.pdf

https://www.doleta.gov/performance/guidance/docs/WIOA_Statistical_Model_Methodology_Report-6-24-2016.pdf

https://wdr.doleta.gov/directives/attach/TEGL/TEGL_26-15-Attachment-II_Acc.pdf

https://wdr.doleta.gov/directives/attach/TEGL/TEGL_26-15-Attachment-II_Acc.pdf


3/23/2017



For reference, details of these sources can be found here: be found here:

https://www.bls.gov/cew/

https://www.bls.gov/lau/

Variables representing Economic Conditions rely on two datasets

provided by the Bureau of Labor Statistics (BLS):

Quarterly Census of Employment and Wages (QCEW) – used

to calculate the percentage of total employment for each

industrial sector within a State

Local Area Unemployment Statistics (LAUS) – used to

calculate the State Unemployment Rate

https://www.bls.gov/cew/

https://www.bls.gov/lau/

Today’s Agenda

3/23/2017







Performance



Negotiated Levels of

Performance

3/23/2017



One part of the negotiations process

WIOA Sec. 116(b)(3)(A)(v) requires consideration of 4 factors in

the negotiation process.

comparison of levels of performance for other States,

the results of an objective statistical model,

continuous improvement in performance accountability by the

State and

the extent to which State performance levels will assist in

reaching GPRA goals.


Performance

3/23/2017



Using the Performance Negotiations Tool

(Titles I and III)

The tool is intended for States and their Local Areas.

The tool provides users with benchmarks stemming from the

statistical adjustment model mandated in statute along with

a review of historical performance.

These benchmarks are intended to inform the negotiations

process between States and their Local Areas.

The tool is a starting point in the negotiations process.


Performance

3/23/2017



Components of the Statistical Model Tool:

Target Outcome

the level at which states are expected to be as indicated by the statistical adjusted model (i.e., the 'national' model).

Predicted Outcome

The level at which states will most likely be as indicated by the statistical model (i.e., the fixed effects model).

Simulated Outcome

The simulated outcomes for the States or Local Area’s prior program year use the individual records for the last program year of each indicator's exiter cohort found in the Wagner-Peyser Employment Service data.


Performance

3/23/2017



Target Outcome Calculation: 1. Multiply each coefficient by the associated input value.

• For example, for Alabama’s Adult Employment Rate in the 2nd

Quarter After Exit, the coefficient for female exiters is 0.05271.

• The result is multiplied by the percent of female exiters for

Alabama which is 0.58237. The product of multiplying these two

numbers is 0.03070.

• This step is repeated for every coefficient and associated input

value.

2. Add up all of the products from Step 1.

• For the Adult Employment Rate in the 2nd Quarter After Exit, the

sum of these products for Alabama is 0.18964.


Performance

3/23/2017



Target Outcome Calculation: (continued)

3. Calculate the average State fixed effect by adding all of the State

fixed effects for the indicator and dividing that sum by the number of

States (52).

• For Alabama the Adult Employment Rate in the 2nd Quarter

After Exit, the average State fixed effect is 0.56105.

4. Add the average State fixed effect from Step 3 (0.56105) to the sum

in Step 2 (0.18964) for a target Adult Employment Rate in the 2nd

Quarter After Exit of 0.75069r 75.1%.


Performance

3/23/2017



Predicted Outcome Calculation: 1. The predicted outcomes are calculated using Steps 1, 2 and 3 for the

Targeted Outcome.

2. In Step 4, instead of adding the average State effect, the specific State effect

is added to get a prediction which accounts for the effect of the State

program.

Simulated Outcome: 1. The simulated outcomes for the States or Local Area’s prior program year are

calculated using the individual records for the last program year of each

indicator's exiter cohort found in the Wagner-Peyser Employment Service

data.

1. effects of the specific State programs as well as other factors


Performance

3/23/2017



Key Elements to be considered in the negotiation process:

Trend analysis/Past Performance

Interpret the range between projected outcome and projected

target

Adjusting some of the model inputs

Comparison across States

Continuous improvement

GPRA Goals

Reaching consensus on a negotiated target

Scenario: Additional

Environmental Factors & Low

Baselines

In one situation, the state wanted to increase its focus on serving a

particular population, in this case TANF.

DOL utilized the statistical model to show what the impact would be

on focusing on TANF by inputting a 200% increase in number of TANF

recipients served, which ultimately indicated minimal impact on

overall performance.

The state asked DOL to consider setting a lower EER goal. They cited

both the uncertainty of working with multiple partners with whom they

had no history and the unknown with regards to how this change

would impact their ability to meet enrollment goals and achieving

positive outcomes.

Today’s Agenda

3/23/2017







Performance



Definition of Terms

3/23/2017



Expected Levels of Performance – provided by States in State Plans

Negotiated Levels of Performance – agreed upon between States

and DOL or ED

Actual Results – Reported in annual Statewide Performance Report

ETA-9169

Adjusted Levels of Performance – calculated after the close of the

program year

Adjusted Levels of

Performance

3/23/2017



Initial Target Outcomes

Calculated using the statistical adjustment model

Provided by DOL prior to the start of the program year

*image from TEGL 26-15 Attachment III-B

WIOA Performance Negotiations Tool

Adjusted Levels of

Performance (cont’d)

3/23/2017



Post-Program Year Target Outcomes

Uses the same model as the initial target outcomes

Calculated using actual participant characteristics and

economic conditions

Provided by DOL after the close of the program year

Adjusted Levels of

Performance (cont’d)

3/23/2017



Adjusted Levels of Performance – for each indicator of performance:

Subtract the Initial Target Outcome from the Post-Program Year

Target Outcome

Add the difference to the Negotiated Level of Performance

Adjusted Level of Performance

(Set after the close of a Program Year)

Negotiated

Level of

Performance +

Difference

between first

and second

SAM outputs =

Adjusted

Level of

Performance

76.0% -1.3% 74.7%

Performance Scores

3/23/2017



For each primary indicator of performance:

Calculation includes the numerator and denominator

(except in the case of median earnings)

Actual performance is divided by the adjusted level of

performance

Actual

Performance

Adjusted Level of

Performance

Performance

Score

20,00030,000

74.7%= 89.2%

For each WIOA title program and Primary Indicator of Performance:

The average performance score is calculated

This calculation has been outlined in Attachment 9 of TEGL 10-16

Average performance scores will be truncated to one decimal place

Average Performance

Scores

3/23/2017

15,90024,00075.2%

+

13,00018,00072.5%

+

20,00030,00074.7%

+

9,00015,00079.7%

+

6,00012,00068.7%

+

31,55542,00075.1%

6

= .89984 = 89.9%

Performance Scores

Average

Indicator Score


Determination of Failure

3/23/2017



Performance Failure occurs when:

Any single Performance Score is less than 50%, or

Any Average Performance Score is less than 90%

Sanctions are applied when:

The State fails to submit a complete and timely annual report, or

Any single Performance Score is less than 50% for 2 consecutive

years, or

Any Average Performance score is less than 90% for 2

consecutive years.

Today’s Agenda

3/23/2017







Performance



Next Steps

3/23/2017



Brief Overview of Feedback

Revisions to the Model

Challenges to the Local Level Model

Next Steps – Feedback

and Possible Revisions

3/23/2017



DOL has completed PY16 and PY17 negotiations for some indicators

where the data was available.

We have received feedback on a range of topics from variable

definitions, clustering, dissemination of the model, and its application

to the local board negotiations tool.

DOL will evaluate the accuracy of the model and make adjustments

Likely revisions include:

Industrial sector groupings

Adding in variable for pre-program wages

Next Steps – Challenges

of the Local Level Model

3/23/2017



WIOA requires States use the statistical adjustment model when

negotiating levels of performance with local boards.

WIOA requires States use the statistical adjustment model to adjust

local levels of performance to reflect the actual economic conditions

experienced and the characteristics of the participants served.

There are two challenges to overcome:

Local area codes change often, which makes curating a dataset

comprised of historical data difficult; and

QCEW and LAUS data is not available at the local level. County-

level LMI data must be aggregated up to the local level, which is

a resource-intensive process.

Any Questions?

3/23/2017



Resources

3/23/2017



Body - Insert text here

Bullet list one

Bullet list two

Bullet list three

Insert more text here

Contact Information

3/23/2017



For questions regarding the information in presentation, send an email to*:

[email protected]

*Please use [INSERT TRACK TITLE HERE] as the subject line.

mailto:[email protected]

3/23/2017



understanding the statistical model - futurework systems...statistical model in the negotiations...

Documents