nine year education gains: project quest’s impact on ... · the economic mobility corporation...

Nine Year Education Gains: Project QUEST’s Impact on Student Success

Anne Roder and Mark ElliottEconomic Mobility Corporation

June 2020

The Economic Mobility Corporation (Mobility) identifies, develops, and evaluates programs and policies that enable disadvantaged individuals to acquire the education, skills, and networks needed to succeed in the labor market so that they can support themselves and their families.

Board of Directors

Cynthia Shoss ChairPlinio AyalaHarry J. HolzerMary PenaRuss PomeranzMark Elliott

Mail:Economic Mobility Corporation, Inc.50 Broadway, Suite 1604New York, NY 10004Telephone:212.280.6975Email:[email protected]


http://economicmobilitycorp.org/

mailto:info%40economicmobilitycorp.org?subject=

Acknowledgments

Funding for the continuation of the Project QUEST evaluation came

from Arnold Ventures. We thank the Evidence-Based Policy team

for its support of this effort, particularly Jon Baron and Amanda

Moderson-Kox. We are also grateful to Howard Rolston for help-

ing make this happen. Support for the initial six-year study came

principally from the Charles Stewart Mott Foundation, with addi-

tional support from the Annie E. Casey Foundation and the US

Department of Health and Human Services through a contract with

Abt Associates.

We thank the Project QUEST staff, participants, and COPS and

Metro Alliance leaders for taking part in the study.

Many people contributed to this report. We thank Heath Prince and

the staff at the Ray Marshall Center at The University of Texas at

Austin for working with us. Ashweeta Patnaik, Greg Cumpton, and

Patty Rodriguez accessed and expertly analyzed the data for this

report. Harry Holzer of Georgetown University provided useful com-

ments on a draft of the report. Caitlin van Dusen edited the report,

and Penelope Malish designed the publication.


Table of Contents

1 Summary

3 Introduction

5 The Study Participants and Targeted Postsecondary Programs

7 Project QUEST’s Impacts on Postsecondary Educational Attainment

14 Comparison of NSC, Project QUEST, and Survey Data

15 Project QUEST’s Impact on Unemployment Benefits Receipt

16 Comparison of Benefits and Costs

18 Conclustions

20 Endnotes

21 Appendix

Figures1 Figure S1: Cumulative Percent Who Earned At Least One NSC-Covered Postsecondary Credential through Each Year

after Random Assignment

2 Figure S2: Project QUEST’s Impact on Average Annual Earnings during Each Year after Random Assignment (N=410)

4 Figure 1: Average Annual Earnings during the Nine Years after Random Assignment among All Study Participants

6 Figure 2: Study Participants’ Intended Health-Care Occupation at Program Entry (N=410)

7 Figure 3: Percent Who Ever Enrolled in NSC-Covered Postsecondary Education during the Nine Years after Random Assignment

8 Figure 4: Percent Enrolled in NSC-Covered Postsecondary Education during Each Year after Random Assignment

8 Figure 5: Percent Enrolled Full Time in NSC-Covered Postsecondary Education during Each Year after Random Assignment

9 Figure 6: Percent of QUEST Participants Who Received Support from Project QUEST during Each Year after Random Assignment

10 Figure 7: Percent of Study Participants Who Earned NSC-Covered Postsecondary Credentials during the Nine Years after Random Assignment

11 Figure 8: Cumulative Percent Who Earned At Least One NSC-Covered Postsecondary Credential through Each Year after Random Assignment

12 Figure 9: Percent of Study Participants Who Earned NSC-Covered Health-Care Credentials during the Nine Years after Random Assignment


13 Figure 10: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Gender

13 Figure 11: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Age

13 Figure 12: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Ethnicity

13 Figure 13: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Education

14 Figure 14: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Marital Status

14 Figure 15: Project QUEST’s Impact on the Likelihood of Earning an NSC-Covered College Credential, by Parenting Status

16 Figure 16: Percent of All Study Participants Who Received Unemployment Benefits during Each Year after Random Assignment

16 Figure 17: Project QUEST’s Costs

23 Figure A1: Average Annual Earnings during the Nine Years after Random Assignment among All Study Participants (2017 Dollars)

23 Figure A2: Project QUEST’s Impact on Average Annual Earnings during Each Year after Random Assignment (2017 Dollars; N=410)

Tables6 Table 1: Study Participants’ Characteristics at Program Entry (N=410)

7 Table 2: Program Requirements and Credentials Earned for Project QUEST’s Targeted Health-Care Career Tracks

9 Table 3: Average Number of Months Participants Received Support from Project QUEST by Program Track

17 Table 4: Comparison of Average Net Earnings Benefits to Net Costs during the Nine Years after Random Assignment (2017 Dollars)

22 Table A1: Comparison of the Baseline Characteristics of Treatment and Control Group Members in the Project QUEST Randomized Controlled Trial

24 Table A2: Analysis of Non-Response Bias in the Six-Year Follow-Up Survey


SummaryProject QUEST, located in San Antonio, Texas, provides comprehensive support to help low-income adults earn postsecondary educational credentials and access well-paying jobs in strong sectors of the local economy. Mobility conducted a ran-domized controlled trial (RCT) study of Project QUEST to assess its impacts on credential attainment and earnings among individuals pursuing health-care careers. With support from Arnold Ventures, we contracted with the Ray Marshall Center at The University of Texas at Austin to examine impacts nine years after study enroll-ment, using state administrative data on earnings and data from the National Student Clearinghouse (NSC) on postsecondary educational outcomes. Our report, Nine Year Gains: Project QUEST’s Continuing Impact, documented that the program had the largest sustained earnings impacts ever found in a US workforce develop-ment program. In this report, we present Project QUEST’s impacts on postsecondary educational outcomes and compare the program’s benefits and costs.

We found that Project QUEST achieved large, sustained educational impacts over the nine years of the study. Specifically, Project QUEST:

• doubled credential attainment after three years.

• increased the percentage of participants earning a credential by 16.4 percent-age points by the end of the ninth year.

• more than doubled the attainment of a health-care credential.

As presented in Figure S1, Project QUEST’s impact on credential attainment grew over the nine years after study enrollment, indicating that Project QUEST set par-ticipants on a path to completing postsecondary education long after they stopped receiving program support. These results almost certainly underestimate Project QUEST’s impact, because the NSC data did not capture one of the primary health-care certificate programs that Project QUEST supported during the study.

Figure S1 Cumulative Percent Who Earned At Least One NSC-Covered Postsecondary Credential through Each Year after Random Assignment

Year 1 Year 2 Year 3*** Year 4*** Year 6**Year 5*** Year 7*** Year 8*** Year 9***0%

10%

20%

30%

40%

50%

60%

■ QUEST group (N=207) ■ Control group (N=203)

Statistical signi�cance levels: *** p<.01; ** p<.05; * p<.10

0.9% 2.1%

10.9%

20.7%

28.1%31.3% 32.5%

34.7%

38.8%40.2%

6.8%

10.6%

15.5%18.4%

21.8% 23.0% 23.3% 23.8%

1 Nine Year Education Gains: Project QUEST’s Impact on Student Success

As we previously reported, QUEST participants earned substantially more than control group members in the third through ninth years after random assignment (Figure S2). Project QUEST’s impact on annual earnings grew to $5,239 in year nine—a difference that was both statistically significant and the largest of any year.

The Project QUEST model includes comprehensive support to help participants earn credentials and partnerships with colleges and employers to ensure that grad-uates have the skills to meet employers’ needs. The study’s results provide strong evidence for both the workforce development and the postsecondary education fields of the benefits of helping low-income adults earn postsecondary educational credentials that have high value in the local labor market.

Figure S2 Project QUEST’s Impact on Average Annual Earnings during Each Year after Random Assignment (N=410)

Year 1** Year 2** Year 3 Year 4** Year 6** Year 7 Year 8 Year 9**Year 5**

Note: Earnings impacts are reported in current dollars and represent differences in average annual earnings between treatment group and control group members in each year. Statistical signi�cance levels: *** p<.01; ** p<.05; * p<.10

-$3,000

-$2,000

-$1,000

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000$4,691

-$1,801-$2,369

$3,925

$1,881

$3,980

$2,176

$2,952

$5,239


IntroductionEmployers have frequently reported difficulties finding workers to fill middle-skill jobs—that is, jobs that require some postsecondary education or training and involve technical, administrative, or communicative tasks.1 At the same time, low-wage workers face substantial barriers to completing postsecondary education and training programs that can enable them to access well-paying jobs. In the early 1990s, two community-organizing groups in San Antonio, Texas, Communities Organized for Public Service (COPS) and Metro Alliance, saw that local employers were filling well-paying jobs with workers recruited from outside the metropolitan area, while local residents struggled in low-wage jobs that did not enable them to support their families. In 1992, COPS and Metro Alliance founded Project QUEST to help San Antonio residents gain skills to meet the needs of employers in strong sectors of the local economy. Project QUEST provides comprehensive support and resources to help individuals complete occupational training programs at com-munity colleges and professional training institutes, pass certification exams, and obtain jobs. Project QUEST’s services include the following:

• financial assistance to cover tuition and fees for classes, books, transportation, uniforms, licensing exams, and tutoring2

• remedial instruction in math and reading to help individuals pass college place-ment tests

• counseling to address personal and academic concerns and provide motivation and emotional support

• referrals to outside agencies for assistance with utility bills, childcare, food, and other services, as well as direct financial assistance with other supports on an as-needed basis

• weekly meetings that focus on life skills, including time management, study skills, critical thinking, and conflict resolution

• job placement assistance, including help with writing résumés and interviewing, as well as referrals to employers that are hiring

In 2005, Project QUEST agreed to take part in an RCT study to assess its impacts on participants’ earnings.3 Between April 2006 and October 2008, 410 individuals enrolled in the study, of whom 207 were assigned to the treatment group, which could receive Project QUEST’s services, and 203 to the control group, which could not. Because individuals were randomly assigned, at the time of study enrollment the two groups were equivalent on measured characteristics, such as age, prior education, and work experience, and unmeasured characteristics, such as motiva-tion. Therefore, any differences found in their outcomes can be attributed to Project QUEST’s services. Additional details regarding the study design and the final sample are provided in the appendix. In this report, we refer to all members of the treatment group as “the QUEST participants” or “the QUEST group.”

In 2017, we reported on Project QUEST’s impacts six years after study enrollment, based on surveys completed with study participants.4 We found that Project QUEST


had a large impact on participants’ annual earnings and that QUEST participants were more likely than control group members to earn vocational certificates or licenses. In 2018, Mobility received support from Arnold Ventures to extend the study to assess Project QUEST’s impacts on participants’ earnings, educational attainment, and public benefits receipt during the nine years after study enrollment. This phase of the study relies on administrative records, including Texas Workforce Commission data on quarterly earnings and unemployment benefits receipt, and National Student Clearinghouse (NSC) data on postsecondary educational out-comes. Mobility contracted with the Ray Marshall Center at The University of Texas at Austin to access and analyze the data.

In our 2019 report, Nine Year Gains, we presented Project QUEST’s impacts on annual earnings and other employment outcomes. As shown in Figure 1, QUEST participants earned substantially more than control group members in the third through ninth years after random assignment. The earnings differences were sta-tistically significant in years four through six after random assignment. The differ-ence narrowed in year seven but began to grow again in year eight. Project QUEST’s impact on annual earnings grew to $5,239 in year nine—a difference that was both statistically significant and the largest of any year.

This brief presents the study’s findings regarding Project QUEST’s impacts on college enrollment and completion as well as receipt of unemployment insurance benefits. We also assess Project QUEST’s net costs and compare those to its net benefits to determine whether it is worthwhile to society to invest in the QUEST model.


Note: Earnings are reported in current dollars. Statistical signi�cance levels: *** p<.01; ** p<.05; * p<.10

$0

$4,000

$8,000

$12,000

$16,000

$20,000

$24,000

$28,000

$32,000

$36,000

Year 1** Year 2** Year 3 Year 4** Year 6** Year 7 Year 8 Year 9**Year 5**

Figure 1 Average Annual Earnings during the Nine Years after Random Assignment among All Study Participants

$10,721

$8,920

$14,993

$12,624

$18,220

$16,340

$22,081

$18,156

$24,311

$20,331

$26,664

$21,974

$27,187

$25,011

$30,315

$27,363

$33,644

$28,404


Limitations of the NSC DataThe NSC holds information on enrollment and degree completion for 3,600 post-secondary institutions nationwide, including two- and four-year, public, private nonprofit, and private for-profit institutions. Its data coverage is better for public institutions than private institutions and has improved over time. However, schools have discretion regarding whether or not to report non-degree-seeking students. This last point is important for the study because our review of the data from Project QUEST, from the six-year survey, and from NSC indicates that enrollment in the medical records coder program that Project QUEST supported during the study was not reported to NSC. Therefore, the findings in this brief underestimate Project QUEST’s full impact on credential attainment.

The Study Participants and Targeted Postsecondary ProgramsDuring the study enrollment period, Project QUEST sought to recruit individuals who were interested in but not currently attending college classes. This included those who needed to improve their basic reading and math skills before they could enroll in college-level classes as well as those who were ready to enroll. Project QUEST staff conducted a thorough screening of applicants to determine their eligibility, evaluate their aptitudes and interests, and match them to appropriate careers and training. Career counselors worked with applicants to develop an academic plan, which included necessary courses and a timeline for completion. Counselors also helped them develop a budget plan to determine whether they would have enough income to cover their expenses during training.

Applicants presented their plans during an interview, after which staff decided whether or not to recommend them for enrollment. Recommended applicants were submitted for random assignment. Career counselors worked with indi-viduals assigned to the treatment group, either inviting them to attend reme-dial classes held at Project QUEST or helping them with the college enrollment process. While control group members did not receive any further support from Project QUEST, the intake process provided them with a de facto road map for completing a certificate or degree program at a local community college for their targeted health-care occupation.

As presented in Table 1, most study participants were female, Latino, had a high school diploma, and had children under age 18 at the time of study enrollment. Most (69 percent) were between the ages of 25 and 64—older than traditional college-age students. Forty-five percent had previously attended college but had not attained a degree. While 84 percent had worked in the previous year, average annual earnings among all participants were only $11,656.

The evaluation focused on individuals who were pursuing training for health-care jobs. Most study participants (85 percent) were seeking to enter non-degree


programs. Sixty percent sought to enroll in licensed vocational nursing certificate programs. A quarter sought to enroll in a medical records coder program at San Antonio College that was new at the time of the study and developed in partnership with Project QUEST. The medical records coder training was a continuing educa-tion program for which students earned a certificate of completion but did not earn college credits. Another 15 percent of participants were interested in pursuing programs that would lead to an associate’s degree, including registered nursing, and radiography, respiratory therapy, sonography, and surgical technician programs (Figure 2). Table 2 provides information about the course requirements for complet-ing each of the targeted health-care career tracks as well as the type of credential each conferred. Once students completed the remedial and prerequisite classes and entered the career track programs, they were required to attend class full-time. When students completed their coursework and clinical rotations, they were required to pass a national certification exam, and in some cases a state exam as well, to work in their chosen profession.

Table 1 Study Participants’ Characteristics at Program Entry (N=410)

Female 88.3%

Latino 74.1%

African American 13.9%

White 9.5%

Age 18 to 24 30.7%

Age 25 to 64 69.3%

Had any children under age 18 71.2%

Married 27.6%

Had a GED 25.1%

Had a high school diploma 70.2%

Had a college degree 4.6%

Had previously attended college but did not earn a degree

44.9%

Was employed at any time in the past year 84.4%

Average annual earnings in the past year $11,656

Registered nurse7%

Licensed vocational nurse60%

Medical recordscoder25%

Figure 2 Study Participants’ Intended Health-Care Occupation at Program Entry (N=410)

Technician8%


Project QUEST’s Impacts on Postsecondary Educational Attainment

Project QUEST significantly increased college enrollment.Project QUEST increased enrollment in NSC-covered college degree and certificate programs. QUEST par-ticipants were significantly more likely than control group members to be enrolled in college at any time during the nine years after random assignment—a 9.4 percentage point difference (Figure 3). As we dis-cuss later in the report, this underestimates Project QUEST’s impact on enrollment, due to the omission of the medical records coder program from the NSC data, which represented a quarter of the study sample. As shown in Figure 4, Project QUEST had a significant, positive impact on college enrollment in the first three years after random assignment. QUEST participants were also significantly more likely than control group members to be enrolled in college full-time during the first two years after random assignment (Figure 5). Research has shown that community college students who attend school full-time for at least a semester are more likely than those who do not to earn a post-secondary certificate or degree.5

TrackNumber of study

participantsProgram requirements Credential

Medical records coder 101 7-month program Continuing education certificate of completion

Licensed vocational nurse 243 1-year program with 3 prerequisite classes Certificate level 2

Registered nurse 30 2-year program with 7 prerequisite classes Associate of applied science

Radiography, respiratory therapy, and sonography technicians 26 2-year programs with 3 prerequisite classes Associate of applied science

Surgical technician 10 1-year program with 6 prerequisite classes Associate of applied science

Table 2 Program Requirements and Credentials Earned for Project QUEST’s Targeted Health-Care Career Tracks

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

QUEST group(N=207)

Control group(N=203)

Statistical signi�cance levels: ** p<.05

Figure 3 Percent Who Ever Enrolled in NSC-Covered Postsecondary Education during the Nine Years after Random Assignment

73.5%

64.1%




Year 1*** Year 2** Year 3** Year 4 Year 6 Year 7 Year 8 Year 9Year 5

Figure 4 Percent Enrolled in NSC-Covered Postsecondary Education during Each Year after Random Assignment

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

66%

45%38%

49%

41%

29% 26%

29%

21%

20%

13%

14%

14%

18%

13%

14% 15%

9%



Year 1*** Year 2*** Year 3 Year 4 Year 6 Year 7 Year 8 Year 9Year 5

Figure 5 Percent Enrolled Full Time in NSC-Covered Postsecondary Education during Each Year after Random Assignment

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

44%

25% 21%

34%

21%

15% 11%

13%9%

7%

7%

5%

7%

6%

7%

5% 4%

7%


The enrollment impacts coincide with the period when participants were supported by Project QUEST. Ninety-five percent of treatment group members received sup-port from Project QUEST during the first year after random assignment (Figure 6). Nearly three-quarters continued to receive support during the second year and 32 percent during the third year. Figure 6 includes Project QUEST’s support of students whose educational activities are not captured in the NSC data, including those who pursued the medical records coder training and those who participated in remedial math and English classes at Project QUEST prior to enrolling in a college program.

Overall, QUEST participants received support for an average of 22.6 months. This varied substantially by program track from averages of 14.6 months for those in the medical records coder track to 3.5 years for those in the registered nursing track (Table 3).

Medical records coder 14.6

Technicians 20.8

Licensed vocational nurse 24.5

Registered nurse 36.5

All 22.6

Table 3 Average Number of Months Participants Received Support from Project QUEST by Program Track

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6

Figure 6 Percent of QUEST Participants Who Received Support from Project QUEST \ during Each Year after Random Assignment

95%

73%

32%

14%

6% 2%


Project QUEST significantly increased attainment of a college credential.QUEST participants were significantly more likely than control group members to earn an NSC-covered college credential during the nine years after random assign-ment—a 16.4 percentage point difference (Figure 7). Project QUEST nearly doubled the percentage of participants who completed a college certificate or diploma program. QUEST participants were also slightly more likely than control group mem-bers to earn a college degree (18 percent versus 14 percent, respectively), but the difference was not statistically significant.

As shown in Figure 8, Project QUEST increased credential attainment starting in the second year after random assignment. By the sixth year after random assignment, when nearly all treatment group members were no longer receiving support from QUEST, there was a 10.7-point difference between the two groups in the percent who had earned at least one NSC-covered credential. The difference between the groups continued to grow in years seven through nine, when an additional 7.7 percent of QUEST participants earned their first credential compared with 2 percent of control group members. Seventy-three percent of QUEST participants who completed their first credential in years seven through nine earned either an associate’s or bachelor’s degree.

QUEST participants were more likely than control group members to earn one NSC-covered credential while a similar percentage earned two or more credentials. Thirty percent of QUEST participants earned one NSC-covered credential during the nine years after random assignment, and 11 percent earned two or more. Comparatively, 13 percent of control group members earned one credential and 10 percent earned two or more.

Figure 7 Percent of Study Participants Who Earned NSC-Covered Postsecondary Credentials during the Nine Years after Random Assignment

Any credential*** Certi�cate or diploma***

Associate’s degree

Bachelor’sdegree

Any college degree


Note: Subcategories do not sum to the total because some participants earned more than one credential.


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

40.2%

23.8% 24.9%

12.6%15.2%

12.6%3.0% 3.4%

17.9%14.3%


Project QUEST more than doubled the attainment of a health-care credential.A main tenet of Project QUEST’s sectoral strategy is to prepare individuals for well-paying careers while meeting the needs of employers in strong sectors of the local economy. As noted earlier, the study focused on Project QUEST’s efforts to help individuals earn credentials and enter careers in the health-care field. At the time of study enrollment, all individuals in the study, including those in the control group, were interested in enrolling in one of the health-care certificate or degree programs that Project QUEST supported.

As shown in Figure 9, QUEST participants were more than twice as likely as con-trol group members to earn a college credential in the health-care field. The larg-est impact was on the percent who completed a health-care certificate program, primarily licensed vocational nursing certificates. The program’s full impact on earning health-care certificates is underestimated because the data do not include the medical records coder program. Project QUEST also had a significant, posi-tive impact on the percent who earned a college degree in the health-care field. Thirteen percent of QUEST participants earned either an associate’s of applied science or a bachelor’s of science in nursing, compared with 7 percent of control group members.

Figure 8 Cumulative Percent Who Earned At Least One NSC-Covered Postsecondary Credential through Each Year after Random Assignment

Year 1 Year 2 Year 3*** Year 4*** Year 6**Year 5*** Year 7*** Year 8*** Year 9***0%

10%

20%

30%

40%

50%

60%



0.9% 2.1%

10.9%

20.7%

28.1%31.3% 32.5%

34.7%

38.8%40.2%

6.8%

10.6%

15.5%18.4%

21.8% 23.0% 23.3% 23.8%


Among study participants who earned any NSC-covered credentials, health-care cre-dentials were the most common for both the QUEST group and the control group. Among QUEST participants who earned any credentials, 89 percent earned at least one credential in the health-care field and 78 percent earned a certificate or degree in nursing. Among control group members who earned a college credential, 64 per-cent earned at least one health-care credential and 38 percent earned a certificate or degree in nursing. The other types of credentials that control group members earned included a mix of occupational programs, including administrative or office skills and early childhood education, and academic programs, including general studies and liberal arts.

Project QUEST had significant positive impacts on credential attainment for most subgroups.Project QUEST had positive impacts on the likelihood of earning any NSC-covered college credential for most subgroups of participants. The one exception was male participants, but the number of males in the sample was small (Figure 10). QUEST participants across age groups were more likely than control group members to earn a credential, but the impact was statistically significant only for 25-to-34-year-olds (Figure 11). Project QUEST’s impact was significant for both Latino and non-Latino participants and for those who had either a GED or a high school diploma at the time of program entry (Figures 12 and 13). This latter finding is important, as some researchers have questioned the wisdom of public investment in training for individuals with GEDs.6

Figure 9 Percent of Study Participants Who Earned NSC-Covered Health-Care Credentials during the Nine Years after Random Assignment

Any health-care credential***

Any health-care certi�cate or diploma***

Any health-care college degree**


Note: Subcategories do not sum to the total because some participants earned more than one credential.


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

15.4%

35.6%

10.7%

24.3%

8.5%

15.3%


Figure 10 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Gender

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

Female***(N=362)

Male(N<50)

-9.3%

19.6%

Within group signi�cance level: *** p<.01

Between group signi�cance level: * p<.10

18 to 24(N=126)

25 to 34(N=186)***

35 to 44(N<100)

45 to 64(N<50)

0%

10%

20%

30%

40%

50%

60%


Figure 11 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Age

8.0%

23.1%

12.2%

18.4%

0%

10%

20%

30%

40%

50%

60%

Latino(N=304)***

Non-Latino(N=106)***


Figure 12 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Ethnicity

14.2%

23.6%

0%

10%

20%

30%

40%

50%

60%

GED(N=103)**

High school diploma(N=288)***

Within group signi�cance levels: ** p<.05, *** p<.01

Figure 13 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Education

16.7% 15.9%


Program impacts on earning a credential were significant for both those who were currently married and those who had never been married, as well as for those with and without children under the age of 18 at the time of program entry (Figures 14 and 15). Differences between the subgroups were not statistically significant.

Comparison of NSC, Project QUEST, and Survey DataIn our earlier publication on Project QUEST’s impacts after six years, we reported findings based on surveys completed with study participants and data from Project QUEST. We found that 98 percent of QUEST participants and 77 percent of control group members reported attending any postsecondary education or training program during the six years after random assignment. Seventy-five percent of QUEST partici-pants and 57.2 percent of control group members reported earning a postsecondary credential, including certificates and degrees. While QUEST participants were sig-nificantly more likely than the control group to report earning a vocational certificate or license (72.6 percent versus 49.1 percent, respectively), they were less likely to report earning a college degree (18.2 percent versus 26.3 percent, respectively).

The NSC findings in this report likely differ from the earlier survey findings for a number of reasons. The survey captured vocational training completed at providers that do not report to NSC. The survey also captured training programs at institu-tions that report to NSC but do not report on all students attending non-degree certificate or diploma programs. This included the medical records coder program

Never married(N=188)***

Formerly married(N=109)

Currently married(N=113)*

0%

10%

20%

30%

40%

50%

60%

Within group signi�cance levels: * p<.10, *** p<.01

Figure 14 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Marital Status

20.0%

10.5%

16.1%

0%

10%

20%

30%

40%

50%

60%

Had at least one child< age 18 (N=283)***

No children < age 18 (N=127)**

Within group signi�cance levels: ** p<.05, *** p<.01

Figure 15 Project QUEST's Impact on the Likelihood of Earning an NSC-Covered College Credential, by Parenting Status

14.9%

20.6%


at San Antonio College (SAC) that Project QUEST supported during the study. A comparison of the survey and NSC data found that 43 QUEST participants and 12 control group members who reported attending a college covered by NSC were not represented in the NSC data. This included 32 QUEST participants who attended the medical records coder program at San Antonio College. If this program had been reported to NSC, the percent of QUEST participants who had enrolled in col-lege would have increased from 74 percent to 89 percent, and the percent who earned any college credential would have increased from 40 percent to 62 percent. In the survey, no members of the control group reported attending the medical records coder program at SAC; therefore, we would not expect this omission to affect the control group outcome. However, control group members may have been more likely to attend other programs not captured in the NSC data.

Other potential reasons for discrepancies between the survey and the NSC data include mismatches due to misspellings of names, resulting in false negative find-ings, and federal privacy laws that allow students to prevent their records from being shared. We also examined whether survey nonresponse bias was a factor but found that this did not contribute significantly to the discrepancies. The results of this analysis appear in the appendix. Another possible explanation for the discrepancy in rates of earning a college degree is that some study participants who completed a certificate program misreported earning an associate’s degree in the survey.

Project QUEST’s Impact on Unemployment Benefits Receipt

Early differences in unemployment benefits receipt faded over time.QUEST participants were significantly less likely than control group members to receive unemployment benefits in the third year after random assignment. The third year represents years 2009 to 2011, depending on when participants enrolled, coinciding with peak unemployment during the Great Recession. The differences between the groups in the percent who received unemployment benefits faded over time, and the rates of receipt were similar in years six through nine (Figure 16). Overall, the difference in the percent of QUEST participants and control group mem-bers who ever received unemployment benefits during the nine years was not sig-nificant (28 percent versus 32 percent, respectively). The average benefit amount over the nine years, including for those who received zero benefits, was $638 for QUEST participants and $878 for control group members; the $240 difference was not statistically significant.


Comparison of Benefits and CostsProject QUEST makes a substantial investment in participants to help them complete postsecondary education and obtain well-paying jobs. To address the question of whether it is worthwhile to society to invest in the Project QUEST model, we com-pared the program’s net earnings benefit to its net costs, including both program costs and the costs of postsecondary education. For purposes of comparison, we converted both benefits and costs to 2017 dollars—the final year in the analysis.

The average cost per participant of Project QUEST was $11,490 in 2017 dollars. Figure 17 presents the breakdown of these costs. Tuition costs—that is, the portion of participants’ community college tuition that Project QUEST paid—accounted for 22 percent of the overall costs. The additional supports that Project QUEST provided made up another 23 percent of costs, the most common of which were books, transporta-tion, certification exam fees, review courses, uniforms, and vaccinations required to enter the health-care field. Salaries and benefits for program staff, includ-ing the career counselors who supported participants from college enrollment through job placement, and other direct program costs accounted for 41 percent of the overall costs.



Year 1 Year 2 Year 3** Year 4 Year 6 Year 7 Year 8 Year 9Year 5

Figure 16 Percent of All Study Participants Who Received Unemployment Benefits during Each Year after Random Assignment

0%

10%

20%

30%

6%

4%3%

5%

11%

5%

7%

10%12%

8%

9%

9% 6%

6%5%

5%4%

6%

Tuition 22%

Student supports 23%

Program staff and other service costs41%

Administration and fundraising13%

Figure 17 Project QUEST’s Costs


We estimated college costs for QUEST participants and control group members using cost data from the Integrated Postsecondary Education Data System from the US Department of Education’s National Center for Education Statistics as well as enrollment data from NSC. Details about the calculation of college costs are included in the appendix. The total costs for QUEST participants include the college costs plus the non-tuition costs of the Project QUEST program.

In the nine years after random assignment, Project QUEST’s average net earnings benefit exceeded its average net college and program costs by $6,869. The net earnings benefit over the nine years was $20,754, while net program and college costs averaged $13,885 (Table 4). In addition to the costs of the Project QUEST program, the QUEST group’s average college costs were higher than those of the control group, due to QUEST participants’ higher rates of college enrollment.

The college costs do not include QUEST participants who attended the medical records coder program. We estimate that including these participants’ costs would increase the average college costs for the QUEST group by about $800. Members of the control group also would have attended college programs not captured in the NSC data, but we do not have data on their enrollment that could be used to estimate those costs. From a societal perspective, additional benefits include an increase in fringe benefits associated with the increase in earnings. While these benefits would be partially offset by an increase in work-related expenses, such as commuting and childcare costs, we expect that the estimate of Project QUEST’s net benefit would be greater using standard methods to estimate fringe benefits and work-related costs.

QUEST group Control group Net

Benefits

Average earnings $207,915 $187,161 $20,754

Costs

Average college costs $14,320 $9,528 $4,792

Average Project QUEST costs(not including tuition) $9,093 $0 $9,093

Average total costs $23,413 $9,528 $13,885

Table 4 Comparison of Average Net Earnings Benefits to Net Costs during the Nine Years after Random Assignment (2017 Dollars)


ConclusionsProject QUEST has demonstrated the largest sustained earnings impacts ever found in a US workforce development program. Not only did Project QUEST have a significant impact on earnings, but also it enabled participants to earn an average of nearly $34,000 nine years after study enrollment—lifting them out of poverty and into San Antonio’s middle class. Project QUEST’s strategy is to provide low-income adults with comprehensive supports to complete the postsecondary edu-cation and training needed to access well-paying careers. This report documents that Project QUEST significantly increased college enrollment and credential attain-ment—in particular, more than doubling the attainment of health-care credentials.

Project QUEST’s 16.4-percentage-point impact on credential attainment nine years after study enrollment is among the highest impacts ever found for comprehen-sive efforts to improve students’ educational success. By comparison, a report on the early impacts from an RCT study of the Valley Initiative for Development and Advancement (VIDA), which is based on the Project QUEST model, found that the program increased credential attainment by 8.3 percentage points after two years.7 RCT studies of two college completion programs—InsideTrack, which offers two years of student coaching, and Stay the Course, which provides students with intensive case management and emergency financial assistance—found impacts on college completion rates of about 4 percentage points two to three years after random assignment.8 The RCT study of CUNY ASAP—which provides up to three years of comprehensive counseling and financial supports, as well as blocked courses and consolidated schedules to help students earn a college credential—found that the program increased credential attainment by 18 percentage points after three years. By the sixth year after study enrollment, ASAP’s impact had nar-rowed to a 10-percentage-point difference.9

The Project QUEST findings are even more impressive given that the impact was not only sustained but grew over the nine years after study enrollment. This indi-cates that the program set participants on a path to complete postsecondary edu-cation long after they had stopped receiving its support. The Project QUEST findings underscore three additional points:

• With the exception of VIDA, the other programs to improve college completion rates mentioned above target services exclusively to students already enrolled in college. For the study, Project QUEST targeted low-income adults who were not enrolled in college but were interested in pursuing a career in health care. The study’s findings demonstrate that it is possible for an organization to success-fully recruit, prepare, and support individuals not currently enrolled in a post-secondary institution. Many of these individuals likely would not have enrolled in, yet completed, a postsecondary education program without Project QUEST’s outreach efforts.


• Project QUEST targets credentials valued by employers in the local labor market, including both certificate and degree programs. The study demonstrated success in the health-care field, in which credentials are essential to accessing well-pay-ing jobs and in which demand for workers remains robust over time.

• Project QUEST had its largest impact among nontraditional-aged college students—those over the age of 24—most of whom had children.

The Project QUEST study demonstrates that the program had a large, positive impact not only on credential attainment but also on earnings and career advance-ment over a nine-year period. Studies of other comprehensive efforts that have shown impacts on attainment of college credentials have not yet demonstrated longer-term impacts on employment and earnings. Long-term follow-up is needed in other studies to determine whether educational gains translate into positive impacts on earnings and career advancement.


Endnotes

1 Holzer, Harry. Job Market Polarization and U.S. Worker Skills: A Tale of Two Middles. Washington, DC: Brookings Institution, April 2015.

2 During the first three years of the study, which covered the period when all study participants enrolled and at least their first year in the program, QUEST paid 100 percent of tuition for participants. After this time, QUEST covered 50 percent of participants’ tuition but continued to cover 100 percent of their fees, as well as the costs of books and other education-related expenses.

3 Project QUEST was initially part of a Public/Private Ventures study we launched of sector-based training programs that included Per Scholas, Jewish Vocational Service Boston, and the Wisconsin Regional Training Partnership.

4 See: Elliott, Mark and Anne Roder, Escalating Gains: Project QUEST’s Sectoral Strategy Pays Off. New York: Economic Mobility Corporation, April 2017. For further details on the program model and participants, see: Roder, Anne and Mark Elliott, Escalating Gains: The Elements of Project QUEST’s Success. New York: Economic Mobility Corporation, May 2018.

5 Center for Community College Student Engagement, 2017, Even One Semester: Full-Time Enrollment and Student Success. Austin, TX: The University of Texas at Austin, College of Education, Department of Educational Administration, Program in Higher Education Leadership.

6 See Clement, Douglas, Interview with James Heckman. Federal Reserve Bank of Minneapolis, June 1, 2005. https://www.minneapol-isfed.org/article/2005/interview-with-james-heckman.

7 Rolston, Howard; Elizabeth Copson; and Karen Gardiner, Valley Initiative for Development and Advancement: Implementation and Early Impact Report, OPRE Report #2017-83, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, US Department of Health and Human Services, October 2017.

8 Evans, William N., et al., Increasing Community College Completion Rates among Low-Income Students: Evidence from a Randomized Controlled Trial Evaluation of a Case Management Intervention. NBER Working Paper 24150. Cambridge, MA: National Bureau of Economic Research, December 2017.

Bettinger, Eric and Rachel Baker, “The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Advising.” Educational Evaluation and Policy Analysis 36(1): 3–19.

9 Gupta, Himani, The Power of Fully Supporting Community College Students: The Effects of the City University of New York’s Accelerated Study in Associate’s Programs after Six Years. New York: MDRC, October 2017.


https://www.minneapolisfed.org/article/2005/interview-with-james-heckman

https://www.minneapolisfed.org/article/2005/interview-with-james-heckman


Study Design and Final SampleThe evaluation of Project QUEST uses a random-ized controlled trial design to assess its impacts on individuals’ outcomes. To assess program impacts, this report relies on data from a baseline survey con-ducted at the time of study enrollment, administrative data on earnings and unemployment benefits receipt from the Texas Workforce Commission (TWC) for the two years before and nine years after study enroll-ment, and National Student Clearinghouse (NSC) data on postsecondary enrollment and completion during the nine years after study enrollment. We used data from the baseline survey to examine the charac-teristics of treatment and control group members in the final sample and to control for any differences in characteristics between the groups in the analysis of program impacts. We also analyzed data from Project QUEST on study participants’ anticipated health-care program track and program outcomes for members of the treatment group.

QUEST staff completed the program’s intake and screening process to identify eligible candidates and obtained their written consent to take part in the study. Staff then sent their contact information to a survey firm, the Institute for Survey Research (ISR) at Temple University, which completed the baseline survey and conducted random assignment. Individuals were blocked according to their intended program track (i.e., registered nursing, licensed vocational nurs-ing, medical records coding, and the various techni-cian tracks), and ISR randomly assigned 50 percent of study participants to the treatment group and 50 per-cent to the control group within these blocks. Between April 2006 and October 2008, 410 individuals enrolled

Appendix

in the study, of whom 207 were randomly assigned to the treatment group and 203 to the control group. QUEST staff invited those assigned to the treatment group to participate in the program. Those assigned to the control group were told that they would not receive support from QUEST but that they could enroll in col-lege or training on their own or seek other services.

Table A1 presents the baseline characteristics of the full sample of treatment and control group mem-bers. There were no significant differences between the groups in key baseline characteristics, including gender, age, highest degree earned, annual earnings, housing status, marital status, and whether partici-pants had any children under age 18. Treatment group members were somewhat less likely than control group members to be Latino and more likely than con-trol group members to have a valid driver’s license and to live in public or subsidized housing. These differ-ences are most likely due to chance, given the number of characteristics tested.

AnalysisThe study uses an intent-to-treat analysis framework to assess program impacts; that is, we examined dif-ferences in the outcomes of all participants randomly assigned to the Project QUEST and control groups. We estimated impacts using multivariate regression analysis, and we report regression-adjusted results. The regression models included the baseline explana-tory variables in Table A1 to control for any differences between the treatment and control group members. The models also included a binary variable for being a treat-ment group member and an index variable indicating study participants’ intended health-care program track.


Treatment group(N=207)

Control group(N=203)

p-value

Gender .705

Female 88.9% 87.7%

Male 11.1% 12.3%

Age .633

18 to 24 29.5% 32.0%

25 to 34 46.4% 44.3%

35 to 44 15.5% 17.7%

45 to 64 8.7% 5.9%

Race/Ethnicity .058

Latino 69.1% 79.3%

Black 16.9% 10.8%

White 12.1% 6.9%

Other 1.9% 3.0%

US immigrant 9.7% 5.4% .164

Highest degree earned .278

GED 24.6% 25.6%

High school diploma 69.1% 71.4%

College degree (associate’s or bachelor) 6.3% 3.0%

Had any health-care certificates or licenses .525

Yes 30.4% 27.6%

No 69.6% 72.4%

Housing status .128

Own 18.4% 14.3%

Rent 63.3% 58.6%

Live rent-free 17.4% 26.6%

Homeless 1.0% 0.5%

Living in public or subsidized housing .061

Yes 15.5% 9.4%

No 84.5% 90.6%

Had a valid driver’s license 92.3% 86.2% .047

Rating of own health .162

Excellent 48.3% 57.6%

Good 44.0% 35.5%

Fair or poor 7.7% 6.9%

Marital status .412

Currently married 30.4% 24.6%

Formerly married 25.1% 28.1%

Never married 44.4% 47.3%

Any children under age 18 in household .407

Yes 67.2% 70.9%

No 32.8% 29.1%

Average annual earnings in the two years prior to enrollment based on state administrative data (including zeros) $21,976 $20,116 .292

Table A1 Comparison of the Baseline Characteristics of Treatment and Control Group Members in the Project QUEST Randomized Controlled Trial


Constant DollarsThe earnings amounts in this brief are reported in cur-rent dollars, unless otherwise noted. Figures A1 and A2 present the earnings impacts adjusted for inflation

and expressed as 2017 dollars. Values were adjusted using the Consumer Price Index Research Series (CPI-U-RS), unadjusted for seasonal variation, from the US Department of Labor, Bureau of Labor Statistics.



$0

$4,000

$8,000

$12,000

$16,000

$20,000

$24,000

$28,000

$32,000

$36,000

Year 1*** Year 2** Year 3 Year 4** Year 6*** Year 7 Year 8 Year 9**Year 5*

Figure A1 Average Annual Earnings during the Nine Years after Random Assignment among All Study Participants (2017 Dollars)

$13,017

$10,231

$15,547

$12,799

$19,092

$17,159

$22,873

$18,635

$24,713

$20,792

$26,995

$21,478

$27,755

$25,228

$29,961

$27,301

$33,496

$28,006

Figure A2 Project QUEST’s Impact on Average Annual Earnings during Each Year after Random Assignment (2017 Dollars; N=410)

Year 1*** Year 2** Year 3 Year 4** Year 6*** Year 7 Year 8 Year 9**Year 5*

Note: Statistical signi�cance levels: *** p<.01; ** p<.05; * p<.10

-$4,000

-$3,000

-$2,000

-$1,000

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000 $5,517

-$2,786 -$2,748

$4,238

$1,934

$3,921

$2,527 $2,660

$5,490


Comparison of Survey and NSC DataWe examined whether survey non-response bias con-tributed to differences between the college enrollment and completion rates reported in the six-year follow-up survey and the NSC results. The response rate on the six-year follow-up survey was 83.7 percent overall, 84.5 percent among treatment group members, and 82.8 percent among control group members. Table A2 presents college enrollment and completion rates based on the NSC data for survey responders and non-responders. Non-response bias does not explain

differences between the survey and NSC data in the percent ever enrolled in college during the nine years. In fact, study participants who did not respond to the six-year survey were somewhat more likely than those who did respond to enroll in college according to the NSC data. Survey non-responders were somewhat less likely than survey responders to earn an NSC-covered credential but given the small number of non-respond-ers and the size of the difference, most of the discrep-ancy between the NSC data and survey data remains after accounting for this difference.

QUEST group Control group

Percent ever enrolled in college during the nine years N Percent N Percent

Survey responders 175 73% 168 62%

Survey non-responders 32 78% 35 71%

Percent who earned an NSC-covered credential during the nine years

Survey responders 175 43% 168 25%

Survey non-responders 32 31% 35 14%

Table A2 Analysis of Non-Response Bias in the Six-Year Follow-Up Survey


Calculation of College CostsWe estimated college costs using cost data from the Integrated Postsecondary Education Data System (IPEDS) from the US Department of Education’s National Center for Education Statistics as well as enrollment data from NSC. Total college expenses are the core expenses reported to IPEDS for each year of the study. Core expenses for public institu-tions include instruction, research, public service, academic support, student services, institutional support, operation and maintenance of plant, depre-ciation, scholarships and fellowships, interest, and other operating and nonoperating expenses. Core expenses for private, not-for-profit and for-profit insti-tutions include instruction, research, public service, academic support, student services, institutional sup-port, net grant aid to students, and other expenses. For all institutions, core expenses exclude expenses for auxiliary enterprises (e.g., bookstores, dormito-ries), hospitals, and independent operations.

We divided total expenses by the 12-month FTE enroll-ment for each year to produce annual expense per FTE amounts for each school and project year. For the four Alamo District Schools in the data, due to a change in reporting that affected the FTE calculation during this period, we divided total expenses by the total number of credit hours and multiplied the per-credit rate by 30 to produce the annual expense per FTE estimates. We then adjusted the annual expense amounts for infla-tion, expressed as 2017 dollars.

We used the annual expense per FTE amounts to calculate the total college costs for each participant over the nine years after random assignment using the NSC enrollment data. The annual expense per FTE amounts were discounted if the participant was

only in school for part of the year and/or was enrolled less than full time (i.e., if the NSC data indicated the person was enrolled three-quarter time, we used 75 percent of the annual expense per FTE; if half time, we used 50 percent; if less than half time, we used 25 percent).

Economic Mobility Corporation, Inc.50 Broadway, Suite 1604New York, NY 10004212.280.6975

www.economicmobilitycorp.org

http://economicmobilitycorp.org/

http://www.economicmobilitycorp.org

nine year education gains: project quest’s impact on ... · the economic mobility corporation...

Documents