evaluating the impact of the ucd new era widening participation initiative

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

1

Evaluating the Impact of the UCD New ERA

Widening Participation Initiative

Kevin Denny, Orla Doyle, Marie Hyland,

Patricia O’Reilly, Vincent O’Sullivan

UCD GEARY INSTITUTE

University College Dublin

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Contents

Contents ACkNoWlEDGEmENTS i

GloSSARY ii

ExECUTIvE SUmmARY 1

1 INTRoDUCTIoN 3

1.1 Background 3

1.2 Formation of the New ERA Programme 4

1.3 Description of the New ERA Programme 5

1.4 Overview of Report 7

2 lITERATURE REvIEW 9

2.1 Policy Context: Access in Ireland 9

2.2 Evaluating the effectiveness of financial and social support 11

3 EvAlUATIoN mEThoDS AND DATA 15

3.1 Evaluation Design 15

3.2 Selecting the Control Group 15

3.3 Description of Data 17

3.4 Data Limitations 18

3.5 Description of Sample 19

4 RESUlTS 29

4.1 Interpreting the Results 29

4.2 Evaluation Results 30

4.3 Main findings in this chapter 35

5 FURThER ANAlYSES 37

5.1 The Impact of New ERA/HEAR on Progression to University 37

5.2 The Impact of New ERA’s Financial Aid on Student outcomes 43

5.3 Likely Consequences of the National HEAR scheme. 44

5.4 Main findings in this chapter 46

6 RECommENDATIoNS AND FINDINGS 48

REFERENCES 50

APPENDICES 53

4

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Acknowledgements

AcknowledgementsThis report was produced under the ‘Evaluating the Impact of the UCD New ERA Widening Participation Initiative’ which was conducted at the UCD Geary Institute and funded by the Strategic Innovation Fund in conjunction with the UCD New ERA Programme and NUI Maynooth Access Programme (MAP).

We would like to thank our colleagues at New ERA and NUI Maynooth in particular Fiona Sweeney (UCD New ERA Coordinator) and Ann O’Brien (Director of Access, NUI Maynooth) for all their help and support. We would particularly like to thank Áine Galvin for supporting and driving this research from the beginning and to Ronan Murphy (acting Director of Access and Lifelong learning) and Anna Kelly (current Director of Access and Lifelong learning) for their time and commitment to this project as members of the Steering Committee.

This study could not have been conducted without the help and support of UCD Registry for assisting us in collating the admissions and exams data used in the analysis. We would particularly like to thank Susan Mulkeen from Admissions, Ciarán Ó hUltacháín and Jill O’Mahony from Assessment, and Paula Tarrant, the Director of Operations. We would also like to thank Maria McDonald from Management Information Services (MIS) in IT services for providing the exams data. Thanks also to James McBride (Director of the Irish Social Science Data Archive) for anonymising the administrative data used in the analysis.

We would like to thank the principals and teachers at the schools that responded to our HEAR survey.

In addition, we would like to thank Susanne Schmidt from the UCD Urban Institute for producing the maps used in the report and also to Patrick McKay for the graphic design work on this report.

Our thanks also to Colm Harmon (UCD), Ian Walker (Lancaster University), Robin Naylor (University of Warwick), Asako Ohinata (University of Warwick) and Arnaud Chevalier (Royal Holloway, University of London) for providing helpful comments and ideas on this evaluation strategy. Finally, we would like to thank the many others who have provided guidance and advice on this study including Danny Moran and Abi Campbell.

It should be noted that the interpretation of the results and the recommendations are those of the research team.

This report was written by the research team at the UCD Geary Institute

kevin Denny, orla Doyle, marie hyland, Patricia o’Reilly, vincent o’Sullivan

i

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Glossary

ii

GlossaryBITE Ballymun Initiative for Third Level Education

CAo Central Applications Office

DEIS Delivering Equality of Opportunity in Schools

DIT Dublin Institute of Technology

GPA Grade Point Average

hEA Higher Education Authority

hEAR Higher Education Access Route

hEDAS Higher Education Direct Application Scheme

hEI Higher Education Institution

mAP Maynooth Access Programme

NESF National Economic and Social Forum

NDP National Development Plan

NUI National University of Ireland

oECD Organisation for Economic Cooperation and Development

RCT Randomised Control Trial

SES Socio-Economic Status

TAP Trinity Access Programme

UCD University College Dublin

Executive Summary

1

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Executive Summary

Executive SummaryThis study is a quantitative evaluation of the effectiveness of New ERA, UCD’s access program. New ERA is

a multi-dimensional programme that works with designated disadvantaged schools to increase the number of

socio-economically disadvantaged students progressing to higher education and to improve student outcomes

once at UCD. Access students entering UCD also receive further support including financial aid, mentoring and

academic supports. A proportion of New ERA students also benefit from a lower Leaving Certificate points

requirements (“Direct” students) with the other New ERA participants (“Merit” students) being required to

obtain the same points as other students.

The study uses a quasi-experimental design to quantify the effect of New ERA on a set of outcomes including

progression to university and the academic outcomes of the students at the end of their first and final year in

UCD. The main results are discussed below.

Participation in New ERA has a positive effect on reducing first year withdrawal rates. New ERA reduces •

the probability of withdrawing prior to first year exams for low point Merit students and high point

Direct students.

New ERA has a positive effect on improving first year exam results by shifting students up the grade •

distribution. Participation in New ERA increases the probability of achieving a First and Second Class

honours and reduces the probability of failing or receiving a Third Class honours/Pass in the first year

exams. These improvements in exam performance only benefit those who enter UCD with more than

400 Leaving Certificate points.

Participating in New ERA increases the probability of graduating from university. The result applies to both Direct

and Merit students.

The programme has relatively little effect on the final degree classification the students receive. •

However, Merit students have an increased chance of attaining higher grades.

Once a secondary school becomes part of the Higher Education Access Route (HEAR) scheme there is •

a higher probability that it will send more students to university. It was found that becoming linked to

HEAR between 2001 and 2007 increased the probability of sending a higher proportion of students to

university by 14%.

Being linked to New ERA increases the proportion of the Leaving Certificate class progressing to UCD •

by about one percentage point.

Changes in New ERA’s financial aid package to students, taking into account the Higher Education •

grant, do not have a measurable effect on student outcomes.

Comparing New ERA students with students who are also socially disadvantaged but who do not •

qualify for the programme (as they attend a non-disadvantaged school), the results show that New

ERA students academically out-perform their comparators. This suggests that the new national

HEAR scheme, which allows students attending non-DEIS schools to participate in New ERA, may be

beneficial.

The report ends with a series of recommendations, some specific to UCD and others which have more general

implications. These include relaxing the criteria for disadvantage status so that more schools are linked to access

programmes, increasing the number of places allocated to “Direct” students particularly in high point courses,

and taking steps to increase the number of “Merit” students by expanding the outreach activities at secondary

level.

CHAPTER 1

INTRODUCTION

Chapter 1Introduction

3


1 Introduction1.1 Background

A person’s education is one of the principle influences on their outcomes in life. Education is an important determinant of an individual’s income, where they live, and how they participate in society. It can also affect how people bring up their children and how they manage their own health. There is ample evidence of significant benefits to being more educated and Ireland is no exception to this trend.1

The ongoing growth of the information society, the emphasis on innovation and increased globalisation are

likely to further increase the importance of education. Therefore, a strong argument can be made that society

should invest in education to ensure that people’s education is not constrained by their parents’ socio-economic

status (SES). Note that a concern over equity is not the only reason why we should be concerned about access

to education: it is also an issue of efficiency. While equity considerations are central for increasing access to

education, another consideration is economic efficiency. For our society to prosper it is essential that skills and

talent of young people are not wasted, just as a firm needs to ensure that its resources are used to the fullest

extent.

Despite the near universal agreement on such equity consideration, it is striking that such a strong correlation

exists worldwide between people’s education and their parents’ SES. While parental background acts as a

constraint to educational attainment for many people, it is worth noting that the extent of this constraint is

probably higher in Ireland than most Western countries. Denny et al. (2009, Table 12.3) measure the correlation

between parental SES and education in 17 OECD countries and finds that father’s education has the highest

impact on the level of education attained by an individual in Ireland. The high dependence of attendance at third

level education on SES has also been well documented in Ireland in a series of reports published by the Higher

Education Authority for example Clancy (1982, 2001).

While children from disadvantaged backgrounds are less likely to go to university, it is not obvious why exactly

this is so. Possible explanations point to a lack of financial and social resources, a lack of knowledge about the

benefits of education, or attendance at schools which do not promote higher education. In reality it can probably

be attributed to some combination of all of these. However, and perhaps surprisingly, we do not have good

estimates for Ireland of the relative contribution of these factors. This represents a significant limitation: if policy

makers are to implement programmes to improve access to education they need to know what the barriers to

education are and where the investment will be most effective. Given the very difficult budgetary situation that

the Irish government now faces this is particularly salient. While the lack of knowledge of the precise causes of

educational inequalities is striking, it is also noticeable that policy makers have been active, particularly in recent

years, in generating initiatives to address these concerns.

This report is one step forward in providing a quantitative assessment of the effectiveness of one such initiative,

the New ERA Widening Participation Initiative which operates at University College Dublin. The primary aim of

the New ERA programme is to promote and support the participation of students from lower socio-economic

groups who, in general, are under-represented in higher education. The access programme provides secondary-

level students from designated disadvantaged schools with supported entry mechanisms to study at UCD and

financial support for the duration of their university life, in addition to a host of support services both before and

after they join UCD. New ERA began in 1997 and now operates under the Office of the Director of Access and

1 See Machin & Vignoles (2005) for an introduction to education economics.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative

4


Lifelong Learning. To date, the effectiveness of the programme has not been assessed by quantitative research. If

the New ERA programme is to develop and maximise its potential, then an evaluation of the programme which

is scientifically rigourous, using the best available data and matched with advanced statistical techniques, is

necessary.

This study conducts a retrospective evaluation of the New ERA programme between 1999 and 2004 across

different outcomes. This time period was prior to the introduction of modularisation in UCD. The study examines

the performance of New ERA students post-entry, specifically the impact of New ERA on first year withdrawal/

retention rates and exam performance. The evaluation also tests the effectiveness of New ERA on final year

graduation rates and degree classification. Finally, the evaluation examines progression rates by quantifying the

effect of a designated disadvantaged school joining the New ERA programme on increasing participation in third-

level education. In addition to examining the impact of the programme for all New ERA students, the study

also distinguishes between students entering UCD on points concession and those who enter without points

concession, i.e. though the normal CAO route. These are called Direct and Merit students respectively. This

distinction will be explained more thoroughly later in this chapter.

The evaluation involves the analysis of a number of secondary data sources – the UCD New ERA database and the

UCD students’ administration database. It is necessary to use both datasets in order to compare the outcomes

of the access students to the students from the general university population. Therefore in the language of the

programme evaluation literature, the current and past New ERA students are the “treatment group” whilst a

matched sub-sample from the general university population, which did not receive the programme, are the

“control group”. By comparing the difference in outcomes between these two groups it is possible to identify the

effect of the programme.

The potential benefits of this study are twofold. First, it will contribute to the international literature on access

programmes, where such rigourous quantitative studies are few. And second, the results from this study will help

determine the effectiveness of the New ERA programme specifically, and help inform policy about the future

development of such programmes internationally.

The reminder of this chapter will discuss the policy context in which New ERA has evolved and developed. Section

1.2 describes the formation of New ERA. Section 1.3 describes the New ERA programme. Finally, section 1.4

presents an overview of the report.

1.2 Formation of the New ERA ProgrammeThe New ERA programme evolved out of a movement, arising in the early 1990s, to improve access to higher

education for people from disadvantaged communities. A significant step forward was the formation of area based

partnerships in 1991. This, along with the community platform, supported initiatives aimed at helping students to

access third level education. One such pilot programme was Ballymun Initiative for Third Level Education (BITE),

established in 1990. This scheme laid the ground work for targeted intervention in Ireland by tackling educational

barriers through homework clubs and Leaving Certificate tuition, social barriers through creating awareness of

third level for both parents and pupils and financial barriers to higher level through the provision of scholarships

at third level. The scheme supported students at primary and second level (pre-entry) and after students entered

higher education (post-entry).

The foundations for New ERA were laid in 1994 by the then registrar, Professor John Kelly through the set up of the

UCD Committee on Equality of Participation, and sustained by his successor Dr Caroline Hussey. The committee

commissioned Lynch and O’ Riordan’s study entitled ‘Social Class, Inequality and Higher Education; Barriers to

Equality of Access and Participation among school leavers’ in 1995 to identify barriers against participation. This

work was commissioned in order to develop a frame of reference for how UCD would tackle access. New ERA

followed in 1997.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

5

In 2001 seven higher education institutes, Trinity College Dublin, University College Dublin, Dublin City University,

NUI Maynooth, NUI Cork, University of Limerick and the Dublin Institute of Technology collaborated to create

the Higher Education Access Route (HEAR) scheme2. Through this scheme students could apply to

the participating institutions through the CAO and a supplementary application. The scheme also allowed Access

Offices to avoid duplicate applications and to pool their reserved places. The HEAR Scheme allows school leavers

from linked schools to apply for reduced point entry to any participating HEI along with post-entry supports.

The scheme was initially supported by the HEA’s Targeted Initiative Scheme Innovation Fund and all other cost is

divided between the seven participating organizations. More recently, the development of the HEAR scheme into

a national scheme has been financed by the Strategic Innovation Fund The HEAR Scheme is, in part, a response to

the suggestion made in the White Paper ‘Charting our Education Future’ that designated disadvantaged schools

should become linked to higher level institutions.

In relation to the financing of access initiatives in Ireland, the HEA provided funding to the universities, under

the Targeted Initiatives, to develop special schemes to improve the participation of students from disadvantaged

social backgrounds (Osborne and Leith, 2000). Also, the National Development Plan (NDP) set aside finances

for a third level access fund for the period 2000-2006, aimed at tackling under-representation by students

from socio-economically disadvantaged backgrounds, mature students, students with disabilities and students

from ethnic minorities. The universities themselves also support the initiatives financially. Funding also became

available directly for students through the European Social Fund. Some of this financial support was kept centrally

and used as a college-wide student assistance fund.

1.3 Description of the New ERA ProgrammeThe New ERA programme has existed since 1997. Its aim is to encourage and facilitate increased participation in

higher education by students from socio-economically disadvantaged backgrounds that are underrepresented at

third level. Broadly speaking, socio-economically disadvantaged students are defined as those from a background

affected by long-term unemployment, low family income and/or little or no tradition of progression to third level

education. Within University College Dublin New ERA operates under the remit of the Registrar and administers

the Higher Education Access Route on behalf of UCD3.

Specifically, a student is considered eligible for New ERA if:

there is no previous history of progression to higher level education in their family •

their family income is below a certain level•

they are students at a designated disadvantaged school•

they are a member of the six under-represented groups outlined by Patrick Clancy in The Social •

Background of Higher Education Entrants (2001).

These under-represented groups are: unskilled manual workers, semi-skilled manual workers, other •

non-manual workers, intermediate non-manual, skilled manual and agricultural workers.

Lynch and O’ Riordan’s (1995) report provided the framework for the development of the New ERA programme.

This report explored and documented viewpoints on disadvantage from four groups experiencing it. These groups

included: community activists, school personnel, low income third level students and second level students

intending to progress to third level.

The report highlighted three main types of barriers, financial, educational and socio-cultural, that hindered students

in accessing and progressing in higher level education. It found that all four groups interviewed considered financial

constraints to be the most substantial barrier to higher level education. Both educational and cultural barriers were

considered important. New ERA aimed to tackle these barriers through providing support to students both before

2 HEAR Scheme has gone through several incarnations and is known as: 2000 - New ERA Direct Entry, 2001 - Common Application Form, 2002 Direct Application Scheme, 2003 to 2004 Higher Education Direct Application Scheme and 2005 - 2009 Higher Education Access Route. For the purposes of this Report it will be known by the most recent title the Higher Education Access Route (HEAR). Appendix A contains a set of maps showing the distribution of link schools and the year in which they joined the HEAR scheme.3 More information on New ERA is at http://www.ucd.ie/access/newera.htm


6


and after they enter third level education.

Prior to entry, New ERA aimed to tackle the socio-cultural barriers to higher level education experienced by

students from lower socio-economic groups through a variety of outreach measures aimed at creating awareness

and providing information to second level students. Pre-entry programmes of action focus on raising student

aspirations, creating an awareness of college, and the provision of academic support for the Leaving Certificate.

New ERA visits designated disadvantaged schools linked to the programme and provides information to students

about the HEAR scheme and its supports. It also organises pre-entry orientation programmes and shadowing

days where second level students followed a third level student through a day in the life of the university. On a

community level it has given presentations to meetings of parents and contributions to community based events.

UCD students provide one to one academic support for Leaving and Junior Certificate students on a voluntary

basis. In addition, Leaving Certificate revision workshops are organized for 6th year students. For younger students,

there is the “Uni4U Summer School” for 2nd year students where participants attend taster lectures, science labs,

field trips as well as sports and social activities.

An example of a collaborative outreach initiative which New ERA is a part of with DCU, DIT, NUI Maynooth and

TCD is the “Take 5” summer project involving participants from designated disadvantaged second level schools

attending each of the five participating higher education institutions for a day. This project aims to introduce

students to the physical, academic, cultural, sporting and social environments of the different institutions

through a range of activities, such as academic workshops, laboratory sessions, and project group work.

New ERA’s outreach activities are not confined to secondary schools, there are also activities involving primary

schools under the “Steeping Stones” programme. This programme works with 31 primary schools that feed into

the linked secondary schools. Activities include a visit to the campus for 5th class students.

Students must apply to New ERA through the HEAR scheme and through the CAO scheme. Applicants must

submit supporting documentation to verify their socio-economic status, references from their school teachers

and meet basic course requirements set by the university. Students are initially deemed eligible for HEAR on the

basis of their socio-economic and once they are deemed academically eligible, are considered for a place on the

New ERA scheme. These eligible students falls into two categories. The first group of students are admitted to the

university under the normal clearing system used in Ireland, which is a nationally administered system4. About

45% (of about 100-140 students per annum) of New ERA students attain sufficient points to meet the minimum

CAO points level and are allocated a place on their preferred course in the usual manner. These students are

known as Merit students.

The remainder of applicants receive preferential treatment in attaining their place in university. These students did

not meet the minimum CAO points level for their chosen course. Instead there are a certain number of places on

each degree programme reserved for these students. These students are known as Direct students. To be offered

one of these places they must meet minimum matriculation and course requirements, but receive a concession

of up to 20% on the CAO entry points required for the programme. In addition, further information showing their

personal circumstances are considered (such as the level of educational qualifications of their parents) as well

as references from teachers. The number of minimum reserved places on each course is calculated on the size

of each course and is relatively fixed. If there is a surplus of suitable and eligible applicants for these places, the

limited places are awarded on the basis of points.

Since 1997, third level tuition has been free to all students in the Republic of Ireland. All students also pay a

registration fee of several hundred Euro, however, for students who are in receipt of the Higher Education Grant,

this fee is covered by their local authority. As almost all New ERA students also receive this grant, they do not

pay this administration fee.

4 This is based on a supply-and-demand based system of allocating degree course places. Students are ranked by converting their leaving certificate results into points using a common scale. The scale takes the best six subjects and has a range of 0 to 600 in increments of 5. The minimum points level for a degree programme fluctuates from year to year. Applications are anonymous in that it does not involve interviews or submission of a personal statement.


7

Post-entry, financial support is provided to all New ERA students in the form of a top-up grant (in addition to

the usual local government grant, which they would normally receive). New ERA also provides a contribution

towards additional course such as lab coats, stethoscopes, etc. In the case of students attending the Quinn School

of Business, this aid is quite considerable as it covers the cost of laptop computers, which are compulsory for all

students in the business school. Students also receive book tokens to contribute towards the cost of textbooks.

Post-entry educational and socio-cultural supports include free additional tuition (if required), mentoring from

student advisors, and a pre-term orientation week where students live on campus with other New ERA students

to encourage early social and academic integration. Post-entry, New ERA aims to tackle social isolation through

personal support and advice, facilitating group events and occasional social events, and by monitoring and tracking

students’ progress. New ERA students can mix with one another and share their experience of university.

1.4 overview of ReportThe remainder of the report is organised as follows: Chapter 2 provides a literature review on the effectiveness

of different forms of international access programmes based on rigorous evaluation. Chapter 3 presents the

methodology and data used to conduct the evaluation. Chapter 4 presents the results of the evaluation of student

exam results. Chapter 5 primarily studies the effect of the programme on progression to university. Issues such as

fluctuations in financial aid are also examined. Chapter 6 presents the report’s recommendations.

CHAPTER 2

LITERATURE REVIEW

Chapter 2literature Review

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review

9

2 literature Review This chapter briefly outlines some of the background to the present study. Section 2.2 describes the policy

background in Ireland noting how thinking on dealing with educational disadvantage in general, and access

programmes in particular, has evolved. It also discusses some of the evaluations of policy that have been carried

out in this area. It should be noted at the outset that these evaluations, which are essentially qualitative reviews

of the issues, are very different from the present exercise which is about quantifying the effectiveness of a

particular intervention i.e. the New ERA programme. In the absence of good estimates of the effectiveness of

access initiatives it is unclear how one can go about designing interventions to tackle educational disadvantage.

It is striking that quantitative assessments of the effectiveness of educational interventions are extremely rare in

Ireland and it is this gap which the present study fills. Studies such as this are more common internationally and

Section 2.3 describes some of these key studies. The aim is not to provide an exhaustive survey of the literature

but to provide a general overview of the research literature and some representative results.

2.1 Policy Context: Access in IrelandA number of government acts and policy documents provide the historical overview of access policy in Ireland which

led to the development of access programmes and initiatives. Key developments included the Higher Education

Authority Act (1971) and the Universities Act (1997) which brought equality in accessing higher education to

the foreground. Several reports were produced that played key roles in setting the agenda for the development

of access initiatives through identifying barriers to higher education and providing recommendations as to how

these barriers could be addressed, these include the Green Paper: Education for a Changing World (1992), the

White Paper: Charting our Educational Future (1995), and The Report for the Action Group on Access to Third level

Education (2001).

The Green Paper: Education for a Changing World (1992) argued that objectives for improving transfer rates to

third level education should include

The development of direct links between third level institutions and selected

schools… [and] … the development of support and access programmes to

increase access and improve retention.

The White Paper Charting our Educational Future (1995) recognised the effectiveness of securing a set number of

places for those from disadvantaged backgrounds as a means of tackling educational disadvantage. The aim was

to admit an additional 500 students from lower-socioeconomic groups into third level institutions each year for

the subsequent five years.

According to Carpenter (2004), the White Paper aims for this policy to be delivered by the third level sector as

a whole and not the universities alone. Furthermore, it was recommended that undergraduate fees in higher

education institutions be abolished from 1996/97, that the criteria for the allocation of student grants be revised,

and that the value of the grant be increased. In order to support students at second level the White Paper stated

that:

each third level institution will be encouraged to develop links with

designated second-level schools, building on existing good practice.

In the short term, it was envisioned that all designated disadvantaged second level schools would become linked

to a third level institution which would support programmes to create awareness of the opportunities for and

advantages of third level education. Some of the strategies outlined include ‘awareness seminars’ and open days

for students. Institutions were also encouraged to help students make the transition from second level education

to third level through post entry support programmes. It should be noted that undergraduate fees were indeed

subsequently abolished in 1996/97. However it is unlikely that this would have led to wider participation by

people from a low income background as they would normally have been in receipt of local authority grants and

hence not liable to pay fees.


10

2: literature Review Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review

The Report for the Action Group on Access to Third level Education (2001) led to the development of the National

Office for Equity of Access to Higher Education in 2003 with the mandate of coordinating efforts to tackle

educational disadvantage and to develop a national strategy. This office is now tasked with facilitating access to

education for groups who are typically underrepresented in higher education.

Evaluations of Irish Access Initiatives:

To date several evaluations have been conducted in an attempt to develop best practice in dealing with access

and participation in Irish higher education. Previous evaluations of such initiatives include: Skilbeck and Connell

(2000), Osborne and Leith (2000) and the Report of the High Level Group on University Equality and Policies,

commissioned by the HEA Equality Review Group in 2004.

Skilbeck and Connell’s report Access and Equity in Higher Education: an International Perspective on Issues and

Strategies (2000) recognizes that the issue of equity in higher education is not just educational but also economic,

cultural and social. It states that educational polices alone cannot abolish educational disadvantage and

recommends that further progress will require more coherent, coordinated approaches across several sectors. It

argues that opportunities for access should be extended through all levels of the education system. The report

recognizes that well-targeted financing will continue to play an important role in equity strategies, but that it is

inefficient to admit large numbers of students and then to accept high failure and drop-out rates.

Osbourne and Leith (2000) in Evaluation of the Targeted Initiatives on Widening Access for Young People from

Socio-economically Disadvantaged Backgrounds recommend that a national strategy should provide a broad

framework in which

individual universities and other institutions can evolve distinctive

practices which reflect their own circumstances.

They also recommend that a national strategy funded by the state needs to have measures in place for progress

to be assessed.

According to the Report of the High Level Group on University Equality and Policies (2004), while there are benefits

for individuals involved in targeted initiatives, the scale and impact of these special initiatives is very small:

according to one university disadvantaged students represent only 3% of the student body. This leads to the

statement that there can be little prospect in the short term that the continuation of these initiatives, even on a

substantial level, will bring about radical changes in representation. However it was recognized by the report that

these initiatives do keep the issue of representation to the forefront, demonstrating what can be achieved on the

ground when resources are specifically targeted at disadvantaged groups.

Throughout the consultation process for the report, the idea emerged that access could be improved by

incorporating weighting in favour of students who are severely disadvantaged, and that, in establishing goals for

the education system generally, the focus should

Shift from the concept of participation (e.g. prevention of early leaving

from school programmes leading to the Leaving Certificate) to benefit

(outcomes for the individual, irrespective of where the learning has taken

place).

The team also highlighted that:

The point was made repeatedly to the review team that intervention should

be made at an earlier stage in second level education.

The report recognized that there was a need for on-going evaluation and improved data collection. It was

acknowledged that the cause of education disadvantage was not limited to the education system. Parents, peer

groups and community factors all play a part in the formation of student expectations and aspirations.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

11

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review

Throughout previous evaluations of access initiatives in Ireland there was widespread recognition that

educational disadvantage did not begin at second or third level and the identification and support of students

at risk needed to begin at an earlier stage in their education. Many studies have been produced which support

this view. According to Carneiro and Heckman (2003), the bulk of the evidence indicates that public expenditure

would be more efficient if resources invested in human capital were more strongly directed towards younger

members of the population. A research paper for the NESF entitled The Economics of Early Childhood Care and

Education (2006) finds that the greatest societal returns to education come from investment in early or primary

education, whereas, in the case of investment in higher education, societal gains are smaller and the returns to

the individual are higher. Researchers at the Geary Institute (UCD) are currently carrying out an evaluation of the

effectiveness of an early childhood intervention programme called Preparing for Life which aims to improve the

school-readiness of children from designated disadvantaged backgrounds entering primary school.

2.2 Evaluating the effectiveness of financial and social supportIt has long been recognised that the rate of return for individuals who invest in higher level education is high. A

study by Harmon, Oosterbeek and Walker (2003) presents estimates on the returns to earnings from an additional

year of education in the U.K. which vary between 7% and 15%, depending on the estimation procedure used. They

also show that the returns to education are higher in Ireland and in the U.K. than in the rest of Europe. However,

not all groups in society invest equally in third level education regardless of the high returns to education and in

spite of the fact that much effort has been made in recent decades to increase the participation and completion

rates of students from underrepresented backgrounds.

Many governments have implemented policies which aim to promote ‘equality of opportunity’ for students

from disadvantaged backgrounds. Universities have established access programmes that are designed to

boost enrolment and persistence of these groups. However the effectiveness of these programmes needs to

be examined in more detail to understand why underrepresentation, withdrawal and non completion remain

enduring problems. The key areas focused on when discussing the effectiveness of access programmes are access/

entry/progression, retention, exam performance, withdrawal/non-completion and graduation.

There are numerous access initiatives worldwide which aim at affecting the outcomes mentioned above, the

majority being implemented in Anglo-Saxon countries. Many of these initiatives focus on tackling barriers to

access, financial barriers in particular, and for this reason there is a large quantity of literature that concentrates

on the effectiveness of financial aid. The two main types of financial aid are need-based and merit-based aid.

Eligibility for needs-based aid is based on certain criteria, such as means testing, and often takes the form of a

grant. Needs-based financial aid is designed to improve the enrolment, retention and graduation rates of students

from socio-economic or minority groups that are often under-represented in higher level education.

Several studies have been carried out to assess the effectiveness of financial aid on improving the outcomes of

students typically under-represented in higher education. A study by Dynarski (2003) found that such aid has

a significant effect on the outcomes of these students. In order to estimate the true effect of aid on student

outcomes the study examines the abolition of the Social Security Benefit Program in the U.S. which ran from

1965 to 1982 and, during this period, paid for millions of students to go to college. This was one of the most

dramatic changes in college aid allocation that has ever occurred in the U.S. Using this policy change, Dynarski

found that needs-based aid significantly affects the probability that a person will enrol in third level education.

The results show that giving a grant of $1,000 to an individual increases the probability that they will attend

college by approximately 3.6 percentage points.

Other studies have also found a positive relationship between needs-based aid and college enrolment. McPherson

and Schapiro (1991) use data on enrolment, tuition and financial aid for population subgroups in the U.S. over

the 1974 – 1984 period and find that an increase in the net costs of college attendance has significant negative

impacts on college enrolment for low income white families. The net cost to students will be affected by the level

of fees and any remission of fees payable by students. Their results also found that there was no significant effect

on higher income families who are not cash constrained.


12

2: literature Review Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review

Brock (2006) studied the effects of a Louisiana needs-based scholarship programme on course completion and

exam performance of low-income parents attending community college. Students were given scholarships of

$2,000 for the year if they attended college at least half-time and attained, on average, a C grade. Students were

randomly assigned to a programme group that could avail of the scholarship, and a control group that could

not. While both groups could avail of counselling services, the programme group were obliged to attend student

counselling in order to receive the financial aid. The programme group also benefit from their counsellors having

a lower caseload. The primary focus of the counsellors was monitoring the academic performance of students

and issuing the scholarships. The difference between the outcomes of the two groups represents the effect of the

programme. The results show that students who were assigned to the programme group were more likely to be

full-time college students, passed more college courses and earned more credits, and were more likely to register

for their second and third years of college.

Bettinger (2004) studies the relationship between needs-based grants and college drop out behaviour and finds

that there is a significant relationship. This study used data related to the Pell Grant programme which is the

largest means-tested financial aid programme available to students across the U.S. The study exploits differences

in the amount of aid paid to students over time to estimate the effect of the grant. The results show that a student

whose grant increases is less likely to withdraw from higher level education; more specifically the results imply

that a $1000 increase in a student’s grant reduces the probability of withdrawal by nine percentage points.

More recently needs based financial aid is being coupled with other forms of aid such as outreach and intervention

programmes that aim to influence access and retention. A study by Bergin, Cooks and Bergin (2007) examines a

programme that aims to increase the participation of youths from typically under-represented groups in higher

education. Students with a B average grade were randomly assigned to a programme or control group. Students

assigned to the programme group participated in activities to increase college awareness; they were also given

academic and financial support. The results showed that the programme did not have a significant effect on

enrolment rates, nor did it improve their high-school results or increase their self-esteem. However, the authors

found that programme participation did increase a student’s desire for further education.

Angrist, Lang and Oreopoulos (2009) conducted an evaluation in which first year college students were

randomly assigned, subject to their written consent, to one of three programme groups or to a control group.

One programme group was given financial support, another was given academic support and another was

given a combination of the two. The evaluation analysed the outcomes of students from each of these three

groups relative to the control group. The results showed that while the effect of academic support was small

and insignificant, both groups in which financial aid was given had positive and significant results, the effect

was stronger in the case where students were given a combination of financial and academic support. Further

analysis showed that the effect is particularly strong for women and that this is driving the overall result. The

programme had, in fact, no significant impact on male outcomes. This result is similar to that of Dynarksi (2008)

who analyses the HOPE programme and finds stronger effects for women. The findings of the Angrist et al (2009)

are particularly interesting as they showed that a combination of social and financial supports are more effective

than either intervention alone.

Finally, Lesik (2006) examines the effects of an academic support programme on student retention. The study

measures the impact of a mathematics programme on the withdrawal rates of students who are, on average,

comparable in all aspects except that one group participate in the development mathematics programme and

the other does not. Students were assigned to the programme based on their results in a mathematics exam

which students must take prior to their first year of college. A cut-off was set and students who scored below

this level were obliged to participate in the programme. The vast majority of students in this study complied

with this assignment rule. Lesik estimated the causal impacts of the programme by comparing the outcomes

of students with results just above and just below the cut-off. The study found a positive relationship between

developmental mathematics programme and student retention: students who participated in the programme

were found to be at a significantly lower risk of withdrawing from third level education.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

13

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 2: literature Review

Overall it has generally been found that social and academic supports have a positive effect on student enrolment

in third level education, on exam performance and on retention rates. The current evaluation of the New ERA

programme, which provides students with a combination of social and financial supports, makes an important

contribution to the existing literature on the effectiveness of educational interventions. It is the first quantitative

evaluation of an access programme that has been carried out in Ireland to date.

The aim of the New ERA programme is to promote and support the participation of students who are typically

underrepresented in higher education, and, using rigorous econometric techniques, the current evaluation aims

to assess the impact of the programme on the outcomes of these students. Evaluations such as this are important

because, in analysing the effectiveness of access initiatives such as New ERA, methods for improving the overall

design of educational interventions can be extrapolated.

Chapter 3Evaluation methods and Data


15

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 3: Evaluation methods and Data

3 Evaluation methods and Data3.1 Evaluation DesignThe key aim of all evaluations is to determine whether the social programme or policy is effective. This involves

finding the counterfactual: what would have occurred in the absence of the programme? In regard to the New

ERA programme, the evaluation needs to determine how the treated students (i.e. those who received the

programme) would have performed if they were not New ERA students. The evaluation addresses how the New

ERA students would have performed in terms of university entrance, first year exam results and final graduation

rates, if they had not taken part in the programme.

A simple evaluation design which compares New ERA students to the general population of the university will

not reveal the treatment effect of the programme, as the estimates of the programme may be biased if the

differences that led one group to enter the New ERA programme and the other group to enter the control group

may also be related to the outcomes of interest, i.e. exams performance etc. For example, New ERA students

may have different characteristics to the general UCD student body as they are selected into the programme

based on certain socio-demographic criteria. New ERA students are, by design, from lower socio-economic

backgrounds, with lower parental education and income. These characteristics are likely to influence a student’s

performance in university. Therefore a direct comparison of the New ERA students to the general student body

would not result in a reliable estimate of the impact of the New ERA programme.

Randomised control trials (RCTs) are considered the ‘gold standard’ in evaluation research (Burtless, 1995) as

they allow one to closely approximate a true counterfactual, i.e. what would have occurred in the absence of

the programme. The use of random allocation to generate unbiased control groups ensures that any observed

differences between the treatment and control group outcomes are likely due to the intervention rather than any

other factors (Oakley et al. 2003). However, RCTs are still relatively rare within the social sciences, especially

in Ireland where rigorous policy evaluation is a relatively new phenomenon. In addition, random assignment is

often not ethically or practical feasible, as is the case for the New ERA programme which has not been operating

on the random assignment of students. Therefore, an RCT evaluation design cannot be used. Evaluations of social

programmes have therefore relied on quasi-experimental methods which simulate the conditions of an actual

experiment without using random assignment (Angrist and Krueger, 1999). The aim of quasi-experimental designs

is to identify a suitable control group which is similar to the treatment group on all observed and unobserved

characteristics, apart from their lack of participation in the programme. Quasi-experiments are also known as

“natural experiments”.

The effectiveness of the New ERA programme is therefore evaluated using the quasi-experimental design. This

evaluation strategy exploits the gradual expansion of New ERA over time to identify a suitable control group.

New ERA has grown in the number of students that participate, but also in the number of “link schools”. By

comparing the outcomes of New ERA students to students who went to New ERA linked schools before they

became linked, we can determine if the programme has an impact. This control group is therefore used to

“difference out” confounding factors and isolate the treatment effect, i.e. the impact of New ERA. The technique

used is commonly employed in the programme evaluation literature. A recent example in a similar context being

Lavy and Schlosser (2005) who evaluate an Israeli education programme aimed at improving exam grades.

3.2 Selecting the Control Group To carry out this technique the control group must be as close to the treatment group in as many respects as

possible. To be eligible for inclusion in the New ERA programme a student’s family has to meet four criteria:

1) income eligibility

2) educational criteria

3) socio-economic status

4) student attended a “link” school.


16

3: Evaluation methods and Data Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 3: Evaluation methods and Data

FAmIlY INComE

The income eligibility for the New ERA programme corresponds to the eligibility for the regular means tested

Local Authority grant which is available to all students whose family income is under these income thresholds,

regardless of the status of the school they attended. Therefore, as the data does not include family income, one

of the selection criteria for choosing the control group is based on receipt of the state grant.

PARENTAl EDUCATIoN & SoCIo-ECoNomIC STATUS

In order to be eligible for the New ERA programme there must be no previous history of progression to higher

level education in the family i.e. students’ parents must not have gone to third level. However, as measures of

parental education are not available in the data, socio-economic status is used as a proxy. The social-economic

status criteria for programme eligibility include unskilled manual, semi-skilled manual, skilled non-manual, and

non-farming agricultural workers. Students whose parents are higher professionals, lower professional, employers

and managers are not eligible for New ERA. It may be the case that there are students whose parents have

some experience of higher level education in the remaining social-economic groups, although the assumption

is made that this is not the case in general. There is anecdotal evidence that farmers and self employed people

circumvent the rules on grant eligibility (Department of Education and Science, 1993). Therefore farmers are

excluded from the control group. It is not possible to identify self-employed people using the socioeconomic

categories observed in the data.

lINkED SChool

In order to be eligible for the access programme, students must have attended a “link school”. When the

programme began in the late 1990’s, certain schools were chosen from the DEIS list, the government’s list of

officially designated disadvantaged schools, and over time more schools from the government’s list were added

to the scheme when the funding allowed it. However, there is a small group (10%) of schools who participate in

HEAR but are not included on the DEIS list. These schools are located in rural areas of the country which have

Objective 1 status under the EU structural funds. Schools are included on the official list of disadvantaged schools

based on a range of socio-economic and educational indicators such as unemployment levels, housing, medical

card holders and information on basic literacy and numeracy, and also some school level factors such as pupil-

teacher ratios. Initially New ERA choose schools which were in their own defined catchment area which were

predominantly highly deprived urban areas.

Given the data available for the analysis only covers 1999 to 2004, schools linked to New ERA in 1999 or before

represent an “always” covered group, i.e. students from these schools, who have satisfied the other eligible

criteria, are always in the treatment group in the analysis. In Appendix A, a set of maps are presented which show

the distribution of link schools across the country and how the HEAR scheme expanded over time. Those schools

who were included in the programme for the first time in 2005, or after, represent a “never” linked group, i.e.

in the data no eligible students from these school have received treatment. Table 1 shows that that number of

schools linked to the New ERA programme has increased substantially over time, however some schools closed

or were amalgamated during the period included in the analysis. In 1999 New ERA was linked to 21 schools with

a further 6 schools joining in 2000. 2001 saw the introduction of the HEAR scheme which resulted in 125 new

schools becoming part of the programme. There were further increases in subsequent years with the largest

expansion occurring in 2003 when 63 new schools joined. By 2007 a total of 310 schools had attached to the

HEAR programme5.

Essentially this study compares New ERA students with students who would have been eligible to join the New

ERA programme had their school been participating in the HEAR scheme at the time they started university.

The reliability of the results in Chapter 4 rests on several assumptions. One is that there was no change in the

characteristics, e.g. student-teacher ratios, quality of teaching, facilities, etc, of the schools across the years.

Given that the period under analysis only covers six years, it is plausible that the majority of schools will not have

significantly changed. Furthermore, it is also assumed that the schools which joined later were not any different

from the schools that joined earlier as this would result in our treatment and control group being different from

one another. If the more disadvantaged schools joined the programme first, then the results may be biased as

5 This is not the current number of active schools as some of the schools which had joined either closed or amalgamated.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 3: Evaluation methods and Data Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

17


the programme group (schools that were part of New ERA first) and the control group (schools that joined New

ERA later) may systemically differ. However, the introduction of HEAR in 2001 should have reduced this problem

as UCD then became linked to schools previously linked with other higher education institutions.

As the above assumptions may be strong, the econometric models estimated in Chapter 4 control as much

as possible for any observed differences in the control and treatment groups. For example, in all of the results

presented in Chapter 4, the Leaving Certificate points of students are taken into account by the econometric

models as the average level of points is different for treatment and control students and because Leaving

Certificate points have a direct effect on student achievement.

Inevitably there are unobservable and perhaps intangible factors that researchers cannot measure (e.g. school

“spirit”, dedication of staff, etc.). This would cause problems for the reliability of any estimated effects of New

ERA if these unobservable characteristics varied between the schools which joined New ERA/HEAR earlier and

those that joined later. The results in Chapter 4 and Chapter 5 do not allow for potential unobservable differences.

To examine if this is a problem, the analysis was repeated using the “difference-in-differences” technique which

is commonly used in econometrics to eliminate unobservable differences between groups. However, broadly

speaking the results are not different when using the differences-in-differences technique, thus the results

presented are reliable indicators of the true effects of New ERA/HEAR. Table 1 below shows how New ERA

evolved in terms of the number of schools joining over time i.e. becoming linked to UCD.

TABlE 3.1Number of schools joining the New ERA scheme.

Number of link schools N

1999 and before 21

2000 6

2001 (HEAR introduced) 125

2002 30

2003 68

2004 2

2005 or later 58

3.3 Description of DataThe study involves an analysis of multiple secondary datasets including the UCD administration database; UCD

exam database; New ERA HEAR database; HEAR school survey and Census data; and Department of Education

school level administrative data.

Data were obtained from the administrative records of UCD and the New ERA HEAR database. The data was

anonymised by the Irish Social Science Data Archive before it was provided to us for analysis. The administrative

data contains information on all undergraduates entering UCD from 1999 to 2004 inclusive, totalling about

30,000 observations. The UCD administration data was used to identify a suitable control group based on

differing characteristics of the students including prior academic achievement, parental socio-economic status,

grant-holder information and school level information. The New ERA HEAR dataset was used to identify the

Merit and Direct New ERA students and to fill-in information that was missing from the main administrative

database. Data from the UCD exams database was used to identify the key outcome variables, namely, retention

rates, first year exam performance and final graduation rates.


18


The HEAR school survey (Denny et al. 2009), conducted in 2008 to collect information on schools participating

in the HEAR scheme, was also used to examine progression rates from second level to third level. In addition, the

administrative data was matched to school-level administrative data made available by the Irish Department

of Education. Finally, census data, such as local unemployment rates, were matched to the school level using

electoral districts. This allowed us to ascertain whether the roll-out of the New ERA programme was random i.e.

whether schools in more/less disadvantaged areas joined the programme first.

A number of students were omitted from the analysis. These include Irish school leavers with missing school

information, international students, those from Northern Ireland and external candidates The working sample

therefore excludes those who have no school-level data and students who entered the university directly rather

than through the university central clearing system for school leavers (e.g. disabled students, certain mature

students, transfers from vocational courses, etc). For much of the analysis, those who switched courses or

repeated a year have had their later observation dropped. Unfortunately, it is not possible to identify students

who have transferred from other universities. A description of the data used and the New ERA sample is below.

3.4 Data limitationsThe dataset used in this study is larger and more wide-ranging in the information contained than any previously

evaluation of university access programmes. However, there are limits on the information in the dataset and

these place restrictions on the precision of the results and the type of questions that can be examined.

Eligibility for the programme is inferred from a binary variable indicating grant holding students. The New ERA

financial eligibility thresholds shadowed the eligibility for the full grant. However, the analysis cannot distinguish

if students are receiving the full grant or partial grants (75%, 50% or 25% of the full amount). Should there be

future studies examining New ERA or the local authority maintenance grant, it is recommended that UCD collect

data annually on the amount of grant received by students rather than just grant holdership.

If household income data were available in the administrative data, it would negate the need to infer New ERA

eligibility. It would also allow future researchers to examine heterogeneity in the effects of New ERA and other

policy intervention. For example, due to the lack of data, this study cannot examine if New ERA affects less well-

off students differently than better-off students.

It is not clear from the data whether some of those who are listed as failing the Summer exams may have in

fact withdrawn by that stage and vice-versa. It is recommended that UCD collect data in such a way that policy

makers can distinguish between students who formally withdraw, withdraw informally by not presenting for

exams and students who fail exams.

Furthermore, with regards to those who do withdraw, it is not known whether those who drop out are transferring

to other institutions or leaving education altogether. It would be helpful if students who formally withdraw were

asked why they were leaving and what their destination was.

Finally, it is noted that high quality secondary school level data is not available in Ireland. Data on schools’

exam results are not made available to researchers. Furthermore, data on transfer rates to third level institutions

for each school are not published regularly. While it is not suggested that such information is made publicly

available, if academic researchers had access to such data (while maintaining strict confidentially of the schools),

more informed policy advice could be given.


19


3.5 Description of SampleFigure 3.1 shows, of the total number of New ERA students in the sample, the percentage admitted each year6.

In total, there are 322 students in the sample, and 8.07% of these were admitted in the first year. The numbers

of New ERA students represent less than 5% of the total admissions to the university each year. As the years

progressed and the programme grew, a higher number of students were admitted to the New ERA programme

each year. There was a noticeably large increase in 2002 and again in 2004. Note that the large increase in 2002

was driven by the introduction of the HEAR programme in 2001. It could be speculated that the administrative

structures for gathering data on HEAR students were not fully in place until 2002.

The figure also provides the breakdown for Direct and Merit students. These sub-groups generally follow the same

pattern as the overall sample. There were a relatively low percentage of students admitted in 1999, with large

increases in 2002 and 2004. However there are three exceptions: less New ERA Direct students were admitted

in 2001 than in 2000, less New ERA Merit students were admitted in 2000 than in 1999, and less Merit students

were admitted in 2003 than in 2002. Nonetheless, the overall trend is the same; the increase was particularly

large for the Merit group in 2002, when the figures increased from 9.4% to 24.2%, and for Direct students in

2004, when the percentage increased from 22.5% to 35.3%.

6 Descriptive statistics in Tabular form are included in Appendix B.

Figure 3.1: Description of the New ERA sample

!

!

"!

#!!

#"!

$!!

%&'()*+,

-+%).+/01+23&

4)532+6

4'733)

4

#888 $!!! $!!# $!!$ $!!9 $!!: ;,36<=+6<<+>*2

"#$%&'(

)%$#'(

1999 2000 2001 2002 2003 2004 Total All

Years Direct 11 19 17 26 39 61 173 Merit 15 12 14 36 30 42 149

!

!

"!

#!!

#"!

$!!

%&'()*+,

-+%).+/01+23&

4)532+6

4'733)

4

#888 $!!! $!!# $!!$ $!!9 $!!: ;,36<=+6<<+>*2

"#$%&'(

)%$#'(


20


SoCIo-DEmoGRAPhICS

Figure 3.2 provides information on the gender breakdown for New ERA students and for the general UCD student

body. It shows that overall there are a lower percentage of male students attending UCD: 46% of the general

student body is male. This trend is even stronger amongst the New ERA students; with a higher percentage

of female students in the New ERA programme overall and a higher proportion among the Merit students

specifically: 40% of New ERA Merit students are male, while only 36% of Direct students are male.

!"#$"%

&'%"() *"%') +,#-+"./012

*34" 56789 8:79; 8<7=>

?"@34" 657<> <=7;5 <87:9

!

"!

#!

$!

%!

&!!

'()*(

+,

-.)(*, /()., 01+20(34567

*34"

?"@34" Figure 3.2: Socio-demographics: Gender

!"#$"%

&'%"() *"%') +,#-+"./012

*34" 56789 8:79; 8<7=>

?"@34" 657<> <=7;5 <87:9

!

"!

#!

$!

%!

&!!

'()*(

+,

-.)(*, /()., 01+20(34567

*34"

?"@34"

Direct merit Non-New ERA

Male 36.42 40.27 45.98

Female 63.58 59.73 54.02

!"#$"%

&'%"() *"%') +,#-+"./012

*34" 56789 8:79; 8<7=>

?"@34" 657<> <=7;5 <87:9

!

"!

#!

$!

%!

&!!

'()*(

+,

-.)(*, /()., 01+20(34567

*34"

?"@34"

!"#$"%

&'%"() *"%') +,#-+"./012

*34" 56789 8:79; 8<7=>

?"@34" 657<> <=7;5 <87:9

!

"!

#!

$!

%!

&!!

'()*(

+,

-.)(*, /()., 01+20(34567

*34"

?"@34"


21


Figure 3.3 gives information on the distribution of New ERA students and the general non-New ERA student

body by socio-economic group. While the most strongly represented group amongst the general student body

are Higher Professionals, there are no New ERA students in this group, as expected based on the programme’s

inclusion criteria. New ERA students are most strongly represented among the lower socio-economic groups

with the Salaried Employees group encompassing the majority of New ERA students (25.3% of New ERA Direct

students and 19.7% of New ERA Merit students), while this group accounts for 16.2% of the general student

body.

Figure 3.3. Socio-demographics: socio-economic status

Farmers & Agricultural WorkersProfessionals & ManagersSalaried EmployeesIntermediate & Other Non-ManualSkilled & Semi-skilled Manual WorkersUnskilled Manual Workers

Non-New ERA

Direct merit

Direct % merit % Non-New ERA %

Farmers & Agricultural Workers 1.15 1.41 11.80

Professionals and Managers 0 0 62.65

Salaried Employees 25.29 19.72 16.24

Intermediate and Other Non-Manual 22.98 23.94 2.91

Skilled & Semi-skilled Non-Manual 36.78 33.80 5.88

Non-skilled Manual 13.79 21.13 0.52


22


UNIvERSITY FACUlTY

Figure 3.4 summarises the percentage of New ERA Direct and Merit students per faculty compared to the general

study body7. For each of the three groups, the highest proportions of students are enrolled in the Arts faculty, with

41% of all university students and 44% of New ERA Merit students being located in Arts, yet a lower proportion

of New ERA Direct students are in the Arts faculty (34%). For the Science faculty there are large differences

across all the three groups. While a high proportion (27.5%) of New ERA Merit students are studying science, and

only 12.9% of the general student body and just 6.4% of New ERA Direct students enrolled in the Science faculty.

Therefore, almost four times as many Merit students are studying Science compared to Direct students. There

are also differences in the percentage of students enrolled in the Commerce faculty, with a significant proportion

of New ERA Direct students (20.8%) studying Commerce, compared to 8% of Merit students and 12.7% of the

rest of the study body. Finally, a higher proportion of Direct students are in the Health Sciences compared to

the Merit students and the general student population. The proportion of students in the remaining faculties is

broadly similar across groups.

Overall, New ERA Direct students more closely resemble the general student in terms of the distribution across

faculty. This may be explained by the relatively high number of Merit students studying lower points courses,

such as Arts and Science. As these are relatively low points courses, many New ERA students do not need to avail

of the points concessions for these courses. The distribution of Direct students across faculty may also resemble

the total distribution across the university as the number of places reserved for Direct students depends on the

number of students in the different faculties.

7 Re-structuring in UCD has replaced the old system of Faculties with a smaller number of Colleges.

Figure 3.4: University Faculty

AgricultureArtsCommerceEngineering&ArchitectureInterfacultyLawMedicineScienceVeterinary MedicineHuman Sciences

Non-New ERA

Direct merit

Direct % merit % Non-New ERA %

Agriculture 0.58 4.70 5.29

Arts 34.10 43.26 40.72

Commerce 20.81 8.05 12.74

Engineering & Architecture 6.36 6.04 9.46

Interfaculty 5.20 1.34 4.41

Law 2.89 2.01 2.99

Medicine 13.87 2.01 5.85

Science 6.36 27.52 12.95

Veterinary Medicine 1.16 1.34 1.81

Human Sciences 8.67 3.36 3.78


23


FIRST YEAR SUmmER ExAmS

Figure 3.5 provides information on first year exam outcomes for New ERA students and the remaining student

body. A higher percentage of New ERA students, both Direct and Merit, were registered for their first year summer

exams compared to the general student body. Approximately 97% of New ERA students sit their summer exams

compared to 94% of remaining students. While Merit students outperform the general study body in terms of

the first year pass rate, only half of Direct students pass their first year summer exams on the first attempt.

This suggests that these two groups may be different in an important way and that the analysis should distinguish

between them. In what follows we see that this is indeed the case.

It is important to bear in mind in looking at these descriptive statistics that one is simply comparing averages

between groups. One cannot conclude that there is any causal relationship.

FIRST YEAR ExAm GRADES

In regards first year exam results, figure 3.6 shows that a higher proportion of Merit students achieve first class

honours (10.3%) than Direct students (only 3%) and the general student body (8.9%). While the proportions

achieving third class honours are roughly the same across all three groups. As noted in above, the percentage of

Direct students failing the summer exams is far higher than either Merit or Non-New ERA students with 50% not

passing the summer examinations. This is further evidence that the Direct and Merit groups are different.

Figure 3.5: First Year Summer Exams

Figure 3.6: First Year Summer Exam Grades

1st Class 2nd Class Upper 2nd class lower 3rd Class/Pass Fail

Direct% 2.98 10.12 14.29 22.62 50

Merit% 10.34 13.1 28.28 21.38 26.9

Non-New ERA 8.89 16.31 21.68 23.77 29.35

!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($

1"#(2%&3 4(#"%&3 567&5(89-:;&3

<$%&=>)$$ ?@AB <C@DE B@BA

?70&=>)$$&FGHH(#I?70&=>)$$&FGHH(#I <C@<? <D@< <J@D<

?70&=>)$$&FK68(#I?70&=>)$$&FK68(#I <E@?A ?B@?B ?<@JB

D#0&=>)$$L&M)$$D#0&=>)$$L&M)$$ ??@J? ?<@DB ?D@NN

!)">& OC ?J@A ?A@DO

0

10

20

30

40

50

60

!"#$%&'"" ()*$%&'""$+,--./0 ()*$%&'""$+123./0 4/*$%&'""5$6'"" 7'8&$

1"#(2%&3

4(#"%&3

567&5(89-:;&3

6./9.)#

!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($

1"#(2%&3 4(#"%&3 567&5(89-:;&3

<$%&=>)$$ ?@AB <C@DE B@BA

?70&=>)$$&FGHH(#I?70&=>)$$&FGHH(#I <C@<? <D@< <J@D<

?70&=>)$$&FK68(#I?70&=>)$$&FK68(#I <E@?A ?B@?B ?<@JB

D#0&=>)$$L&M)$$D#0&=>)$$L&M)$$ ??@J? ?<@DB ?D@NN

!)">& OC ?J@A ?A@DO

0

10

20

30

40

50

60

!"#$%&'"" ()*$%&'""$+,--./0 ()*$%&'""$+123./0 4/*$%&'""5$6'"" 7'8&$

1"#(2%&3

4(#"%&3

567&5(89-:;&3

6./9.)#

!"#$%&'

()#&*

+,

,(#&-

.),

&/#)

0($

!"#$%&'

()#&*

+,

,(#&-

.),

&/#)

0($

!"#$%&'

()#&*

+,

,(#&-

.),

&/#)

0($

1"#(

2%&34

(#"%&3

56

7&5

(8

9-:

;&3

<$%&=

>)$$

?@A

B<

C@D

EB

@BA

?7

0&=

>)$$&FG

HH

(#I

?7

0&=

>)$$&FG

HH

(#I

<C

@<?

<D

@<<

J@D

<

?7

0&=

>)$$&FK6

8(#I

?7

0&=

>)$$&FK6

8(#I

<E

@?A

?B

@?B

?<

@JB

D#0

&=>)

$$L&M)$$

D#0

&=>)

$$L&M)$$

??

@J?

?<

@DB

?D

@NN

!)">&

OC

?J

@A?

A@D

O

0 10 20 30 40 50 60

!"#$%

&'""

()*$%&'""$+,

--./0

()*$%&'""$+12

3./0

4/*$%&'""5$6

'""

7'8&$

1"#(

2%&3

4(#"%&3

56

7&5

(8

9-:

;&3

6./9.

)#

!"#$%& '$#"& ()*+($,-./0

12&-1344$#-.5246-712&-1344$#-.5246-7 89:;; 89:<= 8<:>8

?266-1344$#-7?266-1344$#-7 @A 9<:; 9A:>@

0

10

20

30

40

50

60

70

80

90

100

97.11 97.32 93.69

50

73.1 70.65

Direct Merit Non-New ERA

12&-1344$#-.5246-7

?266-1344$#-7

PercentPercentage of students w

ho sat and passed their sum

mer exam

s


24


Figure 3.7: First Year Autumn Exams

FIRST YEAR AUTUmN ExAmS

For students who either do not sit or who fail the summer exams, they had (prior to modularisation) the option

of repeating the exams in the autumn. Figure 3.7 shows that while a slightly higher proportion of Direct students

sit the autumn exams, the pass rate is broadly similar across all three groups, with 69% of Direct students and the

non-New ERA students passing the repeat exams and 57% of New ERA Merit students passing.

While the data for the summer exams shows that Merit (but not Direct) students were broadly similar to non-

New ERA students (as one might expect given that they also enter via the CAO route) this pattern is reversed

here with Direct students and non-New ERA having very similar pass rates. Of course, students sitting autumn

exams are not representative of the student body in general since only a minority need to repeat.

PRoGRESSIoN To SECoND YEAR

Figure 3.8 shows the proportion progressing to second year. The proportion of students is similar across all three

groups (83-88%), despite the high failure rate of the Direct students on their first exam attempt. This suggests

that any problems that the New ERA students (particularly Direct students) encountered in the summer exams

have been largely ameliorated when the autumn exams are taken into account.

!"#$%&'()#&*+%+,-&./),$!"#$%&'()#&*+%+,-&./),$!"#$%&'()#&*+%+,-&./),$

!"#$%& '$#"& ()*+($,-./0()*+($,-./0

12&-03&34*-5 6789: ;:8<9 ;<86:

=2>>-03&34*-5 <;8?7 7?87; <68@9

0

12

20

32

400

5"#(6% 7(#"% 89-:8(;&.<*

12&-03&34*-5

=2>>-03&34*-5

Direct merit Non-New ERA

Sat Autumn % 95.24 84.62 86.94

Pass Autumn % 68.75 57.58 69.32

!"#$%&'()#&*+%+,-&./),$!"#$%&'()#&*+%+,-&./),$!"#$%&'()#&*+%+,-&./),$

!"#$%& '$#"& ()*+($,-./0()*+($,-./0

12&-03&34*-5 6789: ;:8<9 ;<86:

=2>>-03&34*-5 <;8?7 7?87; <68@9

0

12

20

32

400

5"#(6% 7(#"% 89-:8(;&.<*

12&-03&34*-5

=2>>-03&34*-5

Percentage of students who sat

and passed their autumn exam

s


25


FINAl YEAR GRADUATIoN

Figure 3.9 shows the percentage of students that graduated from university and the final degree classification

for New ERA and non-New ERA students. It should be borne in mind that we have much less data on students in

their final (degree) year. This means that it is much more difficult to discern the impact of New ERA on this group.

Nonetheless, it can be seen below that the Merit students have the highest success rate with 88.9% of students

from this group graduating. This is much higher than the general student body of whom about 78% graduate. The

percentages of students graduating from the Direct group, by contrast, is much the same as the general student

body.

Figure 3.8: Progression to Second Year

!"#$%"#&&'$()*$)+(,)-#."!"#$%"#&&'$()*$)+(,)-#."!"#$%"#&&'$()*$)+(,)-#."

!"#$%& '$#"& ()*+($,-./0

!"$%"#&&)*$)/#0$(,)1#.")212345 16327 11389

3

43

+3

53

63

73

83

93

:3

;3

433

!#"0#

(*

:+<96 :8<+4 ::<57

='"#0* >#"'* ?$(@?#A)BCD

Figure 3.9: Final Year Graduation Rate

!"#$%#&'()*"#&+,!"#$%#&'()*"#&+,

-'"+.&* /+"'&* 0()10+2*3450()10+2*345

!"#$%#&+$*6 7789: ;;8;< 778=

0

10

20

30

40

50

60

70

80

90

100

Percent

77.4288.89

77.5

Direct Merit Non-New ERA


26

3: Evaluation methods and Data Evaluating the Impact of the UCD New ERA Widening Participation Initiative

FINAl DEGREE ClASSIFICATIoN

The New ERA Merit students also perform best in terms of the degree classification awarded with 20.5% of

students from this group graduate with first class honours, compared to 4.3% of New ERA Direct students and

13.8% of the general student body. In addition, a higher proportion of New ERA Direct students receive lower

second class and third class honours compared to the other two groups.

As with the first year exams, there is a striking difference in the achievement of Direct and Merit students with the

latter being much closer to non-New ERA students. Almost one third of the Direct students get a pass/3rd class

honours, compared to less than 20% for the other two groups. At the other extreme, it is noticeable that a high

proportion of Merit students, about 20%, get first class honours.

The data illustrated in these graphs gives a general overview of the academic performance of the New ERA

students and one can see that in some cases they are quite different from the general student population. The

picture is complex with New ERA students sometimes outperforming other students and sometimes lagging

behind. It is essential to realize that these statistics are descriptive only: they cannot tell us how effective New

ERA is. For that, one needs a more sophisticated analysis and this is what is done in chapters 4 and 5.

Figure 3.10: Final Year Graduation Grade

!"#$%&'($)&*(+)((&,%$--"."/$0"1#!"#$%&'($)&*(+)((&,%$--"."/$0"1#!"#$%&'($)&*(+)((&,%$--"."/$0"1#

*")(/0&2 3()"0&2 41#54(6&789&2

:-0&,%$-- ;<=> =?<@: :A<BC

=#D&,%$--&EFGG()H=#D&,%$--&EFGG()H :C<@B =><;> =><?B

=#D&,%$--&EI16()H=#D&,%$--&EI16()H ;@<B: A?<BB AC<:

A)D&,%$--&J&K$--A)D&,%$--&J&K$-- A:<;A :><=A :><?@

!

"#

#$

%&

$'

&!

()*+)

,-

".-/012.. #,3/012../4566)*7 #,3/012../489:)*7 %*3/012../;/(2..

1st Class 2nd Class Upper 2nd class lower 3rd Class/Pass

Direct% 4.29 18.57 45.71 31.43

Merit% 20.51 29.49 30.77 19.23

Non-New ERA 13.78 29.07 38.1 19.05

!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($!"#$%&'()#&*+,,(#&-.),&/#)0($

1"#(2%&3 4(#"%&3 567&5(89-:;&3

<$%&=>)$$ ?@AB <C@DE B@BA

?70&=>)$$&FGHH(#I?70&=>)$$&FGHH(#I <C@<? <D@< <J@D<

?70&=>)$$&FK68(#I?70&=>)$$&FK68(#I <E@?A ?B@?B ?<@JB

D#0&=>)$$L&M)$$D#0&=>)$$L&M)$$ ??@J? ?<@DB ?D@NN

!)">& OC ?J@A ?A@DO

0

10

20

30

40

50

60

!"#$%&'"" ()*$%&'""$+,--./0 ()*$%&'""$+123./0 4/*$%&'""5$6'"" 7'8&$

1"#(2%&3

4(#"%&3

567&5(89-:;&3

6./9.)#

Chapter 4Results


29

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 4: Results

4 Results4.1 Interpreting the ResultsThis chapter presents the results of the econometric analysis examining the effects of the New ERA access

programme on a range of student outcomes. The outcomes that are being studied here are categorical (e.g.

whether a student passed their first year exams or what class of degree he/she received). So the study uses the

appropriate econometric techniques to estimate the effect of different variables on the probability of different

outcomes occurring. While the control group was selected to be as similar as possible to the New ERA programme

group, there may still be variation between the two groups which needs to be taken into account in the analysis.

Therefore the models also include a range of student characteristics which may obscure the effect of the New

ERA programme if they were omitted. These include university faculty, year of university entry, and number of

points attained in Leaving Certificate exams. Results are presented for the main outcomes of interest only.

The models estimate the treatment effect, which is participation in the New ERA programme, relative to the

outcomes of the control group. The control group include financially eligible students (i.e. grant holders), whose

parents are not professionals or employers and who attended schools that subsequently linked to the system.

Students from a farming background are excluded. The results reported on each figure are marginal effects and

associated p-values. Marginal effects show the impact of being in the New ERA programme group, compared to

the control group, on the probability of a achieving a particular outcome. The p-values represent the probability

that the result obtained is due to chance rather than a true relationship between variables. Consistent with the

literature, p-values below 0.10 (10%) are considered to be statistically significant in the present report. A p-value

of less than 0.10 (10%), 0.5 (5%), 0.01 (1%) conveys that the probability that the difference between the two

groups is due to chance is less than 10%, 5%, and 1% respectively.

Note that a result may be statistically significant, but it may not be economically significant or vice-versa. For

example, the impact of New ERA on retention rates may be statistically significant, but the size of the effect may

be trivially small. Alternatively, a result may not be statistically significant, perhaps due to a small sample size,

but it can be economically significant, in that it has a large meaningful impact on the outcome of interest. This is

likely to be an issue when one sub-divides the sample to look at a finer, more detailed analysis as the numbers in

the sub-groups gets smaller.

Three sets of results are presented. The base case examines the impact of New ERA for all Direct and Merit

students. The impact of the programme for New ERA Direct and Merit students are analysed and presented

separately. To recap, Direct students are those who entered the university with a concession on entry scores (i.e.

there is a lower points requirement) and Merit students are those who entered the university with the required

CAO points. Both groups receive the same post-entry supports. Separate results are then presented for students

who attained 400 points or less in their Leaving Cert Examinations, and for students who achieved more than

400 points, to determine if the New ERA programme has differing impact across these groups. Note the choice

of 400 Leaving Certificate points as cut-off is somewhat arbitrary: small changes to this would not make any

difference. Approximately 70% of UCD students enter with 400 or more points while just over 50% of New ERA

students are above this threshold.

It is worth remembering that as far as the New ERA programme is concerned, Direct and Merit students do not

differ prior to entry nor would they appear different post-entry: they simply enter UCD through two different

mechanisms. However what this means is that these two groups may differ in ways that are not observable.

Indeed the descriptive statistics suggest that that they are quite different. What this implies for the effectiveness

of the programme will be explored in this chapter.


30

4: Results Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 4: Results

Figure 4.1 Impact of New ERA on Withdrawal Rates

4.2 Evaluation Results In this chapter a set of results are presented, which show the effect of a student being in New ERA on a series

of academic outcomes. The results presented here cannot shed light on what the exact mechanism of any such

effects might be. It could be through mentoring or through academic support. While it would be extremely useful

to look at this, the data does not permit this detailed investigation8. The results in section 5.2 attempt to examine

the effects of financial supports however the data does not permit a comprehensive analysis to be carried out.

4.2.1 First Year official Withdrawal Rates

The first outcome to be considered is whether students withdraw from UCD without attempting their fits year

exams. This refers to whether students “officially” withdraw. In practice, it seems likely that a significant number

of students effectively withdraw by not turning up for exams but they will still be registered with the university.

In the data here they will be marked as “fails”. Figure 4.1 shows that overall, New ERA Direct students are 5.4%

less likely to withdraw before attempting their exams than students from the control group9. While the effect

is not as strong for Merit students, it is still positive and significant; they are 3.2% less likely to withdraw before

attempting their summer exams than students from the control group. Recall that the p value in the figure is

an indicator of the precision associated with the estimate. Where none is indicated (for example for two of the

results in Figure 4.1) this means that the result is not statistically significant at least at the 10% level.

The results for high (>400 points) and low point (≤400 points) students show that the New ERA programme

has a positive effect on low point Merit students and high point Direct students. Low point Merit students are

8.1% less likely to withdraw before their first year exams, and high point Direct students are 4.5% less likely to

withdraw before their first year exams. The impact of the programme on the withdrawal rates for low point

Direct students and high point Merit students are not statistically significant.

8 Results are illustrated graphically throughout. Details of the results in this chapter with an estimate of their precision are included in Appendix C.9 Note that here and elsewhere in the report effects are measured in percentage points.

!""#$%"&'()*+$$#,)#-.$*

All

Points!400

Points>400

/0

10

20

30

40

50

60

70

8.,(&'.9):

;;#<"

4=30 4=303=40

2=>0

6=10

/=/0!99 ?@&'"*A3// ?@&'"*B3//

)!99)C#D):E!).'F)GH)I&(J)K)L@D)?@&'"*

!"#$%&

'$#"&

p<.05 p<.1 p<.05 p<.05

)))))))!99)))))))))))))))))))))))))))))))))?@&'"*A3//))))))))))))))))))?@&'"*B3//)

0%

!""#$%"&'()*+$$#,)#-.$*

All

Points!400

Points>400

/0

10

20

30

40

50

60

70

8.,(&'.9):

;;#<"

/=/43 /=/43

/=/34

/=/2>

/=/61

/

!99 ?@&'"*A3// ?@&'"*B3//

)!99)C#D):E!).'F)GH)I&(J)K)L@D)?@&'"*

!"#$%&

'$#"&

p<.05 p<.1 p<.05 p<.05

)))))))!99)))))))))))))))))))))))))))))))))?@&'"*A3//))))))))))))))))))?@&'"*B3//)

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 4: Results Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

31


4.2.2 First Year Exam Performance

While it is interesting to know the impact of students withdrawing it is, arguably, much more important to know

how the students perform in their exams since only a minority actually withdraw. The next graph, Figu re 4.2

shows the effect of the New ERA programme on the grade that students get in their first year exams. There are

six possibilities, the first four refer to how students do in the summer from getting a First Class Honours (1.1), an

Upper Second (2.1), a Lower Second (2.2) and either a 3rd class honours or pass grade. The latter two are combined

partly because of the small numbers in the groups and also because not all faculties seem to distinguish between

the two. The other two categories are passing in the Autumn repeats (“Pass-Aut”) and Fail. Note that Fail here

means that either a student failed in the summer and did not repeat in the Autumn or they did repeat but did not

pass. Students who repeat in the Autumn may only get a pass grade: there is no distinction between first, second

class honours etc. Since a student has to fall in to one of these categories (conditional on sitting the exam), the

height of the bars sum to zero. That is if New ERA increases the probability of one outcome it has to be at the

expense of another and the increases cancel out the decreases.

The results are very clear and striking: the probability of the three least desirable outcomes is reduced and the

probability of the higher results (First and Second Class honours) is higher. In other words, what New ERA does is

to shift students up the grade distribution. The failure rate is about four percentage points lower with a slightly

bigger effect for the Direct students. This is unambiguous evidence that the programme has positive effects on

the academic achievements of New ERA students compared. There is some evidence that the effects differ for

Direct and Merit students but the differences are very small. One has to be careful in interpreting this graph: it

does not mean that if a New ERA student repeats in the Autumn that they are less likely to pass. This is because

one is considering all the possible outcomes together. So one can say that a student is less likely to end up passing

in the Autumn because they are less likely to have failed in the summer exam.

Figure 4.2 First Year Exam Performance: Base

First Yr Exam Performance

1.12.12.2

-8%

-4%

-0%

4%

8%

12%

Marginal Effect

1.1 2.1 2.2 Pass/3rd Pass Aut. Fail

DirectMerit

-8%

-4%

-0%

4%

8%

12%

Marginal Effect

DirectMerit

!!!!!!!!!!!!!!!!!! p<.1 p<.05 p<.1 p<.05 p <.1 p<.1 p<.05 p<.1 p<.05

!"!############$"!###############$"$########%&''()*+######%&''#,-."#####/&01

Grade Awarded

Grade Awarded


32


4.2.3 First Year Exam Performance: low points students

In the previous section it was seen that the differences between Merit and Direct students, in terms of the impact

of New ERA, were small and most likely negligible. Since the results control for the students’ Leaving Certificate

points this may not be too surprising. In this section and the next, the distinction is made between students

entering with 400 points or less and those who entered with more than 400 points. Clearly, the points that

students attain in their Leaving Certificate is associated strongly with what degree/programme they enter. The

vast majority of students in the low point category are in Sciences or Arts. The analysis here however controls for

this so the results here do not simply reflect that the two different groups are doing very different degrees.

Figure 4.3 represents the effect of programme participation on first year grades for low point (≤ 400 points)

students. One can see at a glance at the height of the bars in the graph that the effects are very small. More

importantly perhaps, none of them are statistically significant. That is one cannot reject the statistical hypothesis

that the effects that are shown are just due to chance. So these students, once they are in UCD, are neither

helped nor harmed by being part of the New ERA programme.

4.2.4 First Year Exam Performance: high points students

Figure 4.4 illustrates the impact of the New ERA programme on the probability of achieving a particular grade for

High point (>400 points) students. New ERA increases the probability of achieving a higher grade for both Direct

and Merit students. High point Merit students are 7.3% more likely to attain an upper second class honours

grade than students from the control group. While the level of statistical significance falls for Direct students,

programme participation still has a positive and significant effect: New ERA Direct students are 10.7% more

likely to attain an upper second class honours grade relative to the control group.

Figure 4.3 First Year Exam Performance: low Point Students

First Yr Exam Performance

1.12.12.2

-8%

-4%

-0%

4%

8%

12%

Marginal Effect

1.1 2.1 2.2 Pass/3rd Pass Aut. Fail

DirectMerit

-8%

-4%

-0%

4%

8%

12%

Marginal Effect

DirectMerit

!!!!!!!!!!!!!!!!!! p<.1 p<.05 p<.1 p<.05 p <.1 p<.1 p<.05 p<.1 p<.05

!"!############$"!###############$"$########%&''()*+######%&''#,-."#####/&01

Grade Awarded

Grade Awarded


33


High point students, both Direct and Merit, are also less likely to fail their first year exams. Direct students are

3.9% less likely to fail their first year exams, and Merit students are 3.1% less likely to fail their first year exams.

The impact is equally significant for both groups.

These results are very different from those presented in section 4.2.3 for low point students. Taken together, it

shows clearly that the academic benefits to the programme arise from a benefit to students who are academically

strong to begin with. This may have implications for the development of the programme which will be discussed

later.

Why there is such a sharp difference between high and low point students is an interesting question and one for

which this report has no simple answer. One way of thinking about this is that a student’s academic performance

depends on several inputs including their own academic ability, their study effort and any additional support they

receive such as through a programme like New ERA. It seems plausible that these inputs are complementary:

that is the benefit of each is higher the more one has of the other. Simply put, better students are better able

to take advantage of the extra opportunity afforded by the New ERA programme. This might be because such

students have other qualities, such as motivation or self-confidence. In the absence of more data one can only

conjecture.

4.2.5 Probability of Graduating

The results so far have considered only the students performance in their first year exams. It is also important to

consider what happens later during a student’s time at the university. Several outcomes relating to graduation

will be analysed. It is important to note that there is less information available to do this since many of the

students have not had the opportunity to graduate within the time frame of the data. So the dataset pertaining to

final year outcomes has about 40% fewer observations than for their first year outcomes considered so far. This

constrains what one can do in terms of analysing sub-groups. In particular there is data on very few students on

low points, partly because many have not remained in university to this stage. So it is not practical to distinguish

between high and low point groups10. However students are more evenly balanced between Merit and Direct so

that distinction will be considered.

10 The estimation techniques that have been mostly used in the report, based on the method of Maximum Likelihood, require large samples to be reliable.

Figure 4.4 First Year Exam Performance: high Point Students

1.1 2.1 2.2

Direct

Merit

!"!#$%&'& !"'!()*( !"!)&&(+&

!"!#&'&+) !"!(#'(#+ !"!)#%)'(

-8%

-4%

-0%

4%

8%

12%

Marginal Effect

DirectMerit

p<.1 p<.05 p<.05 p<.05 p<.05 p<.1 p<.05 p<.05 p<.05

!!"#"!!!!!!!!!!!!!!!$#"!!!!!!!!!!!!!$#$!!!!!!!!%&''()*+!!!!%&''!,-.#!!!!!!/&01!

Grade Awarded


34


Figure 4.5 represents the impact of the New ERA programme on the probability of graduating from the degree

course. The figure shows that, overall, programme participation has a positive impact on graduation rates. Merit

students are 9.7% more likely to graduate from their degree programme relative to students from the control

group. The effect is bigger (albeit less precisely determined) for Direct students: they are 14.8% more like to

graduate than students from the control group. This is evidence that New ERA has benefits beyond their first year

results and can have a major effect on the lives of those who participate in the programme.

4.2.6 Probability of Graduating on Time

An outcome which has been looked in the international literature is whether students graduate on time, that is

whether they need to repeat one or more years. In this case, as Figure 4.6 shows, there is no evidence that New

ERA has any significant effect. The marginal differences between New ERA students and the control group is very

small, around one percentage point, and not statistically significant.

Figure 4.5 Impact of New ERA on Probability of Graduating

Probabili

ty of

Graduati

ng

Direct Merit

Base case

0.148 0.097

Points!400 0.241 0.182

Points>400 0.054 0.074

!"

#"

$"

%"

&"

'!"

'#"

'$"

'%"

'&"

#!"

()*+,-)./0112

34'$56%"

7568"

9,*234 (2*,4

p<.05 p<.01

Figure 4.6 Impact of New ERA on Probability of Graduating on Time

!"#$%#&'()*+(*&',-!"#$%#&'()*+(*&',-

Direct Merit

Base

Case !"!#$% !"!#&&

Points!400 !"!'%% (!"!)#)

Points>400 (!"!*$# !"!&#)

./

.01/

20./

201/

30./

4#")'(#5*677-

8&1.67%1.44%

9'"-8& 4-"'&

[email protected]


35


4.2.7 Final Degree Classification

The last outcome considered in this chapter is the final degree classification. Note that in this analysis only

concerns students who pass their final degree: whether students pass or not has been considerd already in section

4.2.5. As can be seen in Figure 4.7 there is not much evidence that students degree class is affected although

Merit students may be about 6.1% more likely to get Upper Seconds (2.2’s) compared to lower outcomes.

4.3 main Findings in This ChapterOverall the results indicate the New ERA programme has a number of significant positive effects on the students

participating in the programme.

Participation in New ERA has a positive effect on reducing first year withdrawal rates. •

New ERA reduces the probability of withdrawing prior to first year exams for low point Merit •

students and high point Direct students.

New ERA has a positive effect on improving first year exam results by shifting students up the grade •

distribution.

Participation in New ERA increases the probability of achieving a First and Second Class •

honours and reduces the probability of failing or receiving a Third Class honours/Pass in the

first year exams.

These improvements in exam performance only benefit high point students only, i.e. those •

who enter UCD with more than 400 Leaving Certificate points.

The programme has similar effects for both Direct and Merit students in terms of first year •

exam results.

Participating in New ERA increases the probability of graduating from university. This is striking as one •

might have expected that the barriers associated with low SES might have dissipated by students’ final

year.

The result applies to both Direct and Merit students. •

The programme has no effect on whether students graduate on time. •

The programme has relatively little effect on the final degree classification the students receive. •

However, Merit Treatment students have an increased chance of attaining a higher grade.

Figure 4.7 Impact of New ERA on Final Degree Classification: Base

!"#$%&'()*((&+(,-%.,!"#$%&'()*((&+(,-%.,

base

Direct Merit

/0/ -0.007 0.041

10/ -0.024 0.061

101 0.006 -0.043

2$,,3&4*5 0.025 -0.060

-15%

-10%

-5%

0%

5%

10%

15%M

arginal Effect

1.1 2.1 2.2 Pass/ 3rd

!"#$%&

'$#"&

p<.1

Degree Class Awarded


Chapter 5Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 5: Further Analyses

37

5 Further AnalysesThus far, the research outlined in this report has focused on the impact of the New ERA programme on students

who have entered UCD. The results show that students who participate in the programme, will on average,

experience a number of significant academic benefits. This chapter extends the analysis by examining several

additional outcomes by which one may judge the effectiveness of the programme. This chapter also examines the

potential effects of changes to the New ERA programme.

There are two main strands to New ERA’s activities, pre-entry and post-entry supports with the former (including

admissions) designed to increase the numbers of students progressing to third level from linked schools, and the

latter designed to improve students outcomes once they arrive in university. In addition, the post-entry supports

may also increase progression even though the students are already “on site” since secondary students who

are contemplating applying to university may anticipate the post-entry supports. For example, the post entry

financial support may induce more students to apply to the programme. In addition, the value of the financial

support may have an impact on student outcomes if it reduces the need for student employment whilst at

university and provides additional financial resources to cover basic materials such as books, etc.

While it is very difficult to isolate which specific activity within the programme is more likely to affect which

outcome, the purpose of this chapter is to examine how student outcomes are influenced by varying the level

of New ERA supports. Specifically, this chapter evaluates the impact of the New ERA programme on progression

rates by examining whether a school becoming linked to the programme increases the numbers who subsequently

attend university in general and UCD specifically. Next, it uses the variation in the level of financial support

provided to New ERA participants over time to examine the impact of aid on students’ performances. The level

of financial support tended to vary between 1999 and 2004 due to funding availability and student numbers.

Finally, much of this report is based on the current HEAR system in which, in most cases, only students from

disadvantaged schools can apply to the New ERA programme. However, the introduction of the National HEAR

scheme in 2010 will change the eligibility criteria for joining the programme and this may have consequences for

the effectiveness of the programme identified in this report. The final part of the chapter will examine this issue

in further detail.

5.1 The Impact of New ERA/hEAR on Progression to University The focus of this analysis is to determine whether New ERA is effective at increasing progression rates to higher

level education, However, the data used in the analysis in Chapter 4 cannot be used for this purpose since this

only contains information on students who have progressed i.e. there is no comparison group. In an ideal world,

in order to examine the effect of New ERA/HEAR on the destinations of students from secondary schools, one

would like information about the exact destination of school leavers, on an annual basis and broken down by

each school. However these data are not collected on a systematic basis in Ireland.

5.1.1 The Impact of New ERA/hEAR on Changes in Progression to University

To address this lack of data, the research team conducted a postal survey of all schools linked to the HEAR

scheme (Denny, Doyle, O’ Reilly & O’ Sullivan, 2008). Crucially for the study, data on the proportion of the

schools’ final year students who progressed to university were gathered. Information such as aggregate Leaving

Certificate results, subject choice and student demographics was also collected. The survey questions were asked

in relation to the Leaving Certificate class of 2007 and also in relation to the class of 2001 so that changes in the

characteristics of the schools and the outcomes of their students could be identified.

A total of 158 schools responded to the survey, giving a response rate of 51 percent. Of the schools linked to

UCD, 24 (45% of their total number) responded, 34 (69% of their total number) of the schools linked to NUI

Maynooth and 100 (48% of their total number) of the schools linked to the other universities participating in the

HEAR scheme11.

11 The low response rate could perhaps be explained by a timing clash with the Whole School Evaluations conducted by the Department of Education and with exams as well as a general sense of “survey fatigue” amongst school principals rather than concerns relating to data protec-tion issues.


38

5: Further Analyses Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 5: Further Analyses

Figure 5.1 illustrates the proportion of the Leaving Certificate class in HEAR-participating schools that progressed

to university in the years 2001 and 2007. The data on progression is banded: that is respondents (school principals/

career guidance counsellors) were asked to give responses on a scale with five possible bands. The graph shows that

in 2001 almost 20% of schools reported that less than five percent of their students progressed to university, this

increased slightly in 2007 with 22% of schools reporting that less than five percent of their students progressed

to university. However, this is offset by the fact that from 2001 to 2007 the number of schools that experienced

progression rates of 31% or more increased by almost ten percent.

Figure 5.2 illustrates the overall change in progression rates experienced by schools from 2001 to 2007. The graph

shows that the majority of schools experienced no change in the number of students progressing to university

(52.28%). Approximately 30% of schools experienced an increase in progression rates, while 17% of schools

found that fewer students progressed to university.

However Figures 5.1 and 5.2 are only descriptive in nature and do not infer causality of the effects of joining New

ERA/HEAR (Tables D1 and D2 present these summary statistics in tabular form in Appendix D).

Figure 5.2: Percentage increase or decrease in progression to university

!"#$"%&'(")*%$#"'+"),#)-"$#"'+")*%).#,(#"++*,%!"#$"%&'(")*%$#"'+"),#)-"$#"'+")*%).#,(#"++*,%!"#$"%&'(")*%$#"'+"),#)-"$#"'+")*%).#,(#"++*,%!"#$"%&'(")*%$#"'+"),#)-"$#"'+")*%).#,(#"++*,%!"#$"%&'(")*%$#"'+"),#)-"$#"'+")*%).#,(#"++*,%

!"#$%&'% ()*+,&"-% .%#$%&'%

/%$#%"0&-%*+,&"-%12314 56367 89311

/

01

2/

31

4/

!"#$"

%&'("

2/526

17578

09522

Increase No Change Decrease

Figure 5.1: Percentage of leaving Certificate class progressing to university in 2001 and 2007

!

"!

#!

$!

%!

!&'()'*& +&'()'"!& ""&'()'#!&' #"&'()'$!& $"&',

!""#

!""$

-./0.1(23.')4'

506))75'/.8)/(913

8/)3/.559)1'/2(.5

0% to 5% 6% to 10% 11% to 20% 21% to 30% 31% +

2001 19.9 17 19 20.2 23.9

2007 22 14 18.4 11.8 33.7!

"!

#!

$!

%!

!&'()'*& +&'()'"!& ""&'()'#!&' #"&'()'$!& $"&',

!""#

!""$

-./0.1(23.')4'

506))75'/.8)/(913

8/)3/.559)1'/2(.5

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 5: Further Analyses Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

39

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 5: Further Analyses

The method used for these estimates is somewhat more complicated than those used elsewhere in the report.

While the outcome is ordered (increase, no change, decrease), we allow for the possibility that the length of

time a school is linked to HEAR may not be exogenous. In other words, there may be some systematic difference

between schools that were in the scheme at the beginning from those who joined later. One reason for this

concern is that not all schools responded to the survey (about 120 out of 305). Hence this study uses what is

known as “a simultaneous equations approach” where there an outcome of interest (here, change in progression)

as a function of the treatment variable (how long linked to HEAR) and some covariates with the treatment

variable itself being dependent on some independent variables. The latter are the local unemployment and

education rates12.

Figure 5.3 represents the estimated marginal effect of being linked to the HEAR scheme for an additional year on

the change in the number of students progressing to university. The estimates and the corresponding standard

errors and sample sizes are shown in Table D3 in Appendix D. The graph shows that being linked to HEAR for a

longer period of time has a positive effect on student outcomes. It shows that schools that are linked to HEAR for

an additional year are 12% less likely to have fewer students progressing to university. The effect is even stronger

on the probability of schools sending more students to university; the effect of being linked to HEAR for an

additional year increases the probability that more students will progress to university by 14%. Finally, the graph

shows that the effect of being linked to HEAR for an additional year does not significantly affect the likelihood

that a school will experience no change in progression rates.

12 This approach is similar to “instrument variables regression”, the key difference being that one equation here is non-linear (the ordered probit). Hence the Conditional Mixed Process estimator due to Roodman (2007) is used.

Figure 5.3: The effect of the duration that a school has been linked to hEAR o n the progression of students to university

!"#$"%&&'#()*+,%&!"#$"%&&'#()*+,%&

-%.%")$#'($),#)/('0%"&',12#)34+($% 5#"%)$#'($),#)/('0%"&',15#"%)$#'($),#)/('0%"&',15#"%)$#'($),#)/('0%"&',1

67897 68 7:97

5#"%)$#'($),#)/('0%"&',12#)34+($% -%.%")$#'($),#)/('0%"&',1-%.%")$#'($),#)/('0%"&',1-%.%")$#'($),#)/('0%"&',1

7:97 68 67897

!"#

!$%

!$#

!%

#

%

$#

$%

"#

&'()*+',-.//0

12

$34$

!"

!$"4$

56'(2-7

))))))))))))))))))))))

5#"%)$#'($)))))))))))))))))))))2#)34+($%)))))))))))))))))))))));%&&)$#'($)

,#)/('0%"&',1))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))),#)/('0%"&',1

More going No Change Less going to University to University

p<.01 p<.05


40


5.1.2 Impact of New ERA/hEAR on Changes in Progression to UCD

While the previous analysis examined the proportion of students who progressed to any university as a result of

being linked to the HEAR scheme, it is also possible to model the effect of HEAR/New ERA on the proportion of

students from a particular school’s Leaving Certificate class who attend UCD specifically rather than university in

general. Using the UCD administrative data discussed in Chapter 3, the proportion of students from a particular

school progressing to UCD was calculated for each year. Figure 5.4 shows the proportion of New ERA/HEAR

schools sending at least one student to UCD in a given year. Unsurprisingly, the proportion of New ERA link

schools sending at least one student is always higher compared to that of schools which only participate in HEAR

and are linked to other universities. On would expect that because of location: New ERA schools are, on average,

closer to UCD than other link schools13. For both types of school, the proportion of students who go on to attend

UCD fluctuates from year to year and there is no clear pattern.

13 See Map A1 and A2 in Appendix A for the geographical distribution of linked schools.

Figure 5.4: Average proportion of schools that send at least one student to UCD

!""#$%&''"(#)*+,#-.!#"/01#'0"2#)3-!.#'0"2#)3-!.#'0"2#)

4555 67896 6:8;< 6=84:

9>>> 6>897 ;;8>: <6899

9>>4 <;8>= 6>879 <6899 !"#$%&'('&)*'+$',$-./''0-$)/1)$-2+3$4'52$4)632+)-$)'$789

9>>9 6587: :489< 6686;

9>>= 6989< 6686; 64845

9>>7 67896 ;>8:= 69865

:

;:

<:

=:

>:

?:

@:

A:

B:

C:

;::

;CCC <::: <::; <::< <::= <::>

%2&.2+)

!""#$%&''"(#)

*+,#-.!#"/01#'0"2#)

3-!.#'0"2#)

1999 2000 2001 2002 2003 2004

All Schools % 64.26 60.24 57.03 69.48 62.25 64.26

New ERA link only % 68.75 77.08 60.42 81.25 66.67 70.83

HEAR only % 63.18 56.22 56.22 66.67 61.19 62.69


41


Figure 5.5 Average proportion of leaving Certificate class attending UCD

Figure 5.5 shows the average proportion of each schools’ class who go on to UCD in a given year broken down by

whether the school is a New ERA link school or participating in the HEAR scheme only. Many of these schools

may send a very low fraction of their Leaving Certificate class, if any, to UCD in a given year. Figures 5.4 and 5.5

are shown in tabular form in Table D4 and D5 in Appendix D.

To estimate the effect of being linked to New ERA/HEAR on the probability of students progressing to UCD

a statistical technique called “Tobit” is employed. This is a standard method in econometrics for modelling a

continuous variable where many of the values are clustered at a lower bound14. In this context the continuous

variable is the proportion of students from a particular school attending UCD in a given year which equals zero

when nobody from that school attended UCD in that year.

14 Technically this is called “censoring”. For example many people have zero hours of work and one cannot have less than zero.

!""#$%&''"(#)*+,#-.!#"/01#'0"2#)3-!.#'0"2#)3-!.#'0"2#)4555 6768 97:8 ;758;::: 676< 978= ;7=<;::4 6746 878< ;7=;;::; 678< 9788 ;755;::6 6744 87>9 ;7>;;::8 6794 <7:; ;754?'@A" 676; 97;: ;7=>

!"#$%&'%'&()'*$'+$,-$./011$0((2*3)*#$4-5

0

1

2

3

4

6

7

8

9

10

6777 8999 8996 8998 899: 899;

<2&.2*(

!""#$%&''"(#)

*+,#-.!#"/01#'0"2#)

3-!.#'0"2#)

1999 2000 2001 2002 2003 2004

All Schools % 3.34 3.36 3.13 3.46 3.11 3.51

New ERA link only % 5.04 5.48 4.46 5.44 4.75 6.02

HEAR only % 2.94 2.86 2.82 2.99 2.72 2.91


42


Figure 5.6(a): Effect of being linked on the probability of sending at least one student to UCD

Figure 5.6(b): Effect of being linked on the proportion of students attending UCD

Figure 5.6a shows the estimated marginal effect of a school joining New ERA/HEAR on the probability of sending

at least one student to UCD in a given year. Figure 5.6b shows illustrates the effect on the proportion of students

the school sends to UCD given that it sends at least one student. There are no significant effects on either

outcome for the HEAR-only schools. However there are significant effects for New ERA link schools. The results

show that being linked to the New ERA programme increases the probability that a school will send at least one

student to UCD by approximately 13%. The marginal effect on the proportion of students that it sends (given

that it sends at least one student) is slightly above 1%, i.e. being linked to New ERA increases the proportion of

the Leaving Cert class progressing to UCD by about one percentage point. Given that the average proportion of

students sent by linked schools to UCD is about 5%, this one percentage point increase is a significant gain. The

effects and the corresponding standard errors are displayed in Table D6 in Appendix D.

!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*

.//$01233/4567$89.$/"+:$3+/;<8.9$/"+:$3+/;<8.9$/"+:$3+/;

Marginal Effect of being linked on probability of school sending at least one student to UCD0.073 0.131 0.0030%

2%

4%

6%

8%

10%

12%

14%

16%

!"#$%&"'()**+,-

./01

20/21

3/014''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=

Marginal Effect of being linked on probability of school sending at least one student to UCD

p<.01 p<.05

31

3/?1

3/@1

3/A1

3/B1

2/31

2/?1

2/@1

2/A1

!"#$%&"'()**+,-

0.5%

1.3%

0.0%4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=

!"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI

p<.01 p<.01

!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*

.//$01233/4567$89.$/"+:$3+/;<8.9$/"+:$3+/;<8.9$/"+:$3+/;


2%

4%

6%

8%

10%

12%

14%

16%

!"#$%&"'()**+,-

./01

20/21

3/014''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=


p<.01 p<.05

31

3/?1

3/@1

3/A1

3/B1

2/31

2/?1

2/@1

2/A1

!"#$%&"'()**+,-

0.5%

1.3%

0.0%4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=

!"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI

p<.01 p<.01

!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*

.//$01233/4567$89.$/"+:$3+/;<8.9$/"+:$3+/;<8.9$/"+:$3+/;


2%

4%

6%

8%

10%

12%

14%

16%

!"#$%&"'()**+,-

./01

20/21

3/01

4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=


p<.01 p<.05

31

3/?1

3/@1

3/A1

3/B1

2/31

2/?1

2/@1

2/A1

!"#$%&"'()**+,-0.5%

1.3%

0.0%

4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=

!"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI

p<.01 p<.01

!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*!"#$%&'$()*$)+,$%&'$(-*

.//$01233/4567$89.$/"+:$3+/;<8.9$/"+:$3+/;<8.9$/"+:$3+/;


2%

4%

6%

8%

10%

12%

14%

16%

!"#$%&"'()**+,-

./01

20/21

3/01

4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=


p<.01 p<.05

31

3/?1

3/@1

3/A1

3/B1

2/31

2/?1

2/@1

2/A1

!"#$%&"'()**+,-0.5%

1.3%

0.0%

4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'=

!"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI

p<.01 p<.01


43


5.2 The Impact of New ERA’s Financial Aid on Student outcomes In general this study is concerned with the impact of New ERA taking the program as a whole. That is, the study

does not investigate the different aspects of the program. Ideally, one would do this but the data available to the

research team does not permit such an approach. Nevertheless, in this section an attempt is made to see can one

say anything about the impact of one specific aspect of the program namely the financial aid package provided

to students.

The financial aid package provided to students annually is a key component of New ERA’s post-entry supports.

The amount of this package has changed over time due to funding availability and the number of New ERA

students admitted to the programme. Figure 5.7 shows the total financial aid package in real amounts (expressed

in 2008 prices to adjust for inflation) for the period 1999-2004. As nearly all New ERA students are also in receipt

of the Higher Education grant, the table also shows changes in the value of that grant over time (in 2008 prices)

The sum of New ERA’s financial package and the government grant varies from year to year with the total value

of the package being particularly high in 2000, 2001 and 2003. The average value of the entire financial package

received by a New ERA student in the three years 2000, 2001 and 2003 was €6313 (expressed in 2008 prices).

The average over the years 1999, 2002, and 2004 was €5407.

In order to determine the effectiveness of this financial aid package, the analysis examines whether student

performance in these high value years are different from student performance in the other years. The analysis

rests on the assumption that there were no other differences in New ERA’s activities in these high value years that

may influence outcomes. It also assumes that the characteristics of students in the high value years did not differ

from students in low value years. In order to estimate the effect of receiving a higher value financial package, i.e.

for students that entered in 2000, 2001, or 2003, compared to those receiving the lower package i.e. for students

that entered in 1999, 2002, or 2004, an ordered probit model as used in Chapter 4, is estimated for New ERA

students entering first year between 1999 and 2004.

Although the estimated results presented in Figure 5.8 do follow a pattern suggesting that the extra funding

was beneficial, the first year outcomes for students who received the high value package were not statistically

different from the students who received the lower value package. Furthermore no significant effects of the extra

funding were detected when alternative models were estimated15. The estimated results and their standard

errors are shown in Table D7 in Appendix 7. This does not mean that New ERA’s financial package has no effect on

student performance; however it does imply that increasing the value of the package from an average of €5407

to €6313 (a difference of €906) did not lead to changes in student achievements.

15 The results of these alternative models are available on request from the research team.

Figure 5.7: Financial Aid for New ERA students

!"#$#%"$&'(")!"#$#%"$&'(")

!"#$%&'$()**+,-./+012$')-3+,4-5$6,17-8+-12$()**+,-.8+-12$()**+,-.

9::: ;<=<>: ;<=:<? ;?=9@>

<AAA ;B=<C9 ;<=:A: ;C=9@9

<AA9 ;B=>9< ;<=:>D ;C=BCA

<AA< ;<=?9C ;<=:@< ;?=>D:

<AAB ;B=A:9 ;B=B9C ;C=>A@

<AA> ;<=<B> ;B=B<B ;?=??@

'E",1F" <@:> BACC ?DCA

2000

3250

4500

5750

7000

1999 2000 2001 2002 2003 2004

!"#$%&'$()**+,-.

/+012$')-3+,4-5$6,17-

8+-12$()**+,-.

Financial Support Available to New ERA Students

Financial Support Available to New ERA Students

1999 2000 2001 2002 2003 2004

New ERA Supports €2,249 €3,261 €3,412 €2,516 €3,091 €2,234

Local Authority Grant €2,925 €2,909 €2,948 €2,972 €3,316 €3,323

Total Supports €5,174 €6,171 €6,360 €5,489 €6,407 €5,557


44


There are limits as to how much one can extrapolate from this result. Firstly it should be noted that the sample size

used for this analysis is quite low as only New ERA students can be included; therefore the estimated results may

not be precise. Secondly, based on the data available, it is not possible to predict with any degree of confidence

whether an increase of more than around €900 would have had any effect. Nor is it possible to estimate if a

reduction in the value of the financial package below an amount of, say €5400, would have any effect on average

student performances. However in reducing the amount of financial aid to students, policy makers should

consider the effects of such a reduction on student employment whilst studying full-time. Students may enter

part-time employment to offset a reduction in financial aid. There is currently no consensus in the academic

literature on the causal effects of student employment on academic outcomes. However one recent study by

Stinebrickner and Stinebrickner (2003) was able to exploit a natural experiment in one college and found that

employment was harmful for low-income students’ academic outcomes. Thus the effect of any reduction in the

financial package of New ERA students would need to be closely monitored.

5.3 likely Consequences of the National hEAR scheme. The HEAR scheme will undergo major changes for those applying for entry to higher education institutes in

2010. It is important for policy makers to be informed of the likely consequences of these changes. Until now,

under the HEAR scheme, only students from DESI disadvantaged schools could apply to the university access

programmes. However, the introduction of the National HEAR scheme in 2010 will change the eligibility criteria

for participating in the programme. Under the new system, students who meet the income, socio-economic and

educational eligibility criteria can apply to university through the HEAR scheme even if they are not attending

a DEIS disadvantaged school. Essentially the HEAR scheme now includes all secondary schools in the country.

This new system was introduced as it was recognised that there may be disadvantaged students attending

non-disadvantaged schools who could benefit from the access admissions programme for HEAR. Extending the

scheme to all schools is not the only change: the assessment process now uses six indicators.

Using the UCD administrative data, a further analysis was conducted to attempt to examine the effects of the

Extended HEAR scheme to students from non-disadvantaged schools. The previous analysis compared New ERA

students to students who would otherwise be eligible for the programme but whose school has not yet begun

participating in the HEAR scheme by the time they entered university. The control group is comprised of the

latter group of students. In this new analysis, New ERA students are compared to an alternative control group

who come from similar socio-economic backgrounds but who attend schools that are not currently participating

in the HEAR scheme i.e. the schools which are typically not disadvantaged.

Figure 5.8: Impact of variation in financial aid on first year exam performance

!"#$

!%$

!&$

!'$

!($

#$

($

'$

&$

%$

"#$

)*+,-.*/01223

45

(67$

867$ &6'$

!#67$

!76#$ !76"$

"6" (6" (6( 9*::;<+= 9*::0>?56 @*-/

!"#$"%&

Grade Awarded


45


As in Chapter 4 an ordered probit model is estimated to examine the effects of being a New ERA student relative

to being in the alternative control group. The results presented in Figure 5.9 show that in general the New

ERA students are more likely to have positive outcomes compared to the disadvantaged students from non-

disadvantaged schools who do not currently (pre-2010) participate in the HEAR scheme. One interpretation of

this result is that participation in the New ERA programme has a greater effect on university achievement for

disadvantaged students than the benefits that accrue from attending a non-disadvantaged school. This would

imply that students who meet the minimum income, socio-economic and education eligibility criteria who are

attending non-disadvantaged schools should benefit from the New ERA programme. Overall this suggests that

the new national HEAR scheme may be advantageous.

However in considering this result one should note that the comparison between the New ERA students and the

disadvantaged students in non-disadvantaged schools can be potentially biased by various factors as students

from very different educational backgrounds are being compared. For example, one may argue that the effect

of New ERA on university outcomes is being overstated as the existing New ERA students may have scored

higher Leaving Certificate points had they attended a non-disadvantaged school. Thus comparing New ERA

students to students from non-disadvantaged schools who attained the same Leaving Certificate points might

be misleading.

Figure 5.9: Effect of New ERA relative to being in a non-link school on first year exam performance

!"#$%&'

!"#$%&$&'(!)*'+,-$.$'+#/&$%#0!"#$%&$&'(!)*'+,-$.$'+#/&$%#0!"#$%&$&'(!)*'+,-$.$'+#/&$%#0

1!'0#/&$%#0'2$30/0'435%#'5%&',633$,#'+!780'9%'%6%:;9%<'0,-66;01!'0#/&$%#0'2$30/0'435%#'5%&',633$,#'+!780'9%'%6%:;9%<'0,-66;01!'0#/&$%#0'2$30/0'435%#'5%&',633$,#'+!780'9%'%6%:;9%<'0,-66;01!'0#/&$%#0'2$30/0'435%#'5%&',633$,#'+!780'9%'%6%:;9%<'0,-66;01!'0#/&$%#0'2$30/0'435%#'5%&',633$,#'+!780'9%'%6%:;9%<'0,-66;0

(&( =>=?@

)&( =>=AB

)&) =>=C?

*+,,-./0 :=>=?B

*+,,$123& :=>=BD

!+"4 :=>=CE

!FF$,#'6F'1$G'!*)'3$;5#92$'#6'H$9%4'9%'5'%6%:;9%<'0,-66;0!FF$,#'6F'1$G'!*)'3$;5#92$'#6'H$9%4'9%'5'%6%:;9%<'0,-66;0!FF$,#'6F'1$G'!*)'3$;5#92$'#6'H$9%4'9%'5'%6%:;9%<'0,-66;0!FF$,#'6F'1$G'!*)'3$;5#92$'#6'H$9%4'9%'5'%6%:;9%<'0,-66;0!FF$,#'6F'1$G'!*)'3$;5#92$'#6'H$9%4'9%'5'%6%:;9%<'0,-66;0

567

587

5%7

5.7

5)7

597

)7

.7

%7

87

67

:+/#";+4$<==>

?3

)&(7

%&@7

.&)7

5)&@7

5@&875.&67

p<.05 p<.01 p<.01 p<.05 p<.01 p<.01

(&($$$$$$$$$$$$$$$$$)&($$$$$$$$$$$$$)&)$$$$$$$$$$$*+,,-./0$$$$$*+,,$123&$$$$$$$$!+"4

Grade Awarded


46

5: Further Analyses Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5.4 main Findings in This ChapterIf a school becomes part of the HEAR scheme then there is a higher probability that it will send more •students to university.

Using the period between 2001 and 2007 it was found that becoming linked in that interval o

increased the probability of sending a higher proportion to university by 14%.

Being linked to the New ERA programme increases the probability that a school will send at least one •student to UCD by approximately 13%. The marginal effect on the proportion of students that it sends

(given that it sends at least one student) is slightly above 1%, i.e. being linked to New ERA increases the

proportion of the Leaving Cert class progressing to UCD by about one percentage point.

Changes in the financial aid package to students, taking into account the Higher Education grant, do

not have a measurable effect on student outcomes. Since one only observes variation in this between

years (& not between students) these estimates are less precise.

Comparing New ERA students with students who are also socially disadvantaged but who do not

qualify for the programme (as they attend a non-disadvantaged school), the results show that the

New ERA students in UCD out-perform academically their comparators. This suggests there would be

benefits to allowing students to participate in New ERA even if there school is not linked to HEAR. This

is what the new national HEAR scheme does.


48

6: Recommendations & Findings Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative 6: Recommendations & Findings

6 Recommendations & FindingsThe results of this report have implications both for UCD specifically and for higher education access programmes

generally. A set of general recommendations which have relevance to all higher education institutions have been

identified along with further specific recommendations for the UCD New ERA programme.

UCD RECommENDATIoNS

The following recommendations are based on the results of this evaluation and suggest areas where the New

ERA programme might be refined. While some of the recommendations are based on the assumption that the

financial resources available to the access programme are constant, others would require additional resources.

1. The study has found convincing evidence that the UCD New ERA programme has substantial benefits for students

from lower socio-economic groups. That is, there are more students from disadvantaged backgrounds going to

university and those that do, at least in the case of UCD, perform better on several fronts. It follows that any

contraction in the programme, whether for budgetary or other reasons, is likely to remove these benefits.

2. Given the positive academic benefits to New ERA students throughout their degree, there is an argument for

expanding the capacity of the access programme. While the programme can reserve additional places for Direct

students, it cannot directly affect the number of Merit students entering the programme.

i. It is recommended that the number of places reserved for Direct students be increased,

especially in courses requiring approximately 400 points or more as students attaining such

points generally derive greater benefits from the programme. This is subject to not increasing the

current level of points concession available to these students.

ii. Steps should be taken to actively increase the number of Merit students from lower socio-

economic groups. This could be achieved by providing additional pre-entry support to all

potential students. Increasing the number of Merit students is also likely to be achieved by the

introduction of the national HEAR scheme.

3. The previous recommendations are based on further resources becoming available. If this is not possible we

suggest there could be a reallocation of resources from financial aid to fund these additional places. This is

subject to maintaining the total financial aid package that students receive (including the Higher Education

grant) to be no less than about €5,400 per annum. We recommend this as fluctuations in financial aid do not

appear to have affected student performance. It would be useful to monitor whether changes in financial aid

lead to changes in part-time work by the students.

4. As an alternative to the above recommendation, some of this financial aid could be re-allocated to New ERA’s

other activities (such as academic and social support). This is under the assumption that the current funding

of the programme is unchanged and again subject to the caveat above of not reducing the total package per

student to be less than €5,400 per annum. In the current economic climate it is likely that there will be financial

constraints for Higher Education Institutions and access programmes in the future so prioritizing of the most

effective supports is essential.

5. The research shows that 50% of Direct students succeed at their summer exams on first sitting compared to 71%

of students from the general population. It is therefore recommended that additional supports be put in place

to help prevent this from occurring. This issue is especially important in the modular system where repeating

in the autumn is no longer possible. Preventing students failing during the year will reduce the probability of

students having to carry over modules into second year. For example, the university may wish to consider some

form of “Early Warning System” which flags the presence of students who are at a high of risk of failing.

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative 6: Recommendations & Findings Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

49

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 6: Recommendations & Findings

NATIoNAl RECommENDATIoNS

6. The national roll–out of HEAR is consistent with the evidence that New ERA students perform better than other

disadvantaged students who do not participate in the access programme as their school is not linked. In other

words, for these students, coming from an “advantaged” school does not compensate for not entering the

programme. It therefore follows that allowing such students to participate in the access programme, through

the new HEAR scheme, should generate benefits for them.

7. Evidence in this report suggests schools which are linked to the access programme (and receiving pre-entry

supports) have increased progression rates to higher level institutions. Therefore an argument can be made to

provide pre-entry supports to schools which are currently excluded from outreach activities.

8. Data collection is paramount to facilitating quality research. It is recommended that all higher education

institutions collect information from all students, at the point of registration, on the amount of financial aid

received, parental educational attainment, family composition and levels of household income (using banded

answers). This will facilitate and inform future evaluations of access programmes.

9. Furthermore, access programmes should conduct a survey of all link schools and HEAR participating schools

every two years to build up longitudinal data on the effects of the access programmes’ pre-entry activities.

FURThER RESEARCh

10.

This study has evaluated the overall impact of New ERA on student performance, however it was not possible

to determine which specific components of the programme are most effective. Further research is needed to

determine whether, for example, is it mentoring, financial aid, academic support or a combination of these that

improves student outcomes? To consider this one needs variation in the level of support that students receive

so that some students get different combinations of supports from others. This should be best evaluated as

part of a randomized control trial. While recognizing the practical and ethical difficulties inherent in such an

experiment, a well designed trial could provide valuable information on the design of access programme.

11. The New ERA evaluation is the first quantitative research of an access programme to be undertaken in

Ireland. Quantitative evaluations of other access programmes would contribute significantly to the current

understanding of the effectiveness of access initiatives. This would help inform policymakers in relation to the

national widening participation agenda. HEIs differ in their approach to access both in terms of their history and

approach. Measuring their impacts, aside from its direct relevance to the particular institution, could be very

informative about which type of access programmes are most effective. Research undertaken on a national

level on the impact of the different support measures would inform national policy for the longer term and

would help to identify the best practice in support provision for students from lower socio-economic groups.

12. It would also be beneficial to undertake research into the particular issues associated with widening

participation in programmes of study that are traditionally highly prestigious and financially rewarding, for

example Medicine, Veterinary Medicine , Law and Dentistry. As demand for these courses is very competitive

with high points requirements, there have traditionally been very few low SES students in these courses. A case

study of Medicine may prove valuable as the Irish universities have just moved to a system that reduces reliance

on Leaving Certificate points by also taking into account the scores on the Health Professional Admissions

Test (HPAT). Whether this will change access to medicine for low SES groups is far from clear and needs to be

investigated in a few years when there is sufficient information and the system has had time to mature.


50

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionReferences Evaluating the Impact of the UCD New ERA Widening Participation Initiative

ReferencesAngrist, J. and Krueger, A. B. (1999) “Empirical Strategies in Labour Economics”. In: Ashenfelter, O. and Card, D.,

eds. Handbook of Labor Economics, Vol. 3A: 1339-1344. Amsterdam: North Holland.

Angrist, J., Lang, D. and Oreopoulos, P. (2009) “Incentives and Services for College Achievement: Evidence from a

Randomized Trial”. American Economic Journal: Applied Economics, 1(1): 136-163.

Bergin, D., Cooks, H., and Bergin, C. (2007) “Effects of a College Access Program for Youth Underrepresented in

Higher Education: A Randomised Experiment”. Research in Higher Education, 48 (6): 727-750.

Bettinger, E. (2004) “How Financial Aid Affects Persistence.” NBER Working Paper, No. 10242.

Brock, T. and Richburg-Hayes, L. (2006) “Paying for Persistence: Early Results of a Louisiana Scholarship Program

for Low-Income Parents Attending Community College.” MDRC.

Burtless, G. (1995) “The Case for Randomized Field Trials in Economic and Policy Research.” Journal of Economic

Perspectives, 9(2): 63-84.

Card, D. and Krueger, A.B. (1994) “Minimum wages and employment: A case study of the fast-food industry in

New Jersey and Pennsylvania.” American Economic Review, 84(4): 772-793.

Carneiro, P., and Heckman, J. (2003) “Human Capital Policy.” NBER Working Paper, No. 9495.

Carpenter, A. (2004) “Social Class, Inequality and Higher Education in Ireland. Leaving Early: International

Perspectives on Working Class Student Withdrawal.” Staffordshire University, Stoke-on-Trent, UK.

Chevalier, A., Denny, K. and McMahon, D. (2009) “Intergenerational Mobility and Education Equality.” In: Dolton,

P., Asplund, R. and Barth E., eds. Education and Inequality across Europe. London: Edward Elgar.

Chevalier, A., Finn, C., Harmon, C., and Viitanen, T. (2006) “The Economics of Early Childhood Care and Education.”

Technical Research Paper for the National Economic and Social Forum (NESF).

Clancy, P. (2001) College Entry in Focus: A Fourth National Survey of Access to Higher Education, Dublin: Higher

Education Authority.

Clancy, P. (1982) Participation in Higher Education: A National Survey, Dublin: Higher Education Authority.

Denny, K. Doyle, O. O’Reilly, P. and O’Sullivan V. (2009) “A Profile of HEAR Schools 2008.” UCD Geary Institute.

Department of Education and Science (2001) Report of the Action Group on Access to Third Level Education, Dublin:

Stationery Office.

Department of Education and Science (1995) Charting our Education Future: White Paper on Education, Dublin:

Stationery Office.

Department of Education and Science (1993) Report of the Advisory Committee on Third-Level Student Support

(Committee chaired by Dr. Donal de Buitléir), Dublin: Stationery Office.

Department of Education and Science (1992) Education for a Changing World: Green Paper on Education, Dublin:

Stationery Office.

Dynarski, S. (2008) “Building the Stock of College-Educated Labor.” The Journal of Human Resources, 43 (3): 576-

610.

Dynarski, S. (2003) “Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and

Completion.” American Economic Review, 93(1): 279-288.

Equality Review Team to the Higher Education Authority (2004) Report of the High Level Group on University

Equality Policies, Dublin: Higher Education Authority.

Harmon, C., Oosterbeek, H., and Walker, I. (2003) “The Returns to Education: Microeconomics.” The Journal of

Economic Surveys, 17 (2): 115-155.


51

Evaluating the Impact of the UCD New ERA Widening Participation Initiative References

Lavy, V. and Schlosser, A. (2005) “Targeted Remedial Education for Underperforming Teenagers: Costs and

Benefits.” Journal of Labor Economics, 23(4): 839-874.

Lesik, S. (2007) “Do Developmental Mathematics Programs have a Causal Impact on Student retention? An

Application of Discrete-Time Survival and Regression Discontinuity Analysis.” Research in Higher Education, 48

(5): 583-608.

Lynch, K. and O’Riordan. C. (1996) “Social Class, Inequality and Higher Education: Barriers to Equality of Access

and Participation among School Leavers”, University College Dublin, Registrar’s Office.

Machin, S. and Vignoles, A. (2005) “What’s the Good of Education? The Economics of Education in the UK.” New

Jersey: Princeton University Press.

McPherson, M., Schapiro, M. (1991) “Does Student Aid Affect College Enrollment? Nw Evidence on a Persistent

Controversy.” American Economic Review, 81(1): 309-318.

Oakley A., Strange, V., Toroyan, T., Wiggins, M., Roberts, I. and Stephenson, J. (2003) “Using random Allocation to

Evaluate Social Interventions: Three Recent U.K. Examples”, Annals, AAPSS, 589 (1): 170-189.

Osborne, R. and Leith, H., (2000) Evaluation of the Targeted Initiatives on Widening Access for Young People from

Socio-Economically Disadvantaged Backgrounds, Dublin: Higher Education Authority.

Preparing for Life Programme, UCD Geary Institute. Available from:

< http://geary.ucd.ie/preparingforlife/home> [Accessed 9 September 2009]

Roodman, D. (2007) cmp: Stata module to implement conditional (recursive) mixed process estimator. http://

ideas.repec.org/c/boc/bocode/s456882.html

Skilbeck, M., Connell, H. (2000) Access and Equity in Higher Education: an International Perspective on Issues and

Strategies, Dublin: Higher Education Authority.

Stinebrickner, R. and Stinebrickner T. R. (2003) “Working during School and Academic Performance.” Journal of

Labor Economics, 21 (2): 449-472.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative Appendix

Appendices A: Maps

B: Tables of descriptive statistics for Chapter 3

C: Tables of results for Chapter 4

D: Tables of descriptive statistics and results for Chapter 5

53


54

Appendix Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: IntroductionEvaluating the Impact of the UCD New ERA Widening Participation Initiative Appendix

Appendix A:

Map A1:

Distribution of link Schools by higher Education Instituion as at 2008

1: Introduction Evaluating the Impact of the UCD New ERA Widening Participation Initiative Appendix Evaluating the Impact of the UCD New ERA Widening Participation Initiative Evaluating the Impact of the UCD New ERA Widening Participation Initiative 1: Introduction

55

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Appendix

Map A2:

Distribution of Dublin link Schools by higher Education Institution at at 2008


56


Map A3:

Year School linked to the higher Education Access Route as at 2008


57


Map A4:

Year Dublin Schools linked to the higher Education Access Routes as at 2008


58


Appendix B: Tables of Descriptive Statistics for Chapter 3

Table B1: Description of New ERA sample

Year of entry Total Direct merit

% % %

1999 8.07 6.36 10.1

2000 9.63 11 8.05

2001 9.63 9.83 9.4

2002 19.3 15 24.2

2003 21.4 22.5 20.1

2004 35.6 35.3 28.2Total 322 173 149

Table B2: Descriptive Statistics: Socio-demographics

Direct merit Rest

% % %

male 36 40 46

Average Points 382 429 455

Grant 100 100 16.94

Socio-economic group of father

Farmers/Ag Workers 1.2 1.4 11.8

Higher/Lower Professionals 0 0 40.7

Managers and Employers 0 0 21.9

Salaried Employees 25.3 19.7 16.2

Intermediate Workers 12.6 8.5 1.7

Other non-manual 10.3 15.5 1.2

Skilled manual 24.1 16.9 5.0

Semi-skilled manual 12.6 16.9 0.9Non-skilled manual 13.8 21.1 0.5

N 173 149 16377


59


Table B3: Descriptive Statistics: University Faculty

New ERA Direct New ERA merit General Study Body

% % %

Agriculture 0.58 4.7 5.29

Arts 34.1 43.62 40.72

Commerce 20.81 8.05 12.74

Engineering and Architecture 6.36 6.04 9.46

Interfaculty 5.2 1.34 4.41

Law 2.89 2.01 2.99

Medicine/Nursing etc. 13.87 2.01 5.85

Science 6.36 27.52 12.95

Veterinary Medicine 1.16 1.34 1.81

Human Sciences 8.67 3.36 3.78N 173 149 16337

Table B4: Descriptive Statistics: First Year Exams

Table B5: Descriptive Statistics: Final Year Graduation


First Year Summer Exams: % % %

Registered for Summer Exams 97.1 97.3 93.7

Pass Summer 50 73.1 70.7

Summer Exam Grades

First Class 3 10.3 8.9

Upper Second Class 10.1 13.1 16.3

Lower Second Class 14.3 28.3 21.7

Third Class 22.6 21.4 23.8

Autumn exams

Sat Autumn 95.2 84.6 86.9

Pass Autumn 68.8 57.6 69.3

Progress to Second Year 82.7 86.2 88.4N 173 149 16337


% % %

Graduated 77.42 88.89 77.5

N 93 90 11921

Final Degree Classification

First Class Honours 4.29 20.51 13.78

Upper Second Class Honours 18.57 29.49 29.07

Lower Second Class Honours 45.71 30.77 38.1

Third Class Honours 31.43 19.23 19.05N 70 78 8935


60


Appendix C: Tables of Results for Chapter 4

Table C1: Impact of Access Programme on Retention Rates and Passing First Year Exams

Notes: a Estimated coefficient of linear probability models. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. Direct students are those who entered the university with reduced entry scores. Merit students are those who entered the university without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

Table C2: Impact of Access Programme on First Year Exam Performance

Notes: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%.The treatment effect is participation in the Access programme. Direct students are those who entered the university with reduced entry scores. Merit students are those who entered the university without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

In first sitting of exams Base Points <400 Points >400All Direct Merit All Direct Merit All Direct Merit

First Class Honours 0.014**

(0.007)

0.008

(0.006)

0.015

(0.009)

0.001

(0.008)

N/A

(N/A)

0.001

(0.002)

0.045**

(0.021)

0.039

(0.031)

0.034

(0.022)

Second Class Honours Upper 0.050***

(0.018)

0.047*

(0.027)

0.048**

(0.022)

0.007

(0.009)

0.002

(0.009)

0.005

(0.011)

0.086***

(0.027)

0.107*

(0.063)

0.073**

(0.031)

Second Class Honours Lower 0.055***

(0.019)

0.051*

(0.028)

0.046**

(0.019)

0.026

(0.032)

0.009

(0.042)

0.018

(0.037)

0.023**

(0.010)

0.024**

(0.011)

0.024**

(0.010)

Pass/Third Class -0.013*

(0.007)

-0.008

(0.008)

-0.028*

(0.015)

0.018

(0.023)

0.007

(0.034)

0.009

(0.018)

-0.059***

(0.020)

-0.067

(0.045)

-0.061**

(0.029)

Pass in Autumn -0.054***

(0.019)

-0.052

(0.029)

-0.041**

(0.019)

-0.015

(0.018)

-0.005

(0.026)

-0.010

(0.021)

-0.056***

(0.020)

-0.065*

(0.036)

-0.039**

(0.018)

Fail -0.053***

(0.019)

-0.046*

(0.024)

-0.039**

(0.017)

-0.037

(0.047)

-0.013

(0.060)

-0.023

(0.046)

-0.040***

(0.015)

-0.039**

(0.019)

-0.031**

(0.014)

Sample size680 535 512 303 241 183 377 294 329

Base Points <400 Points >400

All Direct Merit All Direct Merit All Direct Merit

Not Dropping out before

attempting examsa

0.034**

(0.016)

0.054**

(0.025)

0.032*

(0.017)

0.062**

(0.031)

0.054

(0.040)

0.081**

(0.038)

0.012

(0.019)

0.045**

(0.023)

0.000

(0.022)Sample size 707 558 534 322 258 197 385 300 337

Passed first year examsa 0.053**

(0.026)

0.112**

(0.044)

0.029

(0.030)

0.096

(0.058)

0.158*

(0.083)

0.070

(0.069)

0.040

(0.038)

0.111*

(0.060)

0.008

(0.043)Sample size 706 557 533 321 257 196 385 300 337


61


Table C3: Impact of Access Programme on Probability of Graduating, Probability of Graduating on time and Final Degree Classification

Notes: 1 Estimated coefficient of linear probability model. 2 Estimated marginal effects in ordered probit model

conditional on sitting final exams. Marginal effects and standard errors (in parenthesis) reported. Significance

levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. Direct students are

those who entered the university with reduced entry scores. Merit students are those who entered the university

without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose

parents are not professionals or employers and who attended schools that subsequently linked to the system.

Those from farming background excluded. The base specification includes the following control variables: faculty,

year of university entry and number of points attained in university entry exams.

All Base Direct meritProbability of graduating1 0.100***

(0.032)

0.148**

(0.064)

0.097***

(0.035)Sample size 481 391 388

Probability of graduating on time1

0.019

(0.032)

0.017

(0.057)

0.014

(0.036)

Sample size 382 305 310

Final degree classification2 First Class Honours 0.027

(0.019)

-0.007

(0.018)

0.041

(0.032)Second Class Honours Upper 0.053

(0.032)

-0.024

(0.063)

0.061*

(0.037)Second Class Honours Lower -0.026

(0.018)

0.006

(0.014)

-0.043

(0.033)Pass/Third Class -0.055

(0.034)

0.025

(0.068)

-0.060

(0.036)Sample size 383 305 313


62


Appendix D: Tables of Descriptive Statistics and Results for Chapter 5

Table D1: Descriptive Statistics: Percentage of leaving Certificate class progressing to University

0-5% 6-10% 11-20% 21-30% 31%+2001 19.9 17 19 20.2 23.9

2007 22 14 18.4 11.8 33.7

Note: Based on HEAR School Survey carried out by research team in 2008.

Table D2: Descriptive Statistics: Percentage increase/decrease in Progression

Increase No Change DecreasePercentage Change 30.39% 52.28% 17.33%

Note: Based on HEAR School Survey carried out by research team in 2008.

Table D3: Effect of length of Participation in hEAR on Progression of Students to University

more Progressing No Change Fewer ProgressingEffect of time linked to HEAR

on progression to university1

0.141***

(0.47)

-0.020

(0.019)

-0.121**

(0.052)Sample size 116

Note: 1Estimated using limited information maximum likelihood: the outcome is modelled as an ordered probit with the treatment variable a linear function of exogenous variable. The Stata “CMP: conditional mixed process” program is used. We control for school size, rural/urban location and university to which the school became linked. The local unemployment rate and the local education level are used as predictors for how long a school has been linked to HEAR. Marginal effects and standard errors (in parenthesis) reported. Significance levels: ***1%, **5%, *10%. Based on School Survey carried out by research team in 2008.

Table D4: Descriptive Statistics: Average Proportion of Schools that send at least one Student to UCD

All % New ERA link only % hEAR only %1999 64.26 68.75 63.18

2000 60.24 77.08 56.22

2001 57.03 60.42 56.22

2002 69.48 81.25 66.67

2003 62.25 66.67 61.19

2004 64.26 70.83 62.29

N 249 48 201

Note: Based on UCD administrative data 1999-2004.

Table D5: Descriptive Statistics: Average Proportion of leaving Certificate Class Attending UCD

All % New ERA link only % hEAR only %1999 3.34 5.04 2.94

2000 3.36 5.48 2.86

2001 3.13 4.46 2.82

2002 3.46 5.44 2.99

2003 3.11 4.75 2.72

2004 3.51 6.02 2.91

N 249 48 201

Note: Based on UCD administrative data 1999-2004.


63


Table D6: Effect of Being linked on the Progression of Students to UCD

All Schools New ERA link only hEAR onlyProbability of school sending at least one student to UCD2 0.073***

(0.027)

0.131**

(0.051)

0.003

(0.034)Proportion of students from Leaving Cert

class progressing to UCD

0.005***

(0.002)

0.013***

(0.005)

0.000

(0.002)Number of Schools 249 48 201

Note: 2Estimated marginal effect of tobit model, controlling for year and unemployment rates in school locality. Marginal effects and standard errors (in parenthesis) reported. Significance levels: ***1%, **5%, *10%. Based on UCD administrative data 1999-2004.

Table D7: Impact of variation in Financial Aid on First Year Exam Performance

In first sitting of exams New ERA students

First Class Honours 0.027


(0.040)Second Class Honours Lower 0.064

(0.049)Pass/3rd Class in Summer -0.007

(0.009)Pass in Autumn -0.070

(0.051)Fail -0.071


Note: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is being in first year during a high value financial support year (2000, 2001, 2003). The base specification includes faculty and number of points attained in university entry exams.

Table D8: Financial Aid to New ERA Students

New ERA Financial Supports local Authority Grant Total Financial Supports1999

€2,249 €2,925 €5,1742000

€3,261 €2,909 €6,1712001

€3,412 €2,948 €6,3602002

€2,516 €2,972 €5,4892003

€3,091 €3,316 €6,4072004

€2,234 €3,323 €5,557Average over all Years €2,794 €3,066 €5,860

Note: All amounts in 2008 €


64


Table D9: Impact of the New ERA Programme Relative to being a Disadvantaged Student in a Non-Disadvantaged School on First Year Exam Performance

In first sitting of exams New ERA students versus students fromnon-disadvantaged schools

First Class Honours 0.021


(0.018)Second Class Honours Lower 0.032

(0.009)Pass/3rd Class in Summer -0.024

(0.009)Pass in Autumn -0.046

(0.014)Fail -0.038


Notes: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that are not disadvantaged and which will only be eligible to participate in HEAR in 2010. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.


65


evaluating the impact of the ucd new era widening participation initiative

Documents

ucd new era programme

impact of new erahear

support of ucd registry

ucd urban institute

description of data

colm harmon ucd

exams data

administrative data