Evaluating the Impact of the UCD New ERA Widening Participation Initiative by Advantage Point

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction

Evaluating the Impact of the UCD New ERA Widening Participation Initiative Kevin Denny, Orla Doyle, Marie Hyland, Patricia O’Reilly, Vincent O’Sullivan

UCD GEARY INSTITUTE University College Dublin

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Contents

Acknowledgements Glossary

i ii

Executive Summary

1 Introduction

1.1 1.2 1.3 1.4

Background Formation of the New ERA Programme Description of the New ERA Programme Overview of Report

3 4 5 7

2 Literature Review

2.1 2.2

Policy Context: Access in Ireland Evaluating the effectiveness of financial and social support

9 11

3 Evaluation Methods and Data

3.1 3.2 3.3 3.4 3.5

Evaluation Design Selecting the Control Group Description of Data Data Limitations Description of Sample

15 15 17 18 19

4 Results

4.1 4.2 4.3

Interpreting the Results Evaluation Results Main findings in this chapter

29 30 35

5 Further analyses

5.1 5.2 5.3 5.4

The Impact of New ERA/HEAR on Progression to University The Impact of New ERAâ&#x20AC;&#x2122;s Financial Aid on Student outcomes Likely Consequences of the National HEAR scheme. Main findings in this chapter

37 43 44 46

6 Recommendations and findings

References Appendices

50 53

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Acknowledgements

Acknowledgements This report was produced under the ‘Evaluating the Impact of the UCD New ERA Widening Participation Initiative’ which was conducted at the UCD Geary Institute and funded by the Strategic Innovation Fund in conjunction with the UCD New ERA Programme and NUI Maynooth Access Programme (MAP). We would like to thank our colleagues at New ERA and NUI Maynooth in particular Fiona Sweeney (UCD New ERA Coordinator) and Ann O’Brien (Director of Access, NUI Maynooth) for all their help and support. We would particularly like to thank Áine Galvin for supporting and driving this research from the beginning and to Ronan Murphy (acting Director of Access and Lifelong learning) and Anna Kelly (current Director of Access and Lifelong learning) for their time and commitment to this project as members of the Steering Committee. This study could not have been conducted without the help and support of UCD Registry for assisting us in collating the admissions and exams data used in the analysis. We would particularly like to thank Susan Mulkeen from Admissions, Ciarán Ó hUltacháín and Jill O’Mahony from Assessment, and Paula Tarrant, the Director of Operations. We would also like to thank Maria McDonald from Management Information Services (MIS) in IT services for providing the exams data. Thanks also to James McBride (Director of the Irish Social Science Data Archive) for anonymising the administrative data used in the analysis. We would like to thank the principals and teachers at the schools that responded to our HEAR survey. In addition, we would like to thank Susanne Schmidt from the UCD Urban Institute for producing the maps used in the report and also to Patrick McKay for the graphic design work on this report. Our thanks also to Colm Harmon (UCD), Ian Walker (Lancaster University), Robin Naylor (University of Warwick), Asako Ohinata (University of Warwick) and Arnaud Chevalier (Royal Holloway, University of London) for providing helpful comments and ideas on this evaluation strategy. Finally, we would like to thank the many others who have provided guidance and advice on this study including Danny Moran and Abi Campbell. It should be noted that the interpretation of the results and the recommendations are those of the research team. This report was written by the research team at the UCD Geary Institute Kevin Denny, Orla Doyle, Marie Hyland, Patricia O’Reilly, Vincent O’Sullivan

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Glossary

Glossary BITE Ballymun Initiative for Third Level Education CAO Central Applications Office DEIS Delivering Equality of Opportunity in Schools DIT Dublin Institute of Technology GPA Grade Point Average HEA Higher Education Authority HEAR Higher Education Access Route HEDAS Higher Education Direct Application Scheme HEI Higher Education Institution MAP Maynooth Access Programme NESF National Economic and Social Forum NDP National Development Plan NUI National University of Ireland OECD Organisation for Economic Cooperation and Development RCT Randomised Control Trial SES Socio-Economic Status TAP Trinity Access Programme UCD University College Dublin

Executive Summary

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Executive Summary

Executive Summary This study is a quantitative evaluation of the effectiveness of New ERA, UCD’s access program. New ERA is a multi-dimensional programme that works with designated disadvantaged schools to increase the number of socio-economically disadvantaged students progressing to higher education and to improve student outcomes once at UCD. Access students entering UCD also receive further support including financial aid, mentoring and academic supports. A proportion of New ERA students also benefit from a lower Leaving Certificate points requirements (“Direct” students) with the other New ERA participants (“Merit” students) being required to obtain the same points as other students. The study uses a quasi-experimental design to quantify the effect of New ERA on a set of outcomes including progression to university and the academic outcomes of the students at the end of their first and final year in UCD. The main results are discussed below. • Participation in New ERA has a positive effect on reducing first year withdrawal rates. New ERA reduces the probability of withdrawing prior to first year exams for low point Merit students and high point Direct students. • New ERA has a positive effect on improving first year exam results by shifting students up the grade distribution. Participation in New ERA increases the probability of achieving a First and Second Class honours and reduces the probability of failing or receiving a Third Class honours/Pass in the first year exams. These improvements in exam performance only benefit those who enter UCD with more than 400 Leaving Certificate points. Participating in New ERA increases the probability of graduating from university. The result applies to both Direct and Merit students. • The programme has relatively little effect on the final degree classification the students receive. However, Merit students have an increased chance of attaining higher grades. • Once a secondary school becomes part of the Higher Education Access Route (HEAR) scheme there is a higher probability that it will send more students to university. It was found that becoming linked to HEAR between 2001 and 2007 increased the probability of sending a higher proportion of students to university by 14%. • Being linked to New ERA increases the proportion of the Leaving Certificate class progressing to UCD by about one percentage point. • Changes in New ERA’s financial aid package to students, taking into account the Higher Education grant, do not have a measurable effect on student outcomes. • Comparing New ERA students with students who are also socially disadvantaged but who do not qualify for the programme (as they attend a non-disadvantaged school), the results show that New ERA students academically out-perform their comparators. This suggests that the new national HEAR scheme, which allows students attending non-DEIS schools to participate in New ERA, may be beneficial. The report ends with a series of recommendations, some specific to UCD and others which have more general implications. These include relaxing the criteria for disadvantage status so that more schools are linked to access programmes, increasing the number of places allocated to “Direct” students particularly in high point courses, and taking steps to increase the number of “Merit” students by expanding the outreach activities at secondary level.

Chapter 1 Introduction

CHAPTER 1 INTRODUCTION

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction

1 Introduction 1.1

Background

A person’s education is one of the principle influences on their outcomes in life. Education is an important determinant of an individual’s income, where they live, and how they participate in society. It can also affect how people bring up their children and how they manage their own health. There is ample evidence of significant benefits to being more educated and Ireland is no exception to this trend.1 The ongoing growth of the information society, the emphasis on innovation and increased globalisation are likely to further increase the importance of education. Therefore, a strong argument can be made that society should invest in education to ensure that people’s education is not constrained by their parents’ socio-economic status (SES). Note that a concern over equity is not the only reason why we should be concerned about access to education: it is also an issue of efficiency. While equity considerations are central for increasing access to education, another consideration is economic efficiency. For our society to prosper it is essential that skills and talent of young people are not wasted, just as a firm needs to ensure that its resources are used to the fullest extent. Despite the near universal agreement on such equity consideration, it is striking that such a strong correlation exists worldwide between people’s education and their parents’ SES. While parental background acts as a constraint to educational attainment for many people, it is worth noting that the extent of this constraint is probably higher in Ireland than most Western countries. Denny et al. (2009, Table 12.3) measure the correlation between parental SES and education in 17 OECD countries and finds that father’s education has the highest impact on the level of education attained by an individual in Ireland. The high dependence of attendance at third level education on SES has also been well documented in Ireland in a series of reports published by the Higher Education Authority for example Clancy (1982, 2001). While children from disadvantaged backgrounds are less likely to go to university, it is not obvious why exactly this is so. Possible explanations point to a lack of financial and social resources, a lack of knowledge about the benefits of education, or attendance at schools which do not promote higher education. In reality it can probably be attributed to some combination of all of these. However, and perhaps surprisingly, we do not have good estimates for Ireland of the relative contribution of these factors. This represents a significant limitation: if policy makers are to implement programmes to improve access to education they need to know what the barriers to education are and where the investment will be most effective. Given the very difficult budgetary situation that the Irish government now faces this is particularly salient. While the lack of knowledge of the precise causes of educational inequalities is striking, it is also noticeable that policy makers have been active, particularly in recent years, in generating initiatives to address these concerns. This report is one step forward in providing a quantitative assessment of the effectiveness of one such initiative, the New ERA Widening Participation Initiative which operates at University College Dublin. The primary aim of the New ERA programme is to promote and support the participation of students from lower socio-economic groups who, in general, are under-represented in higher education. The access programme provides secondarylevel students from designated disadvantaged schools with supported entry mechanisms to study at UCD and financial support for the duration of their university life, in addition to a host of support services both before and after they join UCD. New ERA began in 1997 and now operates under the Office of the Director of Access and 1 See Machin & Vignoles (2005) for an introduction to education economics.

1: Introduction

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Lifelong Learning. To date, the effectiveness of the programme has not been assessed by quantitative research. If the New ERA programme is to develop and maximise its potential, then an evaluation of the programme which is scientifically rigourous, using the best available data and matched with advanced statistical techniques, is necessary. This study conducts a retrospective evaluation of the New ERA programme between 1999 and 2004 across different outcomes. This time period was prior to the introduction of modularisation in UCD. The study examines the performance of New ERA students post-entry, specifically the impact of New ERA on first year withdrawal/ retention rates and exam performance. The evaluation also tests the effectiveness of New ERA on final year graduation rates and degree classification. Finally, the evaluation examines progression rates by quantifying the effect of a designated disadvantaged school joining the New ERA programme on increasing participation in thirdlevel education. In addition to examining the impact of the programme for all New ERA students, the study also distinguishes between students entering UCD on points concession and those who enter without points concession, i.e. though the normal CAO route. These are called Direct and Merit students respectively. This distinction will be explained more thoroughly later in this chapter. The evaluation involves the analysis of a number of secondary data sources – the UCD New ERA database and the UCD students’ administration database. It is necessary to use both datasets in order to compare the outcomes of the access students to the students from the general university population. Therefore in the language of the programme evaluation literature, the current and past New ERA students are the “treatment group” whilst a matched sub-sample from the general university population, which did not receive the programme, are the “control group”. By comparing the difference in outcomes between these two groups it is possible to identify the effect of the programme. The potential benefits of this study are twofold. First, it will contribute to the international literature on access programmes, where such rigourous quantitative studies are few. And second, the results from this study will help determine the effectiveness of the New ERA programme specifically, and help inform policy about the future development of such programmes internationally. The reminder of this chapter will discuss the policy context in which New ERA has evolved and developed. Section 1.2 describes the formation of New ERA. Section 1.3 describes the New ERA programme. Finally, section 1.4 presents an overview of the report.

1.2 Formation of the New ERA Programme The New ERA programme evolved out of a movement, arising in the early 1990s, to improve access to higher education for people from disadvantaged communities. A significant step forward was the formation of area based partnerships in 1991. This, along with the community platform, supported initiatives aimed at helping students to access third level education. One such pilot programme was Ballymun Initiative for Third Level Education (BITE), established in 1990. This scheme laid the ground work for targeted intervention in Ireland by tackling educational barriers through homework clubs and Leaving Certificate tuition, social barriers through creating awareness of third level for both parents and pupils and financial barriers to higher level through the provision of scholarships at third level. The scheme supported students at primary and second level (pre-entry) and after students entered higher education (post-entry). The foundations for New ERA were laid in 1994 by the then registrar, Professor John Kelly through the set up of the UCD Committee on Equality of Participation, and sustained by his successor Dr Caroline Hussey. The committee commissioned Lynch and O’ Riordan’s study entitled ‘Social Class, Inequality and Higher Education; Barriers to Equality of Access and Participation among school leavers’ in 1995 to identify barriers against participation. This work was commissioned in order to develop a frame of reference for how UCD would tackle access. New ERA followed in 1997.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction

In 2001 seven higher education institutes, Trinity College Dublin, University College Dublin, Dublin City University, NUI Maynooth, NUI Cork, University of Limerick and the Dublin Institute of Technology collaborated to create the Higher Education Access Route (HEAR) scheme2. Through this scheme students could apply to the participating institutions through the CAO and a supplementary application. The scheme also allowed Access Offices to avoid duplicate applications and to pool their reserved places. The HEAR Scheme allows school leavers from linked schools to apply for reduced point entry to any participating HEI along with post-entry supports. The scheme was initially supported by the HEA’s Targeted Initiative Scheme Innovation Fund and all other cost is divided between the seven participating organizations. More recently, the development of the HEAR scheme into a national scheme has been financed by the Strategic Innovation Fund The HEAR Scheme is, in part, a response to the suggestion made in the White Paper ‘Charting our Education Future’ that designated disadvantaged schools should become linked to higher level institutions. In relation to the financing of access initiatives in Ireland, the HEA provided funding to the universities, under the Targeted Initiatives, to develop special schemes to improve the participation of students from disadvantaged social backgrounds (Osborne and Leith, 2000). Also, the National Development Plan (NDP) set aside finances for a third level access fund for the period 2000-2006, aimed at tackling under-representation by students from socio-economically disadvantaged backgrounds, mature students, students with disabilities and students from ethnic minorities. The universities themselves also support the initiatives financially. Funding also became available directly for students through the European Social Fund. Some of this financial support was kept centrally and used as a college-wide student assistance fund.

1.3 Description of the New ERA Programme The New ERA programme has existed since 1997. Its aim is to encourage and facilitate increased participation in higher education by students from socio-economically disadvantaged backgrounds that are underrepresented at third level. Broadly speaking, socio-economically disadvantaged students are defined as those from a background affected by long-term unemployment, low family income and/or little or no tradition of progression to third level education. Within University College Dublin New ERA operates under the remit of the Registrar and administers the Higher Education Access Route on behalf of UCD3. Specifically, a student is considered eligible for New ERA if: • there is no previous history of progression to higher level education in their family • their family income is below a certain level • they are students at a designated disadvantaged school • they are a member of the six under-represented groups outlined by Patrick Clancy in The Social Background of Higher Education Entrants (2001). • These under-represented groups are: unskilled manual workers, semi-skilled manual workers, other non-manual workers, intermediate non-manual, skilled manual and agricultural workers. Lynch and O’ Riordan’s (1995) report provided the framework for the development of the New ERA programme. This report explored and documented viewpoints on disadvantage from four groups experiencing it. These groups included: community activists, school personnel, low income third level students and second level students intending to progress to third level. The report highlighted three main types of barriers, financial, educational and socio-cultural, that hindered students in accessing and progressing in higher level education. It found that all four groups interviewed considered financial constraints to be the most substantial barrier to higher level education. Both educational and cultural barriers were considered important. New ERA aimed to tackle these barriers through providing support to students both before 2 HEAR Scheme has gone through several incarnations and is known as: 2000 - New ERA Direct Entry, 2001 - Common Application Form, 2002 Direct Application Scheme, 2003 to 2004 Higher Education Direct Application Scheme and 2005 - 2009 Higher Education Access Route. For the purposes of this Report it will be known by the most recent title the Higher Education Access Route (HEAR). Appendix A contains a set of maps showing the distribution of link schools and the year in which they joined the HEAR scheme. 3 More information on New ERA is at http://www.ucd.ie/access/newera.htm

1: Introduction

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

and after they enter third level education. Prior to entry, New ERA aimed to tackle the socio-cultural barriers to higher level education experienced by students from lower socio-economic groups through a variety of outreach measures aimed at creating awareness and providing information to second level students. Pre-entry programmes of action focus on raising student aspirations, creating an awareness of college, and the provision of academic support for the Leaving Certificate. New ERA visits designated disadvantaged schools linked to the programme and provides information to students about the HEAR scheme and its supports. It also organises pre-entry orientation programmes and shadowing days where second level students followed a third level student through a day in the life of the university. On a community level it has given presentations to meetings of parents and contributions to community based events. UCD students provide one to one academic support for Leaving and Junior Certificate students on a voluntary basis. In addition, Leaving Certificate revision workshops are organized for 6th year students. For younger students, there is the “Uni4U Summer School” for 2nd year students where participants attend taster lectures, science labs, field trips as well as sports and social activities. An example of a collaborative outreach initiative which New ERA is a part of with DCU, DIT, NUI Maynooth and TCD is the “Take 5” summer project involving participants from designated disadvantaged second level schools attending each of the five participating higher education institutions for a day. This project aims to introduce students to the physical, academic, cultural, sporting and social environments of the different institutions through a range of activities, such as academic workshops, laboratory sessions, and project group work. New ERA’s outreach activities are not confined to secondary schools, there are also activities involving primary schools under the “Steeping Stones” programme. This programme works with 31 primary schools that feed into the linked secondary schools. Activities include a visit to the campus for 5th class students. Students must apply to New ERA through the HEAR scheme and through the CAO scheme. Applicants must submit supporting documentation to verify their socio-economic status, references from their school teachers and meet basic course requirements set by the university. Students are initially deemed eligible for HEAR on the basis of their socio-economic and once they are deemed academically eligible, are considered for a place on the New ERA scheme. These eligible students falls into two categories. The first group of students are admitted to the university under the normal clearing system used in Ireland, which is a nationally administered system4. About 45% (of about 100-140 students per annum) of New ERA students attain sufficient points to meet the minimum CAO points level and are allocated a place on their preferred course in the usual manner. These students are known as Merit students. The remainder of applicants receive preferential treatment in attaining their place in university. These students did not meet the minimum CAO points level for their chosen course. Instead there are a certain number of places on each degree programme reserved for these students. These students are known as Direct students. To be offered one of these places they must meet minimum matriculation and course requirements, but receive a concession of up to 20% on the CAO entry points required for the programme. In addition, further information showing their personal circumstances are considered (such as the level of educational qualifications of their parents) as well as references from teachers. The number of minimum reserved places on each course is calculated on the size of each course and is relatively fixed. If there is a surplus of suitable and eligible applicants for these places, the limited places are awarded on the basis of points. Since 1997, third level tuition has been free to all students in the Republic of Ireland. All students also pay a registration fee of several hundred Euro, however, for students who are in receipt of the Higher Education Grant, this fee is covered by their local authority. As almost all New ERA students also receive this grant, they do not pay this administration fee. 4 This is based on a supply-and-demand based system of allocating degree course places. Students are ranked by converting their leaving certificate results into points using a common scale. The scale takes the best six subjects and has a range of 0 to 600 in increments of 5. The minimum points level for a degree programme fluctuates from year to year. Applications are anonymous in that it does not involve interviews or submission of a personal statement.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction

Post-entry, financial support is provided to all New ERA students in the form of a top-up grant (in addition to the usual local government grant, which they would normally receive). New ERA also provides a contribution towards additional course such as lab coats, stethoscopes, etc. In the case of students attending the Quinn School of Business, this aid is quite considerable as it covers the cost of laptop computers, which are compulsory for all students in the business school. Students also receive book tokens to contribute towards the cost of textbooks. Post-entry educational and socio-cultural supports include free additional tuition (if required), mentoring from student advisors, and a pre-term orientation week where students live on campus with other New ERA students to encourage early social and academic integration. Post-entry, New ERA aims to tackle social isolation through personal support and advice, facilitating group events and occasional social events, and by monitoring and tracking studentsâ&#x20AC;&#x2122; progress. New ERA students can mix with one another and share their experience of university.

1.4 Overview of Report The remainder of the report is organised as follows: Chapter 2 provides a literature review on the effectiveness of different forms of international access programmes based on rigorous evaluation. Chapter 3 presents the methodology and data used to conduct the evaluation. Chapter 4 presents the results of the evaluation of student exam results. Chapter 5 primarily studies the effect of the programme on progression to university. Issues such as fluctuations in financial aid are also examined. Chapter 6 presents the reportâ&#x20AC;&#x2122;s recommendations.

Chapter 2 Literature Review

CHAPTER 2 LITERATURE REVIEW

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

2: Literature 1: Introduction Review

2 Literature Review This chapter briefly outlines some of the background to the present study. Section 2.2 describes the policy background in Ireland noting how thinking on dealing with educational disadvantage in general, and access programmes in particular, has evolved. It also discusses some of the evaluations of policy that have been carried out in this area. It should be noted at the outset that these evaluations, which are essentially qualitative reviews of the issues, are very different from the present exercise which is about quantifying the effectiveness of a particular intervention i.e. the New ERA programme. In the absence of good estimates of the effectiveness of access initiatives it is unclear how one can go about designing interventions to tackle educational disadvantage. It is striking that quantitative assessments of the effectiveness of educational interventions are extremely rare in Ireland and it is this gap which the present study fills. Studies such as this are more common internationally and Section 2.3 describes some of these key studies. The aim is not to provide an exhaustive survey of the literature but to provide a general overview of the research literature and some representative results.

2.1

Policy Context: Access in Ireland

A number of government acts and policy documents provide the historical overview of access policy in Ireland which led to the development of access programmes and initiatives. Key developments included the Higher Education Authority Act (1971) and the Universities Act (1997) which brought equality in accessing higher education to the foreground. Several reports were produced that played key roles in setting the agenda for the development of access initiatives through identifying barriers to higher education and providing recommendations as to how these barriers could be addressed, these include the Green Paper: Education for a Changing World (1992), the White Paper: Charting our Educational Future (1995), and The Report for the Action Group on Access to Third level Education (2001). The Green Paper: Education for a Changing World (1992) argued that objectives for improving transfer rates to third level education should include The development of direct links between third level institutions and selected schools… [and] … the development of support and access programmes to increase access and improve retention. The White Paper Charting our Educational Future (1995) recognised the effectiveness of securing a set number of places for those from disadvantaged backgrounds as a means of tackling educational disadvantage. The aim was to admit an additional 500 students from lower-socioeconomic groups into third level institutions each year for the subsequent five years. According to Carpenter (2004), the White Paper aims for this policy to be delivered by the third level sector as a whole and not the universities alone. Furthermore, it was recommended that undergraduate fees in higher education institutions be abolished from 1996/97, that the criteria for the allocation of student grants be revised, and that the value of the grant be increased. In order to support students at second level the White Paper stated that: each third level institution will be encouraged to develop links with designated second-level schools, building on existing good practice. In the short term, it was envisioned that all designated disadvantaged second level schools would become linked to a third level institution which would support programmes to create awareness of the opportunities for and advantages of third level education. Some of the strategies outlined include ‘awareness seminars’ and open days for students. Institutions were also encouraged to help students make the transition from second level education to third level through post entry support programmes. It should be noted that undergraduate fees were indeed subsequently abolished in 1996/97. However it is unlikely that this would have led to wider participation by people from a low income background as they would normally have been in receipt of local authority grants and hence not liable to pay fees.

1: Introduction 2: Literature Review

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

The Report for the Action Group on Access to Third level Education (2001) led to the development of the National Office for Equity of Access to Higher Education in 2003 with the mandate of coordinating efforts to tackle educational disadvantage and to develop a national strategy. This office is now tasked with facilitating access to education for groups who are typically underrepresented in higher education. Evaluations of Irish Access Initiatives: To date several evaluations have been conducted in an attempt to develop best practice in dealing with access and participation in Irish higher education. Previous evaluations of such initiatives include: Skilbeck and Connell (2000), Osborne and Leith (2000) and the Report of the High Level Group on University Equality and Policies, commissioned by the HEA Equality Review Group in 2004. Skilbeck and Connellâ&#x20AC;&#x2122;s report Access and Equity in Higher Education: an International Perspective on Issues and Strategies (2000) recognizes that the issue of equity in higher education is not just educational but also economic, cultural and social. It states that educational polices alone cannot abolish educational disadvantage and recommends that further progress will require more coherent, coordinated approaches across several sectors. It argues that opportunities for access should be extended through all levels of the education system. The report recognizes that well-targeted financing will continue to play an important role in equity strategies, but that it is inefficient to admit large numbers of students and then to accept high failure and drop-out rates.

Osbourne and Leith (2000) in Evaluation of the Targeted Initiatives on Widening Access for Young People from Socio-economically Disadvantaged Backgrounds recommend that a national strategy should provide a broad framework in which individual universities and other institutions can evolve distinctive practices which reflect their own circumstances. They also recommend that a national strategy funded by the state needs to have measures in place for progress to be assessed. According to the Report of the High Level Group on University Equality and Policies (2004), while there are benefits for individuals involved in targeted initiatives, the scale and impact of these special initiatives is very small: according to one university disadvantaged students represent only 3% of the student body. This leads to the statement that there can be little prospect in the short term that the continuation of these initiatives, even on a substantial level, will bring about radical changes in representation. However it was recognized by the report that these initiatives do keep the issue of representation to the forefront, demonstrating what can be achieved on the ground when resources are specifically targeted at disadvantaged groups. Throughout the consultation process for the report, the idea emerged that access could be improved by incorporating weighting in favour of students who are severely disadvantaged, and that, in establishing goals for the education system generally, the focus should Shift from the concept of participation (e.g. prevention of early leaving from school programmes leading to the Leaving Certificate) to benefit (outcomes for the individual, irrespective of where the learning has taken place). The team also highlighted that: The point was made repeatedly to the review team that intervention should be made at an earlier stage in second level education. The report recognized that there was a need for on-going evaluation and improved data collection. It was acknowledged that the cause of education disadvantage was not limited to the education system. Parents, peer groups and community factors all play a part in the formation of student expectations and aspirations.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

2: Literature 1: Introduction Review

Throughout previous evaluations of access initiatives in Ireland there was widespread recognition that educational disadvantage did not begin at second or third level and the identification and support of students at risk needed to begin at an earlier stage in their education. Many studies have been produced which support this view. According to Carneiro and Heckman (2003), the bulk of the evidence indicates that public expenditure would be more efficient if resources invested in human capital were more strongly directed towards younger members of the population. A research paper for the NESF entitled The Economics of Early Childhood Care and Education (2006) finds that the greatest societal returns to education come from investment in early or primary education, whereas, in the case of investment in higher education, societal gains are smaller and the returns to the individual are higher. Researchers at the Geary Institute (UCD) are currently carrying out an evaluation of the effectiveness of an early childhood intervention programme called Preparing for Life which aims to improve the school-readiness of children from designated disadvantaged backgrounds entering primary school.

2.2 Evaluating the effectiveness of financial and social support It has long been recognised that the rate of return for individuals who invest in higher level education is high. A study by Harmon, Oosterbeek and Walker (2003) presents estimates on the returns to earnings from an additional year of education in the U.K. which vary between 7% and 15%, depending on the estimation procedure used. They also show that the returns to education are higher in Ireland and in the U.K. than in the rest of Europe. However, not all groups in society invest equally in third level education regardless of the high returns to education and in spite of the fact that much effort has been made in recent decades to increase the participation and completion rates of students from underrepresented backgrounds. Many governments have implemented policies which aim to promote â&#x20AC;&#x2DC;equality of opportunityâ&#x20AC;&#x2122; for students from disadvantaged backgrounds. Universities have established access programmes that are designed to boost enrolment and persistence of these groups. However the effectiveness of these programmes needs to be examined in more detail to understand why underrepresentation, withdrawal and non completion remain enduring problems. The key areas focused on when discussing the effectiveness of access programmes are access/ entry/progression, retention, exam performance, withdrawal/non-completion and graduation. There are numerous access initiatives worldwide which aim at affecting the outcomes mentioned above, the majority being implemented in Anglo-Saxon countries. Many of these initiatives focus on tackling barriers to access, financial barriers in particular, and for this reason there is a large quantity of literature that concentrates on the effectiveness of financial aid. The two main types of financial aid are need-based and merit-based aid. Eligibility for needs-based aid is based on certain criteria, such as means testing, and often takes the form of a grant. Needs-based financial aid is designed to improve the enrolment, retention and graduation rates of students from socio-economic or minority groups that are often under-represented in higher level education. Several studies have been carried out to assess the effectiveness of financial aid on improving the outcomes of students typically under-represented in higher education. A study by Dynarski (2003) found that such aid has a significant effect on the outcomes of these students. In order to estimate the true effect of aid on student outcomes the study examines the abolition of the Social Security Benefit Program in the U.S. which ran from 1965 to 1982 and, during this period, paid for millions of students to go to college. This was one of the most dramatic changes in college aid allocation that has ever occurred in the U.S. Using this policy change, Dynarski found that needs-based aid significantly affects the probability that a person will enrol in third level education. The results show that giving a grant of $1,000 to an individual increases the probability that they will attend college by approximately 3.6 percentage points. Other studies have also found a positive relationship between needs-based aid and college enrolment. McPherson and Schapiro (1991) use data on enrolment, tuition and financial aid for population subgroups in the U.S. over the 1974 â&#x20AC;&#x201C; 1984 period and find that an increase in the net costs of college attendance has significant negative impacts on college enrolment for low income white families. The net cost to students will be affected by the level of fees and any remission of fees payable by students. Their results also found that there was no significant effect on higher income families who are not cash constrained.

1: Introduction 2: Literature Review

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Brock (2006) studied the effects of a Louisiana needs-based scholarship programme on course completion and exam performance of low-income parents attending community college. Students were given scholarships of $2,000 for the year if they attended college at least half-time and attained, on average, a C grade. Students were randomly assigned to a programme group that could avail of the scholarship, and a control group that could not. While both groups could avail of counselling services, the programme group were obliged to attend student counselling in order to receive the financial aid. The programme group also benefit from their counsellors having a lower caseload. The primary focus of the counsellors was monitoring the academic performance of students and issuing the scholarships. The difference between the outcomes of the two groups represents the effect of the programme. The results show that students who were assigned to the programme group were more likely to be full-time college students, passed more college courses and earned more credits, and were more likely to register for their second and third years of college. Bettinger (2004) studies the relationship between needs-based grants and college drop out behaviour and finds that there is a significant relationship. This study used data related to the Pell Grant programme which is the largest means-tested financial aid programme available to students across the U.S. The study exploits differences in the amount of aid paid to students over time to estimate the effect of the grant. The results show that a student whose grant increases is less likely to withdraw from higher level education; more specifically the results imply that a $1000 increase in a studentâ&#x20AC;&#x2122;s grant reduces the probability of withdrawal by nine percentage points. More recently needs based financial aid is being coupled with other forms of aid such as outreach and intervention programmes that aim to influence access and retention. A study by Bergin, Cooks and Bergin (2007) examines a programme that aims to increase the participation of youths from typically under-represented groups in higher education. Students with a B average grade were randomly assigned to a programme or control group. Students assigned to the programme group participated in activities to increase college awareness; they were also given academic and financial support. The results showed that the programme did not have a significant effect on enrolment rates, nor did it improve their high-school results or increase their self-esteem. However, the authors found that programme participation did increase a studentâ&#x20AC;&#x2122;s desire for further education. Angrist, Lang and Oreopoulos (2009) conducted an evaluation in which first year college students were randomly assigned, subject to their written consent, to one of three programme groups or to a control group. One programme group was given financial support, another was given academic support and another was given a combination of the two. The evaluation analysed the outcomes of students from each of these three groups relative to the control group. The results showed that while the effect of academic support was small and insignificant, both groups in which financial aid was given had positive and significant results, the effect was stronger in the case where students were given a combination of financial and academic support. Further analysis showed that the effect is particularly strong for women and that this is driving the overall result. The programme had, in fact, no significant impact on male outcomes. This result is similar to that of Dynarksi (2008) who analyses the HOPE programme and finds stronger effects for women. The findings of the Angrist et al (2009) are particularly interesting as they showed that a combination of social and financial supports are more effective than either intervention alone. Finally, Lesik (2006) examines the effects of an academic support programme on student retention. The study measures the impact of a mathematics programme on the withdrawal rates of students who are, on average, comparable in all aspects except that one group participate in the development mathematics programme and the other does not. Students were assigned to the programme based on their results in a mathematics exam which students must take prior to their first year of college. A cut-off was set and students who scored below this level were obliged to participate in the programme. The vast majority of students in this study complied with this assignment rule. Lesik estimated the causal impacts of the programme by comparing the outcomes of students with results just above and just below the cut-off. The study found a positive relationship between developmental mathematics programme and student retention: students who participated in the programme were found to be at a significantly lower risk of withdrawing from third level education.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

2: Literature 1: Introduction Review

Overall it has generally been found that social and academic supports have a positive effect on student enrolment in third level education, on exam performance and on retention rates. The current evaluation of the New ERA programme, which provides students with a combination of social and financial supports, makes an important contribution to the existing literature on the effectiveness of educational interventions. It is the first quantitative evaluation of an access programme that has been carried out in Ireland to date. The aim of the New ERA programme is to promote and support the participation of students who are typically underrepresented in higher education, and, using rigorous econometric techniques, the current evaluation aims to assess the impact of the programme on the outcomes of these students. Evaluations such as this are important because, in analysing the effectiveness of access initiatives such as New ERA, methods for improving the overall design of educational interventions can be extrapolated.

Chapter 3 Evaluation Methods and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

3 Evaluation Methods and Data 3.1 Evaluation Design The key aim of all evaluations is to determine whether the social programme or policy is effective. This involves finding the counterfactual: what would have occurred in the absence of the programme? In regard to the New ERA programme, the evaluation needs to determine how the treated students (i.e. those who received the programme) would have performed if they were not New ERA students. The evaluation addresses how the New ERA students would have performed in terms of university entrance, first year exam results and final graduation rates, if they had not taken part in the programme. A simple evaluation design which compares New ERA students to the general population of the university will not reveal the treatment effect of the programme, as the estimates of the programme may be biased if the differences that led one group to enter the New ERA programme and the other group to enter the control group may also be related to the outcomes of interest, i.e. exams performance etc. For example, New ERA students may have different characteristics to the general UCD student body as they are selected into the programme based on certain socio-demographic criteria. New ERA students are, by design, from lower socio-economic backgrounds, with lower parental education and income. These characteristics are likely to influence a student’s performance in university. Therefore a direct comparison of the New ERA students to the general student body would not result in a reliable estimate of the impact of the New ERA programme. Randomised control trials (RCTs) are considered the ‘gold standard’ in evaluation research (Burtless, 1995) as they allow one to closely approximate a true counterfactual, i.e. what would have occurred in the absence of the programme. The use of random allocation to generate unbiased control groups ensures that any observed differences between the treatment and control group outcomes are likely due to the intervention rather than any other factors (Oakley et al. 2003). However, RCTs are still relatively rare within the social sciences, especially in Ireland where rigorous policy evaluation is a relatively new phenomenon. In addition, random assignment is often not ethically or practical feasible, as is the case for the New ERA programme which has not been operating on the random assignment of students. Therefore, an RCT evaluation design cannot be used. Evaluations of social programmes have therefore relied on quasi-experimental methods which simulate the conditions of an actual experiment without using random assignment (Angrist and Krueger, 1999). The aim of quasi-experimental designs is to identify a suitable control group which is similar to the treatment group on all observed and unobserved characteristics, apart from their lack of participation in the programme. Quasi-experiments are also known as “natural experiments”. The effectiveness of the New ERA programme is therefore evaluated using the quasi-experimental design. This evaluation strategy exploits the gradual expansion of New ERA over time to identify a suitable control group. New ERA has grown in the number of students that participate, but also in the number of “link schools”. By comparing the outcomes of New ERA students to students who went to New ERA linked schools before they became linked, we can determine if the programme has an impact. This control group is therefore used to “difference out” confounding factors and isolate the treatment effect, i.e. the impact of New ERA. The technique used is commonly employed in the programme evaluation literature. A recent example in a similar context being Lavy and Schlosser (2005) who evaluate an Israeli education programme aimed at improving exam grades.

3.2 Selecting the Control Group To carry out this technique the control group must be as close to the treatment group in as many respects as possible. To be eligible for inclusion in the New ERA programme a student’s family has to meet four criteria: 1) income eligibility 2) educational criteria 3) socio-economic status 4) student attended a “link” school.

1: 3: Introduction Evaluation Methods and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Family Income The income eligibility for the New ERA programme corresponds to the eligibility for the regular means tested Local Authority grant which is available to all students whose family income is under these income thresholds, regardless of the status of the school they attended. Therefore, as the data does not include family income, one of the selection criteria for choosing the control group is based on receipt of the state grant. Parental Education & Socio-economic Status In order to be eligible for the New ERA programme there must be no previous history of progression to higher level education in the family i.e. students’ parents must not have gone to third level. However, as measures of parental education are not available in the data, socio-economic status is used as a proxy. The social-economic status criteria for programme eligibility include unskilled manual, semi-skilled manual, skilled non-manual, and non-farming agricultural workers. Students whose parents are higher professionals, lower professional, employers and managers are not eligible for New ERA. It may be the case that there are students whose parents have some experience of higher level education in the remaining social-economic groups, although the assumption is made that this is not the case in general. There is anecdotal evidence that farmers and self employed people circumvent the rules on grant eligibility (Department of Education and Science, 1993). Therefore farmers are excluded from the control group. It is not possible to identify self-employed people using the socioeconomic categories observed in the data. Linked School In order to be eligible for the access programme, students must have attended a “link school”. When the programme began in the late 1990’s, certain schools were chosen from the DEIS list, the government’s list of officially designated disadvantaged schools, and over time more schools from the government’s list were added to the scheme when the funding allowed it. However, there is a small group (10%) of schools who participate in HEAR but are not included on the DEIS list. These schools are located in rural areas of the country which have Objective 1 status under the EU structural funds. Schools are included on the official list of disadvantaged schools based on a range of socio-economic and educational indicators such as unemployment levels, housing, medical card holders and information on basic literacy and numeracy, and also some school level factors such as pupilteacher ratios. Initially New ERA choose schools which were in their own defined catchment area which were predominantly highly deprived urban areas. Given the data available for the analysis only covers 1999 to 2004, schools linked to New ERA in 1999 or before represent an “always” covered group, i.e. students from these schools, who have satisfied the other eligible criteria, are always in the treatment group in the analysis. In Appendix A, a set of maps are presented which show the distribution of link schools across the country and how the HEAR scheme expanded over time. Those schools who were included in the programme for the first time in 2005, or after, represent a “never” linked group, i.e. in the data no eligible students from these school have received treatment. Table 1 shows that that number of schools linked to the New ERA programme has increased substantially over time, however some schools closed or were amalgamated during the period included in the analysis. In 1999 New ERA was linked to 21 schools with a further 6 schools joining in 2000. 2001 saw the introduction of the HEAR scheme which resulted in 125 new schools becoming part of the programme. There were further increases in subsequent years with the largest expansion occurring in 2003 when 63 new schools joined. By 2007 a total of 310 schools had attached to the HEAR programme5. Essentially this study compares New ERA students with students who would have been eligible to join the New ERA programme had their school been participating in the HEAR scheme at the time they started university. The reliability of the results in Chapter 4 rests on several assumptions. One is that there was no change in the characteristics, e.g. student-teacher ratios, quality of teaching, facilities, etc, of the schools across the years. Given that the period under analysis only covers six years, it is plausible that the majority of schools will not have significantly changed. Furthermore, it is also assumed that the schools which joined later were not any different from the schools that joined earlier as this would result in our treatment and control group being different from one another. If the more disadvantaged schools joined the programme first, then the results may be biased as 5 This is not the current number of active schools as some of the schools which had joined either closed or amalgamated.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

the programme group (schools that were part of New ERA first) and the control group (schools that joined New ERA later) may systemically differ. However, the introduction of HEAR in 2001 should have reduced this problem as UCD then became linked to schools previously linked with other higher education institutions. As the above assumptions may be strong, the econometric models estimated in Chapter 4 control as much as possible for any observed differences in the control and treatment groups. For example, in all of the results presented in Chapter 4, the Leaving Certificate points of students are taken into account by the econometric models as the average level of points is different for treatment and control students and because Leaving Certificate points have a direct effect on student achievement. Inevitably there are unobservable and perhaps intangible factors that researchers cannot measure (e.g. school “spirit”, dedication of staff, etc.). This would cause problems for the reliability of any estimated effects of New ERA if these unobservable characteristics varied between the schools which joined New ERA/HEAR earlier and those that joined later. The results in Chapter 4 and Chapter 5 do not allow for potential unobservable differences. To examine if this is a problem, the analysis was repeated using the “difference-in-differences” technique which is commonly used in econometrics to eliminate unobservable differences between groups. However, broadly speaking the results are not different when using the differences-in-differences technique, thus the results presented are reliable indicators of the true effects of New ERA/HEAR. Table 1 below shows how New ERA evolved in terms of the number of schools joining over time i.e. becoming linked to UCD. Table 3.1 Number of schools joining the New ERA scheme.

Number of link schools

1999 and before

2000

2001 (HEAR introduced)

125

2002

2003

2004

2005 or later

3.3

Description of Data

The study involves an analysis of multiple secondary datasets including the UCD administration database; UCD exam database; New ERA HEAR database; HEAR school survey and Census data; and Department of Education school level administrative data. Data were obtained from the administrative records of UCD and the New ERA HEAR database. The data was anonymised by the Irish Social Science Data Archive before it was provided to us for analysis. The administrative data contains information on all undergraduates entering UCD from 1999 to 2004 inclusive, totalling about 30,000 observations. The UCD administration data was used to identify a suitable control group based on differing characteristics of the students including prior academic achievement, parental socio-economic status, grant-holder information and school level information. The New ERA HEAR dataset was used to identify the Merit and Direct New ERA students and to fill-in information that was missing from the main administrative database. Data from the UCD exams database was used to identify the key outcome variables, namely, retention rates, first year exam performance and final graduation rates.

1: 3: Introduction Evaluation Methods and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

The HEAR school survey (Denny et al. 2009), conducted in 2008 to collect information on schools participating in the HEAR scheme, was also used to examine progression rates from second level to third level. In addition, the administrative data was matched to school-level administrative data made available by the Irish Department of Education. Finally, census data, such as local unemployment rates, were matched to the school level using electoral districts. This allowed us to ascertain whether the roll-out of the New ERA programme was random i.e. whether schools in more/less disadvantaged areas joined the programme first. A number of students were omitted from the analysis. These include Irish school leavers with missing school information, international students, those from Northern Ireland and external candidates The working sample therefore excludes those who have no school-level data and students who entered the university directly rather than through the university central clearing system for school leavers (e.g. disabled students, certain mature students, transfers from vocational courses, etc). For much of the analysis, those who switched courses or repeated a year have had their later observation dropped. Unfortunately, it is not possible to identify students who have transferred from other universities. A description of the data used and the New ERA sample is below.

3.4 Data Limitations The dataset used in this study is larger and more wide-ranging in the information contained than any previously evaluation of university access programmes. However, there are limits on the information in the dataset and these place restrictions on the precision of the results and the type of questions that can be examined. Eligibility for the programme is inferred from a binary variable indicating grant holding students. The New ERA financial eligibility thresholds shadowed the eligibility for the full grant. However, the analysis cannot distinguish if students are receiving the full grant or partial grants (75%, 50% or 25% of the full amount). Should there be future studies examining New ERA or the local authority maintenance grant, it is recommended that UCD collect data annually on the amount of grant received by students rather than just grant holdership. If household income data were available in the administrative data, it would negate the need to infer New ERA eligibility. It would also allow future researchers to examine heterogeneity in the effects of New ERA and other policy intervention. For example, due to the lack of data, this study cannot examine if New ERA affects less welloff students differently than better-off students. It is not clear from the data whether some of those who are listed as failing the Summer exams may have in fact withdrawn by that stage and vice-versa. It is recommended that UCD collect data in such a way that policy makers can distinguish between students who formally withdraw, withdraw informally by not presenting for exams and students who fail exams. Furthermore, with regards to those who do withdraw, it is not known whether those who drop out are transferring to other institutions or leaving education altogether. It would be helpful if students who formally withdraw were asked why they were leaving and what their destination was. Finally, it is noted that high quality secondary school level data is not available in Ireland. Data on schoolsâ&#x20AC;&#x2122; exam results are not made available to researchers. Furthermore, data on transfer rates to third level institutions for each school are not published regularly. While it is not suggested that such information is made publicly available, if academic researchers had access to such data (while maintaining strict confidentially of the schools), more informed policy advice could be given.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

3.5 Description of Sample Figure 3.1 shows, of the total number of New ERA students in the sample, the percentage admitted each year6. In total, there are 322 students in the sample, and 8.07% of these were admitted in the first year. The numbers of New ERA students represent less than 5% of the total admissions to the university each year. As the years progressed and the programme grew, a higher number of students were admitted to the New ERA programme each year. There was a noticeably large increase in 2002 and again in 2004. Note that the large increase in 2002 was driven by the introduction of the HEAR programme in 2001. It could be speculated that the administrative structures for gathering data on HEAR students were not fully in place until 2002. The figure also provides the breakdown for Direct and Merit students. These sub-groups generally follow the same pattern as the overall sample. There were a relatively low percentage of students admitted in 1999, with large increases in 2002 and 2004. However there are three exceptions: less New ERA Direct students were admitted in 2001 than in 2000, less New ERA Merit students were admitted in 2000 than in 1999, and less Merit students were admitted in 2003 than in 2002. Nonetheless, the overall trend is the same; the increase was particularly large for the Merit group in 2002, when the figures increased from 9.4% to 24.2%, and for Direct students in 2004, when the percentage increased from 22.5% to 35.3%.

$!!

%&'()*+,-+%).+/01+23&4)532+64'733)4

#"! #"!

#!! #!!

"! "!

! !

#888 #888

"#$%&'( Direct )%$#'( Merit

$!!!

$!!! $!!#

$!!# $!!$

$!!$ $!!9

$!!9 $!!:

$!!:

%&'()*+,-+%).+/01+23&4)532+64'733)4

$!!

;,36<=+6<<+>*2

1999

2000

2001

2002

2003

2004

"#$%&'( 11 )%$#'( 15

19 12

17 14

26 36

39 30

61 42

Total All Years 173 149

Figure 3.1: Description of the New ERA sample

6 Descriptive statistics in Tabular form are included in Appendix B.

1: 3: Introduction Evaluation Methods and Data

"./012 <7=> *"%') +,#-+"./012 "./012 87:9 6789 8:79; 8<7=> <7=> 57<> <=7;5 <87:9 87:9

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Socio-Demographics Figure 3.2 provides information on the gender breakdown for New ERA students and for the general UCD student body. It shows that overall there are a lower percentage of male students attending UCD: 46% of the general student body is male. This trend is even stronger amongst the New ERA students; with a higher percentage of female students in the New ERA programme overall and a higher proportion among the Merit students specifically: 40% of New ERA Merit students are male, while only 36% of Direct students are male.

&!! &!!

%! &!!

#! $!

"! #!

! "!

-.)(*, ! Male *34" -.)(*, ?"@34" Female *34" ?"@34"

-.)(*,

/().,

01+20(34567 01+20(34567

Direct *34" 36.42 /()., ?"@34" 63.58

Merit

Non-New ERA

40.27 01+20(34567

45.98

59.73

54.02

Figure 3.2: Socio-demographics: Gender

'()*(+,

'()*(+,'()*(+,

$! %!

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

Figure 3.3 gives information on the distribution of New ERA students and the general non-New ERA student body by socio-economic group. While the most strongly represented group amongst the general student body are Higher Professionals, there are no New ERA students in this group, as expected based on the programmeâ&#x20AC;&#x2122;s inclusion criteria. New ERA students are most strongly represented among the lower socio-economic groups with the Salaried Employees group encompassing the majority of New ERA students (25.3% of New ERA Direct students and 19.7% of New ERA Merit students), while this group accounts for 16.2% of the general student body.

Non-New ERA

Farmers & Agricultural Workers Professionals & Managers Salaried Employees Intermediate & Other Non-Manual Skilled & Semi-skilled Manual Workers Unskilled Manual Workers

Merit

Direct

Direct %

Merit %

Non-New ERA %

1.15

1.41

11.80

62.65

Salaried Employees

25.29

19.72

16.24

Intermediate and Other Non-Manual

22.98

23.94

2.91

Skilled & Semi-skilled Non-Manual

36.78

33.80

5.88

Non-skilled Manual

13.79

21.13

0.52

Farmers & Agricultural Workers Professionals and Managers

Figure 3.3. Socio-demographics: socio-economic status

1: 3: Introduction Evaluation Methods and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

University Faculty Figure 3.4 summarises the percentage of New ERA Direct and Merit students per faculty compared to the general study body7. For each of the three groups, the highest proportions of students are enrolled in the Arts faculty, with 41% of all university students and 44% of New ERA Merit students being located in Arts, yet a lower proportion of New ERA Direct students are in the Arts faculty (34%). For the Science faculty there are large differences across all the three groups. While a high proportion (27.5%) of New ERA Merit students are studying science, and only 12.9% of the general student body and just 6.4% of New ERA Direct students enrolled in the Science faculty. Therefore, almost four times as many Merit students are studying Science compared to Direct students. There are also differences in the percentage of students enrolled in the Commerce faculty, with a significant proportion of New ERA Direct students (20.8%) studying Commerce, compared to 8% of Merit students and 12.7% of the rest of the study body. Finally, a higher proportion of Direct students are in the Health Sciences compared to the Merit students and the general student population. The proportion of students in the remaining faculties is broadly similar across groups. Overall, New ERA Direct students more closely resemble the general student in terms of the distribution across faculty. This may be explained by the relatively high number of Merit students studying lower points courses, such as Arts and Science. As these are relatively low points courses, many New ERA students do not need to avail of the points concessions for these courses. The distribution of Direct students across faculty may also resemble the total distribution across the university as the number of places reserved for Direct students depends on the number of students in the different faculties. 7 Re-structuring in UCD has replaced the old system of Faculties with a smaller number of Colleges.

Non-New ERA

Agriculture Arts Commerce Engineering&Architecture Interfaculty Law Medicine Science Veterinary Medicine Human Sciences

Merit

Direct

Direct %

Non-New ERA %

Agriculture

0.58

4.70

5.29

Arts

34.10

43.26

40.72

Commerce

20.81

8.05

12.74

Engineering & Architecture

6.36

6.04

9.46

Interfaculty

5.20

1.34

4.41

Law

2.89

2.01

2.99

Medicine

13.87

2.01

5.85

Science

6.36

27.52

12.95

Veterinary Medicine

1.16

1.34

1.81

Human Sciences

8.67

3.36

3.78

Figure 3.4: University Faculty

Merit %

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

First Year Summer Exams Figure 3.5 provides information on first year exam outcomes for New ERA students and the remaining student body. A higher percentage of New ERA students, both Direct and Merit, were registered for their first year summer exams compared to the general student body. Approximately 97% of New ERA students sit their summer exams !"#$%& '$#"& ()*+($,-./0 compared to 94% of remaining students. Merit students outperform the general study body in terms ofPercent 12&-1344$#-.5246-7 89:;; While 89:<= 8<:>8 ?266-1344$#-7 @Astudents 9<:;pass their 9A:>@ first year summer exams on the first attempt. the first year pass rate, only half of Direct This suggests that these two groups may be different in an important way and that the analysis should distinguish between them. In what follows we see that this is indeed the case. It is important to bear in mind in looking at these descriptive statistics that one is simply comparing averages between groups. One cannot conclude that there is any causal relationship.

97.32

70.65

10 0 Direct

Merit

Non-New ERA

B@BA <J@D< ?<@JB ?D@NN ?A@DO

Figure 3.5: First Year Summer Exams

1"#(2%&3 4(#"%&3 567&5(89-:;&3

()*$%&'""$+123./0 4/*$%&'""5$6'""

First Year Exam Grades !"#$%&'()#&*+,,(#&-.),&/#)0($ 1"#(2%&3 4(#"%&3 567&5(89-:;&3 In regards first year exam results, figure 3.6 shows that a higher proportion of Merit students achieve first class <$%&=>)$$ ?@AB <C@DE B@BA honours (10.3%) than Direct students (only and the general student body (8.9%). While the proportions ?70&=>)$$&FGHH(#I <C@<? <D@< 3%) <J@D< <E@?A ?<@JB achieving ?70&=>)$$&FK68(#I third class honours are roughly?B@?B the same across all three groups. As noted in above, the percentage of D#0&=>)$$L&M)$$ ??@J? ?<@DB ?D@NN 1"#(2%&3 Direct students failing the summerOCexams?J@A is far higher !)">& ?A@DO than either Merit or Non-New ERA students with 50% not 4(#"%&3 567&5(89-:;&3 passing the summer examinations. This is further evidence that the Direct and Merit groups are different.

60 7'8&$

40 6./9.)#

5(89-:;&3 B@BA <J@D< ?<@JB ?D@NN ?A@DO

<C@DE <D@< ?B@?B ?<@DB ?J@A

()*$%&'""$+,--./0

?@AB <C@<? <E@?A ??@J? OC

12&-1344$#-.5246-7 ?266-1344$#-7

93.69 73.1

97.11

!"#$%&'""

70 60

<$%&=>)$$ ?70&=>)$$&FGHH(#I ?70&=>)$$&FK68(#I D#0&=>)$$L&M)$$ !)">&

Percentage of students who sat and 30 passed their summer exams

100

6./9.)#

0 !"#$%&'""

()*$%&'""$+,--./0

1st Class Direct% 2.98 1"#(2%&3 4(#"%&3 Merit% 10.34 567&5(89-:;&3 Non-New ERA 8.89

()*$%&'""$+123./0

4/*$%&'""5$6'""

7'8&$

2nd Class Upper 2nd class Lower 3rd Class/Pass

Fail

10.12

14.29

22.62

13.1

28.28

21.38

26.9

16.31

21.68

23.77

29.35

Figure 3.6: First Year Summer Exam Grades

,-&./),$ 1: 3:'$#"& Introduction Evaluation Methods %& ()*+($,-./0 6789: ;:8<9 ;<86: <;8?7 7?87; <68@9

and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

First Year Autumn Exams For students who either do not sit or who fail the summer exams, they had (prior to modularisation) the option of repeating the exams in the autumn. Figure 3.7 shows that while a slightly higher proportion of Direct students sit the autumn exams, the pass rate is broadly similar across all three groups, with 69% of Direct students and the non-New ERA students passing the repeat exams and 57% of New ERA Merit students passing. While the data for the summer exams shows that Merit (but not Direct) students were broadly similar to nonNew ERA students (as one might expect given that they also enter via the CAO route) this pattern is reversed here with Direct students and non-New ERA having very similar pass rates. Of course, students sitting autumn exams are not representative of the student body in general since only a minority need to repeat.

400

400 32 20

12 12

0 0 5"#(6%

89-:8(;&.<*

7(#"% 5"#(6%

7(#"%

Percentage of students who sat and passed their autumn exams

89-:8(;&.<*

Direct

Merit

Non-New ERA

12&-03&34*-5 Sat Autumn % 12&-03&34*-5 95.24 =2>>-03&34*-5 Pass Autumn % =2>>-03&34*-5 68.75

84.62

86.94

57.58

69.32

Figure 3.7: First Year Autumn Exams

Progression to Second Year Figure 3.8 shows the proportion progressing to second year. The proportion of students is similar across all three groups (83-88%), despite the high failure rate of the Direct students on their first exam attempt. This suggests that any problems that the New ERA students (particularly Direct students) encountered in the summer exams have been largely ameliorated when the autumn exams are taken into account.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

3: Evaluation Methods 1: Introduction and Data

433 ;3 :3 93 83

:+<96

:8<+4

::<57

='"#0*

>#"'*

?$(@?#A)BCD

!#"0#(*

53 +3 43 3

Figure 3.8: Progression to Second Year

Final Year Graduation

!"#$%#&'()*"#&+, Figure 3.9 shows the percentage of students that graduated from university and the final degree classification -'"+.&* /+"'&* 0()10+2*345 for New ERA and non-New ERA students. It should be borne in mind that we have much less data on students in !"#$%#&+$*6 7789: ;;8;< 778= their final (degree) year. This means that it is much more difficult to discern the impact of New ERA on this group. Nonetheless, it can be seen below that the Merit students have the highest success rate with 88.9% of students from this group graduating. This is much higher than the general student body of whom about 78% graduate. The percentages of students graduating from the Direct group, by contrast, is much the same as the general student body.

100 90 80 70 Percent

60 50 40 30

88.89 77.42

77.5

20 10 0 Direct

Merit

Non-New ERA

Figure 3.9: Final Year Graduation Rate

1: 3: Introduction Evaluation Methods and Data

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Final Degree Classification The New ERA Merit students also perform best in terms of the degree classification awarded with 20.5% of !"#$%&'($)&*(+)((&,%$--"."/$0"1# students from this group graduate with first class honours, compared to 4.3% of New ERA Direct students and 13.8% of the general student body. In addition, a higher proportion of New ERA Direct students receive lower *")(/0&2 3()"0&2 41#54(6&789&2 second class :-0&,%$-and third class honours compared the other two groups. ;<=> to =?<@: :A<BC =#D&,%$--&EFGG()H :C<@B difference =><;> in the=><?B As with the first year exams, there is a striking achievement of Direct and Merit students with the =#D&,%$--&EI16()H ;@<B: A?<BB AC<: latter being much closer to non-New ERA students. Almost one third of the Direct students get a pass/3rd class A)D&,%$--&J&K$-A:<;A :><=A :><?@ honours, compared to less than 20% for the other two groups. At the other extreme, it is noticeable that a high proportion of Merit students, about 20%, get first class honours.

4(#"%&3 567&5(89-:;&3 <C@DE B@BA <D@< <J@D< ?B@?B ?<@JB ?<@DB ?D@NN ?J@A ?A@DO

()*+),-

! ".-/012..

#,3/012../4566)*7

#,3/012../489:)*7

%*3/012../;/(2..

1st Class

2nd Class Upper

2nd class Lower

3rd Class/Pass

1"#(2%&3 Direct%

4.29

18.57

45.71

31.43

4(#"%&3 Merit% 567&5(89-:;&3 Non-New ERA

20.51

29.49

30.77

19.23

13.78

29.07

38.1

19.05

Figure 3.10: Final Year Graduation Grade

The data illustrated in these graphs gives a general overview of the academic performance of the New ERA students and one can see that in some cases they are quite different from the general student population. The picture is complex with New ERA students sometimes outperforming other students and sometimes lagging behind. It is essential to realize that these statistics are descriptive only: they cannot tell us how effective New ERA is. For that, one needs a more sophisticated analysis and this is what is done in chapters 4 and 5.

)*$%&'""$+,--./0

()*$%&'""$+123./0

4/*$%&'""5$6'""

7'8&$

Chapter 4 Results

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction 4: Results

4 Results 4.1 Interpreting the Results This chapter presents the results of the econometric analysis examining the effects of the New ERA access programme on a range of student outcomes. The outcomes that are being studied here are categorical (e.g. whether a student passed their first year exams or what class of degree he/she received). So the study uses the appropriate econometric techniques to estimate the effect of different variables on the probability of different outcomes occurring. While the control group was selected to be as similar as possible to the New ERA programme group, there may still be variation between the two groups which needs to be taken into account in the analysis. Therefore the models also include a range of student characteristics which may obscure the effect of the New ERA programme if they were omitted. These include university faculty, year of university entry, and number of points attained in Leaving Certificate exams. Results are presented for the main outcomes of interest only. The models estimate the treatment effect, which is participation in the New ERA programme, relative to the outcomes of the control group. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Students from a farming background are excluded. The results reported on each figure are marginal effects and associated p-values. Marginal effects show the impact of being in the New ERA programme group, compared to the control group, on the probability of a achieving a particular outcome. The p-values represent the probability that the result obtained is due to chance rather than a true relationship between variables. Consistent with the literature, p-values below 0.10 (10%) are considered to be statistically significant in the present report. A p-value of less than 0.10 (10%), 0.5 (5%), 0.01 (1%) conveys that the probability that the difference between the two groups is due to chance is less than 10%, 5%, and 1% respectively. Note that a result may be statistically significant, but it may not be economically significant or vice-versa. For example, the impact of New ERA on retention rates may be statistically significant, but the size of the effect may be trivially small. Alternatively, a result may not be statistically significant, perhaps due to a small sample size, but it can be economically significant, in that it has a large meaningful impact on the outcome of interest. This is likely to be an issue when one sub-divides the sample to look at a finer, more detailed analysis as the numbers in the sub-groups gets smaller. Three sets of results are presented. The base case examines the impact of New ERA for all Direct and Merit students. The impact of the programme for New ERA Direct and Merit students are analysed and presented separately. To recap, Direct students are those who entered the university with a concession on entry scores (i.e. there is a lower points requirement) and Merit students are those who entered the university with the required CAO points. Both groups receive the same post-entry supports. Separate results are then presented for students who attained 400 points or less in their Leaving Cert Examinations, and for students who achieved more than 400 points, to determine if the New ERA programme has differing impact across these groups. Note the choice of 400 Leaving Certificate points as cut-off is somewhat arbitrary: small changes to this would not make any difference. Approximately 70% of UCD students enter with 400 or more points while just over 50% of New ERA students are above this threshold. It is worth remembering that as far as the New ERA programme is concerned, Direct and Merit students do not differ prior to entry nor would they appear different post-entry: they simply enter UCD through two different mechanisms. However what this means is that these two groups may differ in ways that are not observable. Indeed the descriptive statistics suggest that that they are quite different. What this implies for the effectiveness of the programme will be explored in this chapter.

1: Introduction 4: Results

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

4.2 Evaluation Results In this chapter a set of results are presented, which show the effect of a student being in New ERA on a series of academic outcomes. The results presented here cannot shed light on what the exact mechanism of any such effects might be. It could be through mentoring or through academic support. While it would be extremely useful to look at this, the data does not permit this detailed investigation8. The results in section 5.2 attempt to examine the effects of financial supports however the data does not permit a comprehensive analysis to be carried out.

4.2.1 First Year Official Withdrawal Rates

p<.05 p<.05 p<.1 p<.1

p<.05 p<.05

/=/61

70 70

p<.05 p<.05

60 60 /=/43

/=/43 /=/34

40 40

20 20 10 10 /0 /0

6=10

/=/2>

30 30 4=30

4=30

8.,(&'.9):;;#<" 8.,(&'.9):;;#<"

50 50

3=40

2=>0 / 0% /=/0 !99 ?@&'"*A3// ?@&'"*B3// !99 ?@&'"*A3// ?@&'"*B3// )))))))!99)))))))))))))))))))))))))))))))))?@&'"*A3//))))))))))))))))))?@&'"*B3//) )))))))!99)))))))))))))))))))))))))))))))))?@&'"*A3//))))))))))))))))))?@&'"*B3//)

!"#$%& !"#$%& '$#"& '$#"&

)!99)C#D):E!).'F)GH)I&(J)K)L@D)?@&'"* )!99)C#D):E!).'F)GH)I&(J)K)L@D)?@&'"* Figure 4.1 Impact of New ERA on Withdrawal Rates

The first outcome to be considered is whether students withdraw from UCD without attempting their fits year exams. This refers to whether students “officially” withdraw. In practice, it seems likely that a significant number of students effectively withdraw by not turning up for exams but they will still be registered with the university. In the data here they will be marked as “fails”. Figure 4.1 shows that overall, New ERA Direct students are 5.4% less likely to withdraw before attempting their exams than students from the control group9. While the effect is not as strong for Merit students, it is still positive and significant; they are 3.2% less likely to withdraw before attempting their summer exams than students from the control group. Recall that the p value in the figure is an indicator of the precision associated with the estimate. Where none is indicated (for example for two of the results in Figure 4.1) this means that the result is not statistically significant at least at the 10% level. The results for high (>400 points) and low point (≤400 points) students show that the New ERA programme has a positive effect on low point Merit students and high point Direct students. Low point Merit students are 8.1% less likely to withdraw before their first year exams, and high point Direct students are 4.5% less likely to withdraw before their first year exams. The impact of the programme on the withdrawal rates for low point Direct students and high point Merit students are not statistically significant.

8 Results are illustrated graphically throughout. Details of the results in this chapter with an estimate of their precision are included in Appendix C. 9 Note that here and elsewhere in the report effects are measured in percentage points.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction 4: Results

4.2.2 First Year Exam Performance While it is interesting to know the impact of students withdrawing it is, arguably, much more important to know how the students perform in their exams since only a minority actually withdraw. The next graph, Figu re 4.2 shows the effect of the New ERA programme on the grade that students get in their first year exams. There are six possibilities, the first four refer to how students do in the summer from getting a First Class Honours (1.1), an Upper Second (2.1), a Lower Second (2.2) and either a 3rd class honours or pass grade. The latter two are combined partly because of the small numbers in the groups and also because not all faculties seem to distinguish between the two. The other two categories are passing in the Autumn repeats (â&#x20AC;&#x153;Pass-Autâ&#x20AC;?) and Fail. Note that Fail here means that either a student failed in the summer and did not repeat in the Autumn or they did repeat but did not pass. Students who repeat in the Autumn may only get a pass grade: there is no distinction between first, second class honours etc. Since a student has to fall in to one of these categories (conditional on sitting the exam), the height of the bars sum to zero. That is if New ERA increases the probability of one outcome it has to be at the expense of another and the increases cancel out the decreases. The results are very clear and striking: the probability of the three least desirable outcomes is reduced and the probability of the higher results (First and Second Class honours) is higher. In other words, what New ERA does is to shift students up the grade distribution. The failure rate is about four percentage points lower with a slightly bigger effect for the Direct students. This is unambiguous evidence that the programme has positive effects on the academic achievements of New ERA students compared. There is some evidence that the effects differ for Direct and Merit students but the differences are very small. One has to be careful in interpreting this graph: it does not mean that if a New ERA student repeats in the Autumn that they are less likely to pass. This is because one is considering all the possible outcomes together. So one can say that a student is less likely to end up passing in the Autumn because they are less likely to have failed in the summer exam.

12%

8% !!!!!!!!!!!!!!!!!! p<.1 p<.05

p<.1 p<.05

p <.1

p<.1 p<.05

Marginal Effect

-0%

Direct Merit

-4%

-8%

!"!############$"!###############$"$########%&''()*+######%&''#,-."#####/&01 Grade Awarded

12%

Figure 4.2 First Year Exam Performance: Base

-0%

Marginal Effect

1: Introduction 4: Results

12%

Evaluating the Impact of the UCD New ERA Widening Participation Initiative 8% !!!!!!!!!!!!!!!!!! p<.1 p<.05

p<.1 p<.05

p <.1

p<.1 p<.05

Marginal Effect

4.2.3 First Year Exam Performance: low points students

-0% section it was seen that the differences between Merit and Direct students, in terms of the impact In the previous of New ERA, were small and most likely negligible. Since the results control for the studentsâ&#x20AC;&#x2122; Leaving Certificate Direct students points this may not be too surprising. In this section and the next, the distinction is made between -4% entering with 400 points or less and those who entered with more than 400 points. Clearly, Merit the points that students attain in their Leaving Certificate is associated strongly with what degree/programme they enter. The vast majority -8% of students in the low point category are in Sciences or Arts. The analysis here however controls for !"!############$"!###############$"$########%&''()*+######%&''#,-."#####/&01 this so the results here do not simply reflect that the two different groups are doing very different degrees. Grade Awarded 12% 8% Marginal Effect

4% -0% -4% -8%

Direct Merit 1.1

2.1

2.2

Pass/3rd

Pass Aut.

Fail

Grade Awarded Figure 4.3 First Year Exam Performance: Low Point Students Figure 4.3 represents the effect of programme participation on first year grades for low point (â&#x2030;¤ 400 points) students. One can see at a glance at the height of the bars in the graph that the effects are very small. More importantly perhaps, none of them are statistically significant. That is one cannot reject the statistical hypothesis that the effects that are shown are just due to chance. So these students, once they are in UCD, are neither helped nor harmed by being part of the New ERA programme.

4.2.4 First Year Exam Performance: high points students Figure 4.4 illustrates the impact of the New ERA programme on the probability of achieving a particular grade for High point (>400 points) students. New ERA increases the probability of achieving a higher grade for both Direct and Merit students. High point Merit students are 7.3% more likely to attain an upper second class honours grade than students from the control group. While the level of statistical significance falls for Direct students, programme participation still has a positive and significant effect: New ERA Direct students are 10.7% more likely to attain an upper second class honours grade relative to the control group.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

12%

p<.1 p<.05

p<.05 p<.05

p<.05

p<.1 p<.05

1: Introduction 4: Results

p<.05 p<.05

8% Marginal Effect

-0%

Direct Merit

-4%

-8%

!!"#"!!!!!!!!!!!!!!!$#"!!!!!!!!!!!!!$#$!!!!!!!!%&''()*+!!!!%&''!,-.#!!!!!!/&01! Grade Awarded Figure 4.4 First Year Exam Performance: High Point Students

High point students, both Direct and Merit, are also less likely to fail their first year exams. Direct students are 3.9% less likely to fail their first year exams, and Merit students are 3.1% less likely to fail their first year exams. The impact is equally significant for both groups. These results are very different from those presented in section 4.2.3 for low point students. Taken together, it shows clearly that the academic benefits to the programme arise from a benefit to students who are academically strong to begin with. This may have implications for the development of the programme which will be discussed later. Why there is such a sharp difference between high and low point students is an interesting question and one for which this report has no simple answer. One way of thinking about this is that a studentâ&#x20AC;&#x2122;s academic performance depends on several inputs including their own academic ability, their study effort and any additional support they receive such as through a programme like New ERA. It seems plausible that these inputs are complementary: that is the benefit of each is higher the more one has of the other. Simply put, better students are better able to take advantage of the extra opportunity afforded by the New ERA programme. This might be because such students have other qualities, such as motivation or self-confidence. In the absence of more data one can only conjecture.

4.2.5 Probability of Graduating The results so far have considered only the students performance in their first year exams. It is also important to consider what happens later during a studentâ&#x20AC;&#x2122;s time at the university. Several outcomes relating to graduation will be analysed. It is important to note that there is less information available to do this since many of the students have not had the opportunity to graduate within the time frame of the data. So the dataset pertaining to final year outcomes has about 40% fewer observations than for their first year outcomes considered so far. This constrains what one can do in terms of analysing sub-groups. In particular there is data on very few students on low points, partly because many have not remained in university to this stage. So it is not practical to distinguish between high and low point groups10. However students are more evenly balanced between Merit and Direct so that distinction will be considered. 10 The estimation techniques that have been mostly used in the report, based on the method of Maximum Likelihood, require large samples to be reliable.

Points>400

1: Introduction 4: Results

0.054

0.074

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

#!" '&" '%"

p<.05

p<.01

'$" ()*+,-)./011234

'#" '!" &" %"

'$56%" 7568"

$" #" !" 9,*234

(2*,4

Figure 4.5 Impact of New ERA on Probability of Graduating Figure 4.5 represents the impact of the New ERA programme on the probability of graduating from the degree !"#$%#&'()*+(*&',info@education.gov.ie course. The figure shows that, overall, programme participation has a positive impact on graduation rates. Merit students are 9.7% more likely to graduate Direct Merit from their degree programme relative to students from the control group. Base The effect is bigger (albeit less precisely determined) for Direct students: they are 14.8% more like to graduate than students from the control group. This is evidence that New ERA has benefits beyond their first year Case !"!#$% !"!#&& results and can have a major effect on the lives of those who participate in the programme.

Points!400 Points>400

!"!'%% (!"!*$#

(!"!)#) !"!&#)

4.2.6 Probability of Graduating on Time An outcome which has been looked in the international literature is whether students graduate on time, that is whether they need to repeat one or more years. In this case, as Figure 4.6 shows, there is no evidence that New ERA has any significant effect. The marginal differences between New ERA students and the control group is very small, around one percentage point, and not statistically significant.

30./

201/

1.67%

1.44%

.01/

./ 9'"-8&

4-"'&

Figure 4.6 Impact of New ERA on Probability of Graduating on Time

4#")'(#5*677-8&

20./

!"#$%&'()*((&+(,-%.,

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

base

Direct -0.007

1: Introduction 4: Results

Merit 0.041

/0/ 10/ -0.024 0.061 101 0.006 -0.043 4.2.7 Final Degree Classification 2$,,3&4*5 The last outcome considered in 0.025 this chapter is -0.060 the final degree classification. Note that in this analysis only concerns students who pass their final degree: whether students pass or not has been considerd already in section 4.2.5. As can be seen in Figure 4.7 there is not much evidence that students degree class is affected although Merit students may be about 6.1% more likely to get Upper Seconds (2.2’s) compared to lower outcomes.

15% 10%

p<.1

Marginal Effect

5% 0% -5%

!"#$%& '$#"&

-10% -15%

1.1

2.1

2.2

Pass/ 3rd

Degree Class Awarded Figure 4.7 Impact of New ERA on Final Degree Classification: Base

4.3 Main Findings in This Chapter Overall the results indicate the New ERA programme has a number of significant positive effects on the students participating in the programme. • Participation in New ERA has a positive effect on reducing first year withdrawal rates. • New ERA reduces the probability of withdrawing prior to first year exams for low point Merit students and high point Direct students. • New ERA has a positive effect on improving first year exam results by shifting students up the grade distribution. • Participation in New ERA increases the probability of achieving a First and Second Class honours and reduces the probability of failing or receiving a Third Class honours/Pass in the first year exams. • These improvements in exam performance only benefit high point students only, i.e. those who enter UCD with more than 400 Leaving Certificate points. • The programme has similar effects for both Direct and Merit students in terms of first year exam results. • Participating in New ERA increases the probability of graduating from university. This is striking as one might have expected that the barriers associated with low SES might have dissipated by students’ final year. • The result applies to both Direct and Merit students. • The programme has no effect on whether students graduate on time. • The programme has relatively little effect on the final degree classification the students receive. However, Merit Treatment students have an increased chance of attaining a higher grade.

1: Introduction

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Chapter 5 Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5: Further 1: Introduction Analyses

5 Further Analyses Thus far, the research outlined in this report has focused on the impact of the New ERA programme on students who have entered UCD. The results show that students who participate in the programme, will on average, experience a number of significant academic benefits. This chapter extends the analysis by examining several additional outcomes by which one may judge the effectiveness of the programme. This chapter also examines the potential effects of changes to the New ERA programme. There are two main strands to New ERA’s activities, pre-entry and post-entry supports with the former (including admissions) designed to increase the numbers of students progressing to third level from linked schools, and the latter designed to improve students outcomes once they arrive in university. In addition, the post-entry supports may also increase progression even though the students are already “on site” since secondary students who are contemplating applying to university may anticipate the post-entry supports. For example, the post entry financial support may induce more students to apply to the programme. In addition, the value of the financial support may have an impact on student outcomes if it reduces the need for student employment whilst at university and provides additional financial resources to cover basic materials such as books, etc. While it is very difficult to isolate which specific activity within the programme is more likely to affect which outcome, the purpose of this chapter is to examine how student outcomes are influenced by varying the level of New ERA supports. Specifically, this chapter evaluates the impact of the New ERA programme on progression rates by examining whether a school becoming linked to the programme increases the numbers who subsequently attend university in general and UCD specifically. Next, it uses the variation in the level of financial support provided to New ERA participants over time to examine the impact of aid on students’ performances. The level of financial support tended to vary between 1999 and 2004 due to funding availability and student numbers. Finally, much of this report is based on the current HEAR system in which, in most cases, only students from disadvantaged schools can apply to the New ERA programme. However, the introduction of the National HEAR scheme in 2010 will change the eligibility criteria for joining the programme and this may have consequences for the effectiveness of the programme identified in this report. The final part of the chapter will examine this issue in further detail.

5.1

The Impact of New ERA/HEAR on Progression to University

The focus of this analysis is to determine whether New ERA is effective at increasing progression rates to higher level education, However, the data used in the analysis in Chapter 4 cannot be used for this purpose since this only contains information on students who have progressed i.e. there is no comparison group. In an ideal world, in order to examine the effect of New ERA/HEAR on the destinations of students from secondary schools, one would like information about the exact destination of school leavers, on an annual basis and broken down by each school. However these data are not collected on a systematic basis in Ireland.

5.1.1 The Impact of New ERA/HEAR on Changes in Progression to University To address this lack of data, the research team conducted a postal survey of all schools linked to the HEAR scheme (Denny, Doyle, O’ Reilly & O’ Sullivan, 2008). Crucially for the study, data on the proportion of the schools’ final year students who progressed to university were gathered. Information such as aggregate Leaving Certificate results, subject choice and student demographics was also collected. The survey questions were asked in relation to the Leaving Certificate class of 2007 and also in relation to the class of 2001 so that changes in the characteristics of the schools and the outcomes of their students could be identified. A total of 158 schools responded to the survey, giving a response rate of 51 percent. Of the schools linked to UCD, 24 (45% of their total number) responded, 34 (69% of their total number) of the schools linked to NUI Maynooth and 100 (48% of their total number) of the schools linked to the other universities participating in the HEAR scheme11. 11 The low response rate could perhaps be explained by a timing clash with the Whole School Evaluations conducted by the Department of Education and with exams as well as a general sense of “survey fatigue” amongst school principals rather than concerns relating to data protection issues.

1: Introduction 5: Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Figure 5.1 illustrates the proportion of the Leaving Certificate class in HEAR-participating schools that progressed to university in the years 2001 and 2007. The data on progression is banded: that is respondents (school principals/ career guidance counsellors) were asked to give responses on a scale with five possible bands. The graph shows that in 2001 almost 20% of schools reported that less than five percent of their students progressed to university, this increased slightly in 2007 with 22% of schools reporting that less than five percent of their students progressed to university. However, this is offset by the fact that from 2001 to 2007 the number of schools that experienced progression rates of 31% or more increased by almost ten percent.

%! $!

-./0.1(23.')4' 506))75'/.8)/(913 8/)3/.559)1'/2(.5

-./0.1(23.')4' 506))75'/.8)/(913 !""# !""$ ! 8/)3/.559)1'/2(.5

!&'()'*&

+&'()'"!&

0% to 5%"!

!""# 2001 !""$ 2007

19.9 22

""&'()'#!&'

#"&'()'$!&

$"&',

6% to 10%

11% to 20%

21% to 30%

31% +

17 14

19 18.4

20.2 11.8

23.9 33.7

Figure 5.1: Percentage of Leaving Certificate class progressing to university in 2001 and 2007

!&'()'*&

+&'()'"!&

""&'()'#!&'

Figure 5.2 illustrates the overall change in progression rates experienced by schools from 2001 to 2007. The graph shows that the majority of schools experienced no change in the number of students progressing to university (52.28%). Approximately 30% of schools experienced an increase in progression rates, while 17% of schools found that fewer students progressed to university.

!"#$"%&'("

17578 2/526 09522

Increase

No Change

Decrease

Figure 5.2: Percentage increase or decrease in progression to university However Figures 5.1 and 5.2 are only descriptive in nature and do not infer causality of the effects of joining New ERA/HEAR (Tables D1 and D2 present these summary statistics in tabular form in Appendix D).

#"&'()'

($) "&',1

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5: Further 1: Introduction Analyses

The method used for these estimates is somewhat more complicated than those used elsewhere in the report. While the outcome is ordered (increase, no change, decrease), we allow for the possibility that the length of time a school is linked to HEAR may not be exogenous. In other words, there may be some systematic difference between schools that were in the scheme at the beginning from those who joined later. One reason for this concern is that not all schools responded to the survey (about 120 out of 305). Hence this study uses what is known as “a simultaneous equations approach” where there an outcome of interest (here, change in progression) as a function of the treatment variable (how long linked to HEAR) and some covariates with the treatment variable itself being dependent on some independent variables. The latter are the local unemployment and education rates12. Figure 5.3 represents the estimated marginal effect of being linked to the HEAR scheme for an additional year on the change in the number of students progressing to university. The estimates and the corresponding standard errors and sample sizes are shown in Table D3 in Appendix D. The graph shows that being linked to HEAR for a longer period of time has a positive effect on student outcomes. It shows that schools that are linked to HEAR for an additional year are 12% less likely to have fewer students progressing to university. The effect is even stronger on the probability of schools sending more students to university; the effect of being linked to HEAR for an additional year increases the probability that more students will progress to university by 14%. Finally, the graph shows that the effect of being linked to HEAR for an additional year does not significantly affect the likelihood 56'(2-7 that a school will experience no change in progression rates.

"# p<.01

p<.05

$% $# $34$

!$"4$

&'()*+',-.//012

!$# !$% !"#

More going to University

No Change

Less going to University

Figure 5.3: The effect of the duration that a school has been linked to HEAR o n the progression of students to university

12 This approach is similar to “instrument variables regression”, the key difference being that one equation here is non-linear (the ordered )))))))))))))))))))))) probit). Hence the Conditional Mixed Process estimator due to Roodman (2007) is used.

1: Introduction 5: Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5.1.2 Impact of New ERA/HEAR on Changes in Progression to UCD

%2&.2+)

While the previous analysis examined the proportion of students who progressed to any university as a result of being linked to the HEAR scheme, it is also possible to model the effect of HEAR/New ERA on the proportion of students from a particular schoolâ&#x20AC;&#x2122;s Leaving Certificate class who attend UCD specifically rather than university in general. Using the UCD administrative data discussed in Chapter 3, the proportion of students from a particular school progressing to UCD was calculated for each year. Figure 5.4 shows the proportion of New ERA/HEAR schools sending at least one student to UCD in a given year. Unsurprisingly, the proportion of New ERA link schools sending at least one student is always higher compared to that of schools which only participate in HEAR and are linked to other universities. On would expect that because of location: New ERA schools are, on average, closer to UCD than other link schools13. For both types of school, the proportion of students who go on to attend UCD fluctuates from year to year and there is no clear pattern.

!""#$%&''"(#) *+,#-.!#"/01#'0"2#) 3-!.#'0"2#)

;:: C: B: A: @: ?: >: =: <: ;: : ;CCC

<:::

<::;

<::<

<::=

<::>

1999

2000

2001

2002

2003

2004

All Schools %

64.26

60.24

57.03

69.48

62.25

64.26

New ERA link only %

68.75

77.08

60.42

81.25

66.67

70.83

HEAR only %

63.18

56.22

66.67

61.19

62.69

Figure 5.4: Average proportion of schools that send at least one student to UCD

13 See Map A1 and A2 in Appendix A for the geographical distribution of linked schools.

4555 9>>> 9>>4 9>>9 9>>= 9>>7

!""#$%&''"(#)*+,#-.!#"/ 67896 6:8;< 6>897 ;;8>: <;8>= 6>879 6587: :489< 6989< 6686; 67896 ;>8:=

;::; 678< 9788 ;755 ;::6 6744 87>9 ;7>; ;::8 ;754 Participation Initiative Evaluating the Impact6794 of the UCD <7:; New ERA Widening ?'@A" 676; 97;: ;7=>

5: Further 1: Introduction Analyses

!"#$%&'%'&()'*$'+$,-$./011$0((2*3)*#$4-5

Figure 5.5 shows the average proportion of each schools’ class who go on to UCD in a given year broken down by whether the school is a New ERA link school or participating in the HEAR scheme only. Many of these schools may send a very low fraction of their Leaving Certificate class, if any, to UCD in a given year. Figures 5.4 and 5.5 are shown in tabular form in Table D4 and D5 in Appendix D.

10 9 8 <2&.2*(

7 6 4 3 2 !""#$%&''"(#) *+,#-.!#"/01#'0"2#) 3-!.#'0"2#)

1 0

6777

8999

8996

8998

899:

899;

1999

2000

2001

2002

2003

2004

All Schools %

3.34

3.36

3.13

3.46

3.11

3.51

New ERA link only %

5.04

5.48

4.46

5.44

4.75

6.02

HEAR only %

2.94

2.86

2.82

2.99

2.72

2.91

Figure 5.5 Average proportion of Leaving Certificate class attending UCD

To estimate the effect of being linked to New ERA/HEAR on the probability of students progressing to UCD a statistical technique called “Tobit” is employed. This is a standard method in econometrics for modelling a continuous variable where many of the values are clustered at a lower bound14. In this context the continuous variable is the proportion of students from a particular school attending UCD in a given year which equals zero when nobody from that school attended UCD in that year.

14 Technically this is called “censoring”. For example many people have zero hours of work and one cannot have less than zero.

1: Introduction 5: Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Figure 5.6a shows the estimated marginal effect of a school joining New ERA/HEAR on the probability of sending at least one student to UCD in a given year. Figure 5.6b shows illustrates the effect on the proportion of students the school sends to UCD given that it sends at least one student. There are no significant effects on either outcome for the HEAR-only schools. However there are significant effects for New ERA link schools. The results show that being linked to the New ERA programme increases the probability that a school will send at least one student to UCD by approximately 13%. The marginal effect on the proportion of students that it sends (given that it sends at least one student) is slightly above 1%, i.e. being linked to New ERA increases the proportion of the Leaving Cert class progressing to UCD by about one percentage point. Given that the average proportion of students sent by linked schools to UCD is about 5%, this one percentage point increase is a significant gain. The effects and the corresponding standard errors are displayed in Table D6 in Appendix D.

16% 16%

p<.01 p<.01

p<.05

p<.05 20/21 p<.05 p<.05

20/21

!"#$%&"'()**+,!"#$%&"'()**+,!"#$%&"'()**+,!"#$%&"'()**+,-

14% 14% 16% 16% p<.01 p<.01 12% 12% 14% 14% 10% 10% 12% 12% ./01 8% 8% 10% 10% 6% 6% ./01 8% 8% 4% 4% 6% ./01 6% 2% 2% 4% 4% ./01 0% 0% 2% 4''(5,677'8 2% 4''(5,677'8

20/21 20/21

3/01 3/01 9+:();4('%&<(7&'= >)4;('%&<(7&'= 9+:();4('%&<(7&'= >)4;('%&<(7&'= 3/01 0% Marginal Effect of of being linked onon probability of of school sending at at least one student to to UCD 0% Marginal Effect being linked probability school sending least one student UCD 3/01 4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'= 4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'= 2/A1 2/A1 Figure 5.6(a): Effect of being linked on the probability of sending at least one student to UCD

MarginalEffect Effect ofbeing beinglinked linkedon onprobability probability schoolsending sendingatatleast leastone onestudent studenttotoUCD UCD Marginal of p<.01 p<.01 p<.01 p<.01 ofofschool

2/@1 2/@1 2/A1 2/A1

p<.01 p<.01

0.5% 0.5% 0.5%

p<.01 p<.01

1.3%

1.3% 1.3%

!"#$%&"'()**+,!"#$%&"'()**+,!"#$%&"'()**+,!"#$%&"'()**+,-

2/?1 2/?1 2/@1 2/@1 2/31 2/31 2/?1 2/?1 3/B1 3/B1 2/31 2/31 3/A1 3/A1 3/B1 3/B1 3/@1 3/@1 3/A1 3/A1 3/?1 3/?1 3/@1 3/@1 3131 3/?1 3/?1

1.3%

0.0% 0.0% 0.5% 4''(5,677'8 9+:();4('%&<(7&'= 4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'= >)4;('%&<(7&'= 0.0% 31 31 !"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI !"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI 0.0% 4''(5,677'8 9+:();4('%&<(7&'= 4''(5,677'8 9+:();4('%&<(7&'= >)4;('%&<(7&'= >)4;('%&<(7&'= !"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI !"#$%&"'()**+,-(7*(C+%&$('%&<+D(7&(E#7E7#-%7&(7*(8-FD+&-8("--+&D%&$(GHI

Figure 5.6(b): Effect of being linked on the proportion of students attending UCD

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5: Further 1: Introduction Analyses

5.2 The Impact of New ERA’s Financial Aid on Student outcomes In general this study is concerned with the impact of New ERA taking the program as a whole. That is, the study does not investigate the different aspects of the program. Ideally, one would do this but the data available to the Financial Support Available to New ERA Students !"#$#%"$&'(") research team does not permit such an approach. Nevertheless, in this section an attempt is made to see can one !"#$%&'$()**+,-. /+012$')-3+,4-5$6,178+-12$()**+,-. say anything about the impact of one specific aspect of the program namely the financial aid package provided ;<=:<? ;?=9@> to 9::: students. ;<=<>: <AAA ;B=<C9 ;<=:A: ;C=9@9 annually is a key component of New ERA’s post-entry supports. The financial aid package provided to students ;B=>9< ;<=:>D ;C=BCA The<AA9 amount of this package has changed over time due to funding availability and the number of New ERA <AA< admitted ;<=?9C ;<=:@< ;?=>D: students to the programme. Figure 5.7 shows the total financial aid package in real amounts (expressed <AAB ;B=A:9 ;B=B9C ;C=>A@ in 2008 prices to adjust for inflation) for the period 1999-2004. As nearly all New ERA students are also in receipt ;B=B<B of <AA> the Higher ;<=<B> Education grant, the table;?=??@ also shows changes in the value of that grant over time (in 2008 prices) 'E",1F" <@:> BACC ?DCA The sum of New ERA’s financial package and the government grant varies from year to year with the total value of the package being particularly high in 2000, 2001 and 2003. The average value of the entire financial package received by a New ERA student in the three years 2000, 2001 and 2003 was €6313 (expressed in 2008 prices). The average over the years 1999, 2002, and 2004 was €5407.

7000 5750

Financial Support Available to New ERA Students

4500 3250

!"#$%&'$()**+,-. /+012$')-3+,4-5$6,178+-12$()**+,-.

2000

1999

2000

2001

2002

2003

2004

1999

2000

2001

2002

2003

2004

New ERA Supports

€2,249

€3,261

€3,412

€2,516

€3,091

€2,234

Local Authority Grant

€2,925

€2,909

€2,948

€2,972

€3,316

€3,323

Total Supports

€5,174

€6,171

€6,360

€5,489

€6,407

€5,557

Figure 5.7: Financial Aid for New ERA students In order to determine the effectiveness of this financial aid package, the analysis examines whether student performance in these high value years are different from student performance in the other years. The analysis rests on the assumption that there were no other differences in New ERA’s activities in these high value years that may influence outcomes. It also assumes that the characteristics of students in the high value years did not differ from students in low value years. In order to estimate the effect of receiving a higher value financial package, i.e. for students that entered in 2000, 2001, or 2003, compared to those receiving the lower package i.e. for students that entered in 1999, 2002, or 2004, an ordered probit model as used in Chapter 4, is estimated for New ERA students entering first year between 1999 and 2004. Although the estimated results presented in Figure 5.8 do follow a pattern suggesting that the extra funding was beneficial, the first year outcomes for students who received the high value package were not statistically different from the students who received the lower value package. Furthermore no significant effects of the extra funding were detected when alternative models were estimated15. The estimated results and their standard errors are shown in Table D7 in Appendix 7. This does not mean that New ERA’s financial package has no effect on student performance; however it does imply that increasing the value of the package from an average of €5407 to €6313 (a difference of €906) did not lead to changes in student achievements. 15 The results of these alternative models are available on request from the research team.

1: Introduction 5: Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

There are limits as to how much one can extrapolate from this result. Firstly it should be noted that the sample size used for this analysis is quite low as only New ERA students can be included; therefore the estimated results may not be precise. Secondly, based on the data available, it is not possible to predict with any degree of confidence whether an increase of more than around €900 would have had any effect. Nor is it possible to estimate if a reduction in the value of the financial package below an amount of, say €5400, would have any effect on average!"#$"%& student performances. However in reducing the amount of financial aid to students, policy makers should consider the effects of such a reduction on student employment whilst studying full-time. Students may enter part-time employment to offset a reduction in financial aid. There is currently no consensus in the academic literature on the causal effects of student employment on academic outcomes. However one recent study by Stinebrickner and Stinebrickner (2003) was able to exploit a natural experiment in one college and found that employment was harmful for low-income students’ academic outcomes. Thus the effect of any reduction in the financial package of New ERA students would need to be closely monitored. "#$ %$ &$ '$

&6'$ )*+,-.*/0122345

867$ (67$

!#67$

!($ !'$

!76#$

!76"$

9*::0>?56

@*-/

!&$ !%$ !"#$ "6"

(6"

(6(

9*::;<+=

Grade Awarded

Figure 5.8: Impact of variation in financial aid on first year exam performance

5.3 Likely Consequences of the National HEAR scheme. The HEAR scheme will undergo major changes for those applying for entry to higher education institutes in 2010. It is important for policy makers to be informed of the likely consequences of these changes. Until now, under the HEAR scheme, only students from DESI disadvantaged schools could apply to the university access programmes. However, the introduction of the National HEAR scheme in 2010 will change the eligibility criteria for participating in the programme. Under the new system, students who meet the income, socio-economic and educational eligibility criteria can apply to university through the HEAR scheme even if they are not attending a DEIS disadvantaged school. Essentially the HEAR scheme now includes all secondary schools in the country. This new system was introduced as it was recognised that there may be disadvantaged students attending non-disadvantaged schools who could benefit from the access admissions programme for HEAR. Extending the scheme to all schools is not the only change: the assessment process now uses six indicators. Using the UCD administrative data, a further analysis was conducted to attempt to examine the effects of the Extended HEAR scheme to students from non-disadvantaged schools. The previous analysis compared New ERA students to students who would otherwise be eligible for the programme but whose school has not yet begun participating in the HEAR scheme by the time they entered university. The control group is comprised of the latter group of students. In this new analysis, New ERA students are compared to an alternative control group who come from similar socio-economic backgrounds but who attend schools that are not currently participating in the HEAR scheme i.e. the schools which are typically not disadvantaged.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5: Further 1: Introduction Analyses

As in Chapter 4 an ordered probit model is estimated to examine the effects of being a New ERA student relative to being in the alternative control group. The results presented in Figure 5.9 show that in general the New ERA students are more likely to have positive outcomes compared to the disadvantaged students from nondisadvantaged schools who do not currently (pre-2010) participate in the HEAR scheme. One interpretation of this result is that participation in the New ERA programme has a greater effect on university achievement for disadvantaged students than the benefits that accrue from attending a non-disadvantaged school. This would imply that students who meet the minimum income, socio-economic and education eligibility criteria who are attending non-disadvantaged schools should benefit from the New ERA programme. Overall this suggests that the new national HEAR scheme may be advantageous. However in considering this result one should note that the comparison between the New ERA students and the disadvantaged students in non-disadvantaged schools can be potentially biased by various factors as students from very different educational backgrounds are being compared. For example, one may argue that the effect of New ERA on university outcomes is being overstated as the existing New ERA students may have scored higher Leaving Certificate points had they attended a non-disadvantaged school. Thus comparing New ERA students to students from non-disadvantaged schools who attained the same Leaving Certificate points might be misleading.

p<.05

p<.01

p<.05

p<.01

87 %7 .7

.&)7 )&(7

597 5)7

5)&@7 5@&87

5.&67

:+/#";+4$<==>?3

%&@7

5.7 5%7 587 567

(&($$$$$$$$$$$$$$$$$)&($$$$$$$$$$$$$)&)$$$$$$$$$$$*+,,-./0$$$$$*+,,$123&$$$$$$$$!+"4 Grade Awarded Figure 5.9: Effect of New ERA relative to being in a non-link school on first year exam performance

1: Introduction 5: Further Analyses

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

5.4 Main Findings in This Chapter •

If a school becomes part of the HEAR scheme then there is a higher probability that it will send more students to university. o Using the period between 2001 and 2007 it was found that becoming linked in that interval increased the probability of sending a higher proportion to university by 14%.

•

Being linked to the New ERA programme increases the probability that a school will send at least one student to UCD by approximately 13%. The marginal effect on the proportion of students that it sends (given that it sends at least one student) is slightly above 1%, i.e. being linked to New ERA increases the proportion of the Leaving Cert class progressing to UCD by about one percentage point.

 Changes in the financial aid package to students, taking into account the Higher Education grant, do not have a measurable effect on student outcomes. Since one only observes variation in this between years (& not between students) these estimates are less precise.  Comparing New ERA students with students who are also socially disadvantaged but who do not qualify for the programme (as they attend a non-disadvantaged school), the results show that the New ERA students in UCD out-perform academically their comparators. This suggests there would be benefits to allowing students to participate in New ERA even if there school is not linked to HEAR. This is what the new national HEAR scheme does.

1: Introduction 6: Recommendations & Findings

Evaluating Evaluatingthe theImpact Impactof ofthe theUCD UCDNew NewERA ERAWidening WideningParticipation ParticipationInitiative Initiative

6 Recommendations & Findings The results of this report have implications both for UCD specifically and for higher education access programmes generally. A set of general recommendations which have relevance to all higher education institutions have been identified along with further specific recommendations for the UCD New ERA programme.

UCD recommendations The following recommendations are based on the results of this evaluation and suggest areas where the New ERA programme might be refined. While some of the recommendations are based on the assumption that the financial resources available to the access programme are constant, others would require additional resources.

The study has found convincing evidence that the UCD New ERA programme has substantial benefits for students from lower socio-economic groups. That is, there are more students from disadvantaged backgrounds going to university and those that do, at least in the case of UCD, perform better on several fronts. It follows that any contraction in the programme, whether for budgetary or other reasons, is likely to remove these benefits.

Given the positive academic benefits to New ERA students throughout their degree, there is an argument for expanding the capacity of the access programme. While the programme can reserve additional places for Direct students, it cannot directly affect the number of Merit students entering the programme.

It is recommended that the number of places reserved for Direct students be increased, especially in courses requiring approximately 400 points or more as students attaining such points generally derive greater benefits from the programme. This is subject to not increasing the current level of points concession available to these students.

ii.

Steps should be taken to actively increase the number of Merit students from lower socioeconomic groups. This could be achieved by providing additional pre-entry support to all potential students. Increasing the number of Merit students is also likely to be achieved by the introduction of the national HEAR scheme.

The previous recommendations are based on further resources becoming available. If this is not possible we suggest there could be a reallocation of resources from financial aid to fund these additional places. This is subject to maintaining the total financial aid package that students receive (including the Higher Education grant) to be no less than about €5,400 per annum. We recommend this as fluctuations in financial aid do not appear to have affected student performance. It would be useful to monitor whether changes in financial aid lead to changes in part-time work by the students.

As an alternative to the above recommendation, some of this financial aid could be re-allocated to New ERA’s other activities (such as academic and social support). This is under the assumption that the current funding of the programme is unchanged and again subject to the caveat above of not reducing the total package per student to be less than €5,400 per annum. In the current economic climate it is likely that there will be financial constraints for Higher Education Institutions and access programmes in the future so prioritizing of the most effective supports is essential.

The research shows that 50% of Direct students succeed at their summer exams on first sitting compared to 71% 5. of students from the general population. It is therefore recommended that additional supports be put in place to help prevent this from occurring. This issue is especially important in the modular system where repeating in the autumn is no longer possible. Preventing students failing during the year will reduce the probability of students having to carry over modules into second year. For example, the university may wish to consider some form of “Early Warning System” which flags the presence of students who are at a high of risk of failing.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

6: Recommendations 1: Introduction & Findings

NaTIONAL recommendations The national roll–out of HEAR is consistent with the evidence that New ERA students perform better than other 6. disadvantaged students who do not participate in the access programme as their school is not linked. In other words, for these students, coming from an “advantaged” school does not compensate for not entering the programme. It therefore follows that allowing such students to participate in the access programme, through the new HEAR scheme, should generate benefits for them. Evidence in this report suggests schools which are linked to the access programme (and receiving pre-entry supports) have increased progression rates to higher level institutions. Therefore an argument can be made to provide pre-entry supports to schools which are currently excluded from outreach activities.

Data collection is paramount to facilitating quality research. It is recommended that all higher education 8. institutions collect information from all students, at the point of registration, on the amount of financial aid received, parental educational attainment, family composition and levels of household income (using banded answers). This will facilitate and inform future evaluations of access programmes. Furthermore, access programmes should conduct a survey of all link schools and HEAR participating schools every two years to build up longitudinal data on the effects of the access programmes’ pre-entry activities.

Further Research

10.

This study has evaluated the overall impact of New ERA on student performance, however it was not possible to determine which specific components of the programme are most effective. Further research is needed to determine whether, for example, is it mentoring, financial aid, academic support or a combination of these that improves student outcomes? To consider this one needs variation in the level of support that students receive so that some students get different combinations of supports from others. This should be best evaluated as part of a randomized control trial. While recognizing the practical and ethical difficulties inherent in such an experiment, a well designed trial could provide valuable information on the design of access programme.

The New ERA evaluation is the first quantitative research of an access programme to be undertaken in 11. Ireland. Quantitative evaluations of other access programmes would contribute significantly to the current understanding of the effectiveness of access initiatives. This would help inform policymakers in relation to the national widening participation agenda. HEIs differ in their approach to access both in terms of their history and approach. Measuring their impacts, aside from its direct relevance to the particular institution, could be very informative about which type of access programmes are most effective. Research undertaken on a national level on the impact of the different support measures would inform national policy for the longer term and would help to identify the best practice in support provision for students from lower socio-economic groups. It would also be beneficial to undertake research into the particular issues associated with widening participation in programmes of study that are traditionally highly prestigious and financially rewarding, for example Medicine, Veterinary Medicine , Law and Dentistry. As demand for these courses is very competitive with high points requirements, there have traditionally been very few low SES students in these courses. A case study of Medicine may prove valuable as the Irish universities have just moved to a system that reduces reliance on Leaving Certificate points by also taking into account the scores on the Health Professional Admissions Test (HPAT). Whether this will change access to medicine for low SES groups is far from clear and needs to be investigated in a few years when there is sufficient information and the system has had time to mature.

12.

1: Introduction References

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

References Angrist, J. and Krueger, A. B. (1999) “Empirical Strategies in Labour Economics”. In: Ashenfelter, O. and Card, D., eds. Handbook of Labor Economics, Vol. 3A: 1339-1344. Amsterdam: North Holland. Angrist, J., Lang, D. and Oreopoulos, P. (2009) “Incentives and Services for College Achievement: Evidence from a Randomized Trial”. American Economic Journal: Applied Economics, 1(1): 136-163. Bergin, D., Cooks, H., and Bergin, C. (2007) “Effects of a College Access Program for Youth Underrepresented in Higher Education: A Randomised Experiment”. Research in Higher Education, 48 (6): 727-750. Bettinger, E. (2004) “How Financial Aid Affects Persistence.” NBER Working Paper, No. 10242. Brock, T. and Richburg-Hayes, L. (2006) “Paying for Persistence: Early Results of a Louisiana Scholarship Program for Low-Income Parents Attending Community College.” MDRC. Burtless, G. (1995) “The Case for Randomized Field Trials in Economic and Policy Research.” Journal of Economic Perspectives, 9(2): 63-84. Card, D. and Krueger, A.B. (1994) “Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania.” American Economic Review, 84(4): 772-793. Carneiro, P., and Heckman, J. (2003) “Human Capital Policy.” NBER Working Paper, No. 9495. Carpenter, A. (2004) “Social Class, Inequality and Higher Education in Ireland. Leaving Early: International Perspectives on Working Class Student Withdrawal.” Staffordshire University, Stoke-on-Trent, UK. Chevalier, A., Denny, K. and McMahon, D. (2009) “Intergenerational Mobility and Education Equality.” In: Dolton, P., Asplund, R. and Barth E., eds. Education and Inequality across Europe. London: Edward Elgar. Chevalier, A., Finn, C., Harmon, C., and Viitanen, T. (2006) “The Economics of Early Childhood Care and Education.” Technical Research Paper for the National Economic and Social Forum (NESF). Clancy, P. (2001) College Entry in Focus: A Fourth National Survey of Access to Higher Education, Dublin: Higher Education Authority. Clancy, P. (1982) Participation in Higher Education: A National Survey, Dublin: Higher Education Authority. Denny, K. Doyle, O. O’Reilly, P. and O’Sullivan V. (2009) “A Profile of HEAR Schools 2008.” UCD Geary Institute. Department of Education and Science (2001) Report of the Action Group on Access to Third Level Education, Dublin: Stationery Office. Department of Education and Science (1995) Charting our Education Future: White Paper on Education, Dublin: Stationery Office. Department of Education and Science (1993) Report of the Advisory Committee on Third-Level Student Support (Committee chaired by Dr. Donal de Buitléir), Dublin: Stationery Office. Department of Education and Science (1992) Education for a Changing World: Green Paper on Education, Dublin: Stationery Office. Dynarski, S. (2008) “Building the Stock of College-Educated Labor.” The Journal of Human Resources, 43 (3): 576610. Dynarski, S. (2003) “Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and Completion.” American Economic Review, 93(1): 279-288. Equality Review Team to the Higher Education Authority (2004) Report of the High Level Group on University Equality Policies, Dublin: Higher Education Authority. Harmon, C., Oosterbeek, H., and Walker, I. (2003) “The Returns to Education: Microeconomics.” The Journal of Economic Surveys, 17 (2): 115-155.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction References

Lavy, V. and Schlosser, A. (2005) “Targeted Remedial Education for Underperforming Teenagers: Costs and Benefits.” Journal of Labor Economics, 23(4): 839-874. Lesik, S. (2007) “Do Developmental Mathematics Programs have a Causal Impact on Student retention? An Application of Discrete-Time Survival and Regression Discontinuity Analysis.” Research in Higher Education, 48 (5): 583-608. Lynch, K. and O’Riordan. C. (1996) “Social Class, Inequality and Higher Education: Barriers to Equality of Access and Participation among School Leavers”, University College Dublin, Registrar’s Office. Machin, S. and Vignoles, A. (2005) “What’s the Good of Education? The Economics of Education in the UK.” New Jersey: Princeton University Press. McPherson, M., Schapiro, M. (1991) “Does Student Aid Affect College Enrollment? Nw Evidence on a Persistent Controversy.” American Economic Review, 81(1): 309-318. Oakley A., Strange, V., Toroyan, T., Wiggins, M., Roberts, I. and Stephenson, J. (2003) “Using random Allocation to Evaluate Social Interventions: Three Recent U.K. Examples”, Annals, AAPSS, 589 (1): 170-189. Osborne, R. and Leith, H., (2000) Evaluation of the Targeted Initiatives on Widening Access for Young People from Socio-Economically Disadvantaged Backgrounds, Dublin: Higher Education Authority. Preparing for Life Programme, UCD Geary Institute. Available from: < http://geary.ucd.ie/preparingforlife/home> [Accessed 9 September 2009] Roodman, D. (2007) cmp: Stata module to implement conditional (recursive) mixed process estimator. http:// ideas.repec.org/c/boc/bocode/s456882.html Skilbeck, M., Connell, H. (2000) Access and Equity in Higher Education: an International Perspective on Issues and Strategies, Dublin: Higher Education Authority. Stinebrickner, R. and Stinebrickner T. R. (2003) “Working during School and Academic Performance.” Journal of Labor Economics, 21 (2): 449-472.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Appendices A: Maps B: Tables of descriptive statistics for Chapter 3 C: Tables of results for Chapter 4 D: Tables of descriptive statistics and results for Chapter 5

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Appendix A: Map A1:

Distribution of Link Schools by Higher Education Instituion as at 2008

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Map A2:

Distribution of Dublin Link Schools by Higher Education Institution at at 2008

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Map A3:

Year School Linked to the Higher Education Access Route as at 2008

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Map A4:

Year Dublin Schools linked to the Higher Education Access Routes as at 2008

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Appendix B: Tables of Descriptive Statistics for Chapter 3 Table B1: Description of New ERA sample Year of entry

Total

Direct

Merit

1999

8.07

6.36

10.1

2000

9.63

8.05

2001

9.63

9.83

9.4

2002

19.3

24.2

2003

21.4

22.5

20.1

2004 Total

35.6 322

35.3 173

28.2 149

Table B2: Descriptive Statistics: Socio-demographics

Direct

Merit

Rest

Male

Average Points

382

429

455

Grant

100

16.94

Socio-economic group of father

Farmers/Ag Workers

1.2

1.4

11.8

Higher/Lower Professionals

40.7

Managers and Employers

21.9

Salaried Employees

25.3

19.7

16.2

Intermediate Workers

12.6

8.5

1.7

Other non-manual

10.3

15.5

1.2

Skilled manual

24.1

16.9

5.0

Semi-skilled manual Non-skilled manual

12.6 13.8

16.9 21.1

0.9 0.5

173

149

16377

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Table B3: Descriptive Statistics: University Faculty

New ERA Direct

New ERA Merit

General Study Body %

Agriculture

0.58

4.7

5.29

Arts

34.1

43.62

40.72

Commerce

20.81

8.05

12.74

Engineering and Architecture

6.36

6.04

9.46

Interfaculty

5.2

1.34

4.41

Law

2.89

2.01

2.99

Medicine/Nursing etc.

13.87

2.01

5.85

Science

6.36

27.52

12.95

Veterinary Medicine

1.16

1.34

1.81

Human Sciences N

8.67 173

3.36 149

3.78 16337

Table B4: Descriptive Statistics: First Year Exams

New ERA Direct

New ERA Merit

General Study Body

First Year Summer Exams:

Registered for Summer Exams

97.1

97.3

93.7

Pass Summer

73.1

70.7

Summer Exam Grades First Class

10.3

8.9

10.1

13.1

16.3

Lower Second Class

14.3

28.3

21.7

Third Class

22.6

21.4

23.8

Sat Autumn

95.2

84.6

86.9

Pass Autumn

68.8

57.6

69.3

Progress to Second Year N

82.7 173

86.2 149

88.4 16337

Upper Second Class

Autumn exams

Table B5: Descriptive Statistics: Final Year Graduation

New ERA Direct

New ERA Merit

General Study Body

77.42

88.89

77.5

11921

4.29

20.51

13.78

Upper Second Class Honours

18.57

29.49

29.07

Lower Second Class Honours

45.71

30.77

38.1

Third Class Honours N

31.43 70

19.23 78

19.05 8935

Graduated N Final Degree Classification First Class Honours

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Appendix C: Tables of Results for Chapter 4 Table C1: Impact of Access Programme on Retention Rates and Passing First Year Exams Base All

Points <400 Direct

Merit

Direct

Merit

All

0.062** (0.031) 322

0.054 (0.040) 258

0.081** (0.038) 197

0.012 0.045** (0.019) (0.023) 385 300

0.000 (0.022) 337

0.029 0.096 (0.030) (0.058)

0.158* (0.083)

0.070 (0.069)

0.040 0.111* (0.038) (0.060)

0.008 (0.043)

533

257

196

385

337

Not Dropping out before 0.034** 0.054** 0.032* (0.016) (0.025) (0.017) attempting examsa Sample size 707 558 534 Passed first year examsa 0.053** 0.112** (0.026) (0.044) Sample size 706 557

Points >400

All

321

Direct

300

Merit

Notes: a Estimated coefficient of linear probability models. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. Direct students are those who entered the university with reduced entry scores. Merit students are those who entered the university without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

Table C2: Impact of Access Programme on First Year Exam Performance In first sitting of exams

Base All

Direct

Merit

Points <400 All Direct

Merit

First Class Honours

0.014** (0.007)

0.008 (0.006)

0.015 (0.009)

0.001 N/A (0.008) (N/A)

0.001 0.045** (0.002) (0.021)

0.039 (0.031)

0.034 (0.022)

Second Class Honours Upper 0.050*** 0.047* (0.018) (0.027)

0.048** (0.022)

0.007 0.002 0.005 0.086*** 0.107* (0.009) (0.009) (0.011) (0.027) (0.063)

0.073** (0.031)

Second Class Honours Lower 0.055*** 0.051* (0.019) (0.028)

0.046** (0.019)

0.026 0.009 0.018 0.023** (0.032) (0.042) (0.037) (0.010)

0.024** (0.011)

0.024** (0.010)

Pass/Third Class

-0.013* (0.007)

-0.028* (0.015)

0.018 0.007 0.009 -0.059*** -0.067 (0.023) (0.034) (0.018) (0.020) (0.045)

-0.061** (0.029)

Pass in Autumn

-0.054*** -0.052 (0.019) (0.029)

-0.041** (0.019)

-0.015 -0.005 -0.010 -0.056*** -0.065* (0.018) (0.026) (0.021) (0.020) (0.036)

-0.039** (0.018)

Fail

-0.053*** -0.046* -0.039** -0.037 -0.013 -0.023 -0.040*** -0.039** -0.031** (0.019) (0.024) (0.017) (0.047) (0.060) (0.046) (0.015) (0.019) (0.014)

Sample size

680

-0.008 (0.008)

535

512

303

241

183

Points >400 All Direct

377

294

Merit

329

Notes: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. Direct students are those who entered the university with reduced entry scores. Merit students are those who entered the university without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Table C3: Impact of Access Programme on Probability of Graduating, Probability of Graduating on time and Final Degree Classification Probability of graduating1 Sample size Probability of graduating on time1 Sample size Final degree classification2 First Class Honours Second Class Honours Upper Second Class Honours Lower Pass/Third Class Sample size

All 0.100*** (0.032) 481 0.019 (0.032) 382

Base Direct 0.148** (0.064) 391 0.017 (0.057) 305

Merit 0.097*** (0.035) 388 0.014 (0.036) 310

0.027 (0.019) 0.053 (0.032) -0.026 (0.018) -0.055 (0.034) 383

-0.007 (0.018) -0.024 (0.063) 0.006 (0.014) 0.025 (0.068) 305

0.041 (0.032) 0.061* (0.037) -0.043 (0.033) -0.060 (0.036) 313

Notes: 1 Estimated coefficient of linear probability model. 2 Estimated marginal effects in ordered probit model conditional on sitting final exams. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. Direct students are those who entered the university with reduced entry scores. Merit students are those who entered the university without reduced entry scores. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that subsequently linked to the system. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Appendix D: Tables of Descriptive Statistics and Results for Chapter 5 Table D1: Descriptive Statistics: Percentage of Leaving Certificate class progressing to University 2001 2007

0-5% 19.9 22

6-10% 17 14

11-20% 19 18.4

21-30% 20.2 11.8

31%+ 23.9 33.7

Note: Based on HEAR School Survey carried out by research team in 2008. Table D2: Descriptive Statistics: Percentage increase/decrease in Progression Percentage Change

Increase 30.39%

No Change 52.28%

Decrease 17.33%

Note: Based on HEAR School Survey carried out by research team in 2008. Table D3: Effect of Length of Participation in HEAR on Progression of Students to University More Progressing 0.141*** (0.47)

Effect of time linked to HEAR on progression to university1 Sample size

No Change -0.020 (0.019) 116

Fewer Progressing -0.121** (0.052)

Note: 1Estimated using limited information maximum likelihood: the outcome is modelled as an ordered probit with the treatment variable a linear function of exogenous variable. The Stata â&#x20AC;&#x153;CMP: conditional mixed processâ&#x20AC;? program is used. We control for school size, rural/urban location and university to which the school became linked. The local unemployment rate and the local education level are used as predictors for how long a school has been linked to HEAR. Marginal effects and standard errors (in parenthesis) reported. Significance levels: ***1%, **5%, *10%. Based on School Survey carried out by research team in 2008. Table D4: Descriptive Statistics: Average Proportion of Schools that send at least one Student to UCD 1999 2000 2001 2002 2003 2004 N

All % 64.26 60.24 57.03 69.48 62.25 64.26 249

New ERA link only % 68.75 77.08 60.42 81.25 66.67 70.83 48

HEAR only % 63.18 56.22 56.22 66.67 61.19 62.29 201

Note: Based on UCD administrative data 1999-2004. Table D5: Descriptive Statistics: Average Proportion of Leaving Certificate Class Attending UCD 1999 2000 2001 2002 2003 2004 N

All % 3.34 3.36 3.13 3.46 3.11 3.51 249

New ERA link only % 5.04 5.48 4.46 5.44 4.75 6.02 48

Note: Based on UCD administrative data 1999-2004.

HEAR only % 2.94 2.86 2.82 2.99 2.72 2.91 201

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix

Table D6: Effect of Being Linked on the Progression of Students to UCD All Schools 0.073*** (0.027) 0.005*** (0.002) 249

Probability of school sending at least one student to UCD2 Proportion of students from Leaving Cert class progressing to UCD Number of Schools

New ERA link only 0.131** (0.051) 0.013*** (0.005) 48

HEAR only 0.003 (0.034) 0.000 (0.002) 201

Note: 2Estimated marginal effect of tobit model, controlling for year and unemployment rates in school locality. Marginal effects and standard errors (in parenthesis) reported. Significance levels: ***1%, **5%, *10%. Based on UCD administrative data 1999-2004. Table D7: Impact of Variation in Financial Aid on First Year Exam Performance In first sitting of exams

New ERA students

First Class Honours

0.027 (0.019) 0.057 (0.040) 0.064 (0.049) -0.007 (0.009) -0.070 (0.051) -0.071 (0.050) 313

Second Class Honours Upper Second Class Honours Lower Pass/3rd Class in Summer Pass in Autumn Fail Sample size

Note: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is being in first year during a high value financial support year (2000, 2001, 2003). The base specification includes faculty and number of points attained in university entry exams. Table D8: Financial Aid to New ERA Students New ERA Financial Supports 1999 2000 2001 2002 2003 2004 Average over all Years

€2,249 €3,261 €3,412 €2,516 €3,091 €2,234 €2,794

Local Authority Grant Total Financial Supports €2,925 €2,909 €2,948 €2,972 €3,316 €3,323 €3,066

€5,174 €6,171 €6,360 €5,489 €6,407 €5,557 €5,860

Note: All amounts in 2008 €

1: Introduction Appendix

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

Table D9: Impact of the New ERA Programme Relative to being a Disadvantaged Student in a NonDisadvantaged School on First Year Exam Performance In first sitting of exams First Class Honours Second Class Honours Upper Second Class Honours Lower Pass/3rd Class in Summer Pass in Autumn Fail Sample size

New ERA students versus students from non-disadvantaged schools 0.021 (0.008) 0.054 (0.018) 0.032 (0.009) -0.024 (0.009) -0.046 (0.014) -0.038 (0.012) 1365

Notes: Estimated marginal effects in ordered probit model. Marginal effects and standard errors (in parenthesis) reported. Significance levels: *** 1%, ** 5%, * 10%. The treatment effect is participation in the Access programme. The control group include financially eligible students (i.e. grant holders), whose parents are not professionals or employers and who attended schools that are not disadvantaged and which will only be eligible to participate in HEAR in 2010. Those from farming background excluded. The base specification includes the following control variables: faculty, year of university entry and number of points attained in university entry exams.

Evaluating the Impact of the UCD New ERA Widening Participation Initiative

1: Introduction Appendix