Assessing the evidence: financial aid by XefNed

ASSESSING THE EVIDENCE: FINANCIAL AID Judith Scott-Clayton CCRC

I. Introduction

Of the various tools at policymakers’ disposal for increasing college access and success, financial aid policy is unquestionably the most researched. This brief review cannot serve as an exhaustive summary of this extensive literature. Instead, we aim to highlight influential articles and offer our own informed perspective regarding the key lessons and directions for future research and experimentation. For recent critical reviews of the literature, we recommend “Into College, Out of Poverty? Policies to Increase the Postsecondary Attainment of the Poor,” by Deming and Dynarski (2009), and “What Is Known About the Impact of Financial Aid? Implications for Policy,” by Bridget Terry Long (2008). For a catalog of research especially in areas where the rigorous evidence is limited, we suggest The Effectiveness of Student Aid Policies: What the Research Tells Us, a volume edited by Baum, McPherson, and Steele (2008). These three broader reviews are themselves summarized in the table accompanying this topic brief. Before discussing the evidence, it is worth explaining some of the challenges that the literature has faced in trying to establish the causal effects of financial aid. Why is it not good enough to simply compare students who receive aid with students who do not? The answer is that these students may be different for many reasons besides just their aid status. Even when studies control for observable differences between aid recipients and non-recipients, there may be reason to think that two groups differ along unobservable dimensions that may influence future outcomes. In the case of need-based aid, students who are receiving assistance tend to be more disadvantaged, so even if they benefit from financial aid, a naïve comparison might suggest negative effects. In the case of merit-based aid, students who are receiving assistance are obviously academically selected, so recipients often perform well compared to non-recipients, but this is not necessarily indicative of the scholarship’s impact. In the case of work-study, the bias is unclear: Students who are eligible for work-study are more disadvantaged in general, but those who actually receive and accept a work-study offer are likely to be positively selected from among those who are eligible. The challenge, then, is to identify some “plausibly exogenous” source of variation in who receives assistance. Following the standards of the Institute of Education Sciences (IES) and the What Works Clearinghouse, we consider findings from explicitly randomized field experiments as the highest standard of evidence, but such designs are relatively rare and in some cases simply impossible to carry out. Among quasi-experimental approaches, regression-discontinuity designs—comparing students just above and below program eligibility cutoffs—can be particularly compelling, though difference-in-difference designs (often based upon changes in

state, federal, or institutional policy) may provide more externally generalizable estimates. Because of the depth of literature on financial aid, we generally focus here on the studies that are at least quasi-experimental in nature; however, in some cases where there are no such studies we take a broader view of the available evidence.

II. Lessons from the Literature

1. Rigorous evidence suggests that money matters. The first lesson, grounded in over thirty years of research, is that money matters for college outcomes. The cost of college is an important factor in students’ enrollment decisions, and when students know that the cost is lower, their enrollment rates increase. Taken together, the quasi-experimental evidence suggests that an additional $1000 of financial aid (or $1000 less in college costs) may increase college enrollment by 4 percentage points (Deming & Dynarski, 2009, p. 11). Key studies showing positive effects include Dynarski’s (2003) study of the Social Security Student Benefit (SSSB) program, studies of the GI Bills (Stanley 2003; Bound & Turner, 2003), Kane’s (2007) study of the Washington, DC, Tuition Assistance Program, and several studies of state merit aid programs (Dynarski 2004; Cornwell, Mustard, & Sridhar 2006; Kane 2003).8 Until recently much of the focus in the financial aid literature was on college entry, rather than post-enrollment outcomes. Several recent studies suggest that financial aid can also improve persistence and completion (Dynarski 2008; Brock & Richburg-Hayes 2008; Scott-Clayton 2009); however, most of the programs studied involve academic achievement incentives and thus the effect of money alone is unclear. Two studies that do examine post-college effects of pure grants are Dynarski’s (2003) study of the SSSB program and Bettinger’s (2004) study of Pell Grants. Dynarski finds positive, but noisy and statistically insignificant effects on completed years of schooling. Bettinger similarly finds suggestive evidence of positive effects on persistence, but the estimates are not always robust to alternative specifications.

2. Rigorous evidence suggests that simplicity matters. An important and puzzling anomaly to the lesson above that “money matters” is the relatively weak evidence regarding the nation’s single largest grant program, federal Pell Grants. The broadest studies of the Pell Grant program—an early study by Hansen (1983) and a subsequent study by Kane (1996)—find no detectable effect of the introduction of Pell Grants on college enrollments for eligible (low-income) populations. One hypothesis for the lack of overall impacts is that the complexity of the Pell eligibility and application process obscures its benefits 8

Note that state merit-aid programs may improve high school achievement, which may increase college enrollment separate from the effects of the money per se. However, the studies mentioned generally look at college enrollments just after implementation of a program, so early cohorts of recipients may have had relatively little opportunity to change their high school behaviors.

and prevents the program from reaching the individuals who need it most—those who are on the fence about college for financial reasons.9 The programs that have demonstrated positive impacts tend to have simple, easy-tounderstand eligibility rules and application procedures. This includes the SSSB program, GI Bills, and state merit aid programs. In contrast, the application process for federal Pell Grants is burdensome, and students do not learn their eligibility until after they are accepted to college. As described in Dynarski and Scott-Clayton (2006), it is difficult for students to respond to a subsidy they do not know about, and even those that know about it might be discouraged by the application. Recent experimental evidence by Bettinger, Long, Oreopoulos, and Sanbonmatsu (2009) suggests that simplifying the Pell Grant application process could have significant effects on college access: providing assistance with completing and submitting the federal aid application increased immediate college entry rates by 7 percentage points for the treated group. The importance of program design and delivery is likely one reason why education tax credits appear to have had little impact (Long 2004). Like Pell Grants, the value of tax credits can be difficult for families to determine in advance of the college enrollment decision. Eligibility depends on family income and tax liability, as well as on year in school and degree intention, and accessing the aid requires filling out a tax form. Importantly, tax relief may not arrive until 16 months after educational expenses are incurred. It is thus perhaps not surprising that take-up rates for the Hope Tax Credit and Lifetime Learning Tax Credit are low. Long (2004) and Dynarski (2004) also find that the benefits of both tax credits and educational savings benefits accrue disproportionately to upper-income families.

3. Rigorous evidence suggests that achievement incentives matter. A third emerging lesson from the literature is that achievement incentives appear to increase effectiveness. This evidence seems especially relevant when the focus is on improving college success (as opposed to simply access). Scott-Clayton (2009) examines a broad-based merit scholarship program in West Virginia that provided free tuition and fees, for up to four years, to eligible students as long as they maintained a minimum GPA and course load in college. She finds that the scholarship increased GPAs and credits completed in the first three years of college, but in the last year of the scholarship—while students are still receiving the money but no longer facing the minimum requirements—the program’s effect disappears. This suggests that the achievement incentives were an important mechanism driving the impacts. Brock and Richburg-Hayes (2006) also find evidence that performance-based scholarships increase GPAs and persistence in community colleges, while Angrist, Lang, and Oreopoulos (2009) find weaker evidence at a large college in Canada.10 9

Note that Seftor and Turner (2002) find positive effects of Pell Grants for older “nontraditional” students and that the Bettinger (2004) study mentioned above finds weak suggestive evidence of positive effects on persistence, conditional on enrollment. Both findings are consistent with a story in which information and experience with bureaucracy is important: Older individuals may have learned about the Pell program over time, and continuing students may learn about the program once they enroll in school. Those who have recently graduated from high school but not yet enrolled may be the least informed and least equipped to figure out the process. 10 They find significant effects of a performance-based scholarship, but only for females who received additional services in addition to the financial incentive. There were no significant effects for the full sample. See accompanying table.

Other relevant evidence comes from Pallais (2009) who finds that a large merit-based scholarship program in Tennessee significantly improved high school achievement as measured by test scores (she finds that the increases in test scores are too large to be explained simply by increases in re-testing). Jackson (2010) finds that a program providing financial incentives to high school students for scoring well on AP exams not only improved AP exam scores, but increased college going rates and increased college performance even for those students who would have gone to college anyway.

4. Evidence is limited on the effectiveness of work-study. A fourth lesson is that while low levels of on-campus student employment (such as would be supported by the Federal Work-Study [FWS] program) may not significantly harm student outcomes, the rigorous evidence for positive effects of such assistance is very weak. The nonexperimental evidence on student employment, including non-work-study jobs, suggests negative effects (see Pascarella & Terenzini, 2005, pp. 414-415, and Hossler, Zisken, Kim, Cekic, & Gross, 2009, pp. 103-104, for recent reviews of this literature). But some studies such as Kalenkoski and Pabilonia (2010) and DeSimone (2008) suggest that the magnitude may be quite small, and some non-experimental studies suggest potentially positive effects of low-level, on-campus work. Importantly, the most credible causal examination of college student employment finds significant negative effects of on-campus work on academic outcomes. Stinebrickner and Stinebrickner (2003) analyze data on students at a small private college in Kentucky at which all students are required to work at a campus job for 10 hours per week, but some jobs offer students the possibility to work more (although the study does not separate out work-study employment, it is likely that many of these jobs are at least partially funded by FWS). Students at the college are randomly assigned by administrators to on-campus jobs, and those who are assigned to a job with additional hours available end up working more than those for whom this is not an option. The authors find that students who worked more because they were assigned to a highavailability job earned significantly lower GPAs, a decline of about 0.162 points per additional hour of work. However, because all students were working at least 10 hours per week, the study cannot address the question of whether lower hours of work might be less harmful.

5. Evidence on the effect of loans is limited, but emerging evidence suggests that design is important. A fifth lesson is that while loans are unpopular, they are a critical element in college financing, and their design might be significantly improved to minimize students’ repayment risks. There is very little rigorous research on the consequences of student loans. Dynarski (2005) finds suggestive evidence of positive effects of student loan expansions in the U.S. in the early 1990s on college attendance, but the estimates are not highly robust to specification checks. Findings from the non-experimental literature “can at best be described as mixed” (Heller 2009, p. 46), perhaps because studies are inconsistent in their choice of counterfactual: Are we comparing $1 of loans to $1 of grants, $1 of work-study, or to no aid at all? Based on the nonexperimental evidence, Heller (2009) concludes that college enrollments are not as sensitive to

loans as to grants. This is unsurprising given that loans are not worth as much to students, but since they also cost less, it is unclear whether loans may still be cost-effective compared to grants. If loans are less effective than grants, one important reason might be debt aversion: Some students simply dislike being in debt, even when that debt enables an investment with high average returns. A recent experiment by Erica Field (2009) finds strong evidence that students (in this case, law school admits) are debt averse. Admitted students at one school were randomly assigned to receive either (1) a public service scholarship which would convert to a loan if students did not pursue public service after graduation, or (2) a loan which would be forgiven if students decided to pursue public service after graduation. The two treatments were financially equivalent, yet framing the program as a “loan which can be forgiven if you pursue public service” was much less effective in inducing students to public service than a “grant which will convert to a loan if you do not pursue public service.” Like the FAFSA simplification study, Field’s findings provide further evidence that the details of program design and marketing can be critical.

III. Open Questions

1. Should aid amounts vary by class year? Are there creative ways to target limited financial aid dollars to maximize the impact on outcomes that matter beyond enrollment? For example, should freshmen and seniors be paid the same or different amounts? Note that some countries (or some universities in some countries) have recently enacted policies in which students are charged more if they are still enrolled beyond 150% of the typical program length. There is some evidence from Italy that such a program improved time-to-degree at one university (Garibaldi, Giavazzi, Ichino, & Rettore, 2007). Note that Pell eligibility currently extends to 18 full-time equivalent terms, or more than 200% of the scheduled length of the typical BA program.

2. Should aid awards by prorated according to students’ course loads? Should awards be prorated by course load? Note that currently Pell Grant policy is asymmetrically generous in that it allows students enrolled less-than-full-time to still claim a prorated grant, but students enrolling for more than the full-time load are not paid more than the full-time amount. Students are considered “full-time” by the federal aid system if they enroll for 12 credits per semester, but typically will need to complete at least 15 credits per semester in order to graduate in four years. Since many state and institutional programs follow the federal “full-time” designation, there are few additional sources of aid for students who want to attend at

a higher intensity and graduate in four years.11 Moreover, the policy may give students the impression that a five-year timeline is typical or even recommended.

3. What are the distributional consequences of achievement incentives? What is the best way to balance achievement incentives with second chances? Is there evidence that some groups of students are more or less responsive to conditional versus unconditional financial assistance? More work could be done to test how financial incentives affect not just overall outcomes, but the distribution of outcomes across different groups.

4. Are some types of student loans more effective than others? Given the widespread reliance on student loans, a more interesting question than whether they work at all is whether they could be made to work better. So, how sensitive are students to the particular design of student loans? Are there ways to make student loans more attractive and less risky for students, without drastically increasing costs? Policy alternatives include incomecontingent loan repayment plans (such as the ones in place in the U.K., Australia, and New Zealand), or “insurance” plans which would help students make their loan payments if they cannot find a job after graduation.12 Alternatively, given that students are unlikely to perceive the difference between subsidized and unsubsidized loans until after they graduate, it would be interesting to test whether students are more responsive to an unsubsidized loan packaged with an upfront grant than to a subsidized loan with the same present discounted value.

IV. Selected Bibliography, by Financial Aid Subtopic

[*] indicates article will be summarized in attached table. Recent reviews of the literature *Baum, S, McPherson, M., & Steele, P. (Eds.). (2008). The effectiveness of student aid policies: What the research tells us. New York, NY: The College Board.

West Virginia’s state merit scholarship, PROMISE, is one notable exception. Students may claim this scholarship in the freshman year even if they enroll at the minimum full-time level, but need to complete 30 credits per year in order to renew for the following year. 12 Note that recent changes in federal loan policy give students additional protections if they experience low income after graduation, but it is unclear whether these changes are transparent enough for students to understand in advance of the decision to take a loan or not.

*Deming, D., & Dynarski, S. (2009). Into college, out of poverty? Policies to increase the postsecondary attainment of the poor (NBER Working Paper No. 15387). Cambridge, MA: National Bureau of Economic Research. *Long, B. T. (2008). What is known about the impact of financial aid? Implications for policy (NCPR Working Paper). New York, NY: National Center for Postsecondary Research. Pell Grants *Bettinger, E. (2004). How financial aid affects persistence. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 207-238). Chicago, IL: University of Chicago Press. Hansen, W. L. (1983). Impact of student financial aid on access. In J. Froomkin (Ed.), The crisis in higher education (pp. 84-96). New York, NY: The Academy of Political Science. *Kane, T. J. (1995). Rising public college tuition and college entry: How well do public subsidies promote access to college (NBER Working Paper No. 5164)? Cambridge, MA: National Bureau of Economic Research. Kane, T. J. (1996). Lessons from the largest school voucher program ever: Two decades of experience with Pell grants. In B. Fuller, R. Elmore, & G. Orfield (Eds.), Who chooses? Who loses? Culture, institutions and the unequal effects of school choice. New York, NY: Teachers College Press. *Mundel, D. (2008). Do increases in Pell and other grant awards increase college-going among lower income high school graduates? Evidence from a ‘natural experiment.’ Unpublished paper, Brookings Institution, Washington, DC. Rice, L. D., & Mundel, D. (2008). The impact of increases in Pell grant awards on college-going among lower-income youth (CCF Brief No. 40). Washington, DC: Brookings Center on Children and Families. Retrieved at http://www.brookings.edu/papers/2008/12_pell_grants_rice.aspx *Seftor, N., & Turner, S. (2002). Back to school: Federal student aid policy and adult college enrollment. Journal of Human Resources, 37(2), 336-352. Stedman, J. B. (2003). Federal Pell grant program of the Higher Education Act: Background and reauthorization. Washington, DC: U.S. Congressional Research Service. Tuition reductions and other non-merit based assistance Abraham, K., & Clark, M. (2006). Financial aid and students’ college decisions: Evidence from the District of Columbia Tuition Assistance Grant program. Journal of Human Resources, 41(3), 578-610.

Angrist, J. D. (1993). The effect of veterans benefits on education and earnings. Industrial and Labor Relations Review, 46(4), 637-652. *Avery, C., Hoxby, C., Jackson, C., Burek, K., Poppe, G., & Raman, M. (2006). Cost should be no barrier: An evaluation of the first year of Harvardâ&#x20AC;&#x2122;s Financial Aid Initiative (NBER Working Paper No. 12029). Cambridge, MA: National Bureau of Economic Research. Bound, J., & Turner, S. (2002). Going to war and going to college: Did World War II and the G.I. Bill increase educational attainment for returning veterans? Journal of Labor Economics, 20(4), 784-815. Dynarski, S. (2002). The behavioral and distributional implications of aid for college. American Economic Review, 92(2), 279-285. *Dynarski, S. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279-288. Garibaldi, P., Giavazzi, F., Ichino, A., & Rettore, E. (2007). College cost and time to complete a degree: Evidence from tuition discontinuities (NBER Working Paper No. 12863). Cambridge, MA: National Bureau of Economic Research. Hansen, W. L. (1983). Impact of student financial aid on access. In J. Froomkin (Ed.), The crisis in higher education (pp. 84-96). New York, NY: The Academy of Political Science. Heller, D. E. (1997). Student price response in higher education: An update of Leslie and Brinkman. Journal of Higher Education, 68(6), 624-659. *Kane, T. J. (2007). Evaluating the impact of the DC Tuition Assistance Grant program. Journal of Human Resources, 42(3), 555-582. *Stanley, M. (2003). College education and the mid-century G.I. bills. Quarterly Journal of Economics, 118(2), 671-708. van der Klaauw, W. (2002, November).Estimating the effect of financial aid offers on college enrollment: A regression-discontinuity approach. International Economic Review, 43(4), 1249-1287. Merit-based grants and financial incentives *Angrist, J. D., Lang, D., & Oreopoulos, P. (2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1(1), 136-163. *Brock, T., & Richburg-Hayes, L. (2006). Paying for persistence: Early results of a Louisiana scholarship program for low-income parents attending community college. New York, NY: MDRC.

Cornwell, C., Mustard, D., & Sridhar, D. (2006). The enrollment effects of merit-based financial aid: Evidence from Georgia’s HOPE scholarship. Journal of Labor Economics, 24, 761786. DesJardins, S. L., & McCall, B. P. (2007). The impact of the Gates Millennium Scholars program on selected outcomes of low-income minority students: A regression discontinuity analysis. Unpublished manuscript, University of Michigan. Retrieved from http://www-personal.umich.edu/~bpmccall/Desjardins_McCall_GMS_June_2008.pdf Dynarski, S. (2000, September). Hope for whom? Financial aid for the middle class and its impact on college attendance. National Tax Journal, 53(3), 629-661. Dynarski, S. (2004a). The new merit aid. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 63-100). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research. *Dynarski, S. (2008). Building the stock of college-educated labor. Journal of Human Resources, 43(3), 576-610. Goodman, J. (2008). Who merits financial aid?: Massachusetts’ Adams Scholarship. Journal of Public Economics, 92(10-11), 2121-2131. Heller, D., & Marin, P. (Eds.). (2003). Who should we help? The negative social consequences of merit aid scholarships. Cambridge, MA: Harvard Civil Rights Project. *Jackson, C. K. (2010). A stitch in time: The effects of a novel incentive-based high-school intervention on college outcomes (NBER Working Paper No. 15722). Cambridge, MA: National Bureau of Economic Research. Kane, T. J. (2003). A quasi-experimental estimate of the impact of financial aid on college-going (NBER Working Paper No. 9703). Cambridge, MA: National Bureau of Economic Research. *Pallais, A. (2009). Taking a chance on college: Is the Tennessee Education Lottery Scholarship a winner? Journal of Human Resources, 44(1), 199-222. *Scott-Clayton, J. (2009). On money and motivation: A quasi-experimental analysis of financial incentives for college achievement. Unpublished manuscript, Teachers College, Columbia University. Retrieved from http://faculty.tc.columbia.edu/upload/js3676/JSC_WVCollIncentives_FullDraft_Oct2009 .pdf Loans Baum, S. (2003b). The role of student loans in college access (National Dialogue on Student Financial Aid Research Report No. 5). New York, NY: College Board.

Burdman, P. (2005). The student debt dilemma: Debt aversion as a barrier to college access. Berkeley, CA: University of California, Center for Studies in Higher Education. Campaigne, D.A., & Hossler, D. (1998). How do loans affect the educational decisions of students? In R. Fossey & M. Bateman (Eds.), Condemning students to debt (pp. 85-104). New York, NY: Teachers College Press. Chapman, C. (1994). Income-contingent college loans: Correspondence. Journal of Economic Perspectives, 8(4), 205-206. Cofer, J., & Somers, P. (2000). A comparison of the influence of debtload on the persistence of students at public and private colleges. Journal of Student Financial Aid, 30(2), 39-58. Callender, C., & Jackson, J. (2005). Does the fear of debt deter students from higher education? Journal of Social Policy, 34(2), 39-58. *Dynarski, S. (2005). Loans, liquidity and schooling decisions. Unpublished manuscript, Harvard University, Cambridge, MA. *Field, E. (2009). Educational debt burden and career choice: Evidence from a financial aid experiment at NYU law school. American Economic Journal: Applied Economics, 1(1), 1-21. Krueger, A., & Bowen, W. G. (1993). Policy watch: Income-contingent college loans. Journal of Economic Perspectives, 7(3), 193-201. Perna, L. (2007). Understanding high school students' willingness to borrow to pay college prices. Research in Higher Education, 49(7), 589-606. Reyes, S. L. (1995). Educational opportunities and outcomes: The role of the guaranteed student loan. Unpublished manuscript, Harvard University, Cambridge, MA. Rothstein, J., & Rouse, C. E. (2007). Constrained after college: Student loans and early career occupational choices (NBER Working Paper No. 13117). Cambridge, MA: National Bureau of Economic Research. Tax benefits Cronin, J. (1997). The economic effects and beneficiaries of the administrationâ&#x20AC;&#x2122;s proposed higher education tax subsidies. National Tax Journal, 50(3), 519-540. *Dynarski, S. (2004b). Who benefits from the education saving incentives? Income, educational expectations and the value of the 529 and Coverdell. National Tax Journal, 57(2), 359383. *Hoxby, C. M. (1998). Tax incentives for higher education. In J. Poterba (Ed.), Tax policy and the economy. Cambridge, MA: National Bureau of Economic Research.

Kane, T. J. (1997). Beyond tax relief: Long-term challenges in financing higher education. National Tax Journal, 50(2), 335-349. *Long, B. T. (2004b). The impact of federal tax credits for higher education expenses. In C. Hoxby (Ed.), College choices: The economics of which college, when college, and how to pay for it (pp. 101-168). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research. On-campus student employment DeSimone, J. (2008). The impact of employment during school on college student academic performance (NBER Working Paper No. 14006). Cambridge, MA: National Bureau of Economic Research. DesJardins, S. L., Ahlburg, D. A., & McCall, B. P. (2002). Simulating the longitudinal effects of changes in financial aid on student departure from college. Journal of Human Resources, 37(3), 653-679. Kalenkoski, C., & Pabilonia, S. (2010). Parental transfers, student achievement, and the labor supply of college students. Journal of Population Economics, 23(2), 469-496. Pascarella, E., & Terenzini, P. (2005). How college affects students, volume 2: A third decade of research. San Francisco, CA: Jossey-Bass. *Stinebrickner, T., & Stinebrickner, R. (2003). Working during school and academic performance. Journal of Labor Economics, 21(2), 473-491. Complexity *Bettinger, E., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2009). The role of information and simplification in college decisions: Results from the H&R Block FAFSA experiment (NBER Working Paper No. 15361). Cambridge, MA: National Bureau of Economic Research. *Dynarski, S., & Scott-Clayton, J. (2006b, June). The cost of complexity in federal student aid: Lessons from optimal tax theory and behavioral economics. National Tax Journal, 59(2), 319-356.

Topic: Effectiveness of Financial Aid Study Design

Sample size, Characteristics, and Power

External Validity

Baum, S, McPherson, M., & Steele, P. (Eds.). (2008). The effectiveness of student aid policies: What the research tells us. New York, NY: The College Board.

n/a

Simplicity and transparency in aid systems is critical; early awareness/understanding of aid may improve college preparation; aid does too little to encourage completion rather than just access; too little support for adult students; inherent tension between simplicity and tailoring programs to diverse student needs.

Suggestive

Deming, D., & Dynarski, S. (2009). Into college, out of poverty? Policies to increase the postsecondary attainment of the poor (NBER Working Paper No. 15387). Cambridge, MA: National Bureau of Economic Research.

n/a

Reducing college costs can increase both college entry and persistence; simplicity and transparency are critical program design features; money linked to incentives and/or services appears to strengthen effectiveness of financial aid.

Strong

Long, B. T. (2008). What is known about the impact of financial aid? Implications for policy (NCPR Working Paper). New York, NY: National Center for Postsecondary Research.

n/a

Information and simplicity are important; grants are more effective than loans or tax credits; need‐based aid is most effective for low‐income students; financial aid is not a panacea. Future research should examine importance of policy design features, whether certain types of aid work better/worse for certain subpopulations (e.g. older students, Hispanic students), and how institutions respond to aid.

Strong

Citation

Summary of Findings

Assessment of Evidence

Notes/Caveats

Subtopic: Recent reviews

Chapters take a broader/deeper survey of the literature than the two recent reviews by economists, but not all studies are critically reviewed, and some important studies are left out (e.g. work by Stinebrickner and Stinebrickner [2003] on student employment). Short piece; explains why correlational evidence is insufficient to establish causal effects of aid on outcomes; briefly reviews the quasi‐experimental and experimental literature; includes summary table of experimental and quasi‐ experimental studies. Broader than Deming/Dynarski but shorter than Baum et al.; five sections covering justifications for aid, role of information/complexity, effectiveness of grants, effectiveness of other aid (loans, tax credits, savings incentives), and institutional responses to aid.

Topic: Effectiveness of Financial Aid Citation

Sample size, Characteristics, and Power

External Validity

Randomized experiment

Initially random sample of H&R Block tax clients meeting demographic requirements yielded 800 dependent and 14,000 independent individuals

Relatively strong (see sample description)

Strong positive effects of full treatment on financial aid application for all groups. Increased college enrollments by 7 percentage points for dependent participants. Increased Pell receipt by all groups.

Strong

An interesting finding is that the treatment involving information‐only, but no assistance in completing the FAFSA, showed no impacts.

Simulations comparing actual aid to simulated amounts calculated under alternative formulae

Nationally representative sample of financial aid applicants from National Postsecondary Student Aid Survey (NPSAS 2004)

Strong

Parents’ adjusted gross income, marital status, family size, and number of family members in college explain over 75% of variation in Pell Grant awards, yet the FAFSA collects over 70 financial items most of which contribute little to targeting. FAFSA is longer and more complicated than an IRS 1040.

Strong

This paper focuses on dependent, full‐time students; later work finds similar results for independent students.

Quasi‐ experimental (difference‐in‐ difference comparing trends over time in states with high and low home values)

18‐19 year olds in CPS 1984‐ 2000; secondary analyses use SIPP data

Strong

Mixed/weak findings. CPS suggests a $1000 increase in loan eligibility increases attendance by 1.7pp, with shift towards four‐year private institutions. SIPP provides much better data on assets, and finds only weak supporting evidence, with very imprecise estimates, and some implications could not be confirmed in these data (regarding effects by income subgroup; using IV for home equity; linking timing to 1992).

Inconclusive

CPS analysis is based on state‐year median home values, not individual measures. Endogeneity of home equity is possible bias in individual data.

Study Design

Summary of Findings

Assessment of Evidence

Notes/Caveats

Subtopic: Complexity Bettinger, E., Long, B. T., Oreopoulos, P., & Sanbonmatsu, L. (2009). The role of information and simplification in college decisions: Results from the H&R Block FAFSA experiment (NBER Working Paper No. 15361). Cambridge, MA: National Bureau of Economic Research. Dynarski, S., & Scott‐Clayton, J. (2006b, June). The cost of complexity in federal student aid: Lessons from optimal tax theory and behavioral economics. National Tax Journal, 59(2), 319‐356.

Subtopic: Loans Dynarski, S. (2005). Loans, liquidity and schooling decisions. Unpublished manuscript, Harvard University, Cambridge, MA.

Topic: Effectiveness of Financial Aid Citation

Study Design

Field, E. (2009). Educational debt burden and career choice: Evidence from a financial aid experiment at NYU law school. American Economic Journal: Applied Economics, 1(1), 1‐21.

Randomized experiment

Sample size, Characteristics, and Power NYU law school students who enlisted in the study, from the classes of 1998, 1999, 2000, and 2001

External Validity Somewhat limited (see sample description)

~1600 college freshmen at large public university in Canada; uses administrative data and student surveys

Relatively strong (based on large sample, but from single university in Canadian context)

Summary of Findings Treatment groups are 14.1 percentage points more likely to take a public‐sector job and 12.2 percentage points more likely to take a clerkship after leaving law school. A $10,000 increase in school debt reduces the likelihood of taking a public interest job two years after law school by approximately 6%.

Assessment of Evidence Strong

Notes/Caveats This study is frequently cited as providing evidence that students are "debt averse." The two treatments were financially equivalent, yet framing the program as a "loan which can be forgiven if you pursue public service" was much less effective in inducing students to public service than a "grant which will convert to a loan if you do not pursue public service."

Subtopic: Merit‐based grants, financial incentives Angrist, J. D., Lang, D., & Oreopoulos, P. (2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1(1), 136‐ 163.

Randomized experiment

No impact of financial incentives alone for any group, but some positive effects for women who received the combined incentive‐ plus‐services treatment.

Strong

Very strong study finding relatively little effect of large incentives. Because students did not receive any benefit from the program "up front," this may have limited their engagement with the program's incentives. Also, incentives were individually‐established and included multiple levels, perhaps making the program confusing?

Topic: Effectiveness of Financial Aid Sample size, Characteristics, and Power ~500 low‐income single parents (mothers) at two community colleges in Louisiana; uses administrative data and student surveys

External Validity Somewhat limited (see sample description)

Quasi‐ experimental (difference‐in‐ difference comparing trends in selected and comparison states over time)

Representative sample of 18‐19 year olds in GA, AK, FL, KY, LA, MS, SC using Current Population Survey data

Strong

Increases the proportion of 18‐19 year olds enrolled by about 5 percentage points.

Strong

Quasi‐ experimental (difference‐in‐ difference comparing trends in selected and comparison states over time)

Cross‐sectional data covering different age groups in Georgia and Arkansas, using Census 2000 data

Strong

Increases the total share of an age cohort with some college by about 1.6pp; share that has a BA by 3 percentage points (from base on 27 percent). Dropout conditional on entry estimated to decrease about 4.3 percentage points.

Strong

Citation

Study Design

Brock, T., & Richburg‐Hayes, L. (2006). Paying for persistence: Early results of a Louisiana scholarship program for low‐ income parents attending community college. New York, NY: MDRC.

Randomized experiment

Dynarski, S. (2004a). The new merit aid. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 63‐100). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research. Dynarski, S. (2008). Building the stock of college‐educated labor. Journal of Human Resources, 43(3), 576‐610.

Summary of Findings Significant, large impacts on persistence into second and third semesters (18pp and 11pp respectively), with all of impact on full‐time enrollment (20pp and 11pp respectively). Increase in GPA in second sem (0.4/2.1) but not significant in first sem. Total impact on credits earned after 3 sems: 3.3/7.7 baseline.

Assessment of Evidence Strong

Notes/Caveats Significant increases in full‐ time enrollment are surprising given the incentive was linked only to half‐time enrollment. Report casts impacts as somewhat small, but could be a story of very large effects for a subset of recipients. In percentage terms effects are quite substantial. This paper is an extension of Dynarski (2000) "Hope for Whom?" on GA Hope; here she broadens to seven states with similar programs. Examines enrollment flows, not "stock" of college graduates. One of the only studies looking at the effect of aid on college completion, not just enrollment.

Topic: Effectiveness of Financial Aid Citation

Study Design

Jackson, C. K. (2010). A stitch in time: The effects of a novel incentive‐based high‐school intervention on college outcomes (NBER Working Paper No. 15722). Cambridge, MA: National Bureau of Economic Research.

Quasi‐ experimental (difference‐in‐ difference comparing before/after changes at schools that had the program to changes at schools that did not) Quasi‐ experimental (difference‐in‐ difference comparing trends in selected and comparison states over time)

Pallais, A. (2009). Taking a chance on college: Is the Tennessee Education Lottery Scholarship a winner? Journal of Human Resources, 44(1), 199‐222.

Scott‐Clayton, J. (2009). On money and motivation: A quasi‐ experimental analysis of financial incentives for college achievement. Unpublished manuscript, Teachers College, Columbia University. Retrieved from http://faculty.tc.columbia.edu/ upload/js3676/JSC_WVCollInce ntives_FullDraft_Oct2009.pdf.

Quasi‐ experimental (regression discontinuity based upon ACT test score cutoff for eligibility; before/after comparison)

Sample size, Characteristics, and Power Population of 10th graders attending Texas high schools from 1999‐2006; uses Texas administrative data

External Validity Strong

Random sample of Tennessee ACT test‐takers using ACT microdata

Strong

6‐8 percentage point (11‐14%) increase in test‐takers scoring 19 or above; cannot be explained by retesting. No significant effects on score‐sending or stated college preferences (if anything, tended to increase favor for out‐of‐state schools).

Strong

Universe of West Virginia public college first‐time enrollees using WV administrative data

Relatively strong (West Virginia is significantly more disadvantag ed and less racially diverse than nation as a whole)

6.7 percentage point (25%) increase in on‐time completion; 3.5 percentage point (7%) increase in completion after 5 years; 6 additional credits after four years; no effects on GPA after 4 years; ‐$10/wk decline in student employment; effects disappear in 4th year when students still receive scholarship but no longer face academic incentives.

Strong

Summary of Findings The program increased college enrollment, as well as GPAs and persistence conditional on enrollment. Program increased college completions for minorities, but not white students.

Assessment of Evidence Strong

Notes/Caveats As the author mentioned, there is some concern that the enrollment effects may partly reflect an increased likelihood of enrolling in Texas rather than some other state.

Other work has found merit aid increases high school GPAs, but ACT scores are more standardized, may be more objective, and may be more difficult to manipulate via changes in coursetaking, etc. Provides strong evidence that the academic incentives embodied in merit aid scholarships are key to their impact; a grant with no strings attached would not have had the same effect.

Topic: Effectiveness of Financial Aid Citation

Study Design

Sample size, Characteristics, and Power

External Validity

Summary of Findings

Assessment of Evidence

Notes/Caveats

Subtopic: Pell Grants Bettinger, E. (2004). How financial aid affects persistence. In C. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 207‐238). Chicago, IL: University of Chicago Press.

Matched/ controlled (looking at same individuals over time, whose financial aid varied); quasi‐ experimental (regression discontinuity based on discontinuous changes in Pell Grant amounts by family size/number in college)

Incoming freshman class in 1999‐2000 school year in Ohio 2 and 4‐year public schools; uses Ohio administrative data

Strong

Results suggest that larger Pell Grants reduce the likelihood of dropout; however the results are not robust to alternative specifications.

Suggestive

Kane, T. J. (1995). Rising public college tuition and college entry: How well do public subsidies promote access to college (NBER Working Paper No. 5164)? Cambridge, MA: National Bureau of Economic Research.

Quasi‐ experimental (difference‐in‐ difference comparing states with bigger/smaller tuition increases over time; comparing eligible and ineligible populations before/after changes in Pell funding)

18‐19 year olds from 1977 to 1993 using October CPS; other analyses using National Longitudinal Survey of Youth (NLSY) 1979 and High School and Beyond (HSB) 1980

Relatively strong (data may be getting somewhat outdated)

College enrollments appear highly responsive to college tuition charges, but less so to need‐based aid. After Pell Grants were introduced, there was no disproportionate increase in enrollments by low‐income youth.

Suggestive

One of the few studies to examine persistence rather than just initial enrollment. The evidence is suggestive, but the estimates are not particularly robust. To the extent they can be taken at face value, these results could indicate that problems of complexity and poor information (which have been cited as explanations for the non‐ impact of Pell Grants on initial entry) may be ameliorated for students after they enroll and learn their aid amount. The results at the time were surprising: that students are sensitive to college costs but not to need‐based aid? Dynarski and Scott‐Clayton (2006) suggest that complexity in the Pell eligibility and application rules may diminish its impact.

Topic: Effectiveness of Financial Aid Sample size, Characteristics, and Power High school graduates from middle, moderate and low‐ income families (1995‐ 2004); uses three waves of the National Postsecondary Student Aid Survey (NPSAS) and CPS data

External Validity Strong (though internal validity is somewhat questionabl e)

Quasi‐ experimental (difference‐in‐ difference comparing eligible and ineligible populations before and after increases in Pell funding)

Independent high school graduates ages 22‐35; uses October CPS for sub‐periods between 1969 and 1990

Relatively strong (data may be getting somewhat outdated)

The difference‐in‐differences estimates of the effect of Pell eligibility on the probability of attending college is 1.5 percentage points for men and 1.3 percentage points for women. When compared to the enrollment rates before the introduction of the program, this reflects a relative growth of 16 percent for men and 40 percent for women.

Strong

No comparison

Households who invest in these savings accounts; “typical “ households with school‐going children; using 2001 Survey of Consumer Finances (SCF) and 2000 NPSAS

Strong

Dynarski finds that the advantages of the 529 and Coverdell rise sharply with income for three reasons. Those in the top tax brackets benefit more from non‐educational use of a Coverdell than those in the bottom bracket gain from its educational use.

Strong

Citation

Study Design

Mundel, D. (2008). Do increases in Pell and other grant awards increase college‐going among lower income high school graduates? Evidence from a ‘natural experiment.’ Unpublished paper, Brookings Institution, Washington, DC.

Quasi‐ experimental (before‐after comparison)

Seftor, N., & Turner, S. (2002). Back to school: Federal student aid policy and adult college enrollment. Journal of Human Resources, 37(2), 336‐352.

Summary of Findings During the 1999‐2004 years, the adjusted immediate college going rate for low income youth increased by roughly 6‐7 percentage points, while the adjusted rate for moderate income youth remained constant, declining by roughly 0‐1 percentage point. The immediate college going rate for middle income youth increased by 4 percentage points.

Assessment of Evidence Inconclusive

Notes/Caveats Paper is a policy brief which does not provide the detailed information needed to evaluate the methodology. Though overall trends are consistent with the findings, other trends could explain the findings as well, such as changes in state and institutional assistance (rather than changes in the Pell grant). (1) The index on which the eligibility to be awarded a Pell grant was calculated using the midpoint of the category as the income and basis for taxes. (2) The authors do not present a comparison of the means to see if treatment and control groups were comparable.

Subtopic: Tax benefits Dynarski, S. (2004b). Who benefits from the education saving incentives? Income, educational expectations and the value of the 529 and Coverdell. National Tax Journal, 57(2), 359‐383.

This is not an impact evaluation, but an examination of how benefits are distributed across the population. However, the suggestion is that the program is unlikely to have much impact on college outcomes if it is primarily utilized by high‐ income families.

Topic: Effectiveness of Financial Aid Citation

Study Design

Long, B. T. (2004b). The impact of federal tax credits for higher education expenses. In C. Hoxby (Ed.), College choices: The economics of which college, when college, and how to pay for it (pp. 101‐168). Chicago, IL: University of Chicago Press and the National Bureau of Economic Research.

Quasi‐ experimental (difference‐in‐ difference between eligible and ineligible individuals before and after the introduction of the credits)

Sample size, Characteristics, and Power College‐going individuals in IPEDS (1993‐1994 to 1990‐ 2000) and October CPS data (1990‐2000)

External Validity Strong

Summary of Findings Tax credits did not increase postsecondary enrollment among credit‐eligible students, nor were students more likely to invest in a 4‐ year (vs. 2‐year) program. The low take‐up rate suggests that not enough families know about the benefit for it to have a discernible impact. Disconnect between the timing of the benefit and college enrollment may limit the effects.

Assessment of Evidence Suggestive

Notes/Caveats The data are somewhat limited for the analysis (e.g. income data is recorded only in brackets, not exact amounts). Measurement error may attenuate results. Not clear that trends for eligible and ineligible students are comparable.

Subtopic: Tuition reductions and other non‐merit based aid Avery, C., Hoxby, C., Jackson, C., Burek, K., Poppe, G., & Raman, M. (2006). Cost should be no barrier: An evaluation of the first year of Harvard’s Financial Aid Initiative (NBER Working Paper No. 12029). Cambridge, MA: National Bureau of Economic Research. Dynarski, S. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279‐288.

Quasi‐ experimental (before‐after comparison)

Data from 16,821 Harvard applications in 2008 and from 19,321 applications in 2009

Somewhat limited (see sample description)

The percentage of applicants from families with incomes of $40,000 or below jumped by more than 20%. Enrollment of students qualifying for the Initiative increased by 11% in one year; while enrollment of students with family incomes below $40,000 increased by nearly 20%.

Suggestive

Quasi‐ experimental (difference‐in‐ difference between children with/without deceased father, before/after the program's elimination in 1982)

12,686 youth from the NLSY 1979; "treated" group include only those whose father was deceased at time of college decision

Somewhat limited (see sample description; data may be becoming outdated)

Dynarski finds that “the elimination of the Social Security student benefit program reduced college attendance probabilities by more than a third. These estimates suggest that an offer of $1,000 in grant aid increases the probability of attending college by about 3.6 percentage points. Aid eligibility also appears to increase completed schooling.”

Strong

The before‐after comparison is somewhat limited in that it relies on only two years of data, and some data are not available for all applicants (e.g., complete financial data is only available for students who enrolled). A seminal study documenting the effect of student assistance on college enrollments.

Topic: Effectiveness of Financial Aid Sample size, Characteristics, and Power Data on first‐time freshmen (DC residents) starting in 1998 (1572 students) until 2002 using Integrated Postsecondary Education Data System (IPEDS) aggregate statistics; financial aid application data; TAG administrative data

External Validity Strong

Quasi‐ experimental (Korean War GI bill analysis); matched/controll ed (WWII GI bill analysis)

Sample of 532 individuals from the Survey of Occupational Change in a Generation (OCG), a supplement to the CPS in 1962 and 1973; sample of 240 veterans from the 1978 Survey of Veterans data

Somewhat limited (see sample description)

The combination of the Korean War and WWII GI bills probably increased total postsecondary attainment among all men born between 1921 and 1933 by about 15 to 20 percent, with smaller effects for surrounding cohorts. The impacts were apparently concentrated among veterans from families in the upper half of the socioeconomic distribution.

Strong

Quasi‐ experimental (instrumental variables analysis; some students worked more because they were randomly assigned to jobs with more hours available)

Sample of 2,372 first year students at Berea College, small private college in Kentucky where all students receive free tuition, room & board; have to work at least 10 hrs per week

Limited (see sample description)

An additional hour of student employment (above 10 hours per week) reduces first‐year GPAs by 0.162 points on a four‐point scale.

Strong

Citation

Study Design

Kane, T. J. (2007). Evaluating the impact of the DC Tuition Assistance Grant program. Journal of Human Resources, 42(3), 555‐582.

Quasi‐ experimental (before‐after, also relies upon staggered timing of effective tuition changes in Maryland, Virginia, and other states)

Stanley, M. (2003). College education and the mid‐century G.I. bills. Quarterly Journal of Economics, 118(2), 671‐708.

Summary of Findings Between 1998 and 2000 the number of D.C. residents attending public institutions in Virginia and Maryland more than doubled. The impact was largest at nonselective public four‐ year colleges, particularly predominantly black institutions. The total number of financial aid applicants, Pell Grant recipients and college entrants from D.C. also increased by 15 percent or more.

Assessment of Evidence Strong

Notes/Caveats It may not be appropriate to attribute all of the increase to the D.C. TAG program. The D.C. College Access Program (D.C. CAP)—was a smaller, but still significant private program that also began operations in six public high schools in D.C. for those graduating in the spring of 2000. It may account for some share of the increase in enrollment observed over that time period.

Subtopic: Work‐study Stinebrickner, T., & Stinebrickner, R. (2003). Working during school and academic performance. Journal of Labor Economics, 21(2), 473‐ 491.

This is virtually the only credible study of student employment at the college level; unfortunately external validity may be limited.