Tutoring program returns to lausd by UCLA Luskin School of Public Affairs

Tutoring Returns to LAUSD Designing a District Tutoring Program Logan Contreras Greg Srolestar Chen Wang 3/26/2012

This report examines current Supplemental Educational Services tutoring in LAUSD and provides guidance to LAUSD as it designs its own SES tutoring program.

Table of Contents Executive Summary ........................................................................................................................ 3 Introduction ..................................................................................................................................... 5 LAUSD ....................................................................................................................................... 5 No Child Left Behind and the Provision of Tutoring ................................................................. 6 Designing a Better Tutoring Program ......................................................................................... 8 Methodology ................................................................................................................................... 9 Identifying Best Practices ........................................................................................................... 9 Interviews with Field Experts ................................................................................................... 10 Interviews with SES providers.................................................................................................. 10 Quantitative Data Analysis ....................................................................................................... 10 Landscape Analysis of Current SES Tutoring ...............................................................................11 SES Utilization by LAUSD Students ........................................................................................11 Characteristics of SES Providers within LAUSD..................................................................... 14 SES Tutoring Practices in LAUSD ........................................................................................... 16 Evaluation Criteria ........................................................................................................................ 18 Analysis of Tutoring Program Design Dimensions ...................................................................... 19 Students Served ......................................................................................................................... 19 Course Subject .......................................................................................................................... 23 Curriculum ................................................................................................................................ 25 Learning Configuration ............................................................................................................. 25 Instructor Training/Qualifications ............................................................................................. 28 Dosage....................................................................................................................................... 32 Varying Dosage Models ............................................................................................................ 35 Learning Site Location .............................................................................................................. 38 Implementation Considerations .................................................................................................... 40 Market Share ............................................................................................................................. 40 Cost Estimate ............................................................................................................................ 40 Staff Considerations .................................................................................................................. 41 Programmatic Costs .................................................................................................................. 43 1

Political Considerations ............................................................................................................ 44 Conclusion .................................................................................................................................... 44 References ..................................................................................................................................... 47 Appendices .................................................................................................................................... 52

Executive Summary The 2001 reauthorization of the Elementary and Secondary Education Act, known as No Child Left Behind, established a tutoring program entitled Supplemental Education Services (SES) for low-income students at poorly-performing schools. The federally-funded program provides eligible students with vouchers to spend at the tutoring provider of their choice. As an added accountability measure, poorly-performing districts like the Los Angeles Unified School District (LAUSD) are disallowed from directly providing federally-funded tutoring. Instead, LAUSD students choose from a selection of approved private tutoring providers with the costs reimbursed by federal funds. In recent years, evaluations of SES tutoring both in LAUSD and elsewhere have found the current private SES tutoring system minimally effective at improving student achievement. In 2005, the U.S. Department of Education initiated a pilot program that permits poorlyperforming districts like LAUSD to serve as SES providers. A recent expansion of this federal program provides a window of opportunity for LAUSD to enter the SES tutoring market. LAUSD asked us to assist in designing a district-provided tutoring program to compete with private providers in the SES market. In completing the program design, we relied on four sources of information and data: analysis of relevant scholarly research, face-to-face interviews with field experts, phone interviews with private SES providers, and quantitative data sets either provided by LAUSD or publically available. We utilized these diverse sources of information and data to conduct the following tasks: (1) perform a landscape analysis of current SES tutoring system in LAUSD; (2) examine key dimensions of tutoring program design across several criteria: effectiveness at improving student learning, parent preferences, cost, legality and political stakes; (3) propose policy recommendations on each dimension of tutoring program: student served, course subject, learning configuration, instructor training/qualifications, dosage, and learning location; and (4) address some further implementation considerations of this tutoring program. Based on our analysis, we provided several recommendations for a LAUSD SES tutoring program: ď&#x201A;&#x; Student served: Given the difficulties in starting a quality, large-scale tutoring program, LAUSD should initially focus on one group of students. We recommend beginning with elementary students primarily because elementary enrollment is much higher. ď&#x201A;&#x; Course subject: LAUSD should focus on both math and ELA instruction and not science instruction. Science enrollment in the elementary grades is negligible and both math and ELA tutoring can significantly benefit LAUSD students. 3

   

Learning configuration: We tentatively recommend one-to-one tutoring for ELA and smallgroup tutoring for mathematics. We prefer small groups of two or three students. Instructor training/qualifications: LAUSD should utilize college students or teachers’ for one-to-one tutoring, and teachers’ aides or certified teachers for small-group instruction. Dosage: We suggest that sessions last between ½ and one hour, meet 2-3 times per week, and last between 10 and 35 weeks. High frequency is a greater priority for ELA instruction. Location: We recommend that LAUSD offer on-site tutoring services at each school that serves students who enroll in the district program.

Introduction As the second largest school district in the nation, when the Los Angeles Unified School District (LAUSD) begins a new program, the effects can be profound. Currently, the district is applying for permission from the California Department of Education to offer federally-funded tutoring services directly to low-income, low-achieving students. If granted approval, the district will need to construct a tutoring program that can cost-effectively improve student achievement within the legal constraints set by the state and federal governments. This report offers LAUSD a blueprint for designing a tutoring program. We begin by explaining the basics of the federally-funded tutoring program called Supplemental Educational Services (SES) and LAUSD’s new opportunity to directly provide these services. Next, we take a clear look at the current state of current SES within LAUSD. We then examine the key components of a tutoring program, and consider how best to tailor each element to the current context. Lastly, we project cost and likely implementation hurdles. In all, this report provides valuable guidance to LAUSD and other districts looking to build an SES tutoring program.

LOS ANGELES UNIFIED SCHOOL DISTRICT The Los Angeles Unified School District (LAUSD) is the largest public school system in the State of California as measured by student enrollment, and second largest in the United States. 1 It serves a Kindergarten through 12th Grade (K-12) student body of 664,233. 2 LAUSD offers K12 instruction for students residing within the City of Los Angeles as well as 31 surrounding communities. 3 The jurisdiction of LAUSD includes approximately 950 school sites, in an area that encompasses 710 square miles of land. 4 This territory is divided into eight Local Districts, each with its own superintendent and support staff. This project was sponsored by Dr. Donna Muncey, the Chief of Intensive Support and Intervention for LAUSD (see Appendix B for full organization chart). Much of our work took place with direction and assistance from Beyond the Bell Branch (hereafter, Beyond the Bell), the subdivision with LAUSD’s Office of Curriculum and Instruction that administers extended learning and enrichment programs outside of the regular school day. 1

Thomas D. Snyder and Sally A. Dillow. “Digest of Education Statistics 2010.” (Washington, DC: NCES, 2011), 138, Table 94.

Los Angeles Unified School District. Office of Communications. “Fingertip Facts 2011-2012.” (Los Angeles, CA: LAUSD,

2011), 1 3

Communications, “Fingertip Facts.”, 1.

Communications, “Fingertip Facts.”, 2. 5

NCLB AND THE PROVISION OF TUTORING The No Child Left Behind Act of 2001 (NCLB) instituted a series of choice and accountability measures including Supplemental Education Services (SES), the focus of this project. Under the accountability provisions of NCLB, all Title I funded schools 5 that do not make Adequate Yearly Progress (AYP) for two consecutive years are identified for Program Improvement (PI). 6 Schools in Program Improvement status must, after one full year, offer low-income students public school choice options and SES tutoring. The purpose of the SES policy is to provide students in poorly performing public schools with access to academic assistance from their choice of tutoring provider. 7 NCLB requires school districts to set aside 20% of their Title I funds to finance SES and school choice transportation. 8 This project centers on Supplemental Educational Services programming in LAUSD. As indicated above, SES is a federally-funded program authorized by NCLB that provides free academic tutoring services for low-income students in low-performing schools. In some districts students may choose between a menu of private tutoring providers and an in-house district provider. Much as poorly performing schools are designated Program Improvement, so too are poorly performing districts. Districts so labeled, including LAUSD, are barred from directly providing tutoring services. 9 LAUSD students are deemed eligible for SES tutoring based on two criteria: the student must attend a school in its second through fifth year of Program Improvement status, and must also participate in the free/reduced priced meal program. 10 These criteria signify that a student attends a poorly performing school and comes from a family with income 185% of the federal

Title I refers to a portion of the Elementary and Secondary Education Act that designates funds for schools with high proportions of low-income students.

AYP is derived from Sec. 1111(b) of the federal No Child Left Behind Act of 2001. The law stipulates that State Education Agencies are required to create a timeline in which all schools must meet or exceed state academic standards. Schools that fail to meet the conditions of this timeline fail to make AYP, and are designated as “in PI”.

United States Department of Education. “Supplemental Educational Services: Nonregulatory Guidance.” (Washington, DC: ED, 2009), 8. http://www2.ed.gov/policy/elsec/guid/suppsvcsguid.pdf

The actual amount of funds spent on SES depends on districts’ choices. But the allocation for SES programs should be between 5% and 15% of the Title I funds.

John E. Deasey. “Letter to Parents/Guardians of Students in the Los Angeles Unified School District.” (letter mailed to parents of LAUSD students, Los Angeles, CA, September, 2011).

Code of Federal Regulations. “SEA responsibilities for supplemental educational services.” Title 34.sec. 200.47(b) (2011). 10

Los Angeles Unified School District, Beyond the Bell Branch, “Free Tutoring Supplemental Educational Services.”Accessed February 8, 2012. http://www.btb.lausd.net/SES/ 6

poverty line. 11 Not all eligible LAUSD students receive SES tutoring. First, only a fraction of the eligible students apply. In 2009-10, of the 382,555 eligible students, only 51,254 (13%) actually applied. 12 Because the program is oversubscribed, only a subset of those who apply can actually enroll in SES tutoring: in 2009-10, 21% of those who applied were waitlisted. 13 Additionally, some accepted students fail to attend. Those openings are filled by waitlisted students. When selecting which students receive tutoring and which are waitlisted, the district is obligated to give priority to the lowest-achieving students, as measured by state test scores, special education status, and other indicators of low achievement. 14 The SES program allows eligible students to select a tutoring provider that meets their preferences. Each student receives a federal voucher (approximately $1,700 in LAUSD) that he may spend on tutoring. 15 SES tutoring covers primarily reading and mathematics, though also science instruction, in the form of one-to-one, small group (2-5 students), group (6-10 students) or online tutoring. State Educational Agencies maintain a list of eligible SES providers. States regulate SES providers although oversight is unfunded and lax. 16 Approved SES providers include public entities such as qualifying schools and school districts, private companies, non-profit organizations and faith-based organizations. Local education agencies (LEAs), in this case LAUSD, manage student applications and funding, and provide some oversight on behalf of the state. As noted above, Beyond the Bell administers the SES program for LAUSD but does not directly provide SES tutoring. Ten years after the passage of NCLB in 2001, research has found private SES tutoring minimally effective. Despite significant financial commitment, SES tutoring has produced very modest

U.S. National Archives and Records Administration. “Child Nutrition Programs – Income Eligibility Guidelines.” Federal Register 72, no. 58 (March 25, 2011): 16724

Melissa K. Barnhart. “The Impact of Participation in Supplemental Educational Services (SES) on Student Achievement: 2009-10.” (Los Angeles Unified School District. Program an Evaluation and Research Branch, Planning, Assessment and Research Division, Los Angeles, CA, May 2011), 2.

Barnhart. “Impact 2009-10.”, 2

LAUSD officials, meeting with the authors, March 8, 2012.

California Department of Education. “Supplemental Educational Services (SES) 2011-12 Per Pupil Amount.” Posted: August 19, 2011. Accessed: March 18, 2012. http://www.cde.ca.gov/ta/ac/ti/documents/sesppa2011.xls

Patricia Burch et al., “Supplemental Educational Services and NCLB: Policy Assumptions, Market Practices, Emerging Issues.” Educational Evaluation and Policy Analysis 29, no. 2 (June 2007): 126. 7

achievement gains. 17 Specifically in LAUSD, multiple evaluations by the district’s research unit have found either no or very small effects. 18

DESIGNING A BETTER TUTORING PROGRAM The underperforming SES system called for policy changes. In 2005, the U.S. Department of Education initiated a pilot program to allow LEAs under Program Improvement to serve as SES providers. 19 Under the program, several underperforming LEAs were permitted to apply for a waiver to enter the SES tutoring market and compete with private providers to tutor eligible students. In those districts, families select either a private tutoring company from the state list or use a district-provided tutor. In this way, the district competes with the private tutoring providers. LAUSD has decided to take advantage of a recent expansion of this waiver opportunity to enter the SES tutoring market. LAUSD hopes to participate in this market for two reasons. First, LAUSD would capture some of the revenue currently spent on private providers, totaling some $54,989,769 in Title I funds. 20 Second, given the unimpressive results of private providers, LAUSD officials believe the district can improve student academic achievement more than the private providers. Our report offers a road map for LAUSD as it constructs a tutoring program that can attract and educate its students. First, we examine the current SES landscape, showing how current providers attract and assist students. Second, we review the empirical literature on tutoring and apply the results produced by credible studies to inform decisions about the design elements of a tutoring program. Lastly, we explore the cost and implementation elements associated with those 17

Shanan Chappell et al. “A Meta-Analysis of Supplemental Educational Services (SES) Provider Effects on Student Achievement.” Journal of Education for Students Placed at Risk 16, no. 1 (2011): 13, Table 8.

Jordan H. Rickles et al., “Supplemental Educational Services Participation and Impact on Student Achievement: The Case of One Urban District over Five Years.” (Paper presented at the Annual Meeting of the American Educational Research Association Symposium: The Effectiveness of Supplemental Educational Services, New York City, March 25, 2008), 33.

Jordan H Rickles and Jeffrey A. White. “The Impact of Supplemental Educational Services Participation on Student Achievement.” (Los Angeles Unified School District. Program and Evaluation and Research Branch. Planning, Assessment and Research Division., Los Angeles, CA, January 30, 2006), 5. Barnhart. “Impact 2009-10.”, 12 19

United States Department of Education. “Non-Regulatory Guidance on Title I,Part A Waivers, B8-B15” (Washington, DC: ED, July 2009).

Based off 15% of LAUSD’s 2010-11 Title I allocation; United States Department of Education, “Fiscal Year 2011 Title I Grants to Local Education Agencies – California,” Accessed March 19, 2012. http://www2.ed.gov/about/overview/budget/titlei/fy11/california.pdf 8

recommendations. While addressing all the key elements of a quality tutoring program are beyond the scope of this project, this report should provide the district with a clear path down which to proceed as it develops its own tutoring program.

Methodology To design an effective SES tutoring program, we needed some understanding of best practices, tutoring realities, the current SES market, parent preferences, costs, and common implementation difficulties. Using a combination of literature review, interviews, and data analysis, we were able to examine many of those key areas.

IDENTIFYING BEST PRACTICES To identify best practices in SES tutoring supported by empirical research, we relied on a range of academic literature. Unfortunately for our purposes, researchers tended to evaluate specific interventions as a whole and not one component at a time. For example, researchers appraised the overall impact of an elementary reading program named Proactive Reading with a very specific set of components. We sought to understand not the impact of a whole program, but its parts; yet, disentangling the effects of intervention length, tutor quality, etc., can be extremely difficult. Additionally, while much of the research covers low-achieving children like the SESeligible population in LAUSD, some programs serve students that vary in racial composition, geography, academic proficiency, and English proficiency from students at LAUSD. We needed research that could give us information about specific program components so we could apply that to the current context. As a result, we relied primarily upon meta-analyses that combine large numbers of studies to estimate effects of different program elements. One difficulty with meta-analyses is that they are only as good as the individual studies they include. The available meta-analyses vary in quality. As education research has improved significantly in recent years, we utilized meta-analyses published no earlier than 2000. Likewise we only used meta-analyses that required studies to include comparison groups and listed clear and reasonable inclusion standards. We rejected two meta-analyses that failed to meet those standards. Even so, the meta-analyses we used are not uniformly high-quality. We carefully considered the rigor of each analysis in drawing conclusions. Moreover, we did not draw firm conclusions from any individual study; instead, we noted findings that are relatively consistent across studies. Effect sizes from meta-analyses use either Cohenâ&#x20AC;&#x2122;s D or Hedgeâ&#x20AC;&#x2122;s G (see Appendix C) as units of measure; both statistics are standardized effect sizes, meaning they express increases (or decreases) in program effect without units, making them useful for comparisons among studies. On occasion we found research that carefully varied individual program elements and tested outcomes. We used those results where applicable. We also looked to see what elements the 9

highest quality programs employ to give us a sense of best practices, even if they were not directly applicable to the LAUSD situation. The literature for elementary school tutoring, the grade levels that are the focus of our analysis, overwhelmingly focused on reading and not math achievement. As a consequence, our analysis of design elements is stronger for improving reading achievement than math achievement. As we discuss later, there is ample reason to believe LAUSD can improve math achievement without detailed guidance. Even so, our analysis in particular and the assessment of math tutoring more broadly would benefit from a stronger research base.

INTERVIEWS WITH FIELD EXPERTS We interviewed a number of field experts to gain a better understand the realities of running a tutoring program. We interviewed Tiffani Chin, Ph.D., an expert on tutoring who runs a tutoring program serving LAUSD students. We consulted with Robert Tagorda, a Program Administrator in the Office of the Superintendent at Long Beach Unified School District, to learn about that districtâ&#x20AC;&#x2122;s internal tutoring program. We spoke with an official from a mid-sized school district (who requested anonymity) that previously obtained a waiver and now provides SES tutoring. We also corresponded and met with Luis Mora, the Administrative Coordinator for SES at LAUSD, and other Beyond the Bell staff members to acquire specific information about the LAUSD context.

INTERVIEWS WITH SES PROVIDERS To perform a landscape analysis of the current SES system in LAUSD, we conducted phone interviews with representatives of 23 private tutoring companies currently providing SES tutoring within LAUSD. The phone interviews typically lasted 15-30 minutes. Interview questions covered various aspects of tutoring practices including learning configuration, tutoring location, tutoring session length, tutor qualifications, training and pay rate. A full questionnaire is in Appendix D.

QUANTITATIVE DATA ANALYSIS DATA COLLECTION: We analyzed LAUSD-provided data to better characterize current SES provision within the district. Due to LAUSDâ&#x20AC;&#x2122;s Institutional Review Board restrictions we were not permitted to interview parents to learn what families want in a provider. Luis Mora of Beyond the Bell furnished us with two datasets on student enrollment and SES provider characteristics that use

numerical identifiers for the students and alpha-numeric codes for the providers. One dataset contains the SES enrollment information for LAUSD students in school year 2011-2012. This dataset consists of records for 86,228 students and includes the following variables: grade level, choice of SES provider, and school zip code. The other data file contains information about the 56 authorized SES providers in LAUSD in the same year. The variables in the SES provider data set include grades served, hours per pupil, subject areas, learning configuration, tutoring location, English Language Learners and students with disabilities served, and whether a computer is provided. These data gave us some understanding of the options available to, and chosen by, students utilizing SES tutoring. ANALYSIS: First, we conducted a descriptive analysis of the SES enrollment data and provider information. We generated descriptive statistics about enrollment by provider, prevalence of tutoring approaches, distribution of tutoring locations, subject served, total services hours, student grouping by grade and schooling levels, etc. This analysis provides useful background on the status quo and helps us to project future enrollment in an LAUSD-administered tutoring program. Second, we conducted a geographic analysis of the distribution of SES applicants by looking at the concentration of students by zip code, and mapped the SES applicant density in the school district. This geographic analysis helped us consider the best model for locating tutoring services.

Landscape Analysis of Current SES Tutoring This section contains a landscape analysis of the existing SES tutoring programs in LAUSD. We focus on factors including utilization rates, course subjects, tutor qualifications and pay rate, tutoring session length, and session frequency. The following analysis mainly relies on the enrollment and provider data provided by Beyond the Bell Branch and information collected from phone interviews we conducted with 23 of the 56 approved SES providers.

SES UTILIZATION BY LAUSD STUDENTS In order to examine the SES utilization by LAUSD students, we first looked at the enrollment data for LAUSD students who applied for SES services between 2002-03 and 2009-10. By examining the historical SES enrollment rate, we observe several significant patterns in SES utilization. Between 2002-03 and 2009-10, while the total number of students in LAUSD dropped, the number of students eligible for tutoring increased by 134%. In 2009-10, eligible students accounted for 56% of the whole student population. This dramatic increase in eligibility was driven by an increase in the number of schools designated Program Improvement (PI). The 11

students in those schools newly designated PI automatically became eligible for SES services. The number of schools in PI grew from 173 schools 2005-06 to 343 schools 2010-11. Table 1. SES Utilization by LAUSD Students School Year

02/03

03/04

04/05

05/06

06/07

07/08

09/10

778,437

779,267

769,231

749,918

723,047

704,806

681,974

Not Eligible

79%

78%

71%

61%

58%

46%

44%

Eligible

21%

22%

29%

39%

42%

54%

56%

Did Not Apply

94%

89%

88%

89%

88%

87%

11%

12%

11%

12%

11%

13%

Did Not Attend

37%

31%

36%

38%

27%

33%

Attended

63%

69%

64%

62%

73%

67%

Attended ELA

54%

44%

37%

31%

41%

43%

42%

High Attendance

22%

19%

21%

19%

12%

71%

89%

Medium

38%

34%

42%

16%

17%

Low Attendance

40%

47%

37%

65%

80%

11%

Attended Math

10%

24%

26%

31%

32%

30%

25%

High Attendance

31%

22%

25%

20%

17%

69%

87%

Medium

20%

31%

30%

15%

10%

18%

49%

48%

45%

65%

74%

13%

Total Number of Students

Applied

Attendance

Attendance Low Attendance

Sources: Melissa K. Barnhart, “The Impact of Participation in Supplemental Educational Services (SES) on Student Achievement: 2009-10.” (Los Angeles Unified School District. Program an Evaluation and Research Branch, Planning, Assessment and Research Division, Los Angeles, CA, May 2011). Jordan H. Rickles and Melissa K. Barnhart, “The Impact of Supplemental Educational Services Participation on Student Achievement: 2005-06,” (Los Angeles Unified School District. Program an Evaluation and Research Branch, Planning, Assessment and Research Division, Los Angeles, CA, May 2011). Melissa K. Barnhart, “The Impact of Participation in Supplemental Educational Services (SES) on Student Achievement: 2007-08,” (Los Angeles Unified School District. Program an Evaluation and Research Branch, Planning, Assessment and Research Division, Los Angeles, CA, March 2009). Notes: 1.

Since LAUSD did not conduct evaluation study in school year 2008-2009, so enrollment data for 08/09 could not be obtained.

Low Attendance=1% to 49% of program hours Medium Attendance=50% to 89% of program hours High Attendance=90% to 100% of program hours

Despite the eligibility increases, the sign-up rate for SES remains very low. In order to utilize SES, an eligible student must first apply for SES tutoring program, pick their top three provider choices, and then be selected and assigned to an SES provider by the district. If we look at the enrollment data, we observe that only a small proportion of eligible students apply for SES programs, around 12% annually since 2004. Moreover, only two-thirds of those who signed up for SES actually attended an SES tutoring program, which implies that only 7%-8% of eligible students actually received SES tutoring. Despite the low sign-up rate, 21% of applicants were waitlisted and could not receive services in 2009-10. 21 This figure indicates that SES providers cannot serve all LAUSD applicants for tutoring services. Even though LAUSD allocates the legal maximum (15% of Title I funds) to finance SES, there were still insufficient funds to offer all applicants SES services; therefore, LAUSD adopted an internal screening selection mechanism to select students. Priority for SES selection was given to students with low California Standardized Tests (CST) scores and who belonged to disadvantaged student groups, e.g., English language learners (ELL) and students with disabilities (SPED). 22 Students in second grade and below who do not take CSTs are low priority. 23 Like the enrollment rate, the attendance rate characterizes the utilization and impact of SES. As shown in Figure 1, the proportion of high-attenders in both English Language Arts (ELA) and Math programs has increased significantly over the years. In school year 2009-2010, nearly 90% of participating students had high attendance, defined as completing at least 90% tutoring hours offered. 24 As we discuss later, the increasing prevalence of providers that give free laptops to high attenders may be partially responsible for this trend.

Barnhart, “Impact 2009-10,” 3.

LAUSD officials, meeting, March 8.

Barnhart, “Impact 2009-10,” 2. 13

Figure 1. ELA and Math Attendance Frequency 25000 20000 15000 Number of Students 10000

ELA Attendance ELA Medium Attendence ELA High Attendence

5000 0

25000 20000 Number of Students

Math Low Attendance

15000

Math Medium Attendence

10000 5000

Math High Attendence

Notes:

Low Attendance=1% to 49% of program hours Medium Attendance=50% to 89% of program hours High Attendance=90% to 100% of program hours

CHARACTERISTICS OF SES PROVIDERS WITHIN LAUSD To deepen our understanding of the current SES system, we conducted an overview assessment of the characteristics of exiting SES providers within LAUSD. Both the Federal and State Departments of Education have promulgated regulations for implementing SES programs, which lay out basic guidelines and criteria for approving SES providers including specific requirements on tutor qualifications, liability coverage, financial reporting, and student improvement reporting. 25 Yet, the guidelines and criteria are ambiguous and oversight is weak. Since the arrival of SES tutoring in LAUSD, a substantial number of private providers have entered the SES market offering significant variation in curriculum, tutoring session length, tutoring approaches, tutor qualifications, and other characteristics. 25

California Code of Regulations. â&#x20AC;&#x153;Supplemental Educational Services.â&#x20AC;? Title 5, sec. 13075 (2011). 14

The number of SES vendors fluctuated initially, but has grown steadily since, reaching 56 providers in 2011-2012. Some small tutoring providers have entered and exited the market within the year due to low enrollment, while others have expanded their market share quickly. Researchers have found substantial year-to-year fluctuations complicate state efforts to identify quality providers and terminate providers that fail to increase student academic achievement. 26 Figure 2 shows that the numbers of applicants varies significantly from provider to provider. Notably, the largest five tutoring providers have attracted 61% of the total applicants, whereas the other 51 providers share the remaining 39% of applicants. Some small providers only attract several applicants. Several of the largest providers give free computers to students, suggesting that a significant fraction of provider competition centers around marketing and gifts. 27 Figure 2. Enrollment by Provider in School Year 2011-2012 20000

No.1, 19268

18000 16000 14000

No.2, 12798

12000 Number of 10000 Students 8000 6000

No.5, 5699

No.3, 7524

No.4, 7234

4000 2000 0 Enrollment by Provider Data source: This figure is generated from the provider information in school year 2011-2012 provided by Beyond the Bell Branch.

SES tutoring cannot improve achievement if students fail to attend the program. Because SES is

Carolyn J. Heinrich and Patricia Burch, “The Implementation and Effectiveness of Supplemental Educational Services: A Review and Recommendations for Program Improvement.” (Prepared for “Tightening Up Title I”, a conference sponsored jointly by the Center for American Progress and the American Enterprise Institute, Washington, DC, March 11, 2011), 11.

LAUSD officials believed that free computers and aggressive marketing were driving enrollment for some providers. Additionally, we regressed the enrollment data by provider on a set of provider characteristic variables and found a positive correlation between the number of applicants and the computer/internet provided variables, significant at the .05 level. 15

a voluntary service, participating students are not required to attend all the tutoring hours that providers offer. Figure 3 shows the program completion rates by SES provider in 2010-2011. The red line shows that the average completion rate of all tutoring programs was 87.2%. This figure implies that on average students attended most of the offered tutoring hours; however, we note that completion rates varied greatly across providers. Because we could not interview parents, we could not determine why some providers had higher attendance than others. High attendance could be a function of perceived effectiveness: families that believe the program is effective may be more likely to make a continued effort to send their child. Some providers offer gifts, most impressively laptops, to students that complete the program with high attendance. These incentives and rewards may be driving attendance, potentially at the expense of achievement. Figure 3. Completion Rates by SES Provider in School Year 2010-2011 120% 100% 80% 60% 40% 20% 0% % Students completed

Average

Data Source: Beyond the Bell Branch Website. http://btb.lausd.net/library/index.php. Note: Providers with over 100% service received additional slots from providers with the lowest attendance service percentages

Given the above analysis, we can observe that the capacities of SES providers in attracting applicants and providing effective tutoring vary significantly. LAUSD can outperform private providers in the local SES market by focusing on succesful practices in curriculum, learning configuration, instructor qualifications, and dosage.

SES TUTORING PRACTICES IN LAUSD Based on the 2011-2012 provider information supplied by Beyond the Bell and our phone interviews with SES providers, we describe current SES tutoring practices in LAUSD. These data show common practices, which serve as a weak-though-helpful guide in predicting parent preferences. First, SES providers favor two of the four types of learning configurations permitted by law: one-to-one tutoring and small-group tutoring (2-5 students) are offered more than group 16

instruction (6-10 students) and online instruction. In Figure 4, the blue bars represent the proportion of SES providers that offer each kind of tutoring approach while the red bars represent the proportion of those that do not. In LAUSD, 91.07% offer one-to-one tutoring, 67% offer small group tutoring, and only 12.5% offer group and online instruction. Second, we turn to learning location. Figure 5 reveals that 86% of all SES providers offer tutoring services at the studentâ&#x20AC;&#x2122;s home; 84% in public locations, such as library or community; and only 32% in the providerâ&#x20AC;&#x2122;s own learning center. Third, in terms of tutoring subject, almost all providers offer both ELA and math tutoring while only 20% provide tutoring services in science. Fourth, SES providers offer a mean of 30 hours with a standard deviation of 5.5 hours; the range is between 20 and 40 hours in a school year. Typically, students are offered 2 to 4 tutoring session per week with sessions lasting one to two hours. Figure 4. Main Tutoring Approaches

91.07%

Figure 5. Learning Locations

67.86% 12.50% 12.50%

Yes

85.71%

Figure 6. Tutoring Subject

High School Students 27%

Yes

19.64% Reading

Math

Public Library

Provider Center

Figure 7. Students Served by Schooling Level

98.21%

Yes 32.14%

In-home

94.64%

83.93%

Science

Middle School Students 21%

Element ary Students 52%

Data source: all the descriptive statistics were generated from the enrollment data and provider information in school year 2011-2012 provided by Beyond the Bell Branch. Note: The number of SES providers in school year 2011-2012 is 56.

One of the most important factors in determining SES effectiveness is tutor hiring. We learned about the practices of tutor hiring from the phone interviews we conducted with SES providers. Most SES providers hire tutors with associate degrees or college graduates, though some recruit credentialed teachers. To assess the qualifications of prospective tutors, SES providers usually conduct an interview and background check. Some providers require new tutors to take a subject test and participate in training. The pay rate for tutors depends on qualifications. On average, payment is $15/hr for college graduate tutors and $25/hr for credentialed teachers. Again, common practices and market rates offer a useful guide for LAUSD as in determining the type of tutors to employ.

Evaluation Criteria This analysis examines several tutoring program design dimensions in detail in the following section. The main criteria by which we evaluate those dimensions are enumerated below. • Effectiveness at improving student learning. Effectiveness is typically measured by test score data. Because funding is fixed for each student, a less expensive program that serves more students is not possible. This provision allows us to focus on maximizing per-pupil growth without worrying that we could increase aggregate achievement by spreading the money across more students. • Cost. We consider the costs of the various program dimensions. While we cannot always estimate a specific cost, we do note which option is more expensive and by how much wherever possible. • Parent/student preference. Given that an internal district tutoring program will compete with private providers for students, families must actively select the district program for it to provide benefits. 28 As a result, the program must appeal to parents and students. • Legality. In this context, legality primarily refers to federal and state legislation; however, it is worth considering certain recommendations may be untenable even if they are not formally illegal. For example, for liability reasons LAUSD will not permit teachers to schedule sessions at students’ homes, even if it is an effective strategy for attracting parents and ensuring high attendance. 29  Political stakes. While most design variations have minimal political implications, a small subset of choices could cause political problems. For example, should the district provide services to certain areas or specific grades only, other members of the 28

It is possible that the entrance of the district will force private providers to improve the effectiveness of their offerings to compete with the district. It is therefore theoretically possible, though not likely, for benefits to accrue to students who remain with private providers.

LAUSD officials, meeting, March 8. 18

community may feel slighted and contact education board members or other representatives.

Analysis of Tutoring Program Design Dimensions The core objective of this project is to identify the most effective template for establishing an internal tutoring program administered by LAUSD. Here we explain and analyze the key design elements of a tutoring program, and recommend a practice or set of practices for each dimension. The program elements include students served, course subject, curriculum, learning configuration, instructor training/qualifications, dosage, and location. We consider program components as independently as possible; however, both in research and practice components interact. In some cases, once we have settled on a particular recommendation, we do not fully consider program designs for alternative options. For example, since we determined that LAUSD should focus on elementary students, we do not evaluate tutoring programs for middle and high school students. This winnowing of policy options was employed to narrow the scope of this paper, both for the authors and reader. Even so, the majority of key policy options are covered herein.

STUDENTS SERVED Recommendation: Given the difficulties in starting a quality, large-scale tutoring program, LAUSD should initially focus on one group of students. We recommend beginning with elementary students primarily because elementary enrollment is much higher. This means any increase in per-pupil improvement is multiplied by a larger number. Students eligible for SES tutoring span across all grade levels; however, a tutoring program must employ different programmatic elements for different grade levels. Skills (and skill gaps) vary considerably by age and grade level. Likewise, a given design component may differentially affect students in different grades. For example, 1st and 2nd graders may respond differently to online learning than high school students. We restrict our analysis to elementary school students for reasons we discuss in depth below. The primary factors determining our focus on elementary school students are rates of application for SES tutoring and attendance rates once accepted, as well as grade level effects. We also briefly explore whether there are likely differences for English Language Learners (ELL) or special education students. The first factor is application and attendance rates. Elementary students are disproportionately inclined to apply for and attend SES programs. Table 2 presents the demographics of LAUSD 19

students eligible for SES services in school year 2009-2010. In focusing on age brackets, we find elementary students (grades 1-5) were over-represented in both categories of applicants and attendees relative to middle and high-school students. Elementary students make up for 38.6% of all eligible students, but 56.6% of applicants and 60% of attendees were elementary students. 30 Because elementary students account for over half of SES students, specializing in the market for elementary students allows LAUSD to pursue a significant segment of students receiving SES tutoring without designing programs for very different grade levels. Table 2. Demographics of LAUSD Students Eligible for SES, 2009-10

Source: Melissa K. Barnhart. (2011). The Impact of Participation in Supplemental Educational Services (SES) on Student Achievement: 2009-10. Los Angeles, CA: Los Angeles Unified School District, Program and Evaluation and Research Branch, Planning, Assessment and Research Division.

Another pattern worth noting is that English language learners and special education students

Barnhart, â&#x20AC;&#x153;Impact 2009-10,â&#x20AC;? 4. 20

were overrepresented among SES users. ELL students comprised 32.8% of all eligible students, yet 41.1% of those who applied and 40.9% of those who attended were designated ELL. Only 27.3% of students who attended SES have never required ELL assistance. Students with disabilities constituted 16.1% of attending students. Both groups comprise significant percentages of students likely to receive LAUSD SES tutoring. The second factor of importance is grade-level effects. LAUSD tutoring should focus not just on the largest number of students it can attract, but also the degree to which it can improve their academic achievement. Ideally, we would look at the achievement of students at each grade-level receiving private SES tutoring in LAUSD. We would then compare this baseline to likely gradelevel achievement in an improved LAUSD program. Unfortunately, projecting grade-level gains in a non-existent LAUSD program is impossible. Instead, we use enrollment figures to demonstrate that achievement gains for older students would need to be much larger than for younger students to achieve the same aggregate gains in student achievement. Current SES achievement is quite low across age groups. A recent evaluation by LAUSD shows very low achievement gains at each age group. 31 Typically, researchers studying SES tutoring have found larger achievement effects for elementary school students though there is certainly disagreement within the research community. 32 Unfortunately, it is very difficult to compare across grades. Larger gains in older grades may be caused by differences in motivation. As Robert Tagorda of Long Beach Unified pointed out, younger students have less agency and attend tutoring at their parents’ behest. 33 Older students can choose whether or not to participate. We would expect that older students who choose to participate come with greater motivation than average. Higher scores for older students may not indicate superior tutoring programs, but instead greater average student motivation. Given the small achievement gains in all cases, we can treat the baseline achievement in each grade level as approximately equal. Determining the potential for achievement gains based on the literature is extremely difficult. Luckily, because of the significant differences in attendance rates, smaller elementary school

Barnhart, “Impact 2009-10,” 12.

Barnhart, “Impact 2009-10,” 9.

Rickles and White, “Impact of Participation,” 8. Chicago Public Schools, “SES Tutoring Programs: An evaluation of year 3 in the Chicago Public Schools,” (Office of Research, Evaluation, and Accountability; Office of Extended Learning Opportunities, Chicago, IL, February2007), 10, Figure 5. Jordan H. Rickles and Melissa K. Barnhart, “The Impact of Supplemental Educational Services Participation on Student Achievement: 2005-06,” (Los Angeles Unified School District. Program an Evaluation and Research Branch, Planning, Assessment and Research Division, Los Angeles, CA, May 2011), 8. 33

Robert Tagorda, Interview by the authors, February 13, 2012. 21

effect sizes result in a much larger aggregate impact (assuming LAUSD can attract a similar proportion of students at each grade level). 34 Given enrollment differences, a high school program would need to be five times as effective as an elementary program to have a comparable aggregate impact. 35 A middle school program would need to be over twice as effective. While we have not examined every possible intervention, we are not aware of middle or high school programs that can provide such a large relative benefit. Not only would tutoring need to provide larger benefits for older students, a large body of research shows interventions become less cost-effective as children age. 36 One reason is that young children are simply more cognitively malleable than their older peers. Additionally, students who fall behind early tend to fall further behind as they age. 37 Failing to learn multiplication tables correctly or read efficiently makes future educational achievement more difficult and less likely. As a result, focusing on elementary school may make sense even if the immediate achievement gains are smaller. None of this means that LAUSD should not, in the future, consider providing SES tutoring to middle school or high school students. An examination of attendance figures suggests middle school would be more productive than high school; however, neither option should be disregarded entirely. We do recommend that should LAUSD decide to build the program incrementally, it should focus on elementary school students initially. While we focus on elementary school students as a group, LAUSD could choose to focus on a subset of elementary school students. We consider elementary school as a single unit for convenience. In LAUSD, most elementary schools serve K-5th grade, although some include 6th graders. While we see little reason to assist only a subset of grades, research examining tutoring outside of SES shows much larger effects in early elementary grades. 38 In keeping with our

If LAUSD’s entrance in the SES market changed the mix of students applying for SES tutoring, increasing the number of middle or high school students, this would mean smaller achievement effects could result in comparable aggregate achievement gains; however, we have no reason to believe this scenario is likely.

Calculation: (Number of elementary students)/(Number of high school students)

James J. Heckman, “Skill Formation and the Economics of Investing in Disadvantaged Children,” Science 312, no. 5782 (June 30, 2006): 1900-1902.

Gary W. Ritter et al., “The Effectiveness of Volunteer Tutoring Programs for Elementary and Middle School Students: A MetaAnalysis,” Review of Educational Research 27, no. 1 (Spring, 2009): 15.

Batya Elbaum et. al., “How Effective Are One-to-One Tutoring Programs in Reading for Elementary Students at Risk for Reading Failure?: A Meta-Analysis of the Intervention Research,” Journal of Educational Psychology 92, no. 4 (2000), 614.

Lauer et. al., “Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students,” Review of Educational Research 76, no. 2 (Summer 2006): 296, 302. 22

recommendation, this analysis will focus on possible designs of an elementary school tutoring program. A very large fraction of LAUSD students are English Language Learners (ELL). According to the California Department of Education, in 2009-10, 26,180 students, or half of LAUSD first graders were designated ELL. 39 ELL students are overrepresented in SES tutoring and, in the past, were more likely to use LAUSD tutoring services. We examined several research reviews to see if there is any reason to believe ELL students require special interventions. While research is certainly not conclusive, the answer appears to be no. The area where research indicates English learners may benefit from different instruction is in curricular focus, an area we do not tackle. 40 Additionally, students with very low English proficiency may need instructors who speak their first language. A LAUSD tutoring program will serve a meaningful portion of special education students. Students with special needs vary considerably in the kinds of assistance they need. Due to lack of expertise, we do not consider students with learning disabilities in our analysis; however, we do recommend the district carefully consider where the needs of special education students differ from their peers and accommodate accordingly.

COURSE SUBJECT Recommendation: We recommend that LAUSD focus on both math and ELA instruction and not science instruction. Science enrollment in the elementary grades is negligible and both math and ELA can offer significant potential benefits to LAUSD students. SES tutoring may cover three subjects: English Language Arts (ELA), mathematics, or science. In LAUSD and elementary school specifically, science enrollment is negligible. LAUSD must then consider whether to focus on math or ELA, or to provide tutoring in both areas. There is fairly consistent evidence that math is easier to teach and tutor. NAEP trend scores indicate that the past 30 years of public policy have improved math scores considerably more than reading scores for nine-year-olds. 41 In SES tutoring literature, we find similar differences,

Data Report from CDE.gov, Source: California Longitudinal Pupil Achievement Data System (CALPADS), accessed: 201010-22

Claude Goldenberg, “Teaching English Language Learners,” American Educator (Summer 2008): 14.

Bobby D Rampey et al., “NAEP 2008 Trends in Academic Progress,” (United States Department of Education. National Center for Education Statistics, Institute of Education Sciences., Washington, DC, 2009), 9, 29. 23

with math effects larger than reading effects. 42 For example, a meta-analysis of SES tutoring found a mean weighted math effect size (0.043) larger than that of reading (0.017), even though both effects were quite small. 43 Moreover, there is evidence that LAUSD has previously constructed a successful math tutoring program. A 2006 evaluation by LAUSD showed that a discontinued Beyond the Bell tutoring program produced notable positive effects in math: elementary math students with high attendance performed .32 standard deviations above those who applied but failed to attend (0.2 standard deviations improvement is considered meaningful in education). 44 This means that just over 62% of students who did not receive tutoring would be below average in the group that received tutoring. 45 Given previous experience and the improved record of interventions in math, there is reason to believe LAUSD will have greater success tutoring math than reading. Even so, we do not recommend LAUSD focus only on math. Improving math achievement is not obviously more important than improving reading achievement. Additionally, enrollment in ELA among elementary-school students is significant: in 2009-10, nearly three quarters of students in grades 1-5 who attended SES tutoring received ELA assistance. This means that an elementary ELA program that is more effective than what private providers offer may produce aggregate academic benefits and capture sizable revenue. Given LAUSD’s record and the broader record of supplemental early reading interventions, there is reason to believe LAUSD can outperform current SES providers. The same 2006 evaluation that showed large benefits for high-attending elementary school students in math showed much smaller, but still positive gains for comparable ELA students (.07 standard deviations). While LAUSD has an unimpressive record of providing effective elementary ELA tutoring assistance, certainly effective programs exist. Interventions that successfully improve reading outcomes among low-achieving children have produced remarkably large gains, often significantly

Experimental designs tend to find larger effects for ELA, suggesting that high-quality ELA programs have great potential, but are more difficult to implement.

Chappell et al. “Provider Effects on Student Achievement,” 8, Table 3.

While this effect could be due to differences in motivation, students with high attendance at the LAUSD program significantly outperformed their high-attendance peers at private providers. Similarly, while many private providers showed positive effects for students who did not attend, LAUSD only had positive effects for students who attended the program, and only meaningful effects for high attenders.

Rickles and White, “Impact of Participation,” 8. 45

Robert Coe, “It’s the Effect Size, Stupid: What Effect Size is and why it is Important,” (University of Durham School of Education, Presented at the Annual Conference of British Educational Research Association, England, September, 2002). 24

narrowing the gap between low-achieving readers and their peers. 46 Conversely, LAUSD evaluations and broader research have found SES providers offer very little, if anything, in terms of boosting ELA achievement. 47 It follows that LAUSD may still reasonably aspire to offer an ELA program that outperforms the average SES program. Given the secondary goal of capturing revenue and the efficiencies of offering broader services to a similar population, LAUSD should only focus on math tutoring if it sees benefit in managing a smaller program, at least initially. As a result, this analysis will consider both ELA and math instruction.

CURRICULUM Recommendation: None A curriculum, in this context, refers to the content students are taught and the teaching strategies employed within each subject. For example, a second-grade reading curriculum could prioritize gaps in reading speed or reading comprehension. Likewise, the curriculum could use instructional techniques stressing inferred or explicit learning. Different subjects (reading and math) necessarily require varying curricula as do different grade levels. SES tutoring requires the use of a set, research-based curriculum. 48 While choice of curricula is clearly very important and can influence the effects of dosage, training/qualifications, and other dimensions we will explore, we have chosen not to evaluate curricula, primarily due to a lack of expertise. We do advise the district take care in selecting a curriculum and ensure that in implementation, tutors do not revert to offering homework help.

LEARNING CONFIGURATION Recommendation: We tentatively recommend one-to-one tutoring for ELA and small-group tutoring for mathematics. The research indicates stronger effects for one-to-one tutoring in ELA. There is less convincing evidence for either approach in math and given the expense of one-toone tutoring, we recommend small-group instruction, though we believe LAUSD should consider groups of two or three students preferable to groups of four or five students. Learning configuration refers to the grouping structures under which SES tutors may educate students. The legally permissible methods include one-to-one tutoring, small group tutoring (2-5 46

George Farkus, “Reading One-to-One: An Intensive Program Serving a Great Many Students While Still Achieving Large Effects,” in Social Programs that Work, ed. Jonathon Crane, (New York: Russell Sage Foundation, 1998), 78. Reading Recovery’ Proactive Reading

Chappell et al. “Provider Effects on Student Achievement,” 8, Table 3.

LAUSD officials, meeting, March 8. 25

students), group instruction (6-10 students) and online instruction (typically individual or small group). Different learning configurations may result in dissimilar academic outcomes and may vary in appeal to parents. We primarily consider one-to-one tutoring and small-group tutoring. We disregard large-group and online tutoring for several reasons. First, an examination of provider offerings suggests families prefer one-to-one tutoring or small group tutoring to large-group instruction. While we were unable to ask families directly, very few providers offer large-group instruction and few families select providers offering large-group instruction. 49 While this decision could be for logistical reasons, we expect this arrangement is partially in response to parental demand; given parent interest in small class sizes, it is difficult to imagine parents view large group tutoring as preferable to small group or individualized instruction. Likewise, finding groups of 6-10 students with matching skill gaps introduces logistical challenges and requires tutoring at centers or schools with large SES enrollment. Lastly, evidence for large-group assistance of elementary students is weak. 50 Online instruction has two basic problems. First, online instruction requires that students have access to computers and the Internet. Since many students do not own computers and young students cannot easily troubleshoot technical problems, online instruction would be difficult. Second, all of the online providers in LAUSD offer free computers to students with perfect attendance. LAUSD is unwilling to compete with providers by offering expensive gifts to students. 51 Research suggests, though not definitively, that one-to-one tutoring is superior to small-group instruction. Many of the largest effect sizes, especially in reading, appear in one-to-one interventions. 52 One study that looks at variations on Reading Recovery, an intensive earlyelementary literacy program, found one-to-one tutoring was a “necessary but not sufficient” condition for large improvements. 53 A meta-analysis looking at out-of-school time interventions among all grades found much larger effects for one-to-one reading programs than large-group programs or those that used variations in arrangement. 54 Additionally, experts like George

Data compiled from phone interviews with private SES providers

Lauer et. al., “Out-of-School-Time,” 296, 302.

LAUSD Officials, meeting, March 8

Robert E. Slavin and Barbara A. Wasik, “Preventing Early Reading Failure with One-to-One Tutoring: A Review of Five Programs,” Reading Research Quarterly 28, no. 2 (April/May/June, 1993), 182.

Gay Su Pinnell et al., “Comparing Instructional Models for Literacy Education of High Risk First Graders,” Reading Research Quarterly , Vol. 29, No. 1 (Jan. - Feb. - Mar., 1994), 31.

Lauer et. al., “Out-of-School-Time,” 306. 26

Farkus argue that one-to-one reading instruction is the only proven intervention for lowachieving readers. 55 Other evidence suggests small-group instruction works just as well as individual tutoring. A meta-analysis of SES tutoring found very small and no difference between individual and smallgroup instruction, for reading and math respectively. 56 The out-of-school time meta-analysis that found larger effects for one-to-one reading programs, found very little difference between math programs using one-to-one tutoring (.22) and group instruction (.18), defined in the study as 10 students or fewer. 57 And one implementation of Reading Recovery and a similar high-quality program called Proactive Reading found significant benefits for expert instructors in small groups. 58 While the research is inconclusive, it suggests that one-to-one tutoring more often delivers large academic benefits. The argument in favor of one-to-one tutoring is stronger for ELA instruction. Given the absence of evidence showing larger effects for one-to-one tutoring in math, we tentatively conclude that small-group and individual tutoring may be similarly effective in mathematics instruction. A key consideration within small-group tutoring is that not all group sizes are necessarily equally beneficial. George Farkus and Rachel Durhan note that there is no useful research on ideal smallgroup size. 59 Yet, groups of two and five are almost certainly different, even though both are classified as “small group.” Given the difficulty of assembling a group of students with similar skill gaps and the likely negative impacts of dividing attention among multiple students, we recommend proceeding with caution with groups larger than three students. We are unable to ascertain if families prefer one-to-one tutoring. While more programs offer oneto-one tutoring, small-group tutoring is also quite common. The modest disparity could be for scheduling or other reasons, and we were unable to contact families to ask them directly what they prefer.

Farkus, “Reading One-to-One,” 78.

Chappell et al. “Provider Effects on Student Achievement,” p. 9, 10.

Lauer et. al., “Out-of-School-Time,” 302.

Patricia Mathes et al., “The Effects of Theoretically Different Instruction and Student Characteristics on the Skills of Struggling Readers,” Reading Research Quarterly 40, no. 2 (April/May/June 2005), p.156, 169-174.

Rachel E. Durham and George Farkas, “The Role of Tutoring in Standards-Based Reform,” in Standards Based Reform and the Poverty Gap: Lessons for No Child Left Behind, ed. Adam Gamoran (Washington DC: Brookings Institute Press, 2007), 226. 27

Because of cost, small-group tutoring does not need to outperform individual tutoring to be preferable. Small group tutoring is much cheaper as one instructor assists between two and five students simultaneously. In most situations, this extra money could be spent assisting other students. Given that each voucher is allocated to a single student, spreading the money across more students is not possible; however, the money may be spent in other ways that improve the impact of SES tutoring on student achievement. The two most expensive areas influencing student achievement are labor costs and dosage, which are discussed below. The important question we cannot decisively answer is whether a given dollar is better spent on individual instruction or some combination of more expensive tutors and greater dosage.

INSTRUCTOR TRAINING/QUALIFICATIONS Recommendation: LAUSD should hire college students or teachers’ aides for one-to-one instruction (ELA only if our learning configuration recommendation is followed). While highlytrained, expert teachers produce the largest effects in ELA, college students do quite well. Given that LAUSD cannot choose the best teachers due to the collective bargaining agreement, selecting the best college students or teachers’ aides at a lower price seems preferable. Where teachers offer the largest advantage is in student management, a skill most relevant for smallgroup instruction; therefore, we recommend utilizing certified teachers or teachers’ aides for small-group math instruction. Tutor quality impacts tutoring effectiveness and potentially program attractiveness. We have identified three categories of tutors: certified teacher, teacher’s aide, and college student. While LAUSD could also use paraprofessional tutors, they may not be available in sufficient numbers and should be thought of as similar to college students. In our interview with tutoring expert Tiffani Chin, she identified three important categories of tutor skill: content knowledge, imparting knowledge, and student management. 60 Content knowledge is a prerequisite for effective instruction. According to Chin, tutoring often requires a particularly thorough command of content as the tutor must quickly diagnose a student’s area of confusion and adapt the lesson accordingly. Chin also argues the process of imparting knowledge is a little different than classroom instruction so both certified teachers and college students require training. Lastly, the importance of student management depends on the learning configuration. One-to-one tutoring requires little experience, yet small-group instruction demands some real ability to control a group of children.

Tiffany Chin, Interview conducted by the authors, February 22, 2012. 28

Credentialed/Certified teachers signal the highest quality tutor, but may or may not deliver on that promise. The signal of a credentialed teacher may be meaningful to parents looking to select an SES program; however, in terms of specific skills they may not be the most cost-effective choice. While teachers likely have better-than-average content knowledge, the union contract forbids selecting among teachers on ability. 61 Instead, tutoring is treated as an extra-work opportunity to be distributed based on factors like seniority. Moreover, the teachers with this yearâ&#x20AC;&#x2122;s extra-work opportunity move to the back of the line next year, 62 suggesting that the SES program will not benefit from staff stability. While seniority may be at least a weak proxy for content knowledge, because we can test and select knowledgeable members from the other categories of tutors, we do not assume credentialed teachers have any advantage on content knowledge. Many would assume that teachers are particularly skilled at instruction. Yet Chin notes that imparting knowledge through tutoring is somewhat different than teaching. While teachers may have some advantage, it may not be particularly pronounced. Where classroom teachers really stand out is in student management. If LAUSD uses small-group instruction, having a credentialed instructor may offer significant, though unquantifiable, benefits. Logistically, teachers are particularly suited for school-site instruction; they are located at the school site and could easily stay at school for another hour or so to tutor. Unfortunately, teachers also cost a good deal more than other tutors, with a projected average cost of $70 per hour. 63 College students offer a very different set of opportunities and costs. College students can be tested to ensure content knowledge. While novice instructors, according to Chin they are often fresher, having spent the day outside of a classroom. With sufficient training, she says, they can provide quality instruction. Where college students come up short is in student management. Unlike credentialed teachers, college students often have difficulty managing even a few students simultaneously, especially initially. Logistically, college students are difficult for LAUSD to employ. Because college students are paid significantly less (an average of $15 per hour) and come from different locations, they need more than one hour at a time to justify the expense and effort of transportation. In addition, we could find no data that would permit us to estimate the supply of qualified college students. If LAUSD needs hundreds of qualified tutors, they may be difficult to find at current prices. Moreover, LAUSD would need to process newly-hired college students through the LAUSD human resources system, incurring some additional cost. 64 Teachersâ&#x20AC;&#x2122; aides offer something of a middle ground. While often less than expert in content knowledge, aides can be tested to ensure proficiency. Like college students they need training in

LAUSD officials, meeting, March 8.

LAUSD officials, meeting, March 8. 29

imparting instruction. Like certified teachers, they have some experience managing children, though much less. While their pay is also set by union contract, the wage is much lower than for certified teachers, approximately between $11.50-$23 depending on experience and specific duties. 65 Logistically, teachers’ aides are conveniently located on school campuses, but perhaps not in sufficient numbers to operate an entire program. Overall, the available effectiveness research shows that uncredentialed instructors can be effective elementary reading tutors. A meta-analysis of elementary reading programs found that college students outperformed paraprofessionals who, in turn, outperformed teachers, with effect sizes of 1.65, .68, and .36 respectively. 66 A separate study of college-student tutors (not screened for ability, but trained for 33 hours) showed they improved elementary reading scores comparable to high-quality programs. 67 68 Confirming the notion that formal credentials are not necessary is a meta-analysis of volunteer elementary reading tutoring that finds a more modest, but meaningful overall effect of 0.23. 69 Conversely, the meta-analysis of SES provider effects found that tutors with college degrees slightly outperformed those without degrees, though this analysis includes older students who may need tutors with greater content knowledge. Hiring a teacher may be particularly advantageous if tutors are not tested for content knowledge or trained in imparting instruction, a common practice among private SES providers. Some of the most effective programs, such as Reading Recovery and Proactive Reading, do use certified instructors; however, those programs have other components that are difficult to replicate. Most prominently, those programs require extensive training of instructors – 42 hours in one implementation. 70 This, coupled with the evidence of comparable effects among welltrained college students suggests extensive training may be an important component. Additionally, these certified instructors were likely not chosen by seniority, but instead selected for their commitment and competence, as demonstrated by their willingness to undergo extensive training. Moreover these programs all offered significant dosages of one-to-one tutoring, 71 an impossibility for LAUSD’s SES program if utilizing certified instructors.

LAUSD official, email to the authors, March 15, 2012.

Effect sizes measured using Cohen’s D.; Elbaumet. al., “One-to-one Reading for Elementary,” 614.

Jill Fitzgerald, “Can Minimally Trained College Student Volunteers Help Young At-Risk children to Read Better?” Reading Research Quarterly 35, no. 1 (2001) 39, 44.

This study includes over 40 classrooms yet fails to follow nesting procedures in the analysis, likely resulting in undersized standard errors. Given the results were highly significant, we still rely on these results, though do so with reservations.

Effect size measured using Hedges G.; Ritter et al., “Effectiveness of Volunteer Tutoring,” 15.

Jason L. Anthony et al., “The Effects of Theoretically Different Instruction and Student Characteristics on the Skills of Struggling Readers,” Reading Research Quarterly 40, no. 2, (2005): 156.

Slavin and Wasik, “Preventing Early Reading Failure,” 182. 30

If parents see certification as a signal of quality, using non-certified instructors may reduce the number of students selecting the LAUSD SES program. We question this assumption given that only a small fraction of tutoring firms primarily hire credentialed teachers. If families do prefer certified instructors and decline to enroll in LAUSD’s program, LAUSD will capture less revenue. It may also negatively affect aggregate student achievement if those students remain in SES but choose a less-effective program. Yet parents may also see the number of tutoring hours offered (preferring a program above the mean of 30 hours) or one-to-one instruction as important signals of quality. LAUSD cannot afford to offer a program with all three signals of quality. Because we were unable to collect good data on parent preferences, we cannot say which signal parents prize most. LAUSD may need to conduct additional market research to ascertain how families will react to variation in these key program elements. Selecting non-union workers over certified teachers or aides could incur some political costs. The teachers’ union will dislike the decision and may contact the district or the board of education to protest. The union may be less upset if the district hires teachers’ aides, as they are also unionized district employees. Any recommendation must consider the impact on academic achievement, the significant cost differences, a possible impact on parent preferences and thus enrollment, and any potential political difficulties. Given the very high cost of using certified instructors, there is no way to offer even close to 30 hours of one-to-one instruction using certified teachers, a minimum number of hours we explore in the dosage section. 72 Any program using certified teachers will have to utilize small-group instruction. Thus the comparison is between three options: smallgroup tutoring with an uncertified instructor, small-group tutoring with an unselected certified instructor, or one-to-one tutoring with an uncertified instructor. As a result we recommend using uncertified tutors in a one-to-one setting for elementary reading, where evidence in favor of a one-to-one approach is strongest. Given the absence of research on the impact of using teachers or non-teachers in elementary math instruction, the value that teachers provide in terms of student management, the political benefits of hiring teachers, and the limited evidence in favor of small-group math instruction, we make only a contingent recommendation: if LAUSD chooses small-group mathematics instruction, LAUSD should employ certified instructors and/or teachers’ aides. LAUSD will also incur costs training tutors. We cannot say if each type of tutor requires the same amount of training; however research and discussions with Tiffany Chin indicate both

As we explain later, dropping below the provider average of 30 hours will very likely reduce enrollment and academic achievement. 31

groups require training. 73 Indeed, if extensive training is a key factor in program success, training may be an important element of a program. Tutors may receive training upon hiring or periodically as they tutor. We could find no research varying the length of tutor training within a range considered viable for this program. We use ten hour trainings in our models, though LAUSD may choose a different length of time.

DOSAGE Recommendation: We recommend a session length between ½ and one hour, meeting at least 2-3 times per week, lasting between 10 and 35 weeks. High frequency is a greater priority for ELA instruction. Dosage refers to the quantity of tutoring each student receives and includes multiple components. First, there is the length of tutoring session. Second, one must consider frequency, or the number of sessions per week. Third, programs can vary in duration, or the number of weeks a program spans. All of these features are interrelated: given a fixed cost constraint, longer sessions mean either fewer sessions per week or a program lasting fewer months. Additionally, this dimension interacts with instructor qualifications and learning configuration through cost. Session length is in many ways the simplest component. In practice, elementary school tutoring typically lasts one hour and very rarely exceeds 1.5 hours, primarily because younger children lack the attention span needed for extended sessions. 74 This figure gives us a fairly hard upper bound on session length. While we found no research exploring the ideal session lengths, the meta-analysis of volunteer tutoring programs included session length figures for each study in the evaluation. The analysis included 15 studies with sessions between 15 and 30 minutes and 11 studies with sessions lasting between 45 minutes and one hour. 75 A review of several highquality programs showed a range from 15-30 minutes. 76 SES tutoring primarily lasts one hour to make the travel time worthwhile for tutors and students. Given the transportation costs for both students and instructors, and the amount of time lost when students transition in and out of tutoring, we do not believe sessions shorter than 30 minutes are feasible. Because shorter sessions are employed in high quality programs and elementary attention spans may diminish after a half hour (after an entire school day), we prefer 30 minute sessions assuming that means twice as many total sessions. Below we consider both 30 minute and hour long sessions.

Elbaum et. al., “One-to-one Reading for Elementary,” 614.

Interviews with SES providers, February 17-24, 2012

Ritter et al., “Effectiveness of Volunteer Tutoring,” 14.

Slavin and Wasik, “Preventing Early Reading Failure,” 182. 32

Before examining the number of sessions per week, we explore total program hours. There is some evidence that after a certain point, increasing the number of hours tends not to improve outcomes. The meta-analysis of elementary reading tutoring found that programs of less than twenty weeks duration significantly outperformed those lasting longer than twenty weeks. 77 Likewise the meta-analysis of out-of-school-time found larger math effects for programs between 46-75 and 76-100 hours than those less or more. 78 The same study identified larger ELA effects for programs between 44 and 84 hours with little difference between longer and shorter programs. While we wonder if those longer programs suffer from weaknesses other than simply length, we might take this as evidence of diminishing returns: as a program gets longer, the marginal benefits of tutoring shrink. Indeed, this result makes sense: if some students have benefitted from tutoring and caught up with their peers, continued tutoring may offer them reduced benefits. The final consideration is the number of sessions per week or the intensity/duration tradeoff. Limited research indicates that more intensive programs have larger effects in reading though there is no corresponding guidance for math programs. The meta-analysis of elementary school reading programs found programs of less than 50 hours total are slightly preferable to programs lasting from 50-150 hours. 79 As noted above, the same study found that programs lasting less than 20 weeks outperform longer programs. As the authors note, this implies intensity is preferable to duration, given a fixed number of hours. This finding is consistent with the research on high quality interventions that meet very frequently. While high-frequency, shorter duration programs appear preferable on average, it is possible this result masks some heterogeneity in benefits. Tiffany Chin argues that struggling students can be divided into two groups: those that have fallen behind and students struggling in the classroom. Those who have fallen behind are not learning in the classroom because they lag far behind their peers and cannot understand the day’s lesson. Chin believes students who have fallen behind benefit from intensive catch-up. Other students are not learning in the classroom, either because they have difficulty in that setting or their teacher is doing a poor job of reaching them. Those students will continue to struggle in the classroom, at least for the remainder of the year, and will benefit most from ongoing assistance. While the former group needs intensive catch up, the latter group may profit more from fewer session stretched over a longer period of time. While it may be possible to identify members of each group from a pre-test, as Chin suggests, we have seen no demonstration of this sorting capacity, either in the literature or in practice. As a result, while we

Elbaum et. al., “One-to-one Reading for Elementary,” 614.

Lauer et. al., “Out-of-School-Time,” 302.

Elbaum et al., “One-to-one Reading for Elementary,” 614. 33

think this distinction merits consideration, we do not recommend designing a program around attempted differentiation without further evidence. Considering a few plausible variations in dosage may help in selecting a preferable option. Looking at provider data for 2011-12, we find the mean and mode number of hours offered is approximately 30 hours per student annually, with a standard deviation of 5.5 hours. 80 Based on our discussions with LAUSD, offering below 30 hours appears inadvisable given parent interest in hours offered. 81 If LAUSD offers the mean number of hours, that allows for 30-60 sessions depending on session length. The chart below shows how varying the number of minutes per session and sessions per week changes the number of weeks of instruction. For example, two 30minute sessions per week results in 30 weeks of instruction. Table 3. Number of Weeks of Instruction (varying by length of session and # of sessions per week)

30 minute sessions

60 minute sessions

1 session/week

60 weeks

30 weeks

2 sessions/week

30 weeks

15 weeks

3 sessions/week

20 weeks

10 weeks

4 sessions/week

15 weeks

7.5 weeks

5 sessions/week

12 weeks

6 weeks

While schools vary in their academic calendars, California currently has a 180-day or 36-week academic year. 82 A program lasting longer than the school year is impractical, eliminating the 60-week option. Given the research on intensiveness, we would not choose a program that meets only once per week. Conversely, the programs that meet four or five times a week would be difficult to staff, requiring either a very large number of instructors or instructors who come daily. Similarly, students have other commitments that will be difficult to accommodate if sessions meet 4-5 times per week. We eliminate these options due to anticipated logistical constraints, which leaves us a smaller menu of choices:

Data set provided to the authors by LAUSD

LAUSD officials, meeting, March 8.

Sharon Noguchi, â&#x20AC;&#x153;California Could Cut School Year by Several Weeks,â&#x20AC;? San Jose Mercury News (May 13, 2011) http://budgetrealities.lausd.net/sites/default/files/San%20Jose%20Mercury%20News516_0.pdf 34

Table 4. Number of Weeks of Instruction (varying by length of session and # of sessions per week)

30 minute sessions

60 minute sessions

1 session/week

60 weeks

30 weeks

2 sessions/week

30 weeks

15 weeks

3 sessions/week

20 weeks

10 weeks

4 sessions/week

15 weeks

7.5 weeks

5 sessions/week

12 weeks

6 weeks

Table 5 presents a version of the same menu of options assuming LAUSD can provide 40 hours. Table 5. Number of Weeks of Instruction, 40 hours (varying by length of session and # of sessions per week)

30 minute sessions

60 minute sessions

1 session/week

80 weeks

40 weeks

2 sessions/week

40 weeks

20 weeks

3 sessions/week

26.7 weeks

13.3 weeks

4 sessions/week

20 weeks

10 weeks

5 sessions/week

16 weeks

8 weeks

Seeking a balance between intensity and program length recommends the above set of four options. Choosing effectively among those options will likely hinge on logistical decisions. Again, should LAUSD tutoring cost significantly more or less than the average cost of approximately $58 per hour, the available options may change. 83 Given current estimates, we recommend a program no fewer than 10 weeks, no longer than 30 weeks, which meets at least two to three times per week, with high frequency considered a greater priority for ELA instruction.

VARYING DOSAGE MODELS Because the design elements do not exist in isolation, it is important to examine how different design components interact. Below, Figures 9, 10, and 11 illustrate how much tutoring LAUSD can provide

The $58 per hour figures is obtained by dividing the mean number of hours (30) into the 2011-12 voucher amount ($1,733);

California Department of Education. â&#x20AC;&#x153;Supplemental Educational Services (SES) 2011-12 Per Pupil Amount.â&#x20AC;? Posted August 19, 2011. http://www.cde.ca.gov/ta/ac/ti/documents/sesppa2011.xls Accessed on March 3, 2012. 35

given different selections of instructor, learning configuration, and session length. In each case, the final number column codes plausible options in black, low dosage options have a strikethrough, while implausibly high options are in white with a strikethrough. Implausibly-high options present the opportunity to reallocate some portion of funds to other areas of expenditure, such as tutor training.

The values in the models are built on assumptions derived from the information we collected. We fix labor costs at 34% of the $1,733 voucher, based on our interview with the anonymous midsized district, a figure consistent with private provider expenditures. If LAUSD uses a higher percentage of the voucher on wages, these calculations would change. Based on LAUSD projections, we treat a teacher as costing the district $70/hour, an aide as $19/hour, and a college student as $15/hour. College students are not considered for five-student groups. The figures show how tutor qualifications can determine the other program components. Certified instructors cannot provide one-to-one instruction. Even with 30-minute sessions, using certified teachers limits program duration under 11 weeks unless groups are composed of more than three students. Conversely, hiring teacherâ&#x20AC;&#x2122;s aides or college students for 30 minute sessions allows for a reasonable duration of one-to-one tutoring. If uncredentialed instructors lead groups, there is little reason to offer shorter sessions.

Figure 8. Horizontal Tree of Interactive Dimension: Teacher

Learning configuration

Session Length

# of sessions after 10 hours training

30 min

Not Possible

60 min

Not Possible

30 min

60 min

30 min

20.4

60 min

10.7

# of Weeks (3 sessions per week)

1:1

Teacher

small group (3 students)

small group (5 students)

Figure 9. Horizontal Tree of Interactive Dimension: Teacherâ&#x20AC;&#x2122;s Aide

Learning configuration

Session Length

# of sessions after 10 hours training

30 min

60 min

30 min

166

55.3

60 min

27.7

30 min

290

96.6

60 min

145

48.3

# of Weeks (3 sessions per week)

1:1

Teacher's Aide

small group (3 students)

small group (5 students)

Figure 10. Horizontal Tree of Interactive Dimension: College Student

Learning configuration

Session Length

# of sessions after 10 hours training

# of Weeks

30 min

19.3

60 min

9.7

30 min

214

71.3

60 min

107

35.7

(3 sessions per week)

1:1 College Students small group (3 students)

LEARNING SITE LOCATION Recommendation: LAUSD should offer on-site tutoring services at every school that serves students who enroll in the district program. Private tutoring companies typically offer education services at the child’s home, a local library, or at a learning center. Possible tutoring locations for LAUSD might include the student’s home, a public library, centrally-located school sites, or on-site, meaning at the enrolled child’s school. LAUSD has rejected home visits for liability reasons 84 and we find no advantage to using a public library over a school campus. The decision to pursue on school-site tutoring versus a center-based approach depends primarily on logistical considerations. LAUSD has many SES participants. Internal enrollment data provided reveals 44,375 elementary school applicants in 2011-12. LAUSD also encompasses a very large territory, totaling 710 square miles of land. As such, one logistical consideration is the distribution of applicants throughout the district. We performed a geographic analysis of the number of elementary school SES applicants by zip code, the results of which can be viewed in Figure 11. The analysis revealed high concentrations of applicants within zip codes in three clusters: (1) the central northern region of the San Fernando Valley, including Pacoima and neighboring communities (2) the harbor area, including Wilmington and San Pedro, and (3) the southeast region, including East Los Angeles, South Gate, South Los Angeles and neighboring communities. Additionally, there is considerable variation in density at the school level. Within highly concentrated zip codes, 25% of schools serve 72% of SES applicants. Moreover, of the 186 elementary schools within highly concentrated zip codes, 50% served fewer than 15 SES applicants. Using a center-based model could be cost-inefficient, given the large number of schools with low concentrations of SES applicants. Busing students to centralized locations costs LAUSD the same amount of money if a bus transports 30 or 3 students. If LAUSD does not offer transportation while using a center-based model, some parents would turn away from the LAUSD program to private providers with more convenient location options. Thus, we recommend having the LAUSD SES program at each respective school site that serves students who enroll in the program. This would simplify transportation logistics and eliminate 84

LAUSD officials, meeting, March 8. 38

Figure 11. Concentration of Elementary SES Applicants in LAUSD

the need for inefficient busing arrangements. It would also help ensure that each student’s voucher would be allocated toward important programmatic elements such as dosage and curriculum development, rather than a cost-inefficient transportation model. Finally, it would reduce parental concern over their children arriving safely at an off-site location, and eliminate travel for teachers and aides who work at a different site during the normal school day.

Implementation Considerations MARKET SHARE To establish a program cost estimate we must project the market share of LAUSD's program. A US Department of Education study of SES in several districts that received an NCLB waiver similar to the one LAUSD is pursuing is instructive. Among districts that implemented a new inhouse SES program, there was a clear inverse relationship between the size of the district and the initial share of SES participants. 85 That is, as the number of SES participants increased the percentage of SES participants served by the in-house programs decreased. The largest of these was Florida's Hillsborough County school district, which captured 16% market share in its first year of operation, constituting 944 out of 6,028 participants. Because the number of LAUSD applicants is much higher than in Hillsborough, we expect its initial market share of SES participants would be somewhat lower. Thus, the Hillsborough market share of 16% serves as the maximum SES market share LAUSD should expect in its initial year of operation.

COST ESTIMATE The SES system in LAUSD currently operates at capacity. In 2009-10, only 34,526 students, 67% of all applicants, were provided with SES. 86 While the system capacity is based upon LAUSD’s annual Title I allocation, we cannot know what the allocation will be in the future. We therefore use the most recent known capacity of 34,526 students as a projection. 87 Elementary school students composed 51.24% of this year’s SES applicants. If we assume the share of elementary enrollment will remain stable, and that the proportion of elementary applicants approximates elementary enrollment, we would anticipate 17,692 elementary school attendees for 2012-13. Projecting 16% market share, the expected number of participants in LAUSD's 85

Andrea Berger et al., “Supplemental Educational Services and Student Achievement in Five Waiver Districts,” (submitted to US Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, by researchers from American Institutes for Research and RTI International, Washington DC, 2011), 15.

Barnhart. “Impact 2009-10.”, 2

Barnhart. “Impact 2009-10.”, 2 40

program in the first year of operation is no more than 2,831 students. Using last year's Title I per pupil allocation of $1,733 per student, the expected total annual budget for the LAUSD program would be approximately $4,906,123 in its first year of operation. This figure assumes that no other funding streams are utilized besides the Title I per pupil allocation for SES. It also assumes perfect attendance, as the vouchers only reimburse monies for tutoring services actually performed.

STAFF CONSIDERATIONS A large expense for this program will be tutor wages. Our anonymous contact in a mid-sized school district that received an NCLB waiver notes her district allocates about one third of its annual SES budget to tutor wages. 88 This is analogous to what the private SES providers currently spend on wages. 89 Therefore, we use a wage allocation of 34% of the projected annual budget, or $1,668,0812 in tutor wages. Data collected from private SES providers yielded an average wage of $15 per hour for college students. Both teachers and teachersâ&#x20AC;&#x2122; aides are unionized, and entitled to their normal hourly rate for extended learning programs like SES, per contractual agreement. LAUSD officials informed us that teachersâ&#x20AC;&#x2122; aide compensation can range from $11.50 to $23 per hour, depending on qualifications. Because the program wages need to be competitive with private providers to ensure adequate personnel, we restrict this range to $15 to $23 per hour for aides, yielding an average wage of $19 per hour. LAUSD teachers receive $60 per hour on average, including benefits. Because the teacher contract specifies that teachers must be hired by seniority, LAUSD estimates an average teacher wage of $70 per hour, which we adopt. 90 We have made separate recommendations for learning configurations between math and ELA, and thus separate subject costs are considered. LAUSD students are placed in either math or ELA tutoring based on student need as measured by California Standards Test (CST) scores. Students may not receive SES tutoring in both subjects. The landscape analysis shows that while participation in both ELA and math has increased steadily over the past decade, ELA participation has grown more rapidly. The latest available year, 2009-10, saw an ELA share of 62.7% and a math share of 37.3%. The increase in ELA participation has differed radically from year to year, making projections difficult. We use the 2009-10 shares for prediction, noting that the ELA share could be higher than 62.7%. Table 6 summarizes the parameters derived from the above sections: 88

Anonymous official at midsized school district, email to the authors, March 16, 2012.

Data compiled from phone interviews with private SES providers

LAUSD officials, meeting with the authors, March 8, 2012. 41

Table 6. Tutor Wage Parameters Parameter Students Served: Wage Budget:

Value 2,831 $1,668,081.82

ELA Share:

62.7%

Math Share:

37.3%

College Student Wage:

$15.00

Teachers' Aide Wage:

$19.00

Teacher Wage:

$70.00

Having established the parameters for tutor wages, we explore how LAUSD might staff the program in the context of our recommendations for instructor qualifications, learning configuration and dosage. Model I – 30 Total Hours, Reading 1:1, Math 1:3 Given 30 hours per student annually, we project 84,930 total student hours, 53,234 student hours in ELA, and 31,696 student hours in math. We recommend one-to-three tutoring for math, making one hour of math instruction equivalent to three student hours. Therefore, we project approximately 10,565 instructional hours for the math component of the program. We recommend a one-to-one configuration for ELA, making student hours and instructional hours equivalent. ELA takes up 83.4% of instructional hours and math comprises 16.6%. Since students are assigned to a subject based on need, $1,391,844 should be allocated to ELA wages and $276,237 allocated to math wages, based on projected enrollment. Math tutors will be teachers and/or teachers’ aides, as we do not recommend college students for one-to-three configuration. With the wages in Table 6 and a math budget of $276,237, we calculate that teachers should work approximately 14% of instructional hours for math. 91 Any larger share and the math component would be too expensive for the projected budget. This allows for a cadre of certified instructors to oversee teachers’ aides or work with students with special education needs. We recommend utilizing college students and teachers’ aides for ELA tutoring. Any combination is feasible, since the cost of using aides exclusively is $1,011,448, well within the suggested budget of $1,391,844. This means that it is feasible to use some teachers for ELA, despite our recommendation to prioritize aides and college students. A combination of teachers and aides yields a maximum teacher share of 14% for ELA instructional hours, and a combination of

For all algebraic cost calculations, please consult Appendix E. 42

teachers and college students yields a maximum 20% share. Therefore, using a combination of all three instructor types, the teacher share could be anything between 14% and 20% of instructional hours. Model II â&#x20AC;&#x201C; 40 Total Hours, Reading 1:1, Math 1:3 We recalculate the model with the same parameters using 40 hours per student annually. This results in 14,087 instructional hours for math and 70,979 instructional hours for ELA, and identical budget shares to Model I. Given a math budget of $276,237, the maximum teacher share for math instruction is only 1%, meaning the district could use very few teachers. Given an ELA budget of $1,391,844, any combination of college students and teachersâ&#x20AC;&#x2122; aides is feasible. As in Model I, LAUSD could use teachers for ELA, with a teacher share of 1.2% to 9%. Table 7 summarizes the above cost models: Table 7. Staffing Cost Models Model I

Model II

Hours per Student

Total Student Hours

84,930

113,240

Math Student Hours

31,696

42,261

Math Instructional Hours 10,565

14,087

ELA Instructional Hours

53,234

70,979

ELA Budget

$1,391,844.63

ELA Teacher Share

14% - 20%

1.2% - 9%

Math Budget

$276,237.19

Math Teacher Share

<=14%

<=1%

The above models suggest that as total hours increase, teachers become less feasible to utilize. If LAUSD seeks to compete based on using credentialed and experienced teachers, Model I is preferable. If LAUSD seeks to compete based on offering more hours than the private providers, Model II is preferred.

PROGRAMMATIC COSTS With wages capped at 34% of the annual budget, this leaves $3,238,041 for other programmatic expenses. These include curriculum development, staff training, college student and teacher aid skills assessments, site coordinator wages, site materials, marketing, and incentives. While we lack the necessary information to cost out these expenses at the time of writing, we submit them as subjects of further consideration.

POLITICAL CONSIDERATIONS Scale Offering the in-house program to all eligible elementary students throughout LAUSD students represents the politically safe option. Our geographic analysis revealed high concentrations of elementary SES applicants in three clusters within LAUSD. Beyond the Bell could launch with a targeted pilot program that only serves the schools with the highest need for SES. This would reduce the number of site coordinators required, simplifying oversight without having to allocate money for transportation for a centralized location scheme. Even so, some local district superintendents or board members will dislike that the in-house program is not offered in their jurisdictions. In anticipation, Beyond the Bell could promise that the program will be scaled up after the first year of operation. While politically risky, we present this option for consideration, given the potential programmatic advantages. Provider Reaction Private SES providers will not be pleased with an in-house LASUD tutoring program. The LAUSD program will deplete their customer base and offer salaries for teachers that providers cannot match. We anticipate that the private providers will aggressively market against the inhouse program; however, this would be true in any district in which a new in-house program is established, and so does not affect our projection of a 16% market maximum. Miramonte On January 30th, 2012 a teacher at Miramonte Elementary School was charged with committing lewd acts upon his students. The Miramonte scandal makes child safety, already a high priority for LAUSD, a very salient concern. We recommend the district carefully implement proper safety procedures, including an open-door rule, administrator walk-bys and, where appropriate, multiple sessions in each room so that no student is left alone with a single teacher. This will allow for an effective learning configuration while reducing the risk of an incident.

Conclusion Based on a comprehensive review of the relative literatures, the experience of experts, and a basic analysis of available data, this project provides instructive guidance in the design of a new LAUSD-run SES tutoring program. We utilized a landscape analysis of the current SES market, a literature review of best practices, a quantitative analysis of enrollment data, and a geographical analysis of applicants to construct recommendations for seven design dimensions: â&#x20AC;˘ First, for the students served dimension, we recommend that the program be initially offered to elementary school students to optimize attendance and effectiveness.

• •

Second, for the course subject dimension, we recommend offering math and ELA tutoring. Third, for the curriculum dimension, we advise LAUSD that an effective curriculum is very important, but we lack the expertise to perform a sound analysis or provide a recommendation. Fourth, for the learning configuration dimension, we recommend utilizing one-to-one tutoring for ELA and small group tutoring for math as a cost-effective balance. Fifth, for the teacher qualifications dimension, we recommend employing college students and teacher’s aides for one-to-one configurations, and teacher’s aide’s or teachers for small group configurations to balance classroom management skills with cost. Sixth, for the dosage dimension, we recommend a program between 10 and 35 weeks, as a balance of intensiveness and administrative burden. Seventh, for the location dimension, we recommend that the program be implemented at all district schools where students apply for SES to simplify transportation logistics and reduce costs.

While we intend these findings to assist LAUSD in answering challenging program design questions, the analysis also raises a number of considerations for the future. First, LAUSD establishes priority enrollment for SES primarily through California Standardized Test (CST) scores, with the first CST examinations given in the third grade. Because there are no CST scores for K-2nd grade students, LAUSD treats these students as low priority regardless of academic ability unless the student is designated SPED or ELL. This practice neglects the age group that is most cost-effective to assist. We urge LAUSD to reconsider this policy, and to adopt a method for identifying K-2 students in need of SES tutoring. Second, serious consideration must be given to students with learning disabilities. Students with special learning needs may require alternative curricula or otherwise differentiated instruction. Indeed, each child enrolled in special education is required to have an Individualized Education Program that specifies how their individual learning needs should be met. 92 Again, we lack the expertise to fully address this concern, but we caution LAUSD to carefully consider the needs of students with learning disabilities. Finally, state regulations allow SES providers to offer incentives for enrollment. Providers may offer incentives worth no greater than $50 except for “computers or other technical equipment

United States Department of Education, “A Closer Look at the IEP,” (Department of Education website, Washington DC, 2007). Accessed 3/20/12 <http://www2.ed.gov/parents/needs/speced/iepguide/index.html#closer> 45

used as the primary instructional tool for the delivery of SES and given to a student after he or she completes the program.â&#x20AC;? 93 This provision allows private providers to attract students with free laptops in return for high attendance. Some providers recruit students with free computers, undermining the program rationale that providers would compete by offering convenient and quality tutoring services. We observe the impact of this provision in the spike in attendance in the 2007-08 school year, the year the regulation was put into effect. 94 In order to ensure that SES remains oriented toward serving students pursuing academic assistance and not free laptops, we urge the California Department of Education to modify this regulation to eliminate the exception for computers.

93 94

California Code of Regulations. â&#x20AC;&#x153;Enrolling Students.â&#x20AC;? Title 5, sec. 13075.9(b) (2011).

LAUSD officials, meeting with the authors, March 8, 2012. 46

References Barnhart, Melissa K. “The Impact of Participation in Supplemental Educational Services (SES) on Student Achievement: 2009-10.” Los Angeles Unified School District. Program an Evaluation and Research Branch. Planning, Assessment and Research Division., Los Angeles, CA, May 2011. Accessed March 18, 2012. <http://btb.lausd.net/library/files/SES/Impact%20of%20SES%202009-10.pdf> Berger, Andrea, Juliette-Marie deSousa, Gur Hoshen, Stephanie Lampron, Kerstin Carlson Le Floch, Megan Petroccia, and Jamie Shkolnik. “Supplemental Educational Services and Student Achievement in Five Waiver Districts.” Submitted to US Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, by researchers from American Institutes for Research and RTI International, Washington DC, 2011. Accessed March 18, 2012. <http://www2.ed.gov/rschstat/eval/disadv/ses-waiver/ses-waiver-report.pdf> Burch, Patricia, Matthew Steinberg and Joseph Donovan. “Supplemental Educational Services and NCLB: Policy Assumptions, Market Practices, Emerging Issues.” Educational Evaluation and Policy Analysis 29, no. 2 June, 2007. DOI: 10.3102/0162373707302035 California Department of Education. “Supplemental Educational Services (SES) 2011-12 Per Pupil Amount.” Posted August 19, 2011. http://www.cde.ca.gov/ta/ac/ti/documents/sesppa2011.xls Chappell, Shanan, John Nunnery, Shana Pribesh and Jane Hager. “A Meta-Analysis of Supplemental Educational Services (SES) Provider Effects on Student Achievement.” Journal of Education for Students Placed at Risk 16, no. 1 (2011): 1-23. DOI:10.1080/10824669.2011.554140 Chicago Public Schools. “SES Tutoring Programs: An evaluation of year 3 in the Chicago Public Schools.” Office of Research, Evaluation, and Accountability; Office of Extended Learning Opportunities, Chicago, IL, February, 2007. Accessed March 18, 2012. http://sesiq2.wceruw.org/documents/chicago_ses.pdf “Child Nutrition Programs – Income Eligibility Guidelines.” Federal Register 72, no. 58 (March 25, 2011): 16724, U.S. National Archives and Records Administration. Accessed March 15, 2012 http://www.fns.usda.gov/cnd/governance/notices/iegs/IEGs11-12.pdf

Chin, Tiffani. Interview by Logan Contreras, Greg Srolestar, and Chen Wang. February 22, 2012. Coe, Robert. “It’s the Effect Size, Stupid: What Effect Size is and why it is Important.” University of Durham School of Education, Presented at the Annual Conference of British Educational Research Association, England, September, 2002. http://www.leeds.ac.uk/educol/documents/00002182.htm Deasy, John E. “Letter to Parents/Guardians of Students in the Los Angeles Unified School District.” Letter distributed to parents of LAUSD students, Los Angeles, September, 2011. Accessed February 8, 2012. http://families.lausd.net/sites/families.lausd.net/files/Document1.pdf Durham, Rachel E., and George Farkas. “The Role of Tutoring in Standards-Based Reform.” In Standards Based Reform and the Poverty Gap: Lessons for No Child Left Behind, edited by Adam Gamoran, 201-228. Washington DC: Brookings Institute Press, 2007. Elbaum, Batya, Sharon Vaughn, Marie Tejero Hughes, Sally Watson Moody. “How Effective Are One-to-One Tutoring Programs in Reading for Elementary Students at Risk for Reading Failure?: A Meta-Analysis of the Intervention Research.” Journal of Educational Psychology 92, no. 4 (2000): 605-619. DOI: 10.1037//0022-0663.92.4.605 Farkus, George. “Reading One-to-One: An Intensive Program Serving a Great Many Students While Still Achieving Large Effects.” In Social Programs that Work, edited by Jonathon Crane, 75-109. New York: Russell Sage Foundation, 1998. Fitzgerald, Jill. “Can Minimally Trained College Student Volunteers Help Young At-Risk children to Read Better?” Reading Research Quarterly 35, no. 1 (2001): 28-46. Accessed March 18, 2012. http://www.jstor.org/stable/748126 Goldenberg, Claude. “Teaching English Language Learners,” American Educator (Summer 2008): 8-23, 42-44. Accessed March 18, 2012. http://www.edweek.org/media/ell_final.pdf Heckman, James J. “Skill Formation and the Economics of Investing in Disadvantaged Children.” Science 312, no. 5782 (June 30, 2006): 1900-1902. DOI: 10.1126/science.1128898 Heinrich, Carol A. and Patricia Burch, “The Implementation and Effectiveness of Supplemental Educational Services: A Review and Recommendations for Program Improvement.”

Prepared for “Tightening Up Title I”, a conference sponsored jointly by the Center for American Progress and the American Enterprise Institute, Washington, DC, March 11, 2011. Accessed March 18, 2012. http://www.americanprogress.org/events/2011/03/av/ses_implementation.pdf Lauer, Patricia A., Motoko Akiba, Stephanie B. Wilkerson, Helen S. Apthorp, David Snow, and Mya L. Martin-Glenn. “Out-of-School-Time Programs: A Meta-Analysis of Effects for At-Risk Students.” Review of Educational Research 76, no. 2 (Summer 2006): 275-313 DOI: 10.3102/00346543076002275 Los Angeles Unified School District. Beyond the Bell Branch. “Free Tutoring Supplemental Educational Services.” Los Angeles, CA, 2009. Accessed February 8, 2012. http://www.btb.lausd.net/SES/ Los Angeles Unified School District. Beyond the Bell Branch. Intervention Staff Member. Email to Greg Srolestar. March 16, 2012. Los Angeles Unified School District. Beyond the Bell Branch. Supplemental Educational Services Staff. Meeting with Logan Contreras, Greg Srolestar, and Chen Wang. March 8, 2012. Los Angeles Unified School District. Office of Communications. “Fingertip Facts 2011-2012.” Los Angeles, CA, 2011. Accessed February 8, 2012. <http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/LAUSDNET/OFFICES/CO MMUNICATIONS/COMMUNICATIONS_FACTS/1112FINGERTIPFACT_SCNOV.8.PDF> Mathes, Patricia G., Carolyn A. Denton, Jack M. Fletcher, Jason L. Anthony, David J. Francis, and Christopher Schatschneider. “The Effects of Theoretically Different Instruction and Student Characteristics on the Skills of Struggling Readers.” Reading Research Quarterly 40, no. 2 (April/May/June 2005): 148-182. doi:10.1598/RRQ.40.2.2 Noguchi, Sharon. “California Could Cut School Year by Several Weeks.” San Jose Mercury News (May 13, 2011). Accessed March 18, 2012. Pinnell, Gay Su, Carol A. Lyons, Diane E. DeFord, Anthony S. Byrk, and Michael Seltzer. “Comparing Instructional Models for Literacy Education of High Risk First Graders.” Reading Research Quarterly 29, no. 1 (January/February/March, 1994): 8-29. http://www.jstor.org/stable/747736

<http://budgetrealities.lausd.net/sites/default/files/San%20Jose%20Mercury%20News51 6_0.pdf> Rampey, Bobby D., Gloria S. Dion, and Patricia L. Donahue. “NAEP 2008 Trends in Academic Progress.” United States Department of Education. National Center for Education Statistics, Institute of Education Sciences. Washington, DC, 2009. Accessed March 18, 2012. http://nces.ed.gov/nationsreportcard/pdf/main2008/2009479.pdf Rickles, Jordan H., and Jeffrey A. White. “The impact of supplemental educational services participation on student achievement.” Los Angeles, CA: Los Angeles Unified School District. Program and Evaluation and Research Branch. Planning, Assessment and Research Division., Los Angeles, CA, January 30, 2006. Accessed March 18, 2012. <http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/F LDR_PLCY_RES_DEV/PAR_DIVISION_MAIN/RESEARCH_UNIT/PUBLICATIONS /POLICY_REPORTS/SUPPLEMENTAL%20EDUCATIONAL%20SERVICES%20REP ORT.PDF> Rickles, Jordan H., and Melissa K. Barnhart. “The Impact of Supplemental Educational Services Participation on Student Achievement: 2005-06.” Los Angeles Unified School District. Program an Evaluation and Research Branch. Planning, Assessment and Research Division., Los Angeles, CA, May 2011. Accessed March 18, 2012. <http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/F LDR_PLCY_RES_DEV/PAR_DIVISION_MAIN/RESEARCH_UNIT/PUBLICATIONS /CONFERENCE_PRESENTATIONS/RICKLESBARNHARTGUALPA_SESOVERFIV EYEARS_AERAPAPER_2008.PDF > Rickles, Jordan H., Melissa K. Barnhart, and Alice S. Gualpa. “Supplemental Educational Services Participation and Impact on Student Achievement: The Case of One Urban District over Five Years.” Paper presented at the Annual Meeting of the American Educational Research Association Symposium: The Effectiveness of Supplemental Educational Services, New York City, NY, March 25, 2008. Accessed Match 18, 2012. <http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/FLDR_ORGANIZATIONS/F LDR_PLCY_RES_DEV/PAR_DIVISION_MAIN/RESEARCH_UNIT/PUBLICATIONS /CONFERENCE_PRESENTATIONS/RICKLESBARNHARTGUALPA_SESOVERFIV EYEARS_AERAPAPER_2008.PDF> Ritter, Gary W., Joshua H. Barnett, George S. Denny, and Ginger R. Albin. “The Effectiveness of

Volunteer Tutoring Programs for Elementary and Middle School Students: A MetaAnalysis.” Review of Educational Research 27, no. 1 (Spring, 2009): 3-38. DOI: 10.3102/0034654308325690 “SEA responsibilities for supplemental educational services.” Code of Federal Regulations. US National Archives and Records Administration, Washington DC, 2011. Slavin, Robert E. and Barbara A. Wasik. “Preventing Early Reading Failure with One-to-One Tutoring: A Review of Five Programs.” Reading Research Quarterly 28, no. 2 (April/May/June, 1993): 178-200 Accessed March 12, 2012. http://www.jstor.org/stable/747888 Snyder, Thomas D., and Sally A. Dillow. “Digest of Education Statistics 2010.” United States Department of Education. National Center for Education Statistics, Institute of Educational Sciences. (Washington D.C.: NCES, 2011), 138, Table 94. Accessed February 8, 2012. http://nces.ed.gov/pubs2011/2011015.pdf. “Supplemental Educational Services.” California Code of Regulations. Title 5, sec (2011). California Office of Administrative Law. Tagorda, Robert. Interview by Greg Srolestar and Logan Contreras. February 13, 2012. United States Department of Education. “A Closer Look at the IEP.” Department of Education website, Washington DC, 2007. Accessed 3/20/12 <http://www2.ed.gov/parents/needs/speced/iepguide/index.html#closer> United States Department of Education. “Fiscal Year 2011 Title I Grants to Local Education Agencies – California.” Washington DC, 2011. Accessed March 19, 2012. http://www2.ed.gov/about/overview/budget/titlei/fy11/california.pdf United States Department of Education. “Non-Regulatory Guidance on Title I, Part A Waivers.” Washington, DC, July 2009. Accessed March 18, 2012. <www2.ed.gov/programs/titleiparta/title-i-waiver.doc> United States Department of Education. “Supplemental Educational Services: Nonregulatory Guidance.” Washington, DC, 2009. http://www2.ed.gov/policy/elsec/guid/suppsvcsguid.pdf. Accessed: February 9, 2012

Appendix A: List of Acronyms AYP – Adequate Yearly Progress BTB – Beyond the Bell Branch CST – California Standardized Test ELA – English Language Art ELL – English Language Learner ESEA – Elementary and Secondary Education Act of 1965 IEP – Individualized Education Program LAUSD – Los Angeles Unified School District LEA – Local Educational Agency NCLB – No Child Left Behind Act of 2001 PI – Program Improvement SEA – State Educational Agency SES – Supplemental Education Services SPED – Special Education

Appendix B: LAUSD Administrative Structure

Source: Los Angeles Unified School District. Accessed: March 18, 2012.

http://notebook.lausd.net/pls/ptl/docs/PAGE/CA_LAUSD/LAUSDNET/ABOUT_US/INFORMA TION/LAUSD%20OVERALL%20--%2011-1-2011.PDF

Appendix C: Supplemental Figures and Tables Cohenâ&#x20AC;&#x2122;s D:

Hedgeâ&#x20AC;&#x2122;s G (pooled standard deviation):

Equations taken from UCLA Stat Computing:

http://www.ats.ucla.edu/stat/mult_pkg/faq/general/effect_size_power/effect_size_power.htm Figure 12. The Eligible Students in LAUSD, School Year 2003-2010 900000 800000 700000 600000 500000 Number of students 400000 300000 200000 100000

Numbe of Students The number of eligible students

Note: Since LAUSD did not conduct evaluation study in school year 2008-2009, so enrollment data for 08/09 could not be obtained.

Figure 13. The Enrollment Rates of SES Programs in LAUSD, School Year 2003-2010

60000 50000 40000 Number of students 30000

Applied Attended

20000

Attended ELA

10000

Attended Math

Note: Since LAUSD did not conduct evaluation study in school year 2008-2009, so enrollment data for 08/09 could not be obtained. Table 8. LAUSD SES Data 2005-2011 Year

05/06

06/07

07/08

08/09

09/10

10/11

Number of Eligible

173

200

235

280

302

343

303,140

310,544

331,275

354,652

382,555

406,121

Number of Providers

Application Deadlines

June 30,

June 29,

June 30,

June 30, 2010

2005 Sept.

2006 Aug.

2007 Aug.

2008 Aug.

2009 Aug.

Aug. 31, 2010

30, 2005

31, 2006

31, 2007

29, 2008

31, 2009

Oct. 15, 2010

Nov. 15,

Oct. 20,

Oct. 19,

Oct. 17,

Oct. 16,

2005

2006

2007

2008

2009

48.50%

62.10%

78.30%

90%

92.80%

98.76

9/05-6/06

9/06-6/07

9/07-6/08

9/08-

9/09-5/10

9/10-5/11

Program Improvement (PI) Schools Number of Students Eligible for SES

Percent Service completed Service Period

5/09 Data Source: Beyond the Bell Branch Website. http://btb.lausd.net/library/index.php.

Table 9. SES Utilization Data 2003-2010 Year

03/04

04/05

05/06

06/07

07/08

08/09

09/10

Number of Students

778437

779267

769231

749918

723047

704806

681974

Eligible

163472

171439

223077

292468

303680

380595

381905

Applied

9808

18858

26769

32171

36442

41865

49648

Attended

6179

13012

17132

19946

26602

30562

33264

Attended ELA

5296

8298

9905

9973

14941

18002

20852

ELA High Attendance

1165

1577

2080

1895

1793

12782

18558

ELA Medium Attendance

2013

2821

4160

1596

1195

3060

1251

ELA Low Attendance

2119

3900

3665

6483

11953

1980

1043

Attended Math

981

4526

6960

9973

11661

12560

12412

Math High Attendance

304

1403

2158

3092

3615

3893

3848

Math Medium Attendance

196

1403

2088

1496

1166

2261

745

Math Low Attendance

481

2172

3132

6483

8629

1633

869

Data source: Data collected from multiple LAUSD studies: The impact of supplemental educational services participation on student achievement. Los Angeles, CA: LAUSD, Program and Evaluation and Research Branch, Planning, Assessment and Research Division. Low Attendance=1% to 49% of program hours; Medium Attendance=50% to 89% of program hours; High Attendance=90% to 100% of program hours.

Note: 1.

Since LAUSD did not conduct evaluation study in school year 2008-2009, so enrollment data for 08/09 could not be obtained.

Low Attendance=1% to 49% of program hours Medium Attendance=50% to 89% of program hours High Attendance=90% to 100% of program hours

Appendix D: SES Providers Survey Questionnaire 1.

What subjects do you tutor? (warm-up question) a. Math b. Reading c. Science

What tutoring approaches do you use? a. One-to-one tutoring b. Small-group tutoring c. Group instruction d. Online tutoring

Where do your sessions take place? a. at the childâ&#x20AC;&#x2122;s home b. at the tutoring center c. another public location (i.e. public library) d. Other (please specify)_______

How long are sessions for math and reading respectively on average? (KEY QUESTION) a. Math:

_________ hour(s)

b. Reading: __________hour(s) c. Science (if applicable): _________ hour(s)

As a parent, why should I send my child to your tutoring program? What do you think are the main features of your tutoring program? Check all that apply. (KEY QUESTION) a. Effective tutoring approaches (i.e. mainly offer one-to-one tutoring/small-group tutoring) b. Instructor training/qualifications (i.e. degreed tutors/experienced teachers/provide initial and on-going training for tutors) c.

Well-designed curriculum

Dosage

Convenient tutoring location

Free gift offered (i.e. free laptop)

Other (please specify)______

Do you ask new students to take a diagnostic test? a.

Yes

b. No

Do you focus on curriculum or homework help? a. Curriculum b. Homework

assistance

c. Both

What qualifications do your tutors typically posses? (KEY QUESTION) a. High school graduates b. Associateâ&#x20AC;&#x2122;s degree holders c. College graduates d. Experienced

teachers

e. Other ___________

How do you assess the qualifications of your new tutors?

10. What is your companyâ&#x20AC;&#x2122;s payment rate for tutors?

11. Have you made any major changes to your program so as to improve the quality of tutoring services in recent years?

Basic Information (for interviewer to record) 1.

Provider Name: ____________________.

Location: _________________________.

How long has you company been working with LAUSD to provide the SES services? ___________ years.

Appendix E: Staff Cost Calculations Model I - Math Let x = Teacher Hours Let y = Teachers’ Aide Hours x + y = 10,565 hours y = 10,565 – x … y = 10,565 – 1480.435 y = 9084.565 hours

$70x + $19y = $276,237.19 … $70x + $19(10,565 – x) = $276,237.19 $70x + 200,735 -$19x = $276,237.19 $51x = $75502.19 x = 1480.435 hours

Model I – ELA Let x = Teacher Hours Let y = Teachers’ Aide Hours x + y = 53,234 hours y = 53,234 – x … y = 53,234 – 7,458.8 y = 45,775.2 hours Let z = College Student Hours x + z = 53,234 hours z = 53,234 – x … z = 53,234 – 10,787.902 z = 42,446.1 hours

$70x + $19y = $1,391,844.63 … $70x + $19(53,234 – x) = $1,391,844.63 $70x + $1,011,446 -$19x = $1,391,844.63 $51x = $380398.63 x = 7,458.8 hours $70x + $15z = $1,391,844.63 … $70x + $15(53,234 – x) = $1,391,844.63 $70x + $798510 - $15 = $1,391,844.63 $55x = 593,334.63 x = 10,787.902 hours

Model II – Math Let x = Teacher Hours Let y = Teachers’ Aide Hours x + y = 14,087 hours y = 14,087 – x … y = 14,087 – 168.32 y = 13,918.68 hours Model II – ELA x + y = 70,979 hours y = 70,979 – x … y = 70,979 – 847.9 y = 70,131.1 hours Let z = College Student Hours x + z = 70,979 hours z = 70,979 – x … z = 70,979 – 6,214.89 z = 64,561.11 hours

$70x + $19y = $276,237.19 … $70x + $19(14,087 – x) = $276,237.19 $70x + $267,653 -$19x = $276,237.19 $51x = 8584.19 x = 168.32 hours $70x + $19y = $1,391,844.63 … $70x + $19(70,979 – x) = $1,391,844.63 $70x + $1,348,601 -$19x = $1,391,844.63 $51x = 43,243.63 x = 847.9 hours $70x + $15z = $1,391,844.63 … $70x + $15(70,979 – x) = $1,391,844.63 $70x + $1,064,685 -$19x = $1,391,844.63 $51x = 327,159.63 x = 6,414.89 hours