Year One Fuller Report

FOUNDING BOARD MEMBER Maconda B. O’Connor, Ph.D. The Brown Foundation EXECUTIVE COMMITTEE CHAIR Bill Crouch Continental Airlines, Inc. PRESIDENT Gasper Mir, III MFR, P.C. TREASURER Thomas L. Elsenbrook Alvarez & Marsal Business Consulting, LLC SECRETARY Yava D. Scott Siebert Brandford Shank & Co., LLC BOARD MEMBERS Susan A. Bischoff Houston Public Library Foundation Jonathan Day Andrews Kurth LLP Bolivar Fraga Neighborhood Centers, Inc. Roberto Gonzalez Employment & Training Centers, Inc. Shawn Gross SAJG Investments, Inc. Scott McClelland H-E-B Karol Musher, M.A., CCC-SLP Texas Children’s Hospital

February 20, 2012

Dear Challenge Network Stakeholder, Just over 18 months ago, Houston A+ Challenge launched an ambitious new initiative aimed at significantly increasing the number of students in our region’s public middle schools who are truly on track for success in high school, college, career and civic life. We began our work with a simple, research-based premise – that schools and families must raise their expectations for all students, because our state’s longheld definition of “good enough” is simply not good enough to ensure the success of our children and our democracy in the 21st century.

The timing for this project could not have been more opportune. This spring, for the first time, Texas schools and students will be faced with new state tests that target post-secondary readiness instead of basic proficiency. Simultaneously, public schools are dealing with the impact of more than $5 billion in cuts statewide to education-related spending made during the during the recent 2011 legislative session. As they navigate this new landscape of higher expectations and fewer resources, our partner districts, principals and teachers have expressed much gratitude for the extra resources and technical support offered by the Challenge Network.

Thus, it is with great pleasure that we share with you the external evaluation of our pilot year of work in Challenge Network schools. The report’s emphasis on quantitative student outcomes – specifically, the number of students who moved from “proficient” (or basic) to “commended” (or college-ready-track) performance – was by design, as the Houston A+ Challenge Board of Trustees recognizes the need to demonstrate early, clear, positive student outcomes in order to justify continued investment from A+, its funders, and its partner districts.

Mary Nesbitt Parent Visionaries Rod Paige, Ph.D. Chartwell Education Group LLC David Ruiz Bank of America J. Victor Samuels Victory Packaging, Inc. J. David Thompson, III Thompson & Horton LLP Bobby Tudor Tudor, Pickering, Holt & Co. Andrea White Community Volunteer ADVISORY BOARD Leonel Castillo Mayor's Office (Retired) Michael Dee Investment Banker H. Devon Graham, Jr. R. E. Smith Interests, Inc. Jenard M. Gross Gross Investments Ann Friedman, Ph. D. Community Volunteer Harry M. Reasoner Vinson & Elkins LLP Rosie Zamora Houston Wilderness, Inc.

We are pleased to quote Dr. Ed Fuller, Associate Professor and Director of the Center for Education Evaluation and Policy Analysis at Penn State University, in reporting that his independent analysis of student data “strongly suggests that the reform effort is having a positive impact on student performance.” Dr. Fuller further concludes that “these initial results are very promising, and the reform effort should be fully supported by school and district personnel as well as funders and policymakers.” Significant findings include: Four out of six Challenge Network schools exhibited positive results. Excluding one school that did not have an A+ Coach for the full year, four of the five schools implementing the reform performed better than or about as well as expected.

Absolute results were strongest at YES Prep West (YES Prep Public Schools), where more students reached the college-ready standard than at any other Challenge Network school and any other campus in the high-performing YES district.

Three schools exhibited very strong relative student performance, meaning A+ Scholars were statistically significantly more likely to achieve commended status than similar peers in comparison schools statewide. A+ Scholars at Horace Mann Junior School (Goose Creek CISD) were more than 10 times more likely than similar students in comparison schools to attain commended performance; A+ Scholars at YES Prep West were 4.6 times more likely to do so; and A+ Scholars at Caraway Intermediate School (Aldine ISD) were 1.6 times more likely to reach the higher bar.

A+ Scholars at O’Donnell Middle School (Alief ISD) were also more likely to achieve commended status than similar students in similar schools statewide, but the measured effect was

Since 1997, serving as a catalyst for change in the public schools that educate nine of every ten children in our region, teaming with principals and teachers in targeted schools to ensure that every student is prepared for post-secondary success.

not as statistically significant. However, O’Donnell students did show strong results in their individual academic growth over time.

Hoffman Middle School (Aldine ISD) was neutral on most measures. Atascocita Middle School (Humble ISD) had the worst performance profile on multiple measures. One influencing factor was a mid-year staffing change at A+, which led to the A+ Coach not being present to coordinate and drive the reform at AMS for the full school year. This was disruptive to A+’s traction with teachers and students at AMS, and likely had a negative impact on the school’s ability to reach higher performance targets.

At least 8 in 10 A+ Scholars surveyed reported that, because of their A+ Coach, they work harder in class, don’t get into trouble as much, pay attention in class more, do better in class, understand the teacher better, and feel more confident about school.

These early results from our pilot year are promising but mixed, and we must keep in mind that history and research have taught us that most reforms take multiple years to bear full and consistent fruit. Education researcher Michael Fullan defines this as “implementation dip”, or the “inevitable bumpiness and difficulties encountered as people learn new behaviors and beliefs.”

In conclusion, we are so gratified to work alongside many brave, dedicated, thoughtful teachers and principals who are rising to the challenge of not only learning, but adopting many new beliefs and behaviors. We are also extremely thankful for the unwavering confidence of the Brown Foundation, Houston Endowment and a host of other individuals, foundations and corporations that support Houston A+ Challenge. And none of our work would be possible without the support of forward-thinking leaders at our Challenge Network partner districts – Aldine, Alief, Goose Creek, Humble and YES Prep – and our talented initiative partners at Communities In Schools and Region 4 Education Service Center.

But most of all, we are grateful for the nearly 900 A+ Scholars who provide the inspiration for all of our efforts. Each number that you read in this report relates directly to a child’s hard work, perseverance and vision for the future. We hope that you will be as inspired as we are by their achievements.

Sincerely,

William K. Crouch President, Board of Trustees

Scott A. Van Beck, Ed.D. Executive Director

Year One Evaluation of the Houston A+ Challenge Network of Middle Schools (2010-11)

Published February 2012

Full Report Available Online at: www.houstonaplus.org/challenge-network/results

Ed Fuller, Ph.D. Associate Professor and Director Center for Education Evaluation and Policy Analysis Penn State University

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

EXECUTIVE SUMMARY The Houston A+ Challenge Network consisted of six middle schools in six districts in the Houston metro area. Each school selected a grade level and subject area to focus on, with one school choosing both math and reading. Table ES-1 provides the details on the districts, schools, grade levels, and subject areas involved.

Table ES-1: School Size, Grade Span, and Targeted Grade Levels for 2010-11 School Year District

School

Number A+

2010 Grade

Target

Subject

Name

Name

Students

Configuration

Grade

Area

05 - 06 07 - 08 07 - 08 06 - 08 06 - 08 06 - 07

5 7 7 6 6 7

Reading Mathematics Mathematics Mathematics Math/Reading Reading

Aldine ISD Aldine ISD Alief ISD Goose Creek ISD Humble ISD YES Prep

Caraway Hoffman O'Donnell Mann Atascocita YES Prep

359 411 625 239 567 144

The primary goal of the reform was to have students “in the middle”—those who had passed TAKS but not attained commended status—improve enough to achieve commended status. The reform consisted of multiple activities that included the following:           

Coaching of principals and school leadership teams; Coaching of teachers; Engaging with and supporting parents; Holding network level events and trainings; Developing Professional Learning Communities (PLCs) within schools; Developing shared responsibility among all school staff for student success; Implementing Assessment FOR Learning; Adopting rigorous, data informed instruction targeted to higher standards; Improving learner behavior and academic discipline; Increasing parental awareness of requirements for post-secondary success; and, Implementing cross-district and charter sharing of best practices

As with most reform efforts, the Houston A+ Challenge (A+) reform effort had mixed results in Year One. This is typical with almost every reform effort, since these efforts typically ask teachers and administrators to make fairly dramatic changes in how they think and behave.

This often causes fear and stress, among other emotions, all of which distract from the focus on teaching and learning. On top of the normal reactions to change were deep budget cuts that resulted in layoffs, cuts in services, and potentially larger class sizes. These factors cannot be discounted when evaluating a reform effort.

With respect to the primary goal of attainment of commended status, Table ES-2 shows mixed results. A+ Scholars in three schools—Caraway, Mann, and YES Prep West—were statistically significantly more likely than their peers in comparison schools to achieve commended status. Although not statistically significant, the results also suggested a possible

2

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

positive impact on O’Donnell students. There was no statistically significant effect at Hoffman, but results suggested a possible negative impact. Atascocita is a special case, as there was a staff change in A+ Coaches during the fall semester, so the school did not receive a full, uninterrupted year of coaching (as did the other schools). This may have had a negative effect on student achievement. Regardless of the reason, A+ Scholars in Atascocita were statistically significantly less likely than their peers in comparison schools to achieve commended status in mathematics.

In reading, the result was not statistically significant, but suggested a possible negative effect.

Table ES-2: Logistic Regression Results for Students in the Middle Attaining Commended Status for A+ Scholars School

Stat.

Odds

Interpretation of

Name

Sig.

Ratio

Odds Ratio

Caraway Hoffman

0.036 0.277

1.586 0.656

58.6% more likely Not stat sig: less likely

Mann Atascocita-Math Atascocita-Reading O'Donnell

0.000 0.046 0.365 0.321

10.747 0.242 0.687 1.462

10.7 times more likely 75.8% less likely Not stat sig: less likely Not stat sig: more likely

YES Prep

0.050

4.658

4.65 times more likely

If Atascocita is removed from the analysis because the school did not fully implement the entire reform package, then three of the five schools fully implementing the reform had A+ Scholars who were statistically significantly more likely to achieve commended status than peers in comparison schools. Moreover, one of the remaining two schools had a positive, but not statistically significant effect. These results strongly suggest that the reform effort is having a positive effect on student attainment of commended status.

Table ES-3 includes estimates of whether students performed greater than, equal to, or worse than predicted based on prior test scores, economically disadvantaged status, and school demographics and achievement. Standardized residual values around zero indicate the school performed about as expected, while positive standardized residuals indicate that the school performed greater than expected.

Three of the six schools performed greater than predicted: YES Prep West, O’Donnell, and Mann. Two schools performed worse than expected: Caraway and Atascocita. One school— Hoffman—performed about as well as expected.

Table ES-3: Standardized Residuals (in Z-Scores) for All A+ Scholar Groups District Name Aldine ISD Goose Creek ISD Humble ISD Aldine ISD Alief ISD YES Prep West

School Name Caraway A+ Scholars Mann A+ Scholars Atascocita A+ Scholars Atascocita A+ Scholars Hoffman A+ Scholars O'Donnell A+ Scholars YES Prep A+ Scholars

Target Grade 5 6 6 6 7 7 7

3

Subject Area Reading Math Math Reading Math Math Reading

Standardized Residual -0.313 0.144 -0.448 -0.297 -0.071 0.123 0.263

Effect Negative Positive Negative Negative Neutral Positive Positive

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Discounting the negative results from Atascocita because the school did not fully implement the reforms, then four of the five schools performed about as well as or greater than expected. Again, this strongly suggests that the reform effort is having a positive impact on student performance.

The final important measure of the reform effort is the direct effect on student behavior.

To assess this impact, a survey was administered to all A+ Scholars. Three of the six schools had large enough response rates to report findings. Responses for the three schools were collapsed together and reported below in Figure EW-1.

Students were asked to respond to eight prompts, each of which started with the phrase: “Because of the A+ Coach …” More than 78% of respondents indicated that their own schoolrelated positive behaviors had increased because of interactions with the A+ Coach. Most impressively, about 88% of the students agreed or strongly agreed that the A+ Coach had made them work harder in class, and 84% of the students believed that they worked harder in class because of the A+ Coach. Although there is no other data to substantiate the perceptions of the students in these three schools, the results do strongly suggest that the reform effort—in particular, the A+ Coach—had positive effects on student behavior.

Figure ES-1: Survey Results for Effects of the Coach over Time

Overall, the data collected and analyzed for the Year One evaluation strongly suggests that the Houston A+ Challenge Network reform effort is having a positive effect on the likelihood of students attaining commended status, student growth as measured through value-added analyses, and student behavior as measured by student self-reports. The effects, however, are unevenly distributed across schools. Additional efforts need to be undertaken to ensure that all of the reform components are fully implemented in each school and additional information needs to be collected to determine the causes of some of the low performance so that the reform can be modified in these schools. In final conclusion, the reform efforts have had a strong Year One start despite not being fully operational until the middle of the academic year. These initial results are very promising, and the reform effort should be fully supported by school and district personnel as well as funders and policymakers.

4

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION I: DESCRIPTION OF THE PROGRAM This section describes the organization implementing the reform effort, the program goals, and the theory of change that undergirds the reform effort.

A. REFORM ORGANIZATION The Houston A+ Challenge has a long-standing track record of supporting school reform efforts across the Houston metro area. Houston A+ Challenge serves as a catalyst for change in the public schools that educate nine of every ten children in the Houston region, teaming with principals and teachers in targeted schools to ensure that every student is prepared for postsecondary success. A+ selected six schools in six districts across the Houston metro area as partners in an effort to increase the percentage of students on track to achieve college-readiness in high school.

Unlike many reform efforts, the Houston A+ Challenge Network focuses on students, teachers, principals, other school staff, and parents. Moreover, the focus of the reform is not simply on increasing student test scores, but on: improving student behaviors; teaching students how to learn; enhancing the capacity of teachers; providing socio-emotional support to students; encouraging parents to support school behaviors and desire to attend college; and, enjoining school staff and leaders in the reform process.

B. PROGRAM GOALS The Theory of Change for this reform effort is based on emerging research that points to the importance of building the capacity of school leaders, teachers, students and parents in order to achieve four outcomes: 1.

Increased percentage of A+ Scholar students on track to reach college-readiness standards in high schools;

2.

Increased percentage of all students on track to reach college-readiness standards in high schools;

3.

Increased awareness of and intentions to enroll in institutions of higher education for all students; and,

4.

A sustained system of success and constant improvement within schools

In this Year One evaluation, the focus is on outcomes one and three. In subsequent years, all four outcomes will be examined in the evaluation.

C. THEORY OF CHANGE The Houston A+ Challenge Theory of Change, as depicted in Figure 1, illustrates the process through which A+ will achieve their articulated goals. A more detailed exposition of each component is provided below. The Theory of Change is divided into five areas: inputs, activities, outputs, intermediate outcomes, and final outcomes.

5

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Figure 1. Theory of Change, Houston A+ Challenge Network

1. Inputs The Houston A+ Challenge Network marshals critical inputs towards the goal of achieving the outcomes outlined above. The inputs include: Coaching for Performance by A+ team; a supportive peer network that promotes learning from collaboration; a research-based focus on and approach towards post-secondary success; district commitments; partnerships with organizations such as Communities in Schools, and funding.

2. Activities Recognizing that improved student outcomes depends on activating all elements of the education ecosystem, A+ Challenge worked with principals, teachers, parents, the district leadership, and students themselves. A+ Challenge aligned their resources to perform five key program activities:

6

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

a) Coaching of principals and school leadership teams Students benefit when schools have a vision, a culture and a plan for ensuring post-secondary success for all. A+ provided coaching to middle school principals and the leadership teams to develop their leadership skills and build and implement a comprehensive, sustainable, research-based action plan for advancing student postsecondary success on their campus. b) Coaching of teachers Students benefit when teachers work continuously to improve their craft and have adequate support for doing so. A+ Performance Coaches built teacher capacity through intensive individualized coaching and targeted professional development that occurred throughout the entire school year. The coaching and professional development are based on attention to data, curriculum standards, lesson planning, instructional strategies, classroom observations, assessment design, critical feedback, and repetition of the cycle.

Teachers are trained to use formative assessment techniques to make their instruction more rigorous, targeted and informed by data. Daily activities include team planning, coteaching, demonstrating lessons, and providing constructive feedback to teachers. c) Supporting and motivating students At each Challenge Network school, the A+ Performance Coach worked with teachers to identify a cohort of middle-school students who scored between “proficient” and “commended” on TAKS in the previous academic year. These students are “in the middle”—not low-performing, but not high-performing either. Since the students had already met the minimum standards on the state-mandated assessment and the accountability system only rewards the attainment of passing, these students often do not receive the extra help they need to become college-ready.

Coaches and teachers spent extra instructional time with these students to ensure that they reach and maintain high levels of performance. The Communities in Schools counselor also worked closely with the coach and teachers to identify any social and emotional needs that struggling students might have and create a plan to address those needs. In addition, students received information on pathways to college – such as preparing for and choosing the best high school coursework and learning about the SATs – as well as invitations to special events such as college visits. d) Engaging with and supporting parents Houston A+ Challenge and its partner, Communities In Schools, worked with school staff to get student and families to sign commitments, helped parents understand that ‘passing’ is not enough, and continually engaged families in having higher expectations and support for their students.

7

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

e) Holding network level events and trainings The A+ Challenge network level events and trainings enabled peer collaborations and opportunities for learning that will hopefully be sustained beyond the A+ Challenge immediate engagement with schools. These forums provided opportunities for district schools to learn from some of the best practices in charters, as well as for charter schools to learn from best practices in district schools.

4. Outputs The activities outlined above were designed to deliver seven process outputs that will lead to the attainment of the four intermediate goals which, in turn, will lead to the attainment of the four over-arching goals described above. These seven process outputs include the following activities: a) Development of Professional Learning Communities (PLCs) within schools Led by the principal, the staff collaboratively and continuously builds its collective capacity to provide high levels of learning for all students. Studies have found that a shared focus that unifies the work of all members of the organization and a collaborative culture are common characteristics of high-performing organizations, including schools. PLCs are identifiable by six characteristics: A focus on high level of learning for all students; a collaborative school culture; collective enquiry into best practices and current reality; action orientation; a commitment to continuous improvement, and a focus on results. b) Responsibility for students’ post-secondary success shared by non-teaching positions Schools will identify and designate a non-teacher leader to be the “School-based Coach” who in Year Two will work closely with the A+ Coach and play the role of A+ Coach with an incoming cohort of approximately 150 students to ensure scalability and sustainability. The School-based Coach drives student achievement gains by building teacher capacity through individualized coaching and targeted professional development based on attention to data, curriculum standards, lesson planning, instructional strategies, classroom observations, assessment design, critical feedback, and repetition of the cycle.

The School-based Coach also drives student achievement gains through working with an identified group of middle school students to improve their achievement and likelihood of being on-track to be post-secondary ready by the time they enroll in high school. Finally, the School-based Coach drives student achievement gains by working with students’ families to ensure they understand the importance of having their child be post-secondary ready entering high school.

8

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

c) Assessment FOR Learning practiced by all teachers Teachers use Assessment FOR Learning techniques to identify the specific learning needs of students, adapt instruction to meet those needs, and motivate students to engage in the learning process. Teachers also provide continuous, consistent, and descriptive feedback to students. Extensive research has documented the significant impact that the effective use of assessments to support student learning has on student performance, including on standardized assessments. d) Rigorous, data informed instruction to aim for higher standards Teachers plan lessons and student interventions based on a rigorous analysis of student data, an awareness of standards, an appreciation of best practices in instruction, and the knowledge of experienced coaches, teachers, and administrators. e) Improved learner behavior and academic discipline Students respond to the increased instruction time and the support and motivation they receive both inside and outside classrooms by increasing their level of effort and engagement, thereby leading to improved learner behavior and academic discipline. f) Parental awareness of requirements for post-secondary success Parents are aware of what it will take for their children to achieve collegereadiness and are fully committed to the goal of ensuring their child’s post-secondary success. g) Cross-district and traditional-charter district sharing School districts and charter schools continuously share what they have learned, as well as best practices, thereby creating a sustained, peer-based support network that collectively strives to ensure that all students within the A+ Challenge Network are on track to reach college-readiness.

5. Intermediate outcomes The activities will lead to four intermediate outcomes that research shows directly translate into the final desired outcomes: a) Increased capacity of principal and school-based coaches Research over the past decade has under-scored the importance of the principal and school leadership team in improving schooling outcomes for students. School leaders improve student outcomes primarily through several key indirect strategies associated with strategically managing the human capital within the school setting and improving the capacity of parents to provide support and assistance to their children.

9

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

b) Increased teacher capacity and efficacy A large and growing body of evidence has identified teachers as the single most important school-based factor that influences student outcomes. There is now consensus among both researchers and policymakers that the capacity of both individual teachers and the team of teachers at a school are inextricably linked to the academic improvement of students in the school. c) Increased student effort and engagement Experience shows that greater effort and involvement by students directly translates into improved scholastic performance. d) Increased parental support and involvement Every study on the factors affecting the academic achievement of students finds that the home lives of students have the largest and most profound effect on student success. Research on improving outcomes has shown that providing non-academic support to students and increasing the knowledge and skills of parents to engage in helping their students and interacting with the school are associated with improving school outcomes for students.

10

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION II: CONTEXT This section provides background and contextual information about the state policy environment and the school characteristics of the schools involved in the reform effort.

A. STATE POLICY The relevant state policy context within which this reform was undertaken includes an increased focus on ensuring a greater percentage of students are well-prepared for college or for employment after high school, declining investments in education, decreasing budgets for school districts, and increasing poverty rates for students enrolled in schools.

1. Focus on College and Post-Secondary Readiness Texas has long been a leader in testing and accountability. Texas implemented minimum competency testing in the 1980s with the Texas Assessment of Basic Skills (TABS) and replaced that Texas with a more rigorous test, the Texas Assessment of Academic Skills, or TAAS. Concomitant with the introduction of the TAAS in the early 1990s was the introduction of the Texas school and school district accountability system. The system held schools accountable for three primary outcomes: test scores, dropout rates, and attendance rates. The novel component of the system was that all the data was disaggregated by student subpopulation in a way that a school’s lowest performing subgroup determined the school’s accountability rating.

TAAS was replaced with the Texas Assessment of Knowledge and Skills (TAKS) in 2003. Again, the rigor of the new test was greater than the test that it replaced. The accountability system changed minimally, but kept the primary driver of focusing on outcomes by student subpopulation intact.

In the late 2000s, an increasing volume of research pointed to the need for schools to focus more on preparing students to be college- and career-ready than on meeting minimum competency standards as indicated by the percentage of students passing state mandated tests.

Thus, policymakers, parents, and the business community started to demand that schools increase their focus on college- and career-readiness.

Consequently, Texas was one of the first states to adopt college- and career-readiness standards. Further, Texas started reporting a college-readiness indicator on the publicly available Texas Education Agency website.

Also at the end of the decade, a number of reports were released that showed that student achievement levels in elementary and middle school could predict college readiness in high school. Further, the reports uncovered that high schools did not have the capacity to take students not well prepared for high school and prepare them for college. In essence, then, the research concluded that middle schools are the last chance for students to get on the college-ready track.

Thus, the 2000s saw an increasing amount of pressure on high schools to push students past simple proficiency and towards being college- and career-ready. Middle schools also felt pressure to ensure that students focused not just on meeting proficiency, but on preparing students to be well-prepared for high school and on the college-ready track.

The adoption of a new state-mandated assessment also drives higher expectations for student achievement as the new tests measures college-readiness across all grade levels.

11

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

2. Declining Budgets Texas has never been a state renowned for its investment in K-12 education. Indeed, Texas has always ranked in the bottom tier of states in terms of investment in K-12 education.

With the recession of 2008 and the school finance/property tax bill of 2006, the available revenue available for education and other areas plunged dramatically. In the 2010 legislative session, legislators cut $4 million from K_12 expenditures over the 2011-12 and 2012-13 school years. The indecisiveness and lack of planning of the Legislature led to widespread confusion and fear among school districts during the spring of 2011. Many districts adopted widespread layoffs of teachers and central office staff. Many districts have now had to ask for waivers from the mandated 22:1 student-teacher ratio in grades one through three.

Thus, the implementation of this reform effort occurs at a time of declining budgets, decrease in resource availability, and an increase in class sizes. These factors could certainly have a negative impact on the efficacy of the reform effort since the schools are being asked to maintain progress with fewer resources.

3. Increasing Poverty and Student Diversity Texas has one of the highest child poverty rates in the country and has seen an explosion in the number of low-income students over the past two decades. The increase has been widespread across the state, but more pronounced in the urban centers and along the TexasMexico border.

Figure 1: Percentage of Economically Disadvantaged Students in Texas Public Schools

Source: Academic Excellence Indicator System (AEIS), Texas Education Agency (TEA)

12

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Figure 2: Percentage of Economically Disadvantaged Students by School Level in Texas Public Schools (1995, 2000, 2005, and 2010)

Source: AEIS, TEA

B. SCHOOLS This section provides information on the school characteristics of the reform schools.

This information includes the school grade configuration and enrollment, student characteristics, and teacher characteristics.

1. School Configuration and Size As shown in Table 1, the six schools varied considerably in size and grade span. YES Prep West enrolled only 144 students in the 7th grade, while Atascocita enrolled 1,705 students in grades six through eight. Research generally finds that larger schools have detrimental effects on student achievement and persistence in schools, particularly for lower-performing and lowincome students. Thus, some schools were in a more advantageous position to advance student achievement than other schools simply based on student enrollment.

Three of the six schools chose to focus on 7th grade mathematics, one school chose to focus on 6th grade mathematics, one school chose to focus on 5th grade reading, and one school chose to focus on 6th grade reading and mathematics.

Schools focusing on reading have a more difficult time producing gains in achievement, because research strongly indicates that reading ability is strongly influenced by the home life of students – in particular, poverty and the factors associated with poverty, such as poor nutrition, lack of health care, and inadequate access to books at home and in the community.

13

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 1: School Size, Grade Span, and Targeted Grade Levels for 2010-11 School Year District

School

School

Size Tgt

2010 Grade

Target

Subject Area Target:

Name

Name

Size

Grade

Configuration

Grade

Math

Reading

05 - 06 07 - 08 06 - 08 06 - 08 07 - 08 06 - 07

5 7 6 6 7 7

none 7 6 6 7 none

5 none none 6 none 7

Aldine ISD Aldine ISD Goose Creek ISD Humble ISD Alief ISD YES Prep

Caraway Hoffman Mann Atascocita O'Donnell YES Prep

749 808 731 1705 1213 144

359 411 239 567 625 144

Source: AEIS, TEA; Evaluation data collected from sites

Table 2 documents the overall student enrollment of the A+ Scholar schools, as well as the average enrollment of the A+ Scholar comparison schools. In general, the five schools with longitudinal data on enrollment have had stagnant or slightly declining enrollment trends. Mann Junior High School in particular experienced declining enrollment, with a decrease of almost 300 students from 2006 to 2010. Stable student enrollments tend to make reform changes easier to implement because of staff and student stability, while rapidly increasing or decreasing enrollments tend to make changes more difficult to implement.

Hoffman, Mann, and YES Prep West had similar enrollments as comparison schools, while Caraway had a smaller enrollment than comparison schools. In contrast, O’Donnell and Atascocita had student enrollments much larger than comparison schools.

Table 2: Student Enrollment for All Grades, 2006-2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 820.0 929.8 971.0 860.8 1263.0 1128.0 1008.0 773.2 1508.0 1148.5 477.0

Spring of Academic Year 2007 2008 2009 780.0 811.0 782.0 933.0 892.3 940.3 885.0 881.0 930.0 834.3 862.3 791.8 1150.0 1143.0 1206.0 1071.8 1033.8 983.4 994.0 978.0 814.0 771.0 807.6 795.4 1652.0 1520.0 1656.0 1250.8 1249.8 1061.0 453.0

SOURCE: AEIS, TEA

14

497.5

496.5

2010 749.0 977.3 808.0 791.3 1213.0 965.8 731.0 785.0 1705.0 1109.0 144.0 248.3

Change: 2006 to 2010 -71.0 47.5 -163.0 -69.5 -50.0 -162.2 -277.0 11.8 197.0 -39.5 na -228.7

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

2. School Demographics a) Economically Disadvantaged Students As shown in Table 3, three of the five schools with longitudinal data had very stable percentages of economically disadvantaged students, while both Hoffman and O’Donnell had slight increases in the percentage. All but Atascocita were predominantly low-income schools, with at least 73% of the students identified as economically disadvantaged. Atascocita, alternatively, only had 20% of students identified as economically disadvantaged.

By design, comparison schools had similar percentages of economically disadvantaged students as the A+ Scholar schools. For five of the A+ Scholar Schools—Caraway, Hoffman, O’Donnell, Mann and YES Prep West—comparison schools had much greater increases in the percentage of economically disadvantaged students over the last five years. The relative stability in the percentage of economically disadvantaged students in A+ Scholar Schools creates a more advantageous environment for implementing reform.

Table 3: Percentage of Economically Disadvantaged Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 85.6 69.8 79.9 67.6 66.9 59.8 81.5 71.2 23.3 24.2

Spring of Academic Year 2007 2008 2009 87.3 86.7 88.2 72.9 74.1 73.9 84.3 83.3 85.7 69.8 71.3 73.6 66.7 62.1 74.3 60.5 59.7 68.6 74.0 77.3 83.0 71.7 73.0 76.6 21.7 14.9 15.4 25.1 20.5 21.9

66.1

73.6

68.7

66.9

2010 86.8 80.5 84.0 78.7 73.5 71.0 81.8 80.8 20.7 23.4 84.7 86.2

Change: 2006 to 2010 1.2 10.7 4.1 11.1 6.6 11.2 0.3 9.6 -2.6 -0.8 20.2

SOURCE: AEIS, TEA

b) Percentage of African-American Students As shown in Table 4, four of the five A+ Scholar Schools with longitudinal data had declining percentages of African-American students. Only Atascocita, with a slight increase in the percentage of African-American students, did not have a decrease. Given that AfricanAmerican students tend to have the lowest achievement levels of all student sub-populations, the decline of African-American students might increase the overall student performance of the school.

Again, by design, comparison schools had relatively similar percentages of AfricanAmerican students as A+ Scholar Schools. The comparison schools had stable or slightly declining percentages of African-American students over the past five years.

15

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 4: Percentage of Economically Disadvantaged Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 55.9 47.9 55.3 54.9 47.7 35.4 20.2 16.2 19.8 19.0

2007 56.4 47.0 53.3 52.1 46.8 34.8 17.8 16.3 20.8 19.9

2008 54.9 46.2 52.3 50.6 45.8 34.2 20.6 15.3 22.6 18.5

2009 52.8 47.2 50.2 48.7 40.2 32.9 13.9 14.6 23.1 20.3

21.3

22.6

25.3

23.1

2010 49.3 48.0 43.4 47.4 37.9 31.9 13.5 13.1 22.8 20.0 29.2 21.0

Change: 2006 to 2010 -6.6 0.1 -11.9 -7.5 -9.8 -3.6 -6.7 -3.1 3.0 1.0 -0.3

SOURCE: AEIS, TEA

c) Percentage of Hispanic Students As shown in Table 5, four of the five A+ Scholar schools with longitudinal data had increases in the percentage of Hispanic students. Only Atascocita did not have an increase. By design, the percentage of Hispanic students in A+ Scholar schools and comparison schools were relatively similar.

Table 5: Percentage of Hispanic Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 42.2 39.7 41.4 35.2 32.5 37.0 63.3 63.5 22.8 18.2 51.9

Spring of Academic Year 2007 2008 2009 41.2 41.9 43.9 41.5 43.2 43.3 44.0 44.9 47.2 38.9 41.0 43.7 35.7 38.0 42.0 38.6 41.7 44.8 66.6 65.5 71.3 64.8 67.7 70.9 24.2 19.9 20.0 20.5 19.6 20.6 54.7

SOURCE: AEIS, TEA

16

54.5

58.0

2010 47.4 43.1 53.7 46.8 42.2 46.8 73.6 73.2 21.1 21.2 54.2 72.2

Change: 2006 to 2010 5.2 3.4 12.3 11.7 9.7 9.8 10.3 9.7 -1.7 3.0 20.3

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

d) Percentage of White Students As shown in Table 6, the percentage of White students was relatively stable at all A+ Scholar schools over the five-year time span. Comparison schools evidenced slight declines in the percentage of White students over the same time period, with the exception of the YES Prep West comparison schools, which had a substantial decline in the percentage of White students— from 23% in 2006 to 5% in 2010.

Table 6: Percentage of White Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 1.2 8.0 2.6 8.8 7.0 16.4 15.1 17.3 52.8 50.7 23.3

Spring of Academic Year 2007 2008 2009 1.4 1.6 2.2 7.2 6.7 5.7 2.1 1.7 1.2 7.6 6.9 6.1 5.3 4.3 5.3 15.6 13.2 11.7 14.3 12.7 13.3 16.0 14.2 11.8 51.2 53.6 52.1 46.6 47.9 46.6 19.9

18.0

17.2

2010 2.3 5.1 1.2 4.7 5.4 10.8 11.8 11.0 51.3 46.2 0.7 5.2

Change: 2006 to 2010 1.1 -2.9 -1.4 -4.1 -1.6 -5.5 -3.3 -6.4 -1.5 -4.5 -18.1

SOURCE: AEIS, TEA

e) Percentage of Mobile Students Mobile students are defined as those students who have not been in attendance at the school for at least six of the 18 weeks at the school. The lack of attendance could be for absences or transferring into or out of the school at such a time that would cause the student to miss at least 17% of the instructional days at the school.

As shown in Table 7, the A+ Scholar schools had either stable or slightly declining percentages of mobile students. The same was true for the comparison schools. Three of the six schools had a substantial proportion of students identified as mobile. Indeed, Caraway, Hoffman, and O’Donnell had 25% of students identified as mobile. Mann and Atascocita had moderate amounts of mobile students, with 16% and 9.5%, respectively. YES Prep West has 0% mobility.

17

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 7: Percentage of Mobile Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 31.2 19.4 26.9 20.7 24.9 18.6 22.1 20.0 13.1 12.7 19.9

Spring of Academic Year 2007 2008 2009 35.5 29.8 26.1 22.7 21.9 19.2 35.5 32.0 28.2 24.4 25.2 21.0 31.4 26.6 24.9 22.0 19.9 17.9 25.0 22.4 16.9 23.2 21.1 17.4 13.3 13.1 11.9 17.3 14.2 10.3 22.7

23.0

21.1

2010 25.3 17.5 25.3 22.7 25.4 16.9 16.3 18.0 9.5 10.7 0.0 15.9

Change: 2006 to 2010 -5.9 -1.9 -1.6 2.0 0.5 -1.7 -5.8 -2.0 -3.6 -1.9 -4.0

SOURCE: AEIS, TEA

Mobility is important because research has found a clear and consistent negative relationship between the percentage of mobile students and achievement. The greater the mobility, the more pronounced the negative effect, because of the greater difficulty in teaching students with different knowledge and skills, as well as the lack of information about the prior achievement and academic strengths and weaknesses of mobile students. f) Percentage of Bilingual Students Bilingual education students are those students enrolled in a bilingual education program.

As shown in Table 8, three of the A+ Scholar schools had large increases in the percentage of bilingual students in the last five years—Caraway, Hoffman and O’Donnell. All three schools had increases of approximately 10 percentage points. These increases were much greater than the increases for comparison schools. Mann and Atascocita had stable bilingual education populations, and YES Prep did not have longitudinal data available.

For Caraway, Hoffmann, and O’Donnell, the increase was far greater than the approximately 3 percentage points for comparison schools. For Mann and Atascocita, the changes were similar to those of comparison schools.

The percentage of bilingual students affects achievement in a number of ways. First, the students taking the Spanish-language version of the TAKS were not included in the student growth analysis. Second, bilingual students are typically more expensive to educate, thus fewer resources may be available for other students.

18

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 8: Percentage of Bilingual Students, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 13.5 12.9 6.6 6.9 10.8 8.9 5.9 11.9 3.2 2.9 10.1

Spring of Academic Year 2007 2008 2009 13.5 18.5 21.4 12.2 15.2 15.0 5.8 6.7 8.6 7.4 7.7 9.3 13.0 16.1 16.9 9.2 10.8 12.5 6.9 5.5 4.1 10.7 12.2 13.0 2.6 1.4 1.4 3.1 2.7 2.4 11.3

13.0

10.9

2010 23.6 16.3 15.7 10.2 20.4 11.4 3.1 15.0 1.9 2.5 22.2 22.1

Change: 2006 to 2010 10.1 3.3 9.1 3.3 9.6 2.5 -2.8 3.1 -1.3 -0.3 12.1

SOURCE: AEIS, TEA

3. Teacher Characteristics Teacher characteristics are often associated with student achievement, although the research in this area is far from conclusive. Regardless, the percentage of teachers who are assigned “in-field” is a fairly good representation of whether the teachers in a school are prepared to teach the subject area and grade level to which they are assigned. Research in Texas has shown that such an indicator is, in fact, related to growth in student achievement.

As shown in Table 9, the percentage of teachers who were assigned in-field varied substantially across the six schools. Both O’Donnell and YES Prep-West had roughly 50% of teachers assigned in-field, while Caraway has about 88% of teachers assigned in-field. Four of the five A+ Scholar schools with available data had increases in the percentage of teachers assigned in-field, while only Mann had a decrease. All but one set of comparison schools (those for Caraway) had a decrease in the percentage of teachers assigned in-field.

Table 9: Percentage of Teachers Assigned In-Field, 2006 to 2010 School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep West YES Prep West Comparisons

2006 60.7 83.2 53.2 71.7 46.7 71.9 82.0 80.9 81.2 80.8 82.0

Spring of Academic Year 2007 2008 2009 65.8 67.0 77.1 74.0 83.5 86.0 46.4 55.8 57.0 56.2 54.9 54.8 51.3 43.8 52.6 61.8 49.2 51.5 75.4 75.4 72.5 76.2 77.2 70.8 81.9 71.3 69.7 78.3 74.0 75.5 93.0

19

75.6

93.9

2010 87.9 86.8 60.7 50.5 51.7 51.9 75.6 69.6 74.8 72.1 52.8 62.2

Change: 2006 to 2010 27.2 3.6 7.5 -21.1 5.0 -20.0 -6.3 -11.3 -6.4 -8.6 -19.8

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SOURCE: Who is Teaching in Texas, TEA

Recent research has found that teacher turnover has a negative effect on student achievement regardless of the teacher “effectiveness” of the teachers leaving the school.

Turnover affects achievement in two primary ways. First, turnover—especially high levels of turnover—results in a loss of information about students. When a teacher leaves the school, the detailed information about the strengths and weaknesses of her students and the instructional strategies that best match each student are lost to the remaining teachers. Thus, the remaining teachers must discover on their own the students’ strengths, weaknesses, and learning preferences.

As shown in Table 10, two of the A+ Scholar schools had very high rates of turnover.

Indeed, both YES Prep West at 26% and Atascocita at almost 38% had very high levels of turnover. Interestingly, their comparison schools also had high levels of turnover as well. Both O’Donnell and Hoffman had moderate levels of turnover—about 20%—while Mann and Caraway had 16% and 17%, respectively.

Caraway had a declining trend in teacher turnover, while Atascocita had a trend of increasing teacher turnover, but only for the last year. The other schools had stable teacher turnover trends or not enough data to have a trend over time.

Three schools had lower turnover than comparison schools, while two schools had greater turnover than comparison schools. Only Mann had a turnover rate similar to the rate for comparison schools.

Table 10: Teacher Turnover Rate School Name Caraway Int Caraway Comparisons Hoffman MS Hoffman Comparisons O'Donnell MS O'Donnell Comparisons Mann JHS Mann Comparisons Atascocita Atascocita Comparisons YES Prep-West YES Prep West Comparisons

06-07 38.9 28.4 24.6 30.6 19.9 22.5 11.2 19.5 21.3 15.6

Spring of Academic Year 07-08 08-09 09-10 38.2 22.7 25.0 23.1 21.7 15.7 21.1 27.5 19.5 29.5 25.5 20.1 36.3 17.6 18.4 30.0 19.5 16.1 20.7 17.6 9.7 18.0 19.7 14.4 23.7 23.5 21.0 23.7 21.7 14.9

19.7

37.5

20.7

25.4

10-11 16.4 23.4 20.0 35.5 19.8 12.9 17.2 16.7 37.5 26.6 25.9 33.3

Change: 2006 to 2010 -22.5 -5.0 -4.6 4.9 -0.1 -9.6 6.0 -2.9 16.3 10.9 13.7

SOURCE: Teacher Responsibility Files, TEA

4. Other Factors Affecting Implementation The school with the worst performance profile is clearly Atascocita. One reason for the low performance is that the A+ Coach assigned to the school was not present for the full length of time as in the other schools. The inclusion and the disappearance of the A+ Coach was likely to be highly disruptive to the teachers and students and could have had a negative impact on the reform effort at that school.

20

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION III: IMPLEMENTATION OF REFORM EFFORTS Michael Fullan (2001) defines the “implementation dip” as “the inevitable bumpiness and difficulties encountered as people learn new behaviors and beliefs.” Fullan continues on to state: The implementation dip is literally a dip in performance and confidence as one encounters an innovation that requires new skills and new understandings. All innovations worth their salt call upon people to question and in some respects to change their behavior and their beliefs — even in cases where innovations are pursued voluntarily.

The Houston A+ Challenge Network reform effort is certainly a serious reform effort. It not only requires educators to alter beliefs about the capacity of students in the middle to achieve at high levels, it also requires principals and teachers to behave in dramatically different ways.

This change causes increased anxiety, fear, and frustration as well as feelings of being overwhelmed and a strong desire to revert back to beliefs and behaviors that are known and easier to the individual.

One consequence of the potential negative emotional effects implementing a serious reform might have on teachers is a decrease in achievement or at least a dampening of the student achievement gains that would have occurred naturally without the reforms. Often, this “implementation dip” in student achievement causes a loss of commitment to the reform, which simply decreases achievement further. This can lead to a vicious cycle of lowering levels of commitment to the reform and concomitant failure to attain desired achievement outcomes.

When the reform effort includes efforts to teach students knowledge and skills that are not directly assessed on the test instrument, the desire to de-commit from the reform is even more powerful. In the Challenge Network effort, teachers are being asked to work with students to build their capacity to be self-regulated learners. This is a difficult, but crucial task in preparing students to be truly college-ready. Yet accomplishing this task takes time away from directly preparing students to do well on the state-mandated assessment. As such, the reform effort may initially decrease student achievement, even though ultimately the students will be far better-prepared for high school and college than other students.

Thus, it is imperative that the reform leaders ensure that everyone stays committed to the program and lower scores do not cause teachers to ignore the reform effort and revert back to previous behaviors or, worse, choose to teach to the test in order to increase student scores.

Leadership is crucial in such a situation, since leadership support and reassurance are desperately needed during the trying times of implementing a significant reform effort.

21

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION IV: STUDENT OUTCOMES This section describes the findings related to student outcomes. The student outcomes in this section include the following: A. Commended Status; B. Vertical Scale Scores Comparison; and, C. Student Value-Added Growth These three metrics provide information that is relevant to the goals and objectives of the project using different perspectives on student growth and achievement. The examination of commended status uses two measures. The first is just a simple examination of the percentage of students attaining commended status in the year prior to reform and after year one of the reform.

The second perspective uses statistical analyses to determine whether students “in the middle” in reform schools were more or less likely than peers in comparison schools to achieve commended status.

Vertical scale score comparison examines the academic growth of students “in the middle” in reform and comparison schools by the prior year’s vertical score. This provides information about the growth of students by lower- or higher-performing students in the middle.

Finally, value-added growth analysis examines the overall student growth on TAKS relative to all schools in the state.

A. COMMENDED STATUS The analyses in this section were based on student-level data. Only students enrolled in the same district for two consecutive years and who had a valid score for the subject area in both years were included in the analyses.

1) Attainment of Commended Status The focus of this reform effort is to elicit large enough academic growth by students “in the middle” so that the students attain commended status. Fuller (2010) found that students achieving commended status by the end of middle school had a very high probability of attaining college-ready status by the end of high school.

This section includes three sub-sections: a) Change in percent of students commended for all students b) Change in percent of students commended for students “in the middle” by TAKS scale score; and, c) Probability of a student “in the middle” attaining commended status. a) Change in Percent of Students Commended for all Students This section examines the attainment of commended status by A+ Scholars. The section relies on individual student-level data from the TEA. The analyses include all students identified as A+ Scholars, regardless of past performance. In other words, the group of students included those not meeting minimum standards, those meeting minimum standards but not commended status, and those meeting commended status.

22

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Extreme caution must be taken when interpreting these results for several reasons. First, the analyses match students across years, thus only students remaining in the same school or district (depending on the grade configuration of the school). Thus, these numbers will be different than those appearing on the TEA website. Second, and more importantly, changes in commended status are driven largely by the distribution of scores around the cut point for commended status. If a relatively large percentage of students scored slightly below the commended status cutoff score, then increasing the percentage of students meeting commended status is much easier than if few students have scores slightly below the commended status cut point. Thus, changes in commended rates do not necessarily reflect the effect of the school on scores. However, the rates do provide valuable information. But the changes in the rates cannot be used to make conclusions about the effect of the school or teachers on student achievement or student growth.

As shown in Table 11, four of the six schools had increases in the percentage of students achieving commended status in 2010 and in 2011. Atascocita essentially remained at the same level of commended status in both 2010 and 2011 while Mann showed a slight decrease.

Caraway A+ Scholars had the greatest increase at 7 percentage points, followed closely by Hoffman with 6.5 percentage points.

Table 11: Percentage of Students Attaining Commended Status in Reading in 2010 and 2011 A+ Scholars in A+ Challenge Network Schools District

Campus

Academic

Year

Change

Name

Name

2010

2011

10 to 11

ALDINE ISD ALDINE ISD ALIEF ISD GOOSE CREEK ISD HUMBLE ISD HUMBLE ISD YES PREPARATORY

Caraway A+ Scholars Hoffman A+ Scholars O'Donnell A+ Scholars Mann A+ Scholars Atascocita A+ Scholars Atascocita A+ Scholars West

15.0% 0.7% 5.6% 20.3% 19.0% 5.0% 50.7%

21.9% 7.2% 9.7% 18.8% 20.7% 5.8% 55.6%

7.0 6.5 4.2 -1.4 1.7 0.8 4.8

SOURCE: Student-level data; TEA

The remainder of this section compares each A+ Challenge network school to a set of comparison schools. At least one comparison school was selected from the same district as a way to control for any district effects that may have influenced any changes in achievement.

While Caraway A+ Scholars had the greatest increase of all the Network schools, the A+ Scholars had about the same growth as other students in the school and smaller growth than most comparison schools.

23

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 12: Percentage of Students Attaining Commended Status in Reading from 4th Grade in 2010 to 5th Grade in 2011 for Caraway A+ Scholars District Name

Campus Name

Grade 4 2010

Grade 5 2011

Change 10 to 11

ALDINE ISD

Caraway Int

15.2%

21.8%

6.6

ALDINE ISD

Caraway A+ Scholars

15.0%

21.9%

7.0

ALDINE ISD EVERMAN ISD ALIEF ISD ALDINE ISD

Parker Int Powell Int Budewig Int Houston Academy

23.6% 19.1% 19.6% 22.2%

21.9% 28.5% 32.0% 30.8%

-1.7 9.4 12.4 8.6

SOURCE: Student-level data; TEA

Both Hoffman and O’Donnell had increases in the percentage of A+ Scholars achieving commended status, while students in comparison schools had a decrease in the percentage of students meeting commended status. Even with the cautions in interpretation described above, these data strongly suggest that A+ Scholars at both Hoffman and O’Donnell outperformed their peers in comparison schools.

Table 13: Percentage of Students Attaining Commended Status in Mathematics from 6th Grade in 2010 to 7th Grade in 2011 for Hoffman and O’Donnell A+ Scholars District Name

Campus Name

Grade 6 2010

Grade 7 2011

Change 10 to 11

22.6%

13.4%

-9.2

0.7%

7.2%

6.5

ALDINE ISD

Hoffman MS

ALDINE ISD

Hoffman A+ Scholars

ALDINE ISD EVERMAN ISD DUNCANVILLE ISD FORT BEND ISD

Teague MS Baxter MS Reed MS McCauliffe MS

23.9% 20.6% 30.1% 18.4%

21.2% 9.2% 14.6% 7.7%

-2.7 -11.4 -15.5 -10.7

ALIEF ISD

O'Donnell MS

30.3%

21.6%

-8.7

ALIEF ISD

O'Donnell A+ Scholars

5.6%

9.7%

4.2

ALIEF ISD ALIEF ISD MESQUITE ISD RICHARDSON ISD PASADENA ISD

Albright MS Holub MS McDonald MS Liberty JHS Beverly Hills Int

39.3% 28.6% 34.8% 51.6% 28.1%

20.9% 14.6% 16.8% 28.3% 27.2%

-18.4 -14.1 -18.0 -23.3 -0.9

SOURCE: Student-level data; TEA

24

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 14, there was very little change in commended rates in mathematics for A+ Scholars at Horace Mann and Atascocita. For Mann, the slight decrease was actually better than most comparison schools, which saw larger decreases than at Mann. Only one comparison school had an increase of all comparison schools. Mann A+ Scholars also had a smaller decrease than the entire Mann Junior High School. This suggests that the A+ Scholars may have outperformed other Mann students.

Table 14: Percentage of Students Attaining Commended Status in Mathematics from 5th Grade in 2010 to 6th Grade in 2011 for Mann A+ Scholars District

Campus

Grade 5

Grade 6

Change

Name

Name

2010

2011

10 to 11

GOOSE CREEK ISD

Mann JHS

35.30%

30.60%

-4.7

GOOSE CREEK ISD

Mann A+ Scholars

20.30%

18.80%

-1.4

GOOSE CREEK ISD SPRING BRANCH ISD DEL VALLE ISD EL PASO ISD GARLAND ISD

Baytown JHS Spring Oaks MS Del Valle MS Terrace Hills Bussey MS

39.30% 18.30% 30.70% 33.50% 32.00%

32.20% 23.20% 24.50% 19.90% 17.30%

-7 5 -6.2 -13.6 -14.7

SOURCE: Student-level data; TEA

As shown in Table 15, A+ Scholars at Atascocita had about the same slight increase in the percentage of commended students in reading as the entire school. The slight increase for the Atascocita A+ Scholars was not much different than for comparison schools with the exception of Dulles Middle School in Fort Bend ISD. In mathematics, the entire Atascocita school had a monumental decrease in the percentage of commended students—31 percentage points. This means that a large number of students had met the commended level in 5th grade, but did not meet the commended level in 6th grade. A+ Scholars, however, showed no change from 2010 to 2011. This makes sense, in that few A+ Scholars would have met the commended status in the 5th grade. All comparison schools had a large decrease in the percentage of commended students from 2010 to 2011.

25

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 15: Percentage of Students Attaining Commended Status in Mathematics And Reading from 5th Grade in 2010 to 6th Grade in 2011 for Atascocita A+ Scholars District

Campus

Grade 5

Grade 6

Change

Name

Name

2010

2011

10 to 11

MATHEMATICS HUMBLE ISD

Atascocita

38.1%

39.7%

1.6

HUMBLE ISD

Atascocita A+ Scholars

19.0%

20.7%

1.7

HUMBLE ISD FRISCO ISD KLEIN ISD FORT BEND ISD

Timberwood MS Stafford MS Schimndewolf Int Dulles MS

31.7% 54.0% 39.9% 45.7%

36.9% 54.4% 41.1% 53.7%

5.2 0.5 1.1 7.9

FRISCO ISD

Roach MS

52.3%

55.8%

3.5

53.7%

22.7%

-31.0

5.0%

5.8%

0.8

41.0% 64.0% 52.4% 61.6% 62.2%

30.1% 50.5% 28.2% 39.6% 44.8%

-11.0 -13.6 -24.2 -22.1 -17.4

READING HUMBLE ISD

Atascocita

HUMBLE ISD

Atascocita A+ Scholars

HUMBLE ISD FRISCO ISD KLEIN ISD FORT BEND ISD FRISCO ISD

Timberwood MS Stafford MS Schimndewolf Int Dulles MS Roach MS

SOURCE: Student-level data; TEA

As shown in Table 16, YES Preparatory West had a similar increase in the percentage of students attaining commended status from the 6th grade in 2010 to the 7th grade in 2011 as comparison schools. One comparison school—Energized for Excellence in Houston ISD—had a decrease in the percentage of commended students.

Table 16: Percentage of Students Attaining Commended Status in Reading from 6h Grade in 2010 to 7th Grade in 2011 for YES Preparatory West A+ Scholars District Name YES PREPARATORY YES PREPARATORY THE EAST AUSTIN KIPP Inc HOUSTON ISD

Campus Name West Bray The East Austin Intrepid Energized for Excellence

SOURCE: Student-level data; TEA

26

Grade 6 2010 50.7% 16.3% 12.0% 26.6% 49.0%

Grade 7 2011 55.6% 20.6% 14.5% 30.4% 45.2%

Change 10 to 11 4.8 4.3 2.4 3.8 -3.8

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

b) Change in Percent of Students Commended for Students “In the Middle” This analysis focuses only on those students “in the middle”—passing the TAKS, but not attaining commended status. Because the starting point for the student markedly impacts the ability to attain commended status, the analysis is by scale score. Each score interval is the equivalent of one question answered correctly. So, working from the highest score at the end of the column, each move up the column marks a one question deviation from passing.

Further, the A+ Scholars for each school are included with students from a large comparison group of schools from across the state. This group of comparison schools includes schools with the same grades served as the A+ Scholar schools, as well as with similar student characteristics in terms of students in poverty and student race/ethnicity.

As shown in Table 15, A+ Scholars at Caraway were generally as or more successful at attaining commended status at each scale score than for comparison schools. This was especially true at the top two scale scores of 667 and 691 where 22% of all Caraway students in the middle started.

Table 17: Students in the Middle Attaining Commended Status by 2010 Mathematics Scale Score for Caraway A+ Scholars Scale Score 554 555 560 564 570 580 585 591 603 607 616 620 631 633 647 649 666 667 691 All

Caraway N % 7 28.6% 3 0.0% 8 0.0% 1 0.0% 17 0.0% 12 8.3% 2 0.0% 11 18.2% 13 15.4% 1 0.0% 18 16.7% 2 0.0% 17 23.5% 2 50.0% 24 25.0% 2 0.0% 4 0.0% 18 55.6% 24 54.2% 186 23.7%

Comparisons N % 345 3.2% 36 0.0% 399 3.0% 30 0.0% 451 5.3% 521 6.1% 45 0.0% 607 10.7% 732 14.6% 68 1.5% 763 18.3% 88 2.3% 873 22.6% 97 3.1% 976 30.7% 108 4.6% 95 3.2% 992 38.9% 1084 49.5% 8421 21.7%

SOURCE: Student-level data; TEA

27

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 18, very few of the Atascocita A+ Scholars advance enough to attain commended status. Even of the 26 students just one question shy of commended in 2010 (scale score of 706), less than 8% attained commended status as compared to almost 33% in comparison schools. Of the 45 students scoring between two and five questions below commended in 2010 (scale scores between 657 and 691), only one achieved commended status.

In the same table is shown the results for Mann A+ Scholars. In contrast to Atascocita, the Mann A+ Scholars had greater attainment rates than their peers in comparison schools. This was especially true for students within four questions of commended status in 2010. While only 3 of 60 such students in Atascocita attained commended status, 11 of 40 (27.5%) of such students at Mann attained commended status. In comparison, only 15.7% of such students n the Mann comparison schools attained commended status.

Table 18: Students in the Middle Attaining Commended Status by 2010 Mathematics Scale Score for Atascocita and Mann A+ Scholars Scale Score

Atascocita A+ N

Atas Comparisons

%

N

%

Mann A+ N

%

Mann Comparisons N

%

603 606 613 621 629 638 647 657 667 678 691 706

5 6 4 8 3 6 4 11 9 17 8 26

0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 11.1% 0.0% 0.0% 7.7%

76 72 106 123 148 164 198 215 271 344 430 519

0.0% 1.4% 1.9% 2.4% 2.0% 3.7% 6.6% 6.0% 8.1% 15.1% 18.4% 32.8%

5 3 4 6 7 7 10 9 13 8 5 14

0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 20.0% 11.1% 23.1% 25.0% 40.0% 28.6%

204 253 254 326 328 325 363 409 425 499 529 607

1.5% 1.2% 2.4% 2.5% 3.4% 6.8% 6.3% 9.3% 8.0% 13.8% 16.3% 22.1%

Total

107

2.8%

2670

13.6%

91

15.4%

4547

9.7%

SOURCE: Student-level data; TEA

28

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 19, the attainment rates for Atascocita A+ Scholars was substantially lower than the attainment rates for comparison schools. Indeed, Atascocita had a greater attainment rate for only one of the nine scale scores.

Table 19: Students in the Middle Attaining Commended Status by 2010 Reading Scale Score for Atascocita A+ Scholars Scale

Atascocita

All Comparisons

Score

N

%

N

%

620 630 640 651 662 675 689 706 725

6 4 3 11 11 7 12 10 9

0.0% 0.0% 0.0% 9.1% 36.4% 0.0% 25.0% 20.0% 33.3%

124 129 171 221 304 453 473 623 789

4.8% 3.9% 13.5% 14.9% 15.5% 23.6% 31.5% 43.5% 52.3%

Total

74

17.6%

3295

32.0%

SOURCE: Student-level data; TEA

As shown in Table 20, YES Prep West had slightly greater attainment rates for those students within two questions of commended status in 2010, but had a 0% attainment rate for all other students. Had even just a few of the 13 students in the 697 to 729 scale score range, YES Prep West would have had a significantly greater attainment rate than comparison schools.

However, the final attainment rate was roughly the same for YES Prep West and comparison schools.

Table 20: Students in the Middle Attaining Commended Status by 2010 Reading Scale Score for YES Preparatory West A+ Scholars Scale Score 644 645 655 665 676 688 701 715 731 751 Total

YES Prep-West N % 1 0.0% 2 0.0% 2 50.0% 4 25.0% 5 0.0% 6 33.3% 9 33.3% 7 0.0% 10 70.0% 17 52.9% 63 36.5%

All Comparisons N % 36 5.6% 42 7.1% 77 7.8% 70 5.7% 78 9.0% 119 6.7% 155 14.8% 184 20.7% 197 25.4% 213 44.6% 1171 20.2%

SOURCE: Student-level data; TEA

29

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 21, A+ Scholars at Hoffman had lower attainment rates of commended status than students in comparison schools. Indeed, only 1 of the 17 students within 3 questions of commended status in 2010 actually attained commended status in 2011.

A+ Scholars at O’Donnell had slightly greater attainment rates than students in comparison schools for most of the scale scores close to commended, although none of the four students with a 2010 scale score of 729 attained commended status. Further, none of the students starting at more than six questions away from commended status in 2010 advanced to commended status in 2011.

Table 21: Students in the Middle Attaining Commended Status by 2010 Mathematics Scale Score for Hoffman and O’Donnell + Scholars Scale Score

N

Hoffman A+ %

Comparisons N %

O'Donnell A+ N %

637 641 648 656 663 671 679 688 697 707 717 729 742 757

4 11 8 4 10 15 10 17 6 12 20 4 9 4

0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 20.0% 5.9% 0.0% 25.0% 15.0% 0.0% 0.0% 25.0%

328 364 373 411 456 467 488 536 508 585 669 691 709 685

0.0% 0.3% 1.1% 0.2% 1.5% 1.1% 2.5% 2.8% 3.3% 6.5% 5.7% 10.6% 12.0% 19.3%

8 5 11 16 10 7 10 19 17 0 7 4 7 14

Total

134

7.5%

7270

5.9%

135

0.0% 0.0% 0.0% 0.0% 0.0% 14.3% 10.0% 5.3% 5.9%

Comparisons N %

14.3% 0.0% 14.3% 21.4%

395 429 446 478 526 553 598 634 581 671 760 782 814 819

0.0% 0.2% 0.4% 0.2% 1.5% 1.4% 1.8% 2.2% 3.8% 6.6% 6.4% 10.0% 11.9% 18.9%

6.7%

8486

5.8%

SOURCE: Student-level data; TEA

c) Probability of a Student “In the Middle” Attaining Commended Status This section focused on only those students “in the middle.” In other words, students who had passed the TAKS, but who had not yet attained commended status. As shown in the section above, the odds of attaining commended status are heavily dependent on how far the students was away from the commended cut score. This makes common sense, as a student who was only one question from attaining commended status has a far easier time making a large enough gain to achieve commended status than a student who was six or seven questions away.

The analyses in these sections relied on logistic regression analysis, which is a statistical approach commonly used when the outcome variable is binary. In this instance, the outcome variable is whether or not a student achieved commended status—yes or no. Several control variables were used in order to assess the independent effect of being enrolled in an A+ Scholar program. These variables included the student’s own prior test score, the student’s economically disadvantaged status, and various combinations of school characteristics such as the percentage

30

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

of economically disadvantaged students, percentage of Black students, percentage of bilingual students, percentage of mobile students, and percentage of special education students. In some cases, school size was also included. Further, either the number or percentage of a school’s students “in the middle” was included as well. The average scale score for all students in the school was also included. Finally, a binary variable identifying whether a student was an A+ Scholar was included. This variable would indicate whether the students in the A+ Scholar group at that particular school were more or less likely than students in comparison school to attain commended status.

The comparison schools in these analyses include the comparison schools listed throughout this report and also a larger group of schools from across the state with relatively similar student populations.

In the tables below, the statistical significance column indicates if the variable was statistically significantly associated with attaining commended status independent of the effect of all other variables included in the analysis, including the prior year’s test score. The accepted statistical significance level is .05, although a result between .05 and .10 suggests a weaker, but statistically significant relationship.

The effect of being in the A+ Scholar program is indicated by the odds ratio. A result greater than 1.00 indicates a positive relationship while a result less than 1.00 indicates a negative relationship. So, for example, if the result was 1.89, then being an A+ Scholar would be associated with being 89% more likely to attain commended status. Or, if the result was 3.56, a student in the A+ Scholar program would be 3.56 times more likely to attain commended status than other students. Alternatively, if the result was .65, then a student in the A+ Scholar program would be 35% less likely (1.00 – 0.65 = .35) than other students to attain commended status.

A number of control variables were included in the analyses. These included:         

Prior year’s TAKS score; Economically disadvantaged status of the school; Percentage of students in the school: economically disadvantaged; Percentage of students in the school: African-American Percentage of students in the school: enrolled in bilingual education; Percentage of students in the school: mobile; Percentage of students in the school: special education; Percentage of students in the school: passing previous year, but not commended; and, Average scale score for all students in the school in the prior year.

These variables were included in order to control for other factors that might explain why a student attained commended status other than the reform efforts being implemented in the school.

After the inclusion of the independent/control variables, the last variable included in the model was a binary variable indicating if the student was an A+ Scholar in the reform effort. The result for this variable would indicate the effect—positive, negative, or neutral—of the reform effort on the likelihood of the student attaining commended status.

Table 22 includes the overall results of the seven separate regression analyses. The column entitled “stat sig” indicates the p-value—or level of statistical significance—of the result.

Any value equal to or less than .05 indicates that the finding was statistically significant at the generally accepted p < .05 level. The odds ratio provides the magnitude of the effect. Results

31

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

greater than zero indicate that A+ Scholars were more likely than other students to attain commended status after controlling for other factors. For example, A+ Scholars in Caraway were 58.6% more likely to attain commended status than peers in comparison schools. For Mann, the result indicates that A+ Scholars were 10.7 times more likely to attain commended status than peers in comparison schools after controlling for other factors.

When the odds ratio is less than zero, then students were less likely to attain commended status than peers in comparison schools after controlling for other factors. For example, A+ Scholars in Atascocita were 75.8% less likely than peers in comparison schools to attain commended status in mathematics, after controlling for other factors.

Table 22: Logistic Regression Results for Students in the Middle Attaining Commended Status for A+ Scholars School Name

Stat.

Sig.

Odds Ratio

Caraway Hoffman

0.036 0.277

1.586 0.656

Mann Atascocita-Math Atascocita-Reading O'Donnell

0.000 0.046 0.365 0.321

10.747 0.242 0.687 1.462

YES Prep

0.050

4.658

SOURCE: Student-level data, TEA; Analysis: Ed Fuller

A+ Scholars at Caraway were almost 59% more likely than students in comparison school to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics.

A+ Scholars in Atascocita were about 76% less likely than students in comparison school to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics. This is consistent with the table from the previous section that showed very few A+ Scholars in Atascocita achieved even the nominal growth needed to attain commended status and that the students had a much lower attainment rate than students in comparison schools.

Mann A+ Scholars were more than 10 times more likely than students in comparison schools to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics. Again, this result is consistent with the table in the previous section that revealed Mann A+ Scholars had greater commended attainment rates across the different scale score levels.

There was not a statistically significant result for A+ Scholars in Atascocita with respect to attaining commended status in reading. However, the effect shown in math was negative, which was consistent with the results from the previous section.

There were no statistically significant results for A+ Scholars in either Hoffman or Oâ&#x20AC;&#x2122;Donnell. If the effects had been significant, the effect for Hoffman would have been negative and the effect for Oâ&#x20AC;&#x2122;Donnell would have been positive.

Finally, students at YES Prep West were 4.6 times more likely than students in

32

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

comparison schools to achieve commended status. However, the finding was statistically significant with p = .05. The inclusion of other variables in the model move result in the finding not being statistically significant. While this result is not necessarily robust, it does strongly suggest that YES Prep West students were more likely than students in other schools to attain commended status.

B. Vertical Scale Score Progress This section examines the vertical scale score growth of students “in the middle” from 2010 to 2011 for each group of A+ Scholars, students from a set of comparison schools, and students from all other schools. Students “in the middle” is defined as the set of students who passed, but did not attain commended status on the previous year’s test. So, the tables below present the vertical scale score growth from one year to the next for students who passed the previous year’s test, but did not attain commended status.

The set of comparison schools were selected from schools across the state that had similar demographics as the target school. In this analysis, the comparison group is larger than the four or five schools in previous analyses. A larger group of schools was selected for the binary regression analysis from the previous section and the same group of schools was applied to this analysis. All other schools include all other schools in the state other than the target school and setoff comparison schools.

Finally, only students who met the following requirements were included in the analysis:  had a valid score in both years;  took the English version of the TAKS both years; and,  took either the TAKS-regular or TAKS-accommodated tests in both years.

As shown in Table 23, most of the Caraway A+ Scholars scale score groups had score changes that were equal to or greater than the score changes in comparison schools. Only three of the eleven groupings had score changes below the average for comparison schools and these three groups only accounted for only 29% of the students.

Table 23: Vertical Scale Score Growth in Reading from 4th Grade (2010) to 5th Grade (2011) for Caraway A+ Scholars, Comparison Schools, and All Other Schools Scale Score 554 560 570 580 591 603 616 631 647 667 691 All Scores

Caraway A+ Comparison Schools All Other Schools Change N Change N Change 7 115.6 414 70.8 6491 84.4 8 50.6 484 72.3 7635 85.2 17 56.5 539 71.5 8947 85.2 12 92.0 635 70.9 10283 83.2 11 75.7 720 74.1 12128 81.0 13 72.2 886 70.6 14039 78.7 18 65.7 925 66.7 16265 77.9 17 76.4 1053 63.3 18494 75.3 24 34.3 1174 64.0 21266 71.5 18 70.9 1191 55.5 24597 66.6 24 51.8 1280 48.8 27311 59.7 169 64.3 9301 64.1 167459 73.8 SOURCE: Student-Level data, TEA; Analysis: Ed Fuller N

33

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 24, most of the Atascocita A+ Scholars scale score groups in reading had score changes lower than the scale score changes for comparison schools. Indeed, six of the nine groups had score changes below that of comparison schools. Only one scale score group— that with a scale score of 662—had score changes greater than comparison schools.

Table 24: Vertical Scale Score Growth in Reading from 5th Grade (2010) to 6th Grade (2011) for Atascocita A+ Scholars, Comparison Schools, and All Other Schools Scale Score 620 630 640 651 662 675 689 706 725 Total

Atascocita A+ N Change 6 11.2 4 46.8 3 55.3 11 34.7 11 55.5 8 -6.6 12 25.7 10 5.1 9 23.0 74 23.8

Comparison Schools N Change 161 47.3 173 44.5 233 47.3 303 47.8 416 44.6 588 48.5 614 44.9 792 50.8 1001 51.0 4294 48.0

All Other Schools N Change 8127 40.1 9773 39.1 11506 38.6 13990 39.3 16554 39.0 19485 38.9 22594 38.6 25343 39.3 28442 38.8 158168 38.0

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

As shown in Table 25, most of the Atascocita A+ Scholars scale score groups in mathematics had score changes lower than the scale score changes for comparison schools.

Indeed, 11 of the 12 groups had score changes below that of comparison schools. Only one scale score group—that with a scale score of 667—had score changes greater than comparison schools. These results detail why the A+ Scholars had lower growth and a lower probability of attaining commended status.

Table 25: Vertical Scale Score Growth in Mathematics from 5th Grade (2010) to 6th Grade (2011) for Atascocita A+ Scholars, Comparison Schools, and All Other Schools Scale Score 603 606 613 621 629 638 647 657 667 678 691 706 Total

Atascocita A+ N Change 6 -13.5 6 1.7 4 22.0 8 -4.5 3 10.3 6 -14.2 4 -49.5 11 -6.2 9 46.4 17 -3.5 7 -24.0 26 -15.5 107 -5.1

Comparison Schools N Change 105 37.2 101 35.2 147 34.7 175 36.0 179 29.7 221 29.5 263 35.7 293 26.6 350 25.6 438 31.5 544 28.6 653 34.7 3470 31.3

All Other Schools N Change 5822 37.3 6463 39.1 7076 38.6 8161 36.6 8829 35.5 9907 32.6 10813 33.1 11950 31.3 13357 31.6 15217 31.4 16773 30.5 18807 28.8 133176 32.8

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

34

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 26, all but one of the scale score groups for Horace Mann A+ Scholars had score changes greater than the scale score changes for comparison schools. Eleven of the 12 scale score groups, in fact, had score changes equal to or greater than that of comparison schools.

Only one scale score groupâ&#x20AC;&#x201D;that with a scale score of 638â&#x20AC;&#x201D;had score changes lower than comparison schools.

Table 26: Vertical Scale Score Growth in Reading from 5th Grade (2010) to 6th Grade (2011) for Mann A+ Scholars, Comparison Schools, and All Other Schools Scale Score 603 606 613 621 629 638 647 657 667 678 691 706 Total

N 5 3 4 6 7 7 9 8 13 7 5 14 88

Mann A+ Change 50.4 40.3 43.5 27.8 43.1 17.6 58.4 36.9 64.5 46.3 55.8 39.4 44.9

Comparison Schools N Change 256 36.8 300 35.6 319 31.8 393 34.3 399 28.7 407 33.5 454 30.7 494 26.1 513 26.9 603 22.7 624 22.5 707 19.6 5469 27.6

All Other Schools N Change 5680 37.2 6272 39.1 6908 38.8 7951 36.6 8608 35.7 9725 32.5 10620 33.1 11759 31.3 13201 31.6 15077 31.7 16706 30.7 18795 29.2 131304 32.9

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

As shown in Table 27, there were mixed results for the A+ Scholars from Hoffmann.

Four of the scale score groups had greater score changes than at comparison schools, five groups had similar score changes, and five groups had lower score changes. Students at the lower end of achievement tended to have lower gains than students with greater scores. Students in the middle ranges tend to have the greatest score changes.

35

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 27: Vertical Scale Score Growth in Mathematics from 6th Grade (2010) to 7th Grade (2011) for Hoffman A+ Scholars, Comparison Schools, and All Other Schools Scale Score 637 641 648 656 663 671 679 688 697 707 717 729 742 757 All Scores

Hoffman A+ N Change 4 14.8 8 15.4 8 52.3 4 15.3 10 23.3 15 23.7 10 52.8 17 41.4 6 4.7 12 28.7 20 21.0 4 -28.5 9 -2.2 4 8.5 131 24.2

Comparison Schools N Change 386 34.7 431 31.3 440 33.0 458 29.2 515 28.8 560 25.0 560 24.2 616 20.5 589 20.3 672 18.1 767 17.1 776 13.5 818 6.5 780 5.8 8368 20.0

All Other Schools N Change 5231 39.5 5816 41.1 6451 39.9 6764 36.9 7377 36.2 8051 34.2 8720 32.5 9473 30.8 9835 29.1 10659 27.5 11362 25.9 12522 25.1 13376 23.1 14628 21.3 130266 29.8

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

As shown in Table 28, all of the O’Donnell A+ Scholar scale score groups had score changes equal to or greater than the scale score changes for comparison schools. Students at the top end of the score distribution (scale score > 688) did particularly well.

Table 28: Vertical Scale Score Growth in Mathematics from 6th Grade (2010) to 7th Grade (2011) for O’Donnell A+ Scholars, Comparison Schools, and All Other Schools Scale Score 637 641 648 656 663 671 679 688 697 707 717 729 742 757 All Scores

O’Donnell A+ N Change 8 31.3 5 41.4 11 32.0 16 33.4 10 37.0 7 54.1 10 35.2 19 22.2 17 27.2 0 na 7 24.9 4 46.8 7 26.7 14 8.2 135 29.6

Comparison Schools N Change 442 35.2 471 29.7 495 32.2 533 28.4 586 30.3 612 24.9 660 23.0 722 19.1 656 19.4 743 16.1 839 15.5 879 11.4 923 6.3 930 5.9 9491 19.1

All Other Schools N Change 5389 39.1 5960 40.7 6552 39.7 6871 36.7 7469 35.7 8160 33.9 8742 32.4 9485 30.8 9891 29.1 10662 27.3 11336 25.9 12498 25.2 13295 23.2 14488 21.3 130804 29.7

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

36

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

As shown in Table 29, all of the YES Prep West A+ Scholar scale score groups had score changes equal to or greater than the scale score changes for comparison schools. The few students at the bottom end of the score distribution (scale score < 676)) did particularly well.

Table 29: Vertical Scale Score Growth in Reading from 6th Grade (2010) to 7th Grade (2011) for YES Preparatory West A+ Scholars, Comparison Schools, and All Other Schools Scale Score 644 645 655 665 676 688 701 715 731 751 Total

YES Prep West N Change 1 111.0 2 75.0 2 78.5 4 89.0 5 73.0 6 79.0 9 92.1 7 41.7 10 87.5 17 61.6 63 73.9

Comparison Schools N Change 36 67.7 42 65.3 77 75.2 70 63.0 78 67.9 119 47.8 155 52.0 184 50.9 197 45.2 213 52.3 1171 54.5

All Other Schools N Change 7626 43.9 9126 50.6 10726 50.5 12481 47.5 14582 47.8 16981 45.6 18969 45.2 21700 45.8 24700 46.3 27476 43.9 164368 46.3

SOURCE: Student-Level data, TEA; Analysis: Ed Fuller

C. Value-Added Growth The value-added growth component of the evaluation focused on all A+ Scholar students, not just students “in the middle.” The results for students in the middle were not significantly different than the results for all students.

This analysis was included because students can attain commended status without making particularly large growth. Indeed, a student one question below the cut score for commended status could answer one additional question correctly and move from “not commended” to commended, while a student scoring 10 questions below the commended cut score point could answer an additional nine questions correctly but still be designated as not attaining commended status.

In short, this analysis examines whether all A+ Scholars exhibited greater than expected growth. The results are reported using standardized residuals from an ordinary least-squares regression analysis. Standardized residuals communicate whether the students scored lower than, equal to, or greater than regression analysis predicted them to score. The standardized residuals are similar to z-scores, in that positive results indicate greater than expected growth and negative numbers indicate lower than expected growth. Scores close to zero indicate growth that was close to the growth predicted by the regression model.

Using regression analysis is important because it can be used to control for factors outside the control of school personnel. In this case, the regression analysis controlled for prior achievement and a host of school demographic variables. Only by using regression analysis can growth be attributed to the efforts of the school personnel rather than to the demographics of the students.

37

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

1) Methodology The regression analysis focused only on those students who met the following criteria:   



Enrolled consecutively for three years and advancing a grade level each year; Had a valid score in the subject area of interest for all three years; Had a score that was not masked to comply with FERPA for three consecutive years in the subject area of interest (only students in small schools or schools with very low numbers of economically disadvantaged or not economically disadvantaged students would have scores that were masked); and, Enrolled in the same school district for three consecutive years.

The regression analysis controlled for the following variables:  Students economically disadvantaged status in 2011;  Percentage of economically disadvantaged students in the school in 2011;  Percentage of mobile students in the school for 2011;  Percentage of bilingual students in the school in 2011;  Percentage of White students in the school for 2011;  Percentage of Asian students in the school for 2011;  Percentage of special education students;  Percentage of students enrolled three consecutive years that were enrolled in the school in 2011;  Percentage of students enrolled in the district three consecutive years for students enrolled in 2011;  School size in 2011;  Average school tests cores in 2010;  Prior test score in 2010;  Prior test score in 2009; and,  2010 test score above a selected cut point at which gains are difficult to make.

The results of the regression analysis are standardized residuals which describe how many standard deviations the schools’ results were above or below the regression line. In other words, the results provide information on (a) whether a school performed worse than, as well as, or better than predicted by the variables included in the regression analysis and (b) the degree to which the school performed worse than, as well as, or better than predicted. The results are reported in Z-scores.

A negative result suggests the school performed worse than predicted, a result close to 0 suggests the school performed about as well as predicted, and a positive score suggests the school performed better than predicted. The greater the distance from 0, the worse (negative Zscore) the school performed and the greater the distance from 0 (positive Z-score), the better the school performed.

38

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

2) Individual School Results As shown in Table 30, there were three results in which A+ Scholars performed better than predicted, three results in which A+ Scholars performed worse than predicted, and one result in which A+ Scholars performed about as well as predicted.

Table 30: Standardized Residuals (in Z-Scores) from the Regression Analyses for All A+ Scholar Groups District Name Aldine ISD Goose Creek ISD Humble ISD Humble ISD Aldine ISD Alief ISD YES Prep West

School Name Caraway A+ Scholars Mann A+ Scholars Atascocita A+ Scholars Atascocita A+ Scholars Hoffman A+ Scholars O'Donnell A+ Scholars YES Prep A+ Scholars

Target Grade 5 6 6 6 7 7 7

Subject Area Reading Math Math Reading Math Math Reading

Standardized Residual -0.313 0.144 -0.448 -0.297 -0.071 0.123 0.263

A+ scholars in YES Prep, Mann (Humble ISD), and O’Donnell (Alief ISD) all performed greater than predicted. Hoffman in Aldine ISD performed about as well as predicted. Caraway (Aldine ISD) and Atascocita (Humble ISD) performed worse than predicted. Note that two of the three negative results were from Atascocita. Thus, in terms of schools, there were three positive results, one neutral result, and two negative results.

3) A+ Scholar School and Comparison School Results: The results for each school and subject area as well as the results for the comparison schools for each school are shown in the Appendix. Note that a different set of comparison schools will have to be selected for YES Prep, since only one of the comparison schools had enough data to generate results. 

In 5th grade reading, Caraway (Aldine ISD) A+ Scholars performed worse than predicted and under-performed compared to students in three of the four comparison schools.



In 6th grade mathematics, Horace Mann (Goose Creek ISD) A+ Scholars performed better than predicted, but performed better than students in only two of the five comparison schools.



In 6th grade mathematics, Atascocita (Humble ISD) A+ Scholars performed far worse than predicted and under-performed students in all but one of the five comparison schools.



In 6th grade reading, Atascocita (Humble ISD) A+ Scholars performed worse than predicted and under-performed students in all but one of the five comparison schools.

39

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller



In 7th grade math, Hoffman (Aldine ISD) A+ Scholars performed about as predicted and performed better than students in two comparison schools, about the same as students in one comparison school, and worse than students in the other comparison school.



In 7th grade mathematics, O’Donnell (Alief ISD) A+ Scholars performed better than predicted and outperformed four of the five comparison schools.



In 7th grade reading, YES Prep West (YES Prep Public Schools) A+ Scholars scored better than predicted and outperformed students in all of the comparison schools.

4) MAP© Test Scores Two schools also participated in an additional assessment—the Measurement of Academic Progress (MAP©)—produced by the Northwest Evaluation Association (NWEA).

Having a second test available is important to ensure that the students are actually learning the material rather than simply learning to do well on a particular test. Indeed, since TAKS is highstakes for schools and often teachers and principals as well, many schools resort to “teaching to the test” which results in the test scores being inaccurate indicators of what students know and can do. A second test that is low-stakes for all involved serves as a check on the accuracy of the scores from the TAKS. Unfortunately, the MAP© test was not taken by all schools. Only two schools had students take the test twice during the 2010-11 school year so that gains could be calculated. But even in these two schools, the “pre-test” was taken in November, so the gains reflect only gains over 6 to 7 months rather than the full 9 months. TAKS, on the other hand, has an entire calendar year between tests. Thus, the MAP© test and TAKS are not directly comparable. However, the MAP© test still serves a very useful purposes and provides excellent information about A+ Scholar students.

In the analysis of MAP© test, students are placed into two groups with respect to growth (above and below average) and two groups with respect to proficiency (above and below average).

As shown in Table 33, 35.2% of the Mann A+ Scholars had greater than average growth.

Most of these students also had greater than average proficiency. In other words, most of the students making above average growth were also above average in terms of proficiency. Almost 65% of students had below average growth. Remember, however, that this is a norm-referenced test and the Mann students only had 6 or 7 months to show growth and are being compared to students across the nation who had up to 10 months to show growth. Thus, it is to be expected that many Mann students would have below average growth. Of those students having below average growth, 50% were above proficient. So, one-half of students with below average growth were achieving proficient status.

Overall, just over 60% of all students achieved proficiency. Thus, as a group, A+ Scholars scored above the national average on the MAP© test.

40

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 31: MAP© Mathematics Results for Mann A+ Scholars Student

Proficiency

Growth

Below

Above

Below

33.8

31.0

Above

5.6

29.6

As shown in Table 32, almost 33% of the O’Donnell A+ Scholars showed greater than average growth, with the majority of such students achieving greater than average proficiency as well. While about 67% had below average growth, the O’Donnell students had fewer months to show growth than the comparison students. With respect to proficiency, almost 56% achieved proficiency. Thus, as with the Mann A+ Scholars, the group of O’Donnell A+ Scholars performed better than average as compared to other students across the nation.

Table 32: MAP© Mathematics Results for O’Donnell A+ Scholars Student

Proficiency

Growth

Below

Above

Below

42.9

25.0

Above

8.3

23.8

5) Summary of Student Outcomes While the results of the analysis of student outcomes are mixed, one must remember that two of the seven results are from Atascocita—a school that did not have an A+ Coach for the entire year of implementation, as did the other schools. If Atascocita is removed from the analysis because it did not fully implement the reform effort, then the results appear much more favorable with respect to the efficacy of the overall reform effort. Indeed, three of the five schools fully implementing the reform had A+ Scholar students that were statistically significantly more likely to achieve commended status than peers in comparison schools, and the other two schools had students that did not perform statistically significantly different than students in comparison schools.

In addition, three of the five schools fully implementing the reform had greater student vertical scale score growth than comparison schools, and the other two schools had vertical scale score growth that was about the same as in comparison schools.

Finally, three of the five schools fully implementing the reform had greater student valueadded growth than predicted, while one school performed as predicted and the last school performed lower than predicted.

Thus, the evidence strongly suggests that the A+ Challenge Network reform effort, when fully implemented, had a positive effect on student outcomes.

41

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION V: STUDENT SURVEY RESULTS Surveys were administered at some schools during Year One of implementation, and three of the schools had high enough response rates to report results. Overall results are reported below; individual school results are included in the appendix.

The survey covered questions about the student’s teacher and A+ Coach, as well as the student’s perceptions of changes that occurred in the school during the academic year. Survey responses were generally in a six-point likert scale format ranging from “strongly disagree” to “strongly agree”. Percentages equal to or greater than 79.5% are highlighted in green, percentages between 60.0% and 79.4% are shaded in yellow, and percentages less than 60.0% are shaded in red. Even though a number of results are shaded in red, it is important to notice that a majority of students agreed or strongly agreed to each statement.

A. Perceptions of Coach and Coaching Efficacy As shown in Table 33, very large percentages of students perceived that the coaches had a profound positive experience on the students. Indeed, at least 78% of students believed the coach had helped them work harder, stay out of trouble, pay more attention in class, perform better in class, better understand the teacher, improved in both reading and math classes, and feel more confident about school. These were all very important outcomes of the reform effort and the coaches appeared to have been quite successful in working with individual students to achieve these outcomes.

Table 33: Survey Results for Effects of the Coach over Time Strongly Disagree or Disagree

Statement Because of the A+ Coach . . .

I work harder in class I don't get into trouble as much I pay attention in class more I do better in class During class, I understand the teacher better I feel more confident about school I have improved in reading or English class I have improved in math class

2.1% 3.6% 3.3% 3.6% 3.1% 4.5% 3.1% 4.6%

Somewhat Disagree or Agree 10.2% 9.9% 12.0% 12.4% 14.1% 14.4% 17.1% 16.9%

Strongly Agree or Agree 87.7% 86.5% 84.7% 83.9% 82.9% 81.1% 79.8% 78.5%

SOURCE: A+ Scholar Survey of Students, Spring 2011

As shown in Table 34, a majority of students agreed or strongly agreed that the A+ Coach was nice, cared for the students, and was liked by the students. In addition, students were also positive about the behaviors of the coaches. Specifically, students perceived that the coaches helped them to do better in class, talked to students about going to college, and expected students to work hard. As with the first set of questions, this set of questions revealed very positive relationships between coaches and students and positive behaviors of coaches with respect to students.

42

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table 34: Survey Results for Perceptions of the Coach Strongly Disagree or Disagree

Statement The A+ Coach: helped me do better on tests was nice and easy to work with talked about me going to college helped me do better in class cared about me explained things in an easy to understand way was someone I I liked working with expected me to work hard

4.9% 4.4% 3.3% 4.2% 8.8% 10.6% 15.8% 14.6%

Somewhat Disagree or Agree 7.2% 19.2% 24.6% 25.4% 26.8% 31.6% 30.1% 31.9%

Strongly Agree or Agree 87.9% 76.4% 72.2% 70.4% 64.4% 57.8% 54.2% 53.5%

SOURCE: A+ Scholar Survey of Students, Spring 2011

B. Perceptions of Teacher As shown in Table 35, students were also generally positive about their teachers and teachersâ&#x20AC;&#x2122; behaviors. Over 70% of students thought that the teachers made the class interesting and cared about the students. Over 60% of students believed that the teacher explains things in an easy-to-understand way and reminds students to think about the future.

Slightly less than 50% of students agreed or strongly agreed that the teacher talked to them about going to college and expected students to work hard. Indeed, 20% of students disagreed or strongly disagreed that teachers expected students to work hard.

Finally, and most importantly, almost 61% of students agreed or strongly agreed that the teacher had improved during the semester. While there is no other data to triangulate with this piece of information, the fact that students perceived improvements in teacher practice is an important outcome.

Table 35: Survey Results for Perceptions of Teacher Strongly Disagree or Disagree

Statement My teacher . . . makes the class interesting cares about me explains things in an easy-to-understand way reminds me that I need to think about my future. has become a better teacher during this semester talks to me about going to college expects me to work hard

SOURCE: A+ Scholar Survey of Students, Spring 2011

43

3.9% 8.6% 6.4% 4.5% 14.1% 13.4% 20.6%

Somewhat Disagree or Agree 24.7% 20.3% 28.9% 31.2% 25.3% 37.0% 32.5%

Strongly Agree or Agree 71.4% 71.1% 64.6% 64.3% 60.6% 49.6% 47.0%

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

C. Perceptions of School Administration Finally, as shown in Table 36, slightly greater than 50% of students believed that the school administration had increased talk about the importance of going to college and exhibited greater concern about student achievement. Similarly, slightly greater than 50% of students believed that school had become safer for the individual respondent and her or his friends.

Almost 75% of students believed that the school had become a “better place to be” over the course of the year.

Table 36: Survey Results Strongly Somewhat Disagree Disagree or or Disagree Agree Compared to the beginning of the school year, the school administration: Talks more about the importance of going to 13.2% 28.6% college More concerned with how well I do in school 14.1% 29.9% Compared to the beginning of the school year, the school seems to be: a better place for students to be 4.2% 20.8% safer for me and my friends 15.8% 30.1% Statement

Strongly Agree or Agree 58.2% 56.0% 74.9% 54.2%

SOURCE: A+ Scholar Survey of Students, Spring 2011

D. Perceptions of School Students were also asked about how they view attending school. This question had five answers and, unfortunately, some students answered with multiple perceptions. Thus, the sum of responses exceeds 100%.

As shown in Table 37, 14% of respondents said they were bored at school and 9% said they disliked school. Alternatively, 16% reported they were excited about school. Finally, 45% believed that school was good while almost 38% said school was okay.

Table 37: Student Perceptions of School Statement I am bored I dislike school I am excited about school School is good School is ok

Percent 14.0 9.2 16.2 45.4 37.5

SOURCE: A+ Scholar Survey of Students, Spring 2011

E. Perceptions of A+ Scholar Reform Finally, students were asked about their perceptions of the A+ Scholar reform effort. A small number of students provided multiple responses, so the data was cleaned by including only each student’s most positive response.

44

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

About 40% of the students were unsure of the program or were unaware of the program.

This suggests that more discussion needs to occur between the adults and students regarding the reform and the reform strategies. On the other hand, about 55% of students were either interested in the program or excited about the program. This suggests that students who know about the program are likely to view it in a favorable light.

Table 49: Student Perceptions of A+ Scholar Reform Response Don't know about it Unsure Don't like Interested in it Excited about it Total

Number 83 69 6 97 97 352

Percent 23.6% 19.6% 1.7% 27.6% 27.6% 100.0%

SOURCE: A+ Scholar Survey of Students, Spring 2011

Summary of Student Survey Results Overall, a large majority of students agreed or strongly agreed that the A+ reform has improved student behaviors. While this data does not have any other data to substantiate the perceptions of the students, the large percentage of students agreeing or strongly agreeing and the consistency in responses across the three schools strongly suggests that the reform modelâ&#x20AC;&#x201D;in particular, the coaching aspect of the modelâ&#x20AC;&#x201D;had very positive effects on student behavior. In addition, students also held favorable perceptions about the coaches, teachers, school administration, and the school overall. Thus, at least from the perspective of the students, the reform appears to have had a large and positive impact on students, teachers, and the culture of the school.

45

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

SECTION VI: CONCLUSION Overall, the data collected and analyzed for the first year evaluation strongly suggests that the Houston A+ Challenge Network reform effort is having a positive effect on the likelihood of students attaining commended status, student growth as measured through valueadded analyses, and student behavior as measured by student self-reports. The effects, however, are unevenly distributed across schools.

Additional efforts need to be undertaken to ensure that all of the reform components are fully implemented in each school, and additional information needs to be collected to determine the causes of some of the low performance so that the reform can be modified in these schools.

Further, the available data suggest that the coach is the critical element in the reform effort. Thus, great attention must be focused on choosing a high-quality coach and ensuring that the coach provides high-quality support to teachers and students.

In final conclusion, the reform efforts have had a strong Year One start despite not being fully operational until the middle of the academic year. These initial results are very promising, and the reform effort should be fully supported by school and district personnel as well as funders and policymakers References Fullan, M. (2001). Leading in culture of change. San Francisco: Jossey-Bass.

46

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

APPENDIX As shown in Table A-1, A+ Scholars at Caraway were almost 59% more likely than students in comparison school to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics.

Table A-1: Logistic Regression Results for Students â&#x20AC;&#x153;In the Middleâ&#x20AC;?

Attaining Commended Status in Mathematics for Caraway A+ Scholars Variable Name Student: Scale Score 2010 Student: Eco disadv School: % eco dis School: % Black School: % Bilingual School: % Mobile School: % Special Education School: % in Middle School: Avg Scale Score Caraway

Stat.

Sig. 0.000 0.000 0.000 0.540 0.000 0.001 0.077 0.223 0.667 0.036

Odds Ratio 1.023 0.722 0.980 1.001 1.014 0.974 1.025 1.010 0.999 1.586

As shown in Table A-2, A+ Scholars in Atascocita were about 76% less likely than students in comparison school to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics. This is consistent with the table from the previous section that showed very few A+ Scholars at Atascocita achieved even the nominal growth needed to attain commended status and that the students had a much lower attainment rate than students in comparison schools.

Also shown in Table A-2, A+ Scholars at Horace Mann were more than 10 times more likely than students in comparison school to attain commended status after controlling for prior scores, economically disadvantaged status, and school characteristics. Again, this result is consistent with the table in the previous section that revealed A+ Scholars at Mann had greater commended attainment rates across the different scale score levels.

47

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table A-2: Logistic Regression Results for Students “In the Middle” Attaining Commended Status in Mathematics for Atascocita and Mann A+ Scholars

Variable Name Student: Scale Score 2010 Student: Eco disadv School: % eco dis School: % Black School: % Bilingual School: % Mobile School: % Special Education School Size School Size Squared School: # Students in the Middle School Effect

Atascocita Stat.

Odds Sig. Ratio 0.000 1.037 0.000 0.585 0.027 1.025 0.000 1.132 0.094 0.975 0.651 1.019 0.811 1.009 0.949 1.000 0.994 1.000 0.776 1.001 0.046 0.242

Mann Stat.

Odds Sig. Ratio 0.000 1.026 0.003 0.663 0.000 0.949 0.003 0.971 0.000 1.052 0.799 1.003 0.200 0.973 0.000 0.992 0.000 1.000 0.004 0.991 0.000 10.747

As shown in Table A-3, there was not a statistically significant result for A+ Scholars in Atascocita with respect to attaining commended status in reading. However, the effect shown in mathematics was negative, which was consistent with the results from the previous section.

Table A-3: Logistic Regression Results for Students “In the Middle” Attaining Commended Status in Reading for Atascocita A+ Scholars Variable Name Student: Scale Score 2010 Student: Eco disadv School: % eco dis School: % Black School: % Bilingual School: % Mobile School: % Special Education School Size School Size Squared School: # Students in the Middle Atascocita

Stat.

Sig. 0.000 0.000 0.878 0.815 0.055 0.293 0.992 0.546 0.793 0.134 0.365

Odds Ratio 1.026 0.722 1.002 1.002 1.046 1.028 1.000 1.001 1.000 0.998 0.687

As shown in Table A-4, there were no statistically significant results for A+ Scholars at either Hoffman or O’Donnell. If the effects had been significant, the effect for Hoffman would have been negative and the effect for O’Donnell would have been positive.

48

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table A-4: Logistic Regression Results for Students “In the Middle” Attaining Commended Status in Reading for Hoffman and O’Donnell A+ Scholars

Variable Name Student: Scale Score 2010 Student: Eco disadv School: % eco dis School: % Black School: % Mobile School: % Bilingual School: % in Middle School: Avg Scale Score School Effect

Hoffman Stat.

Odds Sig. Ratio 0.000 1.033 0.065 0.770 0.947 0.999 0.087 0.992 0.146 1.021 0.919 1.001 0.000 1.105 0.030 0.993 0.277 0.656

O'Donnell Stat.

Odds Sig. Ratio 0.000 1.034 0.000 0.648 0.008 0.977 0.037 0.992 0.818 0.997 0.139 1.011 0.000 1.046 0.026 0.994 0.321 1.462

As shown in Table A-5, students at YES Prep West were 4.6 times more likely than students at comparison schools to achieve commended status. However, the finding was statistically significant with p = .05. The inclusion of other variables in the model move result in the finding not being statistically significant. While this result is not necessarily robust, it does strongly suggest that YES Prep West students were more likely than students in other schools to attain commended status.

Table A-5: Logistic Regression Results for Students “In the Middle” Attaining Commended Status in Reading for YES Prep West A+ Scholars Variable

Stat.

Odds

Name

Sig.

Ratio

Scale Score Gr6 2010 Eco disadv Sch: % eco dis Sch: % Black Sch: % Bilingual Sch: % Mobile Sch: % Special Educ Sch: % in Middle Sch: Avg Scale Score

0.000 0.012 0.454 0.248 0.413 0.045 0.821 0.438 0.529

1.028 0.575 1.008 0.992 0.987 1.029 0.973 0.109 0.992

YES Prep West

0.050

4.658

49

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table A-6: Student Survey Results for A+ Scholars at Caraway Statement

Strongly Disagree

Disagree

Somewhat Disagree

Somewhat Agree

Because of the A+ Coach . . .

I have improved in reading or 2.5% 1.7% 4.2% 11.0% English class I have improved in math class 0.9% 2.6% 1.7% 9.4% I feel more confident abt 3.3% 0.8% 2.5% 11.7% school I do better in class 4.2% 1.7% 2.5% 6.7% I pay attention in class more 4.2% 0.0% 0.0% 10.1% I work harder in class 2.5% 0.0% 0.8% 7.5% During class, I understand the 3.4% 0.8% 2.5% 11.8% teacher better I don't get into trouble as much 2.5% 1.7% 2.5% 6.7% The A+ Coach . . explained things in a way that makes them easy to 7.8% 4.3% 12.2% 25.2% understand expected me to work hard 3.4% 5.9% 8.5% 22.9% was nice and easy to work 0.8% 4.2% 3.3% 15.0% with was someone I liked working 5.2% 6.9% 3.4% 19.8% with cared about me 3.5% 3.5% 4.4% 21.9% helped me do better in class 3.3% 4.2% 5.0% 17.5% helped me do better on tests 3.4% 1.7% 2.5% 3.4% talked about me going to 4.2% 0.8% 4.2% 19.3% college My Teacher . . reminds me that I need to 4.2% 0.0% 2.5% 30.0% think about my future. talks to me about going to 5.1% 2.5% 3.4% 26.3% college explains things in a way that makes them easy to 3.4% 6.0% 6.0% 20.5% understand expects me to work hard 4.2% 16.1% 10.2% 26.3% makes the class interesting 2.5% 0.8% 2.5% 21.8% cares about me 5.0% 3.4% 7.6% 11.8% has become a better teacher 5.8% 12.5% 5.0% 22.5% during this semester Compared to the beginning of the school year, the school seems to be a . . . better place for students to be 3.4% 3.4% 3.4% 16.1% safer place for me and my 5.2% 6.9% 3.4% 19.8% friends Compared to the beginning of the school year, the school seems to . . .

Talk more about the 4.2% 5.9% 10.1% 18.5% importance of going to college More concerned with how 6.0% 4.3% 11.1% 22.2% well I do in school

50

Agree

Strongly Agree

43.2%

37.3%

41.9%

43.6%

47.5%

34.2%

45.4% 42.0% 39.2%

39.5% 43.7% 50.0%

43.7%

37.8%

37.5%

49.2%

27.0%

23.5%

39.0%

20.3%

39.2%

37.5%

43.1%

21.6%

39.5% 47.5% 36.1%

27.2% 22.5% 52.9%

38.7%

32.8%

39.2%

24.2%

47.5%

15.3%

44.4%

19.7%

29.7% 37.8% 37.0%

13.6% 34.5% 35.3%

40.8%

13.3%

44.9%

28.8%

43.1%

21.6%

39.5%

21.8%

38.5%

17.9%

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table A-7: Student Survey Results for A+ Scholars at Hoffman Statement

Strongly Disagree

Disagree

Somewhat Disagree

Because of the A+ Coach . . .

I have improved in reading or 0.0% 0.0% 2.0% English class I have improved in math class 0.0% 0.0% 0.0% I feel more confident abt 0.0% 1.0% 4.9% school I do better in class 0.0% 0.0% 2.9% I pay attention in class more 0.0% 1.0% 2.9% I work harder in class 0.0% 0.0% 1.9% During class, I understand the 0.0% 1.0% 5.0% teacher better I don't get into trouble as much 0.0% 1.0% The A+ Coach . . explained things in a way that 5.2% 3.1% 3.1% makes them easy to understand expected me to work hard 2.7% 11.7% 11.7% was nice and easy to work with 2.7% 0.0% 1.8% was someone I liked working 7.2% 7.2% 9.0% with cared about me 3.6% 3.6% 7.1% helped me do better in class 0.0% 0.0% 7.8% helped me do better on tests 3.6% 0.0% 0.0% talked about me going to 0.0% 0.0% 2.9% college My Teacher . . reminds me that I need to think 1.0% 1.9% 4.9% about my future. talks to me about going to 7.1% 6.2% 8.8% college explains things in a way that 2.7% 0.9% 9.9% makes them easy to understand expects me to work hard 3.6% 13.6% 10.0% makes the class interesting 1.0% 0.0% 5.8% cares about me 4.5% 1.8% 3.6% has become a better teacher 5.4% 5.4% 6.3% during this semester Compared to the beginning of the school year, the school seems to be a . .. better place for students to be 0.0% 0.0% 1.9% safer place for me and my 7.2% 7.2% 9.0% friends Compared to the beginning of the school year, the school seems to . . .

Talk more about the 2.0% 7.8% 9.8% importance of going to college More concerned with how well 8.8% 7.1% 5.3% I do in school

51

Somewhat Agree

Agree

Strongly Agree

18.6%

41.2%

38.2%

25.0%

37.5%

37.5%

16.5%

44.7%

33.0%

10.8% 16.3% 15.5%

43.1% 40.4% 46.6%

43.1% 39.4% 35.9%

11.9%

44.6%

37.6%

14.6%

43.7%

40.8%

25.0%

33.3%

30.2%

19.8% 16.2%

28.8% 44.1%

25.2% 35.1%

28.8%

33.3%

14.4%

22.3% 25.2% 9.0%

39.3% 35.9% 43.2%

24.1% 31.1% 44.1%

31.1%

35.0%

31.1%

30.1%

35.9%

26.2%

41.6%

30.1%

6.2%

20.7%

36.9%

28.8%

20.0% 20.4% 21.4%

30.0% 47.6% 41.1%

22.7% 25.2% 27.7%

13.4%

41.1%

28.6%

26.2%

37.9%

34.0%

28.8%

33.3%

14.4%

16.7%

43.1%

20.6%

25.7%

29.2%

23.9%

Houston A+ Challenge Evaluation

Fall 2011

Ed Fuller

Table A-8: Student Survey Results for A+ Scholars at Oâ&#x20AC;&#x2122;Donnell Statement

Strongly Disagree

Disagree

Somewhat Disagree

Because of the A+ Coach . . .

I have improved in reading or 2.8% 1.9% 0.9% English class I have improved in math class 3.6% 2.7% 5.4% I feel more confident abt 4.5% 3.6% 0.9% school I do better in class 3.7% 0.9% 0.9% I pay attention in class more 3.6% 0.9% 0.9% I work harder in class 3.6% 0.0% 1.8% During class, I understand the 2.8% 0.9% 0.0% teacher better I don't get into trouble as much 4.5% 0.9% 0.0% The A+ Coach . . explained things in a way that 5.5% 5.5% 11.9% makes them easy to understand expected me to work hard 8.8% 11.5% 10.6% was nice and easy to work with 2.7% 2.7% 5.4% was someone I liked working 11.9% 9.2% 10.1% with cared about me 7.9% 4.4% 1.8% helped me do better in class 2.8% 1.9% 4.6% helped me do better on tests 5.1% 0.8% 0.0% talked about me going to 2.7% 1.8% 1.8% college My Teacher . . reminds me that I need to think 5.5% 0.9% 1.8% about my future. talks to me about going to 10.7% 8.9% 7.1% college explains things in a way that 5.3% 0.9% 3.5% makes them easy to understand expects me to work hard 7.7% 16.2% 10.3% makes the class interesting 6.4% 0.9% 3.6% cares about me 5.1% 5.9% 0.8% has become a better teacher 5.2% 7.8% 6.9% during this semester Compared to the beginning of the school year, the school seems to be a . .. better place for students to be 3.6% 1.8% 0.0% safer place for me and my 11.9% 9.2% 10.1% friends Compared to the beginning of the school year, the school seems to . . .

Talk more about the 7.7% 12.5% 6.7% importance of going to college More concerned with how well 9.3% 6.8% 4.2% I do in school

52

Somewhat Agree

Agree

Strongly Agree

15.0%

44.9%

34.6%

17.0%

29.5%

42.0%

7.3%

54.5%

29.1%

13.8% 6.4% 3.6%

41.3% 50.0% 45.5%

39.4% 38.2% 45.5%

11.2%

48.6%

36.4%

6.3%

47.7%

40.5%

16.5%

37.6%

22.9%

22.1% 16.1%

24.8% 38.4%

22.1% 34.8%

19.3%

38.5%

11.0%

22.8% 16.7% 6.8%

37.7% 44.4% 34.7%

25.4% 29.6% 52.5%

15.2%

50.9%

27.7%

24.5%

41.8%

25.5%

24.1%

41.1%

8.0%

26.3%

35.1%

28.9%

20.5% 20.0% 16.1%

30.8% 41.8% 33.1%

14.5% 27.3% 39.0%

21.6%

42.2%

16.4%

15.5%

45.5%

33.6%

19.3%

38.5%

11.0%

24.0%

36.5%

12.5%

21.2%

33.1%

25.4%