Presentacion Margaret Burchinal by Gestión Escolar

Quality Rating and Improvement Systems (QRIS) in the USA: Current Status, Approaches, and Evaluation Margaret Burchinal University of North Carolina June 29, 2011

Quality Rating and Improvement Systems (QRIS): Current Status, Approaches, and Evaluation Kathryn Tout, Ph.D., Child Trends Gretchen Kirby, M.P.P, Mathematica Kimberly Boller, Ph.D., Mathematica September 20, 2010

Why QRIS • Child care experiences can improve children’s cognitive and social skills – linked to quality of care • Growing evidence that there might be “active ranges” in quality

Logic Model for Quality Rating and Improvement Systems

Challenges Underlying Assumptions

Resources/ Inputs

Activities

Outputs

Outcomes

Long-term Impact

Levels to Consider: •Systems •Markets •Programs •Families •Children

Contextual Factors Evaluation

QRIS Structure & Goals • Typical QRIS components: – – – – –

Program standards Monitoring/quality assurance Financial incentives Quality improvement supports Dissemination of ratings

• Goals: – Improve quality – Maintenance of quality over time – Increase demand for quality

Information Source: 2009 QRIS Compendium • Funded by the Office of Planning, Research & Evaluation, ACF, DHHS • Project design informed by an Expert Panel • Data collection occurred in two waves: – Review of public documents – Phone interviews with QRIS administrators to fill in missing information

• Compendium provides comparative information across the 26 QRIS • Individual profiles are available for each QRIS

QRIS Status: Fall 2009 MAINE MINNESOTA 5 pilot area

OREGON

VT NH NEW YORK

IOWA

PENN OHIO ILLINOIS

COLORADO CALIFORNIA Los Angeles County pilot

MISSOURI

DELAWARE

•INDIANA

MARYLAND KENTUCKY TENNESSEE

VIRGINIA 15 pilot areas

WASHINGTON DC

NORTH CAROLINA

OKLAHOMA NEW MEXICO MISS

Key Blue = statewide program Green = pilot or county-wide program

LOUISIANA

• 26 QRIS included in QRS Compendium (Fall, 2009) • New/updated QRIS since then: ID, MA, MT, RI • Most other states conducting field tests or in an exploration/design phase

FLORIDA

Miami Dade and Palm Beach Counties

QRIS Across the U.S. • Various early care and education programs are eligible to participate in the majority of QRIS: – – – – –

Licensed child care centers (26 of 26) Head Start/Early Head Start programs (24 of 26) Licensed family child care programs (23 of 26) Pre-kindergarten programs (18 of 26) School-age (16 of 26)

• Majority of QRS are voluntary (20 of 26) • Density of program participation in QRS is moderate

Density of QRIS Program Participation Percent range of all programs participating in the QRIS

Number of QRIS*

60% or greater

6: New Mexico, North Carolina, Oklahoma, Oregon, Pennsylvania, Tennessee

30 to 59%

3: California (LA County), District of Columbia, Florida Miami Dadea Louisiana

10 to 29%

9: Colorado, Florida (Palm Beach)b, Iowa, Kentucky, Minnesota, Mississippi, Ohio, Vermont, Virginia

Less than 10%

4: Delaware, Maryland, New Hampshire

*Out of 22 QRS with data available. a=child care centers; b=percent of programs serving subsidized children

QRIS Program Standards: 13 Quality Categories Quality Categories

Number of QRIS*

Licensing compliance

Staff qualifications

Environment

Family partnerships

Administration and management

Accreditation

Curriculum

Ratio and group size

Child assessment

Provisions for special needs

Cultural and linguistic diversity

Community Involvement

Health and safety

*Inclusion of quality categories for child care centers

Rating Structure and Levels • Rating Structure – Building block: • all the standards in one level must be met before moving on to the next higher level (13 of 26)

– Points: • points earned for each standard and combined to determine rating level (5 of 26)

– Combination (6 of 26)

• Rating Levels – Licensing requirements are typical starting point (for first level) – Half of QRIS use 5 levels (13 of 26) – Others use 3 or 4 levels

Quality Standards: Director, Staff and Provider Qualifications • All 26 QRIS include quality standards for directors, staff and family child care providers* • Standards may cover: – – – –

Educational qualifications Specialized training Attainment of specific credentials or certificates Experience

• May include completion of a professional development plan *if included in the QRS

Educational Qualification Indicators  Primary caregivers - levels of education recognized in QRS range from high school diplomas to college credits to Ph.D.s  May have require different levels of education at different levels of QRIS  May increase proportion of caregivers with degrees at higher levels of QRIS  Sometimes require degree/certification linked to early childhood education

Educational Qualification Indicators cont.

Separate set of indicators for directors  higher levels of educational attainment than for teachers;  includes years of experience as well  can include a Director’s Credential or educational background in administration and management

Can have another a set of indicators for other staff (such as Assistant Teachers and Curriculum Coordinators)

Quality Standards: Training Requirements • Typically specify minimum hours of annual training at each level – Can have requirements for content, training level, and/or linkage to professional development plan

• Examples of content for training specified in some QRS: – State early learning guidelines – Curriculum planning – Observation and assessment

Content of Training • 23 of 26 QRIS reported that training is available and/or aligned with the QRIS • 18 QRIS reported on the content of the training – 13 – navigation of the QRIS – 15 – environment assessment – 12 – language/literacy, 12 – specific curriculum, 11 – business practices, 10 social/emotional dev., 10 – safety, 9 – child assessment

Quality Standards: Program Administration and Management • Most have indicators related to program administration and management (23 of 26)* – Can include indicators of administration and management such as use of performance evaluations and business plans/philosophy – Wide variation in inclusion of other indicators like: • Staff benefits • Salary scales • Communication strategies with staff *Number of QRSs is lower for family child care programs

Quality Standards: Learning Environment, Curriculum &Child Assessment Global Quality: Most measures learning environment with the Environment Rating Scales Teacher-child Interactions: Growing number use the CLASS (VA and MN) Curriculum: (14 of 26)  Linked to early learning standards and training

Child Assessments (11 of 26)

*Number of QRSs is lower for family child care

QRIS Structure & Goals • Typical QRIS components: – – – – –

Program standards Monitoring/quality assurance Financial incentives Quality improvement supports Dissemination of ratings

• Goals: – Improve quality – Maintenance of quality over time – Increase demand for quality

QRIS Evaluations • Evaluation is a critical tool for: – documenting features of implementation – validating the tools used in QRIS – assessing whether QRIS are producing desired outcomes for children, families, programs, communities and the early childhood system

• 19 of 26 QRIS have had some evaluation conducted* – Size and scope of evaluations vary – Ongoing (10 ) or periodic (9) *updated number since release of the Compendium

Align Evaluation Design with QRIS Stage • Pilot and scale up – Allows for testing approach and assessment of changes in outcomes

• Early operation stage (2-5 years) – Opportunity for large scale longitudinal evaluation

• Mature operation stage (5+ years) – Opportunity to assess trends in quality improvement and outcomes

• At all stages, data can be used for program improvement

Implementation Research • Implementation studies examine the extent to which the QRS is being implemented as planned and can identify emerging areas of success or of challenge in service delivery and assessment of quality. – Tennessee study (Pope, Denny, Homer, & Ricci, 2006) – Colorado study (Zellman, Perlman,Le, & Setodji, 2008)

Financial Incentives • • • •

Tiered reimbursement Quality awards Improvement supports Other incentives: – Scholarships – Wage enhancements – Retention bonuses

Tiered Reimbursement • 18 of 26 QRIS include tiered reimbursement • 3 QRIS offer higher rate to accredited programs • 7 offer a flat rate increase per subsidized child based on star level • 8 QRIS offer a percentage increase that is added to the maximum rate – Lower levels – 3% to 5% – Higher levels – 15% to 25%

• More analysis is needed to know which rate differentials and structures are effective

Quality Awards • 11 of 26 QRIS offer quality awards • Awards are offered based on the size of the facility and level of quality achieved • Awards may be based on the density of subsidized children served • 5 QRIS (DE, IN, KY, VT, VA) offer one-time merit or achievement awards upon receipt of star rating ($250 - $2,500)

Quality Awards (cont.) • 6 QRIS (FL Miami Dade, FL Palm Beach, IA, KY, OH, PA) offer awards to support achievement of quality on an annual or biennial basis (typical range $100- $5,000) • 2 QRIS (OH, PA) offer larger awards – Ohio – uses a base rate and an annual payment to programs based on size, quality level and number of subsidized children (ex. $5,000 + $10,000) – Pennsylvania – merit award based on size, quality level and density of vulnerable children served (ex. $25,000 for above 26% density, star 4, medium center) – OH has requirements for how the funds are used – PA has a best practices document for the funds

Improvement Supports • Similar to quality awards but are tied directly to specific facets of quality improvement – A Quality Improvement Plan is often used as the basis for determining what improvement will be addressed

• 18 of 26 QRIS provide these supports • The majority (14) do not specify the typical amount available but note that it will align with items in the Quality Improvement Plan • It would be useful to know more about how these funds are typically used

Other Incentives • • • •

18 of 26 QRIS offer access to scholarships 5 of 26 QRIS offer retention bonus 6 of 26 QRIS offer wage enhancement These incentives are available to individual staff and are not usually limited to staff in programs participating in a QRIS

Onsite Assistance: Quality improvement supports • All 26 QRIS reported that onsite assistance is available to programs for quality improvement • 18 QRIS provided information about the content – 14 – navigation of the QRIS – 14 – environment assessment – 12 – safety, 12 – business practices, 10 social/emotional dev., 10 – child assessment, 10 – specific curriculum, 9 – language and literacy

• Majority of QRIS reported that the frequency, length, and duration of onsite assistance varied by the needs of the program

Evaluations • Summative • Independent Evaluations – Overall quality – Child outcomes

Q-DOT: Quality, Dosage, & Thresholds • Secondary data analysis funded by OPRE • Examined 8 large US child care studies to look for threshold in associations between school readiness skills and – Quality – Amount or Dosage

Q-Dot • Major findings – Evidence for thresholds in quality-outcomes for • WHEN THERE WAS A THRESHOLD - ACTIVE RANGE IN HIGHER QUALITY CLASSROOMS • INSTRCTIONAL QUALITY AND ACADEMIC SKILLS

– Evidence that more specific measures of quality predict aligned outcomes more strongly than more global quality measures – Evidence for dosage • Head Start: children with 2 years start school with better skills than children with one year

Validation Studies • Validation studies examine how well foundational elements are functioning. – Oklahoma study (Norris, Dunn & Eckert, 2003) – North Carolina study (Bryant, 2001) – Colorado study (Zellman et al, 2008)

• In OK and NC, studies determined QRIS levels were designating differences in quality • In CO, patterns linking ratings and measures were not observed • Further validation studies are necessary to increase the evidence base and inform revision of rating system design.

Quality Improvement Studies â&#x20AC;˘ Evaluations have examined quality improvement over time in Colorado (Zellman et al., 2008), Oklahoma (Norris et al., 2003), Pennsylvania (Barnard, Smith, Fiene, & Swanson, 2006), Colorado (Zellman et al., 2008), and Tennessee (Cheatam, Pope, & Myers, 2005; Pope, Denny, Magda, Homer, & Cunningham, 2007). â&#x20AC;˘ In these studies, quality in participating programs improved over time (though the changes were not always statistically significant). The research designs do not permit causal statements about the role of the QRS in affecting quality improvement, but the results are positive.

Quality Improvement: Impact Study WA Seeds To Success • Study of the effects of coaching and quality improvement grants on child care quality • Half received coaching, quality improvement grants, and professional development funds (“treatment group”); half received professional development funds only (“control group”) Findings: • Seeds significantly improved observed quality of child care • Due to the building blocks scoring system, large improvements in observed quality did not translate into higher Seeds scores

Example from Washington State

WA Seeds To Success Quality Categories The Seeds model includes four areas: Curriculum & Learning Environment

Professional Development & Training

Management Practices

Family Relationships & Community Partnerships 37

What was the Modified Field Test Evaluation? Impact Study  Study of the effects of coaching and quality improvement grants on child care quality  Half received coaching, quality improvement grants, and professional development funds (“treatment group”); half received professional development funds only (“control group”) Implementation Study  Study of services the providers in the treatment group received and what the site coordinators, coaches, and providers thought about those services

How are Seed Scores calculated? Curriculum & Learning Environment ERS and Ratio Score (1 to 5) Professional Development & Training Education Score (1 to 4) Experience Score (1 to 4) ------------------------------------------------------

Lowest score = Seed Score 39

Implementation Findings  On average, the amount of coaching received 

 

adhered to the Seeds model; however, participants had a range of experiences Most QI grants were used for making safety improvements and purchasing materials Some providers accessed professional development, but barriers still exist Providers saw coaches as partners

Impact Study Design  



Key Question: What was the impact of Seeds on quality of child care and on provider education and child care experience? Impact was defined as the difference between treatment and control groups at follow-up, controlling for baseline differences Data sources: – Baseline and follow-up director, teacher, and family child care provider questionnaires – Baseline and follow-up classroom and family child care provider observations

Impact Findings: Services and Supports 



Treatment group family providers and teachers received more coaching services than controls Seeds improved enrollment of center teachers, but not family child care providers, in education and training services – Lead teachers were much more likely to take college courses Possibly due to time constraints, Seeds did not have a positive impact on providers’ and teachers’ completion of degrees and CDAs

Coaching and supports did not significantly increase centersâ&#x20AC;&#x2122; Seed scores

Seeds Component Score

Treatment Control

0 Final Score (n.s.)

ERS

Professional Development (n.s.)

Education (n.s.)

Experience (n.s.)

Source: Follow-Up Child Care Observation, Family Child Care Provider Interview, and Teacher Questionnaire completed in winter 2010. Sample Size: 14 n.s. = not statistically significant.

Conclusions

 Implementing the Seeds model is feasible – Coaching and incentives are sufficient to motivate participation – Overall, providers found coaching very helpful

 Seeds significantly improved observed quality 

of child care Due to the building blocks scoring system, large improvements in observed quality did not translate into higher Seeds scores – Limited timeframe for education outcomes to be observed

Reporting



The full report is available at the Mathematica and Thrive websites. – http://thrivebyfivewa.org/qris.html



Shorter briefs will be available in August

Linkages Between Ratings and Children’s Outcomes • Two states have reported findings on ratings and children’s outcomes – Colorado (no consistent evidence) – Missouri • For all children, greater gains from fall to spring on teacher-rated social behavior among children in high quality programs • Evidence that children in poverty made greater gains from fall to spring on teacher-rated social behavior and on direct assessment of vocabulary and knowledge of letters/sounds (sample sizes in some cells are quite small)

• New studies are underway in IN and MN that will use similar evaluation strategies

New Research Reports to Watch For • State evaluations in 2010 and 2011 – – – – – – – –

• • • •

Georgia (FPG, UNC Chapel Hill) Indiana (Purdue University) Maine (University of Southern Maine) Miami (FPG, UNC Chapel Hill) Minnesota (Child Trends) New York (NCCP) Pennsylvania (Pennsylvania State) Virginia (University of Virginia)

Performance-based Accountability Systems (RAND) QRS Assessment (Mathematica and Child Trends) Quality Improvement in 17 States (NCCP) Coaching in QRIS (Child Trends)

Questions • What resources related to QRIS research and evaluation are needed in your state? • How can research support the inclusion of special populations (special needs, dual language learners) in QRIS? What is needed? • What unintended consequences of QRIS should be examined?

Implications for QRIS • Are all levels created equally? – Where is the crucial “threshold” for child outcomes? – What is the message to parents? Policymakers? Politicians?

• How can the standards more closely reflect practices that will support learning?

Implications for QRIS • What “dose” is required? • Quality improvement – Intensive, intentional efforts to improve quality – How long does a child need to participate in an improved program before we see changes in outcomes?

Moving Forward • On the cusp of a new era of QRIS – More research on quality & child outcomes – More intentional focus on quality improvement – More measures

New Research Reports to Watch For • State evaluations in 2010 and 2011 – – – – – – – –

• • • •

Performance-based Accountability Systems (RAND) QRS Assessment (Mathematica and Child Trends) Quality Improvement in 17 States (NCCP) Coaching in QRIS (Child Trends)

OPRE Research Underway on QRIS • The Child Care Quality Rating System Assessment – Purpose: Provide information, analysis and resources to QRIS stakeholders

• Quality Dosage, Features and Thresholds and Child Outcomes – Purpose: Examine evidence and conduct analyses to understand how quality is linked to children’s outcomes

• Quality Initiatives Research and Evaluation Consortium – INQUIRE – Purpose: Translate policy-relevant research findings and create a learning community for researchers working on QRIS and quality initiatives

OPRE Research Underway on QRIS • State Data and Research Capacity Grants • Child Care Policy Research Grants Grants with QRIS focus: – Midwest Child Care Policy Research Consortium – Extension of Minnesota’s Pilot QRIS Evaluation

Reference â&#x20AC;˘ Tout, K., Starr, R., Soli, M., Moodie, S., Kirby, G. & Boller, K. (2010, April). Compendium of Quality Rating Systems and Evaluations. Prepared for the Office of Planning, Research and Evaluation, ACF, DHHS. http://www.acf.hhs.gov/programs/opre/cc/childcare_ quality/compendium_qrs/qrs_compendium_final.pdf