Peer Learning Community Series for SCA Statewide Recidivism Reduction Grantees
Data Collection & Evaluation: Building Capacity, Measuring Performance, and Developing Strategies to Monitor Public Safety and Program Outcomes Mike Eisenberg, Senior Research Manager Shenique S. Thomas, Policy Analyst
October 9, 2013
http://csgjusticecenter.org/nrrc/ The resource center is
continually updating its website with materials relevant to the reentry field.
Please register for the monthly
NRRC newsletter at: http://csgjusticecenter.org/ subscribe/, and please share this link with others in your networks that are interested in reentry.
Peer Learning Community
• Purpose: Cul5vate learning communi5es between Statewide Recidivism Reduc5on sites to facilitate discussions centered on implemen5ng realis5c research strategies, enhancing data collec5on and evalua5on efforts, and sharing successes, challenges, and lessons learned related to program implementa5on and evalua5on.
BJA’s Expecta5ons around Evalua5on • “In applying for these grants, lead grantees and their sub-‐grantees agree to cooperate in any and all related research efforts and program evalua5ons by collec5ng and providing enrollment and par5cipa5on data during all years of the project.” • “Applicants further agree to implement random or other modes of par5cipant assignment, required by the evalua5on design; cooperate with all aspects of the evalua5on project; and provide comparable individual-‐level data for comparison group members.” • “Applicants are encouraged to consider a partnership with a local research organiza5on that can assist with data collec5on, performance measurement, and local evalua5ons.”
Research Strategy • Realis5c research strategy should increase the organiza5on’s: – Understanding of the mechanism through which the program accomplishes change • What factors influence the change?
– Understanding of the necessary contextual condi5ons to accomplish change • What circumstances/condi5ons are needed to ensure the change takes place?
– Specificity of outcome paWern predic5ons according to context and mechanism triggered • What is the outcome and paWerns that result due to this change?
(Pawson & Tilley, 1997)
Presenters Presenter and Moderator: Mike Eisenberg Senior Research Manager Council of State Governments Justice Center Grantee Presenters: Kansas Department of Corrections Margie Phelps Jeremy Barclay Kristin Bechtel
Ohio Department of Rehabilitation and Correction Ed Rhine Sharon Schnelle
Presenta5on Overview Measuring Recidivism and Other Key Data Indicators
Evalua:ng the Impact of Specific Interven:ons
Case Example: Kansas Department of Correc:ons Case Example: Ohio Department of Rehabilita:on and Correc:ons
Measuring Recidivism and Other Key Performance Measures • Why it ma)ers: Knowing your statewide recidivism rate and having a rich set of data to break down and understand the drivers of the recidivism rate is key to developing effec5ve reentry policies. • What to think about: – Defining recidivism – Effec5vely tracking and measuring statewide recidivism rates on a regular basis – Collec5ng other performance indicators of break-‐outs of the recidivism rate to give more context
Defining and Tracking Recidivism Measures 1. No na:onal standard exists for defining recidivism 2. Agencies use a variety of defini:ons
Arrest Convic:on Return to Incarcera:on
3. Standard follow up periods are necessary to calculate recidivism rates
Follow up maPers – a one year rate will be lower than a three year Percent
One Year Tracking Period
Return to Prison for New Offense or Revoca:on of Supervision
Percent
Three Year Tracking Period
Return to Prison for New Offense or Revoca:on of Supervision
Examples of Key Recidivism Indicators to Inform Policy Recidivism rates by risk level for released popula;on (1 year reincarcera;on) 80%
65%
67%
63%
60% 40% 20%
38%
59% 42%
35%
33%
16%
14%
15%
15%
2009
2010
2011
2012
High Risk Mod Risk
0%
ü To determine if supervision strategies targe:ng risk are effec:ve
Low Risk
Examples of Key Recidivism Indicators to Inform Policy Dis;nguish supervision revoca;ons by technical viola;ons and new crimes 100% 80%
76%
75%
74%
75%
24%
25%
26%
25%
69%
60% 40%
31%
20% 0% FY11, Q4 FY12, Q1 FY12, Q2 FY12, Q3 FY12, Q4 Technical
New Crimes
ü To determine if supervision strategies are impac:ng “behavior” (reduc:on in new crimes) and effec:vely u:lizing progressive sanc:ons for viola:ons
Presenta5on Overview Measuring Recidivism and Other Key Data Indicators
Evalua:ng the Impact of Specific Interven:ons or Programs
Case Example: Kansas Department of Correc:ons Case Example: Ohio Department of Rehabilita:on and Correc:ons
Program Evalua5on Overview
Issue 1: Iden5fying the target popula5on and comparison group Issue 2: Defining and tracking recidivism outcomes Issue 3: Conduc5ng a process evalua5on
Issue 4: Defining successful comple5on
Issue 1: Iden5fying a Target Popula5on and Comparison Group
Target Population Discussion: • High/moderate-‐high risk • IdentiHied needs
Relevant information from the solicitation: “target population should be based on documented groups of offenders that signi;icantly contribute to increased recidivism rates”
• Sample size for evaluation – What is adequate sample size for evaluation – Time to develop sample size and conduct evaluation
Issue 1: Iden5fying a Target Popula5on and Comparison Group Comparison Group Discussion: Tiers of research design quality Random Assignment
Quasi-‐ Experimental
Qualita:ve Evidence
Individuals that qualify for program are randomly assigned to participate, or control group With adequate controls, may include: • Comparison to similar population in different jurisdiction • Comparison to overall population in jurisdiction not receiving services • Pre-‐program/post-‐program comparison of outcomes for the target population
Issue 1: Iden5fying a Target Popula5on and Comparison Group Which is be)er – the “real good reentry” program or the “so-‐so reentry” program? Recidivism Rate “Real Good Reentry” Program
20% ASK
“So-‐so Reentry” Program
40%
Are Popula+ons Served in Each Program Comparable?
Risk
Low Risk
Medium-‐High and High Risk
Age
45+
18 to 30
Female
Male and Female
First :me offenders
2 or more prior felonies
1 year Reincarcera:on
2 year Rearrest
Gender Criminal History Recidivism Measure
These are not comparable programs
Issue 1: Iden5fying a Target Popula5on and Comparison Group Relevant information from the solicitation: “Applicants agree to provide comparable individual-‐level data for comparison group members”
Discussion: In order to compare outcomes, you must control for some of the following factors:
Recidivism Rate Comparison Group
Did Not Par:cipate in “So-‐so Reentry” Program
65%
“So-‐so Reentry” Program
40%
Risk
Medium-‐High and High Risk
Age
18 to 30
Gender Follow-‐up Period Recidivism Measure Criminal History
Male and Female 2 Years Any rearrest 2 or more prior felonies
Issue 2: Defining and Tracking Recidivism Measures Recommended deIinition of recidivism for evaluation: • One year follow-‐up • Rearrest and Reincarceration
Relevant information from the solicitation: “For purposes of this solicita:on, “recidivism” is defined in accordance with the current defini:on u:lized by the applicant agency”
– Misdemeanor or Felony Arrest
– Reincarceration for new offense or revocation for supervision violations
Issue 2: Defining and Tracking Recidivism Measures Recidivism data and follow-‐up period: • What data sources are available for tracking recidivism? Rearrest
Reconvic;on
Reincarcera;on (jail Reincarcera;on or prison) (prison only)
SID #
SID #
SID #
DOC data
• Are you tracking recidivism rates of participants over a standard follow-‐up period?
Issue 3: Conduc5ng a Process Evalua5on
Goals of a process evalua5on: 1. Monitor program implementa5on and changes to the program or service system interac5ons over 5me 2. Determine whether a program is implemented in a way consistent with proven successful interven5ons 3. Fully describe the program and its components for purposes of replica5on
Issue 3: Conduc5ng a Process Evalua5on • A process evalua5on should examine: √ Is program u5lizing a design that has previously demonstrated an ability to reduce recidivism? √ Is the program being implemented as designed? √ Is staff training and experience adequate to deliver program as designed √ Are risk/needs assessed and services delivered consistent with risk and needs? √ Is the delivery of these services consistent over 5me?
Issue 4: Defining successful program comple5on
Successful Comple:on
• Program par+cipant successfully completed the previously defined necessary components of the program.
Drop-‐out/ Mortality
• Program par+cipant disposi+on is unknown and should not necessarily be counted as a successful completer or failure.
Program Failure
• Program par+cipant describes has not made advancements in the program, but instead has transgressed.
Issue 4: Defining successful program comple5on • Process-‐based defini5ons of successful comple5on:
– Program par5cipant has completed 70% of program requirements or case plan within one year. – Program par5cipant at a moderate risk level has completed approximately 100 hours of services, programming, or treatment within one year; program par5cipant at a high risk of reoffending has completed approximately 200 hours of services, programming, or treatment within one year.
Issue 4: Defining successful program comple5on • Outcome-‐based defini5ons of successful comple5on:
– Program par5cipant has shown an improvement on a post-‐test assessment of a behavior related to their risk of recidivism or goals of the program (such as, but not limited to, substance abuse, risk and need level, behaviors and ajtudes, mental health, etc) within one year. – Program par5cipant has achieved core benchmark goals of the program which are not necessarily related to behaviors (such as, but not limited to, aWaining stable housing, aWaining employment, achieving a GED, etc) within one year.
Presenta5on Overview Measuring Recidivism and Other Key Data Indicators
Evalua:ng the Impact of Specific Interven:ons or Programs
Case Example: Kansas Department of Correc:ons Case Example: Ohio Department of Rehabilita:on and Correc:ons
Case Example: Kansas Department of Correc5ons 1. Building the DOC’s capacity and exper5se around data collec5on and evalua5on 2. Evalua5on of Thinking for a Change (T4C) 3. Tracking recidivism and other key performance measures to monitor progress and inform policy
Evalua5on of Thinking for a Change • Project purpose: evaluate effec5veness of T4C, as measured by re-‐incarcera5on, on T4C par5cipants when compared with similarly situated inmates • Ini5al stage of project development and research ques5ons • Methodology – Sample and matching criteria – Implementa5on and program fidelity
• Preliminary findings and next steps
Case Example: Kansas Department of Correc5ons Based on offenders released from KDOC facili;es who return within 3 years. CY 2005 CY 2006 CY 2007 CY 2008 CY 2009 RECIDIVISM 38.62% 34.18% 32.90% 33.64% 33.13% NO RETURN 61.38% 65.82% 67.10% 66.36% 66.87% CY 2005 CY 2006 CY 2007 CY 2008 CY 2009 NEW CONVICTIONS 12.23% 13.21% 12.62% 13.99% 15.14% CONDITIONAL VIOLATION 26.38% 20.97% 20.28% 19.65% 17.99% OFFENDERS RELEASED By type of return and length of follow-‐up period* - The KDOC’s recidivism rate has declined 23.57% since FY 2000 when the recidivism rate was 56.7%. - For every 1% reduc:on in recidivism, the number of crime vic:ms inherently drops and the need for prison beds drops by 34.
Case Example: Kansas Department of Correc5ons SEX OFFENDERS Overall Condi:onal Violators Convicted (New Offenses) GENDER** Male Female RISK LEVELS High Risk Moderate Risk Low Risk
CY 2006 CY 2007 CY 2008 CY 2009 40.47% 42.66% 38.57% 33.33% 36.52% 37.86% 33.71% 27.08% 3.95% 4.80% 4.86% 6.25% 34.17% 33.12% 33.81% 32.46% 18.55% 17.27% 17.88% 15.79% 38.85% 42.30% 41.09% 42.86% 32.97% 31.42% 31.95% 28.77% 26.08% 19.69% 18.04% 15.96%
Presenta5on Overview Measuring Recidivism and Other Key Data Indicators
Evalua:ng the Impact of Specific Interven:ons or Programs
Case Example: Kansas Department of Correc:ons Case Example: Ohio Department of Rehabilita:on and Correc:ons
Case Example: Ohio’s Reentry Efforts • Passage of HB 130 had direct implica5ons for reentry and the Second Chance Act
• Ohio’s HB130 became effec5ve April 1, 2009 • 2 cri5cal components of HB130 that support sustainability efforts for the state -
Established the statewide Ohio Ex-‐Offender Reentry Coali5on to facilitate collabora5on with state and local stakeholders and aid in reentry improvement efforts
-
Catalyst to the development of a 5 year comprehensive strategic plan that contains performance goals and outcomes, a commitment to reduce recidivism by 50%, and updates on progress in Annual Report issued by OERC
Ohio’s scope of Data Collec5on
Dept of Aging
-
# trainings of DRC staff # of trainings Reentry professionals # of inmate trainings(chronic disease) # of inmate trainings (diabetes)
REENTRY EDUCATION EMPLOYMENT VETERANS VICTIMS RECOVERY SERVICES MENTAL HEALTH - HOUSING
- # Courts apply for Certification - # Courts granted Certification - # of SD courts starting up - # of SD courts receiving technical assistance
Dept of Rehabilitation & Correction -
Supreme Court of Ohio
-
Board of Regents
# of One Stops # of Trainings # of Job Fairs # of prison mailings
Veteran’s Services
# of Drug Courts funded # juvenile TASC # adult TASC # clients served # clients successfully complete # clients readmitted DRC
Dept of Education
-
# inmates GED # inmates ABLE # inmates career certificates # inmates apprenticeship program
OHIO Ex-‐Offender Reentry Coalition
Dept of Mental Health*
Governor’s Office of Faith-‐based & Community Initiatives
- # veterans referred - # veterans connected to services - # of veterans released from DRC that connected to services
Dept of Alcohol & Drug Addiction Services*
-
Development Services Agency
Dept of Job and Family Services
Dept of Developmental Disabilities
Dept Of Youth Services -
EDUCATION EMPLOYMENT RECOVERY SERVICES MENTAL HEALTH HOUSING /PLACEMENT MENTORING FAMILY ENGAGEMENT
- # DD determinations - # DD ineligible redeterminations - # MOUs
Dept of Commerce
Rehabilitative Services Commission
Dept of Public Safety
OERC Annual Data Collec5on Tool • Ohio Office of Criminal Jus5ce Services maintains the web-‐ based data collec5on tool • Collabora5ve effort with stakeholders (iden&fy data instruments currently used, and then developed data collec&on tools from there to ensure minimizing double data entry or burdening of the agencies by further taxing limited agency resources. For missing essen&al informa&on needed to demonstrate movement on outcomes included in the strategic plan necessary addi&onal measures were developed and included)
• Effec5ve communica5on of the purpose – mee&ng with stakeholders to acknowledge goal of gathering informa&on that can be used to further leverage scarce state resources
• DATA COLLECTION TOOL located at hJps://www.surveymonkey.com/s/OERCdatacollec&ontool
Barriers / Lessons Learned • • • • • •
Logis5cal -‐ not enough staff resources Fiscal and budgetary crisis at both state/local Data sharing limita5ons and issues Inefficient system of collec5ng informa5on Diversity of data (i.e. collec&on period, methods, measures, etc) Balancing need to meet statutory obliga5ons and u5lity of the informa5on to inform the strategic planning process
Prac5cal Applica5on • Preliminary infrastructure resulted in opportunity to expand reentry efforts to rural communi5es.
Ques5ons?
Thank You! Questions? Mike Eisenberg Senior Research Manager meisenberg@csg.org
Shenique S. Thomas Policy Analyst sthomas@csg.org
http://csgjusticecenter.org/nrrc/
The presentation was developed by members of the Council of State Governments Justice Center staff. Because presentations are not subject to the same rigorous review process as other printed materials, the statements made reHlect the views of the authors, and should not be considered the ofHicial position of the Justice Center, the members of the Council of State Governments, or the funding agency supporting the work.