Data Collection 101 Making the Case for Your Program
JMHCP National Training and TA Event: Collaborating to Achieve and Communicate Positive Public Health and Public Safety Outcomes
Achieving and Communicating Outcomes
Types of Program Evaluation • Process Evaluation – Program Implementation and Operation • How many did you serve? • What services did they receive? • Did the program operate as designed?
• Outcome Evaluation – Program Effectiveness • Did you reduce recidivism?
• Cost-effectiveness – Program costs in relation to benefits • Did benefits exceed program costs?
Achieving and Communicating Outcomes
Program Goals and Objectives Drive Data Collection and Evaluation
• Goal: Broad statement of what the program hopes to accomplish: – Example: “reduce re-offending of mentally ill offenders”
• Objective: Specific and measurable condition that must be attained to achieve program goal – Example: “reduce the percent of mentally ill offenders incarcerated in prison two years after placement in program” Achieving and Communicating Outcomes
Data Elements for Outcome Evaluation • Criminal Justice Measures – Arrests – Convictions – Incarceration
• Mental Health Measures – Hospitalizations – Medication monitoring – Emergency room visits Achieving and Communicating Outcomes
Where Do I Put My Data and How Do I Get It Out?
Paper
Excel
Digital
Access
Criminal Justice/Mental Health Consensus Project: www.consensusproject.org Working with Data for Mental Health Court Practitioners- Parts One and Two (webinars) available online at: http://consensusproject.org/features/webinar-archive-working-with-data-formental-health-court-practitioners-part-one-data-collection-and-manipulation
Achieving and Communicating Outcomes
Data Collection Challenges • Identify sources of data – Law enforcement for arrest data – Case record data from referral source
• Develop data collection protocols – Who gets what from where – Address privacy issues
• Acquire/format data for evaluation – Need research design – Need conceptualization of how you are going to measure outcomes Achieving and Communicating Outcomes
Recidivism as an Outcome Measure • Most common outcome measure in criminal justice • Arrest / Convictions / Incarceration most common measures • “What is the recidivism rate?” is not a complete question – What is the recidivism measure – What is the period of time the measure is tracked
• Sources of recidivism data / capturing state identifier number – Local jail – Department of Corrections – Department of Public Safety
Achieving and Communicating Outcomes
Keys to Outcome Evaluation • Comparison group: Apples to Apples – A group of comparable offenders, not participating in the program, must be identified and tracked in the same way as the program group
• Uniform Follow-up Period – Each offender in the program group and the comparison group must be tracked for the same time period • Example: All offenders in the program and comparison group are tracked for two years
• Program Outcomes – Measure program outcomes by tracking all offenders, program costs, and the recidivism of program completers, Achieving and Communicating Outcomes drop-outs, and comparison groups
Alternatives to Comparison Group for Outcome Evaluation • Developing a Comparison Group can be difficult. Some alternatives to consider for outcome evaluation can include: – – – –
Pre- and post-measures of treatment population Looking at outcomes of waiting list offenders Looking at outcomes of program drop-outs Looking at outcomes of ineligible offenders • Out of service area Achieving and Communicating Outcomes
Outcome Evaluation Example: Mental Health Initiative Outcome Evaluation • Mental Health Initiative placed mentally ill offenders on specialized caseloads and provided case management and medication resources • Goal: Reduce re-offending / Objective: Reduce reincarceration of offenders during 2 year follow-up period • Outcome Measure: Percent Incarcerated 2 years after placement on specialized caseload • Comparison group: – Mentally ill offenders on specialized caseload but no case management or medication services Achieving and Communicating Outcomes
Mental Health Initiative Outcome Results Percent Incarcerated in Two Year Follow-up
MH Initiative
Comparison Group
Overall
13%
20%
17%
High
16%
30%
22%
Medium
12%
18%
14%
Low
9%
10%
9%
Overall Risk Level
Achieving and Communicating Outcomes
Cost-Effectiveness Analysis • •
• •
Do Benefits of Program Outweigh Cost of Treatment?
Cost-effectiveness = Incarceration Costs Avoided – Treatment Cost Incarceration Costs Avoided = Number of recidivists avoided x Incarceration costs o Number of Recidivists avoided = Recidivism Change (Comparison Group – Treatment Group) o = Comparison Group 20% - Treatment Group 13% x Number in Treatment Group (100) o = 7 recidivists avoided o Incarceration Costs Avoided = 7 recidivists avoided x $25,000 (cost of incarceration) = $175,000 Treatment Cost = Average cost of treatment services x Number of offenders receiving treatment= $1,000 per offender x 100 = $100,000 Cost-effectiveness = $175,000 incarceration costs avoided –$100,000 treatment costs +$75,000 net incarceration costs avoided Achieving and Communicating Outcomes
CHATHAM-SAVANNAH MENTAL HEALTH COURT Judge Penny Haas Freesemann, Presiding Superior Court, E.J.C. of Georgia Chief Judge H. Gregory Fowler, Presiding State Court of Chatham County Erica Usher, Director 912-652-7167 Achieving and Communicating Outcomes
In the Beginning… • It began with DATA • Determined if there was a Need • Focused on gathering Criminal Justice data – Research conducted in Chatham County Detention Center • The Chatham County Detention Center houses over 300 inmates with a mental illness on any given day • Cost of $6 million dollars a year to incarcerate these consumers • Mentally ill offenders were spending more time in jail than offenders without a mental illness
– What we found: • Medication budget, detain less people, increased services
Achieving and Communicating Outcomes
Planning Process • Planning Committee – Partnership created between community stakeholders and local treatment providers – Embarked on a 18 month planning process
• Community Mapping – Resources were broken, fragmented, unstable, constant threat of closure or significant reduction in funding for services, etc…
• Development of the Plan – To increase public safety by addressing individuals with mental illness with or without co-occurring disorders who repeatedly come into contact with the criminal justice system through an innovative cross-system collaboration under the direction of a mental health court, which is targeted and responsive to the needs of eligible nonviolent offenders, while preserving the integrity of the court.
• Application for Bureau of Justice Assistance Grant (JMCHP) – Awarded Grant
Achieving and Communicating Outcomes
Collect Data – How Do You‌
make the case for your program?
Collecting Data one piece at a time to prove program impacts
Achieving and Communicating Outcomes
First Year of Implementation September 2007-September 2008 General Data Participants
Diagnosis
Gender
60% Male; 40% Female
Schizophrenia
50%
Race
62% AA; 38% Caucasian
Schizoaffective
25%
Average Age
39
Bi-Polar
20%
Major Depressive
5%
Co-occurring Disorder
81%
Referrals and Participants Reviewed Cases
135
Participants
27
Accepted
27
Terminated
1
Graduated/Completed
0
Achieving and Communicating Outcomes
First Year of Implementation: September 2007-September 2008 Criminal Justice Measures Mental Health Measures – Arrest – Charges – Jail Days • Total Number • 12 months prior • During participation
– Georgia Regional – Memorial Hospital • Admissions • Days – 24 months prior – 24 months following
– 24 months prior – 24 months following
Achieving and Communicating Outcomes
Criminal Justice Measures Criminal Justice Involvement Total # of Arrest in Chatham County
Total # of Criminal Charges in Chatham County
Total:
296
601
12 Months Prior to Program Entrance
44
72
1 (98%)
1 (99%)
Since Admission to the Program
Days Spent in the Chatham County Detention Center Total # of Days
Total Costs at $52.00 per day
Total:
11,644
$607,620.00
12 Months Prior to Program Entrance
2,917
$151,684.00
570 (80%)
$ 29,640.00
Since Admission to the Program
=$122,044.00
Achieving and Communicating Outcomes
Mental Health Measures Georgia Regional Hospital Total # of Admissions
Total # of Days
Total Costs ($498.00 per day)
Total
262
4846
$2,413,308.00
12 Months Prior
28
423
$ 210,654.00
6 (79%)
39 (91%)
$ 19,422.00
During
Georgia Regional Hospital Total # of Admissions
Total # of Days
Total
232
843
12 Months Prior
24
25
10 (58%)
10 (60%)
During
=$144,000.00
Achieving and Communicating Outcomes
First Year of Implementation September 2007-September 2008 General Data Participants
Diagnosis
Gender
63% Male; 37% Female
Schizophrenia
45%
Race
60% AA; 40% Caucasian
Schizoaffective
15%
Charges
76% Felony; 24% Mis.
Bi-Polar
24%
Major Depressive
16%
Co-occurring Disorder
83%
Disabled
61%
Referrals/Participants Reviewed Cases
210
Participants
27
Accepted
38
Terminated
3
Graduated/Completed
5/2
Participants 89% incarcerated at time of acceptance 92% previously used; average age 15 Achieving and Communicating Outcomes
First Year of Implementation: September 2007-September 2008 Criminal Justice Measures Mental Health Measures – Arrest – Charges – Jail Days • • • •
Total Number 24 months prior During participation 24 months following
– Georgia Regional – Memorial Hospital • Admissions • Days – 24 months prior – During participation – 24 months following
– Institutionalized Time vs Free Time Achieving and Communicating Outcomes
Criminal Justice Measures Criminal Justice Involvement
80% Reduction in Jail Days
Arrest and Charges During Treatment
96% reduction in charges 90% reduction in arrests Achieving and Communicating Outcomes
Institutionalized Time vs. Free Time
The graph above represents the participants free/available time verses institutionalized time (incarcerated and hospital days) during the time for data collection.
Achieving and Communicating Outcomes
Outcome Evaluations: Mental Health
Achieving and Communicating Outcomes
Outcome Evaluation: Mental Health Georgia Regional Hospital of Savannah Total # of Admissions
Total # of Days
Total costs at $498.00 per day
Lifetime Total:
267
4999
$2,489,502.00
24 Months Prior to Program Entrance
61
837
$ 416,826.00
Since Admission to the Program
16
130
$
64,740.00
24 Months Following Program
3
24
$
11,952.00
=$352,086.00 Achieving and Communicating Outcomes
The Future • Holds even more DATA… • Implement Database • Identify and Develop Comparison Group • Complete Cost-Benefit Analysis Achieving and Communicating Outcomes
Thank you
For further information & copies of conference presentations please visit www.consensusproject.org This material was developed by presenters for the February 2011 event: “Collaborating to Achieve and Communicate Positive Public Health and Public Safety Outcomes.� Presentations are not externally reviewed for form or content and as such, the statements within reflect the views of the authors and should not be considered the official position of the Bureau of Justice Assistance, Justice Center, the members of the Council of State Governments, or funding agencies supporting the work.
Achieving and Communicating Outcomes