[Funded by the Corston Coalition]
Social Return on Investment Report for SWAN
Harriette Boyden, CEO Relate Northumberland and Tyneside Katie Wheatley SWAN Coordinator, Northumberland
Contents 1.
Executive Summary ………………………………………………………………….. 1
2.
Introduction and Background ……………………………………………………. 3
3.
SROI Methodology …………………………………………………..………………… 4
4.
SWAN ………………………………………………………………………………………… 6
5.
Theory of Change ……………………………………………………..………………. 8
6.
The Impact Map ……………………………………………………………………… … 15
7.
Evidencing Outcomes and Giving Them a Value ………………………… 21
8.
Understanding Impact ………………………………………………………………. 27
9.
Findings ……………………………………………………………………………………… 28
10.
Recommendations and Learning …………………………………………….. … 31
11.
Appendix 1 The Questionnaire …………………………………………………. 32
12.
Appendix 2 Summary of Assumptions (Proxies)…………………………. 35
1. Executive Summary The Corston Coalition has commissioned nef consulting to help the Ministry of Justice Women’s Centre Projects (WCS) develop an outcomes framework to understand the impact of the WCS on individual women and the wider community. Using the principles of measuring what matters1 and drawing on the Social Return on Investment (SROI) methodology, this report presents the theories of change for Northumberland WCS, Support for Women Around Northumberland (SWAN) . SWAN is a Women’s Community Service which received funding from the Ministry of Justice under the Diverting Women from Custody Programme. The objectives of SWAN are to reduce the reoffending of female offenders or those at risk of offending through providing a range of diversionary and preventative interventions. Between February and November 2010 a total of 120 women had been referred to SWAN, with 80 women being assessed and 50 women engaged. The project has resulted in a 70% reduction in arrests, a 73 % reduction in the number of charges made by the police and an 81% reduction the number of convictions amongst engaged female offenders. There had also been a 72% reduction in appearance before magistrates. However, this is not the whole picture. It does not describe all of the value that SWAN has delivered. This research into the Social Return on Investment has allowed us to better understand and represent the changes that occurred as a result of SWAN’s activities. The Theory of Change is fundamental to the story explaining the impact of our service. We have taken a multi-stakeholder approach and measured and valued both outcomes for women and those for the state. In the past, reporting the later has often been our focus, partly because of funding drivers. We believe that delivering the outcomes that women most value ensures that further change can occur and is therefore a fundamental part of the change journey. Thus, being able to communicate and, more-over, report that value to commissioners and funders allows them to make better decisions about the kind of services that will be effective. As practitioners we understand that changes in an individual woman are the initial result of our centres, programmes and initiatives. As individuals change and that change becomes more perceptible, this impacts socially and economically. For example if an individual increases their skills for employability, their income will be greater and poverty rates may decrease. Individual impacts are often not seen by outside agencies as the building blocks of our community change. If individuals change it can result in community change. When the state e.g. local government and institutions like the police and National Health Service begin to notice visible, positive changes such as reduced crime or lower admittance to A&E for substance use, then sectors like the women’s sector can be firm in the conviction that they have influenced and levered change.
1
Lawlor, E et al (2008) Measuring Value: a Guide to Social Return on Investment (SROI) (London:nef)
1
The SROI method – we asked:
What is the change that occurred? And … How much change occurred? And for how long? i. We identified outcomes and outcome indicators for the women and the state and measured them over time (using questionnaires for women and data from the PNC for the state). Our Theory of Change and Impact Map illustrate the impact of our work more clearly than we have ever done before. What would have happened anyway? What change must be attributed to others? What problems have been displaced by our work? i. These three calculations make our findings extremely robust and realistic What is the value of that change? i. This is where we have applied proxies to calculate the value of our outcomes.
Proxies allow us to value the outcomes that we have always known in the sector are important but are those which have no traditional market value for eg well-being. The SROI method looks for proxies for these outcomes. There are things which have well established market prices, such as private healthcare and gym membership. SROI uses these market values to approximate the value of the outcomes most traditionally undervalued, currency of the third sector. Findings The SROI produces a ratio of investment in and value out. Whilst the ratio is indicative of the cost effectiveness of SWAN’s work and incites discussion, it is only one measurement against which decisions should be made. Undertaking a SROI and therefore challenging assumptions and making an organisation open to scrutiny is a valuable process in and of itself. Bringing the total benefits extrapolated to all stakeholders and the total investment over 12 months together and placing them in net present value terms, the Social Return on Investment ratio for SWAN is 1:6.65. This means that for every £1 invested in SWAN’s activities, £6.65 of social and economic value is created for beneficiaries. Therefore the £160,000 invested created £1,064,000 SROI. The value for women was greater than that produced for the state. For women, £748,699 and the state £314,662. Next steps We have learned a lot about how to calculate the Social Return on Investment. It has given us a method and language to communicate with commissioners and funders about what works and why. Our challenge is to increase the opportunities to talk to others about this work and to embed it in our organisations. Our findings highlight the importance of what women value and communicate it with equal status with outcomes for the state, that which we are so used to valuing in £s. The risk of not doing so is that, as commissioning increases and the relative freedom of grant funding becomes extinct, the knowledge and experience of the Women’s Community Services will not be included in commissioning decisions and long-term poor outcomes for women will result. 2
2. Introduction and Background The Corston Coalition has commissioned nef consulting to help the Ministry of Justice Women’s Community Services (WCS) to develop an outcomes framework to understand the impact of the WCS on individual women and the wider community. Using the principles of measuring what matters2 and drawing on the Social Return on Investment (SROI) methodology, this report presents the theories of change for Northumberland WCS, Support for Women Around Northumberland (SWAN) . This work has been conducted as part of a process to assess the practicability of implementing a full outcomes-based measurement and valuation system for all the WCSs across England and Wales. Such an undertaking yields two key benefits: it would help WCS to measure and demonstrate its impact, and it would help it to improve decision-making, ensuring its activites create the desired outcomes. Ongoing evidence of value for money and wider societal impact for material stakeholders is critical to the individual WCS’s organisational sustainability. It is worth noting that this project has the additional objective of capacity-building within services, in order to build on this work in the future and further contribute to its sustainability. This report presents the first step of the SROI methodology: understanding the theory of change. It then presents recommendations for next steps.
2
Lawlor, E et al (2008) Measuring Value: a Guide to Social Return on Investment (SROI) (London:nef)
3
3. SROI Methodology This report on WCS bases its approach to measurement on a methodology known as Social Return on Investment (SROI), which measures all material outcomes that are identified as resulting from an organisation’s or programme’s activities and attaches an approximate financial value. An adjusted form of cost-benefit analysis (CBA), SROI considers costs and benefits across the triple bottom line: social, economic and environmental. This adjustment of traditional CBA gives a fuller picture of the benefits that flow from the investment of time, money and other resources.
Phase 1: Setting parameters and impact map Boundaries
Create the framework for the analysis – what part of the organisation or individual project is to be measured – and prepare background information. In this case, the scope of the analysis is the helpline and advice line that SWAN delivers.
Describe how the project or organisation works and decide the time period for measurement.
Stakeholders
Identify the stakeholders whose costs and benefits – associated with the investment or organisation – are to be measured.
Stakeholders Those people or groups who are either affected by or who can affect the activities.
Prioritise key stakeholders and objectives. Materiality (the accountancy term for ensuring all the areas of performance needed to judge an organisation's performance are captured) is used in the selection of stakeholders and objectives.
Identify common or overriding objectives.
Impact map
Conduct stakeholder engagement to assist in the creation of an impact map that describes how the organisation/investment affects key stakeholders.
Phase 2: Data collection Indicators
Identify appropriate indicators to capture outcomes.
Valuation
Impact map Demonstrates how an organisation’s inputs and activities are connected to its outputs and how these, in turn, may affect stakeholders’ outcomes. Impacts can then be derived from the identified outcomes.
Use findings from stakeholder engagement and existing primary and secondary data (SWAN evaluation completed by Barefoot Research and Evaluation Ltd).
Research to generate proxies to put financial values on outcomes. 4
Data collection
Use tried and tested sources to gather the data – required by the impacts laid out in the impact map – for accurate measurement of identified costs and benefits.
Decide on an appropriate benefit period for each outcome and the extent to which those outcomes drop off over time.
Phase 3: Model and calculate Model and calculate
Create a cost-benefit model using gathered data and projections:
o
Calculate the present value of benefits and investment and the SROI ratio.
o
Account for the displacement, attribution and deadweight of the organisation/investment under review.
Phase 4: Report Report
Consider and present the SROI produced by the organisation/investment.
Identify how the benefits are divided between stakeholders.
Identify the key factors that affect the SROI ratio.
The following sections outline how the first part of this methodology was applied in the context of SWAN’s activities to develop the theory of change.
5
4. SWAN Support for Women Around Northumberland (SWAN) is a Women’s Community Service which received funding from the Ministry of Justice under the Diverting Women from Custody Programme. The objectives of SWAN are to reduce the re-offending of female offenders or those at risk of offending through providing a range of diversionary and preventative interventions. The project became operational in February 2010 and has continued funding for 2011-2012. SWAN is delivered by a partnership of voluntary sector organisations in Northumberland – Escape Family Support (lead agency), the Women’s Health Advice Centre (WHAC), Fourth Action and Relate Northumberland and Tyneside. SWAN Targets The original set targets stated that between February 2010 and March 2011 the project would undertake needs assessments and create individual support plans and risk assessments for 80 women. It also stated that over the course of the project, a total of 50 women would consistently engage with the project. Between February and November 2010 a total of 120 women had been referred to SWAN. With 80 women being assessed and 50 women engaged. The project has resulted in a 70% reduction in arrests, a 73% reduction in the number of charges made by the police and an 81% reduction the number of convictions amongst engaged female offenders. There had also been a 72% reduction in appearance before magistrates. SWAN’s Partner Agencies ESCAPE Family Support has worked in Northumberland for 15 years. ESCAPE provides holistic support to substance users and their families and carers across Northumberland. ESCAPE delivers stand-alone services and interventions in partnership with statutory and non-statutory agencies. It provides Tier 2 (advice, advocacy, information) and Tier 3 services (including person centred counselling and CBT, structured care programmes, therapies, support to employability, brief intervention and 24 hour help line support). ESCAPE’s ethos is to work with individuals and their families at all stages of their treatment journey or their involvement with the criminal justice system: pre arrest, at arrest, pre-sentencing, imprisonment and on release. Recognising and respecting diversity is embedded in all its policies, practice guidelines and procedures. FOURTH ACTION promotes equality with communities. Its main client groups are those facing exclusion and discrimination. Fourth Action is based in rural Northumberland and specialises in rural exclusion issues, carrying out project work with young people who are out of work or learning, young parents and carers, people with learning disabilities, people affected by mental health issues and the migrant community. Fourth Action also offers accredited mentoring skills training to help improve the quality of tailored grassroots support.
6
RELATE (Northumberland & Tyneside) is a member of the Relate Federation. Relate is the UK’s largest provider of relationship counselling and psychosexual therapy with over 70 years’ experience working with individuals, couples, families and children. Relate adheres to the ethical guidelines laid down by the British Association of Counselling and Psychotherapy (BACP). The Relate counsellors are trained in consultation and assessment, adult mental health, adult survivors of sexual abuse and domestic violence/abuse. Relate contributes to the Children, Families and Relationships pathway of SWAN by exploring women’s relationships and the implications for children and the woman’s parenting capacity. WHAC (Women’s Health Advice Centre) has been providing a non-medical service to women for 25 years aiming to help women improve their physical, mental, social health and well-being. This is achieved by offering 1:1 counselling, personal development courses, support and advice services (family law, debt, housing welfare rights), support to women experiencing domestic abuse and sexual assault, 1:1 lifestyle sessions, signposting to other services and volunteering opportunities.
SROI Scope As is evident from the SWAN partners’ activities, the services that SWAN offers are extensive. This report focuses on the holistic support provided by SWAN to its individual clients. The following points set out the scope of the evaluation:
This is an evaluative SROI analysis. This report will look at activities and changes that have already occurred to understand value for money.
The evaluation looks at the benefits generated by SWAN investments. The investment period and measurement period for this analysis is three months.
Data collection will take place from July to September 2011 and will be aggregate to the total number of material stakeholders affected by SWAN’s activities.
7
5. Theory of Change Introduction and Background
Theory of Change offers a pathway of recognition, change and action. It is a theory which results in short and long term outcomes and reflects changes on different levels e.g. individual women, organisations, systems and communities. Within the Women’s Sector it has been identified that individual-centred, empathetic work is a matter of everyday practice however it is often not counted or recognised as a value. In the current economic climate, each sector has had to consider an approach of economic frugality. Often the third sector is hardest hit because it cannot demonstrate the positive economic impact that it can have as the results of the sector’s work are often a hidden economic value. For example an increased sense of self-worth due to a woman attending peer group work could result in decreased GP time for mood stabilizing drugs, but more pertinently create a non-financial contribution to society, that is, the woman now has a feeling of identity, that she is no longer alone and she is valued as an individual. As part of the third sector, it is important that the women’s sector acknowledges and values the impact of the compassionate work that results in “soft outcomes” that are achieved daily and records the far reaching impact that this has on society. The Theory of Change is a turning point for the third sector, as a practical and crucial part of a successful transformation in our vision of our everyday work. A common thread within the women’s sector is a shared value of equality for individual women; to make lasting and positive change to vulnerable women and the relationships that they have. As practitioners, we acknowledge that the positivity of an individual permeates further to influence families, relationships and communities. However, how is this outcome monitored in the third sector? How often do we recognise but do not acknowledge the greater impact we are having within the networks of personal families and communities? Workers understand small changes must occur within an individual’s journey, such as change in knowledge, skills, behaviour, health, family lives for children, adults, families and communities. These changes are often unrecognised but essential, small, “soft” outcomes that we know are vitally important in the journey but do not record. It is appreciated that we have long-term and short-term objectives for the individual and women as a collective. Front line staff will appreciate that essential fire fighting for the individual often diminishes our end objective of a fairer and just society for all. A sense of understanding, justice and empowerment runs through each women’s sector project, it is a part of the running of our everyday services. Unfortunately this sense of social justice does not pay the bills, however the impact of the women’s sector’s social philanthropy can be a means of identifying the impact that we have on society. As practitioners we understand that changes in an individual woman are the initial result of our centres, programmes and initiatives. As individuals change and that change becomes more perceptible, this impacts socially and economically. For example if an individual increases their skills for employability, their income will be greater and poverty rates may decrease. Individual impacts 8
are often not seen by outside agencies as the building blocks of our community change. If individuals change it can result in community change. When the state e.g. local government and institutions like the police and National Health Service begin to notice visible, positive changes such as reduced crime or lower admittance to A&E for substance use, then sectors like the women’s sector can be firm in the conviction that they have influenced and leveraged change.
Stakeholders Engagement Method and Impact Map Stakeholders are defined as those people or organisations that experience change as a result of the activity. The SROI approach does not assume to know what changes (positive or negative) may have occurred for the stakeholders. Therefore, engaging those affected by SWAN’s work is key to understanding true impact. Statutory agencies: In December 2010 a report was produced by Barefoot Research Evaluation (www.barefootresearch .org.uk3) that identified and explored major stakeholders’ perceptions of SWAN. For the purpose of this project, and as the research was completed relatively recently, the engagement of statutory agencies is recorded from this data source. Individual clients In order to gather information from clients, a non-probability sampling approach was adopted using the convenience method and a two-stage approach was employed: 1. Clients were asked if they were prepared to provide feedback on the support that they were receiving from SWAN. If they agreed, they were transferred to another member of SWAN staff, to ensure that they felt comfortable in providing feedback. 2. Once the initial feedback was complete, clients were asked if they were prepared to be contacted three months later to find out about their continued support. Clients were contacted approximately three months after the initial interview. SWAN staff were encouraged to record the feedback in the stakeholder’s own words, or by recorded transcript, where possible. The type of approach to selecting clients was one of convenience because staff used their judgment when asking whether clients would be willing to give feedback i.e. particularly distressed clients or those in crisis were not asked. The interviews allowed for an in-depth exploration of advice given by SWAN partners and which has resulted in or contributed to outcomes for the clients. The material gathered from the interviews forms the basis of the impact map which represents a detailed description of SWAN’s underlying Theory of Change. The impact map formally describes this Theory of Change in terms of how inputs and activities lead to specific outputs, and then in turn how these lead to initial and eventually long-term outcomes. In addition to the impact map, this report examines how SWAN makes a difference, as an attempt to understand how it delivers change to stakeholders.
3
http://www.barefootresearch.org.uk/wp-content/uploads/SWAN-Report-Single-Pages.pdf
9
A focus on outcomes Outcomes are complex and often difficult to measure. They can be negative as well as positive. For example, it is often presumed that gaining an accredited qualification will lead to positive change in terms of the confidence and employability of participants. This, however, is not necessarily the case. If a qualification is at an inappropriate level, it may actually erode a participant’s confidence. By engaging stakeholders in understanding what changes for them, it is possible to measure and value what matters to them – thus understanding the true impact of SWAN’s work. Multi-stakeholder analysis In performing this analysis, it is important to include the material4 outcomes that accrue to all stakeholder groups that are affected by SWAN’s work, above and beyond individual women themselves. These groups include children, families the wider community and the State. Due to the nature of SWAN, this report focuses on individual women as the main beneficiary of the services and the State as the benefits there are significant. Our rationale for materiality is explained fully later in the report. Stakeholders The multi-stakeholder approach in SROI allows the benefits to all stakeholders to be analysed. Stakeholders are those people or organisations that experience change as a result of the activity. SROI does not assume to know what changes (positive or negative) may have occurred for the stakeholders. Therefore, engaging those affected by SWAN’s work is key to understanding true impact. A social sciences approach to saturation sampling5 was adopted to build the Theory of Change. The stakeholder audit trail in the table below presents SWAN’s stakeholders and how they were engaged for this evaluation. Stakeholder audit trail Stakeholder
Number engaged and method of engagement
Women
17 women (aggregated over 12 months – 69) Women were engaged through a questionnaire
State: Police and CJS
Police National Computer data. We cross referenced our stakeholder women with the PNC
4
Materiality is the accountancy term for ensuring that all the areas of performance needed to judge an organisation's performance are captured. 5
Saturation sampling is where the researcher conducts stakeholder engagement until all the outcomes have been identified, i.e. extra engagement produces no new information.
10
It is acknowledged that whilst the numbers of stakeholders engaged may appear low for some stakeholder groups, the evaluators are confident that the sample is representative of wider stakeholders and therefore sufficiently robust to identify outcomes.
11
Increased individual and community safety
SWAN’s Theory of Change
Fewer court appearances
OUTCOMES INPUTS MoJ funding Partner agency contributions Contributions from SWAN partnership
ACTIVITIES Tailored one-to-one and group support both outreach and building based
OUTPUTS 80 women supported in 12 months
• Reduced re-offending • Improved mental health (improved self confidence and self esteem) • Reduced domestic abuse and increased safety • Reduced substance misuse • Increased entry into education and employment
Increased personal and social wellbeing Beneficiary time and commitment Increased effectiveness and efficiency of partner agencies
Volunteering, education and employment 12
Fewer offending events
SWAN’s Theory of Change
Narrative Stakeholders/ beneficiaries Women
Inputs
Activities/Actions
• Funding from Corston Coalition • Contributions from partnership organisations • Contributions from other agencies such as the Police and Probation
Individual tailored support to women using an outreach and building based service model, including: • Accompanying to activities, such as the gym • Advocacy such as to benefit agencies, Core Group or Child Protection meetings, housing, GPs and MARACs • Booking and accompanying onto courses run through ESCAPE such as confidence building • Booking and accompanying onto therapies such as head massage and acupuncture • Cognitive Behavioural Therapy and Motivational Interviewing • Counselling, either provided by Women’s Intervention Workers themselves or referring to WHAC or Relate • Emotional support
Outputs The project undertook need assessments and created support plans for 80 women between February 2010 and March 2011
Primary outcomes • Reduced re-offending • Improved mental health (improved self-confidence and self-esteem) • Reduced domestic abuse and increased safety • Reduced substance misuse • Increased entry into education and employment
13
SWAN’s Theory of Change
Narrative Stakeholders/ beneficiaries State services: Police Probation Social Services Health Services Housing
Inputs
Activities/Actions
• Funding from Corston Coalition • Contributions from partnership organisations • Contributions from other agencies, such as the Police and Probation
• Identification and creation of referral routes • Development of Service Level Agreements
Outputs Increased effectiveness of State services in achieving their objectives SWAN helps these services manage their caseloads, either through taking clients that services are finding difficult to engage with, or taking clients from services earlier and providing support, enabling those services to focus their resources elsewhere.
Primary outcomes Reduction of reoffending Reduction in crime and antisocial behaviour Reduction in domestic violence Improvements in health Reduction of risk of harm to adults and children Maintenance of tenancies
14
6. The Impact Map The formal representation of a Theory of Change is represented by an impact map. An impact map is a table that captures how an activity makes a difference in terms of how it uses its resources to provide activities that then lead to particular outputs and outcomes for different stakeholders.6 SWAN’s activities contribute to creating change for multiple stakeholders and the Theory of Change for each stakeholder is presented in this section through the impact map.
Understanding outputs Whilst the focus on outcomes is crucial to understanding change it is also important to understand the scale of the change and this information is provided by the activity’s outputs. The outputs are the elements that tell us an activity has taken place and provide an indication of scale e.g. number of people attending a group. In the case of SWAN there are four key outputs (key working, counselling, groups and training) that tell part of the story created through their work with women: how long and how frequently women attend the SWAN services. These outputs will allow this analysis to understand the extent to which engagement with SWAN activities has resulted in change (or outcomes) for women. Section 5 presents the outputs and their relationship to outcomes in more detail. Only those outcomes marked in purple on the impact map are included in the SROI analysis. The rationale for this is contained in the text following the impact map.
6
Nicholls J, Lawlor E, Nietzert E, Goodspeed T (2009) A Guide to Social Return on Investment (London: The Cabinet Office).
15
Impact Map Stakeholders/ beneficiaries Women
Inputs Project costs Partner agency contributions Contributions from SWAN Partnership Beneficiary time
Activities/Actions Tailored one-to-one and group support both outreach and building based
Outputs 80 women supported in 12 months
Primary outcomes Reduction of reoffending, police arrests, court appearances, anti-social behaviour and custodial sentences
Reduction in domestic violence Improvements in health Reduction of risk of harm to adults and children Maintenance of tenancies
16
Impact Map Stakeholders/ beneficiaries State
Inputs Project costs Partner agency contributions Contributions from SWAN Partnership Beneficiary time
Activities/Actions Tailored one-to-one and group support both outreach and building based
Outputs 80 women supported in 12 months
Primary outcomes Increased efficiency - more effective management of caseloads
Increased effectiveness - able to better achieve objectives and more quickly
17
Funders and potential funders SWAN’s work is not designed to create change or value for its funders, but instead to the aforementioned stakeholders. This analysis, however, acknowledges that funders of crime reduction provision may derive a benefit in the form of an improved ability to influence the sector.
Stakeholders / Beneficiaries
1.
Funders and potential funders
Inputs
Funding
Activities/Actions
Financial support of SWAN’s activities
Outputs
Key working, counselling, groups and training
Primary Outcomes
Positive outcomes for stakeholders
Secondary Outcomes (taken forward in the SROI analysis) Wider reach
Improved ability to influence the sector
Impact map for funders and potential funders
Materiality Materiality in SROI is used to determine what information and evidence must be included in the accounts to give a true and fair picture, such that readers can draw reasonable conclusions about impact.7 The table below presents the rationale of including or excluding SWAN’s stakeholders in the analysis according to the principle of materiality. No material outcomes are expected to occur for excluded stakeholders.
Stakeholder
Materiality
Rationale
1. Women as members
Included
Key stakeholder that experiences significant change.
2. National Probation Service
Included
Key stakeholder that experiences change.
3. Families and carers
Excluded
Key stakeholder that experiences significant change but unable to monitor as work with the individual client and beyond resources of evaluation.
4. Employees that deliver SWAN activities and SWAN staff
Excluded
Often staff are not considered material in SROI analyses because they are neither primary nor secondary beneficiaries of the organisation’s work.
7
Nicholls J, Lawlor E, Nietzert E, Goodspeed T (2009) A Guide to Social Return on Investment (London: The Cabinet Office) p.97.
18
Stakeholder engagement, however, suggested that SWAN staff derive benefits beyond the purely financial; stakeholder engagement has suggested that there may be significant well-being benefits. 5. Local community (residents and businesses)
Excluded
There are some benefits to community. However, the size of these benefits compared to the overall impact is likely to be small and it is not within the resources of this evaluation to measure these. This stakeholder is not taken forward in the analysis.
6. Local third sector and voluntary organisations
Excluded
Changes for this stakeholder are only important in so far as they create change for their end beneficiary. This analysis is taking account of the end beneficiary so all material outcomes will be captured.
7. Local and national statutory organisations
Included
There is likely to be a number of economic savings that arise indirectly through SWAN’s work to this stakeholder. These will benefit the taxpayer and therefore this stakeholder is a proxy of wider society. This stakeholder is taken forward in the analysis.
8. Funders and potential funders
Excluded
Whilst their input is key to SWAN’s work, this stakeholder does not derive any benefit from SWAN’s activities and is therefore not taken forward in analysis.
Outcomes to be measured Stakeholder
Women
State
Outcomes
Feeling valued
Emotional resilience (positive feelings and optimism)
Empowerment (control over life and positive relationships)
Reduction in arrests
Reduction in court appearances
Reduction in women either in, or at risk of, police or CJS involvement
19
7. Evidencing Outcomes and Giving Them a Value The previous section identified the outcomes that result from SWAN’s work. This section focuses on how these were evidenced and measured in order to understand the extent to which change has occurred.
Outcomes indicators The occurrence of outcomes is often difficult to demonstrate; for example, health and well-being are often subjective and intangible. Indicators are a way to demonstrate that an outcome has taken place. The use of outcome indicators to understand what changes for a stakeholder is consistent with the principles of SROI and the guidance on SROI published by the Cabinet Office.8 While it is acknowledged that distance-travelled indicators are the most effective (and best practice) approach to understanding both binary change and magnitude of change, it was not possible to use this approach in this evaluation owing to the nature and timescales of the analysis. The following approach has, therefore, been adopted: using a selection of objective and subjective indicators to tell us about change, we performed a statistical analysis on the data produced by those indicators. This provided us with information about whether a significant change had taken place for the stakeholders. The following steps involved ascertaining a magnitude of change by applying our findings to average movements on a five point scale. Details of the approach and a worked example are presented later in this section.
Stakeholder
Service Users
Outcome
Indicator
1. Feel Valued Not judged Feel safe in service Feel cared for Trust/confidence
Data was collected for each outcome for the women from new primary data which incorporated the following questions:
2. Emotional Resilience Improved selfesteem In control Optimism To speak to be heard Equal
2.Indicators for Emotional Resilience “In general I feel positive about myself.” Before SWAN/with SWAN’s support? “I am optimistic about the future.” Before SWAN/with SWAN’s support “When things go wrong it generally takes me a long time for things to get back to normal.” Before SWAN/with SWAN’s support.
1. Indicators for “Feel Valued” How did you feel about yourself before SWAN/with SWAN’s support?
8
Nicholls J, Lawlor E, Nietzert E, Goodspeed T (2009) A Guide to Social Return on Investment (London: The Cabinet Office) p.9.
20
3. Feeling Empowered Control Relationship maybe defined by empowerment
State services: Police Crown Prosecution Service Ministry of Justice
1. Police Reduction of numbers of call outs, arrests and charges
3.Indicators for feeling empowered “In general, I feel in control of events in my life.” Before SWAN/with SWAN’s support. “In general, I have good, happy and enjoyable relationships with friends and family.” Before SWAN/with SWAN’s support.
Data was supplied for the state outcomes by Northumbria Police from the Police National Computer 1.
2. Crown Prosecution Service Reduction of number of court appearances 2. 3. Ministry of Justice Reduction of number of women in custody 3.
Number of callouts, arrests and charges made by police on all SWAN clients who have consistently engaged with the project for three months or more – these were counted in the six months before engagement and then during engagement. Number of court appearances by all SWAN clients who have consistently engaged with the project for three months or more – these were counted in the six months before engagement and then during engagement. A distinct cohort of women who are either released on licence and risk breach and being sent again to custody, or who have committed offences in the past which have resulted in them serving less than 12 months in prison. These women were monitored to determine if they had returned to custody after engaging with the project.
Data was supplied by the Police in two periods: July and December 2011.
Outcomes data collection In any analysis, once indicators have been established, it is necessary to collect data to establish the extent to which outcomes have occurred. In the absence of existing outcomes data, primary research must be conducted. For this analysis, a questionnaire was employed to gather primary data. Appendix 2 presents the questionnaires and the following table presents the approaches employed to collect outcomes data.
21
Data collection approach Stakeholder
Mode
Method
Approach
Timeline
Women
Outcomes data questionnaire
Questionnaire
Support offered where needed
July 2011 – Sept 2011
State:
PNC
Barefoot Research contacted Northumbria Police, who kindly agreed to provide the data
SWAN provided the police a list of current clients to be analysed
July 2011
Police and CJS
The stakeholders were sampled as follows. Questionnaires were given to SWAN key-workers who were asked to give them to all SWAN clients by hand. Some were completed by SWAN clients on their own and in some cases key workers gave assistance with literacy and understanding the questions. WHAC and Relate counsellors were not given questionnaires to give to clients as it was felt that the therapeutic relationship would impact more significantly on answers and this may have compromised the therapeutic setting also.
Outcome incidence– how much change occurred for stakeholders? Understanding what changes is the second principle of SROI and crucial to ascertaining the effectiveness of a programme or activity. Outcomes alone tell us about effectiveness, and therefore the value for money of an intervention. This section explains how the amount of change occurring for each stakeholder was understood. We have explored outcomes for two groups; the process by which we have done this is explained in this section. Firstly we examined the outputs for clients i.e. women accessing the service. Outcomes for this group include material improvements in self-esteem, self-confidence and feeling physically safe. We have collected outcome data from this group using a self-reported questionnaire at baseline and then again at three months. The changes between these two surveys are referred to as ‘distance travelled’ outcomes. Secondly, we identified outcomes that accrue to the State, e.g. number of arrests prevented. Here we have examined the number of times this outcome has occurred (outcome incidence) and valued these outcomes. State outcomes may include, for example, reduced anti-social or criminal behaviour and various health outcomes that reduce reliance on NHS funded services. Value to the State has been identified with reference to published research and current information from the Police National 22
Computer. For any given State outcome the self-reported emotional wellbeing scores are replaced by outcome incidences (i.e. number of arrests saved).
Valuing the outcomes Improvements in client ‘wellbeing’ outcomes have been valued using financial approximations of value or ‘proxies’; i.e. market traded goods or services that have the potential to deliver the same or similar outcomes and can therefore approximately represent the importance. The SROI mode uses the following formula for any given service user outcome:
SROI = Self-reported emotional wellbeing score at baseline (i.e. 3 out of 5) – Self-reported emotional wellbeing score at three months (i.e. 4 out of 5) X proportion of the proxy represented by change from 3 to 4 X financial proxy – deadweight – attribution x benefit period – drop off = net present value.
The emotional wellbeing scores were collected from a sample of seventeen service users. Where appropriate, the results were then aggregated to represent the total population of SWAN service users. To identify changes in wellbeing, outcomes data was collected from service users at a baseline stage and then three months later. Changes in these scores are referred to as ‘distance travelled’ outcome measurements. These range between one (being highly negative) and five (generally positive). This represents a quantifiable change in individual women’s wellbeing. Financial proxies were then applied to value these changes. Proportions of these financial proxies are used to value incremental improvements on this five point scale (i.e. people can suffer from degrees of anxiety or low self-esteem on a spectrum scale). To help judge the value of each individual movement on this five point scale, we have undertaken primary research with stakeholders who deliver the programmes. Alongside this we have undertaken a review of relevant academic literature. We have identified, in broad terms, proportions of the financial proxy that should be applied to movements along this five point scale. Our conclusions are summarised in the graph below and explained in the sections that follow.
23
Service user outcome journey 1.2
Proprtion of proxy applied
1
y = 0.4437ln(x) + 0.2572
0.8
0.6
Series1 Log. (Series1)
0.4
0.2
0 1
2
3
4
5
Five point scale Valuing distance travelled In general, the law of diminishing returns applied to outcome incidences. It was more challenging (hence more valuable) to support a service user’s move from ‘one to two’ than from ‘four to five’. This is reflected in figure 1. Moreover, the early stages were particularly challenging (and hence valuable for the purposes of SROI) if the service users where ‘referrals by requirement’ (i.e. as a condition of a community sentence or the CPS), then heir step change along the scale may be greater than those classed as ‘self-referrals’ i.e. choosing to access support largely of their own volition. Data collection performed over a three month period and covering a 12 month period yielded the following results for both stakeholders. Stakeholder 1: Clients The data collected by SWAN showed that the women attributed feeling valued with a net 12 month value of £338,457 as the most important outcome, followed by empowerment, £211,205 and emotional resilience of £199,037. This represents a 70% total net value (£748,699) of the £1,063,361 generated by SWAN working with women in the Criminal Justice System.
24
Stakeholder 2: State The data collected for the State showed that the intervention work that SWAN provides saves the State over one year £108,332 in arrests, £37, 064 in reduction of court appearances and £169,226 in the reduction of women in custody. This represents a 30% net value saving to the state (£314,662) of the £1,063,361 generated by SWAN working with women in the Criminal Justice System.
Financial proxies Financial approximations of value are required to value the things that matter to stakeholders and are one of the seven principles of SROI.9 Using this approach to understand what changes, proxies were applied to value the magnitude of change, denoted by the average movement from the mean. All proxies were divided equally between the number of categories in the independent variable, taking account of behaviour change having already been accounted for by the percentage movement between categories (again, to avoid double-counting). Appendix 2 presents all the financial proxies used in this analysis.
9
Nicholls J, Lawlor E, Nietzert E, Goodspeed T (2009) A Guide to Social Return on Investment (London: The Cabinet Office) p.9.
25
8. Understanding Impact SROI methodology makes an important distinction between outcomes achieved and impact. It defines impact as the difference between the outcome for participants and taking into account what would have happened anyway (deadweight), the contribution of others (attribution), whether a benefit has simply been moved from one place to another (displacement) and the length of time over which outcomes last (benefit period and drop-off). An appreciation of all of these elements is critical to conducting robust cost-benefit analyses. This section explains the approach to these elements of the methodology by working through the previous example. Deadweight Deadweight is an appreciation of what would have occurred anyway, in terms of achievement of outcomes, in the absence of the intervention/activity. Attribution The concept of attribution in SROI is an ‘assessment of how much of the outcome was caused by the contribution of other organisations or people’.10 A highly subjective element of evaluation, credit is usually claimed in its entirety or completely omitted. In organisations engaged in direct delivery, understanding the amount of credit for outcomes can be relatively straightforward through engaging with beneficiaries and wider stakeholders. It becomes more complex when organisations work in partnership with others to create change to beneficiaries who may be far removed from the partner. One of the key principles of SROI is about not over-claiming and this section will explain how this has been applied to this analysis. Displacement This is an appreciation of how much of the outcomes have displaced other outcomes. In other words, has a net benefit been created? This evaluation has analysed that no displacement is present in SWAN’s Theory of Change and therefore does not feature in the calculations. Benefit period and drop-off It is acknowledged that outcomes are not static, but instead dynamic and occur at different points in people’s lives and have different durations. SROI takes into account that benefits may last beyond the period of the intervention and, as such, takes account for this in the modelling of outcomes over time. This is known as the benefit period. Furthermore, SROI acknowledges that outcomes may deteriorate over time and this is also taken into consideration and is known as drop-off.
10
Nicholls J, Lawlor E, Nietzert E, Goodspeed T (2009) A Guide to Social Return on Investment (London: The Cabinet Office).
26
The benefit period and drop-off rates applied to the outcomes for SWAN’s stakeholders vary according to their theory of change.
9. Findings This section presents an analysis of the results from the representative sample if it were applied to all stakeholders, over 12 months. Extrapolation of our sample findings helps demonstrate what change may look like over a whole year period. Representative sample 17 WOMEN
All stakeholders 69 WOMEN and the State (police and CJS)
Top-line results for social value created are presented below, followed by a breakdown of social value by stakeholder group.
Top-line findings Benefits If we were to extrapolate the findings over all of SWAN’s stakeholders, the total added value to all stakeholders, produced over a one year period and attributable to SWAN is £1,063,361.
27
Investment Just as the calculation of the benefits arising from SWAN’s activities takes into account the wider social and economic elements, so must the calculation of the investment. This evaluation seeks to understand how SWAN spends its money. SROI analyses investments and benefits across the triple bottom line. It therefore accounts for both financial and non-financial investments employed to bring about change. The total investment required to deliver SWAN’s work over a 12 month period is £160,000. This figure does not include non-financial investment. SROI ratio Bringing the total benefits extrapolated to all stakeholders and the total investment over 12 months together and placing them in net present value terms, the Social Return on Investment ratio for SWAN is 1:6.65. This means that for every £1 invested in SWAN’s activities, £6.65 of social and economic value is created for beneficiaries. Therefore the £160,000 invested created £1,064,000 SROI. This is a positive return on investment. It is worth noting that a strong air of conservatism has been employed throughout this analysis, adding to the credibility and to the robustness of the results.
Findings by stakeholder The majority of the value (70%) is created for the women, in line with expectations and SWAN’s key aim. However, £314,662 return in benefit to the State is considerable. Stakeholder Women
Value of benefits £748,699
State
£314,662
All stakeholders
£1,063,361
Our external evaluation by Barefoot Research and Evaluation Ltd produces considerable information about the benefits to women and the State. (www.barefootresearch .org.uk11)
11
http://www.barefootresearch.org.uk/wp-content/uploads/SWAN-Report-Single-Pages.pdf
28
Sensitivity analysis Extensive sensitivity analysis on the model yielded the following findings.
Variable Outcome incidence
Activity
Change in ratio
Increase by 50%
From 6.65
16 to 20
To 7.46
12 to 18
0.81 Diff
13 to 19.5 Decrease 50%
To 5.66
16 to 8 12 to 6
Diff 0.99
13 to 6.5 Number of service users
Attribution 0.72
Increased number of service users by 20%
To 7.6
From 69 to 83
Diff 0.95
Increase 20% to 0.86
7.94
Decrease 20% to 0.58
Diff 1.3 5.35 Diff 1.3
Deadweight 0.79
Increase 20% to 0.95
7.99
Decrease 20% to 0.63
Diff 1.34 5.3 Diff 1.35
Proxies
Drop-off
Increase 50% £3100 to £4650
8.33
Decrease 50%
Diff 1.68
£3100 to £1550
4.97
Decrease by 50% 0.85 to 0.43
2.31
0.7 to 0.35
Diff 4.34
0.55 to 0.26 0.4 to 0.2 0.25 to 0.13
10. Recommendations and Learning 29
The outcomes based measurement and valuation system that SROI has provided for WCS has allowed us to highlight the value of outcomes for women in equal terms as that of the State. We believe that the outcomes for the State are dependent on those which women valued. Therefore these must be central to considerations for those commissioning services aimed at reducing offending and providing benefits to the wider community. Calculating the Social Return on Investment has been a challenging task. nef consulting has been an extremely encouraging and patient teacher. Barefoot Research and Evaluation Ltd has also provided invaluable support and assistance. Monitoring, evaluation, outcomes measures and other reporting we have done in the past will tell only part of the story. Our next challenge is to find a way to continue with SROI analysis across all our services and to use it in dialog with commissioners and funders to ensure the most effective use of investment in our sector.
30
Appendix 1 - The Questionnaire Social Return on Investment Questionnaire
1.
How did you feel about yourself?
Before receiving support from SWAN? Bad
Not Good
Not Sure
Good
Great
Not Sure
Good
Great
Good
Great
With support from SWAN? Bad
Not Good
2. “In general I feel positive about myself.” Before receiving support from SWAN? Bad
Not Good
Not Sure
With support from SWAN? Bad
3.
Not Good
Not Sure
Good
Great
“I am optimistic about the future.”
Before receiving support from SWAN? Strongly Disagree
Disagree
Not Sure
Agree
Strongly Agree
Not Sure
Agree
Strongly Agree
With support from SWAN? Strongly Disagree
Disagree
31
4.
“When things go wrong it generally takes me a long time for things to get back to normal.”
Before receiving support from SWAN? Strongly Disagree
Disagree
Not Sure
Agree
Strongly Agree
Not Sure
Agree
Strongly Agree
With support from SWAN? Strongly Disagree
5.
Disagree
“In general, I feel in control of events in my life.”
Before receiving support from SWAN? Strongly Disagree
Disagree
Not Sure
Agree
Strongly Agree
Not Sure
Agree
Strongly Agree
With support from SWAN? Strongly Disagree
6.
Disagree
“In general, I have good, happy and enjoyable relationships with friends and family.”
Before receiving support from SWAN? Strongly Disagree
Disagree
Not Sure
Agree
Strongly Agree
Not Sure
Agree
Strongly Agree
With support from SWAN? Strongly Disagree
Disagree
32
7. None
8.
To what extent has SWAN made a difference to your life? Some
Not Sure
A lot
Greatly
What would you be doing if SWAN did not exist?
Thank you for talking the time to fill out this questionnaire. If you have any questions please call Katie Wheatley at SWAN on 01670 362 491
33
Appendix 2 - Summary of Assumptions The following table presents details of how each outcome incidence was calculated for each stakeholder.
Assumptions Basic Information e.g. demographic info such as number of women 19 women completed questionnaire one. 17 women responded to STATE
Police: Reduction of numbers of call outs, arrests and charges
Crown Prosecution Service: Reduction of number of court appearances
Ministry of Justice: Reduction of number of women in custody
Value 17 women
£1,780
£812
£3423.67 per month
Source We used a non probablility method to select interviewees. The cost of one arrest Total cost £1780 (% distribution based on Devon and Cornwall Constabulary figures)Taken from Page, A. 2011, Counting the Cost: The financial impact of supporting women in the criminal justice system, Revolving Doors Agency. Page 41. Average cost for proceedings in magistrates courts for all offences excluding violent, sexual and motoring offencesTaken from Page, A. 2011, Counting the Cost: The financial impact of supporting women in the criminal justice system, Revolving Doors Agency. Page 41. Average cost for one month in custody. HMPS Annual Report and Accounts 2006-07 Appendix 1 – Statistical Information. Taken from Page, A. 2011, Counting the Cost: The financial impact of supporting women in the criminal justice system, Revolving Doors Agency. Page 41.
34
Outcome data e.g. results from each indicator per outcome Feeling Valued Q1a. How did you feel about yourself? Before receiving support from Q1b. How did you feel about yourself with support form SWAN? Emotional Resilience Self Esteem Q2a. "In general I feel positive about myself." Before receiving support Q2b. "In general I feel positive about myself. After receiving support Optimism Q3a. "I am optimistic about the future." Before receiving support from Q3b. "I am optimistic about the future." With support from SWAN? Resilience
Q4a. When things go wrong it generally takes me a long time for things to get back to normal. Before receiving support fom SWAN?
Q4b. When things go wrong it generally takes me a long time for things to get back to normal. With support fom SWAN?
Empowerment Q5a. "In general, I feel in control of events in my life." Q5b. "In general, I feel in control of events in my life." With support Q6a. "In general, I have good, happy and enjoyable relationships with friends and family". Before receiving support form SWAN? Q6b. "In general, I have good, happy and enjoyable relationships with friends and family". After receiving support form SWAN?
Value 5-point scale. 80%
Source Results from Questionnaire 2
Poss Mobex swan project 5-point scale. 80% Poss Mobex swan project
Results from Questionnaire 2
Poss Mobex swan project
£473. Health insurance (this is for a relatively healthy person – there is very little available data on health insurance for those with health problems – this is therefore lik ely to be a conservative estimate). Alternatively £960. Price of life insurance for a soldier on tour of duty in Afghanistan. (From BBC ‘Boy soldiers’ Source AXA Heath insurance – assumes healthy individual. aired 29th September 2011). This provides a crude indication of how the mark et values risk /dangerous occupations. 5-point scale. 80% Results from Questionnaire 2
·
£542. Cost of home security system.
(Average taken from primary web based research). http://www.premierself-defence.co.uk/cms/Default.asp?Page=20
· £11 self defence class/£48 self defence workshop/£120 one day self defence course.
http://www.yellowdragon-self-defence.co.uk/ http://www.krav-maga.org.uk/index.php?mod=membership
SWAN Police data Number of arrests Number of court appearences Number of women in custody/at risk of custody Q7. To what extent has SWAN made a difference to your life? Other services disengaged after SWAN referral because of their high workloads and primary role to other work There are no other similar mobile services or partnerships Family and Friends
Prior 24. Post 8 Prior 13. Post 1 Prior 14. Post 0 5-point scale. 80%
0 0
PNC
Results from Questionnaire 2
Barefoot research Case worker reported benefits
35
Impact data: deadweight, how much of the outcome would have happened anyway? Q8. What would you be doing if SWAN did not exist? Impact data: displacement, how much of the outcome has been none Benefit period: how long do outcomes last for? SWAN Police data Number of arrests Number of court appearences Number of women in custody Drop off: at what percentage do outcomes drop off (change) over Estimate of fully exited client: 1 year 5% drop off 2 year 10% drop off 3 year 25 % drop off 4 year 40% drop off 5 year 20% drop off
Value
Source
21% Value 0 Value
nef Source 0 women have moved away Source PNC. Data collected from clients engaged in service from 01/07/11
Value
Source
36