a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk
|
tw: @GenChangeUK| t: 0207 014 2690
Community outcomes roundtable – write up When: Thursday 1st May 2014 Where: vInspired, Dean Bradley House, Horseferry Road, London Attendance: Chair:
Hannah Mitchell
Head of Knowledge
vInspired
Paul Adnitt
Curriculum Manager
NCS Trust
Alice Thornton
Head of Impact and Policy
Student Hubs
Xenis Davis
Programme Manager
Envision
Nick Ockenden
Head of IVR
NCVO
Jon Adamson
Senior Research Manager
CFE
Ewan King
Director
OPM
Amy Skipp
Research Director
NatCen
Rania Marandos
Deputy CEO
Step Up To Serve
Background In June 2013, the Young Foundation published a Cabinet Office commissioned scoping paper setting out a Quality Framework for youth social action.1 The framework includes a definition, set of principles and an outcomes framework that young people would build up on their social action journey from aged 10 -20. Generation Change originally set out the brief
for commissioning this framework and was instrumental in setting out the “double benefit” language as a guiding model for quality youth social action.
In December 2013, Generation Change was then commissioned by the Cabinet Office to expand upon the “individual benefits ” part of the double impact framework by developing a
1
Slide 11, “Scoping a Quality Framework for Youth Social Action”, The Young Foundation, June 2013
common metric for organisations to assess outcomes against each of the 8 skills areas. Generation Change brought together stakeholders from multiple sectors and staff from member organisations to identify the challenges of reporting outcomes in this area. Our recommendations were supported by IVR and the Young Foundation, and set out clear options for developing reporting in this area. Double benefit model
Generation Change has now been appointed by Step Up To Serve to lead its project on implementing a quality framework for youth social action programmes, taking forward these developments, and drawing on our delivery organisations. We have established a pilot group of professionals directly involved in evaluation activities in order to support the Step up to Serve campaign by developing common approaches that embed the quality principles and outcomes framework. It is evident that the first step in this project is to expand upon how to evaluate and measure the “benefits to society�, which forms the other half of the double benefit framework. This was one of Generation Change, IVR and the Young Foundation’s recommendation to the Cabinet Office when developing a common metric. With different social action programmes having unique interventions into a range of social causes, it appears more difficult to assign a set of common outcomes which all youth social action programmes could be asked to report on. On 1st May 2014, Generation Change hosted a roundtable with research agencies and evaluation specialists for an initial scoping session for this work. The session was chaired by Hannah Mitchell, Head of Knowledge at vInspired, and written up by David Reed, Forum Coordinator at Generation Change.
a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk
|
tw: @GenChangeUK| t: 0207 014 2690
Defining “community outcomes” A range of possible definitions were suggested at the roundtable:
Community outcomes are… Any outcomes that are not a benefit to the individual. A negative definition, in that beneficial outcomes that result from a social action programme, not directly related to the individual taking part, is considered a “community outcome”. Problems:
Some changes in the individual are considered socially beneficial
•
(e.g. community cohesion, reduction of violence etc) Longer-term impacts derive from behaviour change in the
•
individual; such as an increased propensity to give, being more socially minded in future decision making, etc. Role modelling of positive behaviour – the fact that youth social
•
action is taking place at all, rather than not, could have community outcomes in itself or a knock on effect. Defined by the indicator or cause. Most social action projects take a theory of change approach whereby the outcomes are determined by the analysis of a social problem / need. Each cause will have its own set of indictors. Problems:
•
Endless hair splitting of different causes – e.g. “environment” could be: “biodiversity, conservation, climate change, tidiness, community spaces…” etc.
•
This evaluation approach alone does not leave social action with a framework that is easily comparable.
Beneficiaries / communities that experience impact. Many programmes attempt to identify a beneficiary, or beneficiary community, and measure benefit directly there. Problems:
•
Definition of community can be discrete based on the individual
•
Some interventions have positive impacts that are not easily tied to a specific beneficiary community (e.g. tree-planting)
•
Localities and networks play a significant role in understanding impact, complicating this picture of beneficiaries.
a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk
|
tw: @GenChangeUK| t: 0207 014 2690
Defined by the young person. Many providers – e.g. NCS, vInspired, Envision, Fixers frame the “community impact” part of their programme as up to the young person to decide in terms of the cause and means of addressing it. In this case, each project will have its own approach to defining community impact. Problems:
•
Difficult to compare or contrast different evaluation approaches
•
Social action projects need to support young people to learn about community impact, so this definition does not solve the initial problem. Conclusions of this section:
•
None of these definitions is adequate alone, or suitable in all cases
•
The following areas below appear important to separate: Outputs Positive change added
Reduction in a negative indicator
Impact of young people
Impact of youth social action generally
Immediate / direct impact Behavioural changes Micro impact
•
Outcomes
Cumulative / culture shift Attitudinal changes Macro impact
Whereas there seems to be importance in the inter-relation of the following:
E.g. mentoring, awareness
Mode of intervention
E.g. biodiversity, homelessness,
campaign, fundraising
educational disadvantage
Cause
Beneficiaries
E.g. elderly care home residents, bullied under 16s, my local park
Measuring “community outcomes” Kind of measure used
E.g.
Outputs
…Numbers taking part, attendance of events, tweets
Responses from direct beneficiaries
…Survey of elderly person, report cards, focus groups
Direct set of indicators around a particular issue
…Wellbeing metrics for mentoring of bullying
Universal positive indicators
…Employment rates / voter intention
Responses from immediate social circles
…Teachers / parents / facilitators / friends
Attitudinal statements and future beliefs
…“I believe I am more likely to get a job now”
Second party data / indicators
…attainment, indices of deprivation, local voter turnout
Infrastructural / economic benefits
…capacity of providers, transport links, economic benefits
Media and policy
…news agenda, endorsements, media coverage
A possible means of modelling these measurement approaches was identified: Outputs, direct outcomes and indicators, immediate responses Longer term impact
Close social circle / network
atitudinal statements and beliefs, positive specific indicators, second party data
Individual
Direct beneficiaries
Broader community
Wider / universal data, infrastructure and economics, media and policy
Conclusions of this section: •
Measuring the outcomes of a specific project is different to measuring the broader community outcomes of social action generally, although they are linked.
•
The former might require flexible frameworks for determining appropriate measures, whereas it might be possible to set out general outcomes for the latter (next section)
a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk
|
tw: @GenChangeUK| t: 0207 014 2690
Common long term outcomes Roundtable participants were asked what they would want to be able to measure in an ideal world to demonstrate the long-term community outcomes of social action taking place. The following suggestions made were not intended to be exhaustive: Measurement scale and focus “Individual”
“Communities”
“Society”
Positive traits more common
Responsible and engaged
Active citizenship
and recognised (e.g. social,
communities
ambitious, compassionate, engaged) Changes in behaviours (e.g.
Generational qualities –
giving, decisions, choices)
understanding / acceptance of
Cohesion, tolerance
social differences Increased social capital / life
Social problems alleviated
chances with positive multipliers
(e.g. greater wellbeing)
Voice and influence of young
Changed social attitudes
people
towards young people
Shift in social trends
Participation
Practical challenges Providers cannot be a subject specialist in every cause! Some programmes run a very specific intervention (e.g. mentoring in schools) that is easier to build an impact model around; whereas providers such as vInspired, Envision, NCS and Fixers effectively have multiple interventions, many of which might be up to the young person to decide upon in terms of the cause, means of addressing that cause, theory of change, etc. Solution: Do Something uses standard evaluation frameworks based on the most common intervention approaches (e.g. “mentoring”, “awareness campaign” etc) Requirement for a control group. Proving outcomes puts demands on organisations to use sophisticated approaches such as RCTs which is not feasible to run for multiple ages, contexts and programme lengths in all cases. Solution: Use derived data, external evidence bases, and / or national indicators to provide baseline control areas to compare to. Open data solutions could make it much easier to identify and respond to community outcomes.
a: 58 – 62 White Lion Street, London, N1 9PP w: www.generationchange.org.uk
|
tw: @GenChangeUK| t: 0207 014 2690
Attributing impact over a longitudinal period. Most of the meaningful outcomes organisations need to prove are not captured within the timeframe of a single project, but might be experienced long after a young person has left the programme. Solution: similar to above solution, common data sets needed.
Actions •
Capture and compare different interpretive models for community outcomes for consideration by the group
•
Propose a definition of “active citizenship” to set out a vision for the kind of outcomes youth social action programmes seek
•
Look into existing template frameworks used to help clarify outputs and outcomes for different forms of social action (e.g. “mentoring, planting trees”)
•
Use a deliberative engagement exercise with OPM to capture young people’s own interpretation of the kinds of community impact they’re having and would like to have.
•
Compile and review existing evidence from member programmes in this area and compare to common standards of evidence produced by TSIP
•
Identify solutions with Step Up To Serve to national data indicators and insights that could help establish baseline figures.