Scoping a Common Metric for Young People Participating in the Social Action Trials December 2013
1
Generation Change Generation Change is a partnership of 18 specialist youth social action organisations transforming the UK by empowering over half a million young people to take action in thousands of local communities and schools. Generation Change brings together leaders in the UK youth social action sector, facilitating collaboration between our members, driving quality standards, representing the sector to stakeholders, and creating infrastructure to increase the quantity and quality of social action taken by young people in the UK today. We have a vision of all young people in the UK being involved in double impact social action that builds the skills, confidence and character they need to address our country’s most challenging social issues.
We have four streams of work to achieve this vision:
Quality: Implement a shared quality framework across organisations to help demonstrate the value of social action programmes to employers and funders. Insight: Establish a knowledge hub on youth social action that maps provision of opportunities in the UK, and compiles evidence around demand from young people. Action: Facilitate collaboration across sector to catalyse new initiatives for expanding the reach and impact of youth social action. Awareness: Draw attention to the youth social action agenda to increase the widespread recognition of its role in society and for young people.
Find out more at www.generationchange.org.uk
2
Contents Background: .................................................................................................. 4 Process .......................................................................................................... 6 Example: Scoping a social action metric
.......................................................... 7
Sources .......................................................................................................... 8 Stakeholder feedback .................................................................................... 9 Focus group study ........................................................................................ 10 Creating a core set of statements: ................................................................. 11 Option 1a ..................................................................................................... 14 Option 1b ......................................................................................................15 Option 2....................................................................................................... 16 Recommendations ....................................................................................... 18 Appendix ..................................................................................................... 20 Acknowledgements ..................................................................................... 21 Full references ............................................................................................. 22
3
Background In June 2012, the Young Foundation published a Cabinet Office commissioned scoping paper setting out a Quality Framework for youth social action. 1 The framework includes a definition, set of principles and an outcomes framework that young people would build up on their social action journey from aged 10 -20. This was developed as part of a Quality Framework consultation and drew together work from IVR, Young Foundation and CBI.
Optimism
Communication Creativity Confidence and Agency
Determination
Planning Problem solving Resilience, Grit Leadership
Emotional Intelligence
Relationships Managing feelings, self control Empathy
Generation Change was central to the process of developing this framework. Generation Change is group of 18 national organisations that are either specialist youth social action providers or organisations that have unique importance to the field. Generation Change has a shared vision for increasing participation in social action in the UK, and joint goals, the first of which is to establish a quality framework in the youth social action sector that includes common outcomes metrics. As set out in the Government’s recent report Encouraging Social Action2, the Quality Framework is intended to be tested as part of the Youth Social Action Fund trials with the aim of validating it as a robust model that all organisations delivering youth social action can link up to in the future. This will support the development of a common language about the impact of social action that demonstrates its benefits to educational institutions and employers. Another key objective of this project is to create a large amount of data on outcomes in young people who participate in social action. The Cabinet Office is running a set of trials programmes due to launch early in 2014 to achieve these goals.
1
Slide 11, “Scoping a Quality Framework for Youth Social Action”, The Young Foundation, June 2013 Published, 27 November 2013 [https://www.gov.uk/government/publications/encouraging-socialaction] 2
4
The Young Foundation’s outcomes framework was at the heart of the Quality Framework and is a flexible tool to measure youth outcomes. This framework was developed further to articulate the “double benefit” which is at the heart of the Cabinet Office’s approach as well as the Step Up to Serve / Campaign for Youth Social Action. The double benefit model sets out two distinct, interrelated areas of outcomes that social action achieves; firstly benefitting the individual young person taking part by developing their skills, and secondly benefitting the community by delivering positive social impact. In addition to giving social action providers a means of articulating their own impact, the Quality Framework will allow young people to capture what they learn by taking part in social action in a way that resonates with schools and employers.3
Communication
Optimism
Creativity Confidence and Agency
Determination
Planning Problem solving Resilience, Grit Leadership
Emotional Intelligence
Relationships Managing feelings, self-control Empathy
The outcomes statements should overlap with the CBI First Steps framework, as these are qualities that employers value. Metrics should be backed up with academic metrics that give verified, reliable measures of different character traits. And they should also be relevant to existing outcomes measured by social action providers.
3
Page 33, “First Steps: A New Approach to Our Schools”, CBI, November 2012 [http://www.cbi.org.uk/media/1845483/cbi_education_report_191112.pdf]
5
Process The Cabinet Office commissioned Generation Change from 18th November to 6th December to develop a set of metrics that could sit against the outcomes framework to record and measure the outcomes of social action activity on all young people participating in the Youth Social Action Trials. Previously, Generation Change had provided in depth thinking on implementing common metrics across its membership of specialist providers. The Challenge Network’s POPI metric had widespread support from Generation Change and forms the basis of this proposed metric. We also drew on the CBI “First Steps” report, based on a comprehensive study into the skills most required by employers, and the ways in which employers would expect these skills to be exhibited in the workplace. Finally, Generation Change was grateful to be able to draw on existing toolkits created by the Young Foundation4 and New Philanthropy Capital5 as well as literature review published by the Educational Endowment Foundation.6 Generation Change followed a seven step process: 1. Compiled a bank of self-assessment metrics drawn from Generation Change
member organisations. 2. Conducted a review of existing, verified self-assessment metrics to produce
an extensive bank of relevant metric statements against each characteristic. 3. Consulted a core group of experts on how best to whittle down this list to a
common set of statements. 4. Consulted a range of providers likely to use the quality metric to assess their
programmes. 5. Tested with 8 focus groups of school children via our network, and experts in
the school context, with particular focus on Key Stage 2 and 3. 6. Compared findings with the CBI’s own framework to make it relevant to
employers. 7. Worked with IVR and the Young Foundation to assess our process and
produce a set of recommendations for taking this forward to the trials. This was a very collaborative process, shaped by a number of organisations and individuals already leading similar work, either through their own programmes or at a research level. A full list of acknowledgements can be found at the end of this report. 4
“A Framework of Outcomes for Young People”, The Young Foundation, July 2012 [http://youngfoundation.org/publications/framework-of-outcomes-for-young-people/] 5 The Journey to Employment: a guide for understanding and measuring what matters for young people, Inspiring Impact (New Philanthropy Capital), May 2013 [http://inspiringimpact.org/wpcontent/uploads/2013/03/Inspiring-Employability-final-for-web.pdf] 6 The Impact of Non Cognitive Skills on Outcomes for Young People: Literature Review, The Educational Endowment Foundation, 21 November 2013 [http://educationendowmentfoundation.org.uk/library/the-impact-of-non-cognitive-skills-on-outcomesfor-young-people-full-report]
6
Example: Scoping a social action metric Generation Change believes that an outcomes framework must achieve five goals in the long term:
1) Be Outcome Focused: Until we have proof about which pedagogy/approach delivers high quality outcomes, data must focus on outcomes not just pedagogy or approach. 2) Be distance-travelled: With a clear starting and end point to show change over time. 3) Provide comparable data: Data must of the same type for all delivery. We must be able to ask the same questions and collect comparable data for all social action projects. 4) Be Simple to Collect: Data must be easy to collect that all delivery can be required to collect it at minimal cost, without restricting programmes. 5) Provide Immediate Data: Data must be immediately available – not requiring months/years of longitudinal research – otherwise it will not change behaviour of funders and delivery organisations that operate on much shorter time horizons. Through preliminary workshops, Generation Change found that The Challenge Network’s POPI metric meets these five goals. POPI works in the following way:
Initial Score: At the start of the social action, the young person rates themselves from one (lowest) to 10 (highest) against a set of statements. E.g. a young person rates themselves at 7 out of 10 for “I can achieve a lot if I put my mind to it”
Potential Increase: The potential for increase is calculated for each question by subtracting each young person’s score from 10 (the maximum score). E.g. Their potential increase is 10 – 7 = 3. This is the highest they could score themselves at the end of the programme.
End Score: At the end of the social action, young people rate themselves against the same statements. E.g. At the end of the programme, the young person rates themselves as 9 out of 10; they are now more confident in their ability to achieve.
Increase: calculated by subtracting “Initial Score” from “End Score”. E.g. This is an increase of 2.
POPI: to find out how much of the maximum potential increase each young person has made, the “Increase” is divided by the “Potential Increase”. This gives a figure for how much of the possible improvement has been realised. E.g. This means that the young person made 2 ÷ 3 = 66% improvement of what was possible. Their POPI score for this question is 66%.
7
Sources To scope the common metric, Generation Change compiled self-assessment surveys from three sources:
1) Existing programme providers who have adapted their assessment questionnaires to suit the needs of the programme. This ensures that the metric is relevant to social action providers. 2) Established psychometric tests for each of the outcomes characteristics. We have drawn on prominent literature reviews to identify metrics that are relevant to the 8 skills areas agreed in the framework. This ensures that our final metric is valid measure of each characteristic. 3) Research organisations that have already looked into developing a common outcomes framework for young people. This ensures that our process was comprehensive, and consolidated existing work, rather than re-inventing the wheel. Group
Name
Abbreviation
Programme providers
Envision National Citizen Service vInspired The Diana Award Future Foundations UpRising Student Hubs The Challenge Network The Scouts Association The Citizenship Foundation
Env NCS vInsp Diana FF UpRis SHs TCN Scouts TCF
Academic metrics
Strengths and Difficulties Questionnaires (Goodman 1997) Social Skills Rating System (Elliot & Gersham 1997) Self Description Questionnaire (Marsh, 1990) Rosenberg Self Esteem Scale (Rosenberg 1965) The Duckworth Grit Scale (Duckworth, Peterson et al 2007) The Self-Control Scale (Tangney, Baumeister et al 2004) Adolescent Self-Regulatory Inventory (Moilanen, K. L. 2007) The Leadership Skills Inventory (Townsend, Carter 1983) Roets Rating Scale for Leadership (Roets, 1986) The Baron-Cohen Empathy Quotient (Baron-Cohen, S., & Wheelwright, S. (2004) Generalized Self-Efficacy scale. Schwarzer & Jerusalem (1995) The 7-item Pearlin Mastery scale (Pearlin & Schooler, C. (1978) Toronto Empathy Questionnaire (Spreng, McKinnon et al) 2009 Empathy – Teen Conflict Survey (Bosworth & Espelage, 1995) Development of an empathy index rooted in social cognitive neuroscience and social justice (Gerdes, Lietz, Segal 2011)
SDQ SSRS Marsh Rosenberg Duckworth SCS ASRI Town RRSL BaronCohen GSE Pearlin Toronto Bosworth
CBI “First Steps” Report 2012 Young Foundation - Framework of Outcomes for Young People New Philanthropy Capital – JET Framework Office for National Statistics Subjective Well Being Scale California Healthy Kids Survey (Social Emotional Health Module & Resilience and Youth Development Module) VIA Me! Character Strengths Profile (VIA® Institute on Character (Peterson, Christopher, and Martin E. P. Seligman. Washington, DC: American Psychological Association, 2004.)
CBI YF NPC ONS
Research organisations
Lietz
HKS VIA
8
Stakeholder feedback Generation Change has engaged a set of 41 stakeholders as listed in the acknowledgements section. Below is a summary of the key feedback. The following outlines the range of feedback we have received from the metric:
Statements must be responsive to age range; e.g. 10 year olds are not supposed to be employment ready, therefore skills have a different context.
Academic metrics tend to include at least 10 statements for any one character trait, which for this framework would require nearly 100 statements. However, for social action programmes, such an in-depth psychometric test is inappropriate for young volunteers, or vulnerable groups. A maximum total figure of 20 statements was therefore recommended.
Baseline and follow up self-assessments are not appropriate for certain contexts of social action; e.g. pop-up one-day engagements.
Self-assessment metrics on their own are only useful for testing outcomes as an aggregate – i.e. for a cohort or group of young people – rather than being used to capture an individuals’ progress.
Triangulating outcomes using other measures should be the long term goal of any outcomes assessment. For example, young people building up a portfolio of evidence, and teacher assessment of this, could help create synergies with teachers and education institutions.
The method of data collection is a predictor of response rate, with most providers preferring the flexibility of both an online and offline (paper) system in order to achieve higher percentage of young people completing the survey.
Empathy is an important part of the outcomes framework, but could be better ‘unpacked’ to include other pro-social behaviour / character traits as per findings by the Jubilee Centre.
Some organisations are already using The Young Foundation outcomes framework as part of their own development of impact metrics and analysis. There needs to be flexibility to include their metrics in the roll out of any common set of metrics.
Providers need to have the flexibility to add to metrics to allow additional reporting in areas that are relevant to their funders. The eventual aim of a quality framework should be to give providers flexibility to use tools that are appropriate to their intervention model.
Stakeholders are very keen to start using common metrics and there is considerable motivation to work together to make the framework work across a range of ages and contexts. There is considerable good will and consensus that needs to be harnessed. 9
Programme provider stakeholders are very keen to start using common metrics and there is considerable motivation to work together to make the framework work across a range of ages and contexts.
Representatives in the education and business sector were most interested in seeing metrics be tailored for different age ranges, to see skills built up over the journey from 10 to 20.
Focus group study Generation Change was able to test a draft set of statements amongst children of different ages within the target group of 10 – 20 years old. City Year asked corps members to conduct a self-reflection exercise with young people in years 6 and 7 (ages 10 – 11), as well as years 9 - 11 (ages 14 – 16) using a draft version of 20 statements. They were able to conduct this study across 8 different focus groups in schools in both Birmingham and London. Corps members were asked to record: A) young people’s comprehension and interpretation of the statements and B) their confidence / how they behaved in completing the survey. Simultaneously, Student Hubs were given the same set of statements and asked to consider how relevant the statements were to student volunteers at ages 18 – 20. Key findings:
All age ranges were able to understand the survey statements, although their interpretation of them varied.
A striking finding was that the questions highlighted by corps members as being more difficult for younger age ranges to comprehend, were the same ones that were highlighted by Student Hubs as being the most relevant.
Younger age ranges were more likely to score themselves very highly (9 or 10) if they were conducting the exercise ‘publicly’ with their peer group or the assessor.
There were some surprising results when looking at individual responses. For example, one participant aged 13 responded with either a 9 or 10 to all 20 statements. However, another participant in the same cohort responded with a full range of numbers from 1 to 9. This supports feedback that self-response metric data is most useful when examined as an aggregate.
10
Creating a core set of statements The key objectives for this tool are to firstly ensure a high level of cross comparable data capture, and to balance this with the validity of measures, which necessitates a fit to context. These two aims require a compromise. There is also the ambition to capture data on an aggregate and individual basis. As such, Generation Change sets out three options and recommendations for how a universal outcomes metric could be developed over the life of the trials. Generation Change has compiled a bank of over 450 verified statements used in relevant psychometric self-assessment surveys, existing programme surveys, and organisations that have previously looked into outcomes in young people. Looking at these statements, it quickly became apparent that some of the categories in the outcomes framework do not neatly align with existing research into non-cognitive skills. For this reason Generation Change has separated “confidence and agency” into two reporting areas, and placed “planning” and “problem solving into one reporting area. There is also overlap between metrics intended for measuring leadership, or empathy where there is a clear skills focus in one of the other areas (e.g. communication). From consulting a wide range of stakeholders, (see feedback above), it is clear that young people cannot be expected to respond to more than 20 survey statements. To create an initial shortlist of this wider pool of validated metrics, Generation Change grouped statements that have a high level of cross over within the existing academic metrics that relate to these measures. E.g. I have overcome setbacks to conquer an important challenge Setbacks don't discourage me
Little problems detract me from my longterm plans "Learn to take positives from failure experienced" Even when things are going badly, I usually think things will work out in the end When things go wrong, I don't let it get me down If something goes wrong I am able to bounce back and carry on
N
Duckworth Grit Scale (Duckworth, Peterson et al 2007) Duckworth Grit Scale (Duckworth, Peterson et al 2007) Adolescent Self-Regulatory Inventory (ASRI) CBI
Grit
TCF - Go Givers
Determination
Envision Assessment Survey 2013 -14 (Envision) vInspired evaluation survey (Team V Programme)
Resilience
Grit
Shorter term regulation Grit
Resilience
We suggest that one statement should be included in place of the above 7 sources. There are three ways that this can be chosen:
Combine statements where there is evident parity (e.g. for the above: “When things go wrong, I don’t let it get me down”). The sample size of the trials would then test the reliability of the new wording. (Option 1a)
11
Group statements such as in the above and test all of them across the trials so that each individual participant is only asked one of the statements. This requires a more strategic approach to testing and limits sampling on any one statement. (Option 1b)
Preserve the original wording of statements from academic studies, and make an uninformed choice as to which should be used. Because the decision would be uninformed, this is not a recommended approach.
Because there are 10 outcomes to be tested through these trials, it simply will not be possible to preserve the entirety of an existing metric’s statements where it only measures one character trait. This means that the existing research reliability of the measure as a previously validated resource is necessarily compromised. However, the social action trials offers a unique opportunity to conduct a process of factor analysis, and to compare with evidence collected by researchers, to test the validity and reliability of statements used in a variety of contexts. This can lead to the creation of a robust and validated set of metrics at the end of the trials.
Option 1: Creating a core set of statements for all programmes There are two ways this metric could be developed: a) merging existing academic statements with on the ground practitioner statements and trialling this unvalidated approach across all programmes b) trialling the academic statements in a range of samples to understand which is most reliable and valid. This will capture a large amount of data for each statement. The metric would need to be paired with detailed contextual data to inform analysis at the end of the process. Because this option requires all programmes to use the same questionnaire, there is a strong likelihood that the metric will not be applicable in all cases, and therefore a risk that some providers return results that do not truly reflect their programme outcomes. Generation Change does not recommend this option. Option 2: Use a core set of statements alongside age specific statements The metric would still have a core set of statements that all providers ask young people as above, but added to this would be an additional set of statements chosen by age context. Over such a large sample size (26 trial programmes) this would allow the Cabinet Office to give individual young people less questions, but as a whole test a wider pool of metrics, and in so doing build up a picture of the “journey” of social action. The total number of statements in one survey would be no more than 25 – but over the whole sample, up to 50 statements could be tested.
10-12 – more primary focused language, more knowledge and experience with less focus on work readiness skills 13-16 – building up greater applied skills and knowledge 17-20 – specific work readiness language 12
From our stakeholder consultation there is concern about this. The metrics need to be connected to a progressive framework that creates a pathway where skills and experience is built up. It also gives the Outcomes Framework flexibility to link up to school curricula, such as the Citizenship GCSE or other school based projects.
Option 3: Use detailed analysis to create a core set of statements alongside context specific metrics through the trial process The outcomes framework from the Young Foundation is very much focused on programmes learning as a result of an iterative process by looking at their theory of change, their objectives and then over time aligning their activity. A number of programmes are using this framework already, so it would be useful to have open options so programmes can use the statements that they have developed through their internal, bespoke impact analysis. By including a range of open options to help learn lessons of how the outcomes work in a variety of contexts and across a variety of lengths of programmes this will provide a detailed picture to create a more robust tool that is context and age specific. In the long term, this will also ensure that lessons about how the journey can fit together with different programmes at a variety of stages and ages can create the right social action progression for young people. Such a phased approach to the trials will ensure that a validated final shortlist of statements can be created. IVR and Young Foundation agree a more flexible approach at the outset of the trials is preferable and are now looking into these options as part of their report. See below for illustrative examples of these options. See statements bank for full list of over 450 referenced statements.
13
Option 1a Social action should encourage:
This means to instil the following attributes:
Communication Creativity
Optimism
Confidence Agency
Determination
Planning and Problem solving Resilience, grit Leadership
Emotional Intelligence
Relationships Managing feelings, self control
Empathy
Attitudinal statements that might measure these attributes are:
These statements are a combination of statements found in the following sources:
I find it easy to say what I want to say I'm good at adapting the way I communicate based on the situation I am good at coming up with new ideas I am confident voicing my opinions in public I feel at ease asking people with authority for advice I believe that my actions can make a difference in my community I understand the organisations and people that have influence in my local area I am good at planning a project I manage my time well to get lots done I consider a range of different solutions to a problem When things go wrong, I don’t let it get me down If I start something I usually finish it I am good at motivating others to get things done I try to involve others in activities or conversations I am able to compromise and resolve differences of opinion I get on with people who are very different to me I am good at controlling my feelings in difficult situations I stay focused and am not easily distracted
POPI
vinsp
SSRS
Baron-Cohen
CBI
Env, POPI
VIA
RRSL SSRS (1990 NCS )
UpRis, TCF RRSL
NCVYS Diana Award POPI
Baron-Cohen (2007)
NCS
UpRising
CBI SSRS
UpRising Env, vInsp
vInsp
RRSL
CBI
Env, TCF
vInsp
CBI
Duckworth
NCS
CBI
Duckworth
NCS
CBI CBI
POPI, UpRis SSRS
CBI
I am good at understanding where people are coming from, and can put myself in their shoes. I always consider other people’s feelings when I act
Env, POPI
My own emotions are affected if other people are happy or sad I feel knowledgeable about the key issues affecting people in Britain today
Env, TCF
TCF
Baron-Cohen
CBI
Env, TCF
NCVY S
FF, VIA
vInsp, Diana Baron-Cohen
VIA
Townsend
RRSL
Env
RRSL
VIA, HKS
Diana
NCS, RRSL
CBI
SSRS, SCS
ASRI
VIA, SDQ
CBI
Duckworth, SDQ
Town
HKS, ASRI, SCS
CBI UpRis UpRis
Town, RRSL
Lietz, HKS
Env
VIA
Baron-Cohen
UpRis , TCF
Baron-Cohen, SDQ SSRS, Marsh Bosworth
SSRS
Bosworth
ASRI
Lietz,H KS
Baron-Cohen
CBI
SHs
CBI
14
Option 1b - illustrative These statement wordings are taken directly from academic sources. ----Character measure
Sample 1 I can express the same idea in different ways depending on the situation
Communication
Creativity
Confidence
People usually understand when I give instructions I can bring others’ thoughts together to form new ideas I am always using my imagination to help me think of new ways of doing things I feel confident speaking to people who I don’t know I have goals and plans for the future
Agency
When I make plans, I am almost certain that I can make them work I understand the organisations and people that have influence in my local area. If something isn't going according to my plans, I change my actions to try and reach my goal
Planning and Problem solving
Resilience, grit
Leadership
Relationships
Empathy
I communicate well with others I can simplify complex ideas so that people can understand them I frequently have creative ideas. I am open to new ways of doing things I can do most things if I try I regularly step outside of my 'comfort zone' I am able to make a positive difference in my community What happens to me in the future mostly depends on me. I can break down a plan into clear steps and put them in order
I manage my time well to get lots done
I am good at keeping track of lots of things going on around me, even when I’m feeling stressed
When I am confronted with a problem, I can usually find several solutions
I am good at solving problems
I have overcome setbacks to conquer an important challenge I finish whatever I begin People look up to me as a leader and they give me their trust. I am concerned with how my decisions affect the lives of others I am able to compromise and resolve differences of opinion Other people my age generally like me
Managing feelings, self control
Sample 2
Setbacks don't discourage me After I am interrupted or distracted, I can easily continue working where I left off. Seeing the big picture comes easily to me I enjoy motivating and encouraging others I am very cooperative when I work in groups. I can work with someone who has very different opinions than mine
I am able to work effectively towards longterm goals
I am able to control my anger really well.
I take criticism without getting angry
I review the consequences of my behaviour before I take action.
I’m good at understanding where people are coming from, and can put myself in their shoes. It upsets me to see someone being treated disrespectfully
I have a good understanding of what life is like for people who are different from me I quickly understand how my friends feel when they are angry, upset, or sad.
I am quick to spot when someone in a group is feeling awkward or uncomfortable.
When I talk to people, I tend to talk about their experiences rather than my own.
I try to be nice to other people. I care about their feelings.
I have tender, concerned feelings for people less fortunate than me.
15
Option 2 - illustrative Core
Communication
Creativity
10 – 13
14 - 16
17 - 20
I can choose to express myself passionately if I want to
I can express the same idea in different ways depending on the situation
I can simplify complex ideas so that people can understand them
I frequently have creative ideas
I am always using my imagination to help me think of new ways of doing things
I can think of times when I have built on other people’s ideas
I can break down a plan into clear steps and put them in order.
I can break down a plan into clear steps and put them in order.
I am good at planning a project or campaign.
I am able to see what materials are needed to complete a project.
I know how to set realistic goals.
I am good at keeping track of lots of things going on around me, even when I’m feeling stressed.
When things go wrong, I don’t let it get me down
Setbacks don’t discourage me
I have overcome setbacks to conquer an important challenge
I don’t give up easily
If I start something I usually finish it.
I finish whatever I begin
Confidence
Agency
As above (see option 1)
Planning and Problem solving
Resilience, grit
16
I often lead in projects
I enjoy motivating and encouraging others
I am good at organising myself and others
I know how to decide which ideas are the best ones.
Seeing the big picture comes easily to me
I understand the organisations and people that have influence in my local area.
I listen carefully to other group members what our team is making a decision
I can think of times when I have been a useful team member
When I am working with others I am aware of how my behaviour might affect them.
I understand my moods and feelings
I understand my moods and feelings
I am able to work effectively towards long-term goals
I try to be nice to other people. I care about their feelings.
I often volunteer to help others (parents, teachers, friends)
I feel motivated to take action on social and/or environmental issues.
Leadership
Relationships
Managing feelings, self control
Empathy
17
Recommendations 1. “Empathy” needs to be further explored Currently the outcomes framework lists 8 individual outcomes, plus empathy under ‘benefits to young people’. However it was found that there is considerable importance amongst large, national social action providers such as NCS, The Challenge and Fixers – for measuring other ‘pro-social behaviours’, and that empathy alone is an insufficient descriptor for encapsulating these character traits. The Jubilee Centre has helpfully defined a distinction between ‘civic’, ‘moral’ and ‘performance’ character virtues. This might be a useful reference. (see annex 1). 2. A flexible, holistic evaluation framework is essential Due to the wide spectrum of social action opportunities and the need for the outcomes framework to be relevant to young people on their journey between them, evaluation needs to be driven by context. Instead, the quality framework should provide sufficient methods to link a range of contexts into a broader theory of shared outcomes. The data collected by trials, alongside detailed evaluation, will be able to create this broader theory. 3. Triangulate self-assessment with other forms of assessment For example teacher assessment or adult assessment. It would also be useful for evaluators to visit the trials programmes. With the passport it will be possible to collect both self-assessment and evidence of youth social action. 4. Connect this tool with others to offer wider comparable data Young Foundation – A Framework for Outcomes for Young People. NPC / Inspiring Impact – JET Framework: The Journey to Employment. CBI – First Steps: A New Approach to Schools. ACT – Citizenship curriculum resources for NCS Generation Change – community benefit of youth social action. There is opportunity for much greater synergy with other frameworks being commissioned by the Cabinet Office. For example, the Journey into Employment (JET) framework being developed by New Philanthropy Capital has two outcomes regions which cross over with the outcomes for young people participating in social action (“Experience & Involvement”, “Emotional Capabilities”).
5. Involve funders in this process It is vital that funders recognise the efforts that programmes are undertaking to align with a common outcomes framework. To create a successful quality framework, aligning funders around this is key and the Cabinet Office trials with match funding can support this process. Gaining feedback from other funders throughout the process will help lay the foundations for a common set of metrics for youth social action.
18
6. The “community impact” part of the double benefit In addition the community benefit is the key metric that needs development, this is particularly vital for youth social action. Social impact should be at the heart of social action programmes and will key to driving up the quality standard of youth social action. 7. Phased introduction to ensure it is context specific Furthermore the trials are testing a range of products as well as these metrics. To help create a robust tool at the end of the two years a phased approach would be helpful. For example, Stage 1: simple base metric trialled, with different sets to test a wider pool of statements. Stage 2: data analysed to adapt metric for range of contexts. Stage 3: context-relevant metrics then aligned with education/ curriculum. Stage 4: develop these metrics as a reference for employers. Stage 5: integrate universal metrics into programmes. 8. Engage employers and education in the outcomes framework As the final phase of the trials test the outcomes framework, particularly with the older age group, it is key that education and employers engage with the outcomes so they can understand more about how we can create the shift in the recognition and status youth social action is given as part of their recruitment and assessment processes. The ambition must be that this framework must be flexible to range of stages and ages at which it will be used and that help ensure young people gain the recognition they deserve from school and employers for undertaking “practical action in the service of others”. These statements provide a useful start to develop the needed nuance and flexibility to develop a robust tool.
19
Appendix In July 2013, the Jubilee Centre for Character and Values published a framework on character education that has been sent to schools, politicians, and policy makers. Generation Change would suggest that this maps onto the double benefit model neatly:
Therefore, in the same way that ‘benefits to the individual’ is broken down into a framework of 8 individual traits, ‘emotional intelligence’ – currently included in this paper under the “empathy” skill, but which could be called “pro-social behaviour”, “moral character virtues” or “social motivations” – can likewise be further unpacked to clarify exactly what the outcomes framework intends to develop in young people.
20
Acknowledgements Written by: Fiona Murray David Reed
Management team: Adam O’Boyle, Student Hubs Sophie Livingstone, City Year Jon Yates, The Challenge Network
Core contributors: Duncan Baldwin, The Challenge Network Hannah Mitchell, vInspired Patrick Taylor, Envision Paul Adnitt, The NCS Trust Greg Stewart, The Scouts Association James Probert, City Year Alice Thornton, Student Hubs Ceryse Nickless, The Cabinet Office Rania Marandos, Step Up To Serve Farooq Sabri, The Behavioural Insights Team Matthew Van Poortvliet, The Education Endowment Foundation Nick Ockenden and Kim O’Donohue, The Institute for Volunteering Research Tessa Hibbert, The Young Foundation Tom McClaren Webb and Sara Maden, The BB Group
Produced in consultation with: Generation Change:
Ashoka UK, The BB Group, The British Youth Council, The Diana Award, Fixers, Free the Children, Future Foundations, NCVYS, UnLtd, Year Here,
21
The Wider Voluntary and Youth Sector:
NCVYS, UK Youth, London Youth, Woodcraft Folk Education:
Elizabeth Anderson, The Aldridge Foundation Richard John, Andy Thornton, The Citizenship Foundation Marguerite Heath, former Head Teacher, The Citizenship Foundation. Liz Moorse, The Association for Citizenship Teaching (ACT) 6 focus groups of school children aged either 11 - 13, or 14 – 17, led by young corps members at City Year London and City Year Birmingham. Academic / research based:
Dr Robin Pharaoh Eibhlin Niogain, New Philanthropy Capital Business:
Kate Van Der Plank, National Grid Rob Wall, Head of Employment and Skills, CBI Bethia McNeil, formerly the Young Foundation
22
Full References Organisations: The Association for Citizenship Teaching (ACT) – NCS and Citizenship Curriculum Materials written and developed by ACT in partnership with the NCS. Accessed November 2013: http://www.teachingcitizenship.org.uk/resource/social-actiontoolkit#sthash.evBb9ZHF.dpuf California Healthy Kids Survey – California Department of Education (Safe and Healthy Kids Program Office) and WestEd (Health and Human Development Department). New Philanthropy Capital / Inspiring Impact – “The JET Framework” and “Blueprint for shared metrics”. Accessed November 2013 [http://inspiringimpact.org/resources/resources-funders-commissioners-and-investors/] The Jubilee Centre – ‘A Framework for Character Education in Schools’ (2013) Accessed November 2013 [http://jubileecentre.ac.uk/userfiles/jubileecentre/pdf/other-centrepapers/Framework..pdf] The Young Foundation – Scoping a Quality Framework for Youth Social Action (2013). Accessed November 2013.[ http://youngfoundation.org/wp-
content/uploads/2013/06/Summary-slides-FINAL.pdf] VIA Me! Character Strengths Profile (VIA® Institute on Character (Peterson, Christopher, and Martin E. P. Seligman. Washington, DC: American Psychological. Accessed November 2013 [https://www.viame.org/www/] Literature: Baron-Cohen, S. & Wheelright, S. (2004) The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. Autism Research Centre, Department of Experimental Psychology, University of Cambridge Bosworth & Espelage (1995). “Empathy – Teen Conflict Survey”, Dahlberg LL, Toal SB, Swahn M, Behrens CB. Measuring Violence-Related Attitudes, Behaviors, and Influences Among Youths: A Compendium of Assessment Tools, 2nd ed., Atlanta, GA: Centers for Disease Control and Prevention, National Center for Injury Prevention and Control, 2005. Duckworth, A. L., and Quinn, P. D. (2009). ‘Development and validation of the Short Grit Scale (Grit-S).’ Journal of Personality Assessment, 91, 166-174. Gerdes, K., & Lietz, C. (2013). Measuring Empathy in the 21st Century: Development of an Empathy Index Rooted in Social Cognitive Neuroscience and Social Justice. Research on Social Work Practice July 201323: 447-457, first published on April 9, 2013
23
Goodman R (1997) "The Strengths and Difficulties Questionnaire: A Research Note." Journal of Child Psychology and Psychiatry, 38, 581-586. Gresham, F.M., & Elliott, S.N. (1990). Social Skills Rating System manual. Circle Pines, MN: AGS. Gutman, L & Schoon, I (2013) ‘The Impact of Non Cognitive Skills on Outcomes for Young People: Literature Review’ The Educational Endowment Foundation. Accessed November 2013. [http://educationendowmentfoundation.org.uk/library/the-impact-of-non-cognitive-skillson-outcomes-for-young-people-full-report] Marsh, H.W. (1990). Self Description Questionnaire (SDQ) I: A Theoretical and Empirical Basis for the Measurement of Multiple Dimensions of Preadolescent Self-Concept: A Test Manual and a Research Monograph. Sydney: University of Western Sydney, Macarthur. Moilanen, K. L. (2007). The Adolescent Self-Regulatory Inventory: The development and validation of a questionnaire of short-term and long-term self-regulation. Journal of Youth and Adolescence, 36, 835 – 848. Pearlin, L., & Schooler, C. (1978) ‘The structure of coping.’ Journal of Health and Social Behavior, 19, 2-21. Roets, L.F.(1997). Leadership: Skills training programs for ages 8–18 (8th ed.). Des Moines, IA: Leadership. Rosenberg, M. (1965) Society and the adolescent self-image. Princeton, NJ: Princeton University Press. Schwarzer, R., & Jerusalem, M. (1995). Generalized Self-Efficacy scale. In J. Weinman, S. Wright, & M. Johnston, Measures in health psychology: A user’s portfolio. Causal and control beliefs (pp. 35-37). Windsor, UK Spreng, R. N., McKinnon, M. C., Mar, R. A., & Levine, B. (2009). The toronto empathy questionnaire: Scale development and initial validation of a factor-analytic solution to multiple empathy measures. Journal of Personality Assessment, 91, 62-71 Tangney, J. P., R. F. Baumeister, and A. L. Boone (2004). High Self-Control Predicts Good Adjustment, Less Pathology, Better Grades, and Interpersonal Success. Journal of Personality, 72 (2), 271-324. Townsend, C. D., & Carter, R. I. (1983). The relationship of participation in FFA activities and leadership, citizenship and cooperation. Journal of the American Association of Teacher Educators in Agriculture, 24(1), 20-25.
24