PROGRAMME REPORT 2015–16
Charity No: 1167555
WELCOME It’s hard to believe how far Project Oracle has come. Just three years on from a pilot year to test a hunch we had about the need for our work, we have set sail as an independent organisation, with our charitable status recently approved by the Charity Commission. I want to acknowledge the extraordinary vision and commitment of The Social Innovation Partnership (TSIP) and London Metropolitan University, and particularly of the founding co-Directors, Professor Georgie Parry-Crooke and Stephen Bediako, who joined together to set up the organisation and guide it through to a sustainable future. I also want to thank our loyal funders—the Greater London Authority, the Mayor’s Office for Policing and Crime, and the Economic and Social Research Council—to whom we are indebted and who have continued to fund us for the next phase in our development. I am delighted to say that for the coming year they have been joined by the Paul Hamlyn Foundation. “ WE SEE PROJECT ORACLE AS AN IMPORTANT PART OF THE INFRASTRUCTURE TO HELP ORGANISATIONS GET THE MOST OUT OF EVALUATION, FOR THE BENEFIT OF THE CHILDREN AND YOUNG PEOPLE THEY WORK WITH.” Jane Steele, Director of Impact and Evaluation, Paul Hamlyn Foundation
A new board of trustees will help us to navigate our onwards journey. We will add to the current board members, Georgie Parry-Crooke, Stephen Bediako, Pauline Daniyan and Michael Kell, and the expanded Board will guide an ambitious strategy for the next five years. New developments will include replicating the model outside of London, and expanding the cohort work (Impact Pioneers) to additional subject areas including a Youth Justice Cohort commencing autumn 2016. In this programme report, we look back on the past year and also look forward to the future. It is a future that wouldn’t be possible without the support and willingness of the evidence community in which we sit—and for that we are especially thankful.
Sue Holloway, Chief Executive of Project Oracle
2
OUR STORY—IN BRIEF Project Oracle empowers the children and young people’s (CYP) sector to achieve the best possible outcomes for children and young people. We do this by supporting CYP organisations and funders to produce, use and share high-quality evidence, so that together we can make better decisions to improve how interventions are funded and delivered. Project Oracle was born to address a gap in understanding about which projects deliver positive change for children and young people in the capital. Pockets of disadvantage and violence persist across the UK and, while local budgets continue to shrink, expectations of CYP services seem only to increase. With the parallel movement towards outcome-oriented funding and a focus on “what works”, CYP organisations need to be able to give a reliable indication of their impact. We support CYP organisations to measure and track their outcomes, so they can understand what they are doing well and what could be improved. We simultaneously boost the ability of funders and commissioners to process and use this knowledge, so that public money can be spent on the most promising interventions. Academics and evaluation experts play a part in this by helping to ensure that evidence is of a high quality. We have always depended on three supporting structures for our work:
Managing and growing an online hub
Establishing and nurturing an evidence community
Developing and upholding Standards of Evidence
3
These come together to form an Evidence Hub: a dynamic community where each part contributes to—and gains from—the whole. Around these structures we have developed a range of core support options.
Training
We run full-day and half-day workshops that cover the major areas of evaluation: Theory of Change; evaluation planning, analysis and reporting; and cost benefit analysis. We also run breakfast seminars where topics come in smaller doses.
Research placements
We work with universities to identify masters and PHD students who want to explore the impact of their research in a real-world setting, and then match these researchers with organisations in need of their support.
Evidence Champions
We run a seminar series and peer network designed to embed evidence at every stage in the commissioning process. We teach funders and commissioners to map the needs and demand for a particular service, consider how to manage outcomes-based contracts and monitor service effectiveness.
One-to-one support for validation
Reaching validation against the Standards of Evidence is an iterative process that requires advice and troubleshooting along the way. We are always on hand to provide over the phone and in-person support.
Evidence Competition
Each year, Project Oracle and its supporters offer cash prizes to London-based children and youth programmes striving to better integrate evidence and evaluation into their work. Our aim is to showcase the efforts of those working hard to progress through our Standards of Evidence in order to design and deliver programmes based on evidence of “what works”.
4
THIS YEAR’S DEVELOPMENTS This year, as a newly independent organisation, we took the chance to think about what we really care about: how we want to support the sector; what kind of standard we want to hold ourselves to; and in what manner we want to carry this out. These commitments are now set out in our organisational values as follows:
Responsive
By listening to and understanding an organisation’s needs, we promise to provide sup[port that is both practical and flexible.
Supportive
We are approachable and empathetic, and seek to build long-term relationships that help organisations to achieve their aims.
Knowledgeable
Our team is truly multi-disciplinary, combining research and evaluation expertise, with knowledge and experience of the children and young people sector, to offer a balanced perspective.
Dynamic
We believe in and enjoy what we do, and we convert this into experiences and communications that are interactive and engaging.
Open
We prize reflection, learning and improvement, which requires us to be open and honest with those we work with and also with ourselves. We strive to honour these commitments in all that we do—which is why over the past year we have developed our work in ways that are designed to be responsive, supportive and, above all, targeted at improving the lives of children and young people.
5
IMPACT PIONEERS Our Impact Pioneers programme brings together CYP organisations with relevant funders, and offers them greater scope for change and influence. The concept is simple. We search for agents of change within a particular sub-set of the CYP sector. We then provide a backbone of support tailored to the needs of that group over nine to twelve months. Together we explore the most appropriate ways of performing evaluation, some of which may be new and innovative, and create processes that are engaging and easy to roll out. We then disseminate and communicate this learning to help boost the evidence fortunes of the wider youth sector. Our support is collaborative and seeks to harness a collective energy, to fill in the gaps in evaluation and build consensus around what “good” evidence looks like. Overall it offers an opportunity for organisations to reclaim the evidence agenda—to use evidence to more effectively tell the stories they want to tell, so that they can earn recognition for the valuable work they are doing.
Spotlight on Impact Pioneers: Arts Cohort
For our first cohort, we decided to support arts organisations working with children and young people, with the aim to: 1) prove that it is possible to measure the value of an arts intervention, without sacrificing integrity; 2) co-develop evaluation methods better suited to this endeavour; and 3) share these twin messages and strengthen the evidence base of the wider youth arts sector. As we come to the end of that year, the lessons to emerge from this experience are shared in our learning report, Impact Pioneers: Lessons in arts evaluation, available to download from the Project Oracle website. Key themes include the need for an ongoing avenue or forum for shared learning and reflective practice in the arts; the importance of keeping funders involved every step of the way; and the need for evaluation tools to be developed from the bottom up. “ THE ARTS COHORT HAS EFFECTIVELY HIGHLIGHTED EXISTING GOOD EVALUATION PRACTICE IN THE SECTOR; THE SYSTEMATIC USE OF OBSERVATIONS LOOKS LIKE AN ESPECIALLY PROMISING DEVELOPMENT.” Holly Donagh, Partnerships Director, A New Direction.
As individual organisations, cohort members are better equipped to carry out evaluation and to approach funders with a method that works for them. And, as a group, they are in a better position to share with the wider sector a genuine perspective on what works well in arts evaluation.
6
BESPOKE SUPPORT Over the years we have designed a flexible package of support in response to a wide range of needs in the sector. Every organisation is unique and we tailor our work to fit or, in some cases, create something new. Our bespoke support offer enables us to work intensively with one organisation addressing their issues systematically and driving forward their evidence journey. In the past Project Oracle has designed and delivered bespoke programmes of support for larger funding organisations, including the Greater London Authority and the Mayor’s Office for Policing and Crime.
Spotlight on Islington Borough Council
This year we were approached by Islington Council to help design a programme of support to up-skill and validate a number of organisations operating in the borough. There was a wide range of experience amongst participants, some were new to evidence and a few were even sceptical of its values. We supported a total of eight organisations to create a variety of Theories of Change and evaluation plans. These ranged from simple pre-post evaluations using tools such as the ‘life effectiveness questionnaire’ to more sophisticated, tech-heavy systems designed for iPads and point of sale technology. Some organisations planned Participatory Action Research projects, an advanced approach that aims to evaluate projects through collaborative action and reflection with a strong emphasis on context. This differs from traditional notions of research where evaluators are expected to act as neutral, impartial observers, testament to the forward thinking nature of the group. Overall, the workshops were a success, building on the progress Islington Council has made in the past few years. The flexibility of our bespoke offer allowed participants to think outside the box and experience evidence in a radical new way. Now all of the organisations are set to achieve Standard 1 against the Project Oracle Standards of Evidence, and some are already looking to Standard 2. “ PROJECT ORACLE WORKED FLEXIBLY WITH A RANGE OF PLAY AND YOUTH PROVIDERS IN THE BOROUGH, SUPPORTING THE SECTOR TO BETTER UNDERSTAND, DESCRIBE AND EXPLAIN THE DIFFERENCE THAT THEY MAKE FOR AND WITH CHILDREN AND YOUNG PEOPLE.” Christine Lehman, Commissioning Manager, Islington Council.
The result here is encouraging; although we still encounter uncertainty in the sector, it is often just a question of providing the right kind of support.
7
THOUGHT LEADERSHIP With each year, Project Oracle has been pushing the envelope on the demand for, the generation of and, crucially, the use of evidence. Our aim is to advance understanding of what works throughout the sector to improve the lives of children and young people in London and beyond. This involves testing the boundaries of what has been done before and opening up the conversation to new and innovative ideas. Through this work we help to identify and direct funders’ resources to where they can best make a difference—leading the conversation from innovation to scaled and evidenced programmes. We encourage CYP organisations to view evidence as a part of their culture—that it’s about them and for them—and funders to embed evidence in to their funding decisions ensuring the appropriate and fair distribution of resources based on a high standard of evidence. Project Oracle continues to support and connect these groups facilitating those discussions and practices at our events.
Spotlight on our Annual Conference
This year Project Oracle held its third annual event ‘Breaking barriers: Where evidence goes next’. The conference brought together London’s CYP organisations, funders and researchers for half a day of learning and insight. The aim was to progress thinking about where evidence is heading in the years ahead - to challenge accepted ways of doing things and anticipate new developments in the field - because we believe that together we can contribute to a vision for London to host some of the best youth services in the world. The line-up for the conference was in itself a marker of the relevance of Project Oracle and the need to keep the discussion about evidence going. With speakers from the Metropolitan Police, the Economic and Social Research Council, Turning Point, London Funders, London Youth, Partnership for Young London and the Centre for Youth Impact among many others. We are delighted that so many are committed to continuing the conversation that was started just a few years ago and proud to remain an important voice in that dialogue. “ PROJECT ORACLE WAS CONCEIVED AS A WAY TO CREATE A MORE EFFECTIVE YOUTH SECTOR IN LONDON BY LEADING THE EVIDENCE AGENDA” Munira Mirza, former Deputy Mayor for Education, Greater London Authority.
A highlight of the day was the announcement of Sue Holloway to lead the organisation through the next phase, signalling the continuation of Project Oracle’s leading work - breaking barriers in pursuit of better evidence - and a bright future ahead.
8
OUR IMPACT TO DATE Project Oracle in numbers and words
So far, we have worked with 200 organisations to help them achieve validation of over 220 different projects. We have provided approximately 900 training opportunities, placed over 50 researchers in children and youth organisations, published 6 Synthesis Studies and 3 Evidence Digests and developed the innovative Impact Pioneers Programme.
800+
200+
Training places on evaluation approaches and learning from evidence have been taken up by children and youth organisations in London.
Projects across London have been validated against the Standards of Evidence and we continue to work with more organisations to improve the evidence of their outcomes.
50+ Research Placement students have been provided to youth organisations, supporting them to build their evidence base.
35+ Commissioners of youth projects from local authorities and funding bodies across London have been trained as Evidence Champions.
“ TO DATE WE HAVE FOCUSSED ON TRAINING, VALIDATIONS AND RESEARCH PLACEMENTS AS SEPARATE ACTIVITIES. INCREASINGLY, WE VIEW THEM, TOGETHER WITH THE 1:1 SUPPORT OUR TEAM MEMBERS PROVIDE, AS A PACKAGE OF ACTIVITIES WHICH SUPPORT THE USER JOURNEY TO BETTER EVIDENCE.” Sue Holloway, Chief Executive, Project Oracle
9
Tavistock reports on our impact
In March 2015 Project Oracle commissioned the Tavistock Institute to carry out a full evaluation over the course of a year, with specific focus on our new Impact Pioneers Model. The final report revealed high levels of satisfaction with our work. Participants completing the follow-up survey commented positively on the content and quality of the cohort events and workshops, and expressed particular appreciation for the one-to-one evaluation support. Results indicated that that those surveyed found the bespoke one-to-one sessions particularly effective in troubleshooting specific issues or challenges within an organisation. The report also found evidence of numerous positive impacts in CYP organisations engaging with Project Oracle, ranging from increased confidence in individual staff members to the development of more effective and appropriate evaluation tools and more rigorous evaluation plans. The evaluation also found that CYP organisations were more likely to critically reflect on and improve their delivery of services to young people. The final report is now available in full on the Project Oracle website.
10
NEXT YEAR’S FOCUS In the past few years Project Oracle has gone from strength to strength, not only growing the quantity and quality of evidence in the CYP sector but also developing as an organisation. We have learnt valuable lessons from working in London for the last three years, and our partnership with the Greater London Authority, The Mayor’s Office for Policing and Crime and The Economic and Social Research Council has been key to the success we have enjoyed. “ AS PROJECT ORACLE SETS SAIL AS AN INDEPENDENT CHARITY, WE ARE INDEBTED TO EVERYONE INVOLVED IN GETTING US TO THIS STAGE. THIS INCLUDES OUR LOYAL FUNDERS—THE GREATER LONDON AUTHORITY, THE MAYOR’S OFFICE FOR POLICING AND CRIME, AND THE ECONOMIC AND SOCIAL RESEARCH COUNCIL—ALL OF WHOM HAVE AWARDED US FUNDING FOR THE NEXT PHASE IN OUR DEVELOPMENT.” Sue Holloway, Chief Executive, Project Oracle
We hope, that with your support, we can continue growing. Over the next five years we aim to become a national charity collaborating with others to support the CYP organisation journey towards better evidence in order to improve outcomes for children and young people. To achieve this we will: • Encourage funders and commissioners to increase the demand for high quality evaluation to underpin decisionmaking through our network of Evidence Champions and training offer to them and their colleagues • Enable providers to generate this evidence and progress in their evidence journey by offering training and 1:1 support from our team and by brokering research placements • Engage with academics and experts to maintain a rigorous validation process against our standards of evidence, develop new but robust evaluation approaches which can be implemented by a range of providers, and to offer practical support to providers through our research placements • There is more to do in the capital and we will continue to identify opportunities to work with funders, commissioners and CYP organisations in the London boroughs. At the same time we will begin to explore and develop models for working outside London. We will expand to other parts of the country incrementally, as we identify opportunities for partnership and funding. “ I AM VERY PLEASED TO BE JOINING THE NEW BOARD OF TRUSTEES AT PROJECT ORACLE, TO HELP GUIDE THE CHARITY THROUGH THIS EXCITING PHASE IN ITS EVOLUTION, AS IT WORKS TO SUPPORT IMPROVED OUTCOMES FOR YOUNG PEOPLE THROUGH A BETTER UNDERSTANDING OF EVIDENCE AND EVALUATION IN THE SECTOR.”
Pauline Daniyon, Trustee, Project Oracle
At this crucial stage in our history we hope that our ambitious plan encourages renewed support from friends old and new, allowing us to achieve the best possible outcomes for children and young people in London and across the UK. 11
GET IN TOUCH www.project-oracle.com @Project_Oracle info@projectoracle.com
FUNDED BY
12