L2K Survey Review Carried out by and report produced by the members of Dartford and Gravesham Young Evaluators Project 2012-2013
June 2013
Contents Foreword Background and brief Aims and objectives Challenges and approaches Our findings Survey Recommendations Post review survey Gallery
3 4 4 4 5 6-8 8 9 - 10 11
Foreword Dartford and Gravesham Local Children’s Trust Boards commissioned the current Young Evaluators programme in November 2012. We are a group of 15 young people representing both areas. Our group consists of a mix of those that are less likely to engage or have life challenges which may act as a barrier to their accessing training, as well as those that are already involved in traditional forms of participation. We undertook an intensive training programme that focused on the principles and practices of peer research during a residential training and teambuilding weekend at Arethusa, Upnor. The training was designed around the needs of the group, where we took a lead role in shaping the programme with regard to delivery and approach. This is the first piece of commissioned work for the group. Our training has given us the ability to cover areas such as mystery shopping, peer research, priority planning and influencing boroughwide service specifications. Although this type of project work does not formulate part of the expected range of work we are likely to be commissioned to undertake, the group felt strongly about supporting the Fire Brigade with this piece of work that was built into our training. We know that through having young people involved in designing any aspect of designing, delivering and evaluating services is the best way to make sure that they are going to be “fit for purpose�; and so pushed to take on the project, even though this was a more remote piece of work. This report is a summary of our findings from the L2K survey question set that those attending a L2K event can expect to be asked to complete, to gather feedback on impact and improvement and recommendations, where appropriate. We would like to take this opportunity to thank you for letting us be involved with this project The Young Evaluators Project
3
Background & Brief L2K is a film portraying a group of young people who are going for a day out to the seaside who have an accident, young people are hurt and one killed and a range of personnel from the emergency services are involved. Between 6000 and 7000 6th form students from Kent and Medway schools, see the film/attend the event and are asked to complete an online survey (survey monkey) that usually generates around a 10% response (from 2012 at the time of writing there are 515 responses, approx 9%).
Aims & Objectives The Fire Brigade are unsure as to whether they are asking the right questions in the survey, if the language is right and if the survey is laid out in the right way, so that young people are able to understand and engage with it and subsequently happy and able to complete it. The Fire Brigade are keen to undertake some qualitative evaluation preferably a piece of peer research but appreciate that there is insufficient time for this to take place and that the logistics of arranging this would prove too difficult, at this time but could be considered for the future This bespoke piece of work will be used to inform the reformulation of the L2K post event evaluation carried out via Survey Monkey by young people attending the event.
Challenges & Approaches Our biggest challenge with this piece of work is that none of the group has been given the opportunity of viewing the film or attending the event so from that point it was difficult. However, we used the brief and background information for the context and so were for the most part able to overcome this. Key challenges: - not having viewed the film or attended the event Approaches: - 14 of the group discussed the main body of the survey, the layout, tone and language and then a sub group of 5 young people took part in looking at the detail. - At a following meeting the findings and recommendations were put before the main Young Evaluators Group, amendments made and sign off agreed - To fulfill the brief we looked at four different areas: 1. 2. 3. 4.
Language Tone Layout Content
4
Our Findings Context We felt that it was difficult to understand what the aim of the survey was and felt that young people would want to know why they were being asked the questions and what the information was going to be used for in the future. This is very important for all consultations and will encourage young people to take part or complete any kind of survey, as they will feel it is worthwhile and will feel valued. We felt that if there was a simple opening paragraph that would set this out at the beginning of the survey as well as some basic details around what took place, as this would remind people of the event and the film and act as a memory jogger. Language and tone When we looked at the language used through the survey we found it to be confusing at times and often over complicated. Long sentences, asking too much in too many or the same thing, different ways and difficult words were used when shorter sentences and more simple words would have helped with the flow and understanding of the questions and the survey. There were several questions that asked the same thing and again gave us the feeling that the purpose of what the information was being used for was and why it was being asked, was not clear at the outset of designing the survey , so the questions were randomly put together. This didn’t fill us with confidence and would not, we felt, have encouraged us to continue to go through all the questions and complete the survey, had we been a young person asked to do so. Layout and content After going through the whole survey we found ourselves having to change the order of the questions as they did not flow and we found it confusing the way the survey was laid out. If the survey questions are not set out in a logical order that takes you through the film/show and jumps around in this and the subject area it is referring to then this definitely acts as a barrier to understanding and completing the survey, it is frustrating and also makes you think that it has not been thought through. We believe that the survey needs to be set out in two parts, one dealing with the performance and the other around key messages. We didn’t feel that the survey was particularly too long but were not sure how and when young people would be asked to complete the form. If this was something that was done some time after the event then our comments under “context” would be even more relevant. If it was done immediately afterwards, in a school lesson and with others that took part, with a teacher around for support, then this may help and not prove too long or too difficult. If students are asked to do this some time after the event in their own time then we feel that the survey in its current format is too long and too complicate and will not promote or encourage young people to complete it.
5
The Survey We have gone through each of the questions and made individual comments on each but what needs to be taken into account is that as a group we worked on this from a print out and not taken part in the live survey so unsure as to how it may or may not appear on the screen and how you can interact within it. Neither have we seen the film and feel that both would have helped us more, had we been able to do that. We think that we would be able to provide the Fire Brigade with better feedback if we are able to sit down after seeing the film and work with them to redesign the survey from scratch.
Survey Monkey Questions: 1. 2. 3.
Name of school - Good Gender – Good Age – Good
4.
Which of these factors do you think might have an effect on a persons ability to drive safely? · Exceeding the speed limit · Using a mobile phone whilst driving · Driving under the influence of alcohol · Driving under the influence of drugs · Listening to loud music whilst driving · Changing radio channel or CD/Tape whilst driving · Eating and or drinking · Bad weather conditions e.g. heavy rain, snow, darkness. This question is too complicated and uses words that are too long and unnecessary. We felt that darkness was not a weather condition and would be covered in answers given at original question 5 1. What other factors can you think of which are likely to affect a person’s ability to drive safely? This needs simplifying 6.
Please tick the extent to which you agree or disagree with the following statements · Peer pressure can sometimes lead young peplum into driving more recklessly · I think I’m a safe person anyway, this type of accident is never going to happen to me · Going to this event has made me more aware of the responsibilities and possible consequences of driving a car · Lessons on road safety are important for all road users including pedestrians · The ability to drive a car at speed proves I am a good driver
The question doesn’t have any measure levels attached to it when it asks for the young person to tick the extent to which they agree or disagree with the statements and so these would need to be added. This question, would be difficult for someone who doesn’t drive, to answer. It should be an optional separate question in itself before the rest of the question is asked. The first bullet point asks if peer pressure could make a young person drive more recklessly and this implies that they already drive recklessly and so needs to change! 6 The bullet relating to responsibilities and consequences should be taken out and added to current question 8
7.
What do you feel the main message of the L2K event was for you? · Driving is dangerous · Speeding costs lives · Seat belts must be worn · Driver has the responsibility for the passengers · Concentration is essential when driving
This question should go almost at the end of the survey and should give people the chance to state the order of importance of the messages as well as provide an “other” option as this will give the Fire Brigade a better insight into what young people are really getting out of the film not just what they think they might be getting. 8.
What do you think about the performance you attended? (mark all those you feel apply) · The event was of no benefit to me · The event was informative · The information provided was easy to understand · The event has had an influence on what I now consider to be safe driving · The event will influence the way in which I will drive in the future · The event will help me become a safer driver
This should be moved 9.
You have said the performance was of no benefit to you. Why have you said that?
This question should move and be reformatted as it presumes that the young person has said that the event was of no benefit to them. 10. Many tell us that the performance was powerful; did you think that and if you did why was this so? This is a biased and leading question as it is telling us what others have said so makes us think that we should think that too, otherwise we are a different from others, so is divisive. It is inappropriate in its current format. 11. How do you feel about the length of the performance? Was it: · Too short · About right · Too long This question needs to be moved 12. Do you feel that all drivers would benefit from seeing this event and not just new or potential ones? · Yes · No If you have no driving experience it would be difficult to comment on this statement and so it must be appreciated that it is only opinion being sought and should include the opportunity to state why. 7
13. Do you think there is anything that could be done to improve future events? · Yes · No This should be the last question and should be joined with original question 14 14. As you said yes, what do you feel could be done to improve it? This needs to be joined with question 13 and come at the end of the survey 15. What is your current status with regards driving? · I have not applied for a driving licence yet · I have a provisional driving licence (not currently taking lessons) · I have a provisional driving licence (taking driving lessons) · I have passed my test and hold a full driving licence. This needs to be moved to the beginning of the survey
Recommendations As previously mentioned this type of project does not form part of the expected range of work we are usually commissioned to undertake, however we all enjoyed working on a different challenge. It gave us new perspectives and experiences from which we could learn and use in our future work. We did feel that we would have been better placed to review the survey had we been able to go through the survey in a live situation and most definitely had we had the benefit of seeing the film and the performance this would have helped us also. Language 1. Use simple and less complicated language 2. Use shorter sentences and simple word, not everyone starts from the same place and understands all of the words used in the survey so this needs to be taken into account Context • A simple opening paragraph at the beginning of the survey as well as some basic details around what took place, as this would remind people of the event and the film and act as a memory jogger, especially if the survey was being carried out some time after the event in their own time and not supported at school Layout • So that the survey flows and is easy to complete the survey needs to be split into three sections and made clear what each of these are dealing with, we suggest: o About you and what you think o The event o Main messages and lessons learnt Timing • To maximise the return on the survey that students should be asked to complete in an IT lesson or another similar lesson so it becomes compulsory to complete if students attend – work in partnership with the school to promote this. • If the survey is kept short – to around 13 questions as we have suggested, with simpler answering options then this would not be too difficult for the schools to undertake and a teacher would be there for support.
On the page opposite we have tried to give you some idea of how we feel the survey should look and flow
8
Post Review Survey Section 1 – About you and what you think 1. Name of school 2. Gender 3. Age 4. What is your current status with regards driving I have not started driving yet I hold a provisional licence and am taking lessons I hold a provisional licence and am not taking lessons I hold a full driving licence 5. If you drive do you consider yourself a safe driver? Yes No. 6. Which of these situations do you think might affect someone being able to drive safely? Tick as many as you think. Going over the speed limit Driving after drinking alcohol Listening to loud music Smoking Very bad weather Using a mobile phone Driving after using drugs Changing the music/radio Eating and or drinking Other 7. Please tick whether you agree, disagree or are not sure how you feel about the following statements: Statement I am a safe driver, this type of accident will never happen to me Being able to drive fast proves you are a good driver Road safety lessons are important for drivers and pedestrians Pressure from friends can sometimes lead young people to drive recklessly.
Agree
Disagree
Not sure
Section 2 – The event 8.
Was the event: Too short Too long About right
9
9. Please tell us what you think about the event and tick all those you feel apply I didn’t find the event useful I found the event useful The information was easy to understand I have now changed my view on what I think it is to drive safely The event will change the way I will drive in future The event will help me become a safer driver I feel more aware of my responsibilities as a driver I feel more aware of the consequences of not driving safely
10. If you said you didn’t find the event useful, please tell us why, otherwise go to question 11
Section 3 – main messages and lessons learnt 11 . What do you feel the main message of the L2K event was? Please put the following statements in order of importance. Driving is dangerous Speeding costs lives Everyone in the car must always wear seat belts Drivers are responsible for everyone in the car It is very important to fully concentrate when driving Other: 12. Do you think it would it be useful for all drivers to see the film and talks? Please tell us your reasons for your answer Yes No Why
13. Do you think we could make the event better and if so how. Yes No How
10
11
The Dartford and Gravesham Young Evaluators Project 2012-13 Project coordinator Juli Dosad of Banter Limited Email: juli@banter.uk.net Web: www.banter.uk.net This report was written by the Young Evaluators and formatted by Jasmine Buckley