Rapporto di valutazione POVEL def_lorena_EN

Code JUST/2009-2010/DAP3/AG/1349

Project POVEL Daphne programme

EVALUATION REPORT Edited by

This presentation has been produced with the financial support of the Daphne III Programme of the European Commission.

Contents are the sole responsibility of Scuola Centrale Formazione and its partners and can in no way be taken to reflect the views of the European Commission.

The evaluation of the POVEL project was ensured by the survey activities established by the evaluation plan proposed by SCF to the partnership at the beginning of the project and re-adapted in compliance with the needs which emerged at that time (the survey instruments established by the evaluation plan are attached, except for the monitoring form which is attached to the monitoring report).

The evaluation plan provided for a monitoring action supported by a survey card on the progress of the activities and the problems which emerged in the course of the project and by a satisfaction survey to measure the level of appreciation and effectiveness of transnational meetings. This survey activity allowed us to control the progress of the project and the quality of the products made, especially during the early project phases (research and kit planning phase), the intermediate monitoring report allows us to receive feedback on this part of the evaluation activities.

The second type of evaluation activities focuses mostly on quality and refers to the kit testing phase. As mentioned, the first two phases: research and kit preparation, in

consideration

of

their

characteristics,

were

kept

under

control

through

monitoring. Research indeed was carried out with the aid of a qualified and recognized scientific supervision which ensured the correct preparation of the questionnaire, of data collection and processing. This questionnaire would have made an additional quality evaluation redundant, that’s why only a survey activity was put in place. As concerns the planning phase of instruments and kits, a quantitative monitoring was deemed fit for the assessment of the activities performed since the quality of the products made was evaluated over the course of testing.

For the evaluation of the testing phase three instruments were used: a structured questionnaire to measure the perception of the kits by the students involved (annex 3), a semi-structured journal edited by the coordinators and tutors of the of the activities (annex 1), a semi-structured interview made in the presence of the project partners (annex 2). Some accounts of involved students were also taken into consideration. These instruments allowed us to collect the point of view of the stakeholders involved in the activities: students, trainers and tutors who directly managed classroom testing, coordinating the partnership activities.

The testing of the four teaching kits (Media violence, Peer violence and bullying, violence related to drug assumption, violence related to alcohol abuse) is briefly described in the following chart.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

2

Country

Organization involved in testing

Own kit tested

Target

Media Violence

Italy

CEFAL, CIVIFORM, FOSF Period: March – July 2012

65 (44 questionnaires filled in and returned) + 85 (peer education conference in CIVIFORM)

Peer violence and bullying Spain

French

TRINIJOVE

Violence related to alcohol abuse; Peer violence and bullying; Violence related to drug assumption Period: October 2012 -8 February 2013

40

40

Period: April may 2012

Period: April - may 2012

Violence related to drugs assumption

Violence related to alcohol abuse

ID Formation

34

41

Period: October November 2012

Violence related to alcohol abuse FISSAAJ

Violence related to drug assumption 52 students

Period: October -December 2012

Target

Media Violence

Period: October - November 2012

Belgium

Other kit tested

50 students Period: October 2012 - January 2013

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

3

The point of view of coordinators, trainers and tutors By analysing the outcomes of the semi-structured interviews made to the coordinators of involved partners and the journals written by trainers and tutors what follows emerges.

As mentioned during the monitoring phase, after each partner tested a kit it had not prepared, a phase of adaptation to the national context and the specific needs of final beneficiaries followed, which was especially necessary for the “violence related to drug assumption” kit that had been designed for an older target of beneficiaries than the others. Not all partners chose to carry out all the activities contained in the original kits, as expected, each organization followed and adapted the proposals which would better suit their local needs. In all the accounts gathered, this adaptation phase is deemed as essential for the success of the activities. An adaptation activity was required in some cases over the course of testing: for example while testing the “violence related to alcohol abuse” kit by Civiform problems were noted concerning the themes dealt with which were connected to the personal and family experiences of the students, as a consequence we chose to re-organize some of the set contents to make room for the unexpected issues which emerged. All the contents adaptation activities carried out show that the project request of preparing flexible and adaptable instruments was met.

All the kits provided for informative activities consisting of presentations and interactive activities (video-clips, games, theme discussions...) which would boost the active role of the students in facing the themes under review.

The activities in almost all cases were managed by internal trainers and tutors of the organizations involved in testing. In two cases the management methods partly deviated from this model: The Belgian partner involved independent experts to manage all the set activities, because of the mistrust shown by the school trainers to face alcohol- and drug-related themes, which made it impossible to directly involve them in the activities. In this regard, please note that it would be necessary to work on raising the awareness of these figures on the importance to deal with similar themes on a regular basis in the teaching curriculum. Another organization which, for different reasons, decided to involve experts is Civiform which decided to foster a peer education-based education programme: the themes of the tested kits were proposed and managed by some peer-education experts who worked with small groups of students (12 per kit) who then had the chance, during an ad-hoc event,

to

carry

out

an

information/education

activity

of

their

CFP

peers

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

4

(approximately one hundred). These peer education-trained students were also appointed as contact people for their classmates who would require “advice” on the issues dealt with during the year. In the case of Civiform CFP teachers were still involved and at a later time of training they also directly participated.

Most of the managers of the activities think that creating a motivated and active group is very demanding. To favour the interest of the students the strategies included: -

Favouring the more theoretical phases of presentation in the morning hours and leave the more interactive activities for the afternoon;

-

Involving the students in the planning phase of the activities to be carried out;

-

Involving experts from the reference sector to skilfully and punctually deal with problems concerning the personal experiences of the students;

-

Referring the themes which appear to be more theoretical to the personal experiences of the students favouring the surfacing of their experiences and actual examples.

In all the cases the stress is on the fact that after an initial mistrust of the students (and in some cases of the teachers involved) on the proposed activities a clear increase in interest and participation of the group is shown that is also witnessed by the outcomes of the questionnaires to the final beneficiaries at the end of testing.

From the point of view of the trainers involved some significant elements emerged, in some cases defined as unexpected, in relation to the themes dealt with, that is worth mentioning: •

many of the involved students struggle to catalogue the use of alcohol as an abuse while they are more familiar talking about drug abuse;

•

there is still much confusion about the legislation on abuse despite the fact that some themes are now usually dealt with by media, this shows the ineffectiveness of many of the conveyed messages, on the contrary after completing the proposed kits the students prove to have appreciably increased their awareness.

•

The theme of alcohol abuse in many cases has led to the appearance of problematic personal and family experiences, an element which has us think about the need not to underestimate the importance of regularly addressing this theme in school curricula.

•

As regards the students involved in testing the alcohol and drug abuse kits those involved in the testing of bullying and media violence kits appear to have less prior knowledge and direct experience on the issue.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

5

The point of view of students At the end of testing the students who took part in the testing were asked to answer anonymously a questionnaire which would record their interest in the activities carried out and their perception on the effectiveness of proposed instruments.

Below, there is a series of charts which outline what came out of the answers. For each question there is a corresponding chart which contains the answers expressed as a percentage of all students (notwithstanding the kit they received) and one where it is possible to distinguish the answer percentage in relation to the specific programme (Media violence, Peer violence and bullying, violence related to drug assumption, violence related to alcohol abuse). The percentage data refers to a sample of over 400 students who answered the questionnaires given by the various partners involved.

The first question asked aimed at checking how involved students perceived the usefulness of the programmes in relation to their knowledge.

1)Think about Kit topics:

13% 31%

I didn’t learn anything new I learned some new things I learned many new things 56%

It appears that a minority of students declares that they did not learn anything new, 87% of the total on the contrary declare that they learned some or many new things.

By analysing the distribution of these percentages by kit topic, it appears that the themes on which young beneficiaries declared that they learned more new things are media and peer violence and bullying. In any case it is necessary to consider that those kits were tested on the lowest number of beneficiaries each (93 for the bullying kit, 84 for the media kit).

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

6

1) 100%

16

20

80% 60%

49

53

66

% I learned many new things

57

40% 20% 18 0%

46

45 22

% I learned some new things % I didn’t learn anything new

5

2

Violence Violence Peer Media related to related to violence violence alcohol drug and bullying abuse assumption

The second question aimed at checking the ability of the proposed activities to affect the opinions of involved students.

2) Your opinions about the kit topic:

20% They are changed

35%

They are partially changed They are the same 45%

Almost 2/3 of interviewed students declare that they partly or fully changed their opinions on the themes dealt with. 35% declare that they did not change their opinion. By analysing the comments on this question and the accounts of the students the outcome of the answers is clearer: a large part of the students declares that they already know the danger of some behaviours shown in the programmes, it then appears that most of the students who did not change their

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

7

opinion are those who were already quite aware of the themes. This especially applies to the drug abuse kit, a theme in relation to which the students already appear to be quite aware and informed.

It is still worth noting that from the comments it emerges that a minority of the students who were not knowledgeable and aware of the themes, did not change their opinions (in these remaining cases the kits would appear not to affect the opinion).

2) 100%

15

90% 80%

47

70%

12

54 47

60%

55

They are the same

50% They are partially changed

40% 30%

44

35

20% 10% 0%

They are changed 30

9

41

11

violence violence Peer Media related to related to violence violence alcohol drugs and bullying abuse assumption

The third question asked aimed at checking the interest of the involved students in the teaching methods used.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

8

3) During presentations and discussions:

I w asn’t interested 12%

24% 64%

The topics were interesting but they were addressed in a boring way I w as interested

It appears clear that the large majority of students shows interest in the programmes proposed. However especially for the programmes concerning drug and alcohol abuse the portion of students (relatively 37% and 36%) who shows interest in the themes but does not appreciate the methods adopted, deemed boring (the comments claim that they do not appreciate that too much room is given to presentations, while active teaching methods are more appreciated) is significant. This type of remarks is much less important with reference to the two kits on bullying and media violence, programmes in relation to which the students also showed to know less of.

3) 100% 90% 80% 70%

54

I was interested

45

60%

86

77 The topics were interesting but they were addressed in a boring way

50% 40% 30%

37 36

20% 10% 0%

10 violence related to alcohol abuse

18

5 8

violence Peer related to violence drugs and bullying assumption

10

I wasn’t interested

13 Media violence

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

9

This fourth question, too, aimed at checking the appreciation of the different teaching methods.

4) I was more interested in:

18%

24%

Presentations Discussions Anything else

58%

As already mentioned in the analysis of the outcomes of the answers to the third question, active teaching methods are universally more appreciated for all the programmes.

4) 100% 90%

12

24

80%

35

32

70% 60%

66

50%

60 53

40%

49

10% 0%

Discussions Presentations

30% 20%

Anything else

22

16

13

violence violence Peer related to related to violence alcohol drug and bullying abuse assumption

19 Media violence

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

10

Answers 5 and 6 were asked as a close examination of the first question (Thinking about the topics do you believe you learned some things, many things or nothing?) aiming at understanding how much the information conveyed appeared to be interesting and how much they are actually rooted in the experience of the students.

5) Did you talk about kit’s topics with friends?

49% 51%

YES NO

Approximately half the students (49%) declare that they talked of their own accord with some friends about the themes. It is worth examining this question, since the results of the two pairs of kits are quite different: indeed, for the media violence kit, the percentage of young people who talked to their friends about the topics is 72%, hence broadly above half. For the bullying kit, too, the value exceeds half the youth who tested the kit (60%). This data shows an actual interest in the themes and the perception that the information received can be useful to significant people amid the group of acquaintances. Please note that the sample of youth who has tested the media kit and the bullying kit are less numerous. By analysing the reasons of those who answered “No” many state that there was no chance to talk about it, but that if the chance came up they would. A minority of statements that can be summarized in the ‘my friends would not be interested/I don’t think talking about it is useful’ is also included.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

11

5) 100% 90% 40

80% 70%

66

28

58

60% 50% 40% 60

30% 20%

34

72

NO YES

42

10% 0%

violence violence Peer violence related to related to drug and bullying alcohol abuse assumption

Media violence

6) If your friend was in one of the situations have been described during Kit:

7% 28%

I w ould like to give him/her some advice, but I think I’d not be able to do it I think I w ould be able to give him/her some information I'd not say anything

65%

The effectiveness of the message conveyed by the kits is also confirmed by the answers to the following question: most students show a proactive behaviour, declaring that they feel able to give advice to a peer who would find him/herself in alcohol/drug-related situations. The picture is more or less the same in relation to all kits, a slight difference can be noticed in the drug assumption kit for which more students would not be able to give advice.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

12

6)

100%

6

7

4

5

80% 60%

I'd not say anything 62

77

62

79 I think I would be able to give him/her some information

40% 20%

31

17 0%

22

I would like to give him/her some advice, but I think I’d not be able to do it

16

violence violence Peer Media related to related to violence violence alcohol drug and bullying abuse assumption

This last question was asked to confirm the interest in the programmes performed.

7) Would you like to participate in other training moments like this?

32% YES NO 68%

On average, the majority of students declare that they are inclined to repeat the experience, however 32% say the opposite. The comments to this question show that the making of the question has probably been misunderstood: many students who declare that they would not repeat the experience justify it by saying that the programme was comprehensive and it would not be useful to repeat it, however it would be interesting to propose a similar programme on different topics. These comments show that the figures are misrepresenting the situation.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

13

It is however significant that a much higher percentage of students would like to attend again the programmes on bullying and media violence, themes which the students appear to be less familiar with, also according to the other survey instruments used.

7)

100%

12

16

88

84

90% 80%

43

47

70% 60% 50% 40% 30%

57

NO YES

53

20% 10% 0%

violence related to violence related to Peer violence and alcohol abuse drug assumption bullying

Media violence

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

14

Conclusions To conclude, it is worth summarizing the positive and negative aspects which emerge from the point of view of all the parties involved (students, coordinators, trainers and tutors) in the project testing phase, aspects which should be kept in consideration for future activities which could be proposed again in the future.

Positive aspects: •

Appeal and dynamism of the instruments used with students (in particular those concerning active teaching techniques).

•

Positive response of students: motivated and involved, many students asked to take part in the proposed activities.

•

Greater awareness of the students on the themes.

•

Repeatability of the kits in other contexts (flexibility of proposals).

•

Concreteness and usability of the created and networked instruments.

•

In some cases the fact that the kits were managed by independent experts favoured the honest expression of opinions of students.

•

Possibility of integrating the kits in the teaching curricula of the students (which happened with the Spanish partner).

•

Active role of students (especially clear in the use of peer education which has also guaranteed a broad impact on many students).

•

Growing interest of teacher in the kits (in some cases during the first testing phase they were suspicious, during the second testing phase many asked to be involved).

•

Increased knowledge and awareness of students on the themes.

Negative aspects: •

Logistics and time required to carry out the set activities: problems with identifying the more appropriate moments to place them in the teaching programme (ex. June is the end of courses) with a resulting reduction in the time available for testing.

•

In some cases mistrust of trainers who do not wish to take part in the administration of the kits.

•

Starting phase of a motivated and interested group, especially when participants have very different knowledge and experiences on the themes under examination (ex. some are usual users of drugs others are not).

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

15

•

Problem with completing the programme in the cases in which the involved beneficiaries were not usual participants of long-term training programmes or in the cases in which they dropped out before the end of the programmes.

•

Possibility of using the programme promoted by the kits to work on the classroom climate and the group consolidation.

From analysing these remarks it is clear that the strengths of the experience exceed the criticalities, in relation to which the partnership has also adopted effective response strategies over the course of the project.

We have already mentioned the deliberately flexible nature of the instruments offered in order to permit their subsequent re-use in other contexts. In this regard, to conclude, it is useful to mention some aspects that is worth considering to guarantee the feasibility of this transferability: especially in the cases in which independent experts have been involved in the management of the activities, the reproducibility of the experience is subject to finding new funds which permit their involvement. It also emerges that it would be useful to keep working on the project website in order to make the published instruments increasingly clear and useable.

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

16

Annex 1 Project “POVEL” Daphne Programme

Trial diary Each partner who has to do the trail will describe the mail events that will happen during experimentation specially unexpected events.

The diaries: - will give information to evaluate kits transferability in other contexts - will give useful information for writing guidelines

Date and place Who compile

Event description

Materials produced (describe and attach them)

Things unexpected ?

Any decision taken due to unexpected things?

if yes describe it -

Outcome of unexpected actions

-

-

The trial diaries will be write in english and (valutazione@scformazione.org) at the and of experimentation.

sent

to

Caterina

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

Aimé

17

Annex 2 Project “POVEL�

Daphne Programme

TRIAL INTERVIEW OUTLINE Each partner who did the trail will describe the mail events experimentation specially unexpected events.

This interview is aimed to: - collect information to evaluate kits transferability in other contexts - collect useful information for writing guidelines

that will happen during

1.What Kits have you tested?

1) 2) 2.How many students were involved in each kit tested? The students who are involved in first trial group were the same involved in second group?

1) 2) 3.How long was the trial? And how it was organized? (when it started and ended, how many hours for each day? How many students for each trial group? How many tutors for each trial group? Do you have done trial following the kit or you did some changes?)

4.During trial activities did unexpected things happen? If yes describe it: for example: discussions between students, problematic personal experiences, 5.What did you do to face up to these unexpected situations?

6.Can you tell me 2 of the two most positive aspects that you think came out of the trial?

1) 2) 7. Can you tell me 2 of the two most negatives aspects that you think came out of the trial?

1) 2) 8.Do you think that in the future your organization will continue to use the tested kits with other groups of students?

9.Are there any aspects in kits that you would change after the trial? If yes what and how?

10. Do you have any other comments?

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

18

Annex 3 Project “POVEL” Daphne Programme

POVEL-Kit Trial costumer satisfaction for students At the end of each kit each student will complete this questionnaire. The goal is to collect student’s opinions about activities.

1. Kit Topic tested: µ Violence related to drug assumption µ Violence related to alcohol abuse µ Peer violence and bullying µ Media Violence 2. Think about Kit topics: µ I didn’t learn anything new µ I learned some new things µ I learned many new things 3. Your opinions about the kit topic: µ They are changed µ They are partially changed µ They are the same 4. During presentations and discussions: µ I wasn’t interested µ The topics were interesting but they were addressed in a boring way µ I was interested 5. I was more interested in: µ Presentations µ Discussions µ Anything else (specify):

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

19

6. Did you talk about kit’s topics with friends? µ Yes µ No 7. If your friend was in one of the situations have been described during Kit: µ I would like to give him/her some advice, but I think I’ll not be able to do it µ I think I would be able to give him/her some information µ I did not say anything 8. Would you like to participate in others training moments like this? µ Yes µ No 9. If you have any comments, write them here:

This project has been produced with the financial support of the Daphne III Programme of the European Commission.

20