DEAR DEAR
Monitoring Monitoring &&Evaluation Evaluation
Learning for Development Education and Awareness Raising (DEAR) Lessons to help you maximise the Impact of Your DEAR Project
Monitoring & Evaluation Monitoring Monitoring&& Evaluation Evaluation
DEAR DEAR
DEAR DEAR
Monitoring Monitoring &&Evaluation Evaluation
Monitoring & Evaluation
How to effectively Monitor and Evaluate Your Project Monitoring Monitoring&& Evaluation Evaluation
DEAR DEAR
Challenges Challenges
Monitoring Monitoring &&Evaluation Evaluation
Common Challenges - DEAR Projects Inability to construct working indicators Unaware of different tools to monitor and evaluate In practice, results are often measured without clear definition of objectives or detailed targets. Misinterpretation of the logframe Challenges Challenges
Monitoring Monitoring
DEAR DEAR
Monitoring Monitoring &&Evaluation Evaluation
Interactive Session
Questions & Answers Monitoring Monitoring&& Evaluation Evaluation
DEAR DEAR
Why WhyMonitor Monitor&& Evaluate? Evaluate?
Monitoring Monitoring &&Evaluation Evaluation
Why Monitor and Evaluate? Evaluations Evaluationsare arean animportant importantcontribution contributionto tothe the overall overallimprovement improvementof ofDevelopment DevelopmentEducation. Education.
• An evaluation is a powerful tool. It allows us to self-reflect, learn and improve the quality of our work. • The results of an evaluation should enable us to plan our activities according to our long-term goals. Monitoring Monitoring&& Evaluation Evaluation
Why WhyMonitor Monitor&& Evaluate? Evaluate?
Why WhyMonitor Monitor and andEvaluate? Evaluate?
Monitoring Monitoring &&Evaluation Evaluation
• Continuous monitoring and evaluation improves problem solving skills. • Effective monitoring and evaluation is essential in the construction of interim reports and final report. • Evaluation of performance not only accents our weaknesses and downfalls, but we may uncover hidden strengths within our work practices. “Partners “Partners have have found found that that regularly regularly reviewing reviewing the the project’s project’s achievements achievements and and direction direction not not only only enables enables them them to to take take stock stock of of the the current current situation situation and and identify identify potential potential problems problems early, early, but but is is also also motivating motivating and and inspiring.” inspiring.” -Amnesty Amnesty International’s International’s Education Education for for Human Human Dignity Dignity (E4HD) (E4HD) project project
Why WhyMonitor Monitor&& Evaluate? Evaluate?
Monitoring Monitoring&& Evaluation Evaluation
Monitoring Monitoring
Monitoring Monitoring &&Evaluation Evaluation
What is Monitoring? Monitoring is a continuous action which requires observing and checking the progress or quality of (something) over a period of time and keeping it under systematic review. It involves: •maintaining regular surveillance •examining and reporting •regulating quality standards Monitoring Monitoring&& Evaluation Evaluation
Monitoring Monitoring
Evaluation Evaluation
Monitoring Monitoring &&Evaluation Evaluation
What is Evaluation? Evaluation requires a set of tools providing an evidence-based assessment of results and impacts of interventions. Evaluations focus on issues such as relevance, effectiveness, efficiency, sustainability, utility and added value of EU interventions. Any evaluation includes : •the definition of the subject •the formulation of criteria and indicators •the collection and interpretation of data •the communication about the results
Monitoring Monitoring&& Evaluation Evaluation
Evaluation Evaluation
Monitoring Monitoring
Monitoring Monitoring &&Evaluation Evaluation
Monitoring How Howto toestablish establishaasystematic systematicmonitoring monitoring process processfor forYour YourProject Project
• In the design of an awareness raising campaign plan and construct monitoring systems
Monitoring Monitoring&& Evaluation Evaluation
Monitoring Monitoring
Monitoring Monitoring
Monitoring Monitoring &&Evaluation Evaluation
Results Based Management (RBM) OECD report 2001 emphasises the success of RBM approach in OECD countries. RBM is the process an organization follows to objectively measure how well its targets are being met. It involves: •selecting indicators and setting targets •monitoring performance (collecting data on results) •analysing those results vis-à-vis targets Monitoring Monitoring
Evaluation Evaluation Methodologies Methodologies
Monitoring Measures • • • • •
Barometer Questionnaires Follow –up seminars Expertise databank Meetings with Partners
Evaluation Evaluation Methodology Methodology
Monitoring Monitoring &&Evaluation Evaluation
Evaluation Methodology How Howto toestablish establishthe theright rightevaluation evaluation method methodfor forYour YourProject Project • Evaluation should not be left to the end of the project. In the design of an awareness raising campaign : systematic construction of evaluation and monitoring systems • Use manageable tools and indicators • Qualitative or Quantitative • Internal or External
Monitoring Monitoring&& Evaluation Evaluation
Evaluation Evaluation Methodology Methodology
Monitoring Monitoring &&Evaluation Evaluation
Types of Evaluation Type
Definition
Questions Asked
Formative
Assesses the strengths and weaknesses of campaign materials and strategies before the campaign’s implementation.
How does the campaign’s target audience think about the issue? What messages work with what audiences?
Process
Examines the campaign’s implementation and how the activities involved are working.
How many materials have been put out? How many people have been reached?
Outcome
Measures effect and changes that result from the campaign. Also measures policy changes.
Has there been any affective change (beliefs, attitudes, social norms)? Have any policies changed?
Impact
Measures community-level change or longer-term results that are achieved as a result of the campaign’s aggregate effects on individuals’ behavior and the behavior’s sustainability. Attempts to determine whether the campaign caused the effects.
Has the behaviour resulted in its intended outcomes (e.g. lower cancer rates, less violence in schools)? Has there been any systems-level change?
Evaluation Evaluation Methodologies Methodologies
Some Questions to Answer • Was my action useful and effective? • Did I reach the aim I have set myself? • What were the effects of my action?
Different ways of evaluating in DEAR Two levels of evaluation 1.Evaluation of national campaigns 2.Evaluation of actions carried out at the local level
8 Stages of Self-Evaluation 1. 2. 3. 4. 5. 6. 7. 8.
Define the Action you want to evaluate Clarify the objective of the evaluation Determine who does what? Determine what do you seek from evaluating Choose criteria and their indicators Choose appropriate methods and tools Conducting the evaluation Valorise and use of your evaluation
Choosing Evaluation criteria The most used criteria are: •Effectiveness: compares the results obtained and the results expected •Efficiency :questions the inputs made available to carry out the action in comparison with the results achieved •Relevance: refers to the decision of the action and to values •Coherence: Was the method planned at the beginning the right method to achieve the expected results •Sustainability •Social usefulness •You can create specific criteria close to your expected result
Example of Evaluation Criteria Questions on
Model questions Evaluation criteria Indicators
Approach
Did you choose the right public?
Performance
Is the target groups’ rhythm adequate?
Behaviour of public
Are the partner organizations actively involved in the process?
Meaning given to the action
Does the action provoke a critical change of behavior?
Coherence Does the chosen public match with our educational practice? Decentring Ability to take into account the public’s constraints Readability Ability to make the importance of the theme recognized by the training institution Representations Ability to change and diversify the public’s opinions on the themes covered
Feedback/Reactions from public
Observation of changes in rhythm
The action appear in the activities reports of the partners; frequency of being mentioned How the public views the same concept before and after the activity
Indicators Indicators
Monitoring Monitoring &&Evaluation Evaluation
Evaluation Indicators Develop indicators which are: •Specific •Measurable •Attainable •Realistic •Time Bound Indicators Indicators
Evaluation Evaluation Methodologies Methodologies
The Qualities of an Indicator • Reliability: an indicator must reveal what you are seeking to find out • Easy to establish: The indicator must illustrate clearly and respond to the question asked without ambiguity
Indicators Indicators
Monitoring Monitoring &&Evaluation Evaluation
Consider these Indicators
Indicators Indicators
Evaluation Evaluation Methodologies Methodologies
To Resume • Choose a criteria as close as possible to your objective/result desired • From the criteria obtained, formulate the indicators enabling you to have precise information to collect
Indicators Indicators
Monitoring Monitoring &&Evaluation Evaluation
Indicator: Website hits – Social Media Project Example: Who Cares About Sophie - Denmark After an awareness raising event, effectiveness was monitored by the increase in website views, emails and contacts they received in the aftermath. Project Example: Linking Real Lives – UK The use of broadcasting videos on Youtube allows you to effectively monitor the amount of views the video has had. Youtube does this automatically and it is clearly shown!
Indicators Indicators
Evaluation Evaluation Methodologies Methodologies
Logframe Logframe
Monitoring Monitoring &&Evaluation Evaluation
Evaluation Tools – EC- EuropeAid • • • • • • • • • • • • •
Problem Diagram Objectives and effects diagram Decision diagram Questionnaire Survey Context indicators Interview Focus Group Expert Panel Case Study SWOT (Strengths, Weakness, Opportunities, Threats) Multi-criteria analysis Cost/effectiveness analysis Socio-cultural analysis Logframe Logframe
Evaluation Evaluation Methodologies Methodologies
Evaluation Evaluation Methodology Methodology
Monitoring Monitoring &&Evaluation Evaluation
Evaluation Tool: Survey • Surveys are one of the most popular tools used to measure awareness. • This tool evaluates how aware people are of an issue by asking them directly what they know. • A survey should be carried out pre-implementation and post implementation for more results.
Monitoring Monitoring&& Evaluation Evaluation
Evaluation Evaluation Methodologies Methodologies
Evaluation Evaluation Methodologies Methodologies
Monitoring Monitoring &&Evaluation Evaluation
Surveying Method Methodology Methodologyof ofevaluation evaluationused usedon onPeer PeerUp! Up! Project ProjectininGermany Germany&&Austria Austria • Distribution of a mini-questionaire to targeted officials in order to compose an accurate picture of the inter-relationships established by the contracting agency representatives with national and local officials. • This allowed for composition of a focused questionnaire for the directive interviews that were conducted via e-mail with 2 representatives of each group: World Vision staff directly involved in the implementation, World Vision staff outside the direct implementation, partnering NGOs, initiatives and individuals, the target groups. Monitoring Monitoring&& Evaluation Evaluation
Evaluation Evaluation Methodologies Methodologies
Logframe Logframe
Monitoring Monitoring &&Evaluation Evaluation
The Logical Framework The Thelogical logicalframework frameworkor orlogframe logframeisisan ananalytical analyticaltool toolused usedto toplan, plan,monitor, monitor,and andevaluate evaluate projects. projects. ItItisisbased basedon onlogical logicallinkages linkagesset setout outby bythe theplanner(s) planner(s)to toconnect connectaaproject’s project’s means meanswith withits itsends. ends.
• The Logframe is a tool used to help strengthen project design, implementation and evaluation. • It is a living document • It incorporates: Goal Indicators Purpose Evidence with Outputs Assumptions Activities Monitoring Monitoring&& Evaluation Evaluation
Logframe Logframe
ROM ROM
Monitoring Monitoring &&Evaluation Evaluation
Results Oriented Monitoring (ROM) ROM ROMisisaareview reviewtool toolfor forprojects projectsand andprogrammes programmeswhich whichprovides providesrecommendations recommendationsfor for improvement improvementand andan anoverview overviewof ofEuropeAid's EuropeAid'sportfolio's portfolio'squality. quality. • It involves using a highly structured and consistent methodology • It measures project performance against criteria:
Relevance Efficiency Effectiveness Potential Impact Sustainability
For more information see: http://ec.europa.eu/europeaid/how/ensure-aid-effectiveness/monitoring-results_en.htm Monitoring Monitoring&& Evaluation Evaluation
ROM ROM
ROM ROM
Monitoring Monitoring &&Evaluation Evaluation
ROM – Grid example
Monitoring Monitoring&& Evaluation Evaluation
ROM ROM
Conclusion Conclusion
Monitoring Monitoring &&Evaluation Evaluation
Thank you for your attention! For further information on this presentation or the projects mentioned, please visit the CiSocH website: CiSocH EC-DEAR
Monitoring Monitoring&& Evaluation Evaluation
Conclusion Conclusion