Monitoring & Evaluation Workshop by Cecilia Castillo

Monitoring & Evaluation W orkshop

July, 2006

Table of Contents Presentation .................................................................................................3 Monitoring & Evaluation Workshop: An Introduction ......................4 Defining Monitoring & Evaluation............................................................5 Importance of Monitoring and Evaluation.............................................6 Indicators and Their Use...........................................................................7 Realistic Expectations for Monitoring and Evaluation ......................12 Monitoring & Evaluation Plan .................................................................14 Monitoring and Evaluation Tools...........................................................17 How to Conduct a Project Evaluation.................................................19 Creating an Organizational Culture of M&E.......................................23 Bibliographic References .........................................................................25 Annexes.......................................................................................................26

Presentation Pact is a non-profit, non-governmental organization (NGO) with headquarters in Washington DC and 15 regional offices which oversee activities in 60 countries. Pact has 550 partners throughout Africa, Asia, Latin America, Eastern Europe, and the Middle East. Its main areas of action include: government and democracy, HIV/AIDS, natural resource management, peace promotion, social inclusion, equity, empowerment of vulnerable groups, gender, and micro-credit. Pact has been working in Brazil since 1997 and has contributed to the social development and reduction of the vulnerability of the Brazilian population by networking and partnering with local organizations and governments. In 2003, Pact Brasil inaugurated its country office and has since focused primarily on providing technical, financial, and operational assistance to NGOs implementing activities in the area of HIV/AIDS through the Elos Program. The Elos Program, which is funded by USAID/ Brazil, is comprised of two cooperation agreements - Non-governmental Organizations (NGOs) and Condom Social Marketing (CSM). The main objective of the Elos Program is to help reduce HIV incidence and prevalence within the most vulnerable population groups including homosexuals and other men that have sex with men (MSM), sex workers (SW), transgender population (TG), drug users (DU) and people living with HIV (PLHIV) of the South and Southeast regions of the country. The Elos Program carried out two selection processes for projects to be supported: one within the scope of the CSM component and the other within the NGO component. As a result of these selection processes, 24 agreements were signed with Brazilian NGOs to implement HIV prevention activities and increase access to and use of male condoms. In an effort to effectively implement the Elos Program and complete its mission of- contributing to the social development and reduction of the vulnerability factors faced by the population through the strengthening of civil society and establishment of networks and partnerships with local organizations Pact Brasil provides technical assistance to local NGOs in the following areas: Training in Financial and Administrative Management, Monitoring and Evaluation (M&E), Research, Communication, Human Resource Management, and Project Management. To this end, one of its M&E activities is to facilitate workshops for NGOs to build their capacity in this important area. This guide accompanies the workshop and synthesizes information on the terminology, methodology, and activities included in the workshop. It is important to stress that, in the process of developing and implementing the workshops and this guide, reference materials published by the Brazilian National STD and HIV/ AIDS Program and Center for Disease Control and Prevention were partially used and adapted (see Bibliographic References).

Monitoring & Evaluation Workshop: An Introduction 1. Objective Build the capacity of participating organizations in the area of monitoring and evaluation (M&E) through the presentation and discussion of M&E concepts and methodologies and the application of participatory exercises.

2. Subjects Discussed Defining Monitoring and Evaluation What is Monitoring and Evaluation? The workshop begins with a participatory exercise (Annex I) regarding each participant s current definition of the concepts of monitoring and evaluation (Annex I). Importance of M&E Presentation of why M&E is a crucial component of any organization and/ or institution and review of the primary areas in which it is useful. Indicators and How to Use Them Presentation of the general concept of an indicator focused on their use in the design, implementation, and M&E of projects/ programs. Application of a practical exercise (Annex II) to facilitate debate among participants and review new terminologies using examples from the Elos Program. M&E Plan Presentation of a model M&E Plan based on HIV prevention projects. Presentation highlights the necessary components of an M&E Plan and its importance as part of an organizational culture of M&E. Participants will develop an M&E collectively as part of a group exercise (Annex IV). M&E Tools Discussion and presentation of the primary data collection tools used to monitor and evaluate projects. A group exercise (Annex V) stimulates reflection on the instruments currently used by the NGOs to collect and analyze data. How to Conduct a Project Evaluation Presentation of a practical methodology for conducting a project evaluation with limited financial and human resources. Creating an Organizational Culture of M&E Discussion of the importance of integrating M&A into all phases of projects/ programs. An individual exercise (Annex VI) stimulates reflection and discussion regarding the organization s current monitoring and evaluation activities.

Defining Monitoring & Evaluation 1. What is Monitoring? Monitoring is a systematic process of collecting and analyzing information to accompany the implementation of an action, project or program. It also provides regular feedback that helps track costs related to the development of an action, project or program.

2. What is Evaluation? Evaluation is a systematic process of collecting and analyzing information that determines to what extent an action, project or program has achieved its defined goals and objectives. It involves collecting and analyzing information to explain the results and outcomes of an action, project or program, in an effort to determine its merit or value.

Evaluation is not an audit nor does it seek to place blame.

Importance of Monitoring and Evaluation 1. Monitoring is important because it: Systematically tracks the implementation of actions that form part of a project/program Collects information that may be used to assess the efficiency of projects / programs Provides information to adapt and adjust projects / programs during execution

2. Evaluation is important because it: Contributes to enhancing the effectiveness of an action, project or program by providing information to subsidize strategic management decisions regarding the continuation, modification, or termination of certain activities Provides feedback to help assess the relevance, scope, and impact of an action, project or program Provides information in which to base future actions and proposals

Indicators and Their Use 1. Indicators Indicators are used to measure the process, outcomes, and impact of an action, project, or program. They measure quantitative and qualitative information systematically collected during and/or after the implementation of an action, project or program. Indicators are directly related to the objectives, strategies, and processes of the project / program In general, indicators are divided into three categories: impact indicators, outcome indicators. (short-term and long-term) and process indicators.

Graph 1: Relationship between indicators and objectives and activities Overall Objective

Impact

Specific Objective

Outcome

Outcome Indicator

Strategies

Outputs

Process Indicator

Impact Indicator

I1. What to Monitor and Evaluate A. Inputs Inputs are the financial, human, and material resources used in the project / program implementation. Examples of inputs in an HIV prevention project include: Technical team to facilitate training Technical staff for field activities Condoms Informational, Educational, and Communication (IEC) Material

Input Monitoring Input monitoring involves a systematic process of collecting information regarding the inputs that enable the strategies of a project/program to be implemented. Based on the example above, the input monitoring process should answer the following questions: How many consultants were hired to perform the training? How many people form part of the technical team? How many condoms are obtained every six months? How many IEC materials are available for distribution? The answers to these questions are examples of process indicators, i.e.: Number of consultants hired to facilitate training Number of members in the technical field team Number of condoms obtained every six months by the project Number of IEC materials available for distribution

B. Activities Activities are actions developed as part of the strategies of a project to achieve the expected outcomes and results. In an HIV prevention project, examples of activities include: Technical training of health agents Workshops on how to negotiate condom use Peer education field interventions Distribution of condoms during field interventions Distribution of IEC material during field interventions Monitoring of Activities The monitoring of activities involves a systematic process of collecting and analyzing information related to the immediate effects and short-term results (also called outputs), of the project/program activities. Based on the example above, the monitoring of activities should answer the following questions: How many health agents were trained? How many workshops were held? How many people were reached through the educational activities? How many condoms were distributed? How many IEC materials were distributed? The answers to these questions are the short term outputs of the project implementation process and provide examples of process indicators of a project/program, i.e.: Number of people trained 8

Number of workshops held Number of participants in the workshop Number of people reached in educational activities Number of IEC materials distributed Evaluation of Inputs and Activities, or Process Evaluation A process evaluation will describe in detail how the actions of the project / program were implemented and to what extent the intervention reached the intended population. Based on the examples used, a process evaluation will address the following questions: Were the activities implemented as originally planned? Why or why not? Were the people reached satisfied with the quality of the activities implemented? Did those trained in the workshops increase their knowledge level regarding HIV transmission mechanism? Did the population prioritized by the intervention have access to condoms? If not, what were the barriers to access?

C. Outcomes Outcomes are the effects obtained by the project/ program on the population group reached through prevention activities. They are generally obtained over an intermediate time period and tend to reflect changes in practices, attitudes and behavior of those reached in project/ program activities. Depending upon the specific objectives and time frame of the project or program, outcomes may also include changes in knowledge/ awareness, although generally in longer term projects, increases in knowledge or awareness are considered an output of prevention activities. Based on the examples provided above, the following are examples of possible outcomes of HIV prevention activities: Increase in condom use with regular partner Increase in condom use with non-regular partner Monitoring Outcomes Monitoring the outcomes of a project involves a systematic follow-up of information associated with the expected outcomes and those obtained by the project/ program in the short and intermediate time frame. Based on the examples provided, the monitoring of outcomes should address the following questions: What is the current rate of condom use among the intended population with regular and nonregular partners? The answers to the questions above are outcome indicators of a project/program, such as: Percentage of population reached reporting condom use at last sex with a non-regular partner. Percentage of population reached reporting condom use at last sex with a regular partner.

Outcome Evaluation On outcome evaluation will determine the effectiveness of a project/ program, demonstrating the extent to which the outcomes achieved coincide with the expected outcomes and project objectives. An outcome evaluation emphasizes the causal relationship between the results obtained and activities implemented. An outcome evaluation should address the following questions: Is there any relationship between the increase in condom use and participation in the prevention activities implemented by the project? Based on the example provided above, it is necessary to demonstrate that the increased use of condoms was due to the activities of the project. The explanatory nature of the evaluative process is what differentiates Outcome Evaluation from Outcome Monitoring.

D. Impact Impact is the overall, long-term cumulative effect of one or more projects or programs on the general population. In the case of HIV prevention, examples of impact include: Reduction of HIV incidence Reduction of HIV vertical transmission rate Monitoring Impact In HIV prevention projects, impact monitoring involves a systematic process of collecting and analyzing information related to the epidemiological trends of HIV/AIDS. Impact monitoring responds to the following question: What is the effect of all the projects implemented within a certain timeframe on the incidence rates of HIV? The answer to this question corresponds to two impact indicators; for instance: HIV incidence among MSM Percentage of HIV-infected infants born to HIV-infected mothers Impact Evaluation Impact evaluation determines to what extent the epidemiological tendencies, i.e. increases or decreases in HIV incidence, observed in impact monitoring are attributable to a specific project or program. An impact evaluation should address the following questions: What percentage of variation in the incidence of HIV is due to a specific project or program? What percentage of the total decrease in HIV prevalence was due to the implementation of a specific project or program?

III. Summary Table - Examples of Indicators

T YPE OF I NDICATOR

EXAMPLES

Number of members in the technical team Number of consultants hired to facilitate training Number of health agents trained Process Indicators

Number of workshop participants Number of workshops held Number of people reached in prevention activities Number of condoms distributed Number of IEC materials distributed Percentage of intended population reporting condom use at last sex with a non-regular partner

Outcome Indicators

Impact Indicators

Percentage of intended population reporting condom use at last sex with a regular partner

HIV incidence among MSM Percentage of HIV-infected infants born to HIV-infected mothers

Realistic Expectations for Monitoring and Evaluation 1. Everybody can monitor inputs and activities Techniques Commonly Used Situation Analysis: Analysis of the current situation of the NGO in terms of inputs available and the institutional capacity of the NGO to implement the actions planned as part of project or program. Routine data collection: Systematic collection of information on the financial, human, and material resources of a project or program using specific tools and/or supervisory visits.

2. Most institutions can conduct a Process Evaluation Techniques Usually Used Process Evaluation: A descriptive assessment of project implementation generally conducted by analyzing the data collected through monitoring and supplemented, if necessary, by the application of additional instruments. Case Study: Instead of evaluating a project/ program as a whole, a case study assesses a specific component (geographic or programmatic) of a project/ program, with the goal of understanding and describing a specific situation in greater detail. Cost Analysis: Analysis of the efficiency of a project/ program by comparing the costs per person reached to the benefits of activities performed in an effort to determine to what extent a project was efficient.

3. Some institutions can conduct M&E of outcomes Techniques Usually Used Routine collection of data on outcome indicators: Generally involves the application of specific tools in the field, such as questionnaires, to determine if the action, project or program is achieving the desired outcomes. Outcome evaluation: Application of specific tools evaluative nature (i.e. survey, questionnaire, in-depth interviews), to verify whether the outcomes obtained were due to the actions implemented as part of the project/program.

4. Few institutions can conduct Impact M&E Techniques Usually Used Routine collection of data on impact indicators: Accompaniment of epidemiological surveillance, generally by governmental programs, of HIV incidence, prevalence, and forms of transmission. Impact Evaluation: A comprehensive evaluation involving the collection and analysis of epidemiological data, documented outcomes of project activities, and descriptive data regarding project implementation.

Monitoring & Evaluation Plan A M&E Plan is a fundamental tool for monitoring and evaluating a project or program. The plan organizes all activities related to M&E into a table that generally includes the objectives, activities, indicators, measurement tools, and responsible parties with the goal of ensuring that data are collected and analyzed throughout the project in a systematic and timely manner. A M&E Plan should be: Developed collectively during the development of the project or program Systematically updated throughout the project. The process of developing a M&E Plan can be divided into three basic steps: Definition of the project or program indicators Definition of the measurement tools for the indicators Definition of those responsible for collecting data related to the indicators

1. Definition of Indicators The SMART methodology, traditionally used for defining desired outcomes, is also useful for defining indicators. To be considered SMART, an indicator should be: Specific: The indicator should clearly specify what it will measure Measurable: The indicator must be measurable by quantitative or qualitative mechanisms Appropriate: The indicator must directly relate to the project goals and objectives Realistic: The NGO must have the resources necessary, human and financial, to measure the indicator. Temporal: The indicator must be measurable within the project timeframe. After defining an indicator for an action, project or program, verify the following for each indicator: Does the indicator clearly specify what it will be measuring? What methodologies/data collection tools would be necessary to measure the indictor? Do these tools effectively measure what the indicator proposes? Is the indicator clearly related to the project goals and objectives? Given the available resources and technical expertise of the project team, does the NGO have the capacity to collect and analyze data necessary to report on this indicator? Given the timeframe in which the project will be implemented, is it feasible to expect a change in the indicator? If the answers to all of the questions above are affirmative, the indicator is SMART! 14

2. Definition of Measurement Tools Measurement tools are the instruments that the project/ program will use to measure the indicators. Examples of measurement tools include attendance lists, field reports, questionnaires, focus groups and observations. The measurement tools should be: Relevant to the indicators Feasible in terms of the resources available and project timeline Systematically collected and analyzed

3. Definition of Responsible Parties Although the entire project/ program team should participate in M&E activities, it is essential to define one or two persons who will be in charge of collecting, analyzing, and reporting data on each indicator. They will work in partnership with the rest of the project/ program team to guarantee that the data necessary for each indicator are systematically collected. When defining those responsible for each indicator, it is important to avoid centralizing all activities in the project/program director or coordinator. They generally have many responsibilities in the actual project implementation and may not have the time necessary to dedicate to monitoring and evaluation activities.

Graph 2: Example of M&E Plan for a two-year-project focused on HIV prevention with men who have sex with men General Objective*

Specific Objective

Outcome Indicator

- Contribute to the reduction of HIV incidence in the MSM population in S達o Paulo

- Increase condom use among the MSM population reached through prevention activities in S達o Paulo

- Condom use are last sex with nonregular partners - Condom use are last sex with regular partners

Measurement Tools - KAP questionnaire applied in the beginning and the end of the project

Responsible Parties - External consultant to develop questionnaire, define sample, and analyze results - Researchers to apply the questionnaire

Activity

- Educational workshops on how to negotiate condom use with regular partners - Peer education actions in bars, saunas and night clubs frequented by the MSM population in SP, with distribution of male condoms and IEC material

Process Indicators - Number of workshops - Number of workshop participants - Number of peer education interventions - Number of people accessed in educational actions

Measurement Tools

Responsible Parties

- Attendance list

- Workshop facilitator

- Activity Reports

- Peer educators

- Peer educator field reports - Condom distribution spreadsheet - IEC material distribution spreadsheet

- Number of condoms distributed - Number of IEC material s distributed

*Given that this is an example of a two-year project, it is not possible to measure impact indicators.

Monitoring and Evaluation Tools 1. Examples of M&E Tools Semi-structured Interview This tool is for collecting qualitative information on the population groups reached through project activities. The semi-structured interview guide has open-end questions that allow the interviewee to provide more detailed and in-depth information on the subject in question (see model Annex VII-A). A Voluntary and Informed Consent Form should be signed before conducting an in-depth interview (see model Annex VII-B). Focus Groups Focus groups are ideal for collecting qualitative information with a high degree of depth regarding the perceptions and beliefs of members of a specific population group. The group consists of a group discussion between 6-10 members of the intended population to discuss their perceptions of a specific theme, such as stigma or condom use, in addition to their satisfaction with the quality of activities implemented as part of the project. The focus group guide consists of a several open ended questions designed to introduce the key discussion points into the group (see model Annex VII-C). A Voluntary and Informed Consent Form should be signed by each participant before carrying out a focus group. Knowledge, Attitude and Practice (KAP) Questionnaire The KAP questionnaire collects quantitative data on the knowledge, attitudes, and practices of project participants (see model Annex VII-D) and is useful in the monitoring and evaluation of outcomes. When used in a program evaluation, it also assists in assessing the level of participation in project activities among questionnaire respondents, thereby permitting a more detailed evaluation of the causal relationship between project activities and outcomes. Before applying the KAP questionnaire, it is important that the participants in the study also sign a Voluntary and Informed Consent Form. Forms for Peer Educator Fieldwork Activities Peer educator forms are fundamental for monitoring inputs and activities during the implementation of project activities. Forms generally include the number of condoms and IEC materials distributed, people reached (differentiating between new and regular contacts), and locations visited (see model Annex VIIE). Observations Observations are a useful tool for monitoring the quality of health care services and field interventions. The observation tool is generally a checklist of behaviors and structural factors that one would hope to observe in a high quality intervention or service delivery. The tool is applied in the field, during the intervention or service delivery. Condom and other Prevention Supplies Spreadsheet The purpose of this tool is to document the receipt and distribution of condoms and other prevention supplies throughout project implementation. It is important that the organization specify within the spreadsheet regarding where the shipments come from (i.e. government or partner organizations), the date they were received, where and when they are distributed, and to whom.

Educational Material Spreadsheet The spreadsheet serves to document the receipt and distribution of educational materials. The first entry must be recorded when the organization receives the material from the print shop and/ or from partners, and the distributions must be recorded after each field work activity. It is important that the organization differentiate within the spreadsheet regarding the type of material, who supplies it (i.e. government or partner organizations, where and when it is distributed, and to whom. IEC Material Evaluation Questionnaire Before investing a lot of time and resources into the production of IEC material, it is important to conduct interviews and focus groups with the populations reached in prevention activities regarding their perceptions, opinions, and understanding of the material to be produced. This will guarantee that the material is appropriate for the population group it addresses, and that the message/ information conveyed is clear and pertinent (see model Annex VII-F). Questionnaire for assessing the quality of a specific activity After a workshop or training session, it is important to assess the opinions of the participants with respect to the activity, as well as request suggestions on how to improve it. See example of the questionnaire used to evaluate the quality of this workshop in Annex VII-G. Pre/Post Tests Pre/ post tests are used to measure the effectiveness of educational activities such as trainings and workshops. They test participants on their knowledge regarding the topics covered before and after the activity.

How to Conduct a Project Evaluation The following is an example of how to conduct a project evaluation when you have limited technical and financial resources. It is helpful to conduct this exercise using concrete examples from the organization as this will make developing and visualizing the evaluation process easier. Before beginning any program evaluation, it is crucial to define your study objectives. The objectives must be directly related to the project objectives and indicators. For instance, if the project outcome indicators are knowledge of HIV prevention methods and condom use at last sex , the questionnaires and guides developed must measure these indicators with specific questions on prevention methods and condom use. After the objectives of the evaluation have been defined in accordance with the objectives of the project or program, the following six steps should be followed: Development of the instruments to be applied Definition of sample Interviewer training Application of questionnaire and focus groups Tabulation of collected data Analysis of results

1. Development of appropriate instruments The instruments to be applied should be collectively defined with the project team. In this example, we will focus on two instruments that are frequently used to assess projects: KAP questionnaire and focus groups. The combination of these two methodologies ensures that you collect quantitative data to measure behavior and knowledge (KAP questionnaire) as well qualitative data (focus groups) that will provide a better understanding of the participant s perceptions and opinions regarding the data collected in the quantitative format. In order to develop the KAP questionnaire and the focus group guide, the following steps should be followed: Research examples of questionnaires applied in other projects. Questionnaires assessing HIV prevention projects are used throughout the world and may be used as a basis to create similar instruments. There is an example in this guide (Annex VII-D) in addition to many more on the internet. Develop a draft of a questionnaire with the project team, being careful to phrase questions with vocabulary appropriate for those who will be responding. When developing questionnaires, it is also important to bear in mind the future tabulation of the results. For instance, if the organization does not have a sophisticated statistics program, such as SPSS or Epi-Info, and will be using Excel, it will be easier to tabulate yes and no questions than to use multiple choice questions. Pilot test the questionnaire and focus group guide with members of the population. This is a fundamental step to guarantee that those who will be responding understand the questions, that the answers meet the objectives of the evaluation, and that the questionnaire format is user-friendly for the interviewers. Based on the results of this pilot test, the questionnaire should be revised and finalized. 19

2. Definition of the sample KAP Questionnaire The number of questionnaires to be applied may be defined based on the number of people reached by the project. The number will also depend on the technical and financial resources available for the evaluation. If financial resources are available, it is advisable to hire someone to calculate a representative sample. If financial resources are scarce, the following methodology will assist you in selecting a sample representative of the locations where your intervention will take place. The number of questionnaires to be applied in this case will depend largely on your budget and timeline. In order to select the places where the KAP questionnaire will be applied, data from mapping exercises of intervention sites should be used. It is important that the evaluation guarantee a representative sample of these sites in order to avoid eventual bias by the NGO in choosing the places to include in the evaluation. The following is a suggestion of how to ensure a representative sample without using sophisticated software. It tends to work best in projects that access a variety of bars or nightclubs with relatively small populations. Assign a number for each of the mapped establishments. For instance, if the NGO works in the Las Vegas night club and Bar da Rose , the number 1 may be assigned to Las Vegas and number 2 to Bar da Rose ; Verify the people accessed within each establishment and cut individual pieces of paper according to that number. For instance, if there are an average of 100 people at the Las Vegas night club, 100 individual small pieces of paper must be cut; if at Bar da Rose , there are an average of 15 people, 15 individual pieces of paper must be cut; Place the number of the establishment on each of these pieces of paper. For instance, if the code for Las Vegas night club was number 1, write number 1 on 100 pieces of paper and number 2 ( Bar da Rose ) on 15 pieces of paper. This way, each piece of paper represents one person at each of the establishments; Place all of the pieces of paper with the numbers of the establishments inside an envelope, hat or box and draw the number of questionnaires to be applied; Apply X number of questionnaires in the establishments selected through the random drawing. For instance, if 20 pieces of paper with the number 1 were drawn, 20 questionnaires should be applied at the Las Vegas nightclub; if 5 pieces of paper with the number 2 were drawn, 5 questionnaires should be applied at Bar da Rose.

Focus Group In general, focus groups are comprised of 6 to 12 people. As one of the goals of the focus group is to gain a deeper understanding of the perceptions and opinions of the population, it is important that all the participants have a similar sociodemographic profile (for instance: same sex, social class, profession, etc.) Who you choose to participate will depend largely on your study objectives. For example, if your goal is to evaluate project activities, participants may be selected from the attendance list in the NGO activities. If you are seeking to explore barriers to condom use, participants may be selected from intervention locations. When organizing the focus group, it is important to hand out written invitations to potential participants with the focus group details and cover the costs of transportation of the participants. It is also advisable to invite twice the number of people that are expected to participate in the focus group, given that is common for people not to show up. The number of focus groups to be carried out will depend on the time and resources available and the objectives of the evaluation.

3. Interviewer training KAP Questionnaire Ideally, the interviewers should be either a member of the intended population group or have expertise and/ or experience with the subjects addressed in the questionnaire and population group who will respond. It is essential that all the interviewers be trained to apply the questionnaire in order to ensure the consistency of the data collected.

Focus Group Given that the methodology to be used is more open-ended and qualitative, the focus group should be facilitated by a professional with experience and knowledge in the field/ subject to be evaluated. In addition to the focus group facilitator, it is also important to have a note-taker to record observations on the conversation and non-verbal communication during the focus group.

4. Application of Questionnaires and Focus Groups KAP Questionnaire The drawing will indicate the establishments where the questionnaires should be applied. To avoid any bias in terms of whom responds to the questionnaire, interviewers should define a methodology for selecting participants upon arrival in the establishments. For example, interview every third person encountered, etc.

Focus Group The focus groups should be conducted in a private, safe, and quiet location as they are generally recorded. A snack and small gift (lipstick, nail polish, etc.) to thank participants for their participation are also generally provided.

5. Data Tabulation KAP Questionnaire Depending on the resources available, the data may be tabulated in an Excel database or more sophisticated statistics program (such as Epi Info or SPSS). The most important concern is that the data be tabulated in a way that is easy to understand and analyze. It is vital that the data collected in the KAP questionnaire be tabulated immediately after the application of the questionnaire. This will allow the information to be used to improve the project/program.

Focus Group If there are sufficient financial resources, the audio recording from the focus group should be transcribed. If there are not sufficient financial resources, the facilitator and note-taker of the focus group should write a detailed report on their perceptions of the group with respect to the subjects discussed, highlighting the nuances of the discussion and opinions expressed and including any important quotes.

6. Analysis of the Results The analysis of the evaluation will depend partly on when it is conducted. As described below, it is important to conduct a baseline, mid-term, and final evaluation. The results of the baseline are used to define the most appropriate priorities and methodologies for the actions to be implemented. The results of the mid-term evaluation will indicate whether the project is being implemented as expected and subsidize strategic decisions regarding possible project modifications. The results of the final evaluation of the project determine to what extent the project achieved its goals and objectives in addition to providing data to report on the outcome indicators of the project/program. In order to explain why the project/ program achieved its objectives or not, it is vital to triangulate the information obtained through the questionnaires and focus groups with the information collected throughout the monitoring of the implementation of activities.

7. Evaluation Timeline Below is suggested timeline for a two-year project that includes a baseline, midterm, and final evaluation: Baseline: first month of project, before implementation of intervention actions Apply the KAP questionnaire in establishments where the intervention will take place before initiating any other project activity; Carry out focus groups; Tabulate the KAP questionnaire results in a database; Analyze the information obtained in the baseline and use data to finalize intervention strategies; Implement and monitor the project activities. Mid-term evaluation: One year after the initiation of project activities Apply the same KAP questionnaire and carry out Focus Groups; Tabulate the KAP questionnaire results in a database; Analyze the results obtained and compare them with baseline data; It is imperative that the results be compared with the information collected throughout monitoring in order to determine the effectiveness of the project and any possible causal relationship between the outcomes observed and the project/ program actions. Based on this analysis, the project team should verify the need for changes: intensify actions, adopt new approaches, etc. Final Evaluation: End of project activities Apply the same KAP questionnaire and carry out focus groups; Tabulate the KAP questionnaire results in a database; Analyze the results obtained and compare them with the information collected in the baseline and mid-term evaluations; The final evaluation will determine the overall effectiveness of the project or program. Therefore, it is imperative that the results be compared with the information collected throughout monitoring to establish any possible causal relationship between the outcomes observed and the project/program actions. 22

Creating an Organizational Culture of M&E I. The evaluative way of thinking: The relationship between M&E, planning, implementation and effects It is fundamental that M&E form an integral part of project planning and implementation at all stages. The relationship between evaluation, planning, implementation and outcomes is as follows:

Planning

Implementation

- Baseline - Situation Analisis

- Monitoring of Inputs and Activities - Process Evaluation

Effects

- Outcome Monitoring - Outcome Evaluation - Impact Monitoring - Impact Evaluation

Planning is the basis for M&A

Monitor and Evaluate

Understand

Implement Plan

Example of an evaluative way of thinking: The project initially planned 20 arts and crafts workshops. The activity was carried out, but after the first five workshops, the project team realized that the participants did not return. The team conducted focus groups with workshop participants in an effort to understand the reason for their decision to stop participating. The results of the focus group indicated the participants found the arts and crafts workshops of very little practical use in day-to-day life, and were more interested in sewing. In this case, the activity was modified to fit the interests of the population group, thus ensuring better results with respect to the project objectives. This is what M&E is: the project team accompanied and documented the implementation of an action, observed that it was not meeting its objective, conducted a focus group to understand why, and modified it in accordance to the results obtained. In this case, the M&E Plan should also be updated to reflect the change in project implementation. The project team s lesson will also serve for future planning as they learned that for this specific population group, arts and crafts workshops are not an appropriate intervention methodology.

Bibliographic References BERTRAND, J., Escudero, G. (2002). Compendium of Indicators for Evaluating Reproductive Health Programs. MEASURE Evaluation Manual Series 1(6): USAID. Monitoring & Evaluation Capacity Building for Program Improvement Field Guide - V ersion 1 - December 2003 Global AIDS Program - Centers for Disease Control and Prevention - ORC Macro. Translated by PN-DST/Aids - Evaluation Area: Aristides Barbosa Junior, Elizabeth Moreira dos Santos and Ana Roberta Pati Pascom. With the support of: PN-DST/Aids - Unidade de Desenvolvimento Institucional e Humano: Silvana Solange Rossi and Larissa Polejak, CDC GAP Brasil: Willian Brady and Suzanne Westaman. With the collaboration of: Célia Landamann Szwarcwald, Maria Cristina R. Baggio, Maria Elizabeth da S. H. Corrêa, Maria Sônia Natal, Teresa Seabra Soares and Zulmira Hartz. HARTZ, Z. (Org.). (1997). Avaliação em Saúde: dos modelos conceituais à prática na análise da implantação de programas. Rio de Janeiro: FIOCRUZ. McCOY, K. L., Njeri Ngari, P., Krumpe, E. (2005). Building Monitoring, Evaluation, and Reporting Systems for HIV/AIDS Programs. Washington DC: Pact. MONITORAIDS. System for Monitoring Indicators by the DST and A ids N ational Program. Department of Health. DST Aids National Program. Fundação Oswaldo Cruz. http://www.aids.gov.br/monitoraids PATTON, M.Q. (2005). Os desafios para tornar a avaliação útil. Activity notebooks for professional masters degree in Health Surveillance and Endemic Process Control Program Evaluation. Dimensão Técnica Operacional. Vol 2 (2). REHLE, T., Saidel, T., Mills, S., Magnani. (2001). Evaluating programs for HIV / A IDS prevention and care in developing countries. Washington DC: Family Health International SESSIONS, G. (2001). A valiação em HIV / A IDS: Uma perspectiva Internacional, Fundamentos de A valiação, (2). Rio de Janeiro:ABIA.

Annexes 1. Annex I: Brainstorming Exercise At the beginning of the workshop, before introducing the M&E concepts, ask the participants what their understanding of M&E is. Distribute cards of two different colors and ask participants to write their definition of monitoring on one card and evaluation on the other. Afterwards, request that they glue the cards to the bulletin board or wall and explain their definition to the group. Stimulate a discussion among the participants while also correcting any possible misconceptions of the concepts. The activity is useful for establishing participant s current level of understanding of the terms before beginning the workshop.

2. Annex II: SMART Indicators Exercise Introduce the items below and ask the participants to identify whether the indicators are SMART. 1. 2. 3. 4. 5.

Comprehension of the project as a whole. Ability to apply the instruments. Number of IEC materials distributed. Percentage of transvestites reached in each city that participated in the project. Knowledge of the population in general of the work carried out by the NGO.

The answers to this activity are described below: 1. 2. 3. 4.

Comprehension of the project as a whole No. The indicator is not specific or measurable. Ability to apply the instruments. No. The indicator is not specific. Number of IEC materials distributed. - Yes. Percentage of transvestites reached in each city that participated in the project.. No. The indicator is not measurable or realistic based on the available information and resources. 5. Knowledge of the population in general of the work carried out by the NGO. No. The indicator is not realistic.

3. Annex III: M&E

Definitions and Terminology - Exercise

In this exercise, the participants will receive the following exercise with instructions on how to fill in the blanks. The space for Exercise A should be filled in once the participant has understood whether the sentence corresponds to a monitoring or evaluation action. The space for Exercise B should be filled in once the participant has understood whether the sentence is an example of process, outcome or impact monitoring or evaluation. Instructions for participants Exercise A : Read the description of the intervention below, and answer each question, indicating whether it is a monitoring or evaluation activity. Exercise B: Classify each monitoring or evaluation as process, outcome, or impact.

The intervention An NGO is implementing an intervention for sex workers who work in downtown bars. The NGO planned a group-level action that comprises thirty workshops addressing basic information on HIV transmission and condom use negotiation with steady partners. The expected results are: increased level of knowledge on HIV and increased use of condoms with regular partners. This intervention is estimated to reach between 50 and 75 sex workers quarterly. 1. A questionnaire was applied with the sex workers who participated in the workshops regarding their knowledge of HIV prevention methods, before and after each workshop. Answer to exercise A:

Answer to exercise B:

2. Prior to implementing any project activity, a study on knowledge of HIV prevention methods and condom use with regular partners was carried out with sex workers of the city. One year later, the same questionnaire was applied and the results were compared to those of the previous year. Answer to exercise A:

Answer to exercise B:

3. A KAP questionnaire was applied with the sex workers that participated in the workshops, before and after the entire intervention. The same questionnaire was also applied with a similar group of sex workers that did not participate in the workshops. The results of these observations are compared to see if there were any behavioral changes, and whether these changes occurred in a different fashion in each group.

Answer to exercise A:

Answer to exercise B:

4. The NGO is collecting and analyzing information regarding the number of condoms distributed and the number of workshop participants that participated in each workshop. Answer to exercise A:

Answer to exercise B:

5. The NGO would like to know whether the intervention is making a difference. To this end, it collects several types of data regarding the participants behaviors and quality of the workshops through the application of questionnaires and focus groups. All this data was collected, put in a file cabinet, and never consulted or used. Answer to exercise A:

Answer to exercise B:

3. Annex IV: Exercise to develop a M&E Plan In this exercise, the group is divided into two or three subgroups in order to develop an M&E Plan. The facilitator of the activity, together with the participants, will first define a general objective and two or three specific objectives (depending on the number of subgroups) for a new project. Each subgroup will be assigned a specific objective and then develop a M&E plan in accordance with this objective complete with activities, indicators, measurement tools, and responsible parties. After developing all the above mentioned items, the participants will write them down on separate cards, place the cards on a bulletin board or wall, and collectively discuss each item.

5. Annex V: Reflection on Organizational Processes of Data Collection and Analysis This exercise provokes a discussion regarding the current actions of the organization in the M&E area. To this end, a discussion is initiated on the organizational processes for data collection, analysis and use of the information collected. Before beginning the exercise, write the following terms on separate cards: data collection, data analysis, changes made based on the information collected, and data storage. Prior to starting the exercise, initiate a discussion among participants using the following questions: 1. What are examples of information that you collected during last year? 2. How did you document the HIV prevention activities carried out during the past year? 3. After collecting the data, where did you store the information? Was it in a confidential place? 4. How did you analyze the information? 5. Did you make any changes in the projects based on the information collected? After discussing each question, ask a volunteer in the group to place the cards on the bulletin board in order of organizational ease and difficulty. The card placed in the upper position represents the easiest, and the last represents the most difficult. Discuss the results with the group based on the questions below. 1. Why did you place the cards in that order? 2. Does anybody have a different opinion? Why?

3. If you were to invest money and time in one of these areas in the next six months, which area would you choose? Why? 4. How do you think you could systematize this process?

6. Annex VI: Creating an Organizational Culture of M&E After the group discussion, distribute the sheet with the exercise below for the participants to fill in. Once the sheet has been completed, discuss the answers with the group. Questions 1.

Do we have a clear system to identify the needs of our population?

Is the system we use to assess the needs of the population efficient?

Do we have SMART indicators to M&E our actions?

Does the population we access directly participate in our M&E activities?

Do we have a quality system for collecting programmatic and operational information?

Do we have a quality system for analyzing programmatic and operational information?

Do we make appropriate changes in project implementation based on the analysis of information collected through process monitoring (of inputs and activities)?

Do we systematically document the lessons learned in our projects?

Yes

7. Annex VII: Examples of Monitoring and Evaluation Tools1 A. In-depth Interview Guide for a Process Evaluation of Prevention Activities with Sex Workers I. Demographic Data 1. 2. 3. 4. 5.

City: _____________________________ Age: _______ Workplace: ( ) Street ( ) Night club ( ) Bar ( ) Other:______________ What is the highest grade you completed in school? ______________________________ How would you describe the color of your skin? _____________________

II. Qualitative Script a. Experience with NGO 1. Tell me a little about the type of contact you had during the last year with (name of the NGO) _______________________________. Probe: What was your first contact with the NGO like? Do you remember who spoke with you? In which activities did you participate? Tell me a little about your experience in these activities. What were the strong points in these activities? What were the points that should be improved? What suggestions do you have to improve the activities? What other types of activities are necessary? b. Project Results 2. Tell me a little about how participation in these activities project affected your life. Probe: use of condoms, human rights, health services sought, HIV/AIDS awareness, etc. Probe: What activities of the project most contributed to these changes? c. Material and inputs 3. Did you receive any educational material from the NGO? Probe: What kind? What type of information did it contain? What do you think of the material? Did the person that handed in the material to you explain to you anything about it? What? 4. In addition to the educational material, did you receive any other material? Probe: Condom or lubricating gel

The examples of questionnaires (In-depth Interview Script for analysis of Implementation of Prevention Activities of Sex Workers, script for Focus Groups for an Evaluation of the HIV Prevention Activities with MSM, KAP Questionnaire, IEC Evaluation Questionnaire) are adaptations of instruments used by the following organizations: NEP, NEPS, GAPA/MG, Population Council do Brasil and Family Health International). 30

Probe: What do you think of the number of condoms that the NGO distributed to you? Was it enough? If not, how many would you like to receive? 5. In addition to the NGO, where else to you usually get condoms and/or gel? Probe: Health posts, pharmacies, other NGOs 6. Do you perceive a difference in the quality of condoms distributed by the NGO and those that you got in other places? Probe: 1. Which do you prefer? Why? 2. Has one of the condoms ever broken? Verify if it was a condom from the NGO and see if they remember the color or brand of the condom 7. What are the difficulties that you notice with respect to the use of condoms among sex workers? What could be done to make the use of condoms easier? 8. For you personally, what is it like to use a condom? Probe: Are there any situations when you find it difficult to use a condom? When? Why? 9. Of the activities that you participated in with (name of NGO), did any of them help you improve your ability to negotiate condom use? Which? What about the activity was the most helpful? 10. Is there any other thing you would like to say or ask? Thank the person and end the interview.

B. Example of Voluntary and Informed Consent Agreement2 The objective of this research project is to evaluate the implementation of the HIV prevention actions supported by (name of the NGO), with (population group) in (name of city). The data will be collected through your participation in an individual or group interview, which will be audio-recorded and will last about 40 minutes. Participation in this study is voluntary and your refusal will not adversely affect you, or cause you any harm. Nevertheless, some personal questions may be a little uncomfortable. You may interrupt your participation at any time during the study, even after the interview has been concluded. This study is confidential and you will not be identified at any time. Neither your name nor those of the people you mention in the interview will be released in the results of the study, thus ensuring your privacy and that of the people you may mention. All material related to you will be kept in locked cabinets at the (name of the NGO) office. The information you provide will be used only for purposes of the study. You are free to access the results of the study. You will be reimbursed for any transportation and food costs, provided that you have incurred in such expenses due to the study. You will receive no compensation for participating in this study. Please do not hesitate to contact (name of the person in charge of coordinating the study) at (telephone number) to clarify any questions you may have with respect to this study. Voluntarily and duly informed, I hereby agree to participate in the above-mentioned evaluation.

________________, ________________, 2006

Name of the participant in the study

Interviewer: _____________________________

Signature of the participant

Contact: ________________________

Contact info of the person responsible for the study: _______________________________________

2 Once

signed, the Informed Consent form should be kept in a separate place from the completed questionnaire/guides in order to ensure the privacy of the participant. It is important to stress that the participant does not need to write in his/her real name, if he/she does not wish to do so. The participant must sign two copies of the Consent Form: one copy is for the participant, and the other is for the person who received the signature. 32

C. Focus Group Guide for a Process Evaluation of HIV Prevention Activities with MSM I. Demographic Data (additional demographic data regarding age, educational level and profession of participants should also be collected individually) Date: ________________________________________________________ Number of participants: ________________________________________ City: ______________________________________________________ Location of the Focus Group: __________________________________________ II. Qualitative Guide a. Experience with NGO 1. Tell me a little about the type of contact you had during the last year with (name of the NGO). Probe: What activities did you participate in? Tell me a little about your experience in these activities. What were the strong points of these activities? What were the points that should be improved? What suggestions do you have of to improve them? What other types of activities should be carried out? b. Project Results 1. For you all, what have been the most important results of this project in the MSM community?. Probe: condom use, human rights, health services sought, HIV/AIDS awareness, etc. 2. Tell me a little bit about how participation in the project activities affected your lives. Probe: What activities of the project most contributed to these results? c. Material and inputs 1. Did you receive any educational material from the NGO? Probe: What kind? What type of information did it contain? What do you think of the material? Did the person that handed out the material to you explain anything about it? What? 2. In addition to the educational material, did you receive any other material? Probe: Condom or lubricating gel Probe: What do you think of the number of condoms that the NGO distributed? Was it enough? 3. In addition to the NGO, where else to you usually get condoms and/or gel? Probe: Health posts, pharmacies, other NGOs 4. Do you perceive a difference in the quality of condoms distributed by the NGO and those that you got in other places? Probe: 1. Which do you prefer? Why? 2. Has one of the condoms ever broken? Verify if it was a condom from the NGO and see if they remember the color or brand of the condom 33

5. What are the difficulties that you perceive in terms of condom use among MSM? What could be done to make negotiating condom use easier? 6. Of the activities that you all participated in with (name of NGO), did any of them help you all improve your ability to negotiate condom use? Which? What about the activity was the most helpful? b. Stigma and Discrimination 7. What is it like to be a MSM in (name of the city)? Probe: Do you think it is easy or difficult? Why or why not? Are you open with others regarding your sexual orientation? Why or why not? 8. Have you ever felt discriminated against due to your sexual orientation? If yes, in what situation(s)? 9. Have you already participated in any NGO activity on the subject? Which one/ones? What do think of those activities?

Is there anything you would like to say or ask? Thank the participants and close.

D. Sample - Questionnaire for sex workers

Questionnaire number: ___________________ Interviewer Code: ___________________ Interview Date: _________________________ Neighborhood: ___________________________________

Location of the interview:

Street Night club Sauna Bar Other(Specify):___________________________

How old are you?

__________________________ (in years) Does not know

What is the highest grade you completed in school?

Incomplete Elementary School Complete Elementary School Incomplete High School Complete High School College/ University + Does not know

Do you have children?

Yes How many? _______________ No Does not know

How would you define your skin color?

Black White Indigenous Mixed Other: ________________________ Does not know

How may days a week do you work?

0 1-2 3-4 5-7 Does not know

How many dates do you have in a normal day of work?

1 2 3 4 5+ Does not know

How much do you make a month as a sex worker?

$10-$300 $301-$600 $600-$900 $900 + Does not know

Where were you born?

City in which the person is being interviewed Another city: _____________________ Does not know

What is your marital status?

Married Single Separated Widow Living with someone Does not know

How long have you worked at this point/place?

_______________ (Specify day, month, year) Does not know

Let's talk a little about HIV/AIDS now. For the following questions, you may answer yes, no or I don't know. 12

Do you think that a person can get HIV: A. through a kiss?

Yes

Does not know

B. by hugging?

Yes

Does not know

C. In a public restroom?

Yes

Does not know

D. Drinking out of the same glass as a person with HIV/AIDS?

Yes

Does not know

E. Working with someone who has HIV/AIDS?

Yes

Does not know

F. Practicing anal sex without a condom?

Yes

Does not know

G. Practicing oral sex without a condom?

Yes

Does not know

H. Practicing vaginal sex without a condom?

Yes

Does not know

I. By a mosquito bite?

Yes

Does not know

J. Sharing used needles?

Yes

Does not know

K. From mother to child during pregnancy, delivery or breastfeeding?

Yes

Does not know

HHv 13 May a person with a healthy appearance be HIV positive?

Yes No Does not know

GD 14

If a person is diagnosed with HIV, where could he/she seek care and treatment?

Public health facility Pharmacy Public hospital Private clinic Other: __________________________ Does not know

Would he/she have to pay for care or treatment?

Yes No Does not know

Have you ever been tested for HIV?

Yes No Does not know

The last time you had sex with a new client, did you use a condom?

Yes skip to # 19 No Does not know

(If a condom was not used) Why did you decide not to use a condom?

I didn't have one Heat of the moment Trustworthy partner Use of alcohol or other drugs Client request/demand Other _____________________ Does not know

Do you have a steady client?

Yes No skip to # 22 Does not know

The last time you had sex with a steady client, did you use a condom?

Yes skip to # 22 No Does not know

(If a condom was not used) Why did you decide not to use a condom?

I didn't have one Heat of the moment Trustworthy partner Use of alcohol or other drugs Client request/demand Other _____________________ Does not know

Do you have a boyfriend or husband?

Yes No skip to # 25 Does not know

The last time you had sex with him, did he use a condom?

Yes skip to # 25 No Does not know

(If a condom was not used) Why did you decide not to use a condom?

I didn't have one Heat of the moment Trustworthy partner Use of alcohol or other drugs Client request/demand Other _____________________ Does not know

Now we are going to talk a little about your life here in (name of the city). 25

Are there any people in your day-to-day life that do not know about what you do?

Yes No skip to # 27 Does not know

If you were to tell them, what would be their reaction?

They would have no problem in accepting the situation They would avoid me They would reject me Other: ________________________

Have you ever experienced prejudice or discrimination due to your profession?

Yes No skip to # 29 Does not know

Where did that experience occur?

Street Health facility School Police Station Other: ________________________

Now we are going to talk a little about HIV educational activities. 29

Within the last three months, did you participate in any HIV/AIDS related educational activity?

Yes No end the interview Does not know

In which of the following activities did you participate?

Talk with a peer educator Theatre presentation Workshop Training Received condoms Focus group Other_____________________________

Within the last three months, how many times did you participate in each activity?

Talk with a peer educator _________________ Theatre presentation ____________________ Workshop ____________________________ Training ______________________________ Received condoms ______________________ Focus group ___________________________ Other________________________________

Do you know the name of the NGO/project that organized these activities?

Yes What?____________________________ No Does not know

What do you think of the activities in which you participated?

________________________________________ ________________________________________ ________________________________________ ________________________________________ ________________________________________

Annex E: Example of Field Work Spreadsheet FIELD WORK APPROACH MAP Peer educator: Date: Population group: Total number of contacts:

Location: Total number of first contacts:

Total number of regular contacts:

1) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder ( ) yes ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

Condom use Health services

Human Rights

( )

Testing services

2) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder ( ) yes ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

Condom use Health services

Human Rights

( )

Testing services

3) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder ( ) yes ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

Condom use Health services

Human Rights

( )

Testing services

4) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder yes ( ) ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

Condom use Health services

Human Rights

Testing services

5) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder yes ( ) ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

6) Contact: 1st contact ( ) regular contact ( )

Material distributed: Condom: primer folder ( ) yes ( ) ( ) postcard ( ) no ( )

Subjects discussed: STI/AIDS ( ) Health in general ( )

Human Rights

( )

Suggestions and field observations:

( ) ( ) ( ) Suggestions and field observations:

Use of condoms ( ) Health services ( ) Testing services ( ) Suggestions and field observations: Use of condoms ( ) Health services ( ) Testing services ( )

F. Questionnaire for IEC Material Evaluation I. Ask the following questions for EACH PAGE of the material: 1. Are there any words that you do not understand? Which ones? Can you explain why their meaning is not clear? 2.

Is there anything on this page that you do not believe is true? What?

3. What about the images on this page? Tell me what you see. (Where applicable, ask "What place is this? What type of person is this?") 4. Is there anything on this page or in the images that may offend or embarrass someone? What? What would be better? 5. Is there anything on this page that is confusing? What? How could it be clearer? 6. Is there anything on this page that you really like? What? 7. Is there anything on this page that you really do not like? What? II. Ask the following questions on the material AS A WHOLE: 1. Do you think that the material is suggesting that you do something specific? What? Do you think you would do it? Why or why not? 2. Do you think the material was developed/ created for people like you, or for another type of person? Why? 3. What do you think can be done to improve the material?

G. Evaluation Questionnaire - M&E Workshop3 Score: 1 = Totally disagrees 2 = Partially disagrees 3 = Partially agrees 4 = Totally agrees I. General Perception I learned new things at the M&E workshop. 1

The workshop was useful. 1

My expectations were met. 1

I enjoyed the workshop. 1

II. Facilitators The presentations were clear. 1

The facilitators encouraged participative and collective discussions. 1

The facilitators gave clear answers to the questions asked. 1

III. Material/Information supplied The material was clear and easy to understand. 1

The material was appropriate and relevant to the needs of my organization. 1

The material was complete. 1

This evaluation form is based on the evaluation form used in the Monitoring, Evaluation, and Reporting Workshop for Pact Vietnam partners in January of 2006. 42

IV. Learning I had the opportunity to effectively participate. 1

The three most useful things I learned were: 1. 2. 3. I would have liked to discuss the following subjects in greater depth:

In the future, I would recommend that the workshop...

Other comments:

8. Annex VIII: Example of Schedule - M&E Workshop I. Day one Welcome/Introduction Defining Monitoring and Evaluation: What is M&E? Why is it important? (presentation of slides, collective exercise, and discussion) Break Indicators and how to use them in M&E Part I (presentation of slides and collective discussion) Lunch Indicators and how to use them in M&E Part II (presentation of slides and collective discussion) Break M&E Plan (presentation of slides and introduction to group activity) Evaluation of day one of activities II. Day two Creating an M&E Plan: Group activity Break Monitoring and Evaluation Tools (slide presentation and discussion of tools) Lunch How to Conduct a Project Evaluation (presentation of slides and collective discussion) Break Creating an Organizational Culture of M&E (presentation of slides, exercise, and collective discussion) Evaluation of the workshop (Evaluation instrument is filled in; group discussion)

Pact Brasil Alameda Santos, 1893 Cj. 91 01419-002 S達o Paulo, SP Brazil www.pactbrasil.org