ROPE II: Rights Oriented Programming for Effectiveness

ROPE II

Rights Oriented Programming for Effectiveness Designing and Evaluating Long-Term Programs

LEAD Learning, Evaluation & Accountability Department December 2008

Compiled by the Learning, Evaluation, Accountability Department (LEAD) of Oxfam America.

LEAD would like to make specific mention of Jagabandhu Acharya and John Ambler whose revision of ROPE I formed the foundation of this document. We also wish to thank the staff in all seven regions of Oxfam America who have produced important lessons about how to craft long-term programs. It is through their hard work and commitment that this document is possible. Finally, we wish to thank Regional Program Department (RPD) members in Boston who have worked hard to focus and sharpen this set of ideas.

The current staff of LEAD are Allison Davis, Kent Glenzer, Adinda Van Hemelrijck, Kimberly Miller, Bridget Leigh Snell, and Gabrielle Watson, all of whom contributed to ROPE II. *This version updates and replaces ROPE 1.0 (May 2006).

TABLE OF CONTENTS

A rope symbolizes flexibility, synergy and strength – twisting the individual fibers together multiplies their strength. The ROPE framework symbolizes the weaving together of a number of diverse yet coordinated actions and actors at different levels into a program that is strong, yet flexible enough to produce impact in a dynamic world.

Jagabandhu Acharya, former LEAD Director

Overview of This Document…………….………………………………………………………………...ii 1.

Introduction…….………………………………………………………………………………….1

2.

Principles Of A Rights-Based Programming Approach………………………....……………1

3.

Program Design….……………………………………………………………………....……....3

4.

Program Monitoring, Evaluation, Learning………………...…………….……………...….....7

5.

Management Roles and Responsibilities for External Impact Evaluation……..………….10

6.

GLOSSARY…………..………………………………………………………………………….12

Overview of This Document Purpose This document provides a strategic overview of how Oxfam America approaches long-term, rights-based development. It is not a how-to or detailed guide. However, as you read through this document, you will find certain terms or phrase highlighted in blue. This indicates that tools, advice, and detailed guidance are available from LEAD, or will be available soon.

For additional information on this detailed guidance, please contact Kent Glenzer, Director of Oxfam America’s Learning, Evaluation, and Accountability Department, at kglenzer@oxfamamerica.org or Muthoni Murìu, Director of the Regional Program Department, at mmuriu@oxfamamerica.org.

Intended Audiences The primary audience for this document is Oxfam development and humanitarian programming staff.1 The document should be shared with external stakeholders, supporters, and constituencies. We wish to be transparent about our work, and this document gives others one means to hold us accountable.

1

Oxfam America’s Humanitarian Response Department has more detailed guidance regarding norms, standards, and good practices with regard to rapid response, rehabilitation/reconstruction programming, and long-term resilience building and disaster risk reduction. Contact Michael Delaney for more information (mdelaney@oxfamamerica.org).

ii

1. Introduction Oxfam America seeks to create lasting solutions to accountable. People, we believe, can sustain poverty and injustice. This requires working at positive gains through self-organization, multiple levels to accomplish our mission. We advocacy, and continuous engagement with duty achieve significant and lasting changes in people’s bearers on policies and practices that promote lives through influencing global and national both growth and equity. For this reason, we institutions, policies, and practices. We combine commit to long-term engagements with specific this with work at local levels to disadvantaged groups, and build lasting We understand poverty as change power relations that coalitions and strategic partnerships a problem of power, one are at the root of human rights around clearly identified rights issues. in which people are denial, exclusion, and deprived of opportunities, poverty. resources, public ROPE (Rights Oriented Programming for services, knowledge, and Effectiveness) considers people whose protection.

To create lasting changes rights are being denied as active, capable means we must address the agents, not victims passively awaiting help. It persistent causes of poverty and marginalization. shifts our attention from short-term projects to longWe understand poverty as a problem of power, term programs geared towards achieving impact at one in which people are deprived of opportunities, scale and which lasts without Oxfam America’s resources, public services, knowledge, and continued support. These programs mobilize the protection. Furthermore, they are systematically resources and commitments of others. ROPE excluded from full participation in decision requires us to work together intimately and in a longmaking. Oxfam America’s program approach term manner with community, national, and seeks to address policies, conditions of exclusion international level organizations because the kinds of and discrimination, and above all to empower changes we seek cannot be achieved by Oxfam people themselves to hold powerful actors America alone.

2. Principles of a Rights-Based Programming Approach A rights-based approach seeks to create the conditions for people to live in dignity and fulfill their inherent human potential. The following principles reveal how Oxfam America puts this ideal into practice: Putting people at the center of our work We listen to, and engage as equals with, people living in poverty. We build programs on their existing ideas, capabilities, aspirations, and capabilities. We adopt participatory approaches and seek ever greater levels of authenticity with those approaches. We uncover people’s own ideas of success and build strategies to both achieve them and measure them. We consider people living in poverty with whom we work as

1

primary change agents, and not as passive participants in our programs. This reflects our belief in and commitment to the capabilities and power of people themselves to change the conditions of their own disempowerment, exclusion, and vulnerability. Women and girls are of particular importance to Oxfam America as primary change agents.

Understanding context Our programs are built upon a sophisticated understanding of local context. There are no universal magic bullets – programs are unique, sequenced, long-term sets of actions suited to specific changing situations. We build on demonstrated good practice, adapted locally.

Focusing on leverage points Deep social change is complex, with many forces at work. Yet Oxfam cannot do everything. We carefully choose the most powerful leverage points in the social system. These produce multiplier effects that the same investment, placed elsewhere, would not achieve. Targeted small investments can result in very large change.

Measuring changes, immediate and long-term Our programs identify long term, measurable impacts, and more immediate and intermediate benchmarks, that lead toward those impacts.

We use evidence-based monitoring to measure progress, and do so through both quantitative and qualitative methods.

Working at multiple levels While poverty is seen at the household, group, and community levels, the root causes are often higher. We work at multiple levels, using policy advocacy at national and ROPE global levels.

Promoting knowledge sharing and learning Our programs have monitoring and evaluation systems that enable us to continually learn and improve. We learn as much from our setbacks as our successes. We examine Principles evidence to understand the 1. Putting people at the center dynamics of change and adjust 2. Understanding context 3. Focusing on leverage points our strategies. Most of all, we 4. Working at multiple levels engage people living in 5. Influencing resource poverty, themselves, in these commitments of others 6. Building strategic partnerships learning processes. We find 7. Making long-term new ways to empower them to commitments take a greater role in both 8. Measuring changes, immediate and long term defining “success” and 9.

Promoting knowledge sharing determining if Oxfam’s work has and learning made a difference in people’s 10. Building shared accountability 11. Reflecting on Oxfam’s own lives.

Influencing others’ resources, smartly joining our resources with others The changes we seek are beyond Oxfam’s sole ability to achieve. We influence others to join our efforts. We craft programs in smart ways that contribute Oxfam’s resources to wider efforts and do not act position alone or in a redundant manner. Our own interventions must add value to what others already do.

Building shared accountability We practice the best standards in accountability.

We are accountable, first and foremost, to the people we serve. Next, we are accountable to our partners and allies, to our donors and supporters. We also must be true to our Vision, Values, and Purpose.

Building strategic partnerships Addressing root causes of poverty and rights denial requires us to build strategic partnerships. This means building stronger and longer-term relationships with select partners with whom we share a vision for change. Such partners commit to work together and invest required inputs to achieve change.

Reflecting on Oxfam’s own position We adopt and advocate for new and more equitable relationships between those whose rights are denied, ourselves, other international NGOs, local partners, “experts,” donors, and government entities. Programs require us to constantly challenge ourselves about our own power, our own ideas of success, and our own assumptions about what development is.

Making long term commitments We recognize that a commitment to work on the root causes of poverty is a long-term undertaking.

Our programs make 10-15 year commitments to work with, and for, a specific group of primary change agents.

2

3. Program Design Oxfam America’s approach to programs is defined by what we mean by impact. It is this definition that requires us to become more strategic in our grant-making and partnering practices, engage more in policy advocacy, and devise new measurement approaches and systems.

3.1 Definition of Impact We define impact as a significant and sustainable change in power relations enabling people to assert their rights and to gain access to and influence over the knowledge, resources, services, and decision-making necessary to live in dignity. Such impact is measured at the level of the people themselves, in the concrete changes they experience in their lives, relationships, rights, and conditions. Impact is not impact unless it is felt by people themselves, and recognized as such by them. Programs – designed, implemented, and evaluated as described in this document – are Oxfam America’s primary vehicles for achieving impact.

and knowledge sharing strategy, and a core group of actors who meet regularly to assess performance, change strategy, and (re)allocate resources. Importantly, primary change agents themselves play a prominent role in such thinking and decisions. As a result, a program requires approaches that facilitate their active engagement and continuous, mutual learning.

3.3 Program Design Elements A. Who are the “Primary Change Agents”?

Every program clearly defines a set of primary change agents, the disadvantaged social group(s) who will experience and create, with OA and our partners, the very impacts we aim to achieve. A program identifies primary change agents in terms of where they live, their numbers, and the specific forms – and root causes – of rights denial they experience.

Key here is that primary change agents have a common set of interests, and might be organized and mobilized together around shared goals.

3.2 Definition of a Program A program is a set of strategically aligned, mutually reinforcing interventions – some carried out by Oxfam America, but most by others – that create sustained, positive changes in both material conditions and social positions of a specific group of vulnerable people. At the strategic planning level, a program has a measurable 10-15 year impact goal, an explicit theory of change, a planned sequence of impact benchmarks that relate to the theory of change and lead to achievement of the impact goal. Programs are the umbrella under which shorter term work -- Oxfam grants to partners, policy advocacy and campaigning, day-to-day networking and relationship building, projects funded by foundations, etc. -operate. A program aligns, and connects together, shorter-term work through a common set of indicators and measures, a learning

Ideally, program design includes primary change agents from the outset. This enables us to understand diversity within the group, how they view and cope with rights denial, and how they aspire to realize their rights. A program must lead to concrete results for this group but in a way that enables them to take centerstage in a conscious, organized, and assertive manner.

B. How does power operate?

A power analysis identifies specific actors and the discrimination, control of resources, and life opportunities they experience or force upon others. It takes special note of gender. It focuses on both the visible manifestations and underlying root causes of the rights denial in question. A power analysis and institutional mapping exercise in the design stage reveals key

3

information on the actors and the power dynamics among them and how these could affect the changes we seek. This analysis enables a better definition of the role(s) Oxfam can play.

A power analysis can put people at risk if done insensitively, but we must do it nonetheless.

Ideally, power analysis is done in a participatory manner with primary change agents, partners, and experts such as anthropologists, political scientists or gender specialists.

struggle forward.

D. What’s our theory of change?

Context and power analysis allows us to understand how a social system is currently working. A theory of change is our best guess Power: Five Kinds about how the social Power Over: The ability to make others do system can evolve things, even against their will. between now and 10-15 years from now. A theory Power Behind (Hidden Power): The ability to of change makes explicit, shape the terms of debate, what is discussible, permissible. therefore, the sequence of social changes needed to Power With: Joining with others to arrive at our program accomplish things otherwise impossible. impact goal. In this way, the theory of change Power Within: The capacity to imagine and have hope, self-respect, self-worth. provides us with the short, medium, and long-term Power To: Unique potential of every changes that our individual to shape his or her life and world. monitoring, evaluation, Sources: Lisa VeneKlasen and Valerie Miller, A New and learning (MEL) Weave of People, Power, and Politics: The Action Guide for Advocacy and Citizen Participation, Sterling, VA: systems need to focus on.

C. What is the impact goal?

A program achieves its purpose if it makes a significant improvement Stylus Publications, 2007. John Gaventa, “Finding the in the power relations Spaces for Change: A Power Analysis,” IDS Bulletin Volume 37, Number 6, November 2006. that lead to rights The theory of change is fulfillment. The impact based on expert has to be significant and of a scale that can make knowledge, informed assumptions, and research. an observable difference. Every program We understand that long-term lasting changes in establishes a measurable 10-15 year impact power relationships can only happen when goal, one that specifies the numbers and types of multiple factors coincide to support and sustain primary change agents who will experience that change. A theory of change will typically change and the specific changes in power include changes in elements such as policies, relations and their livelihoods that the program institutions, people’s beliefs, public opinion, aims to achieve. Impact goals are changes to service access, and control/influence over which OA and its partners will contribute, and we strategic assets and resources. Those elements build rigorous yet simple measurement that are the most critical drivers of change in the systems to track and verify that contribution. As program’s context need to be included in a theory a result, an effective impact goal is an answer to of change. the question, “what are the social changes that OA will help catalyze?” rather than to the A theory of change is a hypothesis, and is question, “what results will OA alone be constantly tested, challenged, and improved accountable for?” Impact goals are also based on through our monitoring, evaluation, research, and the idea of sustainability. By sustainable we learning systems. Theories of change will evolve mean the impacts will last beyond the time frame as we and our partners learn. And as we learn of the program and that we have helped primary about whether or not power relations are truly change agents develop the capacities to take their changing, we revisit the theory of change,

4

improve it, and then rethink the strategies and tactics we are employing, moving forward.

one actor/partner among others, our leadership should reflect respect for our partner’s opinions, while at the same time recognizing opportunities to support our partners and strengthen their capacities. We also learn from others’ experience, expertise, and opinions, and cede leadership when required.

E. Who are our Strategic Partners?

The kinds of changes that OA’s programs seek cannot ever be accomplished by Oxfam America alone. Our programs bring together the required set of partners – and their resources – to address the impact goal credibly. These partners form the 4. Mutual accountability and transparency Program Working Group (PWG). The PWG Equal participation implies mutual accountability. consists of partners who share a vision of change We are transparent in the way we engage with our and a commitment to working together to bring partners – including the people we serve and work that change about. Strategic partners co-plan, with, and other key stakeholders. Our relationship co-strategize and commit to learn together. They implies a mutually enforced commitment to deliver contribute resources to achieve the goal. The and respond to our responsibilities in an efficient, PWG may include government bodies, other large accurate and timely way. international NGOs, social movements, the private sector, research 5.

Mutual learning Strategic Partnership institutions, and the Oxfam, as a learning Principles media, in addition to organization, promotes 1. Independence/autonomy small, local NGOs. Over continuous and systematic 2.

Shared vision time, this coalition may learning. We share 3. Equal partnership change and evolve as the responsibility for learning 4. Mutual accountability and transparency program enters new with our partners and use 5. Mutual learning phases and encounters collective reflection and 6. Relevance and flexibility new challenges. analysis to jointly make evidence-based decisions.

Six partnership principles inform our programs: 6.

Relevance and flexibility 1. Independence/autonomy of our partners We construct strategic partnerships purposefully, In all our relationships, we maintain our own based on the program’s requirements. As viewpoint and look for independent partners – with programs unfold, partner and OA roles and their own views and priorities. Our various responsibilities will evolve. perspectives enrich each other’s thinking.

F. How do we move from broad and long-term 2. Shared vision goals and partnerships to shorter-term A shared vision of the program goal needs to be planning, action, and learning? adopted by all strategic partners. The people we Long-term programs are made more concrete, serve and work with should share this vision. To measurable, and operational via the identification this purpose, we have to build mutual trust and of 1) impact benchmarks, 2) strategies with the capacity to work together and create common associated objectives, and 3) explicit learning cause. mechanisms and processes. Impact benchmarks are identified for three, six, and nine year periods.

3. Equal participation Impact benchmarks are built off the program’s We recognize our partners as co-strategists in theory of change and tell us, therefore, how we reflection and decision-making processes. As only are going to measure the social changes – and

5

changes in people’s livelihood conditions – that the theory of change predicts. Program Implementation Plans identify a program’s key deliverables, activities, partners and grants, and connections between short-term outputs and outcomes and longer-term impact for a period of three years.

change, strategic objectives, impact indicators, 3-, 6-, and 9-year benchmarks (i.e., shorter-term outcomes that lead towards the ultimate impacts desired), strategic partners needed, and the broad strategies that the program will pursue in order to make desired changes happen. The PSP also identifies the specific learning processes, mechanisms, and calendar that will be implemented. The length of the internal document is variable. The writing of the document itself should not become an overly time-consuming and difficult activity. We want to stress good quality thinking, strategizing, and ongoing learning rather than writing skills.

H. How much does a program cost?

Every program needs a budget that is tied to a) three-year impact benchmarks for the particular planning period and b) annual work plans. As strategic partnerships are forged, these budgets include all resources, not just Oxfam’s. However, a minimum of 7% of the annual budget needs to be set aside for MEL purposes.

Five-page and one-page summaries are needed for external use. PSPs should be revised at least every five years although modifications to strategy, objectives, and our theory of change should be discussed yearly, in the context of annual planning and budgeting.

I. How do we document a program?

A Program Strategy Paper (PSP) documents the context and power analysis, description of primary change agents, the impact goal, the theory of

Key Elements of Program Design in ROPE 1. Understanding of primary change agents: Who they are, where they are, how many, and their common interest with regard to rights-based impact.

2. Context and power analysis: What prevents people from rights fulfillment?

3. 10-15 year impact goal: Specifies the numbers and kinds of primary change agents who will experience change, and the specific changes in power relations and livelihoods, that will be achieved.

4. Theory of change: A hypothesis to be continuously tested. Makes explicit the sequence of social changes which will lead to the program impact goal.

5. Strategic partners: Who else do we need to make change happen? What other resources, expertise, and experiences does the program need to succeed?

6. 3, 6, and 9 year impact benchmarks: Associated with the theory of change, impact benchmarks show planned progress towards the impact goal.

7. Strategies and objectives for achieving benchmarks: This level of program design sets out the broad set of strategies and associated strategic objectives that will lead to achievement of impact benchmarks 8. Three year implementation plan: Identifies more specific strategies, objectives, activities, grants, impact research, learning processes and products, training needs, and budgets that will lead to accomplishment of three year impact benchmarks. (see Section 4 below for more).

6

4. Program Monitoring, Evaluation and Learning (MEL) We face an important accountability challenge. It is essential that staff, partners, and primary change agents create their own learning processes and engage in sustained critical reflection about what is working and not. Yet, if an accountability system is built only upon internal reflection we risk unacceptable bias and miss the opportunity to gain fresh views and new insights. Oxfam America’s approach to MEL is, therefore, based on balancing internal critical reflection and external evaluators who challenge this perspective. In ROPE, MEL is meant as much to reveal what’s not working, what needs to improve, as it is to show worthy results. The system is intended to gradually position OA and its partners as thoughtful, transparent, and effective organizations that have good data and systems for thoroughly understanding the impacts of its work.

actor. Every program needs an explicit approach and mechanisms for building such accountability. Such a system must be built by committed staff, partners, and people from the bottom up. A crucial element of our accountability is the engagement, from time to time, of reputed external evaluators or researchers who provide program teams with an outside perspective on program successes, challenges, impacts, and strategies.

4.2 Commitment to Participatory Learning Oxfam America is committed to being a learning organization. Organizational learning refers to a set of practices, tangible mechanisms and systems that help us (a) be aware of and use good demonstrated practices, (b) harvest and disseminate new practices/innovations, (c) make course corrections based on new information, and (d) tell our impact story. Every program needs to be explicit about pursuing these requirements.

MEL systems can be quite different depending upon context, the program, and resources. Any system must, however, include explicit indicators for changes in power relations and improvements in rights realization, be clear on how such data will be gathered, and be feasible within the resources available. Importantly, the system has to be comprehensible to our partners and people we work with. This requirement also speaks to the need to keep MEL systems simple, focused, and flexible.

We are also committed to actively engaging primary change agents, partners, and other key stakeholders to define success and validate results. Participatory MEL processes bring different kinds of actors – with different kinds of power and perspectives on change – together to forge common cause.

4.3 Commitment to Rights-Based M&E 4.1 Commitment to Accountability Oxfam America is committed to the highest standards of monitoring, evaluation, learning and accountability. Oxfam America believes it is accountable both for impacts and for our compliance with agreed standards of resource and relationship management. Accountability to the people we serve is our vision, the starting point for our conversations about our duties and responsibilities as a rights-based

7

Whether for a project, a program, or something else, evaluation approaches can take many forms. ROPE calls for rightsbased monitoring and evaluation. M&E is suitably rights-based if it: a. makes changes in power relations explicit and evaluable

b. builds the capacities of primary change agents and partners to think critically about the changes going on around them c. affords primary change agents and partners a real opportunity to hold Oxfam and other partners accountable d. includes specific public transparency mechanisms and processes with regard to disseminating evaluation findings

i.

j.

impact research, or special studies Publish an annual summary of program successes and challenges and collaborate in telling the program’s impact story to external audiences Collaborate on fundraising

B. External Research Institute Every program will build an ongoing relationship with a local/regional research institute. The purpose of this relationship is to a) provide mechanisms through which program staff are challenged about strategies and monitoring and evaluation data, b) channel to the program sophisticated, academic, or expert thinking and good practices from elsewhere, and c) provide objective, external impact research periodically.

4.4 High Level Accountability and Learning Mechanisms A. Program Working Group Program Working Group (PWG) is the coalition of major strategic partners who, together with Oxfam and representatives of the primary change agents themselves, share a commitment to work together toward a common impact goal.

We recognize that such a group is not easy to achieve. While mature rights-based programs at Oxfam should all have a PWG, it may take a number of years to achieve it. The composition of a PWG will also naturally evolve over time, as the program enters new phases and addresses new opportunities and challenges. In any case, once established the functions of this group would typically include: a. Build a shared vision and agenda for positive change b. Discuss external trends, studies, and what other actors are doing c. Agree on norms, standards, and approaches to ensure participatory inquiry and reflection throughout the life of the program d. Review outcome and impact data and determine course corrections or major shifts in strategy and the theory of change e. Conduct annual planning exercises and agree on shorter-term (3-5 year) objectives and the outcomes and impacts these are expected to contribute to f. Discuss each other’s strengths and areas for improvement g. Make good use of the wider networks of influence that each partner has h. Agree on and commission external

C. Program Advisory Committee Some programs may wish to establish a Program Advisory Committee (PAC), although this is not a formal requirement. A PAC consists of external experts and opinion leaders who advise the PWG. A PAC may be very desirable in contexts where stakes are very high, where evidence of impact is likely to be controversial and contested, where OA’s objectivity is questioned, or where policy advocacy goals are assisted by such strategic connections.

4.5 Strategic Monitoring Evaluation and Learning (MEL) System Strategic monitoring, evaluation and learning (MEL) systems focus on programlevel impacts and outcomes. Ideally, it will be developed by the Program Working Group.

Such systems help us look at and understand the combined effects of multiple interventions over the life of the program. Strategic MEL is tied directly to the impact goal, theory of change, and impact benchmarks. As such, it focuses strongly on changes in power relations and the progressive realization of people’s rights. It allows us to track progress

8

4.6 Program Impact Baselines

against our goal, on the critical elements of the theory of change, and to analyze how the theory of change itself needs to be modified, as we learn more about how power operates.

Every program needs an impact baseline against which change will be measured.

Programs can conduct baseline research at different times but it usually will happen in the first year or two of implementation.

Programmatic baselines will differ in scope, complexity, breadth, and depth depending upon resources and the nature of impacts sought.

The strategic MEL system identifies a) a small set of impact indicators we will track over the coming 10-15 years, b) shorter term changes expected by years 3, 6, and 9 (impact benchmarks) that allow us to track progress on impact indicators and on the theory of change, c) who is responsible for gathering and analyzing monitoring and evaluation data, and d) the learning products that will be regularly published and disseminated.

The nature of a program baseline is different from a standard, project baseline. While both establish a present state of affairs against which future change is measured, program baselines are meant to be:

MEL data come from five sources: 1. From partner reports on grants. Such grants incorporate and require reporting on appropriate impact indicators and benchmarks.

2. From policy advocacy and campaigning efforts. Most programs need some kind of policy advocacy to achieve significant change in power relations. This work, like grants, needs to be explicit about which indicators and benchmarks it aligns with and reports on.

3. From research conducted by others.

Other studies may shed light on progress or on our theory of change. This can save OA and its partner’s time and resources.

4. From annual research organized and run by research partners. Every year, a program will commission external research. This research will focus on impacts and benchmarks and on testing fundamental aspects of our theory of change.

5. From internal reflective practice done

by OA staff, partners, and people living in poverty themselves as part of their ongoing collaboration.

1. Strongly suggestive of the current state of important indicators, relationships, and social dynamics without pretending to be definitive studies. Program baselines strive for a level of precision that is realistic and achievable. Changes sought in programs are complex and multi-dimensional and difficult to measure precisely. Our baselines therefore seek to achieve an acceptable balance between certainty and uncertainty while providing a plausible basis for discussing change in the future.

2. Focused on contribution, not attribution. Baselines and strategic MEL both should be focused on helping OA and its partners determine plausible contribution towards changes and guide us to continuously improve that contribution. Classic, social science attribution is not the aim.

4.7 Program Implementation Plans and Operational Monitoring, Evaluation, Learning Programs and strategic MEL systems must be translated into more concrete implementation plans, and connected explicitly with OA’s

9

grants and grant-level monitoring and evaluation. We do this through the use of Program Implementation Plans (PIPs) and Operational MEL systems.

A. Program Implementation Plans A PIP is a management tool, a three-year plan that specifies the results that the program will achieve, strategies and activities that will lead to those results, staff and partner training and development required, resources needed, learning mechanisms and products that the program staff and/or working group will engage in and produce, research needed and its timing, reporting requirements and timelines, and other operational details. The PIP operationalizes the strategic MEL system in the day-to-day management of the program.

B. Operational MEL Systems An operational MEL system identifies: a) which outcomes will be achieved through which grants and other OA-led activities b) how we will monitor grant performance c) how primary change agents will be included in the MEL system d) how the program will be accountable to primary change agents e) which special studies will be conducted, who will conduct them, and the resources needed f) how we will analyze MEL data g) yearly learning products (reports, studies, white papers, etc.) that we will release Budgeting for MEL is crucial. Every year, seven percent of a program budget should be set aside for MEL. This is above and beyond the resources for M&E that are included within every grant.

5. Management Roles and Responsibilities for External Impact Evaluation External impact evaluations will happen periodically during a long term program. At the end of benchmark periods (the PIP period, that is, every three years), the PWG will wish to do a more comprehensive and external evaluation of progress against the baseline.

Such studies represent a more significant investment of funds (10% of budget) and are a crucial mechanism for assessing performance and being accountable to others.

Budgeting for evaluations is the responsibility of (a) Regional Directors, (b) the Director of RPD, and (c) the Senior Vice President of Programs in Boston, who is ultimately in charge of annual budget allocations.

5.2 Terms of Reference Program impact evaluation Terms of Reference are drafted by the PWG, reviewed

by LEAD, and approved by a Regional Director.

5.1 Evaluation Budget Adequate resources have to be budgeted for such periodic evaluations and have to be built into each program’s budget. Every year, seven percent of a program budget should be allocated for MEL. In years three, six, and nine of a program (coinciding with PIPs), 10% should be allocated for more formal impact evaluation done by external researchers.

10

5.3 Evaluating Agency For ensuring credibility and objectivity, program impact evaluation is carried out by external research agencies, ideally through long-term institutional collaborations. The main reason behind our preference for institutions rather than individual consultants is to ensure greater independence and

credibility of the research findings, to maintain continuity and stability of evaluative research on a continuous basis, and to build local research capacity. The composition of the evaluation team is drafted by the PWG, vetted

by LEAD, and signed off by the Regional Director.

5.4 Transparency and Management Response Once completed, the evaluation requires a management response. This is the responsibility of the Regional Director. The response will include any disagreements with the findings and any changes that will be made to the program. All evaluation reports will undergo a brand-risk assessment before being made public. This, too, is the responsibility of the Regional Director, with help from communications staff in Boston. In

any case all impact evaluations must be made public both locally and globally in a form and media that allow all major stakeholders to have information about the quality and results of OA’s work. Such criteria cannot be mandated from above in all of OA’s operating environments. At regional and country levels, the senior OA program staff member connected to the program is responsible for transparent sharing of evaluations. LEAD is responsible for this in Boston, in collaboration with RPD and communications.

5.5 Program adjustments Program Officers are responsible for implementing agreed strategic shifts in program interventions, based on the evaluations and as agreed by the Regional Director. Such strategic shifts, and the assessment of their effectiveness, are documented as part of the overall evaluation strategy.

11

6. Glossary Accountability Accountability is the obligation of a change agent to demonstrate that impacts sought are measured and that the work has been conducted in compliance with agreed norms standards – with respect to good management of resources, knowledge, and relationships. Accountability to primary change agents is the most important kind of accountability.

Evaluation Evaluation involves a comparative analysis of what was hypothesized vs. what actually occurred.

It requires a baseline – it can be qualitative – against which change is assessed. It requires, too, some kind of comparison group, and there are multiple ways to achieve this. Finally, it is also an inquiry into unintended results. It involves an assessment of performance in relation to stipulated standards of quality and accountability. It needs, too, to look at cost efficiency, cost effectiveness, and return on investment. See Monitoring; Evaluation, Project; Evaluation, Program; Evaluation, Rights-Based.

Evaluation, Project A project, as defined here, refers to a contractual obligation that Oxfam America incurs as a result of accepting money for a specific set of deliverables from a donor agency, such as a philanthropic organization. It is a time-bound commitment to deliver on a set of concrete and measurable changes. It often entails a contract between organizations, a grantor and grantee. All projects require ongoing monitoring, reporting, and learning, and all Oxfam grantees are required to submit final reports documenting successes and challenges. “Project evaluation,” however, is more than this. It happens near the close of the project, involves external evaluators to control for bias, and generally focuses on questions of efficiency (outputs compared to inputs such as money and staff) and attribution of outcomes to project interventions. In rights-based approaches to evaluation, it includes primary change agents themselves – to challenge and/or confirm the grantee’s and evaluator’s own understanding of accomplishments. See Project, Program, Evaluation.

Evaluation, Program A program is a 10-15 year effort focused on a clear and measurable impact goal. A program will be made up of many kinds of activities. Program Evaluation is a periodic assessment – every three years in ROPE II – that analyzes that set of activities and determines to what extent, together, they have led to planned results (objectives, outcomes, impacts) and whether our theory of change is accurate. Program evaluation requires the participation of external researchers, who lead the evaluation team in order to minimize the possibility of OA staff bias affecting the evaluation findings. Program evaluation explicitly questions and tests the relevance and appropriateness of our understanding of primary change agents, our impact goal, our theory of change, and the strategies we’ve put in place to achieve long-term impact. Participation of primary change agents in all phases of program evaluation is essential. See Program, Project, Evaluation, Objective, Outcome, Impact.

12

Evaluation, Rights-Based Whether for a project, a program, or something else, evaluation approaches can take many forms. ROPE calls for rights-based monitoring and evaluation. An evaluation is suitably rightsbased if: e. it makes changes of power relations and their measurement explicit and evaluable f. it builds the capacities of primary change agents and partners themselves to think critically about the changes going on around them g. it affords primary change agents and partners a real opportunity to hold Oxfam and other partners accountable h. it includes specific and robust public transparency mechanisms and processes with regard to disseminating evaluation findings Goal A goal is a statement of intended impact. It specifies the extent, scope, and depth of this impact.

A goal specifies the changes in the lives of the primary change agents that we hope to facilitate over the course of a 10-15 year program commitment. While setting an appropriately ambitious impact goal is more art than science, goals should be consistent with the resources we think we can bring to bear on the issues at hand, the skills and competencies we can harness, the political economic context, and the kinds of partners available to work with us. A goal is evaluable, even though it may or may not be purely quantitative. In ROPE’s planning process, field teams will establish a series of impact benchmarks – for three, six, and nine years at a time – and these represent the concrete progress towards the goal that we hope to accomplish in those time periods. See Objectives, Impact.

Impact Impact can be defined as significant, sustainable change in power relations that enable excluded and marginalized people to assert and realize their rights to access, control, and manage resources, institutions, decision-making processes and knowledge, leading to improved wellbeing. Compare to Outcome; Output.

Monitoring Monitoring generally refers to tracking the progress of work against plans in order to enable timely corrections and adjustments. It is also the continuous scanning of the external context for dynamics that are relevant to programs. In monitoring, the focus is on information about activities, procedures, resources and outputs – aggregation and comparison of data and information. Monitoring plays an internal accountability function in ensuring compliance with standards, facilitating cost efficiency, prudent use of resources, and financial accountability.

Cumulative monitoring information supports project and program evaluations. See Evaluation.

Outcomes Outputs lead to Outcomes. Outcomes are the changes in relationships, attitudes, behaviors, ideas, policies, values, practices, institutions, human conditions, and enabling environments as a direct or indirect consequence of a set of actions and their outputs. We always need to ask ourselves the question whether or not such outcomes are sustainable, however. The conditions for sustainability may not be fully met until later in the program, years after a particular outcome has been achieved, when we have altered the power relations that led to such rights denial in the first place. It is more difficult to make direct attribution of such changes exclusively to Oxfam or any individual change agent. Rather, we may be able to assess our contribution towards an outcome, and progress towards a desired outcome. See Outputs.

Outputs Outputs are the immediate results of activities. These are usually tangible, such as numbers of people trained, number of meetings attended, signatures collected, events organized, etc. They are easily measured and attributable. Quantifying and characterizing outputs is an important part

13

of project/grant monitoring. See Outcomes.

Program A program is a set of strategically aligned, mutually reinforcing interventions – some carried out

by Oxfam America, but most by others – that creates sustained, positive changes in both the material conditions and social positions of vulnerable people. A program focuses on a specific population that shares common issues and is experiencing similar rights denial. A program has a measurable 10-15 year impact goal and comprises a coherent combination of much shorter-term kinds of work. A program unites shorter term work via a common set of indicators and measures, a long-term learning and knowledge sharing strategy, and a consistent community of actors who meet regularly to assess performance, change strategy, and allocate resources. See Project.

Programming By programming we mean program design, development and implementation of a program strategy, and the coordination of the interplay among the various program components, with an embedded framework for evaluation and knowledge management.

Project A project is almost always focused on outputs and outcomes, not impacts. It is almost always going to be short term (between 1-5 years). Oxfam’s grants to partners are, in this way, projects.

Likewise, when Oxfam receives a grant, such as from the Gates Foundation, that is a project.

Projects frequently have a contractual element to them: one organization gives specific funds to another, and the other is accountable for those funds. Projects are an important unit of analysis and planning for OA because they need to be carefully aligned with programs so as not to draw us off course. While projects may be organized relatively independently for purposes of funding and management, their organization under a common program is important and modifies their M&E systems and internal management. If we do this well, projects will be strategically linked (“roped”) and be strengthened by the synergy of mutually reinforcing effects. See Program.

14